How total error can save time and money for the lab

Reduce unnecessary QC flags, avoid bad reagent lots, and solve problems quickly by comparing test performance with clinically defined total allowable error limits.

By Zoe Brooks, ART, George Massarella, M.D., FRCP(C), FRCPATH, and David Plaut

T’s HAPPENED AGAIN. You started a new lot of reagents ysterday and today all your quality control results are flagged because the numbers are too high. With a frown on your face, you stare at the analyzer wondering: Is it 0K to report these patient results? Should I stop and try a different lot of reagent? Should I recalibrate, or maybe change the mean? Maybe I should call the hot line or the local QC hotshot. Laboratorians are often faced with the dilemma of having an ana- lytical tst *stem show a shift in the mean or an increase in standard deviation (SD). These occurrences may follow a change in calibrators or reagent lot numbers or a service call on an instrument. Shifts can also happen for no apparent reason whatsoever. Quality control flags alert you to the change so you can investigate the possible error in the system, but the big question is: Do you consider

the test out of control until the prob- lem is resolved or accept the new level of performance? You can sim• plify this decision—and save time and money for your lab—by com- paring the new performance char- acteristics (bias and imprecision) with = total allow. able error (TER) limits.

Bias equals the measured value minus the true value, while total error (T) equals the bias plus SDs. The TEa is the maximum acceptable variation from true value

a Acceptable changes. In the past, many of us have considered a test

Bias

True

n

Brooks is technical coordinator and Messarella is laboratory director with the Cochrane Regional Laborato-y Program in Sudbury, Ontario, Canada. Plaut rs a scientific specialist with a firm,

48 Kwe•nbe 1994

Total error must be less than total error allowed


I s rumeniA

Instrument A (Sodium) Current mean Ta et (“true”,) mean Bias’at current-mean* Current SD . Total error (TE) (Ibiasl + 2SD) -Total etrorallowable (TEa TE/TEa Clinical status,

Month 1

130.0

Month92

130.0

132:o 130.0

: 110

0.25’ 4-•.’OK-•

o.5•o.

4.0

4.0 0.88,-

Initnjhehf B •‘(S8dium)7 Current mean Ta et (“true”) ‘mean

urrent Total + 2SD Totalerror allowable(TEa)

Clinical statusy. •

Month Month 27 *Month 3’

  • 130.0<

2.0 4.0

0K

3.0 ‘4.0

5.0

True 130

Instrument A

TEa 134

50 1994

True 130

Instrument B

134

I

TOTAL ERROR

system’s bias and imprecision as being separate entities. The terms we use to define acceptable in performance crite- ria are critical 55tematic error (SEc) and critical random error (REc). Critical systematic error is the maximum number of SDS the mean value can shift before the instru- men t ical random error is the maximum allowable increase in the standard deviation. Some labs view as acceptable a month-to-month shift in the mean of SD. In the real world, changö in ran• dom and systematic errors do not occur independently. Sometimes you must change reagent lots (and possibly introduce a systematic shift) on an instrument that is already experiencing precision problems. Today’s modern analyzers may yield such small SDS that you can shift much more than one SD before you exceed the clinical limits of acceptable performance. *One SD• may be much larger on an older instrument than on a new one. which rnakB it inappropriate to set one standard as the criteri• on for an acceptable shift in the

Q Solving dilemmas. You can sim- plify many of these daily problems by combining imprecision (SD) and bias (measured value target value) to calculate total error and then relating that TE to the clinical limits of for a particular control and n Igure l, eto error must e <TEa. The total variation from the target value is a combination of bias, which is the absolute value of (measured value true value) plus (1.96 x SD). For the purposes of applying this formula to the every. day monitoring of performance, you can simply estimate bias and add 2


  • Vt-I

IOTAL ERROR

Bias

True 130

Instrument A

Bias

True 130

Instrument B

Automated coagulation (APTT) Current mean

Target (“true”) mean Bias at current mean

Current SD

Total error (TE) (Ibiasl + 2SD) Total error allowable (TEa)

Clinical status

Lot 1

30.0

30.0

0.0

1.0

2.0

4.0

0 50

0K

52 u Nmembet 1994

TEa 134

TEa 134

5

Lot 2

28.5

30.0

-1.5

1.0

3-5

4.0

0.88

0K

Lot 3

27 .o

30.0

-3.0

5.0

1.25

Not 0K

SDs. We use 196 SDS in formal cal- culations because 95% of all data points in a Gaussian distribution fall within ± 196 SDS of the mean. To allow easy comparison. exam- ples shown here use the fornula TE

  • (Ibiasl + 2 SD). The following examples illustrate how total error can be applied in a laboratory set- ting: Example 1: Using TE to monitor monthly QC. You have two elec• trolyte analyzers (either within the same laboratory or at separate sites). You use the same control material to monitor sodium perfor- mance on both instruments. For the last several months, each instru- ment has produced a mean value of 130 mmol/L, which agrees with the interlab comparison testing of the same material. Instrument A has a consistent SD of 0.5 mmol/L, while instrument B has a consistent SD of 1.0 mmol/L The laboratory director has speci- fied a TEa of 4 mmol/L During month 2, you put a new lot of calibrator on instrument A, causing a shift in the control mean (see Figure 2). In month 3, you notice that instrument A develops a contamination problem that in• creases imprecision. In its second month, instrument B also develops a contamination problem that increases imprecision (see figure 3). In month 3, you run a new lot of calibrator on instru- ment B, causing a shift in the con- trol mean. Figure 4 illustrates the change from month I to month 3 for both instruments. Although both analyze ers had the same change in absolute bias and each SD increased by 50% from stable per- formance. instrument A remains within clinically acceptable limits while instrument B does not. During month 2. instrument A

4


experiences a shift in the mean of four SDs. By relating current perfor• mance to clinical TEa, you do not need to spend time and effort trou- bleshooting a change in perfor- mance that remains within IEa lim- its In month 3. however, the higher imprecision of instrument B maks the same absolute bias clinically unacceptable. figure 5 shows that test systems with greater impreci- Sion (larger SDs) cannot tolerate as much bias as more precise systems before exceeding TEa. Example 2: using TE to monitor new reagent lots. Your laboratory measures APIT on an automated coagulation analyzer. You use level 1 control and produce a stable mean of 30.0 seconds. You estab- lish a method bias of zero by com- paring on patient specimens to those that have been run by a referral laboratory. The stable SD is 1.0 second. lhe laboratory director specifies a IEa of 4 seconds. Over the course of several months. you make two reagent lot chang$ (see figure 6). By relating current to clinical TEa, you are able to detect an unacceptable shift that occurred over the course of the two lot changes. Such clinically important shifts can be missed if you examine each lot-telot variation as an isolat- ed event without comparing valuß from the new lot to a •true’ or •tar- get” value. Use the comparison of TE to TER as a tool to qualify new lots of reagents or calibrators prior to putting them into routine use. By identifying unacceptable lots, you can avoid significant errors in

Example 3: IJsing TE to select appropriate QC rules. You use two gas ana yzers to moni- Sta e tor p02. Both use the same control material (see Figure 7). Your cure rent practice is to apply all West•

Blöod gas analyzers (p02) Current mean

mean “Bias at current mean Current SD Total error (TE) (Ibiasl + 2SD) Total error allowable (TEa) TüTEa

Clinical status

Instrument A 175

170 +5

2.0

9.0

10.0

0.90

0K

Instrument B 168

170

-2

1.5 5.0

10.0

0.50

0K

True 170

Instrument A

TEa 180

TEa 160

Instrument B

True 170

  • se na ere xcee Pa. v@tÉiEV.

afiges_ n ea nd/or’émwo_ul

1994 53


ERROR

Muttirule Low stability QCSG

ASE

2.0s - 3.os

ASE

3.os

Moderate

High stability

13/22,/R4/41s

gard multirules to monitor each instrument The laboratory supervi- sor evaluates QC performance and recommends several possible ways in which to improve efficiency. The laboratory director specifies a TEa of IO mm Hg, Instrument B has better precision and accuracy than instrument Because the TE for instrument B is much less than the TEa, you need less stringent QC rules to insure that this system meets clinical require• ments. By using the TEa specified by the laboratory director. you can apply a Westgard Quality Control Selection Grid (see Figure 8) to determine that the recommended QC rules for instrument A are I-3s/2-2s/R-4s/4-ls. Since instru- ment B is more accurate and more precise than instrument A, however. this system only requires applica- tion of the I-3s rule with the ‘I—ls rule used as a warning. By selecting control rules appropriate to the per- formance Of each instrument, you can spend less time investigating false QC flags on test systems that

are performing well and concen- trate your efforts on test systems that are near the clinic.al TEa limits. Well-established idea. Both the theoretical concept and the calcula- tion of total error are well estab- lished. The examples shown in this article illustrate that the relationship of to TEa can be applied practi• cally in the clinical laboratory to monitor QC performance. qualify reagents and calibrators, and select appropriate quality control rules. In order to implement th$e prac- tices successfully, however, you must do the following: I. Set limits of TEa for each control. 2. Establish a “true” or target value for each control. 3. Calculate bias as (measured value target value). 4. Determine SD on a meaningful number of data points. 5. Calculate TE as (Ibiasl + 1.96 x SD). Steps I and 2 are the hard ones. Ilia limits must be clinically mean• ingful. You may set total error limits by asking clinicians to specify how

close to the true value the lab report must be at specific levels. You might also take TEa limits from literature references or base them on regulatory requirements (such as CLIA limits). limits must be control-level specific and may vary from site to site based on clinical practice The system used to assign TEa values within an individual laboratory must be sufficiently flexible to reevaluate and modify ttEe limits if Q Determining bias. In order to cal- culate TE you must know bias, In order to calculate bias you must specify the •true” (or target) value. You can estimate bias by compar. ing: • Patient specimens to the referral• laboratory findings • Control results to the published targets • Control results to a peer group that uses the same test procedure • Proficiency test results to target values Each of tlEe methods has inher- ent problems: a combination of may actualy be the most

Q Detecting error. For the last 2 years we have used the theory of total error in relation to total allow- able error, as described here, to asss the performance of six hospi• tal laboratories. This process has detected unacceptable changes in method performance during our monthly review of QC statistics. We have detected unacceptable re- agent lot numbers and replaced them before any unreliable patient results could be reported. We have saved significant time and effort by customizing QC rule selection to match the bias and imprecision of each test system. In our opinion. applying the theory of total error can save time and money tn today’s lab.

54 MIO “cvembo 1994