Page 1
Critical systematic error supports use of varied QC rules in routine chemistry. Authors: Z. Brooks, Q.I.K. Quality Is Key Ltd.; D. Plaut, Dade-Behring; Conrad Begin, Aline Letourneau, Cochrane Region Laboratory Program Methodology: We analyzed output from Quality Advisor software, © Q.I.K. Quality Is Key Ltd., from 6 Vitros 250 analyzers (Ortho Clinical Diagnostics) in 6 hospital laboratories from October 1998 to August 1999. Target values and TEa limits were specified for each level of each control for each test on each instrument. Target values were based on peer comparison data. TEa limits were specified by the laboratory director to meet clinical requirements. Mean and SD values, generated each month from routine daily QC data, were entered in real time into Quality Advisor. Quality Advisor calculated ∆SEc as ( [(TEa - |Bias|)/SD] – 1.65) and recommended QC rules based on ∆SEc and method stability. For more information, please contact Zoe C. Brooks through UMDNJ WebCT 3 700 600 500 400 300 200 100 0 0 2 5 7 9 11 14 16 18 21 23 25 28 30 32 34 37 39 41 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 5 Critical systematic error supports use of varied QC rules in routine chemistry. ^SEc Vitros 250 23 Tests Oct ‘98 to Aug ‘99 6 Labs AACC 2000 Most controls can shift well over 3 SDs before exceeding the defined quality requirement AACC 2000 Objective: We studied critical systematic error (∆SEc) on monthly QC summary data to evaluate the range of ∆SEc values observed and the efficacy of modifying QC rules based on ∆SEc and method stability. Relevance: International authorities recommend selecting quality control rules to match the varying performance of each analytical method. ∆SEc relates method accuracy and precision to the performance requirement (target value and total error allowed (TEa)) of each control. ∆SEc provides a numerical measure of the number of SD’s a control mean may shift before more than 5% of results will exceed specified error limits. Previous posters and publications claim improved efficiency in QC by selecting QC rules based on ∆SEc and method stability. For more information, please contact Zoe C. Brooks through UMDNJ WebCT 2 Validation: This study examined ∆SEc values and QC rule recommendations on 2712 quality control summary results from 23 routine chemistry tests in six hospital laboratories over 11 months. Data were exported from Quality Advisor to Microsoft Excel and SPSS to create summary graphs. Conclusion: Over 75% of monthly control results for these 23 routine chemistry methods showed SEc values above 3.0. It is appropriate to select QC rules to match ∆SEc and method stability. For more information, please contact Zoe C. Brooks through UMDNJ WebCT 4 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 6 1. 1. Most laboratories use an internally-defined mean and SD to decide each day if each QC data point is “OK” 2. 2. “Performance- Driven Quality Control” selects a QC strategy to maintain current actual method performance within a defined quality requirement © Z. Brooks 1
Page 2
For more information, please contact Zoe C. Brooks through UMDNJ WebCT 7 ∆SEc = # of SDs the mean can shift before >5% of data will exceed TEa For more information, please contact Zoe C. Brooks through UMDNJ WebCT 9 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 11 Critical systematic error supports use of varied QC rules in routine chemistry. 2. 2. Normal distribution assumes 90% of data fall within +/- 1.65 SD of the mean. We subtract 1.65 SD (C) so <5% of data exceed the upper (or lower) TEa limit A C B QC Strategy Tables simplify rule selection * * * *
- Calculate method performance as ∆SEc, (maximum acceptable shift in the mean)
- Categorize method stability based on historical information for each test
Select rules to maintain method performance within defined quality requirements 1. 1. If the mean shifts from its current bias (A) all the way to the TEa limit (B), half the data will exceed the quality requirement
-
- Select the single rule or multi-rule combination based on the calculated ∆SEc for each test/control/ level
-
- When indicated, apply additional QC strategies to increase error detection AACC 2000 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 8 Select appropriate QC rules based on Choose QC Rules to match ^SEc and Method Stability (Significant Errors) 1-3.5s If the mean can shift many SD’s, use a 1-3.5s Rule Use the 10x and 4-1s Rules to detect small shifts Many Small 10x For more information, please contact Zoe C. Brooks through UMDNJ WebCT 10 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 12 ∆SEc to detect shifts in the mean Larger shifts in the mean are easier to detect
-
Specify the “Target” for each control 2. 2. Set a maximum acceptable variation from that target: allowable Total Error limit (TEa) to maintain clinical and proficiency requirements for your institution 3. 3. Calculate a valid mean and SD False Positives Shift Detected QC Rule 1-3 s Many Small 1-2s 1-2.5s Moderate Moderate 1-2.5s 1-2 s 4-1s Few Large 1-3s 10x Rare Large 1-3.5s Mod/many Small 4-1s
-
- The graphic shows mean shifts of 1, 2, 3 and 4 SD
-
- Note the proportion of data above the 2SD and 3SD limits on the QC chart with each shift in the mean 3.3. Larger shifts will generate more QC flags © Z. Brooks 2
Page 3
Z-Bars© show accuracy, precision and ∆SEc… … that you can see at a glance Goal Limit Limit For more information, please contact Zoe C. Brooks through UMDNJ WebCT 13 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 15 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 17 Critical systematic error supports use of varied QC rules in routine chemistry. ∆SEc is the maximum mean shift before the method exceeds TEa ∆SEc varies from lab-to-lab, using the same controls and instruments Method performance is shown graphically Accurate methods are centered about the target Precision is indicated by the width of the Z-Bar Position indicates Bias Z-Bars are color coded like traffic lights: Red = problem * * * * * *
- 1. Z-Bars© (©) show method accuracy and precision relative to the defined target and TEa limit for each control
-
Z-Bars© vary with the data distribution (mean and SD); Tests/controls with low ∆SEc are close to the TEa limit
- 1. The top Z-Bar© shows a method with an ∆SEc of approximately 8. A shift of this magnitude would easily be detected with a 1-3.5s QC rule
-
The other controls can shift 4, 2 and zero SDs. These require closer monitoring with a 1-3s, 1-2.5s, 1-2s or multi-rules; 1(*) needs corrective action. * Target for all labs is 2.95 TEa for all labs is 0.2 ∆SEc SD Mean Lab 3.88 .036 2.95 AG 2.61 .031 2.88 BM 0.39 .027 2.81 HP 6.43 .024 2.96 LM 5.98 .020 2.90 SB 3.56 .029 2.90 SR AACC 2000 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 14 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 16 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 18 Four key numbers indicate method performance ∆SEc and Z-Bars© vary with method accuracy and precision ∆SEc varies from time-to-time within the same laboratory
- 1. The target may be the historical average or peer mean
-
The TEa may be the CLIA limit or a clinically acceptable variation 3. 3. The mean is a current calculated value 4. 4. The SD must represent actual method performance • SD’s assigned from other sources are misleading
- 1. The top Z-Bar© has the same bias as the top Z-Bar© in the previous example, but a higher SD. The ∆SEc in this example is only 2, instead of 8 when the SD was lower.
-
∆SEc = 3. [(TEa-|Bias|)/SD]-1.65 4. 3. ∆SEc combines the 4 key numbers Target for all months is 142.3 TEa for all months is 5.0 ∆SEc SD Mean Date 2.75 1.07 142.0 Sept * * * * * * * 3.83 0.87 142.1 Aug 4.43 0.68 143.2 July 0.18 2.02 141.0 June 0.91 1.40 140.9 May 3.17 0.87 141.5 April © Z. Brooks 3
Page 4
35.0 30.0 25.0 20.0 15.0 10.0 5.0 In Dec. 1998, this method could shift almost 25 SDs. A 1-3.5 s rule would easily detect this. 0.0 Critical systematic error supports use of varied QC rules in routine chemistry. Vary QC rules to match current method performance ^SEc AST Mean of 37 Mean of 115 When the ∆SEc is below 3, a 1-2.5s or 1-2s rule is required, with visual monitoring AACC 2000 When ∆SEc is low, use more or “tighter” rules and watch closely 5.0 4.0 3.0 2.0 1.0 0.0 ^SEc Sodium Mean of 142 Mean of 119 ∆SEc = 4.3; Use a 1-3.5s single rule or 1-3s/(4-1sW) ∆SEc = 1.9; Use a 1-2.5s single rule or multi-rule of 1- 3s/2-2s/R-4s/4-1-s; visually monitor the chart . For more information, please contact Zoe C. Brooks through UMDNJ WebCT 20 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 21 With ∆SEc of 4.0, a shift of 4.0 SDs will cause 5% of points to exceed TEa. This shift is easily detected with a 1-3.5s rule. Simulated shift to ∆SEc limit; 5% of data will exceed TEa For more information, please contact Zoe C. Brooks through UMDNJ WebCT 23 The 1-2s rule produces 5% false positive QC flags 15 “1-2s warnings” with no points > TEa SD = 0.6 No QC Flags when no errors present
- The QC chart and Total Error Index (TEI©) graphs above show 300 data points for Sodium with a mean of 119 and SD of 0.6
- The TEI graph shows that all points are well within the target of 119 and TEa limit of +/- 4.0. 1-3.5s QC Flags are seen as soon as significant change occurs. For more information, please contact Zoe C. Brooks through UMDNJ WebCT 22 With ∆SEc of 1.9, a shift of 1.9 SDs causes 5% of points to exceed TEa. This shift is seen visually, and detected with a 1-2.5s rule. Simulated shift to ∆SEc limit; 5% of data will exceed TEa For more information, please contact Zoe C. Brooks through UMDNJ WebCT 24 A QC Chart does not assess performance against requirements 15 “1-2s warnings” with 11 points > TEa No QC Flags when no errors present SD = 1.0
- The QC chart above, with a mean of 119 and SD of 1.6, looks virtually identical to the chart with SD of 0.6.
- The TEI graph shows that some points exceed the target of 119 by > the TEa limit of +/- 4.0.
- With this low ∆SEc of 0.9, we need to use the 1-2s rule as a reject signal QC Flags and a visible shift are seen when significant change occurs. © Z. Brooks 4
Page 5
Most tests average ∆SEc above 3.0, and can be controlled with a 1-3.5s rule 2. 2. Most tests occasionally approach ∆SEc of zero – at the TEa limit - requiring tighter QC rules, increased visual monitoring and corrective action. TEST 50 40 30 20 ^SEc on Actual SD 10 0 -10 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 25 50 40 30 20 ^SEc on Actual SD Other tests (e.g. Potassium) usually have a low ∆SEc and require close monitoring to meet clinical and proficiency requirements TEST 10 0 -10 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 27 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 29 Critical systematic error supports use of varied QC rules in routine chemistry. ∆SEc varies over time for Ca-T BI-T AST AMY ALP ALB Variations in ∆SEc require * K LI LDH Good QC strategy reduces false positives and false negatives all tests in all labs changes in QC strategy * PHOS THEO NA PROT URAT TRIG UREA LEVEL 1.00 2.00 LEVEL 1.00 2.00 1. False positive QC flags are reduced by selecting QC rules such as 1-2.5s, 1- 3s and 1-3.5s when ∆SEc is high. 2. False negative QC flags are reduced (error detection is increased) by using the 1-2s rule as a reject and visually monitoring charts when ∆SEc is low. Some controls on some tests (e.g. Phosphate) can shift over 20 or 30 SDs before exceeding TEa limits. AACC 2000 16 14 12 10 8 6 ^SEc on Actual SD 4 2 0 -2 TEST For more information, please contact Zoe C. Brooks through UMDNJ WebCT 26 80% of routine chemistry tests could use a 1-3.5s QC rule For more information, please contact Zoe C. Brooks through UMDNJ WebCT 28 QC rules selected by ∆SEc save Previous studies by Q.I.K. show reduced QC flags in other laboratory departments. 350 300 250 200 150 100 50 0 For more information, please contact Zoe C. Brooks through UMDNJ WebCT 30 ∆SEc varies between control levels GLU GGT CRE CO2 CL CK CHOL money in laboratories Number of QC Flags 327 327 down to 107 Saved $13,200 107 Immunoassay Haem/ Coag Chemistry Current System QC Rules to Match Quality Goals 286 70
- 1. TEa limits and method performance (accuracy/ precision) vary between control levels.
-
If one control level is closer to the TEa limit, use tighter QC rules to “watch it more closely”. LEVEL 1.00 2.00 82 35 1.
- Of the 2711 controls monitored in 12 months in 6 labs, 2167 had ∆SEc above 3.0 and could use a 1-3.5s QC rule.
- Only 93 tests required a 1-2s rule and performance improvement.
- Use of a 1-2.5s, 1-3s and 1-3.5s QC rule reduces false positive QC flags.
- In our study, 2618 controls did not require a 1-2s rule.
- At 30 runs per month, these controls represent 78,540 QC data points.
- A 5% false positive rate from the 1-2s rule would have produced 3,927 flags.
- At $10 per repeat/QC investigation, this represents $39, 270.00 © Z. Brooks 5
Page 6
We hope you enjoyed this presentation on QA™ & Z Bars™ are produced and registered by Zoe Brooks Quality Consulting and Q.I.K. Quality is Key Ltd., Worthington, Ontario, Canada. Some slides in this presentation were reproduced with permission For more information, please contact Zoe C. Brooks through UMDNJ WebCT 31 Critical systematic error supports use of varied QC rules in routine chemistry. and please remember… AACC 2000 The End © Z. Brooks 6