Page 1

C L I N I C A L I S S U E S Use some horse sense with QC By Zoe Brooks, ART (Canada) and Carol Wambolt, BS (Chemistry) T he complex world of laboratory quality control (QC) could use a dose of good old-fashioned horse sense. While this may seem an odd cor­ relation, many valuable lessons that result from handling and training horses can be applied to lab QC. The serious gap between quality-control theory and prac­ tice is largely due to a lack of clear com­ munication and the failure to implement concepts developed by experts who agree on the need for performance standards and fundamental requirements for good QC practice. Training horses requires taking the time to understand what makes a horse tick at a very basic level. The trainer then clearly communicates to the horse exactly what he wants. Similarly, a lab QC process can­ not be designed and implemented unless fundamental concepts like accuracy and precision, and the crucial need for perfor­ mance standards are first understood and then communicated to staff. Innumerable bruises, sprains, breaks, and concussions from horses have taught trainers that crystal-clear communication is essential if they want a horse to do what they suggest. Likewise, each person in the lab must be told exactly what to do to en­ sure that every result leaving the laboratory meets required performance standards. Look closely and critically at the laboratory’s quality-control processes. Are the methods being used meeting with the approval of the experts or with CLIA requirements? A straightforward five-step performance-driven QC process presented here can implement recommended QC practices. The need for performance standards The use of performance standards for labo­ ratories was first proposed over 40 years ago. Just what are performance standards? Performance standards define that the ball bearings or cornflakes or coffee cups or gizmos or widgets produced from a specific process for a specific client must be between “x” and “y” units of measure to be acceptable. Performance standards differentiate what is “okay” from what is “not okay.” It makes sense, then, that laboratory tests would use the common concept of performance standards. Ex­ perts and regulators recommend that labs set performance standards that define “if a patient sample has a true value of x units, reported results must be within x units or y % of that value.” Contrary to expert recommendations, however, most front-line workers we have met — not to mention a fair number of senior staff, inspectors, and proficiency- testing officials — still believe that “qual­ ity is good as long as all results are within ±2 SD.” This makes no sense. If the lab’s only performance standard is to be within ±2 SD, its workers may as well say, “whatever I get is okay.” Just because a lab’s QC results are within ±2 SD does not mean they are good. It just means they are the same as they were. What QC practices do experts recom­ mend? If we were to browse any of the more note­ worthy scientific journals today and read articles pertaining to laboratory quality Continues on page 30 28 March 2007 ■ MLO

www.mlo-online.com

Page 2

C L I N I C A L I S S U E S C L I N I C A L I S S U E S Figure 1. Composite of a QC process modeled after expert recommendation, showing the process flow for a performance-driven QC strategy. 3 Yes © Z. Brooks 2007 Design/modify and implement daily QC processes to detect significant change. 4 QC flags or chart examination indicate change in accuracy or preci- sion of daily QC? 1 2 Performance-driven quality control Define performance standards; select candidate method(s); perform validation studies. Method performance (e.g. accuracy, precision, total error) meets performance standards? No Yes Report patient and proficiency data No 5 Validate data used in assessment: mean, SD, True/target, TEa, daily & summary QC Investigate cause of failure to meet perfor- mance standards; Improve or replace result(s); method. control, certain names would appear time and time again: James Westgard, Callum Fraser, George Klee, Ronald Laessig, R. Neill Carey, Carmen Ricos, and many more. Their recommendations seem to parallel one another in most regards: James Westgard: “I think the main ingre­ dient that is missing in laboratory quality management is ‘quality’ — what is needed and how do you know you are achieving it in everyday operation. We need to make our quality practices quantitative and objective, guided by what is needed for patient care.”1 Callum Fraser: “For about 40 years, there has been a steady stream of publications concerned with the generation and ap­ plication of quality specifications. There appeared to be a real conflict about how to set quality specifications, but a decisive recent advance was that a consensus was reached in 1999 on global strategies to set quality specifications in laboratory medicine.”2 George Klee: “A smooth-running system requires careful design and appropriate management. The catchy phrase with five ‘Ps’ is worthy of attention: ‘Purposeful Planning Prevents Poor Performance.’”3 CLIA regulations repeatedly call for clearly defined Performance Specifica­ tions, that they define as “a value or range of values for a performance characteristic, established or verified by the laboratory, that is used to describe the quality of pa­ tient test results.” The experts seem to agree that a cor­ rectly implemented quality-control system requires careful planning that is guided by what is needed for patient care. Step 1: Define performance specifica­ tions for each analyte a. Performance specifications are usually set as total allowable error (TEa) limits. They define the level of quality needed to meet the needs of your patients. TEa limits specify, “If a patient sample has a true value of ‘x’ units, reported results are okay if they are within ‘y’ units or ‘z’ % of that value.” b. The TEa limit may be the CLIA limit, but it is better to select the specification based on the 1999 Stockholm confer­ ence.3 From a practical perspective, the goal is to set performance specifications that are both defensible and attainable. c. Select the TEa that is highest in the hierarchy4 (defensible) — and that the 30 March 2007 ■ MLO

www.mlo-online.com

Page 3

lab staff believes it can meet (attain­ able). Start by choosing the highest goal, usually biological variation, and then compare that value to the current observed total error (TE). If the lab can­ not meet the biological variation goal, then staff should select CLIA or other defensible choices. Step 2: Ensure performance standards are met before reporting any results a. Calculate total error (TE). In theory, this is simple. Total Error = [abs(Bias)

  • 2 SD]. (Labs can and should use other values like 1.65 or 1.96 instead of 2, but let us keep this simple.) b. Bias is simply the current measured mean value minus the true value for a QC sample. [Our preference is to use the QC peer mean as the true value. ] (From over 10 years’ practical application, we have found that the true value based on the peer mean in the first month or two of a new chemistry control will almost always remain unchanged for the full life of that control sample.) Many people find it difficult to grasp that a QC sample has a single true value that does not change. Yet, if the lab aliquotted a patient sample with a true value established at a refer­ ence laboratory, that true value would not change as it was tested over time. The true value of a proficiency sample does not change. While it is common to see a change in the measured mean value of QC samples when reagent lots are changed or when recalibration takes place, that does not mean that the true values changed! The measured mean tells “what you got” for a sample. The true value is “what you should get.” Figure 2: Illustration of concept of margin for error www.mlo-online.com c. Only three numbers are needed to calcu­ late total error: the true value plus the current mean and SD from a single data set (one reagent, calibration, sample). Take care not to calculate the mean and SD on multiple data sets. This practice affects the mean slightly and may cause the SD to appear five or more times larger than it is for a single data set. It works well practically to calculate a mean from as few as six to eight data points and to use the usual or historical median SD. Step 3: Design, modify, and implement daily QC processes to detect signifi­ cant change a. First, select QC samples carefully. Statistical laboratory quality control relies on the founding premise that QC samples mirror patient samples. (Yet, we have seen significant shifts in the entire patient population with no cor­ responding change in QC samples.) In theory, QC samples should reflect medi­ cal decision points. In practice, there is little consensus on medical decision points, and QC samples may not monitor appropriate levels. b. Next, select a QC strategy consisting of frequency of testing, QC rules, and how-tos of creating and examining QC charts. Choose a QC strategy that will detect changes that would cause results to fail to meet the defined performance standard. This is not as complicated as it sounds. The key lies in knowing the mar­ gin for error (ME), similar to Sigma or Critical Systematic Error (SEc). The ME can be calculated from four key numbers Continues on page 33 Q C Visit www.rsleads.com/703ml-038

Page 4

shown in Figure 2: The True Value (1), TEa limit (2), current measured Mean, (3), and SD (4). The Margin for Error is simply the total allowable error minus the total error. Divide it by the SD to know how many SD the mean can shift before results will exceed allowable er­ ror limits. Once the size of shift you need to detect is known, then appropriate QC rules and strategies can be chosen. If the margin for error is small, watch closely and use tight QC rules. If the margin for error is large, check the chart weekly and use broader QC rules. (If you wish to automate this process, there is free software you can download.)5 The experts seem to agree that a correctly implemented quality- control system requires careful planning that is guided by what is Step 4: Do QC flags or chart examina­ tions show change? Test QC samples. Plot all results on QC charts. Apply rules. Examine charts. Pe­ riodically consolidate individual results from a specific data population (process, reagent lot, calibrator, time period) to reflect overall method accuracy and precision. a. If there are no QC flags, report patient results. b. If QC flags indicate that the accuracy or precision of the method has changed, calculate TE and ME with the mean and SD of the current data population. If the changed analytical process is still within TEa limits, adjust the QC process based on ME and carry on. Step 5: Investigate and correct problems If the lab’s method no longer meets performance standards (TE > TEa), stop reporting results while staff makes sure the numbers are correct and take corrective action, if indicated. Summary What do we need to do to improve Lab QC practice? Maybe it is time to take a lesson from the folks with horse sense. Recog­ nize there is a problem. Look closely and critically at QC processes. Help each other improve by benchmarking performance against performance standards and sharing best practices. Begin to change practices www.mlo-online.com needed for patient care. and share stories to encourage others. When a mistake is made when training or riding a horse, repercussions are immediate and personal. But a mistake made in the design or application of quality control means the patient and physician suffer the pain. l Zoe Brooks, a Canadian Advanced Registered Technologist, is a QC consultant, author, teacher, and software designer who reaches out to the world from her farmhouse in Northern Ontario, Canada. To help close the gap between QC theory and practice, Brooks sponsors a website at http://zoebrooks quality.com/harmonize where she hosts discussions and provides free software. An avid horsewoman, Brooks also developed a “better bitless bridle” for horses; visit www. nurturalhorse.com. Carol Wambolt, BS (Chemistry), is a chemist and a QC laboratory supervisor for a beet-sugar processor in Twin Falls, ID. She is currently a Master of Sci­ ence candidate in clinical laboratory science at Idaho State University in Pocatello, with Brooks as her project advisor. References

  1. American Association for Clinical Chemistry. October 2005 Mentor Q&A Session — Ask the Mentor, James O. Westgard, PhD. Available at: http://www.aacc. org/AACC/members/sycl/mentor/past_mentors/2005/ Oct_2005?Mentor_Oct05_Session.htm. Accessed February 15, 2007.
  2. Fraser C. Biological variation and quality for POCT. June
  3. Available at: http://www.bloodgas.org/7302137b- bb49-4ae3-ba26-b49a81088f65.W5Doc?track=poct. Accessed February 15, 2007.
  4. Klee G. Quality management of blood gas assays. June
  5. Available at: http://www.bloodgas.org/7302137b- bb49-4ae3-ba26-b49a81088f65.W5Doc?track=poct. Accessed February 15, 2007.
  6. Fraser CG, Kallner A, Kenny D, et al. Introduction: Strat­ egies to set global quality specifications in laboratory medicine. Scand J Clin Lab Invest. 1999;59:477-478.
  7. Allowable Total Error: Setting Performance Standards. [David G. Rhoades Associates, Inc. website.] July 29,
  8. Available at: http://www.dgrhoads.com/AQText. shtml. Accessed February 15, 2007. Suggested Reading Brooks Z, Plaut D, Massarella G. How Total Error Can Save Time and Money for the Lab. MLO Med Lab Obs. 1994;48-54. Brooks, Z. Quality control in theory and practice — a gap analysis. January 2006. Available at: http://www. bloodgas.org/f276fab5-a3b2-4030-8176-f3a5a8dc1be0. W5Doc?track=qa. Brooks, Z. I found the gap…it’s in the basement. April 2006. Available at: http://www.bloodgas.org/7302137b-bb49- 4ae3-ba26-b49a81088f65.W5Doc?track=poct. Brooks, Z. Quality control…the gap deepens. July 2006. Available at: http://www.bloodgas.org/7302137b-bb49- 4ae3-ba26-b49a81088f65.W5Doc?track=poct. Brooks, Z. What can you do to close the QC gap? October
  9. Available at: http://www.bloodgas.org/7302137b- bb49-4ae3-ba26-b49a81088f65.W5Doc?track=poct. Brooks, Z. Performance-Driven Quality Control. Washing­ ton DC: AACC Press. 2001. Brooks Z. Quality Control — From Data to Decisions. Basic Concepts, Trouble Shooting, Designing QC Systems. Edu­ cational Courses. Zoe Brooks Quality Consulting. 2003. Tonks DB. A study of the accuracy and precision of clinical chemistry determinations in 170 Canadian laboratories. Clin Chem. 1963;9:217-223. Westgard JO, Quam EF, Barry PL. Selection grids for plan­ ning QC procedures. Clin Lab Sci. 1990;3:271-278. Q C Visit www.rsleads.com/703ml-039