Page 1
Quality control design survey reveals 399 ways to do it wrong Zoe Brooks, AWEsome Numbers Inc. ((ElevateQC,) Toronto, ON, Canada). Abstract: Background: The effectiveness of statistical quality control processes to detect unacceptable patient risk depends on the choices that laboratory professionals make for the design of quality control charts and acceptable patient risk. Recommended best practice consists of putting a meaningful recent mean and SD on the QC chart, selecting rules based on the method’s margin for error to unacceptable risk or Sigma value, and basing acceptable risk on clinical need. Methods: A survey was circulated through LinkedIn and Labvine. Forty respondents who included their credentials were included in the final analysis. Respondents were asked to select the most correct answer to each of four questions regarding the parameters that govern the effectiveness of quality control processes.
- The best source of the mean on the QC chart? (Six choices)
- Best source of the SD on the QC chart? (Seven choices)
- Best way to select QC rules for each QC sample? (Six choices)
- If an analytical process “fails,” we should stop reporting before [errors are reported.] (Five choices) A total of 400 potential patterns of QC selections were chosen. (4 * 5 * 5 * 4). Results: When asked acceptable error rate if an analytical process “fails,” responses were: • More than 5% errors are reported - 11%. • Any errors are reported - 66% • It varies with clinical use of the test - 16% • It varies with capabilities of our method - 8% While most respondents chose at least one of the recommended answers for QC chart design and rules, only three chose the best choice for all parameters. Discussion: ISO 15189:2022, CLSI EP23 and other clinical lab guidelines require a risk- based approach to QC, requiring a lab to understand the clinical risks associated with a patient result with a significant error. Usually, this is still mitigated using QC samples intended to trigger a QC flag if their reported result deviates too far from the nominal value for the QC sample. Monitoring the QC measurements over time gives confirmation of the manufacturer’s performance criteria and local clinical need in terms of accuracy, precision and bias, but also any drift in any of these characteristics. Unfortunately, there is no agreed QC flag for ongoing QC monitoring, often with “2s” rules being applied, irrespective to the risk to patient if a measuring system shifts in response. (Rosenbaum et al3). Here we investigated the possibility that the findings of Rosenbaum were a result of insufficient understanding within the labs. Dr. Westgard also conducted a study (The 2021 Global QC Survey Results Are “In” - and “Out“ echoing this wide variety of QC practice.4 Similar studies by the author have shown the same inconsistent practice since 2005.5 References: The best source of the mean on the QC chart? 55% Package insert for QC sample Peer mean value Mean of a recent stable time period Mean established at method validation Best way to select QC rules for each QC sample? 33% 1-2s for all tests at all times 1-3s/2-2s for all tests at all times Westgard rules for all tests at all times QC rules differ by instrument; do not differ by analyte or time QC rules differ by analyte and time period within instruments based on sigma or margin for error A Disconnect Between Detectable Patient Error Rates and QC Understanding Best source of the SD on the QC chart? 25% Package insert for QC sample Peer SD value SD of a recent stable time period SD established at method validation Median or usual SD of several time periods On failure, we should stop reporting before: 15% More than 5% errors are reported Any errors are reported It varies with clinical use of the test It varies with capabilities of our method 30% A B C Points to Ponder: • Most labs do not implement best practice for quality control. • Although recommended by CLSI, most labs do not validate the effectiveness of quality control processes. • Best practice is difficult to find clearly-defined in literature. • QC practice affects ability of labs to detect shifts in accuracy and precision. Should QC software include verification of QC processes to detect the lab-defined acceptable error rate based on their selected mean, SD and rules? Is it time to consider a new approach to teaching, validating and certifying competency in medical laboratory quality control? This chart of a 4.67 sigma method has the recent measured mean and SD assigned. A 1-2.5s rule is applied to provide a 70% probability of error detection in the first QC run. This chart shows the same method with the SD assigned at 2x the measured value. The intended 1-3s rule is actually 6 measured SD above the mean. No results are flagged after the 3 SD shift. The mean here is assigned 1 SD below the measured mean.. The intended 1-3s rule is actually 4 measured SD above the mean. Three results are flagged after the 3 SD shift as 1-3s and there are two 2-2s flags.. 1 https://www.iso.org/standard/76677.html 2 https://clsi.org/standards/products/method-evaluation/documents/ep23/ 3 Rosenbaum MW, Flood JG, Melanson SEF, Baumann NA, Marzinke MA, Rai AJ, Hayden J, Wu AHB, Ladror M, Lifshitz MS, Scott MG, Peck-Palmer OM, Bowen R, Babic N, Sobhani K, Giacherio D, Bocsi GT, Herman DS, Wang P, Toffaletti J, Handel E, Kelly KA, Albeiroti S, Wang S, Zimmer M, Driver B, Yi X, Wilburn C, Lewandrowski KB. Quality Control Practices for Chemistry and Immunochemistry in a Cohort of 21 Large Academic Medical Centers. Am J Clin Pathol. 2018 Jul 3;150(2):96-104. doi: 10.1093/ajcp/aqy033. PMID: 29850771. 4 https://westgard.com/qc-applications/basic-qc-practices/964-2021-global-qc-survey-results.html 5 https://www.researchgate.net/publication/290607153_Benchmarking_Laboratory_Quality#fullTextFileContent