Page 1

2015-03-01 1 Evaluation of Quality OptimiZer TM software (Awesome Numbers Inc.) to simplify application of CLSI EP 23-A, minimize patient risk and reduce clinical cost Zoe Brooks 1, Raymond Gerz 2, Nathalie Lepage 3, David Plaut 4 1,2. AWEsome Numbers Inc. 3. Children’s Hospital of Eastern Ontario 4. David’s Consulting Chicago, 2014 Poster presented Objectives

  1. to evaluate the efficacy of Quality OptimiZer TM software to assess analytical process quality, verify clinical effectiveness of Q.C. processes and recommend a comprehensive Q.C. strategy, .
  2. to quantify the impact of OptimiZer Q.C. processes on patient risk, clinical and laboratory costs, and .
  3. to compare Quality OptimiZer reports to EP 23 recommendations . Relevance: Ineffective quality control practices expose patients to the risk of incorrect or delayed diagnosis and/or treatment. Both CLSI EP23 and CMS Individualized Quality Control Plan (IQCP) require labs to “ensure test result quality is appropriate for clinical use;” validate “the ability of the QC procedures to detect medically allowable error;” and assess “potential costs both in terms of the patient’s well-being and financial liability.”

Page 2

2015-03-01 2 Analytical Process Quality Q.C. Process Strategy (IQCP) Methodology: . We examined analytical processes, Q.C. processes, patient volumes and costs from five analytes, from two instruments in two laboratories. For each Q.C. sample, we gathered 7 ‘stats’ (verifiable facts) 5 6 7 1 2 3 4 Determine Analytical Process Quality Determine Q.C. Process Effectiveness Sodium November 2013 Level 2 Measured Mean 145.3 Measured SD 1.06 True Value 144.8 Medical Goal 4.0 Assigned Mean 146.0 Assigned SD 1.46 QC ‘Stop & Act Rule’ 1-3s 4 Numbers determine analytical process ability to meet medical goals defined by the Lab Director 1 2 3 4 4 Green BarZ show the current measured mean & SD Red BarZ show distribution if 5% of results exceed the medical goal (Tea) 3 4 Performance BarZ© ME = Margin for Error = (TEa-TE)/SD = How close to failure = Zero ME means STOP!

Page 3

2015-03-01 3 Quality OptimiZer TM simulates a clinically significant change Current Performance Simulated 5% Failures After the simulated shift , all patients are at risk; 5% will receive medically-invalid results. Patients will be at risk (or harmed) until QC flags the error and staff stop reporting until acceptable performance is restored. Quality OptimiZer TM recommends and evaluates an individualized 5-part QCP to minimize the number of patients exposed to risk if the method fails. 3 Numbers determine Q.C. process effectiveness 1 2 2 5 6 6 Lab QC Mean, SD & Rules OptimiZed QC Mean, SD & Rules Laboratory QCP Assigned mean & SD OptimiZed Q.C. Measured mean & SD 7 IQCP impacts patient risk & cost 6 rejects after clinical failure 1 warning after clinical failure

Page 4

2015-03-01 4 The root cause of ineffective QC Mean Measured Assigned Chart mean is 0.7 SD < meas mean (0.7/ 1.06 )145.3 146.0 SD 1.06 1.46 Chart SD is 1.38 x Meas SD (1.46/ 1.06 ) Actual QC Rule (148.5 -145.3)/1.06 Assigned QC Rule (150.4 -145.3)/1.06

  • 3 SD Flag at 148.5 150.4 3.0 4.8
  • 2 SD Flag at 147.4 148.9 2.0 3.4 Mean 145.3 146.0 0.0 0.7 -2 SD Flag at 143.2 143.1 -2.0 -2.1 -3 SD Flag at 142.1 141.6 -3.0 -3.5 51 62 5Arbitrarily assigned Q.C. chart values are applied on the QC Chart. What staff perceive as a ‘1-3 s rule failure’ does not occur until a point falls 4.8 meas SD from the meas mean. 4.8 meas SD Assigned Reject Rule Quality OptimiZer’s Risk-Based Quality Grade TM evaluated both the analytical process quality and QC process effectiveness Instr… Instr… Instr… Instr 9 Instr 8 Instr 7 Instr 6 Instr 5 Instr 4 Instr 3 Instr 2 Instr 1 A A- B B- C The R-B-Q Grade TM provided a quick summary for each QC sample, and could be summarized to represent multiple instruments or sites. A D FCB A D FCB

Page 5

2015-03-01 5 Discussion

  1. Error detection is impeded by the common practice of assigning mean and SD values from inappropriate sources, and failing to individualize QC processes in response to analytical process quality.
  2. Laboratory quality would benefit from increased staff focus on clinical quality and the interaction of the seven numbers that drive and assess quality.
  3. OptimiZed Q.C. processes were consistently more effective to detect clinically-significant error.
  4. The R-B-Q Grades , star ratings, and recommended action flags are useful to guide consistent practice. Conclusions Quality OptimiZer reports satisfied EP 23-A recommendations: . /head2right ensures test result quality is appropriate for clinical use /head2right provides suggestions on the frequency of QC sample procedures to have a high probability to detect a stated medically allowable error. /head2right at the least, provides the ability to check if QC procedures can detect medically allowable error. The QC strategy using QC samples should include the following for each method: /head2right The frequency of QC sample test events /head2rightThe type and number of QC samples tested per test event /head2rightThe statistical QC limits used to evaluate the results /head2rightThe frequency of periodic review for detecting shifts and trends /head2rightThe actions taken when results exceed acceptable limits Decisions are based on evaluation of potential costs both in terms of the patient’s well-being and of financial liability of the treating parties vs benefits to the patient. Information: zoe@awesome-numbers.com Download poster