Page 1

The Benchmarking Study, 2001 zoe@awesome-numbers.org 1 Benchmarking Laboratory Quality © Z Brooks 2001 -2016 2 Many thanks are extended to the participating laboratories: Province/ State Cape Breton Healthcare Complex Sydney NS Children’s Memorial Hospital, Chicago Chicago IL Gamma-Dynacare Brantford Brantford ON Gamma-Dynacare London London ON Gamma-Dynacare Ottawa Ottawa ON Hamilton Health Sciences: Henderson Campus Hamilton ON Joseph Brant Memorial Hospital Burlington ON Kaiser Permanente Regional Lab Berkley CA Laboratorio Analisi Chimico Cliniche- Azienda Ospedale Careggi Firenze Italy London Health Services Centre (Westminster Campus) London ON London Health Services Centre (University Campus) London ON London Health Services Centre (Victoria Campus South Street) London ON MDS Laboratories Sudbury ON Phenix Regional Hospital Pheonix City AB St Paul’s General Hospital Vancouver BC Swedish Covenent, Chicago Chicago IL The Hospital for Sick Children Toronto ON Tillsonburg District Memorial Hospital Tillsonburg ON Toronto East General Hospital Toronto ON Toronto Medical Labs Toronto ON Whitehorse General Hospital Whitehorse YK Lab Name City © Z Brooks 2001 -2016 Update: January 2016. This study was presented at a Total Quality Management Symposium in Toronto in 2001. Nobody noticed. The issues highlighted here persist today. Risk Management concepts and software remove the complexity of all these statistics and present the same knowledge in a format that is easy to understand. Maybe now someone will notice. Zoe C. Brooks (It has not been published prior to posting at ResearchGate January 2016.) Zoe C. Brooks Zoe Brooks Quality Consulting 8070 Highway 17 West Worthington, On, Canada P0M 3H0 © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 1

Page 2

The Benchmarking Study, 2001 zoe@awesome-numbers.org 3 Many thanks are extended These laboratories have applied the “Performance- Driven Quality Control” process for two to ten years. © Z Brooks 2001 -2016 4 Bias = (Measured Mean – Target) 165 160 155 150 Measured mean 145 140 135 130 125 120 115 115 120 125 130 135 140 145 150 155 160 165 © Z Brooks 2001 -2016 Each dot = 1 QC sample from one lab/method for 1 month Ansen General Hospital Iroquois Falls ON Bingham Memorial Hospital Mattheson ON Hornepayne Community Hospital Hornepayne ON Lady Minto Hospitial Cochrane ON Notre Dame Hospital Hearst ON Sensenbrenner Hospital Sensenbrenner Hospital ON Smooth Rock Falls Hospital Smooth Rock Falls ON Fort MacLeod Hospital Fort MacLeod AB Pincher Creek Hospital Pincher Creek AB Taber Hospital Taber AB Crowsnest Pass Hospital Blairmore AB Cardston Hospital Cardston AB Lethbridge Regional Hospital Lethbridge AB to the control laboratories Sodium Study Group Control group Zero Bias Target Mean © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 2

Page 3

The Benchmarking Study, 2001 zoe@awesome-numbers.org 5 TEa Source by Department 250 200 Num ber of Controls 150 100 50 0 © Z Brooks 2001 -2016 6 7 6 5 TEa in absolute units 4 3 2 1 0 115 120 125 130 135 140 145 150 155 160 165 © Z Brooks 2001 -2016 Chem Coag Hema Sp Ch Biological variation Clinical CLIA QMPLS TEa = Criteria to measure effectiveness of a process Sodium Study Group Control group Each dot = 1 QC sample from one lab/method Target Mean © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 3

Page 4

The Benchmarking Study, 2001 zoe@awesome-numbers.org 7 TEa = Total Error allowed = Criteria to measure effectiveness of a process 30 25 20 TEa in Absolute Units 15 10 5 0 20 30 40 50 60 70 80 90 100 110 120 130 140 © Z Brooks 2001 -2016 Each dot = 1 QC sample from one lab/method for 1 month 8 CK TEa limits 350 300 250 200 TEa 150 100 50 0 0 200 400 600 800 1000 1200 © Z Brooks 2001 -2016 APTT Study group Control Group Linear (Control Group) Study Group Nov Control Group Target Mean Target © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 4

Page 5

The Benchmarking Study, 2001 zoe@awesome-numbers.org 9 Prothrombin Time TEa limits 10 9 8 7 6 TEa 5 4 3 2 1 0 10 15 20 25 30 35 40 45 50 55 60 © Z Brooks 2001 -2016 10 TSH TEa limits 12 11 10 9 8 7 TEa 6 5 4 3 2 1 0 0 5 10 15 20 25 30 35 40 45 50 © Z Brooks 2001 -2016 Study Group Control group Study Group Control group Target Target © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 5

Page 6

The Benchmarking Study, 2001 zoe@awesome-numbers.org 11 Platelets TEa limits 100 90 80 70 60 TEa 50 40 30 20 10 0 © Z Brooks 2001 -2016 12 Total Error (TE) combines random and systematic error. TE = |bias| + (1.96* SD) Target TEa Limit TEa Limit © Z Brooks 2001 -2016 40 90 140 190 240 290 340 390 440 490 540 590 Total Error is a metric that indicates how far results fall from the True/Target value Study Group Control Group Target © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 6

Page 7

The Benchmarking Study, 2001 zoe@awesome-numbers.org 13 Critical Systematic Error (SEc) measures accuracy and precision relative to performance standards. SEc = The Number of SD the mean can shift before >5% of results from the control exceed TEa Limits. Target TEa Limit TEa Limit © Z Brooks 2001 -2016 14 If method bias or imprecision is increased, the SEc will be decreased. Target TEa Limit TEa Limit © Z Brooks 2001 -2016 SEc is a metric that indicates how close a method is to TEa Limits SEc varies with Target, TEa, Mean and SD SEc SEc © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 7

Page 8

The Benchmarking Study, 2001 zoe@awesome-numbers.org 15 Study group One Bar = percent of QC samples at each TEa level. Below TEa line ‘passes’ © Z Brooks 2001 -2016 16 Control group Study group © Z Brooks 2001 -2016 Total Error as Percent - Sodium 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 7.0% 8.0% 9.0% 10.0% Total Error as Percent - Sodium 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 7.0% 8.0% 9.0% 10.0% TEa 3% (CLIA) Total Error as Percent of Target Study group Study group Control group TEa 3% (CLIA) 90% of 140 QC summary results from the control group met 3% TEa Total Error as Percent of Target 76% of 138 QC summary results from the study group met 3% TEa 76% of 138 QC summary results from the study group met 3% TEa 35 30 25 20 15 10 5 0 60 50 40 30 20 10 0 © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 8

Page 9

The Benchmarking Study, 2001 zoe@awesome-numbers.org 17 Total Error as Percent - Cholesterol trol group 91% of QC summary results from the Study group and 100% from the Control Group met 10% TEa from CLIA. © Z Brooks 2001 -2016 18 Target TEa Limit TEa Limit © Z Brooks 2001 -2016 0.0% 1.2% 2.4% 3.6% 4.8% 6.0% 7.2% 8.4% 9.6% 10.8% 12.0% 13.2% More Total Error as Percent of Target Control group Study group Critical Systematic Error (SEc) measures accuracy and precision relative to performance standards. SEc = The Number of SD the mean can shift before >5% of results from the control exceed TEa Limits. SEc is a metric that indicates how close a method is to TEa Limits SEc 20 18 16 14 12 10 8 6 4 2 0 © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 9

Page 10

The Benchmarking Study, 2001 zoe@awesome-numbers.org 19 If method bias or imprecision is increased, the SEc will be decreased. Target TEa Limit TEa Limit © Z Brooks 2001 -2016 20 SEc (Critical Systematic Error) of Sodium SEc < = 0 These controls do not meet quality specifications Study: 17.4% Vs 2.3% Control Study group Control group © Z Brooks 2001 -2016 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 One Bar = percent of QC samples at each ^SEc level SEc varies with Target, TEa, Mean and SD Control group Study group SEc SEc > 3.0 These controls easily meet quality specifications. The mean can shift > 3 SD before

5% of results exceed TEa. Less QC is required. Critical Systematic Error 30 25 20 15 10 5 0 © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 10

Page 11

The Benchmarking Study, 2001 zoe@awesome-numbers.org 21 SEc (Critical Systematic Error) of Cholesterol Study: 18.9% Vs 0.0% Control Study group Control group © Z Brooks 2001 -2016 22 SEc (Critical Systematic Error) of PT and APTT Study: 10.7% Vs 3.6% Control Study group Control group © Z Brooks 2001 -2016 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 Control group Study group Control group Study group SEc < = 0 These controls do not meet quality specifications Critical Systematic Error SEc < = 0 These controls do not meet quality specifications Critical Systematic Error 14 12 10 8 6 4 2 0 35 30 25 20 15 10 5 0 © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 11

Page 12

The Benchmarking Study, 2001 zoe@awesome-numbers.org 23 SEc (Critical Systematic Error) of Hemoglobin Study: 16.3% Vs 0.8% Control tudy group Control group © Z Brooks 2001 -2016 24 SEc (Critical Systematic Error) of TSH Study: 25.4% Vs 4.1% Control tudy group Control group © Z Brooks 2001 -2016 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 Control group Study group Critical Systematic Error Control group Study group SEc < = 0 These controls do not meet quality specifications Critical Systematic Error SEc < = 0 These controls do not meet quality specifications 14 12 10 8 6 4 2 0 18 16 14 12 10 8 6 4 2 0 © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 12

Page 13

The Benchmarking Study, 2001 zoe@awesome-numbers.org 25 Laboratories used a now discontinued software program created True Limit Limit 26 Z-Bars© show variation in TE and SEc © Z Brooks 2001 -2016 One Z-Bar = 1 QC sample for 1 month by Zoe Brooks to create Z-Bars© to display performance compared to the Target and TEa Target = 123.0 TEa = 4.0 Mean = 123.1 SD = 0.4 Target = 141.8 TEa = 4.0 Mean = 143.2 SD = 2.2 Higher bias and SD  higher TE and lower SEc. Sodium Accurate methods are centered about the target value Precision is indicated by the width of the Z-Bar© Position indicates Bias Z-Bars© are color coded like traffic lights: Red = problem One Z-Bar = 1 QC sample for 1 month © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 13

Page 14

The Benchmarking Study, 2001 zoe@awesome-numbers.org 27 © Z Brooks 2001 -2016 28 Z-Bars© for study group Nov 2000 © Z Brooks 2001 -2016 Z-Bars© for study group Nov 2000 Potassium Calcium Glucose Cholesterol © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 14

Page 15

The Benchmarking Study, 2001 zoe@awesome-numbers.org 29 © Z Brooks 2001 -2016 30 © Z Brooks 2001 -2016 Z-Bars© for study group Nov 2000 Z-Bars© for study group Nov 2000 Prothrombin Time APTT © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 15

Page 16

The Benchmarking Study, 2001 zoe@awesome-numbers.org 31 © Z Brooks 2001 -2016 32 © Z Brooks 2001 -2016 Z-Bars© for study group Nov 2000 Z-Bars© for study group Nov 2000 Hemoglobin WBC © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 16

Page 17

The Benchmarking Study, 2001 zoe@awesome-numbers.org 33 Eosinophil Absolute Count and Percentage © Z Brooks 2001 -2016 34 © Z Brooks 2001 -2016 Z-Bars© for study group Nov 2000 Z-Bars© for study group Nov 2000 Digoxin © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 17

Page 18

The Benchmarking Study, 2001 zoe@awesome-numbers.org 35 © Z Brooks 2001 -2016 36 The patient test volume is ~4x higher; QC test frequency is the same © Z Brooks 2001 -2016 Each dot = 1 QC sample from one lab/method for 1 month Z-Bars© for study group Nov 2000 Ratio of QC Samples to Patient Samples 200 N u m b e r o f Q C S a m p le s 150 100 50 0 0 500 1,000 1,500 2,000 2,500 TSH AST Up to 2,500 Patients per Month Excellent Stability Moderate Stability Number of Patients © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 18

Page 19

The Benchmarking Study, 2001 zoe@awesome-numbers.org 37 AST QC to Patient Samples By Instrument 200 150 N u m b e r o f Q C S a m p le s 100 50 0 500 1,000 1,500 2,000 2,500 Abbott Bayer Beckman-Coulter Dade-Behring Ortho Roche © Z Brooks 2001 -2016 38 2000 1800 1600 N u m b e r o f Q C S a m p l e s 1400 1200 1000 © Z Brooks 2001 -2016 800 600 400 200 0 0 5,000 10,000 15,000 20,000 Ratio of QC Samples to Patient Samples Ratio of QC Samples to Patient Samples Na Excellent Stability Na Moderate Stability 1,000 Patients, 140 QC 1,800 Patients, 50 QC Sodium by Method Stability Number of Patients Number of Patients Na Ex Stability 5,900 patients; 1,900 QC Na Ex Stability 6,300 patients; 150 QC 2,500 Patients, 140 QC © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 19

Page 20

The Benchmarking Study, 2001 zoe@awesome-numbers.org 39 120 100 N u m b e r o f Q C S a m p le s 80 60 40 20 0 © Z Brooks 2001 -2016 40 Does the observed SEc affect the selection of QC rules? We can increase error detection when SEc is low by increasing the number of QC samples or frequency of QC runs. Thus we would expect that the ratio of patient samples to QC samples would be INVERSELY proportional to SEc. A control that can only shift 1 SD (A) requires more frequent QC than one that can shift 10 SD before exceeding quality specifications. (B) © Z Brooks 2001 -2016 Ratio of QC Samples to Patient Samples 0 1,000 2,000 3,000 4,000 Excellent Stability Moderate Stability Poor Stability TSH by Stability Number of Patients A SEc zero to > 10 A = 10 pts/QC B = 100 pts/QC B © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 20

Page 21

The Benchmarking Study, 2001 zoe@awesome-numbers.org 41 350 R a t i o o f P a t i e n t s t o Q C 300 250 200 150 100 50 0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 © Z Brooks 2001 -2016 Each dot = 1 QC sample from one lab/method for 1 month 42 450 400 R a tio o f P a tie n ts to Q C 350 300 250 200 150 100 50 0 0.0 2.0 4.0 6.0 8.0 10.0 12.0 © Z Brooks 2001 -2016 Do labs test more QC samples/ patient when SEc is low? Do labs test more QC samples/ patient when SEc is low? Calcium Ratio of Patients to QC Samples Glucose Ratio of Patients to QC Samples SEc SEc © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 21

Page 22

The Benchmarking Study, 2001 zoe@awesome-numbers.org 43 45 40 R a tio o f P a tie n ts to Q C 35 30 25 20 15 10 5 0 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 © Z Brooks 2001 -2016 44 450 400 350 300 R atio of Patie nts to Q C 250 200 150 100 50 0 0.0 5.0 10.0 15.0 20.0 -50 © Z Brooks 2001 -2016 Do labs test more QC samples/ patient when SEc is low? Do labs test more QC samples/ patient when SEc is low? APTT Ratio of Patients to QC Samples WBC Ratio of Patients to QC Samples SEc SEc © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 22

Page 23

The Benchmarking Study, 2001 zoe@awesome-numbers.org 45 20 18 16 R a tio o f P a tie n ts to Q C 14 12 10 8 6 4 2 0 0.0 1.0 2.0 3.0 4.0 © Z Brooks 2001 -2016 46 75 R a tio o f P a tie n ts to Q C 50 25 0 0 5 10 15 20 © Z Brooks 2001 -2016 Do labs test more QC samples/ patient when SEc is low? Do labs test more QC samples/ patient when SEc is low? Digoxin Ratio of Patients to QC Samples Ratio of Patients to QC Samples Coagulation SEc SEc Hematology Chemistry Special Chemistry © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 23

Page 24

The Benchmarking Study, 2001 zoe@awesome-numbers.org 47 Calcium – Compare 2xCV to TEa as % 14% 12% 10% 2 x CV% (Actual) 8% 6% 4% 2% 0% © Z Brooks 2001 -2016 Each dot = 1 QC sample from one lab/method for 1 month 48 25% 20% 2 x CV% (Actual) 15% 10% 5% © Z Brooks 2001 -2016 Total Error = |Bias| + 2 * SD These controls exceed TEa with 2 x CV 0% 2% 4% 6% 8% 10% 12% 14% Total Error = |Bias| + 2 * SD These controls exceed TEa with 2 x CV 5% 10% 15% 20% 25% WBC – Compare 2xCV to TEa as % Study group Control Group TEa Study group Control Group TEa as Percent TEa as Percent © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 24

Page 25

The Benchmarking Study, 2001 zoe@awesome-numbers.org 49 QC Rules by Percent of Controls 70% 60% 50% 40% 30% 20% 10% 0% 1-2s Only © Z Brooks 2001 -2016 50 © Z Brooks 2001 -2016 1-2s 1-2sW 2-2s 2 of 3 1-3s R-4s 4-1sR 4-1sW 10xR 10xW Other Chemistry Coagulation Hematology Special Chemistry Ratio of Observed to Assigned SD/CV 15.0% C V A s s i g n e d o n Q C C h a rt 10.0% 5.0% 0.0% 0.0% 5.0% 10.0% 15.0% AST Calcium Cholesterol CK GLUCOSE K Na Each dot = 1 QC sample from one lab/method for 1 month Chemistry Measured / Actual CV% © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 25

Page 26

The Benchmarking Study, 2001 zoe@awesome-numbers.org 51 Coagulation 15.0% CV Assigned on QC Chart 10.0% 5.0% 0.0% © Z Brooks 2001 -2016 52 Hematology 20.0% CV Assigned on QC Chart 15.0% 10.0% 5.0% 0.0% © Z Brooks 2001 -2016 0.0% 5.0% 10.0% 15.0% 0.0% 5.0% 10.0% 15.0% 20.0% Ratio of Observed to Assigned SD/CV Ratio of Observed to Assigned SD/CV Hemoglobin Platelets WBC Measured / Actual CV% APTT PT Measured / Actual CV% © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 26

Page 27

The Benchmarking Study, 2001 zoe@awesome-numbers.org 53 Special Chemistry 50% CV Assigned on QC Chart 40% 30% 20% 10% 0% © Z Brooks 2001 -2016 Comments January 16, 2016 54 This body of knowledge existed 10 years ago. Little (if any) improvement has been made in the effectiveness of quality control processes or, surprisingly, in the risk levels of medically-unreliable results. Risk Metrics, Risk Graphics and the process of Mathematically- OptimiZed Risk Evaluation (M.O.R.E.) transform this overwhelming amount of data into meaningful information with recommendations. This new perspective creates a new paradigm in quality management. © Z Brooks 2001 -2016 0% 10% 20% 30% 40% 50% Ratio of Observed to Assigned SD/CV Measured / Actual CV% Digoxin TSH S.Q.C. M.O.R.E. A calculation A process 4 variables 11 Risk Drivers Creates a metric

  • sigma (replaced ^SEc) ^SEc is a far better metric for labs. Margin for Error is better than both. Manages Risk 34 Risk Metrics 5 Benefits 14 Clear text evaluations 15 Recommendations © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 27

Page 28

The Benchmarking Study, 2001 zoe@awesome-numbers.org automates – and verifies- the QC process. 55 © Z Brooks 2001 /2016 Please join me to zoe@awesome-numbers.org www.awesome-numbers.org Mathematically-OptimiZed Risk Evolution virtually explore this new paradigm. © Z Brooks 2001-2016 Presented at TQM Symposium. Toronto 2001 Page 28