The Impact Of T Ea Zoe C. Brooks, ART, AWEsome Numbers Inc. Sudbury, ON, Canada; George Sweeney, MLS(ASCP)CM, M.O.R.E. Quality Consultant, USA
numbers
Obiectives:
- To demonstrate the relationship between the laboratory selection of TEa (total error allowable) limits, patient risk and clinical costs.
- To compare QC (quality control) data for calcium to CLIA (Clinical Laboratory Improvement Amendments) limits, BV (Biological Variation) and NIST (National Institutes of Standards and Technology) medical goal.3
“Incorrect result — result that does not meet the requirements for its intended medical use; NOTE 1: In the case of quantitative test procedures, a result with a failure of measurement that exceeds a limit based on medical utili .
Analytical performance specifications Model 1. Based on the effect of analytical performance on clinical outcomes. This can, in principle, be done using different types of studies:
- Direct outcome studies — investigating the impact of analytical performance of the test on clinical outcomesl Mian Consensus InternatOnal F-ederatOn of Chol ChanStry and Laboratory ‘*ledtne Cincal Chemstry and Laboratory Medtne
What is the cost of analytical error in calcium results?
“Evaluate the potential costs both in terms of the patient’s well-being and in terms of financial liability of the treating parties vs known benefits to the patient.” 2 (CLSIEP 23-A)
Backqround: TEa (allowable error) is “equivalent to the error that does not significantly contribute to wrong clinical decisions.” (ISO 15189:2012 Medical laboratories — Requirements for Quality and Competence). To evaluate risk, laboratories must “compare the estimated risk against given risk criteria to determine the acceptability of the risk” (CLSI Guideline EP23-A). Current TEa limits are typically selected within the laboratory as CLIA or BV limits. Neither of these may be appropriate as a ‘limit of medical utility”. Selection of TEa limits drives the perceived and factual acceptability of patient risk. Medical goals represent best practice.
The Impact of Calibration Error in Medical Decision Making “Based on analysis of over 89,000 patients receiving serum calcium tests at the Mayo Clinic in 1998—1999, we find that the number of follow-up procedures, and hence health care costs, is directly related to initial calcium test values.” 3
“For an analytical bias of 0.5 mg/dL the potential health care cost increase ranged from $34 to $89 per patient having a calcium test. With approximately 3.55 million patients per year receiving screening serum calcium tests being affected by systematic bias, the potential economic impacts range from $60 million to $199 million per year for analytic biases of 0.1 and 0.5 mg/dL, respectively.”3
$2,500
Average per patient follow-up costs
$2,000
$1,500
$1,000
$500
, Tue $0 10.0
Medical Goal
Biological Variation
10.5
CLIA Limit
11.0
11.5
Methodoloqy: We created 3 data sets of recently
- Measured mean;
- Measured SD; and
- Peer Mean with sigma values of 5.7, 5.2 and 3.3.
We entered the data in CatalystQCTM software with
- CLIA limits of 0.25 mmol/L (1.0 mg/dL),
- NIST-recommended Medical goal of 0.125 mmol/L (0.5 mg/dL) mg/dL, and
- Biological variation desirable TEa of 2.55%
We calculated a z-value as (Imeasured mean — TEa limitl)/SD) — equivalent to sigma metric as (TEa-lbiasl)/SD We converted the z-value to percent risk level plus the number and clinical/legal cost of medically-unreliable results (MURs)/year at this level of accuracy and precision.
We compared the interpretation of acceptability of quality and probable action based on the selected TEa limits.
Evaluation of Anal tical Process uali
Patient Samples /Yr. controlled mainly by this QC sample
True Value of this C Sam I Medical Goal (Limit of Medical Utility) or TEa Limi Measured Mean in this time period Measured SD in this time eriod Av . Clinical Le al cost Medicall -Unreliable resul Acce able Risk Criteria ARC as # MUR Year
MUR/ Year - with erformance of this time eriod
Acceptable Risk as Clinical/ Legal Cost of MUR/ Yea
Clinical/legal cost MUR currently reported / year.
A. Baseline 5.7 sigma (z-value) PASS 63,581 10.00 1.0 10.1 0.150 200 1 446 0 $289,200
$0
Ma in for Error # SD from mean to Unacce table Risk z-value (sigma from measured values [(TEa — I bias I )/SD] QC Summary Review - Acceptable Risk Criteria as % Evaluation of QC Process Effectiveness:
3.70 5.70 2.275% PASS
B. Change TEa to BV
FAIL 63,581 10.00 0.255 10.1
c. Change TEa to Medical
PASS 63,581 10.00 0.5 10.1
50 I q 46 11,260 $72,300
$563,000
100 1 446 436 $144,600
$43,600
What is this?
Where Does it come from
valuation Risk Driver isk Driver andard Risk Driver Risk Driver Estimate andard Risk Metric andard
Risk Metric
alculatedb software vailable records m source in lab poli boratory Policy From measured data From measured data linical in ut or studies borato Poli alculated by software boratory Policy
alculated by software
-1.27 0.73 2.275% FAIL
0.37 2.37 2.275% PASS
Risk Metric Statistic atistic valuation
alculatedb software alculated by software alculated by software alculatedb software
R-B- Grade
(TEa Limit)
9.00
A. Acceptable Risk - No Waste
10.00
(TEa umit)
11_00
M.O.R.E. Au -Action Lower Risk and Cost: Lower Medical Goal or Acceptable Risk Criteria to reduce risk Use OptimiZed QC Process No change in QC frequency
Results: We see that by only changing TEa values there is a significant change in patient risk and clinical cost. Looking at case 1 as an example, the medical and legal cost can go from $0 with a TEa of 1.0 mg/dL using CLIA limits to a cost of $320,900 for the year using biological variation desirable TEa of 2.55%. [Due to changes we made in the estimated coast at each level, these numbers are not reflected in the cases shown.]
The patient risk goes from a predicted O medically-unreliable results with CLIA to 3,209 patients at risk using the biological variation TEa. Comparing the results of evaluation with CLIA, Medical, and Biological Variance demonstrates that the value selected for TEa will play a major role in clinical acceptability, patient risk, and cost due to medically unacceptable errors.
The TEa should be validated to be certain that the risk and cost is minimized.
What to do
QC Reject Ruk
{1- 3.5 s
Sample Frequency/ Day
1
Chart Review
Front Line
Supervgor
Ix Week
. Ix 4 Weeks
Change exsting QCP?
Ruk
Chart Freq
Review
Yes
No
No
No
Improve - if possible Accuracy Precision
No
No
R- B - Grade
Medial Goal (TEa Limit)
9.3S
F.
FAIL
1000
Medial Goal (T Ea Limit)
10.26
.O.R.E. Auto-Action to Lower Risk and or Cost: Analytical process quality fails acceptable risk criteria Stop now and ACT to mitigate and reduce risk Refer to Risk Management Procedure
What to do
STOP
STOP!
Improve Method
Accuracy Yes
Yes
R-B- Grade
Medical Goal (TEB Limit)
9.50
A. Acceptable Risk - No Waste
Medical Goal (TEB Limit)
10.50
. .R.E.Au Acti n
r Risk Improve method accuracy & precision Use OptimiZed QC Process Increase fre uen
n
r
What to do
QC Reject Rule
1- 2.0
Sample Frequency/ Day 2
Chart Review
Front Line 2k Week
Supervisor lx Week
Rule
Change existing QP? Chart r req
Review
Yes
Yes
Yes
1m Accuracy Yes
- if •ble Precision
Yes
“If patient result is 10.0, the average O follow-up costs are about $650.
If this sample is reported as 10.5 o mg/dL would, on average, receive an additional $550 in follow-up tests.
If reported as 1 1 .O mg/dL, this error o causes an additional $1 150 in follow- up tests.”
This study analyzed only the cost of follow up tests. (They applied the term bias as total error: inaccuracy/bias of a single sample.) In our study, we conservatively estimated clinical/legal cost of error of 0.5 mg/d/L (0.125 mmol/L) at $100; 1.0 mg/d/L (0.250 mmol/L) at $200 and 0.255 mg/dL (0.06 mmol/L) at $50.
Discussion:
“Risk is the combination of [A] the probability of occurrence of harm and [B] the severity of that harm” (ISO/IEC Guide 51).
You can control severity of harm by setting limits of medical utility (medical goals) — and the associated cost of failure to meet those goals - for each analyte by clinical setting.
You can control probability of harm by setting acceptable risk criteria as the number and cost of Medically-unreliable Results reported, routinely at method review and if the method fails.
M.O.R.E. QualitYTM will design a QC process and guide staff to meet these standards or act as per your procedures. Risk management and Mathematically-OptimiZed Risk EvaluationrM are based on medical goals and acceptable risk criteria by the local laboratory director — and patients, institutes, physicians, and society (“The PIPS”).
Conclusions:
In these case studies the best source for TEa was the CLIA limit throughout. Best practice would be to validate any TEa that is used prior to starting to report patient results. This would save money and reduce patient risk.
References:
- Milan Consensus International Federation of Clinical Chemistry and Laboratory Medicine 2014
Clinical & Laboratory Standards Institute. Guideline EP23-A Laboratory quality control based on risk management. Wayne, PA. 2011 3. Gallaher M, Mobley LR, Klee G, and Schryver P. The impact of calibration error in medical decision making. Prepared for National Institute of Standards and Technology Chemical Science and Technology Laboratory. Planning report 04-1; May 2004. 4. Brooks, Zoe. Managing Acceptable Risk Level 1: Risk Assessor. AWEsome Numbers, Inc. Sudbury, Ontario, Canada. 2016.
Contact Information: Zoe: George:
Zoe Brooks is CEO and Director of Innovation and Research at AWEsome Numbers, Inc. George Sweeney has no financial interest in the company.