Original Articles
Phoenix in the lab: The sigma metrics during Chennai’s worst disaster: Monitoring and management of the Quality Management System (QMS)
Srinivas Chakravarthy1*, Satish Ramanathan2, Smitha S3, Vijayakumar KV4, Thirumalai Nallathambi5, Micheal S6
Author Affiliations
1* Department of Laboratory Medicine, Madras Institute of Orthopaedics and Traumatology (MIOT) Hospital, 4/112, Mount Poonamallee Road, Manapakkam, Chennai, Tamil Nadu, India
2-6 Division of Clinical Biochemistry, MIOT Hospital, 4/112, Mount Poonamallee Road, Manapakkam, Chennai, Tamil Nadu, India
IJPLM. 2017;3(1):OA1
Correspondence: Dr. Satish Ramanathan
Submitted: 16 March 2017, Accepted: 24 March 2017, Published: 24 March 2017
© IJPLM
Abstract
Background: Sigma is a quality management tool that helps in continuous monitoring and improvement of the performance of the analytes in the clinical laboratory. The present study evaluated the performance of our Quality Management System under optimal condition as well as during the adverse condition due to unexpected flooding in Chennai, India in the month of December 2015.
Methods: The performance of the assays in the clinical laboratory was evaluated in sigma scale for 27 routine analytes under optimal condition, adverse condition during the disaster, and after restoring the optimal condition.
Results: The performance of each analyte was analyzed to evaluate the quality of results released under all the three situations viz., optimal condition (Pre-disaster phase); adverse condition (Disaster phase) and after re-establishing the optimal condition (Post-disaster phase). Six analytes showed poor performance with a sigma <3 during the adverse condition at level 1 control and 5 analytes for level 2 control showed sigma <3. After re-establishing the optimal condition, 4 analytes showed an acceptable performance of >3 sigma. Alanine aminotransferase showed a <3 sigma performance at level 1 control and bicarbonate at both levels of quality control even after the optimal condition was restored.
Conclusion: Even in adverse condition, the quality of the results released from VITROS 5600 integrated system was not much impacted except for a few analytes. Analysis using quality goal index (QGI) showed that imprecision was the reason behind the unacceptable sigma. Good clinical laboratory practices aided us in improving the quality of performance of these analytes.
Keywords: Imprecision, Bias, Disaster management, Sigma, QGI, TE
Introduction
Sigma metrics is a management strategy aimed at improving the quality of process outputs by identifying and removing the causes of defects (errors) and minimizing variability. It provides a more quantitative frame work for evaluating process performance in clinical laboratories to assess analytical performance of the laboratory by measuring the process variation and determining process capability in sigma units, thereby to enhance quality and near zero defect rates in healthcare system.1 The sigma metrics value depicts the likelihood of the occurrence of errors; the chances of false test results or defects in lab reports are less likely with higher sigma value.2 In the clinical laboratory system, sigma metrics is used in combination with total allowable error, method imprecision and bias.3
In a tertiary healthcare setting, under favourable circumstances, it is essential to set up a quality system to ensure that all norms are fulfilled and the quality of work is maintained. Several studies have reported on the application of sigma metrics for the assessment and modification of quality control programs in clinical laboratories under favourable conditions. However, there are no published studies on the assessment of performance of a clinical laboratory based on sigma metrics and management of quality during a disaster or adverse conditions.
The purpose of this study was to assess the quality of the results of 27 biochemical analytes during a disaster condition at our hospital when compared to the normal setting. The hospital had encountered a major disaster in the month of December 2015, due to unexpected flooding in Chennai, India. This major natural calamity affected the laboratory operations in many ways including emergency power shut-down due to floods in the hospital premises, continuous power failure for two days, interrupted power supply in the laboratory for a month, erratic ambient temperature and humidity, increased attrition rates of laboratory technologists and breakage of cold chain management of the reagents, quality control materials and calibrators. Even in that difficult circumstance, the clinical laboratory functioned with limited staff and the results were released for the needy patients. The present study evaluated the quality of the results released during the disaster condition based on the sigma metrics and compared the results with those before the flood and after re-establishing the optimal condition.
Materials and methods
The sigma metric study was conducted at the Division of Clinical Biochemistry, Department of Laboratory Medicine, Madras Institute of Orthopaedics and Traumatology (MIOT), Chennai for a period of 8 months from October 2015 to May 2016. The laboratory renders clinical laboratory service to 800 bedded tertiary care multi-speciality hospital. The study was organised into three phases: Pre-disaster phase 1[optimal condition], October to November 2015; Disaster phase 2 [adverse condition], December 2015 to February 2016; and Post-disaster phase 3 [after re-establishing the optimal condition], March to May 2016.
Sigma metric analysis was performed on 27 biochemical analytes using VITROS 5600 integrated system (Ortho Clinical Diagnostics, USA). Out of the 27 biochemical analytes, 26 were based on VITROS microslide technology and one (direct LDL Cholesterol) on VITROS microtip technology.4, 5 The quality control practices employed (as per ISO 15189:2012 and NABL 112:2016) to improve the performance were ensuring: optimum room temperature and humidity of the equipment, uninterrupted power supply of the equipment (Vitros 5600 integrated system) and cold storage space (storage of reagent, QC and calibrators), and routine equipment maintenance though rigorous technologist training. The reagent handling practices including following reagent storage and reagent handling instructions like pre-warming of microslides were stringently followed as per manufacturer’s recommendations, and the quality control practices including QC storage, QC reconstitution and usage as per manufacturer’s instructions were improved and special precautions were taken for volatile analytes like bicarbonate. As a preventive measure, quality control aliquot vial was changed from 1.5 ml to 0.5 ml to reduce the air space and prevent undue evaporation.
Internal quality control analysis was performed twice daily for all the analytes using two levels of controls (Bio-Rad Quality control level 1 and 2). The quality control (QC) outliers were identified based on Westgard multi QC rules adopted by the laboratory, including 1-3s, 2-2s.6 Root cause analysis was performed, documented and the corrections in the outliers were resolved daily. Sigma (σ) value was calculated with the following formula.1
Sigma metrics (σ) = (TEa% - Bias%)
------------------------
CV%
Where TEa% and CV% indicates total allowable error percentage and coefficient of variation respectively. Coefficient of variation (CV%) was derived from calculated laboratory mean and SD of internal QC data using the formula,
CV% = Standard deviation
-------------------------- X 100
mean
CV% was extracted from VITROS 5600 integrated system, as the system has internal software for quality control data analysis. Bias was calculated based on the comparison between laboratory mean and peer group mean available from unity real time data (Bio-Rad unity real time software) for the specific lot of reagents, by using the following formula,
Bias% = Peer group mean – lab mean
----------------------------------------- X 100
Peer group Mean
Where peer group mean is the mean of all QC values of laboratories using the same instrument and method. Total allowable error (TEa) of each analyte was calculated according to CLIA (Clinical Laboratory Improvement Amendment) guidelines.7 The TEa for the analytes like LDL cholesterol, bicarbonate and lipase were not available in CLIA. Hence TEa of these analytes were calculated based on Royal College of Pathologists of Australasia guidelines.8
The analytes were classified based on the performance as follows: ≥6 sigma level: excellent performance, <5.9 to ≥3 sigma level: good to moderate performance, <3 sigma: poor performance.9 Quality tool like quality goal index (QGI) was used to analyse the deviation of bias and precision.10 This was used to analyse the reason for the poor sigma showed by some of the analytes (whether due to imprecision or inaccuracy or both). The QGI ratio was calculated using the following formula,
QGI = Bias/1.5 X CV%
The criteria used for interpreting QGI when test applications fall <6 sigma quality is indicated in Table 1.11
1: Criteria used for interpreting QGI when test applications fall short of six sigma quality11
Results
The present study analysed the sigma of 27 analytes in VITROS 5600. The values of sigma of these analytes (level 1 and level 2 QC) in all the three phases were tabulated (Table 2, Table 3). Analysis in optimal condition (Pre-disaster phase1) showed that among the 27 analytes, 22 and 23 analytes had acceptable performance of ≥3 sigma score at level 1 and level 2 controls respectively. At level 1 control, 16 analytes showed a performance of ≥6 sigma level and at level 2 controls, 18 out of the 27 analytes, showed a performance of ≥6 sigma level. LDL cholesterol showed a performance of 6 at level 1 control, and a performance of 3 to 6 at level 2 control.
2: Percentage of total allowable error (TEa), bias% and CV% for phase 1, 2 and 3 at level 1 and sigma metric of analytes
3: Percentage of total allowable error (TEa), bias% and CV% for phase 1, 2 and 3 at level 2 and sigma metric of analytes
During adverse conditions, the performance of analytes such as total protein and urea, at both levels of controls slipped down <6, but remained within 3-6 sigma level. In addition, amylase slipped down <6 sigma at level 1 control, but remained within 3-6 sigma. Also, creatinine and phosphorus at level 2 controls slipped <6 sigma, but remained within 3-6 sigma.
After restoring optimal condition, creatine kinase, total protein and urea showed performance of ≥6 at both the levels. The improvement in the performance was observed in both accuracy (bias%) and precision (CV%) after restoring optimal conditions. At level 2 control, creatinine showed an improvement in performance from <6 sigma level to ≥6 sigma level after restoring optimal condition. Other analytes also showed a consistent performance during the adverse condition and after restoring optimal condition. QGI was calculated for those analytes with sigma <3 at end of phase 2 and are provided in Table 4.
4: List of analytes performed low during phase 2 assessed for sigma QGI for accuracy and precision problem
Discussion
Sigma metrics is a tool to assess the performance of a process on a universal scale from 1 to 6; six sigma indicates world class performance and 3 sigma is considered as the minimum acceptable performance level.11 CV% (imprecision) and bias% (inaccuracy) are the two statistical indicators that influence the quality of analyte, when expressed in terms of sigma metric. Hence if the analyte is observed to have sigma of <3, it is possibly because of poor precision and/or accuracy.
The present study demonstrated that analytes such as alanine aminotransferase (ALT), bicarbonate, chloride, potassium, iron have showed a consistent poor performance with sigma <3 during phase 1 and phase 2. The study has also noted a drop in the performance of analytes like albumin and sodium, which slipped from ≥3 to <3 sigma, during the disaster (phase 2) compared to their performance in phase 1. An in-depth analysis of CV% and bias% and their contribution to sigma with respect to all above mentioned analytes showed poor performance in phase 2. The study has also noted that poor CV% was the major contributing factor for sigma <3.
The performance of analytes with sigma <3 (during the adverse condition) was analysed after the corrective measures were taken. Iron, sodium, potassium and chloride showed an improvement in their sigma to ≥3, among which the electrolytes like (sodium, potassium and chloride) had a total allowable error of ≤5%. Hence a further improvement in quality of performance of these analytes in terms of sigma (≥6) is practically impossible to achieve. Analytes including ALT (at level 1 control) and bicarbonate (at both levels of control) did not show an improvement in performance and remained at a sigma <3 (Table 2, Table 3).
For ALT and bicarbonate, the QGI was calculated at the end of Phase 3 and showed that the analytes had a significant improvement in bias%, whereas QGI value of less than 0.8 indicated that precision was not improved (Table 5). A performance variable analysis for ALT and bicarbonate was done wherein the lab CV% obtained in phase 3 for these analytes were compared to the manufacturer claim (total CV%). The analysis showed that the lab CV% was comparable to the manufacturer’s claim (Table 5).
5: Phase 3 six sigma, QGI value and CV% for ALT and bicarbonate, and comparison with manufacturer’s claim
The study conducted by Usha et al, in 2015 has reported a sigma value of <3 for urea level 1 and 2, a sigma value between 3-6 at level1, ≥6 at level 2 for total protein, a sigma value <3 for ALT and creatinine at level 1, and sigma between 3-6 at level 2 under normal conditions.12 Similar to these findings, the present study has shown that the performance of urea and total protein at both the levels was between 3-6 sigma under adverse conditions and after restoring the condition, a performance of ≥ 6 sigma was noted at both the levels. ALT remained at a sigma <3 at both the levels, and creatinine showed a performance of ≥6 sigma at both the levels under normal conditions, and dropped between 3-6 sigma at level 2 under adverse condition.
Based on the study findings, we could infer that six sigma is a good quality tool to assess the analytical performance of a clinical laboratory. There are certain limitations in clinical application of six sigma concept with respect to a few analytes for which imprecision (CV%) and bias% prove to be more reliable than sigma to assess the performance of such analytes, provided they are within the total allowable error specific for the applicable analytes. The current study holds significance as there is not much evidence pertaining to the use of six sigma for evaluating the quality of the results of biochemical analytes during a disaster condition.
Conclusion
The present study showed that VITROS 5600 intergrated system could produce consistent and robust quality of results. Except for a few analytes, which showed unacceptable sigma of <3, good clinical laboratory practices were adopted and improved upon in accordance with ISO 15189:2012. These measures resulted in an improvement in performance of most of the analytes on sigma scale.
The study also concludes that sigma is an industrial standard and hence cannot be universally applied to all analytes in clinical laboratory practice, as the clinical significance of the analytes vary. Therefore, laboratory medicine specialists should make a meaningful interpretation of sigma.
Competing interests
The authors declare that they have no competing interests.
Acknowledgement
The authors of this study acknowledge the support of Management of MIOT Hospitals, Chennai for providing the infrastructure and facilities to carry out this work.
References
1. Westgard S, QC W. Six Sigma metric analysis for analytical testing processes. Abbott Laboratories MS-09. 2009; 04:7907.
2. Westgard S, QC W. Quantitating Quality: Best Practices for Estimating the Sigma-metric. Abbott Laboratories MS-10. 2011; 01: 38704.
3. Goel SS, Saini R, Singh SB, Aggarwal O, Goel AK. Six Sigma Metrics and Quality Control in Clinical Laboratory. International Journal of Medical Research and Review [Internet]. 2014 [cited 2017 May 11];2(02). Available from: http://medresearch.in/index.php/IJMRR/article/view/91
4. VITROS® MicroSlide Technology │ Ortho Clinical Diagnostics [Internet]. [cited 2017 May 9]. Available from: https://www.orthoclinical.com/en-in/solutions-products/technologies/vitros-microslide-technology
5. VITROS® MicroTip Technology │ Ortho Clinical Diagnostics [Internet]. [cited 2017 May 9]. Available from: https://www.orthoclinical.com/en-in/solutions-products/technologies/vitros-microtip-technology
6. “Westgard Rules” and Multirules - Westgard [Internet]. [cited 2017 May 9]. Available from: https://www.westgard.com/mltirule.htm
7. CLIA Requirements for Analytical Quality - Westgard [Internet]. [cited 2017 May 9]. Available from: https://www.westgard.com/clia.htm
8. RCPA Allowable Limits of Performance for Biochemistry - Westgard [Internet]. [cited 2017 May 9]. Available from: https://www.westgard.com/rcpa-biochemistry.htm
9. Singh B, Goswami B, Gupta VK, Chawla R, Mallika V. Application of Sigma Metrics for the Assessment of Quality Assurance in Clinical Biochemistry Laboratory in India: A Pilot Study. Indian J Clin Biochem. 2011;26(2):131–5.
10. Quality Goal Index - Westgard [Internet]. [cited 2017 May 9]. Available from: https://www.westgard.com/guest34.htm
11. Westgard JO, Westgard SA. The Quality of Laboratory Testing Today: An Assessment of σ Metrics for Analytic Quality Using Performance Data From Proficiency Testing Surveys and the CLIA Criteria for Acceptable Performance. Am J Clin Pathol. 2006;125(3):343–54.
12. Adiga US, Preethika A, Swathi K. Sigma metrics in clinical chemistry laboratory – A guide to quality control. Al Ameen J Med Sci. 2015; 8(4):281-287.