Original Articles

Precision and diagnostic accuracy verification of VITROS HIV combo assay

Ramanathan Satish^*2, Chakravarthy Srinivas N¹, S Smitha², Nallathambi Thirumalai²

 

Author Affiliations

¹Department of Laboratory Medicine, MIOT Hospitals, Chennai, India

²Department of Clinical Biochemistry, MIOT Hospitals, Chennai, India

 

*Correspondence: Dr. Satish Ramanathan

satishraksha1980@gmail.com

 

IJRCI. 2019;7(1):OA3

 

Submitted: 11 September 2019, Accepted: 27 December 2019, Published: 11 January 2020

© IJRCI

Abstract

Background/Objective: NACO has laid down three structured algorithms (I, II, III) for HIV screen, surveillance and diagnosis. It is the responsibility of the laboratories to select, verify and evaluate the methods fitting into the algorithm of HIV testing. The present study has attempted to evaluate VITROS® HIV combo assay through comparison with Abbott Architect HIV combo assay.

 

Method: The study evaluated VITROS® HIV combo assay through precision verification and diagnostic accuracy assessment by adopting CLSI guidelines. Precision verification of the VITROS HIV combo assay is done based on CLSI EP15A3 and diagnostic accuracy verification based on CLSI EP12A2.

 

Result: Precision verification showed a repeatability CV% (CVR) -2.35% and within lab CV% (CVWL)-3.77%. This was compared against the manufacturer’s claims (σR-3%, σWL-4.8%) and found to be within acceptable limits. Diagnostic accuracy verification showed that difference in paired sensitivity between VITROS® HIV combo assay and Abbott Architect HIV combo assay was zero and difference in paired specificities was 0.3%. This difference was proved to be statistically insignificant based on 95% confidence interval for the paired sensitivities and specificities.

 

Conclusion: The study showed an acceptable performance of VITROS HIV combo assay in terms of precision and diagnostic accuracy.

Introduction

Human immunodeficiency virus (HIV) has emerged as the number one dreadful communicable disease in the last three decades. According to 2016 World Health Organization (WHO) report on global prevalence of HIV infection, 36.7 million people are living with HIV/AIDS worldwide and 1 million deaths have been reported due to HIV-related illnesses.¹ Past decades have witnessed significant advances in the field of HIV prevention, diagnosis and management. Impact of early diagnosis and initiation of anti-retroviral therapy (ART) on long-term survival and infectivity of affected individuals had been studied extensively worldwide.² This has urged an inevitable evolution of HIV testing aimed at early diagnosis. This is expressed as time taken for a particular HIV testing method to detect HIV from the time of infection, otherwise known as ‘window period’.  Various platforms are available worldwide for HIV diagnosis and one of the most commonly used methods is immunoassay. HIV immunoassay has seen the recent addition of fifth generation assay.³ The advancing generations are targeted at substantial reduction in diagnostic window period through an improved efficiency.

 

In spite of the technological advancements, HIV is deprived of a ‘gold standard’ test for disease confirmation. Hence various national and international organizations have developed algorithms for diagnosis of HIV. In India, National AIDS Control Organisation (NACO) has published revised guideline for HIV testing in the year 2015.⁴ Three ‘fit for purpose’ algorithms are available in NACO for HIV testing (algorithm I for blood transfusion donor screen, algorithm II for surveillance, and algorithm III for HIV diagnosis in asymptomatic individuals).⁴ Algorithm III of NACO is being widely used by the clinical laboratories in India for HIV diagnosis. Algorithm III aids in laboratory diagnosis of HIV through a ‘diagnostic review criteria’ established based on findings of a cocktail of three different methods of HIV testing. The first-line testing method should be sensitive to arrive at an early diagnosis of HIV, and the second and third method should be specific to eliminate false positive results. It is the primary responsibility of the clinical laboratories intended to use algorithm III to ensure proper selection, verification and utilization of appropriate HIV testing methods fitting into the algorithm’s groove. The present study has verified the potential of the current  fourth generation VITROS® HIV combo assay to be used as a first-line method for HIV diagnosis (as per algorithm III) as opposed to  the existing first-line HIV testing method (Abbott Architect HIV combo assay) in routine practice. The study has also attempted the verification of VITROS® HIV combo assay based on Clinical Laboratory Standards Institute (CLSI) EP15A3 guidelines (precision verification) and EP12A2 guidelines (diagnostic accuracy verification).⁵

 

Materials and methods

The verification study was conducted at the Division of Clinical Biochemistry, MIOT hospitals, Chennai for a period of three months (August to October 2017). The testing method taken up for the study was fourth generation VITROS® HIV combo assay (Ortho Clinical Diagnostics). VITROS® HIV combo assay is a chemiluminescent immunoassay aimed at detection of anti-HIV 1 and 2 antibodies and p24 antigen. Antibody detection is achieved using recombinant transmembrane envelope proteins for HIV-1 group M and O, and HIV-2.  Antigen detection is accomplished using monoclonal antibodies against HIV p24.⁷ The present study was aimed at verifying the following:

·         Precision verification based on CLSI EP15A3: user estimation of precision and estimation of bias

·         Diagnostic accuracy verification based on CLSI EP12A2: user protocol for evaluation of qualitative test performance

 

a)    Precision verification study

CLSI EP15A3 has been established for precision verification and trueness estimation of quantitative methods. But, VITROS® HIV combo assay is a qualitative immunoassay wherein the patients’ results are interpreted as positive or negative. Though VITROS® HIV combo assay is a qualitative method, the present study adopted CLSI EP15A3 guideline due to the following reasons:

 

1.     Results from VITROS® HIV combo assay are reported as positive/negative, but it provides numeric results for patients (sample/cut off) with a diagnostic cut off 1 (patients’ results if ≥ 1 s/co are interpreted as positive). According to CLSI EP12A2, a qualitative test that produces numeric values can be verified, evaluated and monitored as a quantitative method.⁶ Hence, the present study attempted to evaluate the precision around the diagnostic cut off (1 s/co) of our testing method by using EP15A3.

2.     As per CLSI EP15A3, any testing method validated by the manufacturer adopting EP5A2 guideline, evaluation of precision performance of quantitative measurement methods shall be verified using EP15A3.⁵ Since VITROS® HIV combo assay had been validated by manufacturer for precision using EP5A2.⁷ The study used EP15A3 for precision verification of the same.

 

Testing materials used for precision verification study included two patient sample pools obtained after ethical committee approval. The pools included one positive pool (with value near diagnostic cut off: 1 s/co) and one negative pool. A five-run precision experiment was carried out as per EP15A3, with each run comprising of 5 replicates (5x5). After eliminating the outliers, two measures of imprecision were calculated based on statistical procedures adopted from EP15A3. These included ‘within run imprecision’, otherwise known as repeatability (% CVR), and ‘within lab imprecision’ (% CVWL). These imprecision estimates were compared against the manufacturer’s claims for the same.

 

b)      Diagnostic accuracy verification

The study attempted to measure the diagnostic accuracy rather than the measurement accuracy (comparing a testing method against the diagnostic accuracy criteria). This criterion was established as per NACO algorithm III, based on the findings of three methods of HIV testing for making a final positive/negative classification of infection. The three methodological platforms adopted in our laboratory for HIV diagnosis, as per NACO algorithm III, are as follows (Fig. 1).

1.       Abbott architect HIV combo assay (A1): highly ensitive assay

2.       VIDAS HIV DUO ultra assay (A2): highly specific assay differentiating antigen and antibody⁹

3.       Bio-Rad Geenius HIV ½ confirmatory assay (A3): highly specific assay differentiating HIV 1 and 2 infection¹⁰

 

Fig. 1: Strategy three: detection of HIV infection in asymptomatic individuals

VITROS® HIV combo assay is a fourth-generation immunoassay to detect p24 antigen and anti-HIV antibodies (HIV 1 and 2). This has improved sensitivity (100%) with a shortened window period of detection.³ According to NACO algorithm III, wherein three different methods are used for diagnosis of HIV infection, the first-line method of testing is a highly sensitive screening method. The current study has attempted to make a three-way comparison (based on CLSI EP 12A2) between the candidate method (VITROS® HIV combo assay) and existing first-line ‘highly sensitive’ immunoassay in the laboratory (Abbott Architect HIV combo assay) against the diagnostic accuracy criteria established based on NACO algorithm III.⁸

 

The study considered 4994 samples obtained from hospital-based adult (20 to 60 years) patient population (3400 males and 1594 females). Among these samples, 4913 were confirmed negative and 52 were confirmed newly diagnosed positives (Table 1). Diagnostic accuracy was expressed in terms of sensitivity and specificity. These were estimated for both candidate and existing methods. The paired sensitivities and specificities of these two methods were compared and the differences, if statistically significant, were determined by calculating scored confidence limits based on statistical procedures adopted from EP 12A2.⁵

 

Table 1: Age and sex distribution of study subjects

 

Results

A. Verification of precision

The study followed a stepwise approach to verify precision. The experiment was carried with two patient sample pools, which included a positive pool (with value near diagnostic cut off: 1 s/co) and negative pools. The findings of positive patient samples’ pool have been presented in table 2.

 

Table 2: Compilation of data

 

Step 1: Compilation of data

Results of twenty-five replicates are presented in table 2. Visual inspection of the data showed that there were no gross outliers. Significant statistical outliers were ruled out by Grubbs’ test.

 

Step 2: Grubbs’ test for outliers

According to Grubbs’ test, a result qualifies as an outlier, if that value lies more than the G SDs from the sample mean (Grubbs’ limits) where G is the Grubbs’ factor, and SD is the standard deviation of the raw data including the suspected outliers. Grubbs’ factor G was calculated using Grubbs’ table (Table 3). G was calculated as 3.135. This was followed by calculating Grubbs’ limits.

 

Table 3: Calculation of Grubb’s factor

 

Grubbs’ limits = mean ± G x SD, Where, Mean = 1.404, SD = 0.050 and G=3.135

Grubbs’ limits = 1.404 ± (3.135 x 0.050) = 1.836 ± 0.1567

Grubbs’ lower limit = 1.247 and Grubbs’ upper limit = 1.561

Since all results fell within these limits, statistical outliers were ruled out.

 

Step 3: Imprecision estimate by one-way analysis of variance (ANOVA)

3.1 One-way ANOVA was used to find out the imprecision estimates of HIV within run and between run variability.5 A one-way ANOVA format was prepared using automated ANOVA calculation software (Table 4). From the ANOVA table, DF (degrees of freedom), DF total (total degrees of freedom), MS (mean squares), SS (sum of squares), SS total (total sum of squares) were calculated.

 

Table 4: Imprecision estimate by one-way analysis of variance (ANOVA)

 

3.2 Calculation of variance: The two components of variance were calculated from the ANOVA table (Table 3).

V_w = MS2

V_B = (MS1 – MS2)/n_o

Where, V_W = repeatability (within run) variance, V_B = between run variance, V_WL = within lab variance, n0 =average number of results per run, MS1=mean square of between run variation and MS2= mean squares of within run variation.

 

V_W = MS2 = 0.0011

V_B = (0.0096 – 0.0011)

          -------------------------- = 0.0017

                            5

The sum of two variance (V_w+V_B) yielded within lab variance (V_WL)

V_WL = V_W + V_B = 0.0011+0.0017 = 0.0028

 

3.3 Calculation of imprecision in terms of standard deviation (SD)

The square root of variance yielded the standard deviation. Three components of standard deviation were estimated including,

S_R = √ V_W = √0.0011 = 0.033

S_B = √V_B = √0.0017 = 0.041

S_WL = √V_W + √V_B = √0.0011 + √0.0017 = √0.0028 = 0.053

 

Where, S_R = repeatability (within run) standard deviation, S_B = between run standard deviation, and S_WL = within lab standard deviation.

 

3.4 Conversion of SD to co-efficient of variation (%CV)   

%CV_R = (0.033 x 100)/x, where x = grand mean

%CV_R = (0.033 x 100) / 1.404 = 2.35%

%CV_B = (S_B x 100)/x = (0.041 x 100) / 1.404= 2.92%

%CV_WL = (S_WL x 100)/x = (0.053 x 100) / 1.404 = 3.77%

 

Step: 4 Comparison of imprecision estimates with manufacturer’s claims

The imprecision estimates obtained were compared against the manufacturer’s claims. The repeatability %CV (2.35%) was lesser than the manufacturer’s claim, σ_R (3.0%). Within lab %CV (3.77%) was lesser than the manufacturer’s claim, σ_WL (4.8%) (Table 5).

 

Table 5: Comparison of imprecision estimates against manufacturer’s claims

 

B. Verification of diagnostic accuracy

The diagnostic accuracy was verified by the three-way comparison of VITROS® HIV combo assay (new method), Abbott architect HIV combo assay (old method) and the diagnostic accuracy criteria established according to NACO algorithm III. As per the guidelines, 22 positive cases reported by old (existing) method, turned out to be negative both in candidate method and confirmatory assay. The results were released as indeterminate and later confirmed as true negative by repeat testing after 2 weeks by confirmatory assay (with an ability to differentiate HIV antigen and antibody). The following steps were covered in arriving at the diagnostic accuracy:

 

Step 1: Computation of a three-way comparison (Table 6)

TP = a_pos + b_pos =52+0 = 52, FP = a_neg+b_neg = 0+7 = 7, FN = c_pos+d_pos = 0+0 = 0

TN = c_neg+d_neg = 22+4913 = 4935, N = n_pos+n_neg = 52+4942 = 4994

 

Step 2: Estimation of sensitivity of new (Sens_new) and old method (Sens_old)

     Estimated sensitivity of the new method is:

Sens_new = 100 X [(a_pos + b_pos) / n_pos] = 100 X [(52+0)/52] = 100X1 = 100%

Estimated sensitivity of the old method is:           

Sens_old = 100 X [(a_pos + c_pos) / n_pos] = 100 X [(52+0)/52] = 100X1 = 100%

Estimated difference of Sens_new and Sens_old is:

Sens_new – Sens_old = 100 X [(b_pos - c_pos) / n_pos] = 100 X [(0-0)/52] = 100 X 0 = 0

 

Table 6: A three-way comparison between the candidate method, the old method and diagnostic accuracy criteria

 

Step 3: Finding out the statistically significant difference between the paired sensitivities and specificities of new and old method

As per CLSI EP12A2, scored 95% CI was recommended and statistical procedures provided in the respective guidelines were adopted.

          

Interpretation: The difference between paired sensitivities value is zero (95% CI: -4.93 to +4.93). Hence the difference is not statistically significant.

Step 4: Estimation of specificity of new (Sensnew) and old method (Sensold)

Estimated specificitiy of new method is:

Spec_new = 100 X [(c_neg+d_neg)/n_neg] = 100 X [4935/4942] = 100X0.998 = 99.8%

Estimated specificitiy of old method is:

Spec_old = 100 X [(b_neg+d_neg)/n_neg] = = 100 X [4920/4942] =100X0.995 = 99.5%

 

Estimated difference of Spec_new and Spec_old is:

Spec_new – spec_old = 100X[(c_neg-b_neg)/n_neg] = 100X[(15)/4942] = 100X0.003 = 0.30%

 

Step 5: The 95% confidence interval (CI) for the difference between paired specificities       

Interpretation: The difference between paired specificities value is 0.3 (95% CI:0.43 to +0.57). Hence the difference is not statistically significant.

 

Discussion

Method verification has become the ‘quality mantra of modern era’ of clinical laboratories around the world. There is growing number of laboratories around the globe moving towards the path of accreditation to International Standards Organization (ISO) 15189:2012: ‘International standard for Medical laboratories-quality and competence’. According to the clause 5.5 in ISO 15189:2012, ‘ it is the responsibility of end-user laboratories to perform independent verification of the methods that are already validated by the manufacturer before intending to use these methods for clinical purpose. But ISO 15189 is only a ‘what to do’ document, which focuses on minimum requirements of clinical laboratories for establishing a quality management system in compliance with the international standard. Hence, it is the purview of the individual laboratories to design their quality system procedures to ‘how to do’ to meet the requirements of ISO 15189:2012.

 

With respect to method verification of the analytical procedures in a clinical laboratory, global guidelines are available in CLSI. CLSI is a non-profitable organization aimed at developing standards and guidelines for various processes in a clinical laboratory including verification of validated methods in a clinical laboratory. CLSI has published specific guidelines for verification of quantitative (CLSI EP15A3 guidelines on precision verification and estimation of bias) and qualitative (EP12A2 on evaluation of qualitative test performance) methods accordingly, wherein quantitative methods yield numeric results and qualitative methods yield results reported as: positive/negative, reactive/non-reactive etc.⁶

 

The current study has attempted to verify and evaluate the quality of performance of VITROS® HIV combo assay, which is a qualitative method yielding results as positive/negative, with a diagnostic cut off of 1.0 S/co. Patient values ≥1 are reported as positive and vice versa.

The study has verified the following two essential facets of quality of performance:

1.            Precision verification (around the diagnostic cut off)

2.            Diagnostic accuracy verification

 

1) Precision verification study

The findings of precision verification study of positive pooled patients’ samples yielded imprecision estimates in the form of %CV_R (2.35%) and %CV_WL (3.77%). Both of them were compared against the manufacturer’s claims (σ_R–3% and σ_WL–4.8%). The comparison showed that the actual %CV of repeatability and within lab %CV were within acceptable limits.

 

2) Diagnostic accuracy verification

As per CLSI EP12A2, diagnostic accuracy verification is classified into ‘high level’ and ‘low level’ comparisons. High level comparison comprises of comparing trueness of a testing method against a higher order reference method, if available. In circumstances where a higher order methodology does not exist, the comparison is made against diagnostic accuracy criterion, which refers to a final positive/negative classification of a disease based on combination of the results from different methods suiting the purpose. The estimates of diagnostic accuracy of high level comparison are expressed as sensitivity and specificity, wherein sensitivity (true positive rate) measures the proportion of positives and specificity (true negative rate) measures the proportion of negatives that are correctly identified as such.

 

Low level comparison is characterized by the evaluation of a testing method against a comparative method already in use. The estimates of low-level comparison are expressed in terms of positive and negative agreement and not as sensitivity and specificity, since information on correctness of the testing method is not available against a confirmed diagnosis.

 

In the present study, HIV combo assay does not have a higher order reference method, but a well-defined and established diagnostic accuracy criterion for HIV is available with NACO in the form of Algorithm III. Moreover, the diagnostic accuracy criteria have been established in the laboratory based on a combo of results from Abbott Architect HIV combo assay, VIDAS DUO HIV Ultra assay, and Bio-Rad Geenius HIV 1/2 confirmatory assay (Fig. 1).

 

A stepwise approach was adopted for comparison of accuracy of VITROS HIV combo assay against the diagnostic accuracy criteria. The first essential step in accuracy verification consisted of estimating the sensitivity and specificity of our testing method against the final diagnosis. The corresponding sensitivity and specificity yielded were (Sens_new) 100% and (Spec_new) 99.8%. This was followed by calculation of these estimates for the comparative method (Abbott Architect HIV assay). The estimates were found to have sensitivity (Sen_sold) of 99.8% and specificity (Spec_old) of 99.5%.

 

The next step of investigation comprised of comparing the sensitivities and specificities of old and new methods, otherwise known as paired sensitivites and specificities. The difference (D) between sensitivities of old and new methods (Sens_new-Sens_old) was 0% and the specificities of old and new methods (Spec_new-Spec_old) was 0.3%. The percentage of difference was accountable to the discrepancy in reporting false positive /negative results between the two methods including 29 patient samples.

 

The final goal of the study was to find out whether the differences in paired sensitivities and specificities were statistically significant. For this purpose, 95% CI of the paired data was calculated according to the statistical procedures suggested by CLSI EP12A2. The 95% CI for difference between paired sensitivities was -4.93 to +4.93. Since the actual difference of paired sensitivities was zero, the sensitivity difference was deemed to be statistically insignificant.

 

With respect to specificity comparison, the 95% CI for difference between paired specificities was -0.43 to +0.57. Since the actual difference of paired sensitivities was 0.3%, the specificity difference was deemed to be statistically insignificant.

 

Competing interests

The authors declare that they have no competing interest

 

References

1.     WHO | HIV/AIDS [Internet]. WHO. [cited 2020 Jan 3]. Available from: http://www.who.int/gho/hiv/en/

2.     Davis DHJ, Smith R, Brown A, Rice B, Yin Z, Delpech V. Early diagnosis and treatment of HIV infection: magnitude of benefit on short-term mortality is greatest in older adults. Age Ageing. 2013 Jul;42(4):520–6.

3.     Alexander TS. Human Immunodeficiency Virus Diagnostic Testing: 30 Years of Evolution. Clin Vaccine Immunol. 2016 Apr;23(4):249–53.

4.     National_Guidelines_for_HIV_Testing_21Apr2016.pdf [Internet]. [cited 2020 Jan 3]. Available from: http://www.naco.gov.in/sites/default/files/National_Guidelines_for_HIV_Testing_21Apr2016.pdf

5.     Carey and Clinical and Laboratory Standards Institute - 2014 - User verification of precision and estimation of b.pdf [Internet]. [cited 2020 Jan 3]. Available from: https://clsi.org/media/1431/ep15a3_sample.pdf

6.     Clinical and Laboratory Standards Institute. User protocol for evaluation of qualitative test performance; approved guideline-2nd (ed) Wayne, PA, USA: CLSI; 2008. CLSI document EP12-A2.

7.     HIV Combo Reagent Pack, Pub No. GEM1255_XUS-EN, version 3.0, page: 1-18

8.     HIV Ag/Ab Combo, Pub No. 48-3504/R4, Page:1-8.

9.     HIV DUO Ultra, Ref 3044309691 I – en – 2015/06, Biomerieux SA

10.   Geenius HIV ½ Confirmatory Assay, 883601-2013/01, Page: 1-1