A Software Tool for Exploring the Relation between Diagnostic Accuracy and Measurement Uncertainty
Theodora Chatzimichail, MD
tc@hcsl.com
Aristides T. Hatjimihail, MD, PhD
ath@hcsl.com
Hellenic Complex Systems Laboratory, Kostis Palamas 21, 66131 Drama, Greece.
Theodora Chatzimichail, MD
tc@hcsl.com
Aristides T. Hatjimihail, MD, PhD
ath@hcsl.com
Hellenic Complex Systems Laboratory, Kostis Palamas 21, 66131 Drama, Greece.
Abstract
Abstract
Screening and diagnostic tests are used to classify people with and without a disease. Diagnostic accuracy measures are used to evaluate the correctness of a classification in clinical research and practice. Although this depends on the uncertainty of measurement, there has been limited research on their relation. The objective of this work was to develop an exploratory tool for the relation between diagnostic accuracy measures and measurement uncertainty, as diagnostic accuracy is fundamental to clinical decision-making, while measurement uncertainty is critical to quality and risk management in laboratory medicine. For this reason, a freely available interactive program was developed for calculating, optimizing, plotting and comparing various diagnostic accuracy measures and the corresponding risk of diagnostic or screening tests measuring a normally distributed measurand, applied at a single point in time in non-diseased and diseased populations. This is done for differing prevalence of the disease, mean and standard deviation of the measurand, diagnostic threshold, standard measurement uncertainty of the tests and expected loss. The application of the program is illustrated with a case study of glucose measurements in diabetic and non-diabetic populations. The program is user-friendly and can be used as an educational and research tool in medical decision-making.
Keywords
Keywords
diagnostic accuracy measures; ROC curve; measurement uncertainty; diagnostic tests; screening tests; risk
1. Introduction
1. Introduction
An increasing number of in vitro screening and diagnostic tests are extensively used as binary classifiers in medicine, to classify people into the non-overlapping classes of populations with and without a disease, which are categorized as quantitative and qualitative. The quantitative and many qualitative screening or diagnostic tests are based on measurements. There is a probability distribution of the measurements in each of the diseased and non-diseased populations. To classify the patients with and without a disease, using a test based on a measurement, a diagnostic threshold or cutoff point is defined. If the measurement is above the threshold, the patient is classified as test-positive; otherwise, the patient is classified as test-negative (Figure 1) or inversely. The possible test results are summarized in Table 1.
Out[]=
Figure 1. Probability density function plots . The probability density function plots of a measurand in a non - diseased and diseased population.
populations | |||
nondiseased( D | diseased(D) | ||
testresult | negative( T | truenegative(TN) | falsenegative(FN) |
positive(T) | falsepositive(FP) | truepositive(TP) | |
Table 1. A 2 x 2 contingency table
1.1 Diagnostic Accuracy Measures (DAM)
1.1 Diagnostic Accuracy Measures (DAM)
From the large number of diagnostic accuracy measures appearing in the literature, only a few are used for evaluating the diagnostic accuracy in clinical research and practice [1] . These include the following : 1. Sensitivity (Se), specificity (Sp), diagnostic odds ratio (DOR), likelihood ratio for positive or negative result (LR + and LR −, respectively), which are defined conditionally on the true disease status [2] and are prevalence invariant . 2. Overall diagnostic accuracy (ODA), which is defined conditionally on the true disease status and is prevalence - dependent . 3. Positive predictive and negative predictive value (PPV and NPV), which are defined conditionally on the test outcome and are prevalence - dependent . The natural frequency and the equivalent probability definitions of the diagnostic accuracy measures derived from Table 1 and analyzed by the program are presented in Table 2.
measure | naturalfrequencydefinition | probabilitydefinition |
Sensitivity(Se) | TN FP+TN | Pr(T|D) |
Specificity(Sp) | TP FP+TP | Pr( T D |
PositivePredictiveValue(PPV) | TN FN+TN | Pr(D|T) |
NegativePredictiveValue(NPV) | TN FN+TN | Pr( D T |
OverallDiagnosticAccuracy(ODA) | TN+TP FN+FP+TN+TP | Pr( T D D |
DiagnosticOddsRatio(DOR) | TN TP FN FP | Pr T D Pr T D |
LikelihoodRatioforaPositiveResult(LR+) | TP(FP+TN) FP(FN+TP) | Pr(T|D) PrT D |
LikelihoodRatioforaPositiveResult(LR−) | FN(FP+TN) TN(FN+TP) | Pr T Pr T D |
Youden’sIndex(J) | TN TP-FN FP (FP+TN)(FN+TP) | Pr( T D |
EuclideanDistance(ED) | 2 FN FN+TP 2 FP TN+FP | Pr 2 T 2 Pr(T| D |
ConcordanceProbability(CZ) | TNTP (FP+TN)(FN+TP) | Pr( T D |
Risk(R) | l 0 FN l FN l FP l TN l TP FN+FP+TN+TP | l 0 T l FN D D l FP T D D l TN l TP |
Table 2. Natural frequency and probability definitions of diagnostic accuracy measures.
1.2 Receiver operating characteristic (ROC) curves
1.2 Receiver operating characteristic (ROC) curves
Receiver operating characteristic curves are also used for the evaluation of the diagnostic performance of a screening or diagnostic test [3] . ROC curves are plots of Se against 1 - Sp of the test . A related summary measure of diagnostic accuracy is the area under a ROC curve (AUC) [4, 5] . The area over a ROC curve (AOC) has been proposed as a complementary summary measure of the diagnostic inaccuracy [6] . Recently, the predictive receiver operating characteristic (PROC) curves have also been proposed . PROC curves are plots of PPV against 1 - NPV of the test [2] .
1.3 Objective Functions
1.3 Objective Functions
For the optimization of binary classifiers, objective or loss functions have been proposed . They are based on diagnostic accuracy measures that can be maximized or minimized by finding the optimal diagnostic threshold . These measures include Youden' s index (J)[7], Euclidean distance of a ROC curve point from the point (0, 1) (ED)[8] and the concordance probability measure (CZ)[9] . The abovementioned measures are defined conditionally on the true disease status and are prevalence invariant . Their respective probability and natural frequency definitions are presented in Table 2.
The risk (R) of a diagnostic or screening test is related to its diagnostic accuracy and is defined as its expected loss . Therefore, it depends upon the following (Table 2) :
1. The expected loss for the testing procedure, for a true negative result, for a false negative result, for a true positive result and for a false positive result, defined on the same scale .
2. The probabilities for a true negative result, for a false negative result, for a true positive result and for a false positive result .
Risk is defined conditionally on the true disease status and is prevalence - dependent .
The risk (R) of a diagnostic or screening test is related to its diagnostic accuracy and is defined as its expected loss . Therefore, it depends upon the following (Table 2) :
1. The expected loss for the testing procedure, for a true negative result, for a false negative result, for a true positive result and for a false positive result, defined on the same scale .
2. The probabilities for a true negative result, for a false negative result, for a true positive result and for a false positive result .
Risk is defined conditionally on the true disease status and is prevalence - dependent .
1.4 Measurement Uncertainty
1.4 Measurement Uncertainty
As there is inherent variability in any measurement process, there is measurement uncertainty, which is defined as a "parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand" [10]. The parameter may be the standard measurement uncertainty (u), expressed as a standard deviation and estimated as described in "Expression of Measurement Uncertainty in Laboratory Medicine" [11]. Bias may be considered as a component of the standard measurement uncertainty [12].
The measurement uncertainty is gradually replacing the total analytical error concept [13].
The measurement uncertainty is gradually replacing the total analytical error concept [13].
1.5 Relation between Diagnostic Accuracy and Measurement Uncertainty
1.5 Relation between Diagnostic Accuracy and Measurement Uncertainty
Although the estimation of measurement uncertainty is essential for quality assurance in laboratory medicine [11], its effect on clinical decision-making and consequently on clinical outcomes is rarely quantified [14]. As direct-outcome studies are very complex, a feasible first step is exploring the effect of measurement uncertainty on misclassification [15] and subsequently on diagnostic accuracy measures and the corresponding risk. Exploring this relation could assist the process of estimation of the optimal diagnostic threshold or the permissible measurement uncertainty.
2. Methods
2. Methods
For the calculation of the diagnostic accuracy measures, the following is assumed:
1. There is a reference (“gold standard”) diagnostic method classifying correctly a subject as diseased or non-diseased [16].
2. The parameters of the distributions of the measurand are known.
3. Either the values of the measurand or their transforms [17,18] are normally distributed in each of the diseased and non-diseased populations.
4. The measurement uncertainty is normally distributed and homoscedastic in the diagnostic threshold’s range.
5. If the measurement is above the threshold the patient is classified as test-positive otherwise as test-negative.
Hereafter, we use the term measurand to describe either the normally distributed value of a measurand or its normally distributed applicable transform.
1. There is a reference (“gold standard”) diagnostic method classifying correctly a subject as diseased or non-diseased [16].
2. The parameters of the distributions of the measurand are known.
3. Either the values of the measurand or their transforms [17,18] are normally distributed in each of the diseased and non-diseased populations.
4. The measurement uncertainty is normally distributed and homoscedastic in the diagnostic threshold’s range.
5. If the measurement is above the threshold the patient is classified as test-positive otherwise as test-negative.
Hereafter, we use the term measurand to describe either the normally distributed value of a measurand or its normally distributed applicable transform.
Consequently, if σ is the standard deviation of the measurements of a screening or diagnostic test applied to a population (P), u the standard measurement uncertainty and the standard deviation of the measurand in the population, then we get the following equation:
σ
P
σ=+
2
σ
P
2
u
The definitions of the diagnostic accuracy measures can be expressed in terms of sensitivity (Se) and specificity (Sp). These definitions are derived from Table 2 and presented in Table 3.
measure | definition |
PositivePredictiveValue(PPV) | rSe rSe+(1-r)(1-Sp) |
NegativePredictiveValue(NPV) | (1-r)Se r(1-Se)+(1-r)Sp |
OverallDiagnosticAccuracy(ODA) | rSe+(1-r)Sp |
DiagnosticOddsRatio(DOR) | Se 1-Se 1-Sp Sp |
LikelihoodRatioforaPositiveResult(LR+) | Se 1-Sp |
LikelihoodRatioforaPositiveResult(LR−) | 1-Se Sp |
Youden’sIndex(J) | Se+Sp-1 |
EuclideanDistance(ED) | 2 (1-Se) 2 (1-Sp) |
ConcordanceProbability(CZ) | SeSp |
Risk(R) | l 0 l TN l FN l TP l FP |
Table 3. Definitions of diagnostic accuracy measures against sensitivity and specificity.
The functions of sensitivity (Se) and specificity (Sp), hence the functions of all the above diagnostic accuracy measures, can be expressed in terms of the cumulative distribution function of the normal distribution and therefore of the error function and the complementary error function.
The error function, erf (x), is defined as follows:
The error function, erf (x), is defined as follows:
erf(x)=dt, x≥0
2
π
x
∫
0
-
2
t
e
while the complementary error function, erfc (x), is defined as follows:
erfc(x)=1-erf(x)=dt, x≥0
2
π
∞
∫
x
-
2
t
e
Following the definition of the sensitivity and specificity of a test (Table 2), the respective functions against diagnostic threshold (d) are calculated as follows:
se(d,, ,u)=1-Ψd,,+=1+erfsp(d,, ,u)=Ψd,,+=erfc
μ
D
σ
D
μ
D
2
σ
D
2
u
1
2
-d+
μ
D
2(+)
2
σ
D
2
u
μ
-
D
σ
-
D
μ
-
D
2
σ
-
D
2
u
1
2
-d+
μ
-
D
2+
2
σ
-
D
2
u
where Ψ denotes the cumulative distribution function of a normal distribution; the mean and the standard deviation of the measurand of the test in the diseased population; the mean and the standard deviation of the measurand of the test in the non-diseased population; and u the standard measurement uncertainty of the test.Then, the sensitivity function of a test against its specificity (z) is calculated as follows:
μ
D
σ
D
μ
-
D
σ
-
D
se
sp
z,μ
D
σ
D
μ
-
D
σ
-
D
-1
Ψ
μ
-
D
2
σ
-
D
2
u
μ
D
2
σ
D
2
u
1
2
μ
D
μ
-
D
2+
+2
σ
-
D
2
u
-1
erfc
2(+)
2
σ
D
2
u
The specificity function of a single test against its sensitivity (y) is calculated as follows:
Following Table 3 and the above equations, the diagnostic accuracy measures of a test are defined as functions of either its diagnostic threshold, sensitivity, or specificity. Consequently, the derived parametric equations defining each measure can be used to explore the relations between any two measures.
Following the definition of the ROC curves and assuming a normal probability density function of the measurands of each of the diseased and non-diseased populations, the ROC function is calculated as follows:
Following the definition of the ROC curves and assuming a normal probability density function of the measurands of each of the diseased and non-diseased populations, the ROC function is calculated as follows:
where S denotes the survival function of normal distribution.
Consequently, we get the following:
Consequently, we get the following:
The function of the area under the ROC curve is defined as follows:
where Φ denotes the cumulative distribution function of the standard normal distribution.
The function of the area over the ROC curve is defined as follows:
The function of the area over the ROC curve is defined as follows:
Another ROC curve related quantity is the Euclidean distance (ED) of a ROC curve point from the point (0, 1) or equivalently the Euclidean distance of the point (Se, Sp) from the point (1, 1) of perfect diagnostic accuracy. The respective function is defined as follows:
The predictive ROC (PROC) curve relation is defined as follows [2]:
This relation cannot be expressed in terms of elementary or survival functions.
To explore the relation between diagnostic accuracy measures or the corresponding risk and measurement uncertainty, an interactive program written in Wolfram Language [19] was developed in Wolfram Mathematica®, ver. 12.2 [20]. This program was designed to provide five modules and six submodules for calculating, optimizing, plotting and comparing various diagnostic accuracy measures and the corresponding risk of two screening or diagnostic tests, applied at a single point in time in non-diseased and diseased populations (Figure 2). The two tests measure the same measurand, for varying values of the prevalence of the disease, the mean and standard deviation of the measurand in the populations and the standard measurement uncertainty of the tests. The two tests differ in measurement uncertainty. It is assumed that the measurands and the measurement uncertainty are normally distributed.
Parts of this program have been presented in a series of demonstrations, at Wolfram Demonstration Project of Wolfram Research [6,21-27].
Parts of this program have been presented in a series of demonstrations, at Wolfram Demonstration Project of Wolfram Research [6,21-27].
3. Results
3. Results
Illustrative Case Study
Illustrative Case Study
The default settings of the parameters of the program were derived as following:
1. The bimodal normal distribution was based on log-transformed blood glucose measurements in non-diabetic and diabetic Malay populations, during an oral glucose tolerance test (OGTT) [28]. Briefly, after the ingestion of 75 g glucose monohydrate, the two-hour postprandial blood glucose of 2667 Malay adults, aged 40-49 years, was measured with reflectance photometry. To apply the program, it was assumed that the prevalence of diabetes was 0.067, the measurement coefficient of variation and bias were equal to 4% and 2%, respectively and the log-transformed measurands of each population were normally distributed, as shown in Figure 1. The standardized log-transformed measurand means and standard deviations in the diseased and non-diseased populations, the standard measurement uncertainty and the diagnostic threshold were expressed in units equal to the standard deviation of the log-transformed measurand in the non-diseased population.
2. The standardized log-transformed diagnostic threshold 2.26 corresponds to the American Diabetes Association (ADA) diagnostic threshold for diabetes of the two-hour postprandial glucose during OGTT that is equal to 11.1 mmol/L [29].
3.The standardized log-transformed standard measurement uncertainties 0.023 and 0.23 correspond to standard measurement uncertainties equal to 1% and 10% of the mean of the measurand of the non-diabetic population or equivalently to a coefficient of variation equal to 1% and 10%, respectively.
1. The bimodal normal distribution was based on log-transformed blood glucose measurements in non-diabetic and diabetic Malay populations, during an oral glucose tolerance test (OGTT) [28]. Briefly, after the ingestion of 75 g glucose monohydrate, the two-hour postprandial blood glucose of 2667 Malay adults, aged 40-49 years, was measured with reflectance photometry. To apply the program, it was assumed that the prevalence of diabetes was 0.067, the measurement coefficient of variation and bias were equal to 4% and 2%, respectively and the log-transformed measurands of each population were normally distributed, as shown in Figure 1. The standardized log-transformed measurand means and standard deviations in the diseased and non-diseased populations, the standard measurement uncertainty and the diagnostic threshold were expressed in units equal to the standard deviation of the log-transformed measurand in the non-diseased population.
2. The standardized log-transformed diagnostic threshold 2.26 corresponds to the American Diabetes Association (ADA) diagnostic threshold for diabetes of the two-hour postprandial glucose during OGTT that is equal to 11.1 mmol/L [29].
3.The standardized log-transformed standard measurement uncertainties 0.023 and 0.23 correspond to standard measurement uncertainties equal to 1% and 10% of the mean of the measurand of the non-diabetic population or equivalently to a coefficient of variation equal to 1% and 10%, respectively.
In this case, the measurement uncertainty has relatively little effect on the ROC and PROC curves, on AUC, sensitivity, specificity, overall diagnostic accuracy, positive predictive value, negative predictive value, Euclidean distance and concordance probability of the test, in accordance with previous findings [30,31]. Measurement uncertainty has a relatively greater effect on diagnostic odds ratio, on likelihood ratio for a positive or negative result, Youden's index and risk.
As a result, the measurement uncertainty has relatively little effect on the optimal diagnostic thresholds maximizing the Youden's index or the concordance probability or minimizing the Euclidean distance. Conversely, it has a relatively greater effect on the optimal diagnostic thresholds minimizing risk.
As a result, the measurement uncertainty has relatively little effect on the optimal diagnostic thresholds maximizing the Youden's index or the concordance probability or minimizing the Euclidean distance. Conversely, it has a relatively greater effect on the optimal diagnostic thresholds minimizing risk.
4. Discussion
4. Discussion
The purpose of this program is to explore the relation between diagnostic accuracy measures and measurement uncertainty, as diagnostic accuracy is fundamental to clinical decision-making, while defining the permissible measurement uncertainty is critical to quality and risk management in laboratory medicine. The current pandemic of the novel corona virus disease 2019 (COVID-19) has demonstrated these convincingly [32-37].
There has been extensive research on either diagnostic accuracy or measurement uncertainty; however, such research is very limited on both subjects [14,38,39].
This program demonstrates the relation between the diagnostic accuracy measures and the measurement uncertainty for screening or diagnostic tests measuring a single measurand (Figures 3-19). This relation depends on the population parameters, including the prevalence of the disease and on the diagnostic threshold (Figures 4, 15 and 16). In addition, measurement uncertainty affects the relation between any two of the diagnostic accuracy measures.
This program demonstrates the relation between the diagnostic accuracy measures and the measurement uncertainty for screening or diagnostic tests measuring a single measurand (Figures 3-19). This relation depends on the population parameters, including the prevalence of the disease and on the diagnostic threshold (Figures 4, 15 and 16). In addition, measurement uncertainty affects the relation between any two of the diagnostic accuracy measures.
As the program provides plots of the partial derivative of the diagnostic accuracy measures with respect to the standard measurement uncertainty, it offers a more detailed insight (Figure 16). In antithesis to the complexity of the relation, the program simplifies its exploration with a user-friendly interface.
Furthermore, it provides calculators for the calculation of the effects of measurement uncertainty on the diagnostic accuracy measures and corresponding risk and for calculating the diagnostic threshold optimizing the objective and loss functions of Section 2.
Furthermore, it provides calculators for the calculation of the effects of measurement uncertainty on the diagnostic accuracy measures and corresponding risk and for calculating the diagnostic threshold optimizing the objective and loss functions of Section 2.
The counterintuitive finding that the measurement uncertainty has relatively little effect on the ROC and PROC curves, on AUC, sensitivity, specificity, overall diagnostic accuracy, positive predictive value, negative predictive value, Euclidean distance and concordance probability suggests that we should reconsider their interpretation in medical decision-making. However, further research is needed to explore the effect of measurement uncertainty on diagnostic accuracy measures with different clinically and laboratory relevant parameter settings. Furthermore, clinical laboratories should consider including measurement uncertainty in each test result report.
Compared to the risk measure, a shortcoming of Youden's index, Euclidean distance of a ROC curve point from the point (0, 1) and concordance probability as objective functions is that they do not differentiate the relative significance of a true negative and a true positive test result or equivalently of a false-negative and a false-positive test result. Accordingly, in the case study, the optimal diagnostic thresholds maximizing the Youden's index or the concordance probability or minimizing the Euclidean distance are considerably less than the ADA diagnostic threshold for diabetes of the two-hour postprandial glucose during OGTT (Table 5). Nevertheless, the optimal diagnostic threshold minimizing the risk can be close to the ADA threshold, with specific expected loss settings (Figure 11). Although risk assessment is evolving as the preferred method for optimization of medical decision-making [40] and for quality assurance in laboratory medicine [41], the estimation of expected loss for each test result (Tables 2 and 3) is still a complex task. In the future, as the potential of the data analysis will increase exponentially, expected loss could be estimated by using evidence-based methods.
As the program neither estimates the parameters of the distributions of the measurand, nor calculates any confidence intervals, it is not intended to analyze samples of measurements, but to be used as an educational and research tool, to explore and analyze the relation between diagnostic accuracy measures and measurement uncertainty.
All major general or medical statistical software packages (Matlab®, NCSS®, R, SAS®, SPSS®, Stata® and MedCalc®) include routines for the calculation and plotting of various diagnostic accuracy measures and their confidence intervals. The program presented in this work provides 269 different types of plots of diagnostic accuracy measures, many of which are novel. To the best of our knowledge, not one of the abovementioned programs or any other software provides this range of plots without advanced statistical programming.
5. Conclusion
5. Conclusion
The program developed for this work clearly demonstrates various aspects of the relation between diagnostic accuracy measures and measurement uncertainty and can be used as a flexible, user-friendly, interactive educational or research tool in medical decision-making, to explore and analyze this relation.
6. The Program
6. The Program
6.1 Program Functions
6.1 Program Functions
6.2 Program Interface
6.2 Program Interface
7. References
7. References
8.1 Populations
8.2 Test Outcomes
8.3 Diagnostic Accuracy Measures
DAM : diagnostic accuracy measure
Se : sensitivity
Sp : specificity
PPV : positive predictive value
NPV : negative predictive value
ODA : overall diagnostic accuracy
DOR : diagnostic odds ratio
LR + : likelihood ratio for a positive test result
LR − : likelihood ratio for a negative test result
J : Youden’ s index
ED : Euclidean distance of a ROC curve point from the point (0, 1)
CZ : concordance probability
R : risk
ROC : receiver operating characteristic curve
AUC : area under the ROC curve
AOC : area over the ROC curve
PROC : predictive receiver operating characteristic curve
Se : sensitivity
Sp : specificity
PPV : positive predictive value
NPV : negative predictive value
ODA : overall diagnostic accuracy
DOR : diagnostic odds ratio
LR + : likelihood ratio for a positive test result
LR − : likelihood ratio for a negative test result
J : Youden’ s index
ED : Euclidean distance of a ROC curve point from the point (0, 1)
CZ : concordance probability
R : risk
ROC : receiver operating characteristic curve
AUC : area under the ROC curve
AOC : area over the ROC curve
PROC : predictive receiver operating characteristic curve
8.5 Expected Loss
About
Relation
Version Number 1.0.1
Hellenic Complex Systems Laboratory 2020
First published in: Chatzimichail, T.; Hatjimihail, A.T. A Software Tool for Exploring the Relation between Diagnostic Accuracy and Measurement Uncertainty. Diagnostics 2020, 10, 610. https://doi.org/10.3390/diagnostics10090610
This notebook is distributed under an Attribution - NonCommercial - ShareAlike 4.0 International Creative Commons License.
Version Number 1.0.1
Hellenic Complex Systems Laboratory 2020
First published in: Chatzimichail, T.; Hatjimihail, A.T. A Software Tool for Exploring the Relation between Diagnostic Accuracy and Measurement Uncertainty. Diagnostics 2020, 10, 610. https://doi.org/10.3390/diagnostics10090610
This notebook is distributed under an Attribution - NonCommercial - ShareAlike 4.0 International Creative Commons License.