Paperonium

← Back to the plain-language case

VirtualSweet's reported cross-validation accuracy is impossible given its sensitivity and specificity; supplement shows best-fold values

Nucleic Acids Res (2021) · doi:10.1093/nar/gkab292 · PMC8262722

Summary

Table 1 reports 10-fold cross-validation results for the VirtualSweet model as accuracy 0.88, sensitivity 0.97 and specificity 0.96, and the text repeats the 88%. Accuracy is a prevalence-weighted average of sensitivity and specificity, so these three values cannot describe the same evaluation: with sensitivity 0.97 and specificity 0.96, accuracy must lie between 0.955 and 0.975 even allowing for rounding, and must be at least 0.920 even if the metrics are averages over folds. The paper's supplementary file S2 shows where the row comes from. Each Table 1 cross-validation entry matches the value S2 labels "best" for that metric, and these "best" values cannot all come from one fold. S2 gives the 10-fold mean VirtualSweet accuracy as 0.765 ± 0.114, not 0.88, so the cross-validation performance stated in Table 1, the text and the Conclusions is higher than the cross-validation average the paper's own supplement reports.

What the paper reports

Table 1, "Performance statistics for the VirtualTaste models applied to cross-validation and external validation sets" (excerpt, as archived in PMC):

VirtualTaste models | VirtualTaste models | VirtualSweet | VirtualBitter | VirtualSour
Data sampling method | Data sampling method | SMOTETC | SMOTE VDM | AugRandOS
Cross-validation | Prediction accuracy | 0.88 | 0.94 | 0.98
| Sensitivity | 0.97 | 0.94 | 0.94
| Specificity | 0.96 | 0.92 | 0.97
| ROC–AUC | 0.99 | 0.97 | 0.97
| F-measure | 0.87 | 0.94 | 0.98
External validation | Prediction accuracy | 0.89 | 0.90 | 0.97
| Sensitivity | 0.86 | 0.88 | 0.80
| Specificity | 0.92 | 0.97 | 0.99
| ROC-AUC | 0.95 | 0.96 | 0.99
| F-measure | 0.88 | 0.88 | 0.84

Materials and Methods, "VirtualTaste prediction models", VirtualSweet subsection:

The model achieved a prediction accuracy of 88% on 10-fold cross-validation and 89% on external validation. The AUC–ROC values of cross-validation and external validation are 0.95 and 0.99, respectively (Table 1).

Materials and Methods, "Performance evaluation of the VirtualTaste models", first paragraph:

Each model was validated using 8 different sampling methods for 10-fold cross-validation (22). The data was split into 10-fold, keeping the ratio of active and inactive data constant as published in our previous work (22).

Same section, definition of the metrics:

Prediction Accuracy is defined as the ability of a model to correctly predict the total number of activities and inactives. Sensitivity is the trained model's ability to correctly predict the positive (taste) class. Specificity is defined as the trained model's ability to correctly predict the negative (non-taste) class.

Same section, second paragraph:

The performance statistics both on cross-validation as well as external validation for the top- performing VirtualTaste models are summarized in Table 1. The models achieved a prediction accuracy of 88% and above on both, cross-validation and external validation. The specificity and sensitivity of the VirtualTaste models are balanced and scored 90% and above, except for the sensitivity of VirtualSweet (86%), VirtualBitter (88%) and VirtualSour (80%) on the external validation set.

Materials and Methods, "VirtualTaste prediction models", first paragraph:

The standard deviation and average of the accuracy and other parameters loss are computed (See Supplementary data).

Abstract:

The VirtualTaste models achieved an overall accuracy of 90% and an AUC of 0.98 in 10-fold cross-validation and in an independent test set.

Conclusions and Future Updates, first paragraph:

The models achieved a prediction accuracy of 88% and above both on cross-validation and external validation.

Supplementary file S2.xlsx (Sheet1), VirtualSweet block, row "SMOTETC", internal cross-validation cells (int. ROC-AUC, int. Acc, int. Sens, int. Spec, int. F-Measure):

0.833+/-0.124 | best:0.990 · 0.765+/-0.114 | best:0.877 · 0.740+/-0.185 | best:0.977 · 0.791+/-0.165 | best:0.963 · 0.751+/-0.129 | best:0.870

What the paper's own numbers show

Method. For a single binary evaluation with a share p of positives, accuracy = p × sensitivity + (1 − p) × specificity, so accuracy must lie between sensitivity and specificity for every p. I applied this check to all six rows of Table 1, widening every bound by ±0.005 for two-decimal rounding. For the case where the three metrics are each averaged over the 10 folds, I used the per-fold inequality accuracy ≥ min(sensitivity, specificity) ≥ sensitivity + specificity − 1, which carries over to the averages. I then compared each Table 1 entry with the supplementary file S2.xlsx for the sampling method Table 1 names (SMOTETC, SMOTEVDM and AugRandOS). Every number below is printed by recompute.py.txt.

Quantity Location Reported Recomputed from the paper's own numbers
VirtualSweet CV accuracy Table 1; VirtualSweet subsection ("88%") 0.88 must lie in 0.960 to 0.970 for any prevalence (0.955 to 0.975 with rounding) given sensitivity 0.97 and specificity 0.96
Prevalence needed for accuracy 0.88 derived — p = (0.88 − 0.96) ÷ (0.97 − 0.96) = −8.0, which is impossible
Gap after allowing for rounding derived ≤ 0.885 lowest possible accuracy 0.955; gap 0.070
Lower bound if metrics are fold averages derived — mean accuracy ≥ 0.965 + 0.955 − 1 = 0.920
Balanced accuracy derived — (0.97 + 0.96) ÷ 2 = 0.965
VirtualSweet CV F-measure Table 1 0.87 fits sensitivity 0.97 and specificity 0.96 only at a positive share of 0.133, where accuracy is 0.961; at p = 0.5 F1 is 0.965
Misclassified training compounds (n = 1608) Datasets section accuracy 0.88 → 193 specificity and sensitivity ≥ 0.96 → at most 64
Other five rows (CV Bitter, CV Sour, external Sweet, Bitter, Sour) Table 1 accuracy 0.94, 0.98, 0.89, 0.90, 0.97 each within its sensitivity–specificity range; CV Sour (0.98 vs 0.94 and 0.97) only at the rounding edge (0.975)
VirtualSweet CV entries vs S2 "best" Table 1 vs S2, SMOTETC 0.88, 0.97, 0.96, 0.99, 0.87 S2 "best": 0.877, 0.977, 0.963, 0.990, 0.870 (accuracy, sensitivity, specificity, AUC, F)
Can the S2 "best" values come from one fold? S2, SMOTETC — No: with sensitivity 0.977 and specificity 0.963 a single fold's accuracy lies in 0.963 to 0.977, not 0.877
VirtualSweet CV, 10-fold mean ± SD S2, SMOTETC — accuracy 0.765 ± 0.114, sensitivity 0.740 ± 0.185, specificity 0.791 ± 0.165, AUC 0.833 ± 0.124, F 0.751 ± 0.129; the mean accuracy lies between the mean sensitivity and specificity, so these are mutually consistent
VirtualBitter and VirtualSour CV accuracy Table 1 vs S2 0.94; 0.98 S2 "best" 0.946 and 0.978; 10-fold means 0.845 ± 0.068 and 0.954 ± 0.092
External-validation rows Table 1 vs S2 all 15 cells all agree with the S2 external values to within 0.01, except VirtualBitter specificity (below)
VirtualSweet ROC-AUC, CV / external text vs Table 1 vs S2 text 0.95 / 0.99; Table 1 0.99 / 0.95 S2: CV best 0.990 (mean 0.833), external 0.951; the text has the two values the other way round

Table 1 thus reproduces the S2 "best" values for cross-validation and the single S2 values for external validation. For VirtualSweet, the "best" entries cannot all come from the same fold, because a fold with sensitivity 0.977 and specificity 0.963 cannot have accuracy 0.877. They must have been selected metric by metric. That explains why the printed row is impossible as a single evaluation. It also means the Table 1 "Cross-validation" row is not a cross-validation estimate in the usual sense, which is the average over folds. S2 gives that average for VirtualSweet as accuracy 0.765, which is 0.115 (about one fold SD) below the 0.88 in Table 1 and the text. S2 does not define "best", and it is not always the largest fold value: in two S2 rows the "best" value is below the mean (VirtualSour randOS accuracy, best 0.970 against mean 0.977, and, at the third decimal, VirtualSour AugRandOS sensitivity, 0.94 against 0.941).

The two-decimal Table 1 values follow the S2 values sometimes by rounding (0.877 to 0.88) and sometimes by truncation (sensitivity 0.977 printed as 0.97; VirtualBitter accuracy 0.946 printed as 0.94).

Figure

VirtualSweet's reported cross-validation accuracy lies far outside the range its own sensitivity and specificity allow

Figure 1. The VirtualSweet 10-fold cross-validation row of Table 1. Orange points are the printed sensitivity (0.97), specificity (0.96) and accuracy (0.88). The blue bar is the range that accuracy must take, for any share of sweet compounds and allowing ±0.005 for rounding: 0.955 to 0.975. The dashed blue line marks the weaker bound that still holds if accuracy, sensitivity and specificity are each averaged over the 10 folds: accuracy ≥ 0.920. The printed 0.88 is outside both.

Table 1's cross-validation entries match the supplement's best-fold values, well above the supplement's 10-fold means

Figure 2. Internal 10-fold cross-validation for the three models, using the sampling method named in Table 1. Orange points are Table 1's "Cross-validation" entries. Open diamonds are the values supplementary file S2 labels "best". Open grey circles with bars are the S2 10-fold mean ± SD (the VirtualSour bar is cut at 1.0). Every Table 1 entry sits at the S2 "best" value, not at the mean. For VirtualSweet the means (accuracy 0.765, sensitivity 0.740, specificity 0.791, AUC 0.833, F-measure 0.751) are 0.115 to 0.230 below the printed values.

Why it matters

Explanations considered

Status

Suggested correction

In Table 1, either (a) replace the cross-validation rows with the 10-fold mean ± SD from S2. For VirtualSweet that is accuracy 0.77 ± 0.11, sensitivity 0.74 ± 0.19, specificity 0.79 ± 0.17, ROC-AUC 0.83 ± 0.12 and F-measure 0.75 ± 0.13. For VirtualBitter it is 0.85, 0.90, 0.79, 0.91 and 0.86; for VirtualSour 0.95, 0.94, 0.97, 1.00 (0.996) and 0.94. Or (b) keep the current values but relabel the rows as the best single-fold value of each metric, chosen separately, and state that the values in one column do not come from the same fold. Option (a) gives the conventional cross-validation estimate.

Correspondingly, in the VirtualSweet subsection, replace "a prediction accuracy of 88% on 10-fold cross-validation" with the mean (77%), and correct "The AUC–ROC values of cross-validation and external validation are 0.95 and 0.99, respectively" to the Table 1/S2 values (cross-validation 0.83 as a mean, or 0.99 as the best fold; external 0.95). Revise the "88% and above" and "balanced and scored 90% and above" sentences in the Methods and Conclusions, and the abstract's "overall accuracy of 90% and an AUC of 0.98 in 10-fold cross-validation", to match whichever summary is adopted. The authors may also wish to check the VirtualBitter specificity cells (Table 1: 0.92 cross-validation, 0.97 external; S2: 0.973 best, 0.919 external) and the VirtualSour cross-validation AUC (Table 1 and text 0.97; S2 0.996 mean, 0.998 best).