A practical guide to sensitivity, specificity, predictive values, discrimination, calibration, thresholds, uncertainty, and workflow-level evidence for medical AI.
By ModAstera
11 Aug 2026
A medical AI model reports 94% accuracy. Is that good?
Not enough information.
The same number could describe a useful model, a model that misses most clinically important cases, or a model that predicts the majority class almost every time. It does not tell us how errors are distributed, whether predicted probabilities are trustworthy, how performance changes across sites, or whether the output improves the real workflow.
Medical AI evaluation should therefore begin with the decision the model is meant to support, not with a favorite metric. The right evidence depends on intended use, population, operating threshold, human role, and the consequences of false negatives, false positives, delayed review, and uncertain outputs.
Accuracy is the fraction of evaluated examples classified correctly. It is intuitive, but it combines different errors into one average.
Imagine a dataset where 95 of every 100 examples are negative. A model that predicts every example as negative reaches 95% accuracy while detecting none of the positive cases. The arithmetic is correct. The conclusion that the model is useful would not be.
Even in a balanced dataset, the same accuracy can arise from very different error patterns. One model may miss positive cases. Another may flag too many negative cases. Those failures can have different safety, workload, cost, and user-trust consequences.
This is why a credible evaluation should show the confusion matrix at a stated operating threshold and report the errors separately.
Before selecting metrics, define:
A triage system that prioritizes cases for review has a different objective from a confirmatory classifier. A quality-control model that rejects poor inputs has a different objective from a risk score. A model that proposes a region for expert review should not be evaluated as though it makes an autonomous diagnosis.
This intended-use framing is central to validation-first medical AI. Metrics become meaningful only when they are connected to the action, user, and failure boundary they are supposed to support.
For a binary classification task, the confusion matrix separates four outcomes:
Several familiar metrics follow from these counts.
Sensitivity, also called recall for the positive class, asks: among positive cases, how many did the model identify?
Specificity asks: among negative cases, how many did the model correctly leave negative?
Precision, often equivalent to positive predictive value in a defined evaluation population, asks: among positive predictions, how many were actually positive?
Negative predictive value asks: among negative predictions, how many were actually negative?
These metrics answer different questions. Increasing sensitivity by lowering a threshold often increases false positives and reduces specificity. Raising the threshold may reduce review burden but increase missed positives. There is no universally correct tradeoff. The acceptable balance depends on the intended workflow and the available fallback or human-review process.
Always report the threshold used. Sensitivity or specificity without the operating threshold and evaluated population is incomplete evidence.
Discrimination describes how well a model ranks examples, placing positives above negatives across possible thresholds.
The area under the receiver operating characteristic curve, or AUROC, is a common discrimination measure. It is useful for comparing ranking behavior, but it does not tell a team which threshold to deploy. Two models with similar AUROC can behave differently at the high-sensitivity or high-specificity region that matters operationally.
AUROC can also look reassuring when the positive class is rare and the practical concern is the number of false alerts among positive predictions. Precision-recall curves can make that burden more visible because precision responds directly to false positives and class prevalence.
Neither curve replaces threshold-specific reporting. A review package should include the part of the curve relevant to intended use, the selected operating point, and the confusion-matrix counts behind the headline rates.
A model can rank cases well and still produce unreliable probability estimates.
Calibration asks whether predicted risks match observed frequencies. Among cases assigned a predicted probability near 0.8, approximately 80% should have the outcome in a well-calibrated model for that population and context.
A poorly calibrated score can create false confidence even when AUROC is strong. This matters when probability influences escalation, scheduling, consent discussions, resource allocation, or a decision to abstain.
Useful calibration evidence may include:
Calibration should be assessed on data not used to fit the calibration transformation. It can also deteriorate after deployment as prevalence, acquisition, or workflow changes. Model monitoring should therefore track both input conditions and outcome-linked performance where labels become available.
Sensitivity and specificity are properties of performance in an evaluated population, but predictive values depend strongly on prevalence.
If a condition is rare, even a model with high sensitivity and specificity can generate more false-positive than true-positive alerts. If a test dataset artificially balances classes, its precision and negative predictive value may not describe routine use.
Teams should report the evaluation prevalence and explain whether it reflects the intended deployment population. When a curated challenge set changes class balance to expose failure modes, that is useful, but predictive values from that set should not be presented as ordinary operational yield.
Case mix also matters beyond a single prevalence number. Severity, referral patterns, inclusion criteria, missingness, device type, image quality, and site workflow can all change observed performance. The population definition belongs next to the metric, not in an appendix nobody reads.
A threshold is a model-development decision.
It may be chosen to meet a minimum sensitivity, cap false-positive workload, optimize a stated utility function, or define an abstention region. Whatever the rule, it should be selected using training and validation evidence, not by searching the final test set for the most attractive result.
The final test set should estimate performance after the model, preprocessing, calibration method, threshold, and evaluation plan are stable. If test results drive another round of threshold tuning, the set has become part of development and the strength of the final claim should be reconsidered.
Preserve:
That record connects performance claims to AI traceability instead of leaving the chosen operating point as an unexplained constant in code.
An aggregate result can hide concentrated failure.
Depending on intended use and lawful data availability, evaluate performance across relevant contexts such as:
The purpose is not to produce dozens of unstable slices and declare differences from noise. It is to test known risks and deployment conditions. Report the sample size and uncertainty for each slice. Small groups may support a risk signal or further-data requirement rather than a definitive comparison.
External evaluation at another site can strengthen evidence, but it does not establish universal generalizability. Claims should remain bounded by the populations, settings, and systems actually studied.
Point estimates alone can create false precision.
Confidence intervals help show how much a metric could vary because of finite sample size. They are especially important for sensitivity when there are few positive cases, for subgroup analyses, and when several records come from the same patient or case.
The method should respect the sampling structure. Treating many correlated images from one patient as independent can make uncertainty look smaller than it is. Patient- or case-level resampling may be more appropriate when those are the independent units.
Error review is equally important. Examine false negatives, false positives, high-confidence mistakes, and abstained cases. Look for repeated patterns involving artifacts, missing data, labels, acquisition conditions, or workflow shortcuts. A metric tells the team that a failure exists. Case review helps explain why.
A clinically adjacent product is usually more than a model endpoint.
If a human reviews, confirms, overrides, or escalates predictions, evaluation should include that system. Relevant measures may include:
A threshold that looks optimal offline may overload reviewers. A model that improves case ranking may still fail if users cannot understand uncertainty or if integration delays make the prediction irrelevant. Conversely, a model with modest standalone performance may provide value inside a carefully designed human-review workflow.
Model metrics, human factors, and workflow outcomes should be reported separately. One should not be used to imply another without evidence.
Before accepting a medical AI result, confirm:
Medical AI evaluation is not a contest to find the most impressive number. It is a structured argument that the model can support a defined decision under defined conditions, with understood errors and uncertainty.
Accuracy can be part of that argument, but it cannot carry it alone. Stronger evidence connects threshold-specific errors, discrimination, calibration, prevalence, subgroups, uncertainty, and workflow outcomes to the intended use. It also preserves enough lineage for another reviewer to understand how the result was produced.
If your team is preparing a medical AI benchmark or validation package, ModAstera can help review metric selection, threshold evidence, and traceability before a headline score hardens into a product claim.
Sizing a medical AI dataset is arithmetic. Keeping that dataset provable, versioned, and reusable across years of development is the part that fails, and it is what MAEA is built to hold together.
There is no single number. There is a data budget, a timeline, and an endpoint that finally makes the number computable. A full report on sizing evidence for regulated medical AI.
A practical guide to testing whether a locked medical-AI system transfers beyond its development data, including cohort design, thresholds, subgroup analysis, workflow evaluation, and failure review.
