Early Sepsis Prediction with Machine Learning
Sepsis kills more hospital patients than any other single condition, and treating it an hour sooner saves lives. So a model that flags it early should be a slam dunk. Instead the largest real-world deployment scored barely above a coin flip, and the story of why is the best lesson clinical machine learning has to teach.
In 2019 a man walked into an emergency room with what looked like a bad case of the flu. The staff told him it was just a virus and sent him home. A few days later he was dead of sepsis. When the New York Times told his story, what stuck with me was not the missed diagnosis itself, which is tragically common, but a detail that surfaced when doctors discussed it afterward. The hospital’s electronic health record almost certainly fired a sepsis alert at some point. And almost certainly someone clicked it away.
I keep coming back to that, because it is the whole problem in miniature. We have spent more than a decade building machine-learning models to catch sepsis early, the single highest-value prediction task in hospital medicine, and the dominant story is not one of triumph. It is a story of a model that fired so often, so uselessly, that clinicians learned to dismiss it like a seatbelt chime. The technology arrived. The benefit mostly did not. Working out the gap between those two things turns out to teach you more about clinical machine learning than any success story could.1
What Sepsis Actually Is, and Why the Clock Matters
Sepsis is the body’s response to infection turning on itself. An infection somewhere, lungs, urinary tract, abdomen, bloodstream, triggers an immune response so dysregulated that it starts damaging the body’s own tissues and organs. The progression from infection to organ dysfunction to septic shock to death can happen with frightening speed. A patient who looks stable at 2 PM can be in multi-organ failure by 6 PM.2
The numbers are the reason anyone cares. Sepsis affects roughly 1.7 million adults in the US each year, kills around 270,000 of them, and runs up more than $60 billion in annual hospital costs, which makes it the most expensive condition treated in American hospitals. Globally the toll is far larger. And survival is exquisitely sensitive to time. The most cited figure, from Kumar et al (200620ya), is that each hour of delay in appropriate antibiotics during septic shock raises mortality by an estimated 4 to 8%.3
Survival in septic shock falls with each hour that passes before effective antibiotics. This time-dependence is the entire argument for early prediction: the value of a warning is highest in the first few hours, exactly the window where the diagnosis is hardest to make. Chart: Ian Furst, CC BY-SA 4.0, via Wikimedia Commons.
So the pitch writes itself. Here is a deadly, common, expensive condition whose outcome depends on catching it early, and whose early signs are buried in exactly the kind of high-dimensional, time-series clinical data that machine learning is supposed to be good at. If ML cannot help here, where can it help?
That question has a real answer, and it is more uncomfortable than the pitch suggests.
Why Early Detection Is Genuinely Hard
Sepsis is hard to catch early for a reason that has nothing to do with machine learning: its first signs are nonspecific. Fever, a fast heart rate, an elevated white blood cell count, low blood pressure. These are the classic SIRS criteria, and they show up in dozens of conditions that are not sepsis: post-surgical patients, dehydration, pain, anxiety, a transfusion reaction. The signal that says “this person is becoming septic” looks, in its early stages, almost identical to the noise of a busy hospital ward.
The medical community has tried to sharpen the definition. The current Sepsis-3 consensus, published in 2016, ditched the loose SIRS framing for something built on the Sequential Organ Failure Assessment (SOFA) score, which measures actual organ dysfunction. Sepsis-3 is more specific. It is also, for prediction purposes, almost self-defeating, because it tends to identify sepsis only once organ damage has already started. By the definition’s own logic, the moment you can confidently say “this is sepsis” is the moment you are no longer early.4
The workflow makes it worse. On a normal hospital ward, vital signs get measured every four to eight hours. Labs come back once or twice a day. Nursing notes live in free text that nobody is mining in real time. The data that would reveal a developing sepsis exists, but it arrives in dribs and drabs, scattered across systems, read by different people at different times, none of whom has the whole picture in their head at once.
This is, on paper, the perfect job for a model: stitch together intermittent, heterogeneous, time-stamped streams and surface a pattern a human would miss in the churn. The research literature is full of models that appear to do exactly that. The deployments are where the appearance and the reality come apart.
A bedside ICU monitor: heart rate, blood pressure, oxygen saturation, respiration. The data a sepsis model watches is mostly this, sampled intermittently and pooled with labs and notes. The signal of impending sepsis is in here somewhere, tangled up with everything else a sick patient’s body is doing. Photo: Quinn Dombrowski, CC BY-SA 2.0, via Wikimedia Commons.
The Model Landscape
The first generation of sepsis warnings were not machine learning at all. They were scoring systems: add up a few abnormal vital signs, ring a bell past a threshold. The Modified Early Warning Score (MEWS) and the qSOFA score are the canonical examples, simple enough to compute at the bedside on a notecard. They are also mediocre, with AUROC values typically in the 0.65 to 0.75 range and false-positive rates high enough to limit how much anyone trusts them.5
Machine learning promised to do better by throwing more variables and temporal structure at the problem. A handful of models defined the field:
InSight, from the company Dascena, used gradient-boosted trees on vital signs, labs, and demographics, and claimed an AUROC above 0.90 for predicting sepsis onset four hours ahead. It became one of the first ML sepsis tools to clear FDA review.
The Epic Sepsis Model (ESM), built by the company that dominates the US hospital EHR market, was switched on across hundreds of hospitals. That alone made it, by sheer reach, the most widely deployed clinical machine-learning model in history. Most clinicians who have ever been nagged by a sepsis pop-up were being nagged by this thing.
TREWS, the Targeted Real-Time Early Warning System out of work later commercialized through Bayesian Health, took a deep learning approach to EHR time series and, importantly, obsessed over deployment and clinician workflow rather than just the offline score.
And Google’s EHR work (Rajkomar et al, 2018) showed that deep nets fed the full firehose of longitudinal EHR data, including clinical notes via natural language processing, not just tidy vitals and labs, could predict a range of outcomes including sepsis. It was a proof of capacity more than a product.
The PhysioNet/Computing in Cardiology Challenge 2019 tried to make the field comparable by handing everyone the same benchmark, pooled from three hospital systems. The result was clarifying in a deflating way: even the winning models, scored on a custom utility function that rewarded early calls and penalized false alarms, would have produced two to three false alarms for every true one if you translated their scores into bedside terms.6 That is not a footnote-sized caveat. That is the ballgame.
The Epic Sepsis Model Audit
The most important paper in sepsis prediction is not a methods paper. It is an autopsy. In 2021 Wong et al published, in JAMA Internal Medicine, an external validation of the Epic Sepsis Model as it actually ran at the University of Michigan. They looked at 27,697 patients across 38,455 hospitalizations, of which sepsis occurred in 7%. The results were brutal.
The ESM scored an AUROC of 0.63. For a model Epic had validated at 0.76 to 0.83, that is a collapse, and 0.63 is close enough to a coin flip that the word “model” starts to feel generous. It missed 67% of sepsis cases outright, a sensitivity of 33%. And of the alerts it did fire, the overwhelming majority were wrong. The summary one Hacker News commenter (skmurphy) pasted straight from the paper said it plainly: “The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer.” Another commenter (kritiko) distilled the whole episode into one sentence: “A widely used sepsis prediction tool demonstrated poor performance in an independent validation, with worse results than clinician judgement and an increased burden of alert fatigue.” Worse than the doctors, and more annoying. That is the worst of both worlds.
How does a model lose 15 to 20 points of AUROC between the brochure and the bedside? Three failures, and they recur everywhere in clinical ML.
Distribution shift. The model was developed on one patient population and deployed to another. Case mix, documentation habits, lab-ordering patterns, the local definition of when someone “counts” as septic, all of it drifts between sites. A model that memorized the statistical fingerprints of its training hospital does not transfer, and one HN commenter (haldujai), a clinician, named exactly this as the thing that should temper radiology-AI hype too: “validating this across multiple institutions while making sure population drift doesn’t happen (see the Epic sepsis AI predicting failure in 2022) will take 5-10 years.”
Label leakage. This one is subtle and it is the deepest cut. Many sepsis models use antibiotic administration as part of the definition of the sepsis label they train against. But antibiotics get ordered because a clinician already suspected infection, based on the same vitals and labs the model reads. So a model that “predicts” antibiotic administration is not predicting sepsis. It is predicting that the doctor has already figured it out. The alert, when it fires, arrives after the human decision it was supposed to beat. It is a stopwatch that starts when the race is already over.7
Evaluation that measured the wrong thing. Epic’s validation, like most, was retrospective AUROC on an internal dataset, with labels stamped at fixed time points. That number cannot tell you whether the alert fires early enough to matter, whether the information is actionable, or whether the clinician already knew. It measures discrimination on history, not impact on the future. The Wong paper’s real contribution was to show that the entire standard evaluation ritual of clinical ML, the one that produced all those 0.9-something AUROCs in all those papers, was insufficient to tell you whether a model would help a single patient.8
A central monitoring station: one screen, many patients, every one of them generating the vitals a sepsis model ingests. Scale up the alerting and you scale up the false alarms in lockstep. This is the environment a sepsis warning has to earn attention in, and most of them don’t. US Navy photo by Shane T. McCoy, public domain, via Wikimedia Commons.
There is a more human version of this failure, and the clinicians on Hacker News told it better than any paper. Discussing the Times ER death, one physician (cfu28) described what the alert actually looks like at 3 AM: “You’ll get something like ‘sepsis criteria triggered by wbc 13, cr 1.5, hr 101, rr 22.’ And that’s it, usually in the middle of a night on a new patient I just got a page for. Can’t open documentation to see the patient’s med history. It’s ridiculous.” An anesthesiologist (devilbunny) described the lockout mechanics: the warning “prevent[s] you from leaving the screen until you have dealt with them,” and to clear it you might have to order the entire sepsis bundle or dismiss the alert permanently, sometimes against a supervising physician’s correct judgment to wait for labs first. A third (modmans2nd) offered the closest thing to a defense: “Prior to this algorithm most Epic customers had an alert for providers to check for Sepsis in a patient. In most cases the Epic tool is an improvement over [the previous] over-alerting.” Faint praise. The bar was on the floor.
Why This Is Harder Than the Pitch Admits
Strip away the Epic-specific failures and you are left with a few structural problems that make sepsis a genuinely treacherous target, problems no amount of better modeling fully dissolves.
When does sepsis “start”? The honest answer is that there is no clean onset. Infection and organ dysfunction develop gradually and asynchronously, so any “onset time” you pick to train against is a convention, not a fact. Worse, different onset definitions yield different datasets, which train different models, which report different metrics, all nominally on the “same” problem. A 2018 HN commenter (telchar), who had worked on this exact task, flagged the deeper issue with calling any of it “prediction” at all: “To properly claim prediction of the future disease state you would need to show that the patient does not have the disease at the time of the prediction,” which the papers mostly do not do. Much of “sepsis prediction” is closer to “sepsis recognition slightly faster than the doctor,” which is a real but far more modest thing.
The math of rare events is unforgiving. Sepsis hits maybe 2 to 7% of hospitalized patients, and only a sliver of any patient’s stay is the “pre-sepsis” window you care about. So your model has to find the relevant handful of hours among thousands of ordinary patient-hours. Even a genuinely accurate model, run at a sensitivity high enough to catch most real cases, will generate far more false alarms than true ones, simply because the negatives so vastly outnumber the positives. This is the base rate problem, and it is not a tuning issue you can engineer away. It is arithmetic.9
Alert fatigue is a real, measured phenomenon, not a vibe. Clinicians in ICUs and on wards already field hundreds of electronic alerts a day, and they override the vast majority. Ancker et al (2017) found override rates above 90% in many settings. Drop a low-specificity sepsis alert into that environment and you are not adding signal, you are adding to the noise that trained everyone to stop looking. The man who died of sepsis after being sent home from the ER died in a system where the alert may well have fired and been dismissed, because dismissing alerts is the rational adaptation to a firehose of mostly-wrong ones.
The treatment contaminates the data. This is the causal trap, and it is the one I find most genuinely hard. The interventions that prevent sepsis from progressing, fluids, antibiotics, vasopressors, get given on the basis of the same signals the model watches. So in your retrospective data, the patients who did not progress to sepsis are a mix of two invisible groups: those who were never going to, and those who were effectively treated. The model cannot tell them apart, which means it cannot cleanly learn who is at risk, and your evaluation cannot cleanly measure whether it would have helped. The act of acting on the prediction changes the thing being predicted. This is the same feedback loop that haunts any deployed decision-support system, and sepsis has it in an acute form.
What Actually Works
After all that, you might expect me to conclude the whole enterprise is hopeless. It is not. Some deployments have shown real clinical benefit, and the interesting thing is what they have in common, because it is almost never “a better model.”
Henry et al (2022) published a prospective study of TREWS across five Johns Hopkins hospitals, and it is the most encouraging result in the field. When clinicians actually engaged with the alert, time to antibiotics dropped and mortality fell. The design choices behind that are the whole story. The system was not built as an alarm; it was built as clinical decision support. It showed the evidence behind each alert, the specific vitals and labs that triggered it, and it required the clinician to actively engage rather than passively dismiss. It answered the exact complaint the HN physician made about the Epic alert: it told you why. The Duke connection here is not incidental, one HN commenter (fny) mentioned in passing that they had “recently been working with a team at Duke building sepsis prediction models,” a reminder that this was a live, hands-on engineering effort at multiple academic centers, not a single vendor’s product.
Sepsis Watch at Duke took a different and equally instructive route. Instead of alerting the bedside clinician, who is already drowning in alerts, it routed the model’s output to a centralized rapid response team whose entire job was to evaluate flagged patients and act. That sidesteps alert fatigue not by making the model quieter but by sending its alarms to people who are paid to respond to alarms. It is an organizational fix to what everyone keeps treating as a technical problem. (The Duke program also surfaced an uncomfortable equity finding, that the model performed differently across racial groups and had to be recalibrated, a reminder that models trained on biased data reproduce the bias.)10
The pattern is consistent. The systems that helped were not the ones with the highest offline AUROC. They were the ones designed around the human: integrated into the workflow instead of bolted on as a pop-up, transparent about their reasoning instead of barking a number, routed to someone with the time and mandate to act, and deployed at an operating point chosen to keep false alarms survivable. The model is a component. The system is the product. Almost everyone who failed had it backwards.
What Sepsis Prediction Taught the Rest of Clinical ML
I think this story matters beyond sepsis, because the same mistakes are being made, right now, in every corner of medical AI that is rushing from a good validation number to a deployment.
Retrospective AUROC is not clinical utility. It measures how well you sort historical patients, not whether you change a real decision in time to matter. A 0.92 in a paper became a 0.63 in a hospital. Any model whose entire evidence base is internal retrospective discrimination should be treated as unproven, full stop.
External validation is non-negotiable, and the field mostly skips it. Park et al (2021) reviewed clinical prediction models and found only about 15% had been externally validated at all; of those, the average performance drop was around 12 points of AUROC. The gap between “works on my data” and “works on yours” is not an edge case in clinical ML. It is the median outcome.
When humans can act on the prediction, the prediction changes the outcome. This breaks the tidy supervised learning assumption that your labels are independent of your model. In medicine they are not, and pretending otherwise corrupts both training and evaluation. You cannot evaluate a sepsis model the way you evaluate an image classifier, because the cat does not change species when you point at it.
The deployment is the intervention. Alert timing, alert content, who receives it, what protocol fires, how feedback flows back: these determine whether a model helps, and they have nothing to do with the loss function. The field’s slow, grudging shift from “prediction model” to “clinical decision support system” as the unit of evaluation is the single most important thing it has learned. It is also harder, slower, and less publishable than chasing AUROC, which is exactly why it took a public failure at the scale of Epic to force the lesson.
I do not think sepsis prediction is a failure. I think it is a field growing up in public, and the growing up looks like failure because the early confidence was unearned. The honest version of the pitch is narrower than the original: machine learning can help catch sepsis earlier, in some hospitals, when it is wired into the workflow with care, watched for drift, and aimed at people whose job is to respond. That is a real and worthwhile thing. It is just not the magic the brochures promised, and the distance between the two is measured in the patients who got a pop-up nobody read.
The next generation of these systems will look less like classifiers and more like carefully engineered decision support built around the clinician. That is progress. It is also an admission that the hard part was never the model.
The central audit discussed here is Wong et al, “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients” (JAMA Internal Medicine, 2021). The encouraging prospective result is Henry et al (Nature Medicine, 2022) on the Johns Hopkins TREWS deployment. Background facts on sepsis are drawn from the CDC, Rhee et al (2017), and Kumar et al (200620ya). The clinician and practitioner voices are from public Hacker News discussions of the Epic sepsis findings and the 2024 New York Times report “It’s Just a Virus, the E.R. Told Him. Days Later, He Was Dead.” For the broader machine-learning context, see The Transformer Architecture and Attention Is All You Need.