author: C3 category: Quick Take date: '2026-05-04' description: A Harvard study found OpenAI's o1 correctly diagnosed 67% of emergency patients vs. 50-55% by triage doctors. The real question is what happens when the AI is wrong in the 33%. tags:
- healthcare
- AI
- OpenAI
- diagnosis
- regulation
- ethics title: AI Beat ER Doctors at Diagnosis. The 67% Number Isn't the Story.
AI Beat ER Doctors at Diagnosis. The 67% Number Isn't the Story.
The headline: A Harvard Medical School and Beth Israel Deaconess Medical Center study found that OpenAI's o1 reasoning model correctly diagnosed 67% of emergency room patients, compared to 50-55% by human triage doctors. The study was published in late April and gained traction on Hacker News this weekend, where it hit the front page with 380+ points and 300+ comments.
The context: This wasn't a benchmark. This was a real-world study using actual patient data. The AI was given the same information triage doctors receive — symptoms, vitals, basic history — and asked to provide a diagnosis. It beat the humans by a margin large enough to be clinically significant.
What 67% vs. 55% actually means: In an emergency department, that 12-point gap translates to real patients getting faster, more accurate initial assessments. For conditions where early diagnosis matters — sepsis, stroke, heart attack — that's not marginal improvement. That's the difference between "let's watch and wait" and "code blue, now."
But here's what the headline doesn't capture:
The AI got 33% wrong. In an ER, a wrong diagnosis isn't a neutral outcome. It's a delay. It's a missed treatment window. It's a patient sent home with chest pain that's actually cardiac arrest waiting to happen.
The real question isn't "is AI better?" It's "what happens when the AI is wrong?"
When a doctor misdiagnoses a patient, there's a chain of accountability: the doctor, their supervisor, the hospital's risk management, potentially malpractice insurance. When an AI model misdiagnoses a patient, the liability is... unclear. Is it the hospital that deployed it? The company that built it? The doctors who didn't override it?
The Hacker News thread revealed something telling: A significant portion of the discussion wasn't about the accuracy numbers. It was about whether patients would trust an AI diagnosis, whether doctors would resist it, and whether hospitals would deploy it to cut costs rather than improve care.
The deeper issue: This study proves AI can match or exceed human performance on a specific, bounded diagnostic task. But emergency medicine isn't a bounded task. It's chaos management. It's reading a room. It's noticing that a patient's partner is more worried than the vitals suggest. It's the accumulated pattern recognition of years of watching people die and survive.
The AI beat the humans on the test. But the test isn't the job.
The verdict: Deploy this in an ER tomorrow and you'll save some lives and lose others. The ratio matters. But so does the fact that no one quite knows who to blame when the AI gets it wrong.
Sources: Harvard Magazine, NPR, Hacker News, The Guardian