Journal of Medical Internet Research, 2026

A Multi-Agent Large Language Model Framework for Emergency Treatment Recommendation in Acute Ischemic Stroke

A structured, auditable LLM workflow for guideline-concordant AIS treatment recommendation, TOAST classification, safety assessment, and human-AI clinical validation.

Bicong Yan, Ruipeng Zhang, Yanfeng Fan, Ying Li, Li Chen, Xinyu Song, Yixiao Tang, Zhongzheng Cao, Li Shen, Mengfei Wang, Zhuo Li, Yijia Xiong, Yuehua Li

Paper Code BibTeX

Study framework for the HR-LLM-Stroke multi-agent pipeline — Multi-agent framework, multisource AIS cohorts, model benchmarking, safety analysis, and physician validation.

Abstract

Clinical decision support under acute stroke time pressure

Acute ischemic stroke treatment selection requires rapid integration of clinical, imaging, and laboratory information, including therapeutic windows, contraindications, stroke severity, and imaging eligibility. This process is expertise-dependent and vulnerable to safety-critical errors.

HR-LLM-Stroke develops and validates a structured multi-agent LLM framework for real-world AIS decision support. The framework combines a workflow-oriented summarization agent, guideline-concordant reasoning-path CoT, and a multiple-choice constraint agent to generate treatment recommendations and TOAST subtype classifications with auditable intermediate outputs.

Framework

Structured reasoning before final selection

The public release supports two reproducible inference modes: a standalone LLM baseline and the framework-augmented workflow. The framework first summarizes long clinical narratives, then applies guideline-oriented reasoning, and finally constrains the answer to clinically interpretable options.

Summarization Agent

Extracts AIS-relevant evidence from lengthy case narratives.

Reasoning-Path CoT

Enforces concise, guideline-concordant diagnostic and treatment reasoning.

Multiple-Choice Constraint

Maps outputs to treatment and TOAST categories for consistent evaluation.

Main Results

Framework augmentation improved accuracy, safety, and physician performance

2,081

Evaluated cases

Retrospective, prospective, multicenter, and PubMed-derived AIS scenarios.

+18.9%

Average gain

Mean framework improvement over standalone LLMs in treatment recommendation experiments.

0.687 -> 0.847

DeepSeek-R1 Group A

Treatment recommendation accuracy increased with framework augmentation.

4.7% -> 3.1%

Hallucination rate

Framework outputs reduced hallucination rate versus standalone outputs.

0.600 -> 0.846

Junior non-specialists

Physician treatment decision accuracy improved under AI assistance.

0.667 -> 0.833

Junior specialists

AI assistance delivered large gains for less-experienced clinicians.

Figures

Experimental evidence

Standalone and framework-augmented LLM performance across cohorts — Standalone versus framework-augmented treatment recommendation accuracy across model families.

F1 score comparison and treatment recommendation flows — F1 comparisons and treatment-flow changes after framework augmentation.

Ablation study of alternative framework compositions — Ablation study comparing alternative agent combinations with the full framework.

Reasoning case examples, harmfulness, instruction following, hallucination, and omission analysis.

Human-AI evaluation across physician groups — Human-AI evaluation across physician seniority, specialty, and perceived usefulness.

Open Release

Reproducible prompting and evaluation

The repository provides prompt templates, batch inference scripts, a public example case, example outputs, and a lightweight exact-match evaluator. Private clinical datasets and production clinical deployment code are not included.

Output tags

<think>: Diagnostic reasoning retained for research audit.
<diagnosis>: TOAST classification.
<treatment>: Emergency treatment recommendation.

Citation

Cite HR-LLM-Stroke

@article{yan2026hrllmstroke,
  title = {A Multi-Agent Large Language Model Framework for Emergency Treatment Recommendation in Acute Ischemic Stroke: Development and Validation Study},
  author = {Yan, Bicong and Zhang, Ruipeng and Fan, Yanfeng and Li, Ying and Chen, Li and Song, Xinyu and Tang, Yixiao and Cao, Zhongzheng and Shen, Li and Wang, Mengfei and Li, Zhuo and Xiong, Yijia and Li, Yuehua},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  doi = {10.2196/96304},
  note = {Epub ahead of print, June 23, 2026}
}