Clinical AI Red Teaming for Healthcare Security
Key Takeaways
- In clinical settings, a manipulated AI output can reach a clinician or patient, so tests should follow harm paths, not only data-leak paths.
- Peer-reviewed research shows medical LLMs can be steered and poisoned. A 2025 JAMA Network Open study found injected instructions succeeded in 94.4% of simulated medical dialogues, and a Nature Medicine study found poisoned models can look normal on standard benchmarks.
- Governance points toward testing but doesn’t define it. The Joint Commission and CHAI guidance, FDA cybersecurity guidance, and the proposed HIPAA rule all touch on security testing, but none, in the summaries reviewed, prescribes AI-specific adversarial methods. Your organization has to define scope.
- Evaluate providers on clinical workflow understanding, safe-testing practices, ePHI handling, and reporting in patient-safety terms.
What Counts as Clinical AI, and What Does Red Teaming Add?
Clinical AI is any AI that touches care decisions, clinical documentation, or patient interaction: ambient documentation, clinical decision support, clinician copilots with chart retrieval, symptom triage chatbots, inbox-drafting tools, imaging and diagnostic models, and agents with scheduling or order access. Some are FDA-regulated devices, and many are enterprise LLM features that are not.
Clinical validation asks whether the AI is accurate, stable, and fair. Red teaming asks a different question: can an adversary, or an accidental malicious input, make it wrong, make it leak, or make it act? An AI can pass validation and still follow instructions hidden in a patient message or a retrieved document.
Where Can Clinical AI Be Attacked?
The table maps common clinical AI workflows to attack paths and to the OWASP Top 10 for LLM Applications (2025). It’s illustrative, not a list of documented incidents.
| Clinical workflow | Illustrative attack path | Category | Patient-safety and privacy consequence |
|---|---|---|---|
| Ambient documentation (scribe) | Injected instructions or manipulated transcript content alter the generated note or proposed orders | LLM01 Prompt Injection; LLM05 Improper Output Handling | A wrong medication or allergy enters the chart and informs later decisions |
| Clinician copilot or decision support with retrieval over guidelines and charts | A poisoned guideline, or patient-authored content with hidden instructions, steers the output; retrieval returns another patient’s record | LLM01 (indirect); LLM04 Data and Model Poisoning; LLM08 Vector and Embedding Weaknesses | Unsafe recommendation or cross-patient disclosure |
| Patient-facing triage or symptom chatbot | Direct injection or a jailbreak steers advice or extracts other users’ information | LLM01; LLM02 Sensitive Information Disclosure | Delayed or inappropriate care guidance and PHI exposure |
| Agentic workflows with EHR or scheduling write access | An agent is manipulated into actions beyond the user’s authorization | LLM06 Excessive Agency | Unauthorized changes to records, orders, or referrals |
| Patient-message drafting | Injection in inbound patient messages influences drafted replies | LLM01; LLM05 | Misleading communication sent under a clinician’s name |
| Imaging and diagnostic ML | Adversarial inputs, poisoned training data, or membership inference against the model | Adversarial machine learning | Missed findings or exposure of training records |
What Does the Research Show About Medical LLMs Under Attack?
Two peer-reviewed studies set the evidence base. In a quality improvement study published in JAMA Network Open in 2025, researchers ran standardized patient-LLM dialogues across 12 clinical scenarios against three lightweight commercial models, the kind commonly deployed in resource-constrained health care settings. Across 216 evaluations (108 injection and 108 control), injection attacks succeeded in 94.4 percent of cases (102 of 108) at the primary decision turn and persisted in 69.4 percent (75 of 108) of follow-ups. The authors concluded that current safeguards remain inadequate to prevent injection that could induce unsafe clinical recommendations.

The second study, in Nature Medicine, examined data poisoning. Researchers found that replacing just 0.001 percent of training tokens with medical misinformation produced models more likely to propagate medical errors, and that the corrupted models performed comparably to clean ones on standard medical benchmarks. That matters operationally: benchmark scores and clinical validation results can’t be treated as evidence that a model hasn’t been tampered with.
Preprint research on EHR agents with persistent memory also studies memory poisoning, where malicious instructions corrupt an agent’s long-term memory and influence later responses. It’s early work, but it names an attack path specific to agents embedded in clinical systems.
These studies have limits. The injection study used simulated dialogues and lightweight models, and the poisoning study concerned models trained on web-scraped data, which is mainly a risk for model developers and a supply chain question for hospitals using third-party or fine-tuned models.
What Do Governance Frameworks Expect?

| Framework | Status | What it means for clinical AI testing |
|---|---|---|
| Joint Commission and CHAI, Responsible Use of AI in Healthcare (RUAIH) | Released September 17, 2025; non-binding | Seven elements, including data security and data use protections (vendor security assessments, data use agreements with audit rights) and voluntary, blinded reporting of AI safety events. The summaries reviewed describe security assessments and vulnerability scans, not adversarial testing of AI behavior |
| FDA cybersecurity in medical devices guidance | Final June 27, 2025; reissued February 2, 2026 for QMSR alignment | Recommends threat modeling and security testing, including penetration testing, for device premarket submissions. The FDA defines the “medical device system” to include connected systems such as healthcare facility networks, so a manufacturer’s testing doesn’t replace testing of your integration |
| HIPAA Security Rule | Current rule in force; update proposed January 6, 2025, not final | The current rule requires risk analysis and periodic evaluation. The proposal would require vulnerability scanning at least every six months and penetration testing at least every 12 months. OMB’s non-binding target for final action is now July 2027. An AI system handling ePHI belongs in your risk analysis either way |
How Should Clinical AI Be Tested Safely?
These are practical recommendations, not regulatory requirements, and they should be agreed in the rules of engagement before testing begins:
- Use synthetic patients and fabricated or de-identified data in non-production environments. Attack scenarios shouldn’t run against live clinical workflows unless explicitly authorized.
- Prevent test output from reaching real charts. Use isolated test accounts, block write paths to production records, and plan cleanup for anything generated.
- Add clinical review. A clinician should review attack scenarios and outputs so unsafe content isn’t left where it could be mistaken for guidance.
- Decide ePHI handling up front. If the tester may encounter ePHI, confirm whether a business associate agreement applies and is signed before work begins.
- Confirm authorization for third-party AI. EHR-embedded and vendor-hosted AI features may be contractually off-limits to testing. Test your configuration and integration layer, and ask the vendor for its own testing evidence.
- Define stop conditions. Agree what triggers an immediate halt, such as evidence that real patient data is exposed.
Which Cybersecurity Firms Provide AI Red Teaming and LLM Penetration Testing for Healthcare?
Several firms publicly list AI red teaming or LLM penetration testing. None of the pages reviewed, including VerSprite’s, describes a clinical-workflow-specific methodology or a patient-safety severity scale, so ask each provider how it handles clinical context. The table reflects each firm’s own public pages as of November 2026.
| Provider | AI testing scope stated | Healthcare-specific content stated | Frameworks stated | Pricing |
|---|---|---|---|---|
| VerSprite | Human-led testing of models, LLM applications, RAG pipelines, AI agents, and MCP servers; Knife automated testing included in its PTaaS offering | Testing clinical and diagnostic AI for data poisoning, model inversion, and unauthorized inference against systems processing ePHI; HIPAA named among obligations in scoping | OWASP LLM Top 10 (2025), MITRE ATLAS, NIST AI RMF and its Generative AI Profile, ISO/IEC 42001, EU AI Act | Not published |
| Bishop Fox | LLM-enabled applications, AI agents, and surrounding architecture: prompt injection, data exposure, excessive agency, insecure integrations; open-source AIMap tool for exposed AI agent infrastructure | Not found on the pages reviewed | Not stated on the pages reviewed | Not published |
| Bitdefender | Consultant-led LLM and AI penetration testing: scoping, adversarial testing, controlled exploitation, and reporting; AI-assisted scans support manual testing | Not on the service page; compliance reports named for MAS TRM, PCI DSS, ISO 27001, and GDPR (HIPAA not named) | OWASP LLM Top 10, MITRE ATLAS | Scoped in person-days, not published |
| CrowdStrike | AI Red Team Services: penetration testing of LLM applications against the OWASP Top Ten, adversary emulation, and red team and blue team exercises | Not on the service page | OWASP Top Ten for LLM applications | Request info |
| Schellman | AI red teaming combining manual and automated testing | Not found on the page reviewed | OWASP LLM Top 10, NIST AI RMF | States about two weeks and a floor of $16,000 per engagement; token exhaustion attacks excluded unless requested |
VerSprite
- Method: VerSprite applies PASTA, a seven-stage risk-centric methodology its CEO co-created, to AI systems, tying each test to the business objectives and regulatory obligations of the system under test.
- Scope areas listed: LLM and generative AI applications, AI agents and MCP, RAG pipelines and vector stores, adversarial machine learning, and AI infrastructure and MLOps. Engagements are scoped to your systems.
- Deliverables stated: Attack scenario documentation, validated findings, framework mapping, business risk prioritization, remediation guidance, and executive and technical reporting.
- Timeline: A few weeks to a phased engagement, depending on the number of models and APIs, pipeline complexity, and scope.
- Not stated on its page: a clinical-workflow methodology, a patient-safety severity scale, business associate agreement handling, FDA or medical-device mapping, and published pricing.
How Should Healthcare Security and Privacy Leaders Evaluate Providers?
Evaluate how well the firm understands care delivery, how it tests safely, and how it reports. Ask each provider:
- How do you test clinical workflows, not just chat interfaces? Look for scenarios built around scribes, decision support, triage, and agent actions.
- How do you express severity? Ask whether findings can be framed in patient-safety and privacy terms alongside technical severity, and who reviews clinical impact.
- How do you test safely? Ask about synthetic patients, isolation from production charts, stop conditions, and cleanup.
- How do you handle ePHI? Confirm whether you’ll sign a business associate agreement, and what tester access to PHI would look like.
- How do you cover retrieval and agents? Cross-patient retrieval, indirect injection through documents and messages, and agent permissions need specific tests.
- How do you handle third-party and EHR-embedded AI? Ask what’s authorized and how findings are routed to vendors.
- Which frameworks do findings map to? OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF, and ISO/IEC 42001 are the common ones, and your governance committee may want the Joint Commission and CHAI elements referenced too.
- How are retests and cadence handled? Models, prompts, retrieval sources, and tool permissions change, and each change can reopen a finding.
- How does reporting support AI safety event processes? Reports should be usable by the governance committee and consistent with how you report other safety events.
Limitations
The injection figures come from a simulated study with lightweight models, and results may differ for other models and real deployments. The poisoning study concerned models trained on web-scraped data. Some details, such as the memory poisoning work, are preprints. Governance summaries reflect published guidance as of November 2026, the HIPAA rule remains proposed, and its target date is a planning estimate. The attack-path table is illustrative, not incident-based. Provider descriptions come from their own pages, so verify them directly, and VerSprite publishes this guide.
Final Takeaway
Clinical AI changes what a security finding means: a manipulated output can reach a patient, and standard validation can’t show whether a model was tampered with. The practical response is adversarial testing scoped to clinical workflows, run safely against synthetic data, and reported in terms your governance committee already uses for patient safety and privacy.
Sources
- JAMA Network Open (2025), Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice
- Nature Medicine, Medical large language models are vulnerable to data-poisoning attacks
- Joint Commission and Coalition for Health AI, Guidance on the Responsible Use of AI in Healthcare (September 17, 2025)
- FDA, Medical Device Cybersecurity resources and updates
- HHS, HIPAA Security Rule NPRM (January 6, 2025) and Unified Agenda entry RIN 0945-AA22
- VerSprite, AI Red Teaming and LLM Penetration Testing
- Bishop Fox, AI and LLM Security Assessment
- Bitdefender, LLM and AI Penetration Testing
- CrowdStrike, AI Red Team Services
- Schellman, AI Red Teaming
Frequently Asked Questions
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /