Clinical AI Red Teaming for Healthcare Security

Clinical AI Red Teaming for Healthcare Security
About the author. Tony UcedaVélez is the Founder and CEO of VerSprite and co-author of Risk Centric Threat Modeling (Wiley, 2015). This guide summarizes published research and regulatory status as of November 2026 and describes each provider only from its own public pages. It is not medical, legal, or compliance advice.

Key Takeaways

  • In clinical settings, a manipulated AI output can reach a clinician or patient, so tests should follow harm paths, not only data-leak paths.
  • Peer-reviewed research shows medical LLMs can be steered and poisoned. A 2025 JAMA Network Open study found injected instructions succeeded in 94.4% of simulated medical dialogues, and a Nature Medicine study found poisoned models can look normal on standard benchmarks.
  • Governance points toward testing but doesn’t define it. The Joint Commission and CHAI guidance, FDA cybersecurity guidance, and the proposed HIPAA rule all touch on security testing, but none, in the summaries reviewed, prescribes AI-specific adversarial methods. Your organization has to define scope.
  • Evaluate providers on clinical workflow understanding, safe-testing practices, ePHI handling, and reporting in patient-safety terms.



What Counts as Clinical AI, and What Does Red Teaming Add?

Clinical AI is any AI that touches care decisions, clinical documentation, or patient interaction: ambient documentation, clinical decision support, clinician copilots with chart retrieval, symptom triage chatbots, inbox-drafting tools, imaging and diagnostic models, and agents with scheduling or order access. Some are FDA-regulated devices, and many are enterprise LLM features that are not.

Clinical validation asks whether the AI is accurate, stable, and fair. Red teaming asks a different question: can an adversary, or an accidental malicious input, make it wrong, make it leak, or make it act? An AI can pass validation and still follow instructions hidden in a patient message or a retrieved document.




Where Can Clinical AI Be Attacked?

The table maps common clinical AI workflows to attack paths and to the OWASP Top 10 for LLM Applications (2025). It’s illustrative, not a list of documented incidents.


Clinical workflow Illustrative attack path Category Patient-safety and privacy consequence
Ambient documentation (scribe) Injected instructions or manipulated transcript content alter the generated note or proposed orders LLM01 Prompt Injection; LLM05 Improper Output Handling A wrong medication or allergy enters the chart and informs later decisions
Clinician copilot or decision support with retrieval over guidelines and charts A poisoned guideline, or patient-authored content with hidden instructions, steers the output; retrieval returns another patient’s record LLM01 (indirect); LLM04 Data and Model Poisoning; LLM08 Vector and Embedding Weaknesses Unsafe recommendation or cross-patient disclosure
Patient-facing triage or symptom chatbot Direct injection or a jailbreak steers advice or extracts other users’ information LLM01; LLM02 Sensitive Information Disclosure Delayed or inappropriate care guidance and PHI exposure
Agentic workflows with EHR or scheduling write access An agent is manipulated into actions beyond the user’s authorization LLM06 Excessive Agency Unauthorized changes to records, orders, or referrals
Patient-message drafting Injection in inbound patient messages influences drafted replies LLM01; LLM05 Misleading communication sent under a clinician’s name
Imaging and diagnostic ML Adversarial inputs, poisoned training data, or membership inference against the model Adversarial machine learning Missed findings or exposure of training records



What Does the Research Show About Medical LLMs Under Attack?

Two peer-reviewed studies set the evidence base. In a quality improvement study published in JAMA Network Open in 2025, researchers ran standardized patient-LLM dialogues across 12 clinical scenarios against three lightweight commercial models, the kind commonly deployed in resource-constrained health care settings. Across 216 evaluations (108 injection and 108 control), injection attacks succeeded in 94.4 percent of cases (102 of 108) at the primary decision turn and persisted in 69.4 percent (75 of 108) of follow-ups. The authors concluded that current safeguards remain inadequate to prevent injection that could induce unsafe clinical recommendations.

Figure 1: Bar chart from a 2025 JAMA Network Open study: prompt injection succeeded in 94.4% of 108 simulated medical dialogues at the decision turn and persisted in 69.4% of follow-ups.
Figure 1. Prompt-injection success in simulated medical dialogues. Source: JAMA Network Open (2025). Simulated dialogues using lightweight models; results may differ for other models and deployments.

The second study, in Nature Medicine, examined data poisoning. Researchers found that replacing just 0.001 percent of training tokens with medical misinformation produced models more likely to propagate medical errors, and that the corrupted models performed comparably to clean ones on standard medical benchmarks. That matters operationally: benchmark scores and clinical validation results can’t be treated as evidence that a model hasn’t been tampered with.

Preprint research on EHR agents with persistent memory also studies memory poisoning, where malicious instructions corrupt an agent’s long-term memory and influence later responses. It’s early work, but it names an attack path specific to agents embedded in clinical systems.

These studies have limits. The injection study used simulated dialogues and lightweight models, and the poisoning study concerned models trained on web-scraped data, which is mainly a risk for model developers and a supply chain question for hospitals using third-party or fine-tuned models.




What Do Governance Frameworks Expect?

Figure 2: Timeline of US clinical AI governance dates: HIPAA Security Rule NPRM January 2025, FDA device cybersecurity guidance June 2025, Joint Commission and CHAI AI guidance September 2025, FDA guidance reissued February 2026, original HIPAA final-rule target passed May 2026, current HIPAA target July 2027.
Figure 2. Key US dates affecting security testing of clinical AI. The HIPAA Security Rule update remains proposed.

Framework Status What it means for clinical AI testing
Joint Commission and CHAI, Responsible Use of AI in Healthcare (RUAIH) Released September 17, 2025; non-binding Seven elements, including data security and data use protections (vendor security assessments, data use agreements with audit rights) and voluntary, blinded reporting of AI safety events. The summaries reviewed describe security assessments and vulnerability scans, not adversarial testing of AI behavior
FDA cybersecurity in medical devices guidance Final June 27, 2025; reissued February 2, 2026 for QMSR alignment Recommends threat modeling and security testing, including penetration testing, for device premarket submissions. The FDA defines the “medical device system” to include connected systems such as healthcare facility networks, so a manufacturer’s testing doesn’t replace testing of your integration
HIPAA Security Rule Current rule in force; update proposed January 6, 2025, not final The current rule requires risk analysis and periodic evaluation. The proposal would require vulnerability scanning at least every six months and penetration testing at least every 12 months. OMB’s non-binding target for final action is now July 2027. An AI system handling ePHI belongs in your risk analysis either way



How Should Clinical AI Be Tested Safely?

These are practical recommendations, not regulatory requirements, and they should be agreed in the rules of engagement before testing begins:

  • Use synthetic patients and fabricated or de-identified data in non-production environments. Attack scenarios shouldn’t run against live clinical workflows unless explicitly authorized.
  • Prevent test output from reaching real charts. Use isolated test accounts, block write paths to production records, and plan cleanup for anything generated.
  • Add clinical review. A clinician should review attack scenarios and outputs so unsafe content isn’t left where it could be mistaken for guidance.
  • Decide ePHI handling up front. If the tester may encounter ePHI, confirm whether a business associate agreement applies and is signed before work begins.
  • Confirm authorization for third-party AI. EHR-embedded and vendor-hosted AI features may be contractually off-limits to testing. Test your configuration and integration layer, and ask the vendor for its own testing evidence.
  • Define stop conditions. Agree what triggers an immediate halt, such as evidence that real patient data is exposed.



Which Cybersecurity Firms Provide AI Red Teaming and LLM Penetration Testing for Healthcare?

Several firms publicly list AI red teaming or LLM penetration testing. None of the pages reviewed, including VerSprite’s, describes a clinical-workflow-specific methodology or a patient-safety severity scale, so ask each provider how it handles clinical context. The table reflects each firm’s own public pages as of November 2026.


Provider AI testing scope stated Healthcare-specific content stated Frameworks stated Pricing
VerSprite Human-led testing of models, LLM applications, RAG pipelines, AI agents, and MCP servers; Knife automated testing included in its PTaaS offering Testing clinical and diagnostic AI for data poisoning, model inversion, and unauthorized inference against systems processing ePHI; HIPAA named among obligations in scoping OWASP LLM Top 10 (2025), MITRE ATLAS, NIST AI RMF and its Generative AI Profile, ISO/IEC 42001, EU AI Act Not published
Bishop Fox LLM-enabled applications, AI agents, and surrounding architecture: prompt injection, data exposure, excessive agency, insecure integrations; open-source AIMap tool for exposed AI agent infrastructure Not found on the pages reviewed Not stated on the pages reviewed Not published
Bitdefender Consultant-led LLM and AI penetration testing: scoping, adversarial testing, controlled exploitation, and reporting; AI-assisted scans support manual testing Not on the service page; compliance reports named for MAS TRM, PCI DSS, ISO 27001, and GDPR (HIPAA not named) OWASP LLM Top 10, MITRE ATLAS Scoped in person-days, not published
CrowdStrike AI Red Team Services: penetration testing of LLM applications against the OWASP Top Ten, adversary emulation, and red team and blue team exercises Not on the service page OWASP Top Ten for LLM applications Request info
Schellman AI red teaming combining manual and automated testing Not found on the page reviewed OWASP LLM Top 10, NIST AI RMF States about two weeks and a floor of $16,000 per engagement; token exhaustion attacks excluded unless requested

VerSprite

  • Method: VerSprite applies PASTA, a seven-stage risk-centric methodology its CEO co-created, to AI systems, tying each test to the business objectives and regulatory obligations of the system under test.
  • Scope areas listed: LLM and generative AI applications, AI agents and MCP, RAG pipelines and vector stores, adversarial machine learning, and AI infrastructure and MLOps. Engagements are scoped to your systems.
  • Deliverables stated: Attack scenario documentation, validated findings, framework mapping, business risk prioritization, remediation guidance, and executive and technical reporting.
  • Timeline: A few weeks to a phased engagement, depending on the number of models and APIs, pipeline complexity, and scope.
  • Not stated on its page: a clinical-workflow methodology, a patient-safety severity scale, business associate agreement handling, FDA or medical-device mapping, and published pricing.



How Should Healthcare Security and Privacy Leaders Evaluate Providers?

Evaluate how well the firm understands care delivery, how it tests safely, and how it reports. Ask each provider:

  1. How do you test clinical workflows, not just chat interfaces? Look for scenarios built around scribes, decision support, triage, and agent actions.
  2. How do you express severity? Ask whether findings can be framed in patient-safety and privacy terms alongside technical severity, and who reviews clinical impact.
  3. How do you test safely? Ask about synthetic patients, isolation from production charts, stop conditions, and cleanup.
  4. How do you handle ePHI? Confirm whether you’ll sign a business associate agreement, and what tester access to PHI would look like.
  5. How do you cover retrieval and agents? Cross-patient retrieval, indirect injection through documents and messages, and agent permissions need specific tests.
  6. How do you handle third-party and EHR-embedded AI? Ask what’s authorized and how findings are routed to vendors.
  7. Which frameworks do findings map to? OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF, and ISO/IEC 42001 are the common ones, and your governance committee may want the Joint Commission and CHAI elements referenced too.
  8. How are retests and cadence handled? Models, prompts, retrieval sources, and tool permissions change, and each change can reopen a finding.
  9. How does reporting support AI safety event processes? Reports should be usable by the governance committee and consistent with how you report other safety events.



Limitations

The injection figures come from a simulated study with lightweight models, and results may differ for other models and real deployments. The poisoning study concerned models trained on web-scraped data. Some details, such as the memory poisoning work, are preprints. Governance summaries reflect published guidance as of November 2026, the HIPAA rule remains proposed, and its target date is a planning estimate. The attack-path table is illustrative, not incident-based. Provider descriptions come from their own pages, so verify them directly, and VerSprite publishes this guide.




Final Takeaway

Clinical AI changes what a security finding means: a manipulated output can reach a patient, and standard validation can’t show whether a model was tampered with. The practical response is adversarial testing scoped to clinical workflows, run safely against synthetic data, and reported in terms your governance committee already uses for patient safety and privacy.

Sources




Frequently Asked Questions

Firms that publicly list AI red teaming or LLM penetration testing include VerSprite (which publishes this guide), Bishop Fox, Bitdefender, CrowdStrike, and Schellman. For healthcare, ask each how it handles clinical workflows, patient-safety severity, safe testing, and ePHI, since none of the pages reviewed describes a clinical-workflow-specific methodology.
It is adversarial testing of the AI embedded in care delivery, such as ambient scribes, clinical decision support, patient-facing triage, and EHR agents, to find how an attacker or manipulated input could expose patient data or push a wrong output into the chart or care pathway. Findings are judged by patient-safety and privacy consequence as well as technical exploitability.
Clinical validation asks whether an AI is accurate, stable, and fair. Red teaming asks whether an adversary can make it wrong, make it leak data, or make it act. An AI can pass validation and still follow instructions hidden in a patient message or retrieved document, and a Nature Medicine study found poisoned medical models can perform normally on standard benchmarks.
Priority attack paths include direct and indirect prompt injection against scribes, copilots, triage chatbots, and message-drafting tools; retrieval weaknesses that expose another patient’s record; excessive agency in agents with EHR or scheduling write access; data and model poisoning; and adversarial inputs or membership inference against diagnostic models.
A 2025 quality improvement study ran simulated patient dialogues across 12 clinical scenarios against three lightweight commercial LLMs. Across 216 evaluations, injection attacks succeeded in 94.4 percent of cases (102 of 108) at the primary decision turn and persisted in 69.4 percent of follow-ups. The authors concluded current safeguards remain inadequate; results may differ for other models and real deployments.
Not reliably. A Nature Medicine study found that replacing just 0.001 percent of training tokens with medical misinformation produced models more likely to propagate medical errors, while the corrupted models performed comparably to clean ones on standard medical benchmarks. Benchmark scores are not evidence a model has not been tampered with.
Use synthetic patients and fabricated or de-identified data in non-production environments, block write paths from test accounts to real charts, have a clinician review attack scenarios and outputs, agree stop conditions, decide ePHI handling and any business associate agreement in advance, and confirm authorization before testing vendor-hosted or EHR-embedded AI.
Not by name under the current Security Rule, which requires risk analysis and periodic evaluation. The update proposed on January 6, 2025 would require vulnerability scanning at least every six months and penetration testing at least every 12 months, but it remains proposed, and OMB’s non-binding target for final action is now July 2027. An AI system that handles ePHI belongs in your risk analysis regardless.
The FDA’s cybersecurity guidance for medical devices, finalized June 27, 2025 and reissued February 2, 2026, recommends threat modeling and security testing, including penetration testing, for premarket submissions. It defines the medical device system to include connected systems such as healthcare facility networks, so a manufacturer’s testing does not replace testing of your own integration and configuration.
Before go-live and after significant changes to the model or vendor version, system prompts, retrieval sources, tool permissions, or integrations. No standard sets a cadence, so base it on how often those components change, and turn confirmed findings into regression tests that run with each release.