Unsanctioned AI in clinical settings: what health institutions need to know
Shadow AI creates specific confidentiality, regulatory, and auditability risks in pharmacovigilance and toxicovigilance workflows.
The behaviour is documented and, in most contexts, rational. A medical assessor at a national pharmacovigilance centre or a physician responding at a poison control unit faces a complex case, ambiguous causality, scattered references, time pressure, and opens a public AI assistant. The tool produces a coherent clinical summary in seconds. The team is small. The institutional system offers nothing equivalent. The decision follows logically from the constraints.
What follows from it, regulatorily, is the subject of this article.
What leaves the institution when a case enters a public AI tool
An ICSR is not a generic clinical document. It contains information that identifies, directly or by inference, the patient, the reporting clinician, the product, and the therapeutic context. In toxicovigilance cases, the exposure narrative, circumstances, substance, quantity, clinical trajectory, can be specific enough to identify the incident without naming anyone. When an assessor submits this content to a public AI assistant to reformulate a narrative, structure a differential, or support a causality hypothesis, that content exits the institutional perimeter.
The issue is not whether the platform will misuse the data. The issue is evidentiary: the institution can no longer demonstrate that the information remained within its controlled environment. For agencies operating under national data protection frameworks, and for those aligned with the AMRH's emerging standards on health data governance, that gap is not theoretical. It becomes concrete at the moment of an audit, an incident review, or a cross-border regulatory inspection.
The audit trail problem
GVP Module VI requires that every material step in the processing of a case be traceable: who acted, at which stage, on the basis of which information. The EMA's guidance on electronic records extends this to the systems themselves, the audit trail must allow reconstruction of the sequence of events surrounding any record's creation or modification. A public AI tool produces none of this. No institutional log of the query. No record of the output. No version identifier for the model. No linkage between the AI-generated content and the case it informed. The assessor's conclusion enters the case file. The reasoning that shaped it does not. For an internal review, the institution cannot demonstrate that its workflow followed approved procedures. For an external inspection, by a national authority, a regional body, or an international partner, the absence of a traceable reasoning chain in a contested case is a material weakness. Not a formality gap. A substantive one.
Causality assessment and what "defensible" actually means
The Naranjo algorithm, the French imputability method, WHO-UMC criteria, these are structured processes precisely because causality conclusions must be reconstructable. If the case is later reviewed, challenged, or cited in a safety signal, the reasoning has to hold under scrutiny: defined methodology, identified data, qualified assessor.
When part of that reasoning was conducted in a session on a third-party server, generating no institutional trace, applying no validated methodology, engaging no documented accountability, the recorded assessment becomes structurally incomplete. The conclusion is on file. The process that produced it is not, and cannot be retrieved.
The practical consequence for agency directors and compliance officers is narrow but firm: an institution that cannot reconstruct how a case was assessed cannot fully defend how it was closed.
Health vigilance systems, whether in a national pharmacovigilance centre, a regional toxicovigilance unit, or a ministry-level post-market surveillance agency, operate on the premise that every decision affecting public health can be accounted for. Unsanctioned AI use does not threaten that premise by intent. It threatens it by architecture.