People ask general-purpose assistants about their symptoms. They did so before anyone shipped a health feature and they will continue regardless of what any terms of service say. This is the fact that regulatory frameworks in this area have to accommodate, and mostly do not.
When software becomes a device
Medical device regulation turns substantially on intended use, which is established by the claims a manufacturer makes. Software that diagnoses, treats or prevents disease is generally a device; software that provides general information generally is not. The boundary was workable when a product did one thing.
A conversational system does whatever the conversation leads to. The same interface, unchanged, provides general information in one exchange and something much closer to individualised clinical advice in the next, depending on what the user asks and how specifically the system answers. The developer controls the marketing claim. They do not control the conversation.
What HIPAA does and does not reach
A common misunderstanding is worth stating plainly: HIPAA regulates covered entities — providers, plans, clearing houses — and their business associates. It does not regulate health information as such. An individual describing their symptoms to a consumer product is generally not within its scope, and the protections people assume apply usually do not.
What governs that data instead is a patchwork of consumer protection law, state privacy statutes of varying strength, and the developer's own commitments. Where a developer makes specific commitments about health conversations, those commitments become the operative protection, which is a fragile place for the protection to live.
Why disclaimers underperform
The standard mitigation is a notice that the system does not provide medical advice. In a conversational interface this is shown at a moment when the user has no health question, and is absent at the moment they do. Its effect on behaviour in the exchange that matters is, on the available evidence, small.
Controls that plausibly work are behavioural rather than textual: recognising presentations that warrant urgent care and responding accordingly, declining to give specific dosing or diagnostic conclusions, and surfacing emergency routing prominently at the point of relevance. These are engineering decisions with measurable outcomes, and they are what an assessment of a health-adjacent product should be examining.
The question litigation will settle
Where harm follows an interaction with a general-purpose system, the contested question is whether the developer owed a duty of care in respect of that interaction, and what a reasonable standard of care would have been. No settled answer exists. It will be established case by case, and the first decisions will shape product design across the sector more than any framework currently published.