California Enacts New Laws Governing Use of AI in Health Care

Bias Mitigation Requirements

SB 503 requires developers and deployers, including health facilities, clinics, group practices, and physicians, of clinical decision support (CDS) systems to “make reasonable efforts to identify [CDS] systems developed for use by deployers that are known or have a reasonably foreseeable risk of biased impacts,” such as diminished access to care, quality of care, or outcomes on an individual based on their protected characteristics such as race, sex, or national origin.

The bill defines CDS as “an AI system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decision [m]aking related to timing of care, diagnosis, or treatment.” However, it specifically excludes systems that “provide appointment management such as booking, canceling and rescheduling appointments, appointment reminders, patient education and pre-visit materials and preparation, and payment processing, to the extent the independent performance of these activities by the system does not require a professional license.”

SB 503 also requires developers to provide deployers of CDS systems with a statement of intended uses and known or reasonably foreseeable risks associated with use of that AI system, in addition to providing documentation disclosing:

• what type of data was used to train the CDS system, including demographic representativeness,
• how the CDS system was evaluated for performance, limitations, and mitigation of bias impacts,
• intended benefits and expected outputs of the CDS system,
• any known or reasonably foreseeable risk of bias impacts,
• recommendations on how the CDS system should be used and monitored,
• summaries of data governance measures used to evaluate training datasets that help determine the suitability of data sources and possible biases, and
• any other documentation reasonably necessary to help the deployer understand the outputs and monitor performance of the CDS system.

As a result, health care providers using external CDS tools will need to ensure that, prior to deployment of the tool, they make reasonable efforts to identify or address known or reasonably foreseeable risks of biased impacts associated with the tool. The bill also creates an ongoing obligation to monitor CDS tools for adverse impacts and other indicators of bias. Although the statute does not define what constitutes “regular” monitoring, the dynamic nature of AI models suggests that more frequent review intervals may be better suited to detecting adverse outcomes or performance that departs from expectations. In developing governance structures around these requirements, organizations may consider what monitoring cadence is operationally feasible and whether review schedules should be standardized across AI tools or tailored based on each tool’s risk profile and use. Model cards are one approach being used by some providers and hospital systems to address this requirement, with periodic updates to the information they contain helping to facilitate ongoing monitoring of the AI tool’s performance and use.

Professional Judgment Must Remain with the Provider

AB 1979 requires health facilities, clinics, group practices, and physicians to take “reasonable steps” to ensure their licensed health care professionals are able to exercise independent professional judgment over their patient’s care when that care has been informed by a CDS system (using the same definition as SB 503). AB 1979 also explicitly prohibits health care providers from using or deploying an AI tool, system, or device to independently perform a clinical function that by law must be performed by a licensed professional, reaffirming existing requirements under the state’s Medical Practice Act that the practice of medicine can only be carried out by a licensed “person.” Additionally, it prohibits providers from using or deploying an AI tool, system, or device to direct, guide, or supervise an unlicensed professional in performing those same functions.

The law carves out certain uses of CDS tools for administrative functions, such as automated messages informing patients of updates to their health records and generated appointment reminders. However, the law also amends the state’s Confidentiality of Medical Information Act (CMIA) to extend certain privacy obligations to businesses offering health care chatbots that allow consumers to manage their health information, including by assisting in diagnosis or treatment of a medical condition.

Violations of AB 1979 are enforceable by the Medical Board of California or the Osteopathic Medical Board, and may result in an injunction, issuance of a restraining order, or other remedies available to the Boards.

Disclosures for AI Chatbots

Lastly, while AB 1609 is primarily a customer-service chatbot law, it broadly applies to businesses with annual gross revenue exceeding $500 million nationally, which will sweep in a number of health care companies. Among other things, it requires these businesses to disclose to customers when they are interacting with an AI chatbot, and to connect the customer to a human agent within 15 minutes of a request during regular business hours. Notably for health care providers, AB 1609 does not apply to hospital communications related to the provision, coordination, management, administration, payment, or operation of health care services.

Violations are subject to up to $5,000 in civil fines for an initial violation and up to $10,000 for each subsequent violation. There is no private right of action for customers and exceptions exist for unforeseen circumstances and emergencies.

Vetoed AI in Health Care Bills

Governor Newsom also vetoed two bills that would have impacted the use of AI within California’s health care system.

SB 903 would have prohibited mental health service providers from delegating therapeutic decision-making to AI systems or allowing AI systems to communicate directly with clients to provide therapy. The bill also sought to provide greater privacy protections for patients, prohibiting the use of AI to record or transcribe psychotherapy communications unless the patient provided consent.

In vetoing SB 903, Governor Newsom reasoned that it was overly broad and would drastically limit a clinician’s use of AI technology that benefits the delivery of care. However, it is worth noting that the provisions of the bill parallel similar legislation that has been enacted in other states, most notably in Illinois. Against the backdrop of ongoing concerns regarding the use of AI in mental health care, driven in part by several high-profile lawsuits within the state, proponents may introduce a narrower version of this bill in future sessions.

In addition to SB 903, Governor Newsom vetoed AB 2575, which sought to protect the professional judgment of providers who override AI-assisted outputs in the delivery of care. The bill was introduced to address growing concerns about accountability and oversight as the use of AI expands into tools, such as EHRs, that are embedded in clinical decision-making and care delivery processes. Governor Newsom reasoned the bill would be impractical to enforce because it linked the scope of practice requirements to the state’s anti-retaliation protections, tying the state Labor Commissioner’s hands in establishing a violation.

Key Takeaways

California, like many other states, continues to enact legislation governing the development and use of AI, including in health care, despite objections from the Trump administration. These laws reflect a clear theme: AI may support clinical care and health care operations, but licensed professionals and other responsible individuals must retain appropriate oversight and accountability. California health care providers deploying AI tools should begin evaluating their operations and preparing to comply with the new statutory requirements.