Objective

To assess various New York State agencies’ progress in establishing an appropriate artificial intelligence (AI) governance structure over the development and use of AI tools and systems. The audit covered the period from January 2019 through November 2024.

About the Program

AI-powered tools have an increasingly significant role in industry operations including agriculture, health care and medicine, manufacturing, transportation, and government, enabling entities to improve government operations. In recent years, the use of AI has rapidly gained popularity amongst public and private institutions. As AI has been growing and benefiting business, it is also giving rise to a host of unintended consequences.

AI systems create unique challenges in accountability as their inputs and operations are not always visible. Additionally, AI systems are frequently complex making it oftentimes difficult to detect and respond to failures when they occur. Governance over AI systems, including transparency, should be established to promote accountability and responsible use of such systems. The U.S. Government Accountability Office noted that a lack of transparency reduces effective oversight in identifying errors, misuse, and bias. Therefore, it is essential to establish governance structures over AI to ensure that its use is transparent and accurate and does not generate harmful, unintended consequences.

New York State (NYS) and some of its agencies have been using AI to aid their operations. For example, the Department of Motor Vehicles (DMV) uses AI to power its facial recognition technology, improve highway safety, and deter identity fraud, and the Department of Corrections and Community Supervision (DOCCS) uses AI in its telephone system that allows incarcerated individuals to communicate with family and friends.

The Office of Information Technology Services (ITS) was established in 2012 to centralize IT services and develop technology services for State government. ITS provides statewide direction, directs IT Policy, and delivers centralized IT products and services that support the mission of the State. In January 2024, as directed by the Governor, ITS issued the Acceptable Use of Artificial Intelligence Technologies policy (AI Policy), providing guidelines and requirements for the acceptable use of AI technologies by State Entities (e.g., State Agencies).

Key Findings

NYS does not have an effective AI governance framework. While the AI Policy outlines certain requirements or recommendations for agencies to consider, it lacks adequate guidance and procedures on how these agencies can meet these expectations.

We judgmentally sampled four State agencies: Office for the Aging (NYSOFA), DOCCS, DMV, and the Department of Transportation (DOT). Based on our survey results, we found the AI governance at the four sampled agencies varies significantly. Some agencies have identified key risks and taken steps to address those risks, while others have not created any AI-specific policies or taken other steps toward effective AI governance. These incomplete approaches to AI governance do not ensure that the State’s use of AI is transparent, accurate, and unbiased and avoids disparate impacts. For example, none of the agencies required or developed specific procedures to test AI systems in order to evaluate whether outputs were accurate or biased. Several factors contributed to the inconsistent and inadequate AI governance and compliance with ITS’ AI Policy, including lack of statewide guidance, poor identification of AI technology, and non-existent training.

Key Recommendations

To ITS:

• Amend the Acceptable Use of Artificial Intelligence Technologies policy and provide additional guidance or procedures to assist State Entities with adopting AI technologies.
• Coordinate with all State Entities to provide support in developing AI governance structures as necessary.
• Develop and coordinate statewide training to ensure key AI risks and risk management options are understood by appropriate staff.

To NYSOFA, DOCCS, DMV, and DOT:

• Review ITS’ Acceptable Use of Artificial Intelligence Technologies policy to identify areas within the agency that need to be strengthened.
• Implement policies to create an effective AI governance structure.
• Coordinate with ITS on developing AI governance structures as necessary.

To DMV:

• Review the facial recognition system with ITS to determine compliance with ITS’ Acceptable Use of Artificial Intelligence Technologies policy.