Public and/or Shared AI Models Cannot be Trusted Until an AI Bill of Materials Become the Norm

Last Updated on January 14, 2024

Public and/or shared AI models cannot be trusted to return untainted data unless a comprehensive AI Bill of Materials (AI BOM) is available.

Similar to the Software Bill of Materials (SBOM) concept that is increasingly taking hold, especially in the US government sector and the military, an AI BOM would catalog the components of a Large Language Model (LLM) or other AI model to help identify vulnerabilities, manipulation, reduce risk, and better secure the AI supply chain.

Why do AI BOMs matter?

The cyber risk associated with AI models is massively compounded by sharing and reuse. Everyone wants to build AI/ML applications, but few have the time, computational resources, and technical expertise required to architect them from scratch. Hence a growing reliance on third-party APIs and pre-trained models.

But along with that reliance comes a requirement to understand those third-party upstream components. If their creators cannot ensure traceability and trustworthiness, then neither can any downstream system that consumes these components. An organization’s vendor risk management program and subsequent questionnaires should ask AI related questions (as appropriate) aligned with a trusted framework such as the NIST AI Risk Management Framework.

LLMs like ChatGPT, Google Bard, and Meta’s recently leaked LLaMA are incredibly powerful and beginning to have massive social, economic, and cyber risk impacts. Some of the business risks and AI governance challenges that LLMs pose or intensify include discrimination, bias, toxicity, adaptability to organized disinformation/misinformation campaigns, and increased risk to privacy and information security.

Manipulating people and events with disinformation is already happening on social media, and LLMs can amplify the harm by targeting these campaigns more effectively and making it even harder to sort legitimate from manipulative content. Likewise, data privacy violations are already rampant online, and LLMs can make it even easier to extrapolate personal identities and undermine privacy safeguards.

What should AI BOMs cover?

Consumers of AI need to know that risks like poisoned datasets and hidden backdoors aren’t present in the code they are reusing. But there is currently no practical approach to establish the provenance of an AI model, including the algorithms and data used during training. What’s needed to ensure trustworthiness with shared LMMs is a way to know about the traceability and observability of third-party AI components and algorithms, the features and parameters tested, and the datasets used.

To highlight in real-world terms how LLMs can be compromised, researchers recently modified an open-source AI model to report misinformation and distributed it on Hugging Face. Their experiment proved that open-source LLMs can be straightforwardly exploited to disseminate “fake news” while performing other tasks as expected. The motivation behind this hack is to “raise awareness of the critical importance of a secure LLM supply chain with model provenance to guarantee AI safety.”

The call for greater government and private sector regulation of the AI supply chain, including a mandate for AI BOMs, is growing louder—but how to enforce it? The reality is that AI systems are increasing their sphere and level of impact far ahead of the pace of government action. Consumer demand will play an important role in protecting the AI supply chain. But leaving governance up to the marketplace has been proven again and again to be inadequate.

Another challenge or point of resistance with implementing AI BOMs is the potential that data requested for transparency could enable experts to reverse-engineer a system and compromise the vendor’s intellectual property. Proprietary algorithms and other AI-related IP needs to be protected. But not at the expense of escalating cyber risk across the board for businesses.

What’s next?

Software supply chain security is one of the biggest challenges facing our industry, and the potential for misuse of AI models is surely one of the fastest-growing sources of cyber risk for organizations individually and collectively.

Governing application security starts with a best-practice approach that covers all the many facets of the software development lifecycle, including identifying vulnerabilities in APIs and other third-party code, implementing DevOps-friendly security testing, and validating “provable security” against widely respected guidance.

Contact Pivot Point Security to speak with an expert about how best to reduce your company’s cyber risk from AI by aligning your organization with the principles and actions outlined in the NIST AI Risk Management Framework.