There’s a moment most engineers hit after shipping their first AI feature. The model is live. Users are interacting with it. And then something unexpected happens, a hallucinated response, a sensitive data leak in a prompt, an output that embarrasses the company. You start wondering: was there a checklist I was supposed to follow? A review process? Someone I should have talked to?
The answer, increasingly, is yes. And that answer has a name: AI governance.
This is not a compliance lecture. Compliance is what happens when lawyers talk to each other. Governance, done right, is what happens when engineers think carefully before they ship. This blog is about the practical side of that: what you actually need to understand, what you need to build, and why it matters more now than it did two years ago.
Why Governance Became an Engineering Problem
For a long time, “responsible AI” was treated as someone else’s problem. Ethics teams handled the philosophy. Legal handled the risk. Engineers handled the code. That separation made sense when AI was mostly research-grade.
That separation no longer makes sense.
When you integrate an LLM into a product, you’re making dozens of small decisions that have real governance implications. How are you handling user data before it goes into a prompt? What happens when the model returns something harmful or wrong? Do users know they’re interacting with a model? Who reviews edge cases? What’s the fallback when the API is down?
These aren’t policy questions. They’re engineering decisions that happen in pull requests.
In July 2024, NIST released its Generative AI Profile (NIST AI 600-1) as a companion to the existing AI Risk Management Framework (AI RMF 1.0). Rather than treating generative AI as a single risk category, the profile highlights multiple risks unique to or intensified by generative AI, including confabulation, data privacy, information integrity, information security, harmful bias, intellectual property, human-AI configuration, and value-chain risks. None of those risks live in a policy document. Every single one of them lives in your codebase.
The Frameworks Worth Knowing
You don’t need to read every governance framework cover to cover. But you should know what they say about the things you’re building. Here’s the short version of what actually matters:
NIST AI RMF organizes AI risk into four functions: Govern, Map, Measure, and Manage. Think of it as a lifecycle framework. Govern means having clear ownership and accountability. Map means understanding what risks your system could create in its specific context. Measure means evaluating those risks with actual testing, not just assumption. Manage means having documented responses when things go wrong. These four words are worth putting on your team’s internal wiki.
OWASP Top 10 LLMs and GenAI Apps is probably the most immediately useful thing for engineers. It reads like a security checklist you can actually act on. Prompt injection sits at the top, and rightfully so. If user input can manipulate your model’s behavior in ways you didn’t intend, that’s a vulnerability, not a UX quirk. The 2025 OWASP Top 10 for LLMs and GenAI Apps expands the security lens beyond prompt injection to include sensitive information disclosure, supply-chain risk, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. If you’ve shipped web features before, the mental model maps directly.
The EU AI Act entered into force on 1 August 2024 and applies progressively. Prohibited AI practices and AI literacy obligations began applying from 2 February 2025, while governance rules and obligations for general-purpose AI models began applying from 2 August 2025. For product teams, the important lesson is not just the legal timeline, but the risk-based structure: systems used in high-impact areas such as employment, education, credit, healthcare, law enforcement, or critical infrastructure may face stricter obligations around documentation, oversight, transparency, and risk management. Even companies outside the EU should pay attention, because the Act can apply where AI systems are placed on the EU market or where their outputs are used in the EU.
ISO/IEC 42001:2023 is the first international management system standard specifically for AI. It’s most relevant for companies building AI into their core product offering rather than just integrating an API. Think of it as ISO 27001, but for AI risk management. If your organization is pursuing any kind of enterprise sales, expect procurement questionnaires to ask about this.
Google’s SAIF (Secure AI Framework) and Microsoft’s Responsible AI Standard v2 are useful benchmarks for what mature AI engineering organizations actually require internally. They’re both publicly available, and reading them is genuinely informative, not because you’ll implement everything, but because they show what responsible defaults look like at scale.
What Governance Actually Looks Like in Code
Frameworks are useful orientation. But what does governance actually look like when you’re writing a PR? Here’s where each principle lands in practice.
Data Handling
Before a user’s input touches a model, ask: should it? PII, names, emails, addresses, financial data, health information, should be stripped or pseudonymized before hitting the prompt. Not after. Not sometimes. Always, by default, in code.
Log what goes into your prompts. Not the sensitive data itself, but metadata: session IDs, timestamps, prompt templates used, approximate token counts. If something goes wrong later, you need to be able to investigate. If you can’t audit what inputs caused what outputs, you don’t have a debuggable system, you have a black box that occasionally misbehaves.
For teams storing prompt history or fine-tuning on user interactions, the data lifecycle needs the same treatment as any other user data: retention limits, deletion rights, access controls, encryption at rest. The model makes it feel different. It isn’t.
Output Validation
Model outputs are untrusted input. Full stop. Treat them the way you’d treat form submissions from the internet.
For structured outputs, validate the schema before using the data downstream. A model told to return JSON will usually return JSON. “Usually” is not a contract. Parse defensively, catch exceptions, and have a fallback path when the format is wrong.
For natural language outputs going to users, think about what the model could plausibly return that would be harmful, factual errors, hallucinated citations, toxic content, personally identifying information about third parties. The risk profile depends entirely on your use case, but you need to have thought about it before you ship, not after the first incident.
Output filtering isn’t perfect, but it’s not optional either. At minimum, test your system against adversarial prompts before launch. Red-team it yourself. The OWASP Top 10 for LLMs and GenAI Apps gives you a starting vocabulary for the kinds of attacks to test against.
User Consent and Transparency
Users should know when they’re talking to a model. This seems obvious. In practice, it’s often deprioritized in the sprint where “polish” gets deferred.
Beyond the basic disclosure, there are design questions that have governance implications: Can users opt out of AI-generated responses in contexts where that matters? Are you using their conversations to improve the model? Can they delete their data from training pipelines if they ask? These aren’t edge cases, they’re things users will increasingly expect and regulators will increasingly require.
Transparency is becoming both a product expectation and, in many contexts, a legal requirement. Under Article 50 of the EU AI Act, providers of AI systems intended to interact directly with people must ensure users are informed that they are interacting with an AI system, unless this is obvious from the context. The same article also includes transparency obligations for AI-generated or manipulated content, such as marking synthetic audio, image, video, or text outputs where applicable.
Audit Logs
If you’re deploying AI in any context where decisions matter, customer support escalations, content moderation, financial advice, healthcare information, you need audit logs. Not application logs. Audit logs: structured records that capture what input came in, what the model returned, what action the system took, and what the user did next.
This serves multiple purposes. It enables post-incident investigation. It enables bias and quality audits. It gives you the evidence base to improve your system over time rather than guessing. And in regulated industries, it’s often a legal requirement.
Design your logging schema before you ship, not after. Retrofitting audit infrastructure onto a running system is miserable, and you’ll inevitably find you missed capturing something important.
Explainability and Human Oversight
“The model said so” is not a satisfying explanation for a user who was denied service or shown a harmful output. In high-stakes contexts, someone needs to be able to explain, at least at a general level, why the system behaved the way it did.
For engineers, this usually means two things. First, don’t use AI for decisions that have no human review path when the stakes are high enough to warrant it. Second, log enough context that a human reviewer can understand what happened without reverse-engineering the prompt from scratch.
NIST AI RMF’s Govern function specifically calls out the need for clear accountability; someone has to own the system’s behavior. On a product team, that ownership should be named and explicit before you launch, not discovered during a crisis.
Safe Fallback Behavior
What does your system do when the model returns garbage? When the API is down? When confidence is too low? When the output fails validation three times in a row?
Every AI feature should have a defined fallback state: a non-AI path that handles the request gracefully. This might mean surfacing a manual support option, returning a cached response, or simply showing an error message that doesn’t expose implementation details. What it should never mean is silently returning an empty or malformed response and hoping users don’t notice.
Build fallbacks into the design, not as an afterthought. The moment you treat the model as a dependency that can fail , which it can, like any other dependency, your architecture decisions improve.
On Security: The OWASP Angle
If you take one thing from the security side of AI governance, make it this: prompt injection is SQL injection for the LLM era. An attacker who can control what goes into your prompt can potentially control what comes out of it. If your system acts on model outputs, sends emails, queries databases, executes code, the blast radius is significant.
Mitigations follow familiar patterns: sanitize and validate input, separate user-controlled content from system instructions clearly, don’t give the model more permissions than it needs, treat the model’s output as untrusted before any downstream action.
The other OWASP categories worth engineering attention are sensitive information disclosure (models can leak training data in edge cases), insecure plugin/tool design (if your model has tools, scope them tightly), and supply-chain vulnerabilities (the model you’re using is a dependency you need to evaluate like any other).
A Pre-Ship Checklist for AI Features
Before you open a launch PR for an AI feature, these questions should have answers:
- Where does user input go before it reaches the model, and is PII handled appropriately at each step?
- What’s the worst realistic output the model could return, and how does the system handle it?
- Are users informed they’re interacting with an AI where required?
- What’s the fallback behavior when the model fails or returns invalid output?
- Are outputs validated before being used downstream, especially for structured data?
- Do audit logs capture enough context to investigate a user complaint or incident?
- Is there a named human who owns this system’s behavior and can be escalated to?
- If your users are in the EU, does this system fall under the EU AI Act’s risk categories?
This isn’t an exhaustive compliance checklist. It’s a minimum sanity check. For teams building in regulated industries or high-stakes domains, the bar is substantially higher, but this is a reasonable starting point for most product engineers shipping AI features for the first time.
The Bigger Picture
Governance frameworks can feel like they were written for legal teams rather than engineers. They often were. But the risks they describe are real, and the responsibility for managing them increasingly falls on the people who write the code.
The good news is that most of what responsible AI engineering requires isn’t exotic. It’s the same things good engineering has always required: thinking carefully about failure modes, testing against adversarial inputs, designing for auditability, building fallback paths. The frameworks give those instincts structure and language. The OECD’s AI Principles, adopted back in 2019, describe the goal simply: AI should be trustworthy. Robust. Transparent. Accountable. That’s not a compliance target, it’s an engineering standard.
The engineers who understand this now will be the ones trusted to build more ambitious AI systems later. The teams that treat governance as a tax will eventually pay it in incidents instead.
Ship thoughtfully. It’s faster in the long run.
