AI Governance by Design: Building Controls Into the Development Lifecycle
Instead of Auditing After Most AI governance programs are assembled in the wrong order. A model goes into production, someone in legal or risk asks what it was trained on, and a project starts to reconstruct the answer months after the people who made the decisions have moved on.
The reconstruction rarely works. Training data provenance cannot be recovered retroactively if nobody logged it. Bias testing performed after deployment tells you what is already affecting users. A threat model written after the architecture is fixed becomes a list of things you now cannot change.
The alternative is not more governance. It is governance placed at the points in the lifecycle where decisions are actually made, in a form that produces evidence as a byproduct of the work rather than as a separate exercise.
This article sets out six control gates, what each one produces, who signs, and what to do when the organization does not have the roles the framework assumes.
Why AI Breaks the Standard SDLC Model
Conventional software assurance rests on an assumption that does not hold for machine learning systems: given the same input and configuration, the system produces the same output, and testing that output establishes correctness.
Three properties break this.
The behavior is learned, not specified. There is no requirements document against which model behaviour can be verified. The training data is the specification, which means data governance is now a functional safety control, not an administrative one.
The system degrades without any change to it. Data drift and concept drift mean a model that passed acceptance testing can fail eighteen months later while every line of code remains identical. Nothing in a standard release process detects this.
The attack surface includes the input itself. Prompt injection, model inversion, and training data poisoning have no equivalent in deterministic systems. Input validation is not sufficient because the input is also the instruction.This is why the gates below sit where they do. Each one addresses a failure mode that cannot be caught later.
The Six Gates
Gate 1 — Before Anything Is Built: Impact Assessment
The decision being made: whether to build this at all, and under what risk classification.
What happens. The business sponsor completes an impact assessment before resources are allocated. The assessment establishes the use case, the population affected, the consequence of an incorrect output, and whether the system makes or materially influences a decision about a person.
That last question is the one that determines almost everything downstream. A model that drafts marketing copy and a model that screens job applicants are not in the same regulatory universe, and the difference should be established on day one rather than discovered during legal review.
Map the outcome to whichever regime applies: risk classification under the EU AI Act, the Map function under the NIST AI RMF, or the impact assessment obligations in ISO/IEC 42001 Clause 6. If more than one applies, use the strictest.
Who signs. Whoever can stop the project. In a large organization this is a risk committee. In a smaller one it is the person who owns the budget — but the sign-off must be recorded, because “we assessed it” without a dated artifact is indistinguishable from not assessing it.
What it produces. A dated impact assessment, a risk classification with the reasoning behind it, and a scope statement identifying what the system will and will not be used for.
The common failure. Assessment performed as a form-filling exercise after the build decision was already made. If the assessment has never once resulted in a project being modified or stopped, it is not functioning as a gate.
Gate 2 — Architecture: Threat modeling and Data Design
The decision being made: what the system is permitted to touch, and what happens when someone attacks it.
What happens. Threat modeling covers the AI-specific vectors — prompt injection where the system processes untrusted input, model inversion and membership inference where the model is exposed to users, training data poisoning where data is sourced externally or contributed by users.
At the same time, establish data lineage tracking and the boundary controls: what data classes may enter training, how personal data is minimised or excluded, and who holds access to the training corpus and the model artefacts. Role-based access control applied to the application but not to the training data leaves the sensitive asset unprotected.
Where a data protection impact assessment is required, it is written here, not after deployment.
Who signs. Security architecture and whoever owns privacy. If those are the same person, that is workable; if that person also built the system, it is not.
What it produces. A threat model, a data flow diagram covering the training and inference paths, a provenance record for training data, and the DPIA where applicable.
The common failure. Threat modelling the application while treating the model as a black box supplied by a vendor. If the model came from outside, the vendor’s controls are now your controls, and you need evidence of them rather than assurance about them.
Gate 3 — Build: Reproducibility and Pre-Training Checks
The decision being made: whether what was built can be explained a year from now.
What happens. Automated logging of data transformations, hyperparameters, and random seeds, so that a given model version can be reconstructed. This is not academic. When a regulator, a customer, or a plaintiff asks why a system produced a particular output, an unreproducible model means the question has no answer.
Bias metrics are computed on the training data before training, not after. Demographic parity and disparate impact ratios calculated on outputs tell you what has already gone wrong; the same measurements on inputs tell you what is about to.
Where the pipeline supports it, configure baseline skew to halt the run. A gate that requires a human to notice a metric is not a gate.
Who signs. The data science lead, with compliance reviewing the explainability output. Feature attribution methods such as SHAP or LIME are useful here, with an important
caveat: they explain what the model weighted, not why it is correct. Treat them as investigative tools, not as evidence of fairness.
What it produces. A model card, data lineage logs, and pre-training bias measurements with disposition.
The common failure. Reproducibility logging configured but never tested. Attempt torebuild an earlier model version once, deliberately, before you need to.
Gate 4 — Verification: Adversarial Testing
The decision being made: whether the system fails safely when someone tries to break it.
What happens. Functional testing establishes that the system works. Adversarial testing establishes what happens when it does not. The testing should cover prompt injection against any input path that includes untrusted content, boundary testing at the edges of the intended use case, out-of-distribution inputs, and — where the system generates factual claims — measurement of the rate at which it generates false ones.
Test the failure mode, not only the failure rate. A model that is wrong 3% of the time and signals uncertainty is operationally very different from a model that is wrong 3% of the time with full confidence, even though the metric is identical.
Who signs. Security, with independent verification. “Independent” means someone who did not build the system. In a small organization this may mean an external party for the initial assessment.
What it produces. A test report with methodology, findings, severity, and remediation status. Findings accepted rather than remediated should be documented with the acceptance rationale and the approver.
The common failure. Testing performed once at launch and treated as durable. Model updates invalidate prior testing. Tie re-testing to the change, not to the calendar.
Gate 5 — Deployment: Limiting the Blast Radius
The decision being made: how much damage a bad release can do before it is stopped.
What happens. This is the gate where conventional software resilience practice transfers cleanly. The CrowdStrike outage in July 2024 was not an AI failure — it was a configuration update distributed globally at once, to a component with kernel-level privileges, without staged rollout. The lesson is about deployment topology, and it applies with equal force to model updates.
Release in stages: a canary population, health metrics validated in real time, then progressive expansion. Configure automated rollback triggered by defined thresholds , output distribution shift, latency, error rate, guardrail activations — rather than by someone watching a dashboard.Input and output validation should sit outside the model, at the API boundary. Guardrails implemented inside a prompt are instructions, and instructions can be overridden.
Who signs. Whoever operates the infrastructure, confirming that rollback has been tested rather than merely configured.
What it produces. Deployment configuration records, rollback procedure with evidence of testing, and an incident playbook naming who decides to roll back and who they call.
The common failure. Rollback that restores the previous model but leaves the changed data pipeline in place. Test the whole path.
Gate 6 — Operation: Monitoring and Scheduled Re-Evaluation
The decision being made: whether the system still deserves the authorisation it was given.
What happens. Monitor for data drift, concept drift, and anomalous usage patterns. Feed AI-specific telemetry into existing security monitoring — a spike in guardrail activations is a security signal, and it should reach the same people who see the rest of them.
Schedule re-evaluation independent of incidents. The original risk classification was made against a legal and business context that changes. A system classified as low-risk in 2024 may not be low-risk under an obligation that took effect since.
Who signs. Whoever owns enterprise risk, on a defined cycle, with three available outcomes: continue, retrain, or decommission. If decommission has never been exercised, verify that it is genuinely available.
What it produces. Monitoring records, drift analysis, re-evaluation decisions with reasoning, and a retraining log.
The common failure. Monitoring dashboards nobody is accountable for reading. Assign the eview to a named person with a defined cadence, or it will not happen.
Accountability
The framework assumes distinct roles. Most organizations do not have them. The principle that matters is not the org chart — it is that the person who builds the system does not approve it. Where that separation is impossible, document the limitation and compensate:
If You Are a Team of Eight
Most published AI governance guidance assumes a risk committee, a red team, and a DevSecOps function. If you have twelve people and two of them touch the model, the framework above still applies — the gates are decision points, not departments. What
changes is the implementation.
Collapse the roles, keep the separation. One person can hold Gates 1, 2, and 6 provided they did not build the system. The separation that matters most is between builder and approver at Gate 4.
Buy independence where you cannot staff it. External adversarial testing at Gate 4 for the first release of a high-risk system is the single highest-value expenditure in this framework for a small organization.
Reduce the artefacts, not the decisions. A one-page impact assessment that is genuinely completed and dated is worth more than a fifteen-page template that is copied forward. The evidentiary value is in the record of the decision, not its length.
Start at Gate 1 and Gate 6. If you can only implement two gates, implement the one that prevents building the wrong thing and the one that catches the system going wrong after launch. The middle four are recoverable; those two are not.
What This Is Actually For
Governance produces two things. The first is fewer failures. The second — the one that
matters when something does go wrong — is a record of how each decision was made, bywhom, and on what basis.
Organizations that struggle under regulatory scrutiny are usually not the ones with weak controls. They are the ones who cannot demonstrate that a decision was considered at the time it was made. Every gate above is designed to generate that record as a byproduct of doing the work, rather than as a separate exercise performed under pressure afterwards.
Build the controls in. Reconstructing them later does not work.
This article is general guidance and does not constitute legal advice. Applicable obligations depend on jurisdiction, sector, and use case.
Where to start
If you are building this from nothing, the first decision is which gate you cannot afford to skip. That depends on what you have in production and who is asking you about it.
If you are a defense contractor, the answer is usually Gate 1 — because scoping determines whether the system touches CUI, and that determination drives every obligation that follows. The related question of how AI vendors fall inside your assessment boundary is covered in Securing the Defense AI Supply Chain.
Cyber DSC works with defense contractors and their suppliers on CMMC Level 2 readiness, NIST SP 800-171 alignment, and the AI governance questions that now sit alongside them.
If you would find it useful to talk through where your program actually stands, book a 30-minute call. No slides. You leave with the two or three things worth doing first, whether or not you work with us.