AI safety is mostly systems design
The safety meeting after the demo
The demo had gone well in the usual way. The model read a stack of internal documents, answered a policy question, suggested a next step, and produced a neat summary for the case file. The room nodded. Someone asked whether it could be connected to the workflow system. Someone else asked how soon. Then the safety officer asked what would happen if the model was wrong but persuasive. The room became very interested in the biscuits.
That question is where practical AI safety starts. Not in the abstract, and not in a slogan. In the shape of the system around a model that will sometimes be wrong, sometimes stale, sometimes overconfident, sometimes underconfident, and sometimes correct for the wrong reason. The safety issue is not only model behaviour. It is what the system allows that behaviour to do.
A model that drafts a sentence is one risk. A model that updates a benefit decision is another. A model that calls a tool with write access is another. A model that routes a patient, rejects a claim, changes a credit limit, or controls equipment sits in a different hazard class entirely. The same underlying model can be harmless, useful, risky, or unacceptable depending on boundaries, state, evidence, authority, monitoring, and recovery. This is why AI safety is mostly systems design.
Mostly is doing work in that sentence. Model research matters. Data quality matters. Alignment methods matter. Evaluation science matters. But when an organisation deploys AI, safety becomes operational. Who may ask. What may the model see. What may it change. What proof is required. What state is preserved. What happens when sources disagree. When must a human approve. How is drift detected. How is a bad release rolled back. These are design questions before they are ethics statements. The ethics statement may be sincere. The system does not execute sincerity.
Safety is a control problem
When people hear safety, they often think of values, training data, refusal policies, and model alignment. Those are part of the picture, but a deployed system also needs controls. A control is something that changes what can happen: an access check, a schema, a timeout, a threshold, a human approval, a sandbox, a rate limit, a verifier, a rollback, a kill switch, a record that cannot be edited by the serving path. Controls are boring until they are missing. Then they become the entire meeting.
A safe AI system assumes the model may fail and designs the surrounding machinery accordingly. It separates suggestion from action. It limits tool authority. It checks evidence before state changes. It makes uncertainty visible. It records model version and source scope. It refuses when required fields are missing. It gives operators a way to pause automation. It treats a model output as a candidate, not as a decree from a very articulate spreadsheet.
This is ordinary safety engineering with a language model in the middle. Aviation, medicine, rail, industrial control, and finance have all learned that important systems need layered controls because no component is perfect. The AI version feels new because the component speaks. Speech makes failure socially persuasive. A wrong answer that sounds calm can slide past a tired operator more easily than a blinking red error. The control design must therefore be more stubborn than the prose.
Control design starts by asking what the system is allowed to affect. If the AI can only draft, safety focuses on source quality, hallucination checks, user interface clarity, and review. If it can route work, safety adds state tracking, queue integrity, service levels, and appeal paths. If it can call tools, safety adds permission scopes, argument validation, transaction logs, and compensating actions. If it can affect rights or physical systems, safety requires proof, formal review, restricted autonomy, and serious incident planning. The model is one part. The authority envelope is the other.
The boundary is the product
The most important line in an AI system is often not visible in the interface. It is the boundary between reading and writing, recommending and deciding, summarising and recording, assisting and acting. Weak boundaries are how harmless prototypes become unsafe services. A model that begins as a drafting aid gets a button to apply the draft. Then it gets a tool to update the case. Then it gets a routing rule because the team is busy. Six months later nobody can explain which decisions are automated and which are merely assisted. This is not malice. It is scope creep wearing comfortable shoes.
Good boundaries are explicit and inspectable. The model may read these sources, not those. It may propose these fields, not commit them. It may call this tool with these arguments, not arbitrary text. It may route low-risk cases, not contested cases. It may answer only when evidence is current. It may not use private notes for public explanations. It may not cross from one tenant, region, account, or legal basis into another because the next paragraph looked helpful.
Boundaries also need to be enforced outside the model. A prompt that says do not write to the customer database is not a boundary if the tool has write access and no gateway. A system message that says cite sources is not a boundary if answers can be stored without citations. A policy paragraph that says human approval is required is not a boundary if the workflow lets automation mark approved. Safety instructions inside the model are useful, but they are not enough. The system has to make unsafe paths difficult or impossible.
This is the unromantic work that actually protects people. Typed tool calls. Narrow scopes. Read-only defaults. Separate approval services. Idempotent operations. Audit logs. Versioned policies. Queue states. Explicit refusal states. The vocabulary sounds like a backend engineer had a quiet weekend with a clipboard. Good. Safety in production usually looks like that. It is not trying to win a slogan contest.
State is where safety becomes real
AI conversations can make systems look stateless. A user asks, a model answers, the screen changes. Production is not stateless. Cases move. Tickets age. Permissions expire. Documents are corrected. A model version changes. A human overrides a recommendation. An appeal arrives. A user asks why something happened last month. Safety depends on whether the system preserved the state needed to answer.
State tells the system what has already happened and what may happen next. Without it, the model sees a slice and guesses the rest. That is how duplicate actions happen, deadlines vanish, approvals are assumed, and old evidence re-enters a new decision. A safe workflow records explicit states: draft, evidence missing, policy blocked, review required, approved, executed, challenged, corrected, recalled. These states are not decorative. They prevent the model from treating a consequential process like a chat transcript with ambitions.
State also gives humans leverage. An operator can inspect why a case is in review. A manager can see whether automation is creating a backlog. An auditor can replay which source versions were active. A user can challenge an outcome with something more solid than a screenshot. A developer can measure whether a new model increases refusal, escalation, or rework. Without state, every safety conversation becomes theatre. People talk about responsibility, but the system has misplaced the nouns.
For AI, state should include model and policy version, prompt or template identifier, source set, retrieval timestamp, tool calls, confidence or uncertainty, refusal reason, human action, and downstream effect. Not all of that belongs in one log line, and not all of it should be visible to everyone. Privacy and security matter. But the information must exist under controlled access. Otherwise safety becomes dependent on memory, and memory is a famously poor database with excellent excuses.
Evidence is the safety rail
A model answer is safer when it is connected to evidence. That sounds obvious until the system is under pressure. Evidence is skipped because it adds latency. Citations are stored as text rather than identifiers. Retrieval scores are not preserved. Source versions are overwritten. The model writes a confident explanation from a document that has since been corrected. Later, everyone asks why the answer was wrong. The answer is that the rail was decorative.
Useful evidence is not a footnote. It is a chain. The request entered with an identity and purpose. Sources were selected under permission. Retrieval found specific chunks or records. The model produced an output under a named version. A verifier checked claims or thresholds. A human accepted, edited, or rejected. The action changed state. The system stored enough of that chain to replay or challenge it. This is not bureaucracy for its own sake. It is how safety survives time.
Evidence also protects the model from being asked to do impossible work. If the system requires current sources, the model can refuse stale input. If the system requires two matching records, it can flag disagreement. If the system requires source identifiers, it can stop unsupported claims. The model no longer has to turn missing proof into fluent uncertainty management. It can return a safe state. This is kinder to the model and much kinder to the person affected by the output.
There is a balance. Evidence capture should not become surveillance. Sensitive content may be hashed, redacted, separated, or retained for limited periods. The point is not to record everything forever. The point is to preserve the facts needed to inspect important behaviour. Safety without evidence is just trust with a nicer jacket. It may look fine in the lobby. It does not hold up in an incident.
Humans need designed authority
Human in the loop is often used as a magic phrase. It is not magic. A tired reviewer with weak evidence, unclear policy, and a queue of two hundred cases is not a safety system. That is a person placed at the end of a conveyor belt and asked to be governance. It may work briefly, especially if the person is experienced and stubborn. It is not a design.
Human authority has to be specific. Which cases require review. What evidence does the reviewer see. Can the reviewer change structured fields or only approve prose. Does the reviewer know whether the model was uncertain or whether policy required escalation. Are disagreements fed back into evaluation. Are reviewers protected from automation bias. Can they pause a workflow. Can they escalate to a specialist. Can they see similar past cases. Can they explain a decision to the affected person. These details decide whether human oversight is real or ornamental.
Good oversight also separates judgement from clerical salvage. People should not spend their time finding missing source identifiers, deduplicating retries, translating model prose into workflow states, or guessing which policy version applied. The system should package the case. The human should apply judgement where judgement is needed: contested meaning, exception, proportionality, context, empathy, and responsibility. If the human is mostly cleaning up infrastructure gaps, the organisation has confused staffing with safety.
The design must also allow humans to disagree with automation without friction becoming punishment. If overriding the model takes eight clicks and accepting takes one, the interface has a policy. If escalation makes a team look slow, people will avoid escalation. If metrics reward throughput only, safety reviews will become ceremonial. Systems design includes incentives, screens, queues, and defaults. The model may not know the metric, but the people certainly will.
Locality changes risk
Where AI work happens matters. A system that sends every document to a remote model has different risks from one that performs extraction near the data and sends only a derived field. A system that stores traces in the serving service has different risks from one that keeps independent audit records. A system that relies on a single external router has different recovery options from one that can degrade locally. Locality is not nationalism with a network diagram. It is the practical question of which risks move when data, compute, policy, and evidence move.
Some work should happen close to sensitive data because moving the data creates exposure. Some work should happen close to users because latency affects safety. Some work should happen in a central service because specialised review or hardware is needed. Some evidence should be stored away from the model serving path because the serving path may be the thing under investigation. The safe design is rarely everything local or everything remote. It is deliberate placement.
Locality also affects legal and organisational accountability. If a model call crosses regions, which law governs the data. If an index is built outside the source system, how are deletions propagated. If a supplier changes a model, how is notice received and testing performed. If logs sit in a vendor portal, can the organisation export them during an incident. These are not abstract sovereignty questions. They are safety questions because they determine whether the organisation can control and explain the system under stress.
The wrong placement can make safe behaviour expensive. If every verification step requires a remote round trip, teams may sample instead of checking every case. If audit data is hard to join, investigations become slow. If private data must travel for simple classification, risk increases for no gain. Good locality design reduces the cost of doing the safe thing. This matters because controls that are too expensive will eventually be described as optional, which is how many bad stories begin.
Evaluation has to follow the system
Many teams evaluate the model and forget to evaluate the system. They test whether answers are correct on a set of prompts, then deploy into a workflow with permissions, source freshness, queues, tool calls, human review, retries, and changing policies. The model benchmark may still be useful, but it no longer describes the whole risk. Safety failures often live in the joins.
A system evaluation tests end-to-end behaviour. It asks whether the right source is retrieved for the right user. It checks whether stale data is refused. It verifies that tool calls are blocked without permission. It measures whether high-risk cases reach humans. It tests rollback. It tests queue delay. It tests model upgrade paths. It tests weird but ordinary cases: missing date, duplicate customer, region exception, appeal after correction, source deletion, partial outage, overloaded reviewer. Production has a taste for the cases nobody thought were dramatic enough.
Evaluation should include negative capability. The system must be good at not doing things. Not answering without evidence. Not crossing permission boundaries. Not escalating to action when only drafting is allowed. Not hiding uncertainty. Not using a source after deletion. Not continuing when a tool returns inconsistent state. Safety depends as much on refusal and pause as on accuracy. The most important output of a safe AI system may be not enough evidence, which is a sentence no demo team ever puts on a banner.
System evaluation also changes release management. A prompt change, model change, index rebuild, policy update, tool permission change, or reviewer interface change can alter safety. Each needs a test scope and rollback plan. The team should know which behaviours changed and which risks increased. This is tedious in the way seatbelts are tedious. It becomes interesting only when absent, which is too late for comfort.
The lesson
AI safety is mostly systems design because harm usually travels through systems. A model output becomes risky when a workflow trusts it too much, a tool gateway is too wide, evidence is missing, state is vague, review is overloaded, logs are weak, or rollback is imaginary. The model may be the visible source of the sentence, but the system decides whether the sentence becomes action.
This view is less dramatic than many AI safety debates, and more useful for organisations deploying systems now. It asks teams to draw boundaries, name authority, preserve evidence, design states, place work deliberately, evaluate end to end, and give humans real power. It treats the model as a powerful component in a safety case, not as the whole safety case. That is the adult position. It also gives engineers something to build besides another policy PDF, which is kind to everyone.
The safety officer's question after the demo was exactly right. What happens if the model is wrong but persuasive. A safe system should have an answer: it can only draft here, it must cite current sources there, it cannot call that tool, this case goes to review, this action leaves a receipt, this workflow can be paused, this release can be rolled back, this decision can be challenged. Those answers are not vibes. They are design.
There will always be uncertainty in AI. The practical task is to stop uncertainty from becoming unbounded authority. That is done with interfaces, states, controls, evidence, locality, evaluation, recovery, and human judgement that has been properly equipped. In other words, systems design. Not because values do not matter, but because values have to be made executable before they can protect anyone.