Why the maker could not build a creature that only does good
The obvious objection to my first installment in this series goes like this: if a maker knew things would go sideways with their creation, why not build creatures that simply could not do harm? Remove the option. Problem solved.
The biblical story answers this question with a tree. Eden has a tree, and the instruction attached to it is not in fine print. It is a neon sign: one rule, stated plainly, with the consequence spelled out. Whatever else you believe, the design logic is clear. The maker did not hide the line or set a trap. And if a creature cannot choose otherwise, its obedience means nothing, and what you have built is not an agent. It is an appliance. A toaster never disobeys. It also never writes a poem, closes a deal, or catches the mistake buried in paragraph nine of your contract.
That is the heart of this post. You cannot remove the ability to do harm without removing the ability to do good, because both come from the same place: the ability to act beyond the script. In the real world, the forbidden fruit grows on the same tree as everything else.
Two connected dials
Think of two dials, one for capability and one for safety. Capability brings the reward, and it brings the risk with it. Safety holds the risk down, but every control has a cost, and too much of it can crush the opportunity it was meant to protect. The problem is that nothing stops us from cranking the first to eleven without touching the second. The dials are real, and balancing the two is the job, because they are wired together. The more you ask of the system, the more you need around it, and the more you build around it, the more you can safely ask of it.
Here is what wired together looks like. The same model that drafts a grant proposal in four minutes also drafts a phishing email in four minutes. The system that can read ten thousand pages of discovery in an afternoon can read ten thousand pages of anything. The assistant that writes a persuasive legal brief can invent a court case, with the same confidence, and nobody has to push it. In each case, the risk lives in the gap between what we built it for and what it can do. Sometimes a person finds the gap. Sometimes the system wanders into it on its own.
Yes, the guardrails exist, and the good ones work most of the time. Makers train their models to refuse clearly harmful requests, layer instructions on top, and filter what goes in and comes out. But a guardrail is a control, not a removal of the capability. Intent is invisible, so the same phishing request might come from a criminal or from the team building your staff training. Controls are made of language and probability, not locks, so clever rephrasing or a task broken into harmless-looking pieces can slip past them. And some models are copied and modified with the safeguards removed. A guardrail lowers the odds. It does not change what the system can do.
Organizations fail in both directions. Greed turns up the reward and leaves the controls behind. Leadership underestimates how tightly the two are connected. And fear piles on so many controls that the opportunity never gets a chance.
The employee bargain
I have been fortunate to run my own business for more than 20 years, and here is what that taught me: every employee you hire can steal from you. Every one of them can misuse a customer list, shade a number, or take a shortcut that hurts someone. I have never led a team or run an organization where I could prove that nobody would. I have, however, decided that the value of trusting capable people outweighed the risk, and then built controls around that trust.
Those controls are familiar. Segregation of duties. Approval limits. Audit trails. Consequences. Nobody builds a company out of people who are physically incapable of misconduct, because that company has no employees. It has furniture.
AI deserves the same bargain, stated honestly. We give it capability because capability is the point, and we govern it because capability cuts both ways.
Overcorrection is also a failure
A tempting way out is to lock everything down. A system that refuses anything remotely risky can never embarrass its maker. But look at how the two kinds of failure get noticed. When an AI helps someone commit fraud, there is a headline, a lawsuit, and a postmortem. When an AI refuses a legitimate request, such as the nurse checking a drug interaction or the investigator studying a fraud pattern, nobody writes anything. Organizations learn to fear the first failure and ignore the second, and they drift toward overcontrol.
Anyone who has run a process knows the pattern. Make an approval take three weeks and people stop asking for approval. They build a side channel. The temple that locks every door gets a thriving alley out back. In AI terms, over-restricted tools push people toward less careful ones, and the net result is less safety, not more.
Autonomy raises the stakes
Both dials get heavier as AI moves from answering questions to taking actions. A model that suggests an email is one thing. An agent that sends it, moves money, or changes a record in your system of record is another. Eden began with one rule when the creatures were few and the stakes were small. The rules multiply as the creatures spread and gain power, and that is not an accident. Proportionality is the principle: the more a system can do without a human in the loop, the more you need to know about what it did and why.
The working question is how to set the two dials so the value justifies the controls, and what those controls must be. That question has no answer, and the Old Testament spends several hundred pages demonstrating it.
Next up: the maker, as it turns out, tried just about every control there is.