Room 5 of 5
Limits
Last room. This is the one that matters most.
Everything you’ve seen so far tells you what the model can do. This room tells you how it fails.
And it fails in the most dangerous way possible.
It fails silently.
Look at the widget below. Four constraints are already loaded: four rules the model has been told to follow. “Be factual.” “Be creative.” “Be concise.” “Cite sources.”
Now look at the failure indicator. One constraint has already been quietly dropped. “Cite sources”: the model stopped doing it. It didn’t tell you. It didn’t flag an error. It just stopped.
And the output still reads fine.
Real models don’t drop constraints in this neat order — they fail unpredictably, which is worse. This demo shows the principle; reality is messier.
Try removing a constraint. Take one off. Watch the failure indicator go green: the model can handle fewer rules without dropping any.
Now add a fifth. “Match tone.” Watch what happens.
Two constraints dropped. The output still sounds fluent. Still reads like it was written by a professional. The kind of output that would sail through a boardroom review. But two of the rules you explicitly gave it have been silently abandoned.
Add a sixth. A seventh. Watch the gap between what the model appears to do and what it actually does widen with every constraint you add.
Go back to Room 1. Remember the temperature dial? The model sounds equally fluent whether it’s correct or guessing at random. This is the same phenomenon, at scale. It will never raise its hand and say “I can’t do this.” It will do it badly, and it will sound perfect doing it.
What this means when you use AI
Never trust AI output on complex tasks without human verification. The more constraints you give it — tone, format, factual accuracy, citations, length, audience — the more likely it is to silently drop one.
Build a verification step into your workflow. For important outputs, check: did it actually do everything I asked? The model will never tell you it failed. You have to check.
This is also why AI “hallucinations” are dangerous. The model can generate plausible-sounding but entirely fabricated information — invented citations, fictional events, made-up statistics — and present them with the same fluency as verified facts. The output sounds perfect either way.
The people who understand this will use AI brilliantly.
The people who don’t will use it confidently.
Which is worse.