The bug
A virtual agent handled subscription cancellations. When the caller declined a retention offer, it was supposed to capture their account number and transfer them. It transferred them, but never asked for the account number.
The instruction was one dense sentence. It contained a “don’t ask the caller for X” qualifier and, separately, an “ask for Y if it’s missing” step. The model over-generalised the “don’t ask” to both.
It was sneaky because the correctly scoped half of the same sentence worked in testing, which made the whole instruction look trustworthy.
The fix
I split the compound instruction into explicit lettered sub-steps, each with one unambiguous scope. The data capture worked from the next test onwards.
The rule I now follow
One conditional or qualifier per sentence. Never two bundled together. Agent instructions are code. Write them like code.