Why 'Just Be Careful Next Time' Never Reaches an AI
In the field of supporting AI adoption at companies, I keep hearing the same complaint. "I keep correcting the same mistake." "I explain carefully and nothing changes." "Maybe my prompt is the problem."
At first, everyone is excited. The AI drafts an email, summarizes a meeting, runs through research faster than they could. This is going to change how we work. Then they keep using it, and every time they end up retouching the same parts. They make their instructions more careful, they write rules, nothing changes. The conclusion they slide toward is: "Maybe I'm not communicating well enough."
That feeling is real. But it is not a communication problem. It is a structural one.
The mechanism: "yesterday" doesn't reach the AI
The fact that AI doesn't carry memory across sessions is not a bug. It is by design.
Most AI services start from zero every conversation. You can carefully explain "review before marking complete" and "don't touch the numbers" today, and tomorrow you'll ask the same task and the AI will skip the same step. This is not poor performance. The conversation ends, the context is gone, the next conversation starts blank. That is the design.
Some people write the rules into a config file. "Confirm before completing." "Do not perform tasks that were not requested." It seems to work for the first few uses. Then the same thing happens in the same place.
The strange part is that the AI is not unaware of the rules. Ask "do you remember this rule?" and it will recite the content accurately. Point out the violation and it will say "you are right." It still skips the rule mid-task.
For an AI, "knowing something" and "stopping at that thing during a workflow" are different things. Putting the rule in the context doesn't guarantee that the workflow's processing order will respect it. As you add more rules, attention to each individual rule thins out. There is a report of someone who built an aggressive prompt-injection mechanism, found it had ballooned to 56,000 characters, watched the AI stop functioning, and had to cut it back under 1,200 characters.
Adding more rules has a ceiling. Beyond a point, "lower the probability of the same mistake by writing more carefully" stops working.
What the numbers say
"It never changes no matter how often I say it" is not a niche complaint about niche usage.
JUAS, in its 2025 IT Trends Survey, reports that among companies that adopted generative AI, only 4.0% said "the effect significantly exceeded expectations." "Roughly as expected" came in at 33.1%, "some effect" at 36.1%. Add the positive evaluations and you get around 70% — but the slice that says "this met the expectations we had going in" is small.
MIT's "The GenAI Divide: State of AI in Business 2025" puts the share of corporate generative-AI tools that reach production and produce outcomes at roughly 5%. "Tried" and "in actual operation" are not the same thing.
Persol Research's February 2026 survey on generative AI and work found that the share of users who actually reduced their working hours via AI was about 25.4%. One in four. There is also a counterintuitive finding: people who use AI more often have longer overtime.
The experience of "I introduced AI but my work didn't shrink" is not rare. The reason isn't the user. It is the structure of the counterparty you are talking to. Polishing your communication harder, against this structure, mostly adds wear without adding result.
From "ask it to be careful" to "build a path it can't skip"
When you change framing, the way you organize work changes.
Pick the places in an AI-assisted workflow where skipping a step would actually hurt. For those places, don't tell the AI "be careful here." Build a path where confirmation cannot be skipped.
For example: when the AI changes something, surface the impacted areas automatically. Or block progress to the next step until the confirmation completes. You are not asking the AI to be careful. You are narrowing the road down to a single lane. The AI didn't get smarter. The road got narrower. The same kind of mistake stops happening because there is no longer a side path to take.
This is not a story about raising AI accuracy. It is a story about designing without depending on accuracy. Any AI, however carefully instructed, has some probability of skipping. Start from that assumption and either make skipping harmless or make skipping impossible.
Tools like Cursor (the coding tool) with its "Rules for AI" and `.cursorrules` for git-managed per-project rules, or Devin with its Knowledge and Playbook features that share coding conventions across a project — these are extensions of this idea. Instead of teaching the AI the same things each session, they make sure the AI walks the same path each session.
Separate "let it run" from "put a stopper here"
When you hand work to an AI, there are two kinds of moment.
"If it skips here, I'll catch it later" moments, and "if it skips here, things break" moments.
The first kind, hand off generously. Email drafts? You read them before sending. The AI can make mistakes and there's a natural stopping point. Hand off whatever you can.
The second kind, change how you hand off. If you're asking the AI to produce a deliverable with numbers in it, don't trust the numbers the AI wrote. Separate "cross-reference against the source data" into its own step. Keep an "unverified" marker on the deliverable until that step completes. The act of removing the marker stays in human hands. That is a path where confirmation cannot be skipped.
This is not "can you trust the AI." It is "you cannot manage something without continuity using management designed for something with continuity."
A real-estate tech company called estie implemented this in practice: they ran the same task through multiple AI models (GPT-4, Claude, Gemini), automatically flagged the items where the models disagreed, and routed only those items to human review. Instead of humans checking everything, humans see only what the AIs themselves were uncertain about. This is not "design that prevents skipping" — it is "design that detects what got skipped and routes it to a human." Same direction.
Japan's METI/MIC "AI Operator Guidelines (v1.2)" from March 2026 also names "inserting human judgment at appropriate timing" as a principle. As AI agents act more autonomously, "where the human checks" becomes the design question.
Escape the "review everything / trust everything" binary
There is a pattern that AI users fall into. Either everything bounces back for human review, or everything proceeds without oversight. Both fail.
The first overloads humans. The second loses visibility. Neither is operational.
The only exit from the binary is to make the escalation criteria explicit.
Can't resolve in 2 minutes → switch approach
Stuck for 15 minutes → return to human
Design change beyond scope → always return to human
Otherwise → proceed autonomously
Put rules like this into the AI's system prompt. "When the AI should stop, and when it shouldn't" gets defined. Unnecessary interruptions drop. Reflexive "approve every action" monitoring tends to add friction without adding meaningful safety.
The model shifts from "approve everything in advance" to "monitor while it runs, intervene when needed." That shift is what makes AI actually usable for work.
Three months later, the surviving deployments have a fixed home
Watching AI rollouts across multiple clients, the difference between "still being used" and "quietly abandoned" usually comes down to: does the AI have a fixed place in the workflow?
For meeting notes: the AI drafts the notes after every meeting. The owner doesn't review the full phrasing — they only check the decisions and action items for omissions. No one has to decide "should I use AI here?" each time. They know what to look at after the AI runs.
For customer support: the AI drafts the first reply. The owner doesn't reread the whole message — they confirm that customer-specific context is reflected and that numbers and conditions are correct. Confirmation cost doesn't grow. The target of confirmation gets narrower.
For deck creation: the AI does structure draft and background research. The flow-of-argument decisions stay with the human. Without this split, "have AI make the deck" turns into reviewing the whole thing end to end, and you end up thinking "I should have just done it myself."
Without the line, the field hesitates constantly. Just the decisions? Or omissions too? With any uncertainty, people fall to the safe side: check everything, redo the research, return to source documents. The AI stops being something that reduces work and starts being one more thing to confirm.
The metaphor that fits is: it is the guardrail that lets you move forward. When you don't know how far you can trust the AI, the conscientious people are the ones who can't use it. Once "this much is OK to hand over" is visible, the workflow stabilizes.
Persol's survey has another number worth holding: among users who saved time with AI, 61.2% of the saved time was reinvested into work. The hours come back as more work. "Where do we cut" is the wrong question. "Where do humans actually own the call" is closer to the real one.
Move the placement of your expectation
If you are someone who keeps thinking "I tell the AI carefully and it still fails in the same place" — the first thing I want to say is: it isn't a communication failure.
AI is built that way. It doesn't carry yesterday's instruction into today. This is the current spec, not a skill issue on the operator side.
Once "this is a design problem, not a communication problem" lands, the view changes. You start thinking ahead — "this is a place where skipping would hurt, so I'll put a physical stopper" — instead of "I'll write the instruction more carefully." When the same mistake recurs, the thought becomes "I didn't put a stopper here." Design problems are things you can change yourself.
The AI moves from being something to blame to being something to design around. When the placement of your expectation moves, the texture of the work shifts.
There are counterparties for whom "be careful next time" works, and counterparties for whom it doesn't. Keep looking for the way through to the second kind, and the wear builds up. Move to the side where you design the path, and the relationship with the AI settles.
References
- JUAS "Corporate IT Trends Survey 2025": https://juas.or.jp/cms/media/2025/02/it25_2.pdf - MIT "The GenAI Divide: State of AI in Business 2025": coverage at https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/ - Persol Research "Generative AI and Ways of Working" (Feb 2026): https://rc.persol-group.co.jp/thinktank/data/generative-ai/ - estie "Human-in-the-loop while running": https://www.estie.jp/blog/entry/2025/05/29/094554 - AI Agent memory design patterns: https://zenn.dev/deskrex/articles/9ee6c17f4a420b
---
*Originally published in Japanese at [note.com/nomuraya](https://note.com/notes/n6c8ea041b0b2). I write under "nomuraya / shimajima / 中翔" — the same person across media. The English version is adapted rather than literally translated.*