Meta’s engineers asked their AI coding agent to change the group home screen in a new app, and the agent changed the home feed instead [1]. Other times it guessed at screen coordinates [1]. Meta calls this its main failure mode: the agent losing track of where it was in the app [1]. Meta describes the general cost of unchecked output like that: code that “looks finished and isn’t”, which an engineer has to read, debug, and often discard [1].
The app was Forum, Meta’s standalone app for Facebook Groups, which a small team built from scratch with an agent in the development loop [1], [2]. The first fix most of us reach for in that spot is the prompt. We describe the screens more carefully, add a line that says “check you are on the right screen”, and try again. Meta went the other way, and its write-up says so in one sentence: “The biggest gains came from changing our code and tooling, not our prompts” [1].
We’ll walk through 5 changes that make a repo checkable by an agent. They come from Meta’s Forum build and from Elie Steinbock, an open-source maintainer who reached the same place on his own project. They are: let the agent look at real app state, write a rule once a mistake repeats, test against stateful fakes of the services the app calls, attach screenshots to every change, and keep prototype code compiled out of production. For each one, we say what it fixed and what it still left broken.
Figure 1 - Change the Codebase, Not the Prompt: A better prompt gives the agent a better description to guess from. Changing the repo gives it something to check: the real screen, the written rule, a fake service, a screenshot, and a production build prototype code cannot reach.
Why a better prompt doesn’t stop wrong-screen edits
Start with what the agent had to work with. Forum is a phone app. To change a screen, the agent has to know which screen it is on, which elements are on that screen, and how to get from where it is to the one it was asked to change. Without tools for that, it only has what we told it in the prompt, plus what it can work out from the code.
That is a description of the app, and a description can be wrong. It can be out of date the moment someone adds a screen. It can use a name the code doesn’t use. An agent reading “the group home screen” has to match those words to a file, and “home feed” is a close enough match to pick by mistake.
Rewriting the prompt improves the description. It still gives the agent no way to find out whether its guess was right.
Meta names the cost directly. Without verification, AI-generated code gives you “plausible output that no one has confirmed actually works” [1]. Weak signals “don’t remove the constraint”, Meta writes. They “move it from writing code to reviewing code that looks finished and isn’t” [1].
Meta’s goal for Forum made this worse. It wanted to move from human-in-the-loop to human-on-the-loop. In human-in-the-loop, an engineer drives every step and the AI helps inside it. In what Meta calls human-on-the-loop, the agent runs the plan, implement, build, test and correct cycle by itself, while the engineer sets the goal, watches, and approves the result [1]. An agent that edits the wrong screen can’t run unsupervised. Someone has to catch every wrong guess.
Figure 2 - The Wrong Screen: Picture the request naming the group home screen. An agent matching words in a description to files edits the home feed instead. The edit looks plausible, so the mistake only shows up when a person reads it.
Let the agent look at the real screen
Meta’s first change was to stop describing the app and give the agent tools to look at it. First, 2 terms. A simulator is a program on a developer’s computer that runs a phone app as if it were on a real phone. The view hierarchy is the tree of views (everything drawn on a screen), from the screen itself down to each list, each row in the list, and each button in the row.
The agent got 3 tools, used before it acts: read the live view hierarchy from the simulator, list the elements actually on screen, and look up the path to a target view [1]. Reading the hierarchy tells the agent what’s on screen, by name, right now.
With those tools, the agent no longer has to match the words “change the group home screen” to a file. The agent can ask the simulator which screen is open and see that it is the home feed. Then it can find the path to the group home screen before it edits anything. Meta reports the result in 4 words: “Misdirected edits dropped sharply” [1]. Meta gives no number for that drop, before or after, so we can’t say how sharp.
Meta then applied the same idea everywhere the agent touched its systems. In its words, “give the agent a way to check state directly rather than a description to reason from” [1]. These tools inspect state. How many tools an agent should see at once is a separate question.
Figure 3 - A Way to Look: On one side, the agent reads a description and guesses. On the other, it calls 3 tools before it acts: read the live view hierarchy, list the elements on screen, and find the path to the target view.
Those tools cut the wrong-screen edits, but they gave the agent no memory between sessions. Each agent session starts fresh. When a session figures out that a particular screen is easy to confuse with another one, that lesson is gone when the session ends, and the next session can make the same mistake.
KEY INSIGHT: When an agent keeps acting on the wrong thing, check whether it can see the thing it is acting on. Give it a tool that reads the real state before you spend another hour on the prompt.
Write a rule once a mistake repeats
Meta’s answer to repeated mistakes is a threshold. “When the agent repeats a mistake more than twice, we encode the correction once, for everyone” [1]. The correction becomes a rule, which Meta defines as a permanent project instruction [1]. Meta doesn’t name its file. In most coding agents, a rules file is a plain text file in the repo (such as CLAUDE.md or AGENTS.md) that the agent reads at the start of every session.
The threshold matters most. A one-off mistake doesn’t earn a rule, so the rules file stays short. A mistake that keeps coming back gets written down once, so every session and every engineer’s agent starts with the correction, not only the one that hit it.
Meta also changed the code itself, for the same reason. “Agents learn patterns from the code they read,” the post says, so a weak pattern left in the code shows up again in everything the agent writes next [1]. Its “What Worked” table puts it as “Codified Fixes Compound”: “The codebase is part of the context, so improving it compounds across every agent that reads it” [1].
Rules are one part of a larger setup Meta describes [1]. Skills turn team conventions into steps the agent can load, and hooks are automatic triggers, such as running a linter (a tool that checks code for style problems and common errors) after every agent turn [1]. If you already use a coding agent with a rules file and hooks, you have a version of that setup. What’s new is the habit of turning a repeat mistake into a rule on purpose.
Figure 4 - A Repeat Becomes a Rule: The same mistake shows up in session after session. Once it repeats, the correction goes into the project rules once, and every later session and every engineer’s agent starts with it.
Now the agent can see the screen and remembers its past mistakes. It still can’t check work it can’t run. A lot of real code calls services the agent has no safe way to touch.
Test against stateful fakes of the services you call
Elie Steinbock builds Rakazo, an open-source alternative to Grok Bot, a chat-based AI assistant in which each bot gets its own computer. He hit this wall while having an agent build the first version [3], [4]. His app talks to outside services such as Gmail and Slack. As he puts it, if the agent needs to sign into Gmail to test that it can send an email, “Google’s going to block the AI. You’re going to hit rate limits” [3]. The agent can run a database such as Postgres, or a cache such as Redis, on its own machine. It can’t run Slack.
The usual answer is a mock, a stand-in that returns a canned response when the code calls it. Steinbock uses an emulator instead, and he’s clear about the difference [3]. An emulator is a basic clone of the service with its own small database, usually held in memory, so it remembers what happened earlier. Send a message to an emulated Slack, send another, and have a second bot read the earlier ones, and it all works the way the real Slack would [3]. A mock would answer each call and forget it.
That memory is what “stateful” means here. It matters since real integration bugs are usually about sequences: a message that should appear in a thread, a payment that should show up after a charge, a pull request that should exist after it was opened.
The tool he points to is emulate, a Vercel Labs experiment on GitHub [5]. Vercel Labs is where Vercel publishes experimental projects. Running npx emulate starts local copies of services such as GitHub, Slack, Stripe and Google, with Slack answering on localhost:4003 (port 4003 on the same machine), and “No config file needed” [5]. Its README calls it “Fully stateful, production-fidelity API emulation. Not mocks” [5]. Since it needs no config file and no network, it runs in CI (continuous integration, the service that runs a project’s tests automatically on every change). The project site puts the offline side as “No API keys. No network. Not mocks” [6]. Steinbock says he uses it in his other product, Inbox Zero, and that Rakazo itself doesn’t use it [3]. Inbox Zero’s package file pulls it in through a fork, @inbox-zero/emulate, rather than the Vercel Labs package [7].
For Rakazo he went further. Some services he needed had no emulator anywhere [3]. One is Composio, a service that connects AI agents to outside apps. Others are sandbox providers, services that rent out isolated cloud machines for running agent code. So he had the agent look up each provider’s API online and write an emulator that mimics it [3]. Those emulators are in the repo: the adapters folder holds emulators for Composio, Stripe and the sandbox providers (Daytona, E2B and Box), among others [8]. The project’s AGENTS.md makes it a rule for every new integration: “New providers must reuse shared contracts and deterministic offline conformance tests” [9]. Emulate supports this too, with npx emulate init --custom to scaffold an emulator for a service it doesn’t cover [5].
Figure 5 - Mock Versus Emulator: A mock returns a canned reply and forgets the call. An emulator keeps a small in-memory database, so a message sent in one call can be read back in the next, the way the real service behaves.
Figure 6 - Fakes on Localhost: In CI, the app under test calls emulated services on localhost instead of the real ones. The agent can check an integration end to end without a real account or an API key.
Now the agent can run the whole app and check its integrations. What reaches a reviewer is still a pull request (PR), a branch and a green test run. To see what actually changed, the reviewer has to pull the branch and run it.
KEY INSIGHT: Pick the 3 integrations your agent touches most and give each one a stateful fake that runs locally. Without one, the agent has no way to test those integrations itself.
Attach screenshots to every change
Steinbock started with a rule for the agent. While the agent was building Rakazo’s first version, he told it he needed screenshots of every page: “I don’t want you to pretend or guess that this is working” [3]. He ran the build as a goal loop, a mode in which the agent keeps working through a plan until every item is checked off, and made the screenshot the finish line. “If you don’t have a screenshot, then you haven’t hit your goal, and you need to keep going” [3].
That rule changes what “checked off” means. A page counts as done only when the agent has a picture of it.
He calls the result a product built in a day. By his own account, the day doesn’t count about an hour of planning beforehand, a goal run that went overnight and had to be repeated, or a security scan afterwards [3].
Then he kept the rule for everyone else. “Very early on in the project, I decided to add screenshots on every pull request” [3]. In the repo, this is an end-to-end (E2E) test job, one that drives the real app through a browser the way a user would. The job runs on GitHub Actions (GitHub’s built-in CI service), is named “Web E2E” and runs Playwright, a browser automation library, against the app [10]. The job’s sandbox provider defaults to fake, so it runs against the project’s own emulators and needs no real vendor account [10].
A second workflow publishes the screenshots as a gallery and posts or updates one comment on the pull request that links to it [11]. The AGENTS.md file turns that into a contributor rule: “For UI changes, link the CI E2E screenshot that shows the change on the PR; add the web test that opens that screen if it is missing” [9]. Steinbock says he doesn’t look at every screenshot on every run, only at the ones that changed [3].
Figure 7 - No Screenshot, No Goal: The agent’s loop does not end when it says a page works. It ends when there is a screenshot of every page. A page with no screenshot sends the agent back to work.
Figure 8 - Screenshots on Every PR: A pull request triggers the Web E2E job against fake services. A second workflow publishes the screenshot gallery and posts one comment on the pull request that links to it, so the reviewer sees the change without running the branch.
He also describes an agent that wakes every few hours, reviews new pull requests and merges simple bug fixes on its own, while bigger features wait for him with screenshots attached [3]. That agent isn’t among the repo’s workflow files, so we couldn’t check it. It stays his claim.
Now the agent can see, remember, test and show its work. That freedom brings the last problem. An agent that can try things will produce experiments, and those experiments live in the same codebase as the app real people use.
Keep prototype code compiled out of production
Meta hit this once Forum had real users. It wanted prototypes and agent-generated experiments to keep moving fast while the production app stayed limited to finished, reviewed work [1].
The common tool for this is a feature flag, a switch in the code that hides unfinished work from users. Meta wanted more than that. With feature gating, as Meta puts it, “prototype code still exists inside the production build” [1]. The code is hidden, but it ships, and a mistake in the switch exposes it.
So Meta built a separate build flavor. A build flavor is a second version of the app produced from the same source code with different settings. Meta’s sandbox flavor “shares the exact same codebase, frameworks, and design system as production, but compiles prototype code out of the production build entirely via a plugin/socket architecture” [1]. Meta doesn’t describe the plugin/socket mechanism further. The outcome is what counts: the production build never contains the prototype code at all.
Meta lists 3 results [1]. Designers and other partners could prototype with AI tools without an engineer in the loop for each change. Anything built in the sandbox could move to production with no rebuild. Prototype code could not crash the production app, because “it physically didn’t exist in that build” [1].
Figure 9 - Compiled Out of Production: A feature flag hides prototype code that still ships in the production build. A separate build flavor shares the same codebase but leaves prototype code out of the production build entirely.
The first 4 changes help the agent check its own work. This one limits how far an unchecked mistake can travel. It doesn’t check anything itself, though. A broken prototype still lands in the sandbox build, and someone still has to review what moves to production.
Why the 5 changes work together
Go back to the wrong-screen edit. It happened because the agent guessed. Each of the 5 changes replaces one kind of guess with a check:
- Which screen is the agent on? The view tools read the real screen instead of a description of it.
- Has the agent made this mistake before? The rules file answers that for every session.
- Does the integration work? The stateful fake answers the way the real service would, offline.
- What did the change look like? The screenshot shows it to the agent and to the reviewer.
- Can this experiment reach users? The build flavor makes the answer no.
None of them is enough alone. Meta’s “What Worked” table says this outright under “No Single Tool Was Enough”: “Screen inspection, simulator control, and end-to-end tests each had blind spots the others covered, so the reliable setup was all of them layered” [1]. Meta also sets the finish line at verification, not compilation: “a task is complete only once automated tests and visual checks have passed” [1].
A prompt lives in one conversation. A view tool, a rule, an emulator, a screenshot job and a build flavor live in the codebase, so every session and every person’s agent gets them without asking.
Figure 10 - 5 Guesses, 5 Checks: Each change answers one question the agent used to guess at: where it is, whether it has made this mistake before, whether the integration works, what the change looks like, and whether an experiment can reach users.
What did each team get? Meta says misdirected edits “dropped sharply”, and gives no number [1]. Meta also ran stress runs: repeatable scenarios that mimic aggressive scrolling, random group browsing and rapid navigation. They surfaced a memory issue that the agent then isolated and fixed during development, before the app reached users [1]. Meta says the fix ended up “roughly halving memory use and page faults” [1]. A page fault happens when the app reaches for memory the operating system has to load or set up first, and each one costs time. Meta also says Forum reached core feature parity with Facebook Groups in 8 weeks [1]. These are Meta’s own figures. The post gives no baseline for the memory number, no comparison team for the 8 weeks, and no definition of core feature parity, so we treat them as Meta’s account rather than a measured gain.
Steinbock’s result is easier to check, because it’s in the repo: a screenshot gallery linked from each pull request, a contributor rule that asks for it, and emulators that let CI run with no vendor accounts [9], [10], [11]. He also gives counts of stars, contributors and merged pull requests in the video. The star count he gives doesn’t match the one on the repo page when we checked [3], [4], and the others can’t be checked, so we leave them out.
KEY INSIGHT: Put each check in the repo, where every session finds it, instead of in a prompt that one conversation reads. Then layer them, because each check covers blind spots the others leave.
Where it stops working
Forum was a new app. The first row of Meta’s “What Worked” table is “Fresh Codebases Accelerate AI Development”: starting on a new codebase let the agents iterate faster, since the code they needed to reason about “was compact and self-contained” [1]. Where reuse blocked the agents, Meta rebuilt from scratch [1]. We wouldn’t expect an older repo with tangled code to get the same speed just by adding these 5 changes.
Passing checks can still hide shortcuts. Meta’s “What Did Not” table says agents “produced pull requests that technically passed while cutting corners, so quality signals must be rigorous enough to catch the shortcuts” [1]. Checks have to be strict as well as frequent.
An emulator can pass a test the real service would fail. Neither Steinbock nor the emulate README covers that gap, and we haven’t seen it measured anywhere. A fake that drifts from the real API gives the agent false confidence, so we’d still run a smaller set of tests against the real service before a release.
A screenshot shows that a page rendered. It says nothing about whether the page behaves. Rakazo’s own homepage step is labelled “Homepage screenshots only: no copy or layout assertions” [10]. The screenshot is for a person to look at, so a person still has to look.
The tooling has rough edges. Meta found remote iOS simulator connections “unreliable enough to slow the loop” [1], and its table heading for that lesson is “Local Development Beats Remote For Simulator Work” [1].
People still sign off. Meta writes that “Every shipped outcome still carries an engineer’s sign-off” [1], and calls human-on-the-loop “the target we are driving toward, not a finished state” [1]. Steinbock’s account is one maintainer’s teaching video about one open-source project, not a measured study [3].
Figure 11 - The Checkability Audit: The 5 yes-or-no questions to ask of a repo before giving an agent more autonomy. Each “no” is a place where the agent is still guessing.
Conclusion
Meta’s agent made far fewer wrong-screen edits once it got a way to look at the screen. The other 4 changes extend that idea. The agent needs to see the state it is changing, keep the corrections it has already earned, test against fakes that behave like the real services, show its work as a picture, and stay unable to ship an experiment by accident. All 5 are changes to the repo, and none needs a new model. Each can start small: a rules-file entry, one emulator, one screenshot job. The build flavor is the heaviest of the 5, and it is easier on a new codebase than an old one.
The next step is an audit of your own repo, before you give the agent more autonomy. Ask 5 questions. Can the agent read the real state of what it changes? Are repeated corrections written into a project rules file? Do the 3 integrations it calls most each have a stateful fake? Does every pull request carry a screenshot? Is prototype code kept out of the production build? Fix the first “no” before you touch the prompt again.
References
[1] S. Guo, B. Luo, S. Sarkar, V. Qu, M. Fan, P. Jain, M. Plumpe, C. Abhyankar, and R. Tandon, “How Meta Built Forum With AI-Native Engineering,” Engineering at Meta, Oct. 15, 2026. https://engineering.fb.com/2026/10/15/ml-applications/how-meta-built-forum-ai-native-engineering/
[2] Meta, “Find Your Community With Forum, a Dedicated App for Facebook Groups,” Meta Newsroom, Sep. 29, 2026. https://about.fb.com/news/2026/09/find-community-forum-dedicated-app-facebook-groups/
[3] E. Steinbock, “How I Built an Open Source Grok Bot in a Day,” YouTube, Oct. 15, 2026. https://www.youtube.com/watch?v=xk-aLMQRYfk
[4] E. Steinbock, “rakazo: Open-source Grok Bot alternative. Choose your own model and sandbox.,” GitHub, accessed Oct. 16, 2026. https://github.com/elie222/rakazo
[5] Vercel Labs, “emulate: Local API emulation for CI and no-network sandboxes,” GitHub, accessed Oct. 16, 2026. https://github.com/vercel-labs/emulate
[6] Vercel Labs, “emulate: Local API Emulation for CI & Sandboxes,” emulate.dev, accessed Oct. 16, 2026. https://emulate.dev
[7] E. Steinbock, “apps/web/package.json,” elie222/inbox-zero, GitHub, accessed Oct. 16, 2026. https://github.com/elie222/inbox-zero/blob/main/apps/web/package.json
[8] E. Steinbock, “packages/adapters/src,” elie222/rakazo, GitHub, accessed Oct. 16, 2026. https://github.com/elie222/rakazo/tree/main/packages/adapters/src
[9] E. Steinbock, “AGENTS.md,” elie222/rakazo, GitHub, accessed Oct. 16, 2026. https://github.com/elie222/rakazo/blob/main/AGENTS.md
[10] E. Steinbock, “playwright.yml,” elie222/rakazo, GitHub, accessed Oct. 16, 2026. https://github.com/elie222/rakazo/blob/main/.github/workflows/playwright.yml
[11] E. Steinbock, “publish-playwright-report.yml,” elie222/rakazo, GitHub, accessed Oct. 16, 2026. https://github.com/elie222/rakazo/blob/main/.github/workflows/publish-playwright-report.yml
Building production AI, or modernizing a legacy system?
That is the kind of work we do at Dotzlaw Consulting. Book a free 20-minute intro call and tell us what you are trying to build, or what is slowing you down.