Tests that prove something

Greg Tatum – July 26, 2026

In my work on high quality code generation, it’s more important than ever to create cheat-proof tests. Testing is the way that you can assert program behavior in an automated fashion without actually spinning up real infrastructure, loading up a UI manually and clicking things, or booting up your game and using a controller. It’s a fully hands-free approach to run your code. In order for agentic coding practices to work effectively, these tests need to actually mean something if they pass, and reflect the target environment as closely as is reasonable. When tests are designed this way, the speed at which you can review and have confidence in an agent’s output increases dramatically. If the test passes, you will know it can pass when you actually run the application.

This way of thinking shouldn’t be confused with unit test vs regression test vs acceptance test. A test in a realistic environment can be all of these things at once. A unit test can test a smaller scope of your application while still operating in a realistic environment.

PageExtractor example

An example for this type of test I will give is the PageExtractor component in Firefox’s Smart Window project. This component is responsible for taking a live web page and translating it into plain text that can be given to a language model. The live web page is full of rich structure, semantic meaning, and accessible controls. There is a lot of subtlety and behavior to figure out when encoding this information into a text format.

In order to make this test run in as realistic a manner as possible, I want to be able to load a real browser, use the same wiring that the PageExtractor component uses, and then create a very human-readable flow. The focus should be on writing declarative code instead of imperative code.

const { html } = await serveHTMLInTab({ browser: gBrowser });
const { getPageExtractor } = await html`
  <article>
    <h1>Hello World</h1>
    <p>This is a paragraph</p>
  </article>
`;
const actor = getPageExtractor();

is(
  (await actor.getText()).text,
  ["Hello World", "This is a paragraph"].join("\n"),
  "Text can be extracted from the page."
);

From: toolkit/components/pageextractor/tests/browser/browser_dom_extractor.js

The setup and utility functions are doing some heavy lifting. serveHTMLInTab invokes a real web server that can serve a file, and then feeds that into a tab on a real browser. The html template literal may be surprising syntax, but it completes the serving of the HTML in a declarative way which is easy to read. When I’m reading this test, or any other generated test case, I can easily determine what the input is without loading up imperative code in my brain.

From there getPageExtractor loads the real PageExtractor component in the environment. I would consider this test a unit test. It’s testing the PageExtractor at the unit level, as it’s consumed downstream by UI code in a broader product. I really don’t want the product flows relying on specifics of DOM processing behavior. The unit of work boundary here is this internal component, but it’s running that test in the most realistic environment possible.

Finally the test reads the final result and checks it in a way that keeps the focus on declarative structure. I can quickly scan the file and find behavior quirks for the component.

Security testing

Another example of cheat-proof testing is with the security properties for the Smart Window project. Security properties are ways that we track the lethal trifecta when working with language models and browser data. We need flags to track private data or untrusted content. Here private data would be something as simple as what tabs you have open. Untrusted content would be web page data. If our tests aren’t cheat-proof here, then we can’t be sure whether we’ve sufficiently locked down the product. While bugs can always slip in, or assumptions can be wrong, if we’re executing the code in a sufficiently realistic environment then we can have confidence in our initial position. When a bug comes in, we have tools to reproduce the bug in the environment, and fix it with confidence.

The problem with this type of regression testing is that LLM resources are expensive and unpredictable. A regression needs to be reproducible, and we can’t rely on an external service. We could mock out all of the environment and write some tests to check the box ✅ yes I wrote some tests, but the real work here is to only do a mock where it’s really needed.

const { html } = serveHTMLInTab();

const { url, cleanup: removeNewsArticle } = await html`
  <h1>News Article</h1>
  <p>This is a news article about technology.</p>
`;

Here we can start with the previous approach of serving up some HTML in a tab for a real webpage. We’re starting with a realistic environment.

await typeInSmartbar(
  sidebarBrowser,
  "What is the title of this page? Don't look at the page content."
);
await submitSmartbar(sidebarBrowser);

Then the behavior of the application can be exercised with the real UI loaded and actions performed against it. Ideally the utility functions follow the guiding principles from Testing Library where buttons are clicked and accessibility-focused code is what drives the tests. This gets you behavior correctness and cheat-proof accessibility.

From here we need the response from the language model. While you could try provisioning some CI services for language models, the responses have the reliability issues and complexity I mentioned earlier. This is the point where we need to find the right way to mock it. For my case, I built a specialized MockLLMEngine that sits deep within the engine management infrastructure, but only mocks at the point where a real LLM would be called. It matches the text streaming API, but strips out the language model. I wired this into the Smart Window internals and provided an ergonomic testing API.

Within the test setup function I create my mock:

function setupSecurityTest() {
  ...
  const mockEngineManager = new MockEngineManager();
  ...
  return { mockEngineManager, ... }
}

Then anywhere in my test where I need to control the language model response, I can do so.

const { mockEngineManager, ... } = setupSecurityTest();
...
await mockEngineManager.respondTo({
  purpose: "chat",
  response: "This page has no title.",
});

This gets rendered through the testing infrastructure as a real browser with a real response. This is an adequately realistic environment to show that a component works.

This work to build the mock took time, and it even has its own unit tests. I had to invest time to make this happen outside of code generation. I literally couldn’t prompt this piece, and it required good ol’ hands on the keyboard to make happen. This investment meant that from here, I could write cheat-proof tests agentically, and I can confidently review my peers’ contributions.

Ultimately with this test, I cared about the SecurityProperties component, a component I was owning reviews for and collaborating on with others. This was the unit of work that I cared about, and the test covered its integration into the application. I can test that the properties get set with the internal representation we care about, and assert the actual environment of the browser.

Assert.equal(
  conversation.securityProperties.privateData,
  true,
  "The conversation gets marked as private as the tab info is added to it."
);
Assert.equal(
  conversation.securityProperties.untrustedInput,
  false,
  "Nothing untrusted is added to the conversation."
);

For the full test see: browser/components/aiwindow/ui/test/browser/browser_security_chat.js

Conclusion

Building these tests took engineering work. I had to build the browser fixtures, declarative test APIs, and realistic mocks before I could effectively write tests using it all. That investment wasn’t specific to AI-assisted development. It made every test easier to write and much more meaningful. This improves the situation for both traditional and AI-assisted workflows.

Language models are really good at writing plausible-looking code. When tests aren’t cheat-proof, even a domain expert can be convinced something works when it actually doesn’t. That’s why I care so much about running tests in realistic environments. If a test passes, it needs to show something meaningful about how the code will behave when it leaves the test harness.