How We Use AI to Smoke Test TrueTest

AI coding agents are rapidly shortening implementation cycles. Developers can move from an idea to working code faster than before, but release lifecycles do not automatically become faster as a result. Testing can become the new bottleneck: the volume and pace of change increase while quality teams still need enough evidence to decide whether the whole product is safe to release.

AI can already execute and investigate individual test cases. That is valuable, but release confidence cannot be built one isolated case at a time. As AI-generated changes become more frequent, repeatable automated end-to-end regression testing is more necessary than ever. Teams need to validate complete user and system workflows continuously, catch failures that emerge between components, and produce evidence that can be trusted at release time.

Testing an AI-powered product makes that challenge even more demanding because the output is not just a page, an API response, or a database record. It is a chain of transformations.

For TrueTest, now part of Katalon True Production Insights, that chain begins with activity in an application. TrueTest captures the activity as user sessions, organizes it into journey maps and flows, and generates automated tests from the behavior it observes. A useful smoke test therefore has to prove more than “the application loaded.” It has to prove that meaningful behavior survived the entire journey from interaction to executable test.

We use AI agents to coordinate that test, investigate failures, and assemble the evidence. Deterministic tools still do the measuring: Playwright drives the browser, APIs expose the data moving between layers, and Katalon Runtime Engine executes the generated tests. Human-reviewed test cases define what success means.

That combination gives us speed without asking anyone to trust a black box.

We test the pipeline, not just the interface

A traditional UI smoke test might sign in, open a few pages, and verify that important controls are present. We do that too, but it covers only the surface of TrueTest.

Our smoke test follows the product's core value stream:

1. Generate a small set of realistic, synthetic user journeys in a sample ecommerce application.
2. Confirm that the TrueTest tracking agent received the expected interactions.
3. Find the resulting user sessions using a unique smoke-run identifier.
4. Review captured steps, routes, timestamps, and session replay.
5. Generate a new journey map from only that traffic.
6. Inspect representative flows and confirm that their pages and actions match the source sessions.
7. Generate automated test cases from the reviewed flows.
8. Inspect the generated repository artifacts and execute every generated test.

The workflow is built around the way the TrueTest engine turns application behavior into executable tests. Under each engine stage, we add a corresponding test and evidence check:

Codex orchestrates the complete execution rather than replacing the tools in each layer. It loads the human-reviewed contract, runs the appropriate automation, correlates identities and evidence, investigates failures, makes targeted automation corrections, and assembles the result. Katalon True Platform remains the durable system of record for the suite, authored steps, execution results, evidence, and reporting.

The test crosses several boundaries: browser events, tracking requests, session processing, journey modeling, code generation, source control, and test execution. A green navigation check at the beginning cannot compensate for a missing action or an unusable test at the end.

This is why we treat smoke testing as an end-to-end data-integrity problem.

Human-reviewed intent remains the contract

At the start of each run, the agent reads the current smoke suite from Katalon True Platform. It does not rely on a stale copy embedded in the automation.

The suite contains human-reviewed test cases and authored steps for capabilities such as:

• finding and reviewing captured user sessions;
• replaying a session;
• opening journey maps and inspecting their flows;
• generating a new map from a controlled traffic set; and
• verifying that generated test cases are complete and executable.

Playwright is the vehicle used to perform those checks, not the source of truth for what the checks mean. The agent maps browser evidence back to the authored steps and calls out anything that was skipped or only partially covered.

This distinction matters. AI can help coordinate and interpret a test, but it should not quietly redefine the acceptance criteria while it is running.

We use Playwright as an independent outer harness so TrueTest-generated Katalon tests are not validated only through the same toolchain that produced them; every generated test still runs with Katalon Runtime Engine in its intended runtime.

Controlled traffic makes the result explainable

Production-like data is valuable, but an open-ended stream of sessions is a poor smoke-test fixture. It is difficult to know which interaction produced which result, and unrelated traffic can make a weak assertion appear to pass.

Instead, our Playwright harness creates a compact batch of deliberate scenarios: browsing products, searching and filtering, viewing product details, adding and changing cart items, abandoning a cart, and completing a synthetic checkout. Each browser context is isolated, and every session is tagged with a unique smoke-run identifier.

Before the workflow continues, the harness verifies that the correct tracking configuration was loaded and that each important interaction produced a successful tracking request. If tracking is unavailable, the run stops there. Waiting for downstream processing would only turn a clear collection failure into a confusing “sessions not found” failure later.

Once processing is complete, the same run identifier becomes the filter used throughout the rest of the test. The agent can now compare a known source batch with the sessions, flows, and generated artifacts derived from it.

TrueTest supports filtering journey-map data by session attributes and URL keywords, which makes this kind of isolated validation possible. The product documentation explains how captured interactions become journey maps and how reviewed flows can be turned into test cases in a configured repository: Generate user journey.

We verify meaning at every transformation

Counts are useful, but counts alone are not enough.

Imagine that the traffic harness clicks Add to cart, and the session contains that action, but the generated flow omits it. The number of sessions may still be correct. The journey map may still reach an active state. A generated test may even exist. Yet the behavior has been materially changed.

Our smoke workflow retains evidence from each layer and compares them:

Playwright interaction
  → tracking request
  → captured user-session event
  → journey-map page and action
  → generated test object and script
  → runtime result

For session review, we inspect both browsing and checkout behavior rather than opening an arbitrary row and declaring success. For replay, we confirm that recorded content is visible and that selecting another step changes the replayed state. For journey maps, we open representative flows and inspect their page and action details. For generated tests, we inventory the objects, test cases, and scripts, compare them with the original scope, and run them.

This layered evidence tells us not only that something failed, but where information first disappeared or changed.

The agent investigates; it does not merely retry

Browser automation fails for many reasons: a selector is outdated, a response is slow, an environment is misconfigured, or the product has a real defect. Blind retries blur those categories and can convert useful evidence into a vague flaky-test label.

When a Playwright check fails, the AI agent follows a structured investigation:

1. Preserve the original trace, screenshots, video, network evidence, console output, and assertion details before any rerun.
2. Identify the exact authored test step, expected result, actual result, and relevant run or artifact identifiers.
3. Recheck the behavior in a signed-in browser and inspect the corresponding API payloads or generated files.
4. Classify the failure as an automation defect, timing issue, expected behavior, product defect, environment problem, or insufficient evidence.
5. If the automation is wrong, correct the selector, wait, parser, or assertion and run the smallest retest that proves the change.
6. If product behavior is uncertain, ask Thor for a verdict and retain that decision with the report.

The agent is especially useful here because the evidence spans multiple tools and representations. It can connect a browser action to an event payload, a flow step, a generated locator, and a runtime failure without forcing a tester to reconstruct the chain manually.

Thor is our internal AI diagnostic assistant. The smoke-testing agent brings Thor a compact evidence package—what was expected, what happened, where it happened, and what the UI, API, and artifacts show. Thor helps determine whether the behavior is intended, a product defect, or a question for a specific owner. That response is recorded as a product-context verdict, not treated as permission to rewrite the test or hide a failure.

The guardrail is equally important: we do not weaken an assertion or edit generated product output simply to make a red test green.

Generated does not mean validated

The final stage is intentionally demanding. A “test case generated” notification is not our finish line.

We verify that all expected generated artifacts are present and readable, including the object repository entries and executable scripts. We compare the generated scope with the behavior that produced it. Then we execute every generated test through Katalon Runtime Engine.

This closes a common gap in AI evaluation. A generated artifact can look plausible in a UI or code review and still fail at runtime because a locator is invalid, a dependency is missing, or an interaction was translated incorrectly.

TrueTest stores generated artifacts in the repository configured for the application under test. Its documentation describes both journey-map generation and the newer path for generating tests from selected user sessions: Generate test cases from User Sessions.

We only consider the generation path healthy when its output runs.

Evidence is part of the product

Every smoke run produces a conclusion-first HTML report. It states the overall result, maps results to the human-authored smoke cases, and links each failure to the evidence needed to understand it.

For a successful run, that evidence provides traceability. For a failed run, it becomes a head start on a bug report: expected and actual behavior, reproduction context, screenshots, traces, network or API details, generated-artifact comparisons, and focused retest results.

We also preserve the first failure even if a corrected automation run later passes. The history matters. It shows whether the product changed, the test changed, or the environment recovered.

When results need to be shared more broadly, the agent can publish step-level outcomes back to the managed test run. Katalon True Platform provides a central place for test cases, executions, and results, while keeping human oversight over AI-assisted work.

What AI changes for our QA team

The biggest benefit is not that AI clicks faster than a script. Scripts were already good at clicking.

AI reduces the coordination cost around the script:

• It reloads the current test contract and maps automation to authored intent.
• It follows identifiers across browser, API, repository, and runtime evidence.
• It separates likely automation defects from likely product defects.
• It chooses focused retests based on what changed.
• It turns raw artifacts into a report a tester, developer, or product manager can use.

That leaves people focused on the decisions that require judgment: which behaviors matter, whether the evidence is sufficient, what risk a failure represents, and whether the product is ready.

The future of testing is agentic and framework-agnostic

AI testing will not succeed by replacing every tool with one new tool. Quality teams already have valuable test suites, execution frameworks, CI systems, device clouds, observability data, and domain-specific practices. The more durable opportunity is to place an intelligent orchestration layer above them.

Our current smoke workflow uses Playwright, Katalon True Platform, APIs, Git repositories, and Katalon Runtime Engine. The reasoning pattern is not tied to that exact stack:

test intent
    + execution adapter
    + observable evidence
    + independent runtime check
    + human-governed disposition

A different team could use Selenium, Cypress, WebdriverIO, Appium, an API test framework, or a mixture of them. The agent's job is to understand the contract, choose the appropriate execution capability, correlate evidence across layers, and return results in a shared language.

That is what we mean by framework-agnostic AI testing. The agent does not erase specialized frameworks; it makes them easier to combine. Teams can keep deterministic checks where precision matters, use AI where context and investigation matter, and change an execution tool without redesigning the entire quality process.

What we learned

Several principles have made the workflow more reliable:

• Use deterministic assertions for deterministic facts. AI can reason about evidence, but exact counts, response statuses, identifiers, and runtime results should be checked directly.
• Give every synthetic batch a unique identity. Traceability begins before the first click.
• Fail at the earliest broken layer. A tracking failure should not masquerade as a journey-generation failure.
• Preserve original evidence before retrying. The first failure often contains the best diagnostic signal.
• Validate generated output by executing it. Plausibility is not the same as correctness.
• Keep humans in control of intent and disposition. Agents accelerate the work; they do not silently move the quality bar.

Closing the loop

We use AI to smoke test TrueTest because the product itself is a connected system of observation, modeling, generation, and execution. An agent can follow that system across boundaries in a way that a single browser script cannot.

But our approach is deliberately hybrid. Human-reviewed tests provide the contract. Playwright and APIs provide objective measurements. Runtime execution proves that generated assets work. AI connects the evidence, investigates ambiguity, and communicates the result.

AI testing is the future not because it removes frameworks or people, but because it can connect both into a faster learning loop. That is the standard we want for AI-assisted quality engineering: not less rigor, but more of the system tested—and a clearer explanation of what happened.