The paper, presented as a YouTube video by Hoe-won Jeong from Naver Financial, addresses the critical challenge of ensuring code quality and reliability in the era of AI-driven software development. While AI agents significantly accelerate coding (evidenced by a 1.5x increase in merged Pull Requests), validation often becomes a bottleneck. The core problem is that AI-generated code, especially at the integration and end-to-end levels, can introduce subtle yet critical defects that unit tests fail to catch. AI models, like humans, tend to be lenient on their own code and may report completion prematurely without adequate validation.

The proposed solution is Harness Engineering, which involves providing AI agents with the necessary environment, tools, and context to self-validate their work. This goes beyond mere model weights and includes directive files (e.g., agent.md), tools (e.g., MCP skills), domain knowledge encoded in LLM wikis, and crucial validation tools like automated tests. The presentation specifically champions the use of End-to-End (E2E) testing with Playwright as a cornerstone of this harness.

Why E2E Testing is Crucial for AI Agents:
The speaker references Kent C. Dodds' testing pyramid, highlighting that while static checks and unit tests (e.g., with Vitest or React Testing Library) are cost-effective, they provide limited confidence in the system's overall functionality. Unit tests might pass, but "component boundary defects" (e.g., data format mismatches, state propagation issues, API call failures in real environments) often occur when components interact. E2E tests, which simulate actual user flows in a real browser, provide the highest level of assurance, making them indispensable when AI agents are composing complex systems.

Advantages of Playwright for AI Agent Validation:

1. Real User Experience Validation: Playwright tests operate in actual browser environments, verifying user flows like button clicks and page navigations, offering validation beyond simple function calls.
2. Test Code as Living Specification: Playwright test code naturally reads like user specifications (e.g., "User navigates to loan page, clicks button X, expects result Y"). This means the test code serves dual purposes: an executable sensor for validation and a human-readable, living specification for the AI agent, eliminating the common issue of out-of-sync documentation.
3. Rich, Structured Reports for AI Agents: Playwright generates comprehensive HTML reports with screenshots, execution videos, DOM snapshots, network requests, console logs, and error traces. This structured data is directly consumable by AI agents, allowing them to perform sophisticated root cause analysis and propose corrective actions, mimicking a human developer's debugging process with browser developer tools.

Core Methodology: Building and Operating E2E Tests with AI Agents (Self-Correction Loop):

The methodology focuses on integrating Playwright E2E tests into an AI-driven development workflow, enabling a self-improving loop for AI agents.

4. Identifying Critical User Flows: Given large codebases, testing everything is impractical. The team focuses on "critical user flows"—scenarios whose failure directly impacts business (e.g., revenue, data integrity, user trust). This prioritization ensures that the most impactful areas are covered.

5. AI-Assisted Test Creation:
   • playwright codegen (Human-Assisted Start): Initially, human interaction recording via codegen generates rough Playwright test scripts.
   • AI Agent Refinement: An AI agent then refines these scripts to conform to team conventions, adding necessary setup (e.g., authentication using storageState fixtures), organizing steps with descriptive names, and ensuring linting/type correctness.
   • Full AI-Driven Test Generation (Playwright Agent System): Playwright provides a dedicated AI agent system comprising:
     • Planner: Explores the codebase and generates a markdown-based test plan/scenario outline for human review. This separates planning from implementation.
     • Generator: Takes the planner's output and translates it into executable Playwright code. It validates each step by actually executing it in a browser, ensuring the generated code works as intended.
     • Healer: Automatically diagnoses and fixes failing tests. It leverages Playwright's trace files (containing comprehensive execution data: DOM, network, console) via specific CLI skills (playwright cli, playwright trace) to identify root causes. It then modifies the test code, re-runs it, and iterates until the test passes.

6. Harness Engineering for Context and Environment:

For AI agents to effectively use these tools, specific context and environment setup are crucial, akin to onboarding a new human developer:

• agent.md: A comprehensive document describing the service, core flows, authentication, and access control.
• E2E Test Guidelines: A dedicated, single source of truth for E2E testing conventions, execution environments, authentication scenarios, and prior decisions on what to test/not to test.
• Utility/Helper Functions: Pre-built modules for common, repetitive tasks like authentication, state initialization, and data seeding to keep test code clean and focused.
• MCP Server Integration: Allows AI agents to access internal project contexts like design documents, issues, and PRs, providing rich domain knowledge beyond just code.

7. Managing External Dependencies and Flakiness:
   • Test Isolation via API Seeding: To prevent redundancy and inter-test dependencies, tests are designed to start from specific points in a user flow. This is achieved by using test-specific APIs to pre-set user states (e.g., user has completed terms agreement and phone verification) before the test begins, allowing it to focus on subsequent, divergent paths.
   • Mocking External Dependencies: For external services (e.g., third-party authentication, payment gateways), Playwright's page.route() is used to mock responses at the browser level for client-side calls. For server-side rendering (SSR) or Backend For Frontend (BFF) calls, environment variables direct requests to fixed mock responses. This isolates the tests from external instability and speeds up execution.
   • Preventing Flakiness:
     • Conditional Waits: Avoid hard-coded waitForTimeout in favor of Playwright's smart, condition-based waits (e.g., waitForSelector, waitForLoadState) that wait only as long as necessary.
     • Semantic Selectors: Prioritize robust selectors based on role or name attributes over brittle CSS class names or DOM structures, making tests resilient to UI changes.
     • Burn-in Testing: Running tests multiple times (--repeat flag) immediately after creation to catch intermittent failures (flakiness) early, as fixing them post-merge is significantly more expensive.

8. CI/CD Integration and Self-Improvement Loop:

AI agents raise Pull Requests. The Continuous Integration (CI) system runs E2E tests. If tests fail, the PR is blocked, and the CI system comments on the PR with feedback. Crucially, the Playwright trace file (the comprehensive log of test execution) is uploaded as an artifact. The AI agent can then download this trace, use playwright trace (via an e2e_debug skill) to analyze the failure, employ the Healer agent to propose fixes, and submit a new commit/PR. This creates a self-correction loop where the AI agent learns from its mistakes, analogous to a human developer debugging failed CI runs.

The Role of Humans in Harness Engineering:
Despite AI's capabilities, human intervention remains critical. Humans design the self-correction loops, provide the necessary tools and context (e.g., agent.md, domain knowledge), and adapt the harness as models evolve. The core principle is "making it easy for the agent to do the right thing," which involves continuous refinement of the environment and feedback mechanisms. The E2E test code serves as both a "guide" (specification for the AI) and a "sensor" (validation mechanism), closing the self-improvement loop for AI agents.