Tests Are The New Moat | Daniel Saewitz
Key Points
- 1Open-source projects mature into complex systems, creating a tension between maintaining backward compatibility and modernizing, particularly for commercial ventures balancing altruism with business incentives.
- 2As AI advances, it can readily clone well-documented projects with comprehensive test suites, transforming these previously beneficial assets into a competitive vulnerability for their original developers.
- 3This shift suggests that commercial open-source projects will increasingly adopt a strategy like SQLite's, keeping their extensive test suites proprietary to establish a "moat" against easy replication and protect their investment.
Open-source projects, which typically begin lean and grow incrementally, often face a "Theseus paradox" where their foundational infrastructure becomes outdated. Historically, this has led to forks or total rewrites, where new projects leverage existing contracts (e.g., API surfaces) but rebuild on modern foundations (e.g., S3-compatible APIs, Redpanda as a Kafka-compatible rewrite). While this evolution is largely unproblematic for purely altruistic Free and Open-Source Software (FOSS), commercial open-source projects face a unique tension between their stated altruistic goals and underlying business incentives.
The paper argues that with the increasing capabilities of Artificial Intelligence (AI), the value proposition of open-source assets is shifting, making comprehensive test suites the new "moat" for commercial projects. Previously, elements like good documentation, strong contracts, well-designed interfaces, and robust test suites were seen as beneficial, fostering trust and enabling further development. However, these very assets paradoxically make it easier for competing entities to rebuild or "clone" a project without inheriting its legacy debt, such as historical backward compatibility requirements or outdated architectural choices.
The core methodology for this "cloning" facilitated by AI involves the AI ingesting the target project's documentation, API specifications, and most crucially, its comprehensive test suite. These tests serve as precise, executable specifications of the software's behavior and contracts. An AI can then be prompted to generate a new implementation that passes these tests, thereby adhering to the defined API surface, but built upon a completely modern and potentially more performant foundation.
A prime example cited is Cloudflare's development of a Next.js alternative (their Workers framework offering Next.js compatibility). Vercel, the creator of Next.js, invested heavily in detailed documentation, well-defined software contracts, and an extensive test suite (including Vitest and Playwright tests). Cloudflare, without needing to maintain backward compatibility or Vercel's historical architectural decisions (like building Turbopack as a Webpack alternative), could prompt an AI to rebuild the Next.js API surface. The tests provided by Vercel were instrumental in validating Cloudflare's re-implementation, allowing them to claim 94% API surface coverage. The AI's ability to interpret these tests meant Cloudflare could instruct it to build upon a more modern and performant bundler like Vite, rather than inheriting Vercel's specific foundational choices. This resulted in a "sleeker, slimmer ship" from which to start, demonstrating how the very assets that signify quality (tests) can be leveraged to create a competitive product.
The paper points to SQLite as a pioneer of this "tests as a moat" strategy. While SQLite's core database engine is open-source, its rigorous test suite, comprising an estimated 92 million lines of code (592 times the amount of core source code), is kept closed-source. This decision is presented as a deliberate commercial choice to maintain sustainability and protect their investment by controlling the definitive, executable specification of their product's behavior. While this approach might be seen as hostile to traditional open-source ethos, it is framed as a pragmatic business decision in the face of increasingly sophisticated cloning capabilities.
In conclusion, the paper argues that in an AI-driven world where code generation becomes increasingly trivial, the true value shifts from the source code itself to the precise definitions of software contracts, API surface areas, and especially, comprehensive test suites. This creates a fundamental clash of incentives for commercial open-source projects: the desire to clearly define and communicate their software's APIs (which benefits users and adoption) directly conflicts with the need to protect their intellectual investment, as such clarity makes cloning easier. The implication is that more commercial open-source projects will likely move towards privatizing their test suites or similar mechanisms to build defensible moats, redefining what constitutes "valuable" open-source contributions in the commercial sphere.