Andrej Karpathy's "LLM Wiki" proposes a fundamental shift from traditional Retrieval-Augmented Generation (RAG) paradigms by enabling Large Language Models (LLMs) to actively accumulate and manage knowledge, addressing RAG's core limitation of lacking knowledge persistence and compounding. While RAG treats the LLM as a "search engine + summarizer" that operates on a per-query basis, LLM Wiki transforms the LLM into an "editor + librarian," continuously structuring and maintaining knowledge from the moment new data is ingested.

The core problem identified with RAG is its "single interest" approach to knowledge. When presented with a query requiring synthesis from multiple documents, an LLM using RAG performs the aggregation for that specific query, but the synthesized knowledge is not retained or integrated for future use. Subsequent similar queries necessitate the entire process to be repeated, preventing knowledge from compounding.

LLM Wiki solves this by adopting a "compound interest" model for knowledge, where previously synthesized information is integrated and built upon. This is facilitated by a three-layer architecture:

1. Raw Sources (Immutable): This layer comprises the original, curated documents (e.g., academic papers, articles, data files, images). The LLM is restricted to read-only access, ensuring these sources remain the untainted "source of truth."
2. The Wiki (LLM-maintained): This is a dynamic collection of Markdown files generated and continuously updated by the LLM. It encompasses various types of pages, including summary pages, entity pages, concept pages, comparison analyses, and synthesis documents. When new information is introduced, the LLM updates relevant existing pages, refreshes cross-references, and flags inconsistencies. Humans primarily consume information from this layer.
3. The Schema (Rules): Consisting of configuration files (e.g., CLAUDE.md, AGENTS.md), this layer defines the structural rules, conventions, and operational workflows for the Wiki. It dictates how summary pages should be formatted, the sequence for integrating new material, and other operational guidelines, allowing humans to guide and refine the LLM's behavior and quality of output. This clear separation ensures source integrity, wiki flexibility, and controlled LLM behavior.

The LLM Wiki operates through three primary, continuously running processes:

4. Ingest: When new source material is added to the Raw Sources directory, the LLM processes it. This involves extracting key information, writing new summary pages, updating index files, and modifying related entity or concept pages. A single new document might trigger updates across 10-15 existing wiki pages. Karpathy suggests a sequential, single-document ingestion for human oversight, though batch processing is also feasible.
5. Query: Users interact with the Wiki by posing questions. The LLM first consults the index.md to identify relevant wiki pages, then reads those pages to formulate a comprehensive answer. Crucially, high-quality responses—especially those involving novel insights, comparative analyses, or identified connections—are subsequently saved back into the Wiki as new pages, transforming query interaction itself into a knowledge accumulation process.
6. Lint: Periodically, the LLM performs a health check on the Wiki. This involves identifying internal inconsistencies, outdated claims invalidated by newer data, "orphan" pages lacking cross-references, important concepts mentioned but without dedicated pages, and missing cross-references. The LLM can also suggest new research questions or sources to investigate.

For navigation and internal coherence, the Wiki utilizes two special Markdown files:

• index.md: A content-centric catalog providing links, one-line summaries, and metadata for all wiki pages, organized by category. The LLM uses this as a primary lookup for relevant information, reducing the need for embedding-based RAG infrastructure for wikis up to several hundred pages.
• log.md: A chronological record of all activities, including ingestions, queries, and linting sessions, facilitating tracking and auditing of the Wiki's evolution.

The efficacy of LLM Wiki stems from its ability to nearly eliminate the maintenance burden that typically leads to human wikis' failure. Tasks such as updating cross-references, re-summarizing, and resolving inconsistencies—which are tedious and costly for humans—are handled by the LLM at virtually zero marginal cost. This ensures that as the Wiki grows, its value increases exponentially due to enhanced connectivity and coherence, rather than diminishing under the weight of maintenance.

In practical implementation, Karpathy uses Obsidian (a Markdown-based note-taking application) as the Integrated Development Environment (IDE) for the human user, with the LLM agent serving as the "programmer" that edits the Wiki ("codebase"). The entire Wiki can be managed as a Git repository, inherently providing version control, branching, and collaboration capabilities. Auxiliary tools like Obsidian Web Clipper, image local storage, Obsidian's graph view for visualizing connections, Marp for presentations, Dataview for querying metadata, and qmd for local search enhance the workflow.

Historically, LLM Wiki can be seen as the modern realization of Vannevar Bush's Memex (1945), a hypothetical device for personal knowledge storage and associative linking. The critical challenge Memex faced—the labor-intensive process of maintenance and connection building—is precisely what LLMs can now address.

The profound implication of LLM Wiki is the redefinition of the LLM's role from a reactive query-response tool to a proactive, persistent agent for structuring and maintaining knowledge. This shift requires surprisingly little complex infrastructure, relying primarily on Markdown files, Git, and the LLM's read/write capabilities, suggesting a new avenue for knowledge management beyond complex vector databases and RAG pipelines. While scalability for thousands of pages and the quality of "schema engineering" remain areas for exploration, this pattern offers a promising path towards truly compounding knowledge systems.