
@karpathy: This works really well btw, at the end of your que...
Key Points
- 1Experimenting with LLMs to structure responses as HTML significantly improves output visualization, leveraging humans' preference for visual information over plain text.
- 2The ideal human-AI interaction progresses from audio input to sophisticated visual outputs, evolving from markdown towards interactive neural videos and simulations.
- 3Future AI output will blend procedural software with neural artifacts to create dynamic, interactive experiences, necessitating complementary advancements in input methods like gesturing.
The paper discusses the evolution and optimization of human-AI interaction, particularly focusing on improving AI output formats to better align with human cognitive processing. The core methodology presented is a user-side prompt engineering technique, where users explicitly instruct a Large Language Model (LLM) to "structure your response as HTML" or to "present its output as slideshows." This leverages the LLM's capacity, honed during training on vast text corpora (including web data), to generate complex structured textual formats beyond plain markdown. By requesting HTML, the user obtains a text string conforming to HTML syntax, which can then be viewed in a web browser to provide a richer, visually organized experience, moving beyond the limitations of raw text or markdown.
The author posits that while audio is a preferred input modality for humans to AIs, vision is the preferred output modality from AIs to humans, given that approximately one-third of the human brain is dedicated to visual processing. This physiological preference underpins the proposed progression of AI output formats:
- Raw text: Characterized as hard and effortful to read.
- Markdown: Offering basic formatting like bold, italics, headings, and tables, which is somewhat easier on the eyes and represents the current default.
- HTML: Though still procedural code, it provides significantly more flexibility in graphics, layout, and even interactivity. This is identified as an early but rapidly forming new default.
- Interactive neural videos/simulations: The extrapolated ultimate output form, leveraging technologies like diffusion neural networks to generate dynamic, interactive visual content, potentially blending procedural "Software 1.0" artifacts with neural-generated elements.
Beyond output improvements, the paper also touches upon necessary advancements in input mechanisms, suggesting that current methods (audio, text, video alone) are insufficient. It highlights the need for intuitive interactions like pointing or gesturing on a screen, mirroring physical human-to-human communication alongside a shared computer display.
In summary, the paper advocates for a deliberate shift towards more visually rich and structured AI outputs, emphasizing that this "mind meld" between humans and AIs is an ongoing process with significant room for progress, focusing on practical, achievable enhancements like HTML output over more distant technologies such as Neuralink-esque Brain-Computer Interfaces (BCIs). The concrete advice offered is to experiment with asking LLMs to output responses structured as HTML.