Anthropic Open-Sources Performance Evaluation Take-Home Assignment | GeekNews
Key Points
- 1Anthropic has released an open-source performance take-home challenge, inviting participants to optimize code to surpass Claude Opus 4.5's best record of 1487 simulated clock cycles.
- 2The challenge measures performance in clock cycles, setting a specific benchmark for human developers to beat, with successful submissions leading to a potential recruitment opportunity.
- 3This novel recruitment method has sparked significant discussion within the tech community regarding its technical complexity, value as an alternative to standard interviews, and the insights it provides into low-level code optimization.
Anthropic has open-sourced a performance evaluation take-home assignment on GitHub, designed to identify candidates capable of optimizing code beyond the capabilities of their advanced AI models, particularly Claude Opus 4.5. The primary objective for participants is to achieve a performance level below 1487 clock cycles on a simulated machine, the best recorded performance by Claude Opus 4.5 over an 11.5-hour harness run. Successful candidates are invited to submit their code and resumes to Anthropic.
The assignment's historical context reveals an evolution in difficulty. Initially, it featured a 4-hour time limit, but this was reduced to 2 hours after Claude Opus 4.5 consistently outperformed human participants. The starting codebase has also seen changes, reverting from a 18532-cycle baseline (used in the 2-hour version) to the slowest available baseline, while retaining the latest structural elements. Performance is measured in simulated machine clock cycles.
Key performance benchmarks provided:
- 2164 cycles: Claude Opus 4 (long run)
- 1790 cycles: Claude Opus 4.5 (general code session)
- 1579 cycles: Claude Opus 4.5 (2-hour test harness run)
- 1548 cycles: Claude Sonnet 4.5 (long run)
- 1487 cycles: Claude Opus 4.5 (11.5-hour harness run) - the target to beat.
- 1363 cycles: Claude Opus 4.5 (improved harness environment)
The core technical challenge involves low-level code optimization for a specific, likely custom, simulated architecture. Discussions surrounding the assignment highlight the task's complexity, balancing ALU (Arithmetic Logic Unit) and VALU (Vector Arithmetic Logic Unit) throughput, and addressing potential memory load bandwidth bottlenecks. Assumptions such as a constant starting index of 0 are noted for achieving very low total loads. The problem implicitly involves deep understanding of the underlying computational model, possibly a Random Forest prediction or similar, and requires fine-grained control over execution.
Effective optimization strategies and techniques mentioned by participants include:
- Manual PTX (Parallel Thread Execution) code writing, implying an architecture akin to a GPU or a highly parallel processor.
- Leveraging SIMD (Single Instruction Multiple Data) instructions for parallel data processing.
- Utilizing profiling tools like Chrome tracing or Perfetto to identify performance bottlenecks.
- Employing advanced compiler and runtime optimizations such as vectorized hashing, speculative execution, and static code generation per stage.
- Careful management of pipeline stages, including specific prolog and epilog sequences.
- Identifying and exploiting opportunities for parallel computation, for example, observing that bits 16 and 0 of stage 4 allow for parallel calculation of stage 5's parity.
- Minimizing or bypassing memory load bottlenecks.
The problem is considered a test of expertise in low-level systems programming and micro-architectural optimization, often requiring significant time investment even for experienced developers. The current public version allows for unlimited time to attempt the challenge, with submissions tested via python tests/submission_tests.py.