Projects

Chessformer ICLR 2026

Beat Stockfish to win three TCEC computer chess championships.

  • A model organism for mechanistic interpretability, studied by Jenner et al. (NeurIPS 2024, Stuart Russell’s group), Cruz (TMLR 2025), Sandmann et al. (NeurIPS 2026 oral) and Lin et al. (2026).
  • Outperforms DeepMind’s chess transformers with 30x less inference compute.
  • A geometry-aware attention mechanism that matches the strength of a 2.5x larger model at ~15% higher latency.

Paper Blog post

Attention maps from a Chessformer model

NanoGPT Speedrunning

Two independently verified record submissions to the GPT-2 pretraining speedrun on 8×H100.

  • 725 → 685 training steps from validation-time hints drawn from all unused data shards, with 4-token context and top-5 candidate continuations.
  • 1.0% less wall-clock time at lower loss from token normalization before n-gram embedding hashing, inspired by DeepSeek’s Engram.

PR #381 PR #375

Diagram: differently formatted n-grams normalize to one string and share one embedding slot

Generalized Advantage Estimation for NLAs

Generalized advantage estimation on per-token rewards for Anthropic’s NLA reduces training-time-to-performance by 30%.

Code

Held-out FVE versus RL step: GAE (lambda = 0.97) is ahead of GRPO at every evaluation

Stockfish Development

Merged 15 Elo playing strength improvements into Stockfish, the world’s strongest chess engine.

  • 2.5% speedup from permuting weights for block sparsity using CuPy.
  • Search heuristics that use evaluation uncertainty to guide pruning and depth reduction.
  • Each change validated by sequential probability ratio testing over tens of thousands of games.

Contributions