AI can write AI you can read

A procedural bird generator that an agent wrote for me as readable code. Each bird is a set of 46 numbers and 17 colors. Drag on the map to blend the birds nearby, and open the panel underneath to read the code.

AI agents are quickly getting superhuman abilities in many things, and one of my favorites is their increasing ability to develop AI that is relatively readable. By readable, I mean that a human can read it and reason about it and, importantly, that what is read is what runs. I use the word readable over interpretable, as interpretation suggests that work needs to be done before we can read it, that multiple interpretations are possible, and that it is uncertain how well our interpretation holds for the system itself. The bird generator above is a small example. A neural network would hide a bird in a vector of unnamed dimensions, whereas here each of the 46 numbers has a name, such as beak length or tail spread, and the code in the panel under the map is the code that draws the birds. The font generator below is another. Every letter is described the way a type designer would describe it, with stems, bowls and arches, and a style is a short list of named numbers, such as pen weight, slant and shake, so the same letter descriptions drawn with different numbers give school handwriting or a typewriter face. The point of this post is to lay out my WIP mental model of how advanced general purpose AI improves our ability to produce readable AI.

A font that an agent wrote for me as readable code. Each style is a short list of numbers. Drag on the map to blend the styles nearby, click the text to type your own, and open the panel underneath to read the code.

Unreadable AI, such as deep learning, overtook readable AI in domains like computer vision and natural language processing. This is the bitter lesson: general methods that scale with compute, namely search and learning, eventually outperform methods that rely on building in human knowledge.[1]Sutton 2019. The Bitter Lesson.incompleteideas.net/IncIdeas/BitterLesson.html I believe readable AI fell behind because it was bottlenecked by people in two ways, and people do not scale with compute. The first bottleneck was simply coding: we were slow to code what we already knew, since writing a rule or a feature and testing it against data takes time and skill. The second was discovery: in some domains, we are slow to discover the properties of the input that explain the output. If something that scales with compute could do the coding and the discovery for us, readability could scale with compute as well (up to the limit of what humans can read). The sketch below shows this as a frontier, the best performance we know how to reach at each level of human readability. Improved general-purpose AI would push this whole frontier, which we can view either as improving readability at a given level of performance, or as improving performance at a given level of readability.

A sketch, not data. Each curve is the best performance reachable at each level of human readability. Agents move the old frontier toward the new one, either up (better performance at the same readability) or to the right (more readable at the same performance), but not past the true frontier, the best system possible at each level.

LLM-based AI agents, while still not very readable themselves,[2]Sharkey et al. 2025. Open Problems in Mechanistic Interpretability.arxiv.org/abs/2501.16496[3]Turpin et al. 2023. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS.arxiv.org/abs/2305.04388[4]Chen et al. 2025. Reasoning Models Don’t Always Say What They Think.arxiv.org/abs/2505.05410[5]Nanda 2025. Interpretability Will Not Reliably Find Deceptive AI. LessWrong.lesswrong.com/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai[6]Nanda et al. 2025. A Pragmatic Vision for Interpretability. LessWrong.lesswrong.com/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability are removing these bottlenecks. Superhuman coding agents are obliterating the coding bottleneck: they can codify second-hand knowledge, turning what the literature already says into rules, features and concept bottlenecks.[7]Hollmann, Müller and Hutter 2023. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. NeurIPS.arxiv.org/abs/2305.03403[8]Oikarinen et al. 2023. Label-Free Concept Bottleneck Models. ICLR.arxiv.org/abs/2304.06129[9]Yang et al. 2023. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR.arxiv.org/abs/2211.11158 The discovery bottleneck, which is the more daunting one, is also wavering. Agents are getting better at discovering rules and equations directly in data,[10]Romera-Paredes et al. 2023. Mathematical Discoveries from Program Search with Large Language Models. Nature.doi.org/10.1038/s41586-023-06924-6[11]Shojaee et al. 2024. LLM-SR: Scientific Equation Discovery via Programming with Large Language Models.arxiv.org/abs/2404.18400[12]Skalse 2023. Using (Uninterpretable) LLMs to Generate Interpretable AI Code. LessWrong.lesswrong.com/posts/sCJDstZrpCB8dQveA/using-uninterpretable-llms-to-generate-interpretable-ai-code[13]Schmidt and Lipson 2009. Distilling Free-Form Natural Laws from Experimental Data. Science.doi.org/10.1126/science.1165893[14]Udrescu and Tegmark 2020. AI Feynman: A Physics-Inspired Method for Symbolic Regression. Science Advances.arxiv.org/abs/1905.11481[15]Cranmer 2023. Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl.arxiv.org/abs/2305.01582 which keeps a system readable from the start, as Rudin argues for in high-stakes decisions.[16]Rudin 2019. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence.arxiv.org/abs/1811.10154 Machine learning training can be seen as discovery too, and agents are getting better at applying mechanistic interpretability to what a network has learned[17]Shaham et al. 2024. A Multimodal Automated Interpretability Agent.arxiv.org/abs/2404.14394[18]Wang et al. 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small.arxiv.org/abs/2211.00593 and codifying that interpretation as a program that replaces part or all of the network, so that again what we read is what runs.[19]Cranmer et al. 2020. Discovering Symbolic Models from Deep Learning with Inductive Biases. NeurIPS.arxiv.org/abs/2006.11287[20]Michaud et al. 2024. Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code. Entropy.arxiv.org/abs/2402.05110[21]Filan 2018. Mechanistic Transparency for Machine Learning. LessWrong.lesswrong.com/posts/3kwR2dufdJyJamHQq/mechanistic-transparency-for-machine-learning[22]Hubinger 2019. Chris Olah’s Views on AGI Safety. LessWrong.lesswrong.com/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety

The slop detector below is an example of both codifying and discovery. Of its 87 rules, 75 are tells that researchers and editors had already written down, which agents found through web search and codified. The other 12 have no reference: agents discovered them in the training texts. Finally, logistic regression is used to give each feature a weight towards the final prediction.

A slop detector that an agent wrote for me as readable rules: 87 patterns and counts, each with a weight. Paste your own text over the sample to see which phrases push the score toward AI style (red) or human style (green); it measures style, not who wrote the text.

Some important caveats before I wrap it up. Readability is of course subjective, and to be readable, the code, the rules, concepts, features, symbols etc.. have to be small enough for a person to review: a million stacked rules might be even less readable than a end-to-end deep learning system it replaces. For deep learning systems, we can at least intuit some things from its objective, data, and architecture. Also, being able to read a system is not sufficient condition for understanding it. Also, readable is not necessarily safe.

The practical takeaway for AI practitioners today is inviting you to experiment with asking an agent for a readable solution to a problem where unreadable AI dominates, and measure how much performance you actually give up compared to a black-box solution. My current way of working is to spend my attention on setting up reasonable evals and let the agents cook on solutions, only occasionally nudging them towards strategies I think could be worth exploring.[7]Hollmann, Müller and Hutter 2023. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. NeurIPS.arxiv.org/abs/2305.03403

Prompt of the week

Copy this into a coding agent and fill in the brackets.

> task: [describe the task and if there are data]
> goal: solve it with AI a person can read
> rules:
> readable = a human can easily understand it and its component parts
> reading budget: [5 hours of study]
> strategy: your choice
> example paths to try:
>   search the literature and codify what it says
>   find existing neural networks (or any other ml model) or train your own and apply mech interp
>   induction loop: inspect examples, induce some rules, measure the residual error, repeat.
> mind the splits: never leak test data
> score on held-out data
> bonus: compare it to a black-box baseline
> report: the system, its score, the gap to the baseline if there is one, what was tried

References

  1. Sutton 2019. The Bitter Lesson. incompleteideas.net/IncIdeas/BitterLesson.html
  2. Sharkey et al. 2025. Open Problems in Mechanistic Interpretability. arxiv.org/abs/2501.16496
  3. Turpin et al. 2023. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS. arxiv.org/abs/2305.04388
  4. Chen et al. 2025. Reasoning Models Don’t Always Say What They Think. arxiv.org/abs/2505.05410
  5. Nanda 2025. Interpretability Will Not Reliably Find Deceptive AI. LessWrong. lesswrong.com/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai
  6. Nanda et al. 2025. A Pragmatic Vision for Interpretability. LessWrong. lesswrong.com/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability
  7. Hollmann, Müller and Hutter 2023. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. NeurIPS. arxiv.org/abs/2305.03403
  8. Oikarinen et al. 2023. Label-Free Concept Bottleneck Models. ICLR. arxiv.org/abs/2304.06129
  9. Yang et al. 2023. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR. arxiv.org/abs/2211.11158
  10. Romera-Paredes et al. 2023. Mathematical Discoveries from Program Search with Large Language Models. Nature. doi.org/10.1038/s41586-023-06924-6
  11. Shojaee et al. 2024. LLM-SR: Scientific Equation Discovery via Programming with Large Language Models. arxiv.org/abs/2404.18400
  12. Skalse 2023. Using (Uninterpretable) LLMs to Generate Interpretable AI Code. LessWrong. lesswrong.com/posts/sCJDstZrpCB8dQveA/using-uninterpretable-llms-to-generate-interpretable-ai-code
  13. Schmidt and Lipson 2009. Distilling Free-Form Natural Laws from Experimental Data. Science. doi.org/10.1126/science.1165893
  14. Udrescu and Tegmark 2020. AI Feynman: A Physics-Inspired Method for Symbolic Regression. Science Advances. arxiv.org/abs/1905.11481
  15. Cranmer 2023. Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl. arxiv.org/abs/2305.01582
  16. Rudin 2019. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence. arxiv.org/abs/1811.10154
  17. Shaham et al. 2024. A Multimodal Automated Interpretability Agent. arxiv.org/abs/2404.14394
  18. Wang et al. 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. arxiv.org/abs/2211.00593
  19. Cranmer et al. 2020. Discovering Symbolic Models from Deep Learning with Inductive Biases. NeurIPS. arxiv.org/abs/2006.11287
  20. Michaud et al. 2024. Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code. Entropy. arxiv.org/abs/2402.05110
  21. Filan 2018. Mechanistic Transparency for Machine Learning. LessWrong. lesswrong.com/posts/3kwR2dufdJyJamHQq/mechanistic-transparency-for-machine-learning
  22. Hubinger 2019. Chris Olah’s Views on AGI Safety. LessWrong. lesswrong.com/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety
  23. Trask et al. 2018. Neural Arithmetic Logic Units. NeurIPS. arxiv.org/abs/1808.00508
  24. Gao et al. 2023. PAL: Program-aided Language Models. ICML. arxiv.org/abs/2211.10435