AI can write AI you can read
AI agents are quickly getting superhuman abilities in many things, and one of my favorites is their increasing ability to develop AI that is relatively readable. By readable, I mean that a human can read it and reason about it and, importantly, that what is read is what runs. I use the word readable over interpretable, as interpretation suggests that work needs to be done before we can read it, that multiple interpretations are possible, and that it is uncertain how well our interpretation holds for the system itself. The bird generator above is a small example. A neural network would hide a bird in a vector of unnamed dimensions, whereas here each of the 46 numbers has a name, such as beak length or tail spread, and the code in the panel under the map is the code that draws the birds. The font generator below is another. Every letter is described the way a type designer would describe it, with stems, bowls and arches, and a style is a short list of named numbers, such as pen weight, slant and shake, so the same letter descriptions drawn with different numbers give school handwriting or a typewriter face. The point of this post is to lay out my WIP mental model of how advanced general purpose AI improves our ability to produce readable AI.
Unreadable AI, such as deep learning, overtook readable AI in domains like computer vision and natural language processing. This is the bitter lesson: general methods that scale with compute, namely search and learning, eventually outperform methods that rely on building in human knowledge.[1]Sutton 2019. The Bitter Lesson.incompleteideas.net/IncIdeas/BitterLesson.html I believe readable AI fell behind because it was bottlenecked by people in two ways, and people do not scale with compute. The first bottleneck was simply coding: we were slow to code what we already knew, since writing a rule or a feature and testing it against data takes time and skill. The second was discovery: in some domains, we are slow to discover the properties of the input that explain the output. If something that scales with compute could do the coding and the discovery for us, readability could scale with compute as well (up to the limit of what humans can read). The sketch below shows this as a frontier, the best performance we know how to reach at each level of human readability. Improved general-purpose AI would push this whole frontier, which we can view either as improving readability at a given level of performance, or as improving performance at a given level of readability.
LLM-based AI agents, while still not very readable themselves,[2]Sharkey et al. 2025. Open Problems in Mechanistic Interpretability.arxiv.org/abs/2501.16496[3]Turpin et al. 2023. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS.arxiv.org/abs/2305.04388[4]Chen et al. 2025. Reasoning Models Don’t Always Say What They Think.arxiv.org/abs/2505.05410[5]Nanda 2025. Interpretability Will Not Reliably Find Deceptive AI. LessWrong.lesswrong.com/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai[6]Nanda et al. 2025. A Pragmatic Vision for Interpretability. LessWrong.lesswrong.com/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability are removing these bottlenecks. Superhuman coding agents are obliterating the coding bottleneck: they can codify second-hand knowledge, turning what the literature already says into rules, features and concept bottlenecks.[7]Hollmann, Müller and Hutter 2023. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. NeurIPS.arxiv.org/abs/2305.03403[8]Oikarinen et al. 2023. Label-Free Concept Bottleneck Models. ICLR.arxiv.org/abs/2304.06129[9]Yang et al. 2023. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR.arxiv.org/abs/2211.11158 The discovery bottleneck, which is the more daunting one, is also wavering. Agents are getting better at discovering rules and equations directly in data,[10]Romera-Paredes et al. 2023. Mathematical Discoveries from Program Search with Large Language Models. Nature.doi.org/10.1038/s41586-023-06924-6[11]Shojaee et al. 2024. LLM-SR: Scientific Equation Discovery via Programming with Large Language Models.arxiv.org/abs/2404.18400[12]Skalse 2023. Using (Uninterpretable) LLMs to Generate Interpretable AI Code. LessWrong.lesswrong.com/posts/sCJDstZrpCB8dQveA/using-uninterpretable-llms-to-generate-interpretable-ai-code[13]Schmidt and Lipson 2009. Distilling Free-Form Natural Laws from Experimental Data. Science.doi.org/10.1126/science.1165893[14]Udrescu and Tegmark 2020. AI Feynman: A Physics-Inspired Method for Symbolic Regression. Science Advances.arxiv.org/abs/1905.11481[15]Cranmer 2023. Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl.arxiv.org/abs/2305.01582 which keeps a system readable from the start, as Rudin argues for in high-stakes decisions.[16]Rudin 2019. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence.arxiv.org/abs/1811.10154 Machine learning training can be seen as discovery too, and agents are getting better at applying mechanistic interpretability to what a network has learned[17]Shaham et al. 2024. A Multimodal Automated Interpretability Agent.arxiv.org/abs/2404.14394[18]Wang et al. 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small.arxiv.org/abs/2211.00593 and codifying that interpretation as a program that replaces part or all of the network, so that again what we read is what runs.[19]Cranmer et al. 2020. Discovering Symbolic Models from Deep Learning with Inductive Biases. NeurIPS.arxiv.org/abs/2006.11287[20]Michaud et al. 2024. Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code. Entropy.arxiv.org/abs/2402.05110[21]Filan 2018. Mechanistic Transparency for Machine Learning. LessWrong.lesswrong.com/posts/3kwR2dufdJyJamHQq/mechanistic-transparency-for-machine-learning[22]Hubinger 2019. Chris Olah’s Views on AGI Safety. LessWrong.lesswrong.com/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety
The slop detector below is an example of both codifying and discovery. Of its 87 rules, 75 are tells that researchers and editors had already written down, which agents found through web search and codified. The other 12 have no reference: agents discovered them in the training texts. Finally, logistic regression is used to give each feature a weight towards the final prediction.
Some important caveats before I wrap it up. Readability is of course subjective, and to be readable, the code, the rules, concepts, features, symbols etc.. have to be small enough for a person to review: a million stacked rules might be even less readable than a end-to-end deep learning system it replaces. For deep learning systems, we can at least intuit some things from its objective, data, and architecture. Also, being able to read a system is not sufficient condition for understanding it. Also, readable is not necessarily safe.
The practical takeaway for AI practitioners today is inviting you to experiment with asking an agent for a readable solution to a problem where unreadable AI dominates, and measure how much performance you actually give up compared to a black-box solution. My current way of working is to spend my attention on setting up reasonable evals and let the agents cook on solutions, only occasionally nudging them towards strategies I think could be worth exploring.[7]Hollmann, Müller and Hutter 2023. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. NeurIPS.arxiv.org/abs/2305.03403
Prompt of the week
Copy this into a coding agent and fill in the brackets.
> task: [describe the task and if there are data] > goal: solve it with AI a person can read
> rules:
> readable = a human can easily understand it and its component parts > reading budget: [5 hours of study] > strategy: your choice > example paths to try: > search the literature and codify what it says > find existing neural networks (or any other ml model) or train your own and apply mech interp > induction loop: inspect examples, induce some rules, measure the residual error, repeat. > mind the splits: never leak test data > score on held-out data > bonus: compare it to a black-box baseline > report: the system, its score, the gap to the baseline if there is one, what was tried
References
- Sutton 2019. The Bitter Lesson. incompleteideas.net/IncIdeas/BitterLesson.html
- Sharkey et al. 2025. Open Problems in Mechanistic Interpretability. arxiv.org/abs/2501.16496
- Turpin et al. 2023. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS. arxiv.org/abs/2305.04388
- Chen et al. 2025. Reasoning Models Don’t Always Say What They Think. arxiv.org/abs/2505.05410
- Nanda 2025. Interpretability Will Not Reliably Find Deceptive AI. LessWrong. lesswrong.com/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai
- Nanda et al. 2025. A Pragmatic Vision for Interpretability. LessWrong. lesswrong.com/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability
- Hollmann, Müller and Hutter 2023. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. NeurIPS. arxiv.org/abs/2305.03403
- Oikarinen et al. 2023. Label-Free Concept Bottleneck Models. ICLR. arxiv.org/abs/2304.06129
- Yang et al. 2023. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR. arxiv.org/abs/2211.11158
- Romera-Paredes et al. 2023. Mathematical Discoveries from Program Search with Large Language Models. Nature. doi.org/10.1038/s41586-023-06924-6
- Shojaee et al. 2024. LLM-SR: Scientific Equation Discovery via Programming with Large Language Models. arxiv.org/abs/2404.18400
- Skalse 2023. Using (Uninterpretable) LLMs to Generate Interpretable AI Code. LessWrong. lesswrong.com/posts/sCJDstZrpCB8dQveA/using-uninterpretable-llms-to-generate-interpretable-ai-code
- Schmidt and Lipson 2009. Distilling Free-Form Natural Laws from Experimental Data. Science. doi.org/10.1126/science.1165893
- Udrescu and Tegmark 2020. AI Feynman: A Physics-Inspired Method for Symbolic Regression. Science Advances. arxiv.org/abs/1905.11481
- Cranmer 2023. Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl. arxiv.org/abs/2305.01582
- Rudin 2019. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence. arxiv.org/abs/1811.10154
- Shaham et al. 2024. A Multimodal Automated Interpretability Agent. arxiv.org/abs/2404.14394
- Wang et al. 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. arxiv.org/abs/2211.00593
- Cranmer et al. 2020. Discovering Symbolic Models from Deep Learning with Inductive Biases. NeurIPS. arxiv.org/abs/2006.11287
- Michaud et al. 2024. Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code. Entropy. arxiv.org/abs/2402.05110
- Filan 2018. Mechanistic Transparency for Machine Learning. LessWrong. lesswrong.com/posts/3kwR2dufdJyJamHQq/mechanistic-transparency-for-machine-learning
- Hubinger 2019. Chris Olah’s Views on AGI Safety. LessWrong. lesswrong.com/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety
- Trask et al. 2018. Neural Arithmetic Logic Units. NeurIPS. arxiv.org/abs/1808.00508
- Gao et al. 2023. PAL: Program-aided Language Models. ICML. arxiv.org/abs/2211.10435