Neural networks are clean

Nick is a force of nature. Highly recommend following him on X if you don't already.

His launch post gives good context not found in the blog:

> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had

> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

https://x.com/nickcammarata/status/2110757819427574056?s=46

The claims seem to be an overreach of what is actually proven. We've known about progressive feature complexity and the impacts of early training structures on later training for a really long time (since CNN's, The Lottery Ticket Hypothesis, etc...) and neither of those phenomenon are mutually exclusive with parameter overloading. The Multimodal Gap work (https://arxiv.org/abs/2203.02053) seems to imply networks have a tendency towards condensed representations that are clearly not semantic and a bunch of other considerations are already on the table for discussion. I don't think these have all been considered sufficiently to start claiming we are around the corner from mechanistic interpretation.

It reminds me of this quote from Judith Butler:

"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."

https://www.theguardian.com/books/1999/dec/24/news

this article seems to take it on faith that neurons are aeasily findable, even if their meaning is tricky to untangle: it would be sharply improved by description of how the neurons are being isolated before the meat of the article, the naming of the neurons, is performed. In particular axis aligned semantic directions are incompatible with the classic superposition description, where the meaningful directions are nearly orthogonal, allowing many more to be packed onto the sphere than it has dimensions. (i’m also aware of work showing that Adam or relu pushes semantics onto the axes, but it’s not clear that that’s what he thinks is happening)

The tone reminds me a little of the enthusiasm from the Anthropic mechinterp team around the time of the first large-scale SAE results. A kind of hypomanic energy around the idea that we've discovered the key paradigm necessary to understand models, and all that's left is to scale it up.

To cut through some of all this text - here's a link he calls "the first non-trivial neural network I feel I mostly understand:

https://already-clean.goodfire.com/microscope/medium.html?la...

Yudkowsky seems unconvinced.

https://x.com/allTheYud/status/2110805233454731449?s=20

This is the primary reason, to me, that open weights are so important. If OpenAI and Anthropic want these models to be as autonomous as possible they also need to provide the means to understand them fully.

A TLDR, since most comments here seem to misunderstand the result.

The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.

How do you do that? Theres two ways:

1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.

2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.

Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.

However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.

So we have here a massive evidentiary brief, the point of which is to argue that it is easier than we thought to, like, "sequence the AI genome" (in my estimation). We revisit famously-messy neurons, examine various tool failures, check places where messy things still hide, and catalog how "polysemanticity" is represented variously in the network.

Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.

I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.

I'm not sure I buy the author's arguments for mechanistic interpretability being easy, but it sure is a well-made and very interesting post. Would be better without the completely superfluous laggy 3d city model, though.

Isn't it ironic that an article on AI explainability is itself so hard to understand?

I think certain classes of artificial neural networks are definitely more interpretable than others.

For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.

The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.

I only skimmed through this post. But I think this is a crux:

> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.

I have yet to fully finish going trough this but I would not rush to say that superposition is completely wrong. I understand that the message is instead of first learning a new representation (e.g. SAEs/transcoders) and then interpreting those learned features, you try to understand the neurons and circuits that already exist in the model.

This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).

I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.

This line of work is a mirage. You're much better off investing in a project like latentpedia.org and building a new class of LLMs which support dual use. Deterministic and less compressed or probabilistic and fit in GPU memory.

I'd be interested to see where this goes, but it seems to rely on LLMs being mostly or entirely compositional (i.e. ultimately, they can be taken apart to be understood). But if LLMs have notable emergent properties - which seems likely - then those parts of the LLM will be effectively off-limits to this sort of interpretation.

And emergence doesn't need to involve much of the LLM's weights:

> In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood

If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks.

Seems like the TLDR is that the neural network is not randomly placing things together due to constraint, it is actually finding some commonality from the input dataset. We just need to spend more effort trying to trace it.

Perhaps our anthropomorphic understanding of data is biasing what objective features there actually are in the dataset.

I wonder if we can study the graph similarities of various models, do we see a convergence of structures that encapsulate certain concepts. And would that convergence be due to just how humans have narrowly defined the concept in our language, or is there something inherent to the concept that allows it to converge. If we compare the same concept in different languages (ie. chinese vs english), does this convergence disappear?

Interpretation comes down to ultimately arbitrary human assignment of a succinct and compressed syntax and semantics to summarize a big data set; so yeah eventually something new to describe models will emerge. It's how it goes.

Also how it goes is this new consensus will be a leaky abstraction and we'll keep conjuring new words to describe new rhetorical features and bloat the corpus of philosophy until everyone is over the whole concept and then we'll start over with a whole new arbitrary academic philosophy for "running algebra on a computer"

> When I retrain its original architecture, which still produces messy deep layers, and follow the least legible neurons I can find, they turn out to be overgrowing locally into reasonable, adjacent territory.

It's interesting to observe that human reasoning is still the barometer of what "reasonably adjacent" means in the context of a NN's structure.

This is amazing. Feels like it will be the start of a new era for interpretability research

Intuitively this makes sense. When you have small models, they need to compress more by leveraging superposition and polysemanticity.

Nick is looking at a 32B model, which certainly has to compress less... Now we have 10 trillion parameter models... do they need to compress at all? As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates

On a practical level when could I take an open weight LLM and get a print out of what each parameter does?

I've read shorter books. Great piece, at least the parts that I managed to read until I ran out of time. Interactive teaching aids, too.

Is the use of "generalization" like this standard in mech interpretability? Seems undesirable given the typical definition in ML.

JShelter alerts me of active fingerprinting on the website. Why, Nick.

Glad to hear it is easier to clean your room than ever. Congrats to Nick

Feels like I'm back in the distill.pub days!

> The bad news about modern neural networks is that they're enormous. The good news is that the enormity helps...

I always catch on misuse of enormity

It's now so widespread the misuse has gained despairing acceptance, as with "infer" for "imply"... but that doesn't make it any less wince-inducing.

For the layperson who doesn't want to read 20,000 words, many of which are quite redundant due to the way the article was built, this is a version which adds a glossary, preserves the original language (except the added intro and conclusion which I felt was lacking) but removes all repeated material and claims, of which there was a massive amount.

https://pastebin.com/raw/p11vzFnB

There is a philosophical problem hidden in the core of interpretability: Can a mind understand a mind much greater than itself?

"Early model organisms"? Really???

Oof, right at the top it points to how people were apparently convinced of this situation from a single crappy example

You work in data science and you over indexed on a singular example to explain away an entire set of phenomenon??

My experience with more.... directly testable NN models that are small seems to be that they absolutely can be far too interconnected to make strong explanations. The structure and pathways they fall into don't have to make inherent sense to anything or anyone.

Is it possible that almost all nodes in a network being explainable with a singular feature that makes obvious sense to us is evidence of poor efficiency? The network hasn't been made to collapse learning or discover more effective organizational structures than what might make sense to us.

I don't think I have the fundamentals here to contribute though.

Working with some neural networks, you can somewhat feel the structure when you read between the lines of the output. You can sometimes feel that a threshold somewhere tripped and set the signals down an alternative path.

But confirmation bias and all that.

I hope interpretability has a breakthrough. It would make my job far easier and cooler.

I had figured the "Recursive generalization" was like, understood? Maybe I'm just too pop-sci brained. "Split categories into more on this new boundary" and "Combine two separate categories that aren't different enough" I thought were always the core verbs for working with bulk data. My mental model has never been formalized though, so is invalid.

I guess I never got a good sense of the examples they had which made it look like what was "obvious" or made sense to me wasn't always a good explanation in general. Neat.

It's interesting that meaningful examples both ways care about shapes so much, as your eyes encode edge finding and shapes really early in the signal path. It's why some optical illusions are so effective. It's awesome.

The whole "how do you test a neuron: Find a group of nouns and find where you can split them" is just how I find I think learning works. You build your mental model, and find the split points where it diverges from reality and you are wrong, and you refine. I feel like this is how I learned Random Forests conceptually, though surely that's not a valid comparison?

Apologies, not sure I have actually contributed anything here! I bet I misunderstood the entire article somehow. Specifically, I do not have the fundamentals of looking into visual models that he is really working from. But I always figured that when Data Scientists talk of "Features" in larger models, they explicitly meant these far more abstract representational "features". Doesn't the premise of model hijacking and abbliteration sort of presuppose that you have abstract "Reject" or "Safety" neuron paths? I've always understood this stuff as that neuron path, without weighting too much on individual neurons.

>I used to think of neurons as human-shaped nouns: one is a car, another is a curve.

I didn't think anyone was ever supposed to think this for human neurons! The way humans identify something as even a "Chair" is insanely abstract and cultural! Very specifically, the "building sub branches to disambiguate" is just, like, how you would expect a hierarchical human brain to self organize, right?

Some of this is why I've been so displeased with the interpretability research from Anthropic, where it felt like they just pointed new NNs at the disparate neurons of another NN and post hoc named them. I felt the paths were of importance at least but had no idea how to convey, interpret, or model that. Learn you a good math kids! Not having any formalisms hurts.

Reading this made me recognize that better interpretability yields not just better aligned but more compact models too. Which makes a lot of sense in hindsight, and even has some ironic ring to it when considering the unending amount of anthropocentric takes regarding what an "understanding" means.

Wow, I thought it was a GIF, but it's actually interactive. The website is just too heavy, though.

> After many months studying neurons and circuits in image and language models

There are no neurons or anything like them in these image and language models.

Aye, will we stop hammering nails on God's coffin?

What's next? Maybe in a little bit we will discover that we H. Sapiens suffer from severe polysemanticity in our wetware because our individual neurons are messy and massive like an ENIAC while the corvids in the park have neat and efficient little brains that don't need unfolding ?... How's the pope (may God stop afflicting him with cancer or may Man make him live forever) going to explain that?

This website crashed my phone - let's not have auto-playing embedded 3D renders, or whatever that was?

[dead]

[flagged]

Why does this post have a very laggy city gif thing at the top?

16,000 words into the essay: "What I mean by clean"

...and it still doesn't answer it. What is this actually about?

[flagged]

[flagged]