The 'Astra can read and write weights to improve Qwen' post is misleading

I work in this field (mech interp research) and I agree, the original post is definitely misleading. Some overselling in research is normal but it's a bit much.

First, the key result has nothing to do with making Qwen better at math, it just reduces reasoning loops.

Reasoning loops are a simple problem: the model generates the </think> token (which closes reasoning) with too low probability. You can fix this with simple steering approaches where you add a vector to different components of the LLM (https://arxiv.org/abs/2503.22048, https://arxiv.org/abs/2506.18167, etc). The method Astra used for identifying the vector is the standard procedure humans use (contrastive analysis). Such steering vector creation is basically the standard "intro-level interp project" for human researchers.

If you can code and understand LLM internals, you can do this yourself in 4-8 hours. The SWE equivalent would be doing binary patching with a debugger. Cool that frontier models can do this, but not unexpected since humans can do it without issue.

Speaking for myself. I’ve been aware of a lot of concepts around training, but stayed away from it until very recently.

A few weeks ago, I formed a hypothesis that I could get a smaller model good at the specific standards I have for my git commit messages, which are hard to get out of most LLMs today.

I spent the last four to five weeks on traditional training. It improved in a lot of areas, but not enough to be usable for me.

Yesterday, the article the author of this post is critical of gave me enough of an aha that maybe the problems I was seeing could be solved by having a larger LLM change the weights directly, without going through training.

In the last twelve-ish hours I’ve made more headway than I have in 2+ weeks. I’m using one of my last well performing checkpoints (full fine-tune + changes to the decision head/LoRA, etc.) and gave the model free rein to change any of the layers.

Its last two actions: (a) removed 16 participant-order directions from a head matrix, which improved one type of reasoning and degraded another, and (b) restored one of the 16, which reversed the degradation while keeping the core improvement.

All that to say, this might be known to people who’ve been around training for years, even decades, but it’s new to a lot of us. And we’re excited about it.

Wasn’t all that obvious to anyone who read the original post when they got to the section that explicitly explained what the author of this post is now summarizing?

Sure if you only read the title and brief at the top you’d walk away without the full picture though.

This is a better response imo: https://news.ycombinator.com/item?id=50344861

I feel like a lot of this Astra editing Qwen stuff is regurgitation of existing relatively well known techniques, for example: https://arxiv.org/html/2406.11717v3 and conceptually similar to: https://arxiv.org/abs/2512.05648 (tho this is done in training iirc)

I'm still impressed by the "Weight-Space Programming" idea. Perhaps its because I didn't know that LLMs could understand or successfully modify weights at all previous to reading it. However, I think the examples provided in the original piece were eye-opening. Yes, humans can build tooling to modify weights, but the author here agrees that it's tough. The examples where Astra was interacting with weights are still novel, and Astra seemed to handle them with surprising ease.

There was some really good comments by @yurimo on the 'Astra can read and write weights to improve Qwen' hacker news post https://news.ycombinator.com/item?id=50341174

But theoretically, what would prevent the labs from RL'ing the models on "weight-space programming" tasks so that they get better at them? Perhaps we're just at the beginning, like how coding agents sucked 2 years ago.

It still seems like a promising approach for RSI, distillation, fine-tuning, and a whole host of other tasks.

Some of original blog post really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.

So I do agree it is misleading and trying to present work as groundbreaking.

The original post explains what they did and why better than this post. This post taught me nothing that wasn’t in the original.

I called this in the comments of the post yesterday. Astra understands mechanistic interpretability literature and linear algebra and Python. So go figure what it would do with that knowledge.

Agreed theres a lot between "editing raw weights" via LLM and a bunch of code/programming. But in my case, the technique itself was new to me, and worked for a small ASR model I've been training based on moonshine + claude being able to actually reason about what was causing loops. This is something I sure couldn't/wouldn't do manually.

https://quickthoughts.ca/projects/welsh-asr/ (scroll down to the project updates to read how much impact it had)

Here is a measured take from Neel Nanda, who is quite publicly involved in serious interpretability research:

> This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted!

> This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!

https://xxcancel.com/NeelNanda5/status/2117375052618539485#m

Recent and related: Astra can read and write weights to improve Qwen - https://news.ycombinator.com/item?id=50341174 (870 comments)

LLMs are not superhuman at reading and writing neural network weights - https://news.ycombinator.com/item?id=50344861 (10 comments)

There is something about AI related posts that makes it irresistible for people to post a “rebuttal” that actually makes the same claim as the original post just with different vibes. It feels like a lot of discourse is spent on setting the vibe over actually adding anything to the conversation.

The post wasn’t misleading, the usual horde of online clickbait it generated was misleading. That isn’t on the researchers. Their writeup was very detailed and informative. The problem is that there’s no room for nuanced discussion on any online forum. People seek out and amplify low quality drama and then complain about it.

I pretty much got that idea when reading the post, on how it didnt have improvements on a Gemma 270M model which didn't have any looping issue. However, it still made me look at some bottlenecks in models I had trained (like MLP network for stylus input automation and a 17M reranker with NLI stance and noul heads) and allowed me to finetune without training, but just by adjusting some weights. But what's pretty cool is the zero shot sandbox development method they showed for a Go model. I tried to do that for a 17M re-ranker from zero weights and Astra built it using its reasoning without me having to use a GPU and find training datasets that are not CC by NC, and add some hours of tweaking and checkpoints. I don't know if its the best way to build something, but we might be building small models (maybe not LLMs) like with just programming the weights instead of finetuning.

In the end I got to replace a 68M model with a 17M model, and improve stlyus tool recognition accuracy from 88 to 96%. Was a pretty fun experiment too.

Why this post is here in the front page? The Qwen vector steering performed was a very selective operation that interested only specific layers and weights set, it is non trivial to identify and fix. This is remarkable. Also Experiment 3: no access to forward pass, so why here this post says the contrary of what the paper shows and everybody upvoted it? I don't understand.

>The 'Astra can read and write weights to improve Qwen' post is misleading

Does the author mean the "LLMs can natively read & write neural network weights" post? Did the researchers change the name or something? Because when I read it, it made it very clear what they were doing and none of it was misleading. It all aligns with what this author is saying.

Thank you. For some reason the voices of sanity got drowned in the original discussion. LLMs do not have an uncanny ability to speak weights. Yet.

I was blown away even by #1, and I don't feel after reading this that I was misled.

I don't get it. Because python was used, it's supposed to not be cool? IMHO, working with raw weights with Python is still working with raw weights.

Thank god someone finally brought attention to this. It’s amazing how I see this paper everywhere like it did some Herculean task but the paper is honesty misrepresenting (intentionally or not) the findings and making their write ups seem like they did something that it simply did not. We need more of this kind of critical thinking calling out bullshit papers and experiments..

Recently, as has probably been the case in other fields as well, many people feed the public cherry-picked results to hype up what AI can do or to stir up FOMO.

I used to think that in the dev community this sort of thing was less common than elsewhere... because people want to share their experience, what they found, etc.

Now that AI is here, people's experience and knowledge have basically been reset. So it's no different here now, which is a shame (I'm not saying it was better before, though)

Misleading how exactly when the original post states _exactly_ the same things as outlined in this objection?

Just like Graduate Students are good at Gradient Descent, they can see patterns and fine tune parameters 100000 times until the result looks good. So too can LLMs with their ability to examine patterns and brute force things.

Do you not think that it’s cool / unsettling how in theory we can just “ask” for certain behaviours etc to be added / removed?

I’ve experimented with this today and have been able to add “refusal” categories by ~following the paper, with pretty good results.

If nothing else I think this will spark more targeted research

Either way, a lot of devs will be brute forcing explorations in this direction today.

It doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.

None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.

One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.

We lack a build tool to handle an edge case yet unseen, no means exist to input the new weights and get a better model

Big deal; have some bpf probe handle kick off of the rebuild when the when new weights are tested by some verification process

This is just CI/CD work now

I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...

(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)

Just one word - MARKETING

oh man someone should make this into a loop where the model can alter it weights after an inference. some kind of descent of the local gradient.

Wow that was fast

something i found 15 minutes ago is that the car network test is "very very easy", as in Opus 5.5 in Claude Code did not even need to see the weights of the network to solve it correctly. i can give the transcript of the session. i'm pretty sure Opus didn't cheat

I completely agree with you Matthew, but I see the thing go in right direction in research as well, I am working in the same field where I am building an agent where it will improve the base model too !

Why doesn't Astra improve it's own weights

“Mech interp” has to be the most pretentious term I’ve heard in a while. Any normal group of people would call it MI.

That "Likewise" is quite load bearing in the original article :)

So in effect, all what we're quarreling about here is whether it is _misleading_ to use "Likewise" to transition between two paragraphs without implying that the details of the first apply to the second?

That’s a distinction with only some difference.

“You can’t have a machine for testing” is a weird restriction on AI, especially if you entertain even the remotest notion of possible AGI.

I for one think it's pretty cool how far too many people bend the truth or outright lie in tech these days.

There is going to be a reckoning for tech in the near future, because people will remember the lies and it's gonna be hard to beat the history.

Welcome to publications involving ML in 2026... Glad to see someone else write this though because it's important to start having these discussions. Too much overselling has been happening for too long. There is so much great about ML. IMO it hurts the field long-term to continue overselling claims like this.

In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.

edit: lo and behold! https://x.com/SchmidhuberAI/status/2117446822855888956

Waiting for

The "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading.

But can we use the idea inside the training loop, as an optimizer instead of (or next to) e.g. Adam?

Who watches the pruners?

A better idea is to have Astra modify it's own weights and let it try if it can RSI, but you gonna need to closely monitor for actual leap in capabilities so you can keep a check on alignment.

This isn't a good look. The original article clearly states what the model was doing, and this new hit piece is supposed to land because they are straw manning the comments on the original post? This isn't reddit dude.

This could have been a comment on the original article.

This goes to show that AI psychosis on HN is real, especially when many AI researchers have repeatedly dismissed this paper as highly misleading.

[flagged]

[flagged]

[dead]

[dead]

[dead]

[dead]

[flagged]

> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights.

The author is being equally disingenuous.

That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.

The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.

A tangent, but I’m now realizing that the “recursive self improvement” thing that these LLM companies are talking about is (probably?) not about unsupervised Claude Code instances improving the training process, but rather LLMs modifying their own weights through tooling directly. It would certainly open up a whole new process for model optimization.

The blog post exhibits the same flawed logic that many academic reviews do — looking for reasons to reject a work rather than accept it. It comes off cynical and contrarian. It's easy to find issues with an academic paper, but ACL has consistently pushed for a higher standard [1].

Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).

[1]: https://aclrollingreview.org/reviewerguidelines#-task-3-writ...