Discovering Cryptographic Weaknesses with Claude

I find that some of my friends and acquaintances have gotten obsessed with prompting style, "prompt engineering", which skills to use, which skills to build, "context engineering", and a billion other variations on "how to write smart things so the model does good".

Friends, look at the prompts that Anthropic's own people are putting into the machine:

> A few hours after the first message, we found that Claude was still searching for simple attacks and sent a message: “no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks”;

> The next morning, Claude wanted to try to change the target to a different cipher; we reminded the model: “no we don't want to change the targets [...] agian [sic] we need to find something that worth [sic] publishing”;

> That night, we sent one final message offering words of encouragement: “again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.”

All of that RLHF and fine-tuning effort is going toward making prompts like this, or worse, work with no fuss.

When high quality effort is applied to a tool, such as AES or the linux kernel, we intuit that it "hardens" the tool. That is, it makes the tool more correct, more resilient, less assailable, etc.

Similarly, when effort is applied to an open problem, such as the Riemann hypothesis or P v NP, without progress, it "hardens" the problem: it makes the problem feel more daunting to whoever takes a stab at it next.

Andrew Wiles, whose interview also hit the homepage today (https://news.ycombinator.com/item?id=49075264), couldn't just tackle Fermat's Last Theorem head on, he had to wait until a different, modern problem reduced to it, because FLT had gathered this mystique of unassailability through its 300 years of existence.

A thing I worry about is that as AI transmutes tokens into effort, it'll split the world into two: some problems will yield, making human effort entirely unnecessary, and others will harden to the point where human effort will feel increasingly less worthwhile, because "even AI couldn't solve it". I don't like this. AI is spiky, so I suspect it'll continue having major blind spots, and yet its mere presence will probably have a chilling effect on what would have otherwise been useful human effort.

“Each of the results cost roughly $100,000 in API cost to develop.”

And

“Over the course of a week, one Anthropic researcher worked together with Claude to develop the HAWK attack, and another researcher built a scaffold4 that allowed Claude to fully autonomously discover the AES attack.”

Spending $100k in tokens in a week is an impressive feat even with massive parallelization. I suspect the TPS their internal folks have access to is far higher than their bulk public endpoints.

There’s a tech aristocracy rapidly emerging in our society and it’s going to tear us apart.

I can already picture the faces of national security directors everywhere.

"The attacks described in these two papers are the strongest attacks we have found to date. We are sharing them after a period of consultation with US government and industry leaders. But as we develop increasingly powerful cryptanalytic results, it would be prudent to consider how researchers should react if a language model were to discover vulnerabilities in cryptosystems where attacks do have an immediate real-world impact. We believe answering this question will require input from academia, government, and industry. We hope that our work here will help launch these conversations."

And a veiled pitch to real cryptanalysis researchers: "Researchers at Anthropic then spent several hundred hours learning enough cryptography research to validate the model’s claim"

> The multi-agent workflow led to interesting dynamics. For example, the key idea in producing this attack was discovered by a pair of workers working together. Both started investigating the idea; the first worker prematurely rejected the idea as infeasible, but the second found a way to fully exploit it. The pair kept exchanging messages, and eventually both agreed they had found an effective attack.

this is pretty interesting. the way it is written doesn't make it sound like the collaboration actually led to the discovery, but rather just the stochastic nature of each thread in the search. it would be interesting to replay and repeat the search (possibly with prior/context pertubations) to get a sense for how often it finds or misses the known working path.

> But as we develop increasingly powerful cryptanalytic results, it would be prudent to consider how researchers should react if a language model were to discover vulnerabilities in cryptosystems where attacks do have an immediate real-world impact.

How would they react if a human were to discover vulnerabilities in cryptosystems?

Interesting they are still using "Mythos Preview" instead of "Mythos 5"; I had read from others who had access to both that Mythos 5 is less capable.

This seems like a bit of an overstatement:

  Despite HAWK having survived two rounds of expert human review over a period of two years, Mythos was able to improve the best-known attack on it in just 60 hours of work—effectively cutting its key strength in half.

since, later:

  Mythos’s attack works by finding a specific, previously unexploited symmetry called a nontrivial automorphism in the lattice used by HAWK. Prior work proved that efficiently finding such an automorphism would permit an attack, but did not answer if such an automorphism was accessible in the lattice used by HAWK. The automorphism discovered by Mythos allows a faster enumeration attack that, while still exponential, means that one needs to double the size of HAWK keys to achieve the same level of security.

Not downplaying Mythos's contribution here[1], but that first paragraph strongly hinted (at least to me) that there were no known weaknesses. "Discovering a weakness that had previously been only theoretical" is vastly different from "discovering an unknown weakness." Again: very cool Mythos was able to do this. It just seems like another case of "LLMs are good at finding concrete mathematical (counter)examples" - which is also cool! But the PR here is cynical.

...and it is kind of incredible to think that they spent $100,000 over 3 days looking for an automorphism. Not the possibility of an automorphism, that was already known. Man.

[1] ... or focusing too hard on the strange use of mathematical language...

TL;DR: They marginally improved on the best known academic attack on 7-round AES-128 (which normally uses 10 rounds - you do not need to worry about AES being broken).

The attack on HAWK is perhaps more interesting - they were able to halve the effective key length. HAWK is a candidate for NIST standardisation. It has been studied academically, but isn't really deployed anywhere (because it hasn't been standardised!)

One attack weakens HAWK, a post-quantum cryptography cipher candidate. I don't trust these PQC things one bit. I'll use them in combination with a strong clasically-resistant cipher (in so-called hybrid encryption modes), but not alone.

There's a push to turn off the classical modes and rely entirely on PQC for both quantum and classical security. Uh... no, thank you? Why would we want to do that at this point? The classical cipher component isn't hurting anything. Awfully creepy to pushing reliance on the new thing alone.

... especially now that we have LLM-discovered attacks on the new things.

Great now can you make opus 5 work please

[flagged]

[dead]

[flagged]