Malware developers added nuclear and biological weapons text to to their spyware
- elashri - 4108 sekunder sedanI still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing.
So I wouldn't be able to develop a nuclear weapons with the resources of drug cartal (as an example) using Claude in secret.
- sciencejerk - 2004 sekunder sedanIf you actually read the Tweet, the exploit doesn't work against Fable, Opus, Grok...at least, in the examples.
Jailbreaks do work against the models (look on Github), and they do use similar strategies of mixing SAFE text with malicious text, or malicious with even more malicious, etc, but the working Jailbreaks I've seen are pretty long and complicated and even...creepy.
- ofjcihen - 4638 sekunder sedanWorked a contract where this succeeded in pushing through a fail open design.
It also should be a warning to everyone that these groups are now aware of analysis and deobfuscation using AI and to take using a sandboxed environment more seriously.
I’ve personally had about 20% success rate getting opus 4.8 to download a package and install it using a breadcrumb trail technique that would be trivial for threat actors to replicate in their malware in order to target responders/automated scanning/curious devs.
- - 1540 sekunder sedan
- strenholme - 3796 sekunder sedanThe solution is simple: If using an AI-assisted scanner and a guardrail gets hit, then the code is obviously malicious and needs to be automatically flagged (and refuse to run the code!).
As an aside, I got hit by the “PC App store” adware when trying to download Foobar2000 on a new computer; Google ads allowed a deceptive “Download” button to appear, and PC App store gave the file the name setup.exe. I removed the program and ran an Avast free scan to ensure I didn’t have malware, but I also installed uBlock Origin in Firefox to make sure I don’t see Google Ads anymore; they have become a delivery mechanism for malicious (or at least unwanted) software.
- logancbrown - 4393 sekunder sedanWould this realistically be a problem for code going through LLM-based code-review? Presumably if a LLM reviewer agent hits this commentary, it would produce a failure to analyze and exit, thus failing the automated code review and forcing a human to read through it which they would subsequentially catch and revoke.
- y-curious - 2774 sekunder sedanMy friend made this in jest (code very NSFW, ironically):
https://github.com/thebabush/mcp-job-security
Same energy and kind of a funny, low tech solution to frontier model analysis.
- carlsborg - 4072 sekunder sedanPipeline is then: Cheap open source model for flagging potential LLM refusal content -> main LLM check
- elevation - 5038 sekunder sedanWhy would a malware scanner read the comments?
- charcircuit - 4939 sekunder sedanThe sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".
- ipython - 5562 sekunder sedangood news, now we have pretty much a clear signal that there's something nefarious going on... after all, the first step to analyzing malware is to determine if it's malware at all.
- hurtigioll - 5189 sekunder sedandevs will say this is proof we need to remove all biological guardrails. think about that for a second
Nördnytt! 🤓