We have a year to fix security everywhere

I'm confused why the worry about LLMs that will answer "how do I build a pipe bomb". That information is easily available other places. The anarchist cookbook has been around and available for 55 years, and yet pipe bombs are not going off all around us.

> On September 22, Apple is releasing the M5 Mac Studio with 256 GB of unified memory [..] it will probably [..] enough to write this snippet of code in 3 seconds

The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.

Zzzzz, we should have gotten security right a few decades ago. But security costs money and isn't a flashy feature to attract new customers, or cuts into your margin if you're a "real" business producing stuff or offering some service. Or whatever the decision makers in Berlin were thinking when they ignored security.

Yeah, we would still see hacks, but we would see less of them if security wasn't optional.

Maybe the AI craze helps by forcing more decision makes to see security as imperative, and by giving us another powerful tool for our tool box.

N.b.: I work in the security industry, our customers obviously want to improve their security. We've been seeing an uptick in awareness, but that's mostly due to NIS2 and other legislative efforts. Those force them to do something. AI is a curiosity for small talk to many of them.

Here's an idea: as a first step, simplify everything, and make sure you're aware how your stack works, and what it imports.

As an example: WordPress is a horrible thing, but the core has been through so much, that it's suprisingly secure. Then plugins and themes come, and whoosh, the security is gone.

We need a new KISS: keep it simple, stupid, secure.

I don't think we even have a year. The current batch of LLMs are ferociously good at identifying vulnerabilities.

I see a lot of people focused on servers and production environments, and of course that's needed, because that's business after all — but the personal computer seems to be absent from this discourse. Not everyone can buy a spare mac studio, and they might still need to install these tools on their personal computing devices, like their personal/home laptops. At that point, it's not even about whether a Claude Code, an OpenCode, or a Pi will steal/sniff personal data, but whether it can — though I think saying "it's a matter of 'when'" might be hyperbole. As of now, it's just: keep giving access and permissions or struggle while working, or create another user, or use Docker, run inside sandbox-exec, a VM, etc. As an end user, I am really scared. Someone who has been very disciplined and vehemently privacy- and security-conscious feels the ground below has just shifted.

OEM/OSes don't seem to have woken up to it yet. A mild proof is Apple's own special folder access reporting. When you go to Privacy & Security > Files & Folders, for a certain app, "Full Disk Access" is shown greyed out and mentioned in both cases — whether you had given Full Disk Access to that app or not. This directory-level permission UX is itself broken — there's Full Disk Access, and there's Files & Folders, and Full Disk Access gets shown in Files & Folders as well. This is, for lack of a better word, such an undesirable mess.

As of now I am debating between: creating a new user and just move everything work/learning to that user. Or just run all of it inside sandbox-exec (and maybe even block it from the shell if it tries to run outside it). Or use a tool that makes the latter easier and better. I even came across such a tool here on hn few weeks ago. agent-safehouse, yet to try it.

I think we have less time and the only remaining limitation is the actual cost to run such hacking campaigns. It does not appear expensive, but is not free, and there is a LOT of things to scan for vulnerabilities.

The models are already here, and one can rent a GPU cluster to run such workloads at speed - no need to play with slow local machines. I'd assume one can host the thinking at an unsuspected public cloud provider, proxy the network traffic to some botnet to evade blocking - and the only thing remaining is time and cost.

I do wonder what tools exist for boring, legitimate companies to try and do the same to their own systems to find the vulnerabilities before the bad guys do. The paradox here is I can't run a de-restricted chinese model with the same tools that hackers are using - but I think enterprises actually HAVE to do it in order to stand a chance in preparing for the onslaught.

1 year left for cybersecurity hardening, I thought so as well. The issue is, even if we get it done: in 1 year the models will be so good in social-engineering that they will be able to extract any information they want anyway. Happy to be falsified here, if anyone has evidence-based arguments.

EDIT: by social-engineering I mean for example: recon company structures, gathering and merging people's data from the dark-web, then using it to bribe/pressure/deceive users.

> This probably sounds like nonsense words or hysterical overreacting to most people, so here's what that means: "GLM" is a kind of LLM (AI) [...]

The post also sounds like that to people that understand the technology.

Calling that out like this and trying to pin that assessment to lack of knowledge is not a get-out-of-jail-free card, nor a good move.

__

Edit: Having spent some time letting the article marinate in my mind.

On the defending side, it is written that

> LLMs are good at writing patches, but not as one-off-prompts.

But this for me kinda conflicts with what is written on the attacking side:

> GLM 5.3-flash is so good at those tasks that human involvement in those tasks can be negligible. As a result, we are now in a world where cybersecurity attacks can be run in a for loop.

What is it? Can it be this autonomous terrifying entity or can it not be?

Yes, yes, attackers only need to win once, whereas defenders need to win every time, but that's not my point.

I sympathize with the sentiment but the suggested/implied guidance to fix bugs is wrong.

The overall game is increasing costs to exploit so much that attackers give up. Fixing 10 most obvious bugs, just very slightly increases costs, they would just a few more tokens to find another bug.

As someone said "I had infinite bugs, I fixed 1000, I still have infinite bugs".

To significantly increase exploit costs software/security has -1 years to do:

- Defense in Depth - Sandbox everything - Zero trust - Canary tokens - Split data from code (lol) - App Whitelisting - Reduce attack surface - Etc.

In other words, the only path is investing heavily on the "game changers" we have already discovered... but we are too cheap/lazy/coward/incompetent to apply.

And if we feel specially brave, changing the liability laws regarding software. Open Source & Proprietary code is so crappy because no gets jailed or fined when one of its dumb decisions results in millions of people have their data stolen.

Going to be hard to fix security when the frontier labs won't let us fix bugs in our own codebases without them offering refusals or bans.

The standard strategy of a security salesman since 1945. Develop dangerous weapons, show the damage they can do, and sell security cover to the terrified people.

Every single piece of technology did this. As a side effect or direct effect, they make bad guys more powerful and then keep on piling up new tech to deal with that. The cycle continues.

No you have a few years before we go back to the feudal era where almost everything is owned by a few and the rest are serfs. We are fast moving towards that world and this security bullshit is also about the same as they will use it to stop revolutions that will erupt.

Not sure, the labs will probably just cripple the security features of these models for a while I think and even potentially put back doors into systems for the security services…

A positive way to spin this is: we have a year to break in into any IT system. After that, it will be all either fixed or broken into, and all is fixed ever after. :)

As LLMs make formal verification cheaper (they can generate proofs that can then be automatically checked) many of the verifiable components of software systems, such as compilers and microkernels, will be verified. I suppose the issue is that the critical bugs are rarely in compilers and microkernels, but more often in applications, such as web browsers, which are more difficult to formally verify.

> And a big fuck you to DeAlignAI, Z.ai, and everyone else who's been participating in this race to the bottom.

I'm so glad frontier level AI isn't in the hands of just the Altmans and that other cult leader who are currently live testing their products in actual conflicts in the middle east and Ukraine.

Mmm, a world where a defender-LLM is essentially required is great news for people selling inference.

How to secure our identity layers(AuthN & AuthZ)? Let alone the products.

I don't see much hope since I last explored some github repositories. There was a time when a successful repo had about 10 - 20k stars and usually those older repos stay around this level. But now there is a ton of vibe coded slop 50k + stars. Most of them have a "nice look", maybe even extensive docs but are usually build with no security considerations at all. One recommended to provide a "google app password" to the agent which has the same permissions as your regular login. Another was a browser plugin with permissions to read all cookies, inject js, open background tabs etc. You would probably assume the chrome store would at least put some visible warnings on the app store page or force the user to actively confirm those permissions. But because they are already stated in the manifest there is only a small footnote and it's even "recommended by google".

We had decades to avert the worst effects of climate change…batten down the hatches.

The title reads like a Diary of a CEO thumbnail but unlike those discussions this article has a point.

Impotent slop code on one side and potent automated vulnerability exploitation on the other will lead to fun times.

Just like Cryptolocker, this will be the "Finding Out" phase for everyone who has been putting off best practice security.

But, lets be clear, Best Practice will save you. We can engineer assuming there are zero days in path. Go to your CTO now cap in hand and ask for overlapping controls, wafs, application monitoring, backups and all the other shit you haven't been doing.

Because when you find out, I will laugh, it will be very very very funny to me.

Maybe all these vital infrastructure companies should not have spent the past decades in a race to the bottom of cybersecurity. There is going to be a reckoning.

> Invest in formal verification, fuzzing and property testing, and memory-safe languages. LLMs are good at writing Lean and fuzz tests. I don't care whether you use Go or Rust but for the love of god please don't use C or C++ for new code.

How accepted is this thinking in your respective domains?

More tired “don’t use C or C++” advise.

It's just the same advice as ever: be extremely, exceedingly careful in what you expose to any network. When I set up machines for production, they don't respond to pings and they don't even have an SSH port open without knocking. There are also ways to eschew the need for an SSH port entirely.

People who never took that seriously will never take this seriously either, and that's their loss. (And loss of the commons, unfortunately.)

There's just also new advice: you can't afford to expose an unsecured system to the internet even for a moment. Think of those IPv4 address space scanners, except this time any one of them could be capable of developing individualized attacks in mere minutes. They don't sleep, they don't take breaks.

Another similar issue, with similar consequences and timeline, is Post Quantum Cryptography.

Clanker fodder. Unless the We are heads of states/heads of spooks or the AI powers that be (praise be) that there is no power and will to do that in one year or even ten years.

Remember how GLM 5.3 was going to cause massive hacks, break banks and ruin everything (it was even newsworthy since media picked up how people were working overtime in preparation).

And yet here we are.

[dead]

[dead]

[flagged]

Or we could just dump Linux and Windows and switch to a microkernel operating system, which is much more secure.

These endless patching cycles are simply not going to work in the long run. Operating systems get orphaned all the time, especially the ones in cheap Chinese stuff.