RipGrep musl binaries occasionally segfault during very-large searches

Heh, from the kernel patch: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...

> I saw a fun bug report in ripgrep and a studious but pretty bad AI-generated analysis

Referring to https://github.com/dfoxfranke/ripgrep-3494-analysis which I indeed thought "that's an awful lot written to have been written by a human."

Looks like that thread is from...today!

I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.

mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.

I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.

Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple users running these types of jobs simultaneously to bring a high-bandwidth filesystem to its knees. Just stop it already.

The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis.

So, why the bug triggers only with muslc and not other libc ?

I would normally suspect musl's thread stack size. Is the kernel bug confirmed?

[dead]

[dead]

[flagged]

wow, everything is broken lol

No wonder search in codex is so a$$

This is a bit sad. Here we have an example of rust and musl struggling whereas C/C++ and glibc does not. That's an oversimplification, but we also had not long ago the rewrite of ... I believe it was coreutils, in rust (or was it utillinux ... but I think it was coreutils), also have issues. Rust needs to toughen up here when it really wants to replace all of C in the linux ecosystem. Alternatively it could admit failure, then people can say that C will be - and remain - the forever king.