Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug

> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future.

Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.

Very nice article, and I appreciate SQLite's explanation of the bug too. And how extremely cool Tailscale appears to have been about it (paying for the VFS shim, etc.).

I'd have liked to have heard more about the decision to checkpoint so frequently that put them on this path though. Presumably that's to keep the WAL tiny for very fast recovery. Trying to mitigate some of the deleterious effects of inserting a DBMS into your network layer, I suppose? Tricky stuff. Wonder how that compares to typical etcd snapshot frequencies too.

Well written post, really enjoyed reading it.

> A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used.

This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQLite page[0] outline that it can only ever occur if there are multiple connections open, so the writer and the checkpointer must have been on different threads.

[0] https://sqlite.org/wal.html#the_wal_reset_bug

SQLite: 92 million lines of tests

Dijkstra: Tests can only prove the presence of bugs, never their absence!

This was a great technical writeup and very interesting to read, but it's not clear to me why once the suspected source of the bug was identified, they seemingly didn't build a automated way to trigger the condition? It seems like that could have cut down on the uncertainty of whether the fix worked over a painfully long period of time.

Great read. So glad they took the time to tell this story. (And glad they, as a for profit corporation, took out a support contract with SQLite. I hope they continue to do so even though this problem is resolved.)

Maybe it's just me, but the explanations of the cause don't align.

One clue was that during corruption incidents, our metrics showed that SQLite would report copying more pages from the WAL file than were actually available. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong.

vs

it thinks some of the pages have been copied from the WAL into the main database file, but they haven’t. Those pages never get written to the database file, and that data is permanently lost.

The first says "more were copied than existed" but the second says "fewer were copied than should have been."

Like I said, it's probably just me interpreting something incorrectly.

This was really, really interesting - what a triumphant adventure.

A few (very, very, very pedantic) things that stood out:

> We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky).

(Emphasis mine) - it would be "risky", not "potentially risky" - then the "calculated risk period" starts and it's "potentially problematic".

In the SQLite report[0] (11.2) I wish they downplayed this less - a mention of the rarity, then technical details - I'm friendly with a few of the devs/previous-devs, have the utmost respect for their skill and accomplishments (and by extension, faith that the developers I do not personally interact with are also excellent), appreciation and fondness for the huge accomplishment that is SQLite, and on and on... this is world-class work. Maybe section 11.2 wasn't really aimed at me, or I'm too critical. To be fair to all involved, what a minor quibble for such an interesting problem/fix. I hope my comment isn't a fly in the ointment.

Last bugfix point[1] - ugh. What a sinking feeling that must've been to deploy a fix then be flooded with not-green - and a lesson[2] against smuggling other changes in a changeset "just because we're already here"? Happy it turned out non-catastrophic, but did result in a rare (not remembering other instances of top of head) recall[3] from SQLite. That it was throwing errors at the same time SQLite and Tailscale were testing the other WAL-issue bug must've upset some stomachs for a moment.

[0] https://sqlite.org/wal.html#the_wal_reset_bug

[1] https://tailscale.com/blog/sqlite-wal-reset-bug#fixed-with-a...

[2] Nobody conceptually learned anything here - we're all just reminded of what we know: that sometimes "perfect storms" do actually occur.

[3] https://sqlite.org/releaselog/3_52_0.html

Great writeup, and it was great to see them step in an pay the developers of SQLite to help them fix the bug. I get tired of corporations asking open source authors to fix problems that affect the corporation for free. And while I'm sure it was frustrating for folks to have these outages, I find such puzzles pretty fun to get to the bottom of.

>In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups.

> running boring technology in a non-standard way is a risk.

It was a good read and reminder that the industry is loosing experts gradually. I am not a DBA and yet I have heard about this behavior at least couple times in the past as something to avoid. Its just one of those things which didnt get a chance to be documented cause experts avoided it and regulars didn't get into

Such a good write up . Having explored a little bit of sqlite internals for a codecrafters challenge i was mildly happy i could follow along what was happening .

What a brutal bug. I'd never entertain a that bug in SQLite could be causing problems in code I wrote.

Which SQLite driver for Go does Tailscale use?

The irony is that the SQLite developers get a support contract iff someone runs off the path in anger and finds an ancient bug. But perhaps that's part of what make it a quality team: devotion thriving without adverse incentives.

Tracking down a 16-year-old edge case in database internals is peak engineering perseverance. Incredible deep dive.

Block device upfuckery layers are powerful against databases. Years ago some colleagues wrote one that provides most of the hazards described by "Parity Lost and Parity Regained"[1] to test FoundationDB, which immediately uncovered several flaws in a project that described itself as well-tested. It's easy to do this with all the probing features that Linux (and others) provide today.

1: https://www.usenix.org/legacy/event/fast08/tech/full_papers/...

> Now we’re in summer, we’re confident that we’ve found the bug, that we understand it—and more importantly, that we’ve fixed it.

This is the feeling I chase as a software engineer. It's the greatest motivator.

Similar bug to the one that plagued Codex until 3 months ago.

Learned something new today, thanks

> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window.

Gotta love single points of failure...

you gotta admire the power of using json/b and simple KV stores.

so many people sleep on that.

I imagine the SQLite eschews AI generated code, but using it for testing (vulnerability, performance, etc) would seem like an easy win.

I know their proprietary testing framework is their secret sauce so we may never know...

While technically true as written, it seems to downplay the significance:

> The bug is a data race with tight timing constraints. It is unlikely to occur in common use.

A large customer did experience this corruption, so it's important for people with tailscale's setup update immediately.

> The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the the bug in order to verify that the issue has been fixed.