Asynchronous I/O in DuckDB: Work, Thread, Work

Does having the worker pool hold as many threads as cores work well alongside the async pool? It is basically oversubscribed by design.

I built a system once which had (this is Rust) a Rayon worker thread pool of 4 threads and a Tokio async pool of 2 (multithreaded runtime). On a system of 6 vCPU. This ended up working fine. Tokio was not starved so handled network requests at low latency.

One difference is DuckDB is a pure network client. If one of its async threads is starved it is not the end of the world (e.g. k8s does not kill your pod for failure of replying to health checks).

Using 512gb of ram for a 22gb remote file does feel a bit weird for a benchmark but maybe they couldn’t get a large number of cores without lots of memory?

Deep dive into asynchronous I/O architectures like this is pure engineering gold for high-performance data processing. Excellent breakdown.

I wonder how this would work in trying to parallelize the worker threads (multiple duckdb instances) coordinating them via Quack.

Ducks all the way down!

is there any aggregator for official docs such as this for db/systems/distributed arch at scale?

Do they have SSL updates yet? Signing is great, but using https means not fighting firewalls to start a job

DuckDB is trending towards becoming a query engine, specifically the fastest analytical query engine. This is very good.

This is such a long waited feature!

Brilliant deep dive into asynchronous I/O and execution thread models. Essential reading for high-performance data engineering.