Docker is weird

Hi, I'm the creator of Docker and came across this post. It's always fun to see other engineers think about how you designed a system. I haven't been involved in developing Docker in many years, but it's a fun trip down memory lane.

I'm happy to answer any questions you may have.

What made Docker unique IMO is not the container technology itself. I used chroot 20 years ago and it worked just fine. Yes, no network namespaces; no process namespaces, but I don't even think these are that important.

What made Docker unique were there things:

1. Dockerfile concept. Building images layer by layer. Caching layers. Everything in a single file. It made possible to iterate on image quickly. It made possible to implement CI builds efficiently.

2. Docker network registry concept. Putting universal images onto a server and share them with the world.

3. Momentum. A lot of software that was traditionally complicated to deploy, suddenly appeared in docker servers. I remember how fascinated I was, when I could deploy wordpress website with tiny docker-compose. And how contained it was. I could just delete every trace of it from my host. I always felt uneasy installing various things onto my host and docker just solved that uneasiness.

When Docker came out, I was a heavy user of OpenVZ.

My bash build scripts looked like Dockerfiles.

Most of the container code in Linux came from those russian guys.

Took me 6 months to find out the difference between stateless and stateful containers, after Jerome Petazzoni published a blog post opposing the usage of an SSH server inside containers.

I prefer Podman most of the time, but Docker is/was amazing and a great DX put on top of the so called "5 different linux kernel features in trenchcoat". That's the selling point, it's super lightweight and takes full advantage of the OS features that were already there (or being developed) but were awkward to use and required tons of specialized knowledge.

> Docker’s container runtime, runc, is written in Go.

I worked at Docker from 2014-2019. The most important part is actually written in C.

[I wrote a whole comment here about to explain how the hack works because I was too fixated on how technically incorrect "written in Go" is. Then I went back to read the rest of the article only to see that the paragraph immediately after this kind of explains it in its own way! Let that be a lesson to always finish reading the article before posting about it.]

Lots of positive stuff in this thread which I won't bother repeating. Instead, I'll add a few ambivalent points:

- Today, a lot of functionality still in Docker has been upstreamed into the (Linux) kernel, so containers can be much simpler (and rootless). Docker is still carrying around some of that cruft, which is what makes it slower and heavier than Podman.

- As a devex technology, Docker/containers have been massively successful, but not as infrastructure. Containers never became secure enough to run multitenant workloads without some other form of isolation, so they ended up being redundant with KVM (or gVisor, or a similar second layer) in production a lot of the time.

That second contradiction led to lots of inefficiency. For example, on Google App Engine, for years when you ran a deploy it would create a VM just to run a single container.

Always nice to see someone demystifying a complex thing by learning the building blocks it's made from.

Knowing this, you might find rootless Podman's fork-exec method of starting containers to be more sensible. I certainly do. No root, no daemon, no socket.

If you want to learn more about the Linux Kernel namespaces which work together to make a container, this LWN series is really good: https://lwn.net/Articles/531114/

Every statement in the article make me ask: "what's weird about it?"

Everything in that list feel natural when you know how things evolves over many years. Maybe it is just me getting older and less curious.

To solve the networking setup, typically some sort of "slirp" is needed. Slirp on one hand takes (fake or real) packets from somewhere and then on the other side pass it to a normal Linux process that does CONNECT / BIND / SEND / RECV.

It turns out you can do all the packet processing magic (connection tracking, syn handling, etc) in userspace totally fine, and then indeed hand it over this to userspace syscalls.

Being upset with classic C-based slirp and rust smoltcp (not enough features) I've dug up Fuchsia userspace tcp/ip stack - netstack - and wrapped it in slirp a-like envelope.

https://github.com/cloudflare/slirpnetstack

So this beast is basically that - on one hand you pass it packets (potentially from inside container) on the other hand it does connect/bind/etc to host OS.

This approach, in one form or another, is used by tailscale, podman and many other container runtimes.

Poke around in bocker[1]. For technical people, it makes the same points the article makes in a way that's very clear.

[1] "Docker implemented in around 100 lines of bash" https://github.com/p8952/bocker

I am a huge Docker fan! I would reiterate what several commenters have said already that it had a huge impact on my career. My question is relatively pedestrian compared to what others have asked already. I absolutely loved the "adjective-scientist/nerd" container names Docker generates. In fact, that is one of my most favorite things about Docker. How did that come about and who chose those combinations?

> I decided to make my own container runtime for fun [...] Unlike VMs, containers are just processes

Try to make your own VM runtime next, you might be surprised!

Interesting read! Thanks.

Though it would be better if it compared Dockers ways to say Podman, which does things a little different I think.

For others, like me:

Footgun:

A footgun is a programming or system design term for a feature, tool, or API that makes it very easy for a user or developer to shoot themselves in the foot—meaning it causes self-inflicted damage or severe errors while working exactly as intended.

I started hating Docker after seeing all the cargo culting on Reddit. Whenever someone asks about deploying an app on Linux, there's always at least one comment saying, Just use Docker. We finally ended up building our own deployment automation tool for our entire stack that doesn't use Docker at all.

That part about the threading was a huge source of issues back in the early days. Goroutines could hop threads and it's the threads namespace/cgroup configuration that gets inherited when forking..

One of the gnarliest issues I recall was where network namespaces could occasionally not be cleaned up correctly. So "randomly" (but more likely under certain loads) the networking of a reused thread would get jacked and Docker could no longer pull images.

I have recently discovered INCUS, very nice alternative to docker when a simple container is not sufficient https://linuxcontainers.org/incus/

A good talk on the tech behind containers.

Containers From Scratch • Liz Rice • GOTO 2018

https://www.youtube.com/watch?v=8fi7uSYlOdc

interesting read as a relatively less experienced developer.

I created Range (for fun). No daemon. No root.

A few cool tricks:

`range shell python:3.12` - opens a shell image without pulling the entire image.

`range shell ubuntu --commands` - you are in Ubuntu. Type any command Ubuntu doesn't have and it comes from the image that has it. On the fly.

`range run --network none ubuntu` - network-isolated sandbox

range run ebs://snap-... -- cat /etc/os-release - opens an EBS snapshot with no EC2 instance

Another cool trick:

`range run python:3.12 --mount hf://moonshotai/Kimi-K2-Instruct:/model -- du -sh /model` says 959G instantly :)

Check it out here - https://getrange.sh

Most importantly, Range doesn't need any daemon, any container runtime, any registry or any image conversion. One binary only. On a Mac it boots its own tiny Linux VM per session, and on Linux it uses KVM.

> And if you know where to look, you can even find that process in /proc.

This is very important for debugging! Not just that process, but the processes running in that Docker can be found in /proc.

Why this is useful is that you can run gdb outside of that docker and point it to a process in the docker, including setting the file system root to the docker one, so that it gdb reads the correct objects from the correct paths, to get the right symbols and addresses.

If you're old enough to remember some techniques that people used before docker (like chroot(2)-ing daemons for security), the "containers aren't real" observations are not really new info. When containers came on the scene it felt like an outgrowth or evolution of that technique, which was extremely crude and simple.

Docker question. Who prefers docker for local dev vs. run the thing directly. I usually prefer run the thing for debugging and hot reloading. Sure: build the containers and test them in CI. Dependencies like PG sure run in containers as they change less often and you save the install.

This only scratches the tip of the iceberg. People's minds are blown when they realize Linux has no concept of a container.

It's not a vm: It's all namespaces, baby! Unshare one or more of your mnt, pid, net, ipc, uts, user, cgroup, or time namespaces and you have a "container".

Oh man, what a fun thing to do! I may have to replace what used to be my fascination with programming languages and compilers with virtualization or containerization now.

This is a better way to understand it: we can think of a container as an isolated process. When deploying containers, having a clear understanding of the boundaries between containers and the host machine in terms of permissions, resources, and data is extremely helpful for troubleshooting issues.

I can't use Docker locally only because lack of enough space on my SSD.

Today, I would definitely choose rust over go. And I lament that so much container software uses go rather than rust. But back when docker started rust was a lot less mature and probably wouldn't have been a good choice.

Just use Podman

> Unlike metal servers, VMs are just operating systems being gaslit by a hypervisor. Your actual CPU is still executing those instructions! And if you know where to look, yyou can even read the other VMs’ memory in L1D

> Go is kind of a terrible language for a container runtime

I love Go and use it for everything I can but will absolutely concede this point. It's clearly possible to write the most successful container runtimes on earth in Go, but you absolutely have to fight the language/runtime sometimes. Since containers are based on processes, handing off a container spec to a minimal rust runtime seems like what we should all be doing by now. crun has just proven good enough that AFAIK no one has RIIR.

For as "terrible of a language" Go is alleged to be for a container runtime, its not like crun or youki are exactly faster in many meaningful ways unless you count churning containers running true and exiting.

And implying the system kernel still has to run the container kinda ignores kata containers and gvisor runtimes.

Docker does not need root.

Side note almost nothing about Docker is weird. This article chooses rather boring design decisions and picks on them.

> Next up: Databases are weird

> Did you know that databases are just a file?

oOoOoOoOo

Mostly joking. It's cool to see tech demystified, but I feel like a lot of things we use and take for granted are like this.

Honestly it took me so long to pick up containerisation because of how it's marketed. "Orchestrate a zillion fake computers, ultralight VMs, livestock not pets" big whoop.

"It's an idempotent language for OS level dependencies so you can be sure exactly what is running on your server"

Gimme gimme gimme!

Docker hasn't required root for 7 years, I wish people would stop repeating that. It uses a root daemon by default because it makes things easier. Also, Containers are only as fake as a "Desktop" in an OS, which isn't real either. It's a user, network & process namespace with a control bus and a bunch of other weird things. A Desktop is a Container with a GUI.

Other weird facts about OSes: a filesystem is a database, the Linux kernel is a monolithic program, "hidden" dot-files were a Unix bug that got carried forward, deleting a file doesn't remove the file from your machine, virtual memory is a lie, your computer can't run multiple programs at once (it's just running one at a time real fast), in 12 years most systems will crash due to one number getting too big, you technically never need to reboot because you can launch a different version of Linux while already running Linux, you can freeze a program to a file and then later resume running that same program in the same state (but nobody uses it), files aren't actually stored in directories, there's a maximum number of files (and size of file) on most filesystems (which you can use to lock up a machine, despite file quotas existing and nobody using them), you can lock up most systems by forking too many times / using too much RAM / accessing disk too much, there's a default number of open files and/or TCP ports which is artificially low and you can use to again lock up a machine, pipes are just two file descriptors connected to one memory buffer, an IP address is just one long number split up into bytes, etc.

One of my deeply unpopular opinions is that Docker only exists because Linux as a bad operating system that is incapable of deploying Docker. I have not once worked on a project where “we’ll solve this with Docker!” didn’t result in never ending Docker fixes and workarounds and hacks.

This is not a slight on Docker itself. It’s a slight on the conditions that induced Docker to exist.

One of the great selling points of one of my projects is “this is easy to run without needing Docker”.

[flagged]

>>Images are just folders that Docker puts somewhere and then calls chroot (or pivot_root) to make that folder your container process’s new root directory.

And folders are graphical representations of directories--sometimes, not always--and not directories themselves. When will the newbies get this?