Kubernetes on Oxide: How customer needs shaped our integrations

Disclaimer: I'm totally biased here.

I talked with a colleague from Oxide in 2024 about your Kubernetes story and he said back then "not yet but soon-ish". Seems like soon-ish is now :)

We said we'd talk again when that happens but he's since left Oxide. If you (or well...your customers) are interested in a Kubernetes native data platform 100% open source we'd be very happy to talk about how that could work easily. As it's "just" Kubernetes it should be trivial but we'd be happy to test and add you to our list: https://docs.stackable.tech/home/stable/kubernetes/#supporte...

The offer stands. If you're interested, my mail is in my HN profile. https://stackable.tech

I'm interested to see how the `oxide-cloud-controller-manager` is being built for "modern" Kubernetes and if it leads to any signify difference compared to CCMs that originated in-tree. Given the way Oxide engineer their solutions this could be really interesting.

FYI I've got `karpenter-provider-oxide` on my bingo card...

I would absolutely kill for them to open source their documentation system.

I am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal?

At first look, it feels like oxide is equivalent to proxmox or some virtualization tool, may be it uses qemu stuff underneath.

Just curious.

The reason I am asking this is that, we have lot onprem scenarios in our business. We are tightly coupled with k8s, to solve this we started building an internal project that is kubernetes API compatible [1] but runs containerd or WASM or our platform natively.

Just curious how oxides work in this scenario

[1] https://github.com/debarshibasak/superkube

I have seldom wanted anything as much as I want an Oxide rack at home. Maybe in 40 years we'll start to see the first ones show up in surplus auctions...

Kubernetes is where Oxide, from my armchair, doesn't yet quite match the public cloud.

On AWS EKS Fargate, each pod runs in its own dedicated VM. With Oxide each k8s node is a VM, so you still need something like Talos.

On networking it looks like it is getting closer. Where you can have external subnet give pod routed IPs without overlay. But the gap, as marked by the article, is also the load balancing.

It would be nice if Kubernetes were a native feature out-of-the-box. Also integrated within the existing user/access control.

Love to see the CAPOx provider and buy in to Cluster API

So you’d be attaching a new volume to the running worker VM for each PVC? This seems a little odd to me. Could you attach a single large volume, and do path-based provisioning on that?

How are folks managing stateful workloads like databases on Oxide?