Air-gapped Kubernetes embedded in trucks, all over the world

Where they started

Hand-configured servers around a rough Docker Swarm cluster. It met none of the client’s expectations, and the team had to build much of what a modern orchestrator provides out of the box: service restarts, scaling, configuration management.

We came in as a subcontractor to the client’s software partner.

The challenge

Running Kubernetes without internet access, for installation and day-to-day operation alike, at a time when neither the tooling nor the documentation covered this case:

• installing without access to public repositories;
• hosting a local image registry, down to the system images everyone forgets, such as the pause image behind every pod;
• updating with no connection;
• all for a team that was new to container orchestration.

The critical moment

Kubespray had next to no support for air-gapped installs, and errors surfaced late, sometimes deep into testing. Every fix uncovered a new problem, to the point where delivering on time looked out of reach.

We changed our method: a fully automated platform build, failing fast on the developer’s machine, and a very thorough test suite that blocked regressions and drastically shortened the feedback loop.

What we delivered

• A tailor-made, 100% offline Kubernetes cluster: Kubespray, containerd, local-path-provisioner for storage, an embedded Docker registry.
• A fully declarative platform: every deployment is an Ansible playbook, versioned in Git and applied automatically. The client can rebuild and evolve it on its own.
• Scaling: without touching the client’s proprietary software, its application now handles a much heavier load.

Outcome

A self-contained platform, deployed in trucks all over the world. The client took ownership of it and keeps iterating on it, years later. They would have liked us to stay.

“We would never have made it without you.”