Lab · kind + podman + Kamaji

Hosted Control Planes on a laptop

A management cluster runs the control planes of other clusters as pods, and separate workers talk only to their own API through a published endpoint. It has the same shape as OpenShift HCP (ROSA HCP), built from upstream pieces so you can break it and understand it.

1 management cluster 2 hosted clusters 1 worker per NodePool ~3.3 GB of RAM in total
Architecture

What runs where, and how the pieces talk

Management cluster and hosted control planesData plane (NodePools)
Your machine hkubectl tenant-a … 127.0.0.1:31441 → NodePort MANAGEMENT CLUSTER · kind hcp-mgmt hcp-mgmt-control-plane Management API NodePorts 31441–31443 Kamaji ≈ HyperShift operator MetalLB IPs 10.89.0.200–250 ns clusters-tenant-a Control plane pod (Deployment) kube-apiserver controller-manager scheduler konnectivity-server dedicated etcd StatefulSet + PVC Svc LoadBalancer 10.89.0.200 runs on hcp-mgmt-worker2 · v1.35 ns clusters-tenant-b Control plane pod (Deployment) kube-apiserver controller-manager scheduler konnectivity-server dedicated etcd StatefulSet + PVC Svc LoadBalancer 10.89.0.201 runs on hcp-mgmt-worker · v1.36 DATA PLANE outside the management cluster podman kind network 10.89.0.0/24 tenant-a-worker-1 NodePool tenant-a-workers kubelet kube-proxy konn. agent kindnet CoreDNS your apps kindest/node container · kubelet v1.35 tenant-b-worker-1 NodePool tenant-b-workers kubelet kube-proxy konn. agent kindnet CoreDNS your apps kindest/node container · kubelet v1.36 kubelet → :6443 tunnel :8132 logs · exec kubelet → :6443 tunnel :8132
Each hosted cluster lives in its own clusters-<name> namespace on the management cluster, with its own etcd. The workers are not management cluster nodes: they join the tenant's API through the LoadBalancer IP. The konnectivity-agent opens the tunnel from the worker, and kubectl logs, exec and port-forward travel back down that same tunnel. From your machine, each tenant's API is published on a fixed NodePort.
Mapping

What you are looking at, translated to HCP

OpenShift HCP / HyperShiftIn this lab
Management clusterkind cluster hcp-mgmt
Nodes for control planes (label hypershift.openshift.io/control-plane)hcp-mgmt-worker and hcp-mgmt-worker2, with the same label
HyperShift operatorKamaji, in kamaji-system
HostedCluster + HostedControlPlaneTenantControlPlane (kubectl get tcp -A)
Namespace clusters-<name>Namespace clusters-<name>
etcd por hosted clusterdedicated etcd por tenant (chart kamaji-etcd)
API publishing (NLB / Route)LoadBalancer Service via MetalLB
Konnectivitykonnectivity-server in the control plane pod, agent as a DaemonSet in the tenant
NodePool04-nodepool.sh: kindest/node containers
Ignition / node bootstrapkubeadm join with a 15-minute token
Control plane upgrade (spec.release)spec.kubernetes.version on the TCP
NodePool upgrade (upgradeType: Replace)Rolling replacement, one node at a time
OVN-Kuberneteskindnet
Upgrades

The control plane and the workers upgrade separately

Same as HCP: first you bump the control plane version, which only restarts pods on the management cluster. The workers stay on the previous version until you upgrade their NodePool.

Start

control plane
v1.35.8
workers
v1.35.8

1 · Control plane

control plane
v1.36.4
workers
v1.35.8
kubectl patch tcp … version v1.36.4

2 · NodePool

control plane
v1.36.4
workers
v1.36.4
04-nodepool.sh tenant-b 1 v1.36.4

Rejected

control plane
v1.35.8
workers
v1.36.4
workers newer than the control plane break the version skew policy
Get started

Up and running in about 5 minutes

Runs on macOS (podman machine) and on Fedora, CentOS Stream or RHEL hosts with 6 CPUs and 10–12 GB of RAM. The scripts download kind, helm and kubectl into the repo and use their own kubeconfigs, so they never touch ~/.kube/config.

git clone https://github.com/mcipamo/hcp-kind-lab.git && cd hcp-kind-lab

# Linux (Fedora / CentOS Stream / RHEL), once:
./scripts/00-linux-host-prep.sh
# macOS, once, if the podman VM is too small:
podman machine stop && podman machine set --cpus 6 --memory 12288 && podman machine start

./scripts/01-mgmt-cluster.sh               # management cluster
./scripts/02-addons.sh                     # cert-manager, MetalLB, Kamaji
./scripts/03-hosted-cluster.sh tenant-a 1  # hosted cluster in slot 1 → 127.0.0.1:31441
./scripts/04-nodepool.sh tenant-a 1        # one worker in its data plane
./scripts/status.sh                        # overview

source env.sh
kubectl get tcp -A                         # control planes, seen from the management cluster
hkubectl tenant-a get nodes                # inside the hosted cluster

./scripts/stop.sh                          # pause for the day
./scripts/start.sh                         # resume, ~3 minutes
Practice

Exercises

  1. The tenant cannot see its control plane. hkubectl tenant-a get pods -n kube-system shows no apiserver and no etcd.
  2. No konnectivity, no logs. Delete the konnectivity-agent DaemonSet and try logs: No agent available. Kamaji brings it back in about 10 seconds.
  3. The control plane goes down, the apps keep running. Scale the tenant's Deployment to 0 and check crictl ps on the worker.
  4. Control plane HA. Set replicas to 2 and watch them spread across both management nodes.
  5. Two-phase upgrade. The TCP first, then the NodePool.
  6. Scale the NodePool. 04-nodepool.sh tenant-a 3 and back to 1, drain included.
  7. Lose a management cluster node. podman stop hcp-mgmt-worker2: which control plane goes down? What about its etcd on a local PVC?
Keep in mind

Differences from real HCP

  • No OpenShift components: no openshift-apiserver, oauth, CVO or Routes. It is upstream Kubernetes with the same topology.
  • HCP runs each component as its own Deployment. Kamaji packs apiserver, controller-manager, scheduler and konnectivity-server into one pod with 4 containers.
  • Workers are not in another account or VPC: they share the kind network with the management cluster.
  • There is no NodePool CRD: a script plays the controller.
What broke while building it

Three problems you will run into

kube-proxy: nf_conntrack_max: permission denied

kube-proxy rewrites the value whenever it is not exactly 32768 × CPUs, and inside a container it is not allowed to. 01-mgmt-cluster.sh sets that exact value on the container host. Re-run it after restarting the VM or changing its CPUs.

Pods stuck in ContainerCreating with flannel

flannel delegates to the bridge CNI plugin, and kindest/node images do not ship it. That is why the lab uses kindnet, which comes inside the image itself.

logs and exec: No agent available

The konnectivity-agent is not running, almost always because pod networking is broken. Since kubectl logs does not work either, debug from the node: podman exec <worker> crictl logs ….