CRC Crash Recovery
A physical host crash left a nested OpenShift cluster hung mid-boot. Here’s how I confirmed the hang, rebuilt the cluster, fixed two real deployment bugs along the way, and turned the recovery into a runbook so it isn’t tribal knowledge next time.
What Went Wrong
A silent hang, not a clean failure
In September 2026, a crash on the physical Windows host left the CRC (CodeReady Containers) VM’s boot process genuinely hung. It wasn’t erroring, just silent, roughly 40 seconds into boot.
Not a clean crash
Nothing on the OpenShift side reported an error. The CRC VM simply stopped progressing partway through boot, with no console output past that point and no obvious failure to chase.
The easy fix didn’t work
A misbehaving CRC instance usually recovers with a plain crc stop followed by crc start. This time it came back up and hung in the exact same place.
Diagnosis
Confirming the hang before touching anything else
Rather than guessing, I checked libvirt’s own console capture for the CRC VM at
/var/log/libvirt/qemu/crc.log. It showed the boot process had genuinely
stalled, confirming this was an actual hung guest and not a display or reporting
glitch, before committing to a rebuild.
Recovery Process, Start to Finish
Rebuilding the cluster, then rebuilding what ran on top of it
With the hang confirmed, the fix was to rebuild rather than keep troubleshooting a hung
VM: crc stop, then crc delete, then a fresh crc start,
using the same persisted configuration (4 CPU / 16GB RAM / 50GB disk) so no flags were
needed. That brings OpenShift itself back, but AWX doesn’t come back with it. It
has to be redeployed into the new cluster from scratch: git fetch --tags in
~/awx-operator to pick up the latest release, then make deploy,
which pulled in AWX Operator 2.19.1 at the time. Two things broke on that redeploy:
The operator’s sidecar wouldn’t pull
awx-operator-controller-manager stuck with its kube-rbac-proxy sidecar in ImagePullBackOff, blocking the whole operator from coming up.gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0) no longer resolves. Google Container Registry has been migrating images over to registry.k8s.io, and this one had already moved.oc set image deployment/awx-operator-controller-manager kube-rbac-proxy=registry.k8s.io/kubebuilder/kube-rbac-proxy:v0.15.0 -n awx repointed it at the new location and the operator came up clean.No Route, no way in
service_type: ClusterIP is the default and was fine on its own, but ingress_type: Route has to be set explicitly in the AWX custom resource on OpenShift, or nothing gets exposed externally.ingress_type: Route to the AWX custom resource and reapplied it. The Route appeared and AWX was reachable again.Once those two were resolved, AWX rebuilt successfully, but everything that had lived in the old cluster’s database was gone with it: the Project, both Credentials (Source Control and Machine), the Inventory and its Inventory Source, every Job Template, the Team/RBAC setup, and the custom Execution Environment registration. None of that lives outside the cluster, so all of it has to be recreated from scratch on every rebuild, not just the automation itself.
Safeguards Added Afterward
Turning a one-off recovery into a repeatable process
VMware snapshots as a floor
Added VMware-level snapshots of the lab VMs as a baseline safety net, starting with control.lab, so a future hang has a known-good rollback point instead of a rebuild from zero.
A runbook, not tribal knowledge
The same rebuild sequence had to be repeated again later for an unrelated reason (reducing CRC’s memory allocation to relieve pressure on the host). That second rebuild is what actually pushed me to write it down: the full rebuild sequence, every object that has to be recreated, and every gotcha along the way, so the next rebuild doesn’t require re-deriving any of it. Read the full runbook →