Skybaer Labs logo
SKYBÆR LABS
Technical Reference

CRC + AWX Rebuild Runbook

The full procedure for rebuilding CRC/OpenShift and AWX from scratch, plus every gotcha hit along the way, written so a future rebuild doesn’t require re-deriving any of it. Referenced from the CRC Crash Recovery write-up.

View on GitHub

When to Use This

  • The CRC VM fails to boot or hangs after a host crash, and a plain crc stop / crc start doesn’t fix it
  • Intentionally resizing CRC’s memory, CPU, or disk allocation (a crc config set change only takes effect after crc delete and crc start)
  • Any other situation that requires crc delete

1. Confirm the CRC VM Is Actually Broken

Crash scenario only

  • Check /var/log/libvirt/qemu/crc.log on rhel10-crc for a hung, silent boot. The confirmed pattern here is the boot going silent roughly 40 seconds in.
  • Try crc stop and crc start first. If that resolves it, stop here. No rebuild needed.
  • If it’s still hung, move on to the full rebuild below.

2. Rebuild CRC

This wipes and recreates the entire cluster: a new SSH keypair, a new kubeadmin password, and all in-cluster state gone. Sizing persists across a plain crc start unless it’s explicitly changed via crc config set first.

shell 
crc stop
crc delete
# only if intentionally changing sizing:
crc config set memory <MB>
crc config set cpus <N>
crc start

3. Redeploy the AWX Operator

Rebuilding CRC brings OpenShift itself back, but not AWX. It has to be redeployed into the new cluster from scratch.

shell 
cd ~/awx-operator
git fetch --tags   # pull the latest operator release
make deploy

kube-rbac-proxy sidecar ImagePullBackOff

Problem
gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0 no longer resolves. Google Container Registry has been migrating images over to registry.k8s.io, and this one had already moved.
Fix
oc set image deployment/awx-operator-controller-manager kube-rbac-proxy=registry.k8s.io/kubebuilder/kube-rbac-proxy:v0.15.0 -n awx

AWX custom resource needs an explicit Route

Problem
This operator version does not default to creating an OpenShift Route on its own. service_type: ClusterIP is fine as the default; no change needed there.
Fix
Set ingress_type: Route explicitly in the AWX custom resource.

4. Recreate AWX Database Objects

Everything below lived in the deleted cluster’s database

All of this has to be rebuilt from scratch after every crc delete, since none of it lives outside the cluster itself.

Execution Environment

Administration → Execution Environments → Add. Image: ghcr.io/jayberryman/skybaer-awx-ee-custom:<current tag>, a public image so no credential is needed. Pull policy Always is worth keeping while still iterating on the image. Assign it to each Job Template that needs it: anything using password_hash() needs passlib, anything using community.windows.win_lineinfile needs that collection.

Credentials

A Source Control credential (deploy key) for the private lab automation repo, and a Machine credential for SSH. Linux hosts authenticate as root; the Windows host needs ansible_user=jayberryman set as a host var override, since Windows has no root account. On the classic UI’s Create Project form, the Source Control Credential field is a lookup widget. Typing the credential name directly into it sets a plain string instead of a real reference, and silently fails validation with no visible error. It has to be selected through the picker.

Project and Inventory

The Project points at the private lab automation repo using the Source Control credential above. The Inventory has a Source Control-based Inventory Source pointing at the inventory file already committed there. Project sync and Inventory Source sync are two separate operations: syncing the Project only refreshes AWX’s checked-out copy of the repo, it doesn’t reparse anything into Groups and Hosts. The Inventory Source itself has to be synced separately before new groups or hosts show up.

Job Templates and Teams

Every Job Template currently in use has to be recreated by hand: Lab Shutdown (with its multi-choice Survey), Apache Install, Password Policy Enforcement, User Account Creation (Survey: target_group multi-choice, target_user_ID text, target_comment text, target_temp_passwd Password type with a 14-character minimum), and anything else in use at the time. The Team and its scoped, execute-only permissions on the relevant Job Templates need recreating too.

5. Inventory Variable Gotcha

ansible_shell_type

A host var that wouldn’t stick

Problem
Setting ansible_shell_type via group_vars/host_vars files placed alongside the SCM-sourced inventory doesn’t reliably survive AWX’s sync. Matches the pattern described in upstream ansible/awx GitHub issues #15438 and #12829.
Fix
Set it directly inside the flat inventory file itself, using INI-style = syntax in a [windows:vars]-style section, not YAML : syntax and not a separate group_vars/host_vars file.

6. CRC Node Clock Drift

Still open, no permanent fix yet

The lab runs two layers of virtualization: the physical host, the rhel10-crc VM, and CRC’s own inner VM. The inner VM doesn’t reliably resync its clock when the physical host wakes from sleep, the way the outer VM does through VMware Tools.

Symptom & diagnosis

AWX’s UI shows stale or wrong timestamps even after a fresh login, which rules out browser cache. Diagnose with oc debug node/<nodename> (get the node name via oc get nodes), then chroot /host, then date. crc ssh doesn’t exist in current crc CLI versions, and crc console only opens the OpenShift web console URL. Neither gets you a shell.

Stopgap vs. real fix

A crc stop / crc start corrects the clock immediately, but that’s a stopgap, not a fix. The planned permanent fix is a MachineConfig object shipping a proper chrony config via Ignition to the CRC node’s MachineConfigPool, chosen deliberately for OpenShift admin practice and because it preserves the host’s power-saving settings instead of requiring sleep to be disabled. Not implemented yet.

7. After Rebuild

  • Confirm job templates run clean, starting with something low-stakes (Lab Shutdown or Apache Install) before trusting anything more complex
  • Take a fresh VMware snapshot of rhel10-crc once the new build is confirmed stable. An old snapshot taken before a rebuild restores the previous config if ever reverted to, which is an easy thing to forget
  • Note whatever CRC memory/CPU sizing was used, and check whether rhel10-crc’s own VMware-assigned memory needs a matching downward adjustment. Host-level RAM oversubscription is a known constraint in this lab, and CRC generally needs to run in relative isolation or alongside only light guests
Questions about the lab?
Contact ·
← Back to CRC Crash Recovery