Red Hat Lightspeed Integration
Red Hat Lightspeed and the Hybrid Cloud Console, integrated into the Skybaer Labs RHEL environment for centralized inventory, advisory and vulnerability visibility, and remote remediation management, kept deliberately separate from the lab’s existing Foreman setup rather than folded into it.
Two Architectures, One Platform
RHEL 9’s rhcd and RHEL 10’s Yggdrasil turned out to be genuinely different
Lightspeed went onto four RHEL 10.2 hosts (control, rhel10-crc, node1, node2) and the lab’s RHEL 9.8 Foreman host. The two major versions turned out to use different remote-management stacks under the hood.
RHEL 10.2: Yggdrasil-based
Built on rhc, insights-client, and rhc-worker-playbook, with yggdrasil.service connecting outbound to Red Hat over HTTPS/WebSocket. Startup log lines like “failed to publish message: not Connected” are expected during initialization, not failures.
The playbook worker is D-Bus activated
com.redhat.Yggdrasil1.Worker1.rhc_worker_playbook.service starts on demand via D-Bus rather than being enabled directly. Seeing it sit inactive while idle, shown as “(activatable)” in busctl, is the expected state, not a problem to chase.
RHEL 9.8: the older rhcd daemon
The Foreman host has no standalone Yggdrasil package. Instead rhcd runs with its own child workers, rhc-package-manager-worker and rhc-worker-playbook.worker, under a simpler single-daemon model.
The lesson
RHEL 9 and RHEL 10 shouldn’t be treated as having identical Lightspeed remote-execution architectures. The differences are substantial enough to change configuration steps, troubleshooting paths, and what “working” actually looks like on each version.
The Canonical-Facts Blocker
Every prerequisite was green, and remediation still wouldn’t run
RHEL 10 stuck at “Not Configured”
rhc-worker-playbook was installed, yet the Hybrid Cloud Console kept reporting RHEL 10 systems as Connection status: Not Configured, which blocked remote remediation entirely./var/lib/yggdrasil/canonical-facts.json existed but held only an empty object. Yggdrasil had no canonical system identity information for the remote-management stack to use.rhc-canonical-facts.service populated the file with real identity data: insights_id, machine_id, bios_uuid, subscription_manager_id, IP and MAC addresses, and the FQDN. Enabling rhc-canonical-facts.timer kept it refreshed automatically, and restarting yggdrasil.service loaded the new facts. rhc status then showed Remote Management as active end to end.RBAC and a Missing Menu
The right roles, and one documented screen that never showed up
Two predefined roles, not a custom one
Remote remediation required the predefined Remediations administrator role plus RHC administrator, assigned directly to the account running remediations rather than anything custom-built.
A documented screen that wasn’t there
Some Red Hat documentation references Inventory → System Configuration → Remote Host Configuration, but that menu never appeared in the Skybaer Labs console despite the account holding the right admin roles. Current release notes suggest Remediations no longer depends on the older RHC Manager permission setting for systems connected through rhc, so the missing menu wasn’t pursued further.
Remediation Workflow
From a flagged advisory to a verified fix
Lightspeed flags an advisory or vulnerability, a remediation plan gets built against the affected systems, target connection state is checked, the plan executes, and Execution History records what happened. The part that isn’t obvious is what happens after.
The console doesn’t update itself
Even after a successful remediation, Content and Vulnerability findings could stay visible in the console until new system data was uploaded. Checking locally with dnf updateinfo list and forcing a fresh run of insights-client confirmed the remediated advisories were actually gone, with only genuinely outstanding items left behind.
Collection runs on its own schedule
insights-client.timer collects roughly once a day on a randomized schedule. No changes were made to that interval in the lab; the post-remediation refresh above was run manually to confirm results immediately rather than waiting on the timer.
The Disconnected-System Finding
A behavior pattern found once, then deliberately reproduced
The most significant result of this integration wasn’t getting remediation working. It was what happened when a target system was offline during a run, a pattern that showed up once by accident and was then reproduced on purpose under controlled conditions.
Four systems, one powered off
A controlled two-node reproduction
Execution Scope vs. Plan Scope
Why “Succeeded” doesn’t mean what it sounds like it means
Two different definitions of done
A remediation plan’s system count reflects what an administrator intended to remediate. Its success criteria is based only on the systems that actually took part in that one execution. A plan built for two systems can report Succeeded off the strength of just one.
Nothing flags the gap afterward
A later look at the console can show both systems Connected and the remediation marked Succeeded, with no visible indicator that one of them missed that particular run. Spotting the discrepancy means comparing a plan’s intended membership against the specific systems listed in that execution’s own history entry.
No dedicated notification for it either
Hybrid Cloud Console notification events cover things like task started, task completed, and job failed, but Remediations has no notification category of its own, so there’s no built-in alert for an omitted system either.
Escalating It
A question for Red Hat Support, and what this finding isn’t
The question for Red Hat Support
“Is it expected behavior for Red Hat Lightspeed Remediations to omit a disconnected system from an execution and report the overall remediation as Succeeded, even though that system remains part of the remediation plan and never received it?” Follow-up questions cover where the audit record for an omitted system would live, if this filtering is intentional, and why execution was allowed to start at all with one target visibly disconnected. The two-node reproduction above, backed by screenshots at each stage, is the supporting evidence.
What this isn’t
Some documentation suggests an unavailable target should surface as Unreachable from Ansible. That isn’t what happened here. The verified observation is narrower: node2 stayed part of the plan but never appeared in the execution record at all, which is a different and more surprising behavior than a simple reachability failure.
Current State and What’s Next
Operational, with one open question still outstanding
Where things stand
Remote remediation works on both RHEL 10.2 (Yggdrasil) and RHEL 9.8 Foreman (rhcd), RBAC is configured, and every currently targeted, powered-on lab system has been patched through the integration.
No built-in rollback
Lightspeed Remediations has no native rollback feature. A possible next step is pairing it with the lab’s existing VMware/Terraform automation: snapshot before a remediation, run it, validate, then drop the snapshot once the result checks out. Not yet built.
Still open
The disconnected-system behavior remains an open question, flagged for a Red Hat Support ticket rather than treated as solved.