Splunk Log Aggregation & Security Monitoring
A standalone Splunk Enterprise deployment built as centralized log aggregation and security monitoring for the Skybaer Labs environment: a Deployment Server pushing forwarder configuration out to Windows and RHEL hosts, feeding a hand-built security dashboard.
Architecture
One indexer, a deployment tier, and two very different forwarders
One box, two roles
splunk.skybaerlabs.net runs as both the Splunk Enterprise indexer (web UI on port 8000, receiving forwarder data on 9997) and the Deployment Server (port 8089) that pushes configuration out to forwarders, rather than standing up a separate deployment tier.
Two hosts, two kinds of logs
Universal Forwarders run on control.skybaerlabs.net (RHEL 10, shipping syslog and audit logs) and DC01.skybaerlabs.net (Windows Server 2025, shipping Security Event Logs), the lab’s existing control node and AD domain controller.
Deployment Server pattern, not hand-configured forwarders
Rather than configuring each Universal Forwarder by hand, serverclass.conf maps clients to deployment apps by hostname pattern: TA-windows-server to DC01, TA-windows-workstation to any future DESKTOP* machine, and TA-rhel-servers to any *.skybaerlabs.net RHEL host, with DC01 explicitly excluded from that last pattern. Adding a forwarder later means matching a hostname pattern, not writing a new config by hand.
What Each Deployment App Collects
Different log sources, same centralized push
TA-windows-server
Pulls a wide set of Windows Event Log channels from DC01: Security, System, Application, Setup, and Forwarded Events, plus the Applications and Services logs for AD Certificate Services, AD Domain Services, AD Web Services, Directory Service, DHCP Server, DNS Server, DFS Replication, Remote Desktop Services, and IIS, all indexed into main.
TA-rhel-servers
Monitors /var/log/messages, /var/log/secure, and /var/log/audit/audit.log on RHEL hosts: general system messages, SSH and authentication events, and the Linux audit framework.
Four Real Problems, Four Fixes
What actually broke getting data flowing end to end
Deployment Server unreachable on port 8089
firewall-cmd.Indexer refusing forwarder connections on 9997
[splunktcp:9997] stanza to inputs.conf on the indexer; nothing was listening for forwarder traffic until that existed.DC01’s inputs.conf never reached the forwarder
inputs.conf had to live under TA-windows-server’s local/ directory, not default/, for the Deployment Server to actually push it out.serverclass.conf stanza syntax
[serverClass:Name:app:AppName].The Windows Security Overview Dashboard
Five panels built from DC01’s Security Event Log data
Built in Splunk Web as the portfolio-facing piece of this project: KPI tiles up top, then statistics tables and a timeline underneath, all scoped to the last 24 hours and user-adjustable from the dashboard header.
Row 1: KPI tiles
Three single-value panels: total Security events from DC01 in the last 24 hours, a count of distinct Account_Name values generating events, and a count of distinct Source_Network_Address values.
Rows 2–4: statistics tables
Event Code Distribution (which security events fire most often: 4624, 4634, 4672, 5058, and so on), Top Users (which accounts, service and human, generate the most activity), and Top Source IPs (where that activity originates on the lab network).
Row 5: event timeline
A line chart of event volume by EventCode over the last 24 hours, the one panel that makes a spike or a pattern change visible at a glance rather than buried in a table.
index=main host=DC01 source="*Security*"
| stats count by EventCode
| sort - countindex=main host=DC01 source="*Security*"
| timechart count by EventCodeVerifying the Data Flow
Confirming forwarders phoned home, not just that install finished
control.skybaerlabs.net, confirmed
Collecting /var/log/messages, /var/log/secure, and /var/log/audit/audit.log, with over 2,788 events indexed and verified flowing.
DC01, confirmed
Windows Security Event Log channels confirmed flowing once TA-windows-server actually reached the forwarder, after tracking down the local/ vs default/ directory issue above.
Watching the Deployment Server itself
Verified through the Deployment Server console that forwarders were phoning home over HTTPS and that apps were deploying according to the serverclass.conf hostname patterns, rather than assuming a clean forwarder install meant data was actually arriving.
What’s Next
Rolling the rest of the lab onto this pipeline
Rolling out further
Extending Universal Forwarders to the Windows workstation fleet (the DESKTOP* naming pattern the app already matches) and the remaining RHEL nodes (node1–4, CRC, Foreman), to bring the whole lab onto one dashboard.
Alerting, not just dashboards
Building Splunk alerts for failed logins and account lockouts, plus summary indexing for performance as the data volume grows.
Documenting the operational side
Backup, index rotation, and user management procedures, so the deployment reads as something actually run day to day, not just stood up once.