Implementing Infrastructure Auditing and Logging with Auditd

Auditd system auditing represents the primary mechanism for subsystem tracking within the Linux kernel; it provides a transparent and robust method for monitoring security-relevant events. In complex technical environments such as high-density data centers, water treatment control systems, or cloud-native microservices, maintaining an immutable record of system calls is critical. The “Problem-Solution” context focuses on the inherent lack of visibility into privileged user actions and kernel-level modifications that evade standard application logs. Without Auditd, an administrator cannot definitively prove who modified a configuration file or executed a specific binary at a specific microsecond. Auditd solves this by intercepting system calls directly at the kernel interface, ensuring that even if a userspace process is compromised, the audit trail remains intact. This implementation ensures regulatory compliance and provides a forensic foundation for post-incident analysis; it effectively closes the gap between standard logging and true infrastructure observability.

TECHNICAL SPECIFICATIONS

THE CONFIGURATION PROTOCOL

Environment Prerequisites:

Successful deployment requires a Linux distribution with a modular or monolithic kernel supporting CONFIG_AUDIT. The auditor must possess root or sudo privileges. All dependencies, specifically auditd and audispd-plugins, must be verified against current repository versions. In high-concurrency environments, ensure that the underlying storage subsystem can handle the synchronous write throughput required by the logging daemon to avoid I/O bottlenecks.

Section A: Implementation Logic:

The engineering logic of Auditd relies on the encapsulation of system calls. When a process initiates a syscall (e.g., open, execve, chmod), the kernel’s kauditd component evaluates the call against a set of active rules. If a match occurs, a record is generated and sent via a netlink socket to the user-space auditd daemon. This architecture minimizes latency by performing the initial filtering within the kernel while delegating the disk-intensive logging to user-space. This design ensures an idempotent auditing state; the rules are applied consistently across every process, regardless of the user’s shell environment or application-level logging capabilities.

Step-By-Step Execution

1. Installation of the Audit Subsystem

Execute apt-get install auditd audispd-plugins or yum install audit.
System Note: This command installs the user-space daemon and the plugins necessary for multiplexing audit events to other services. The installer registers auditd as a system service via systemctl, which will manage the daemon’s lifecycle.

2. Verify Kernel Status

Run systemctl status auditd and auditctl -s.
System Note: The auditctl -s command queries the kernel to report the current status of the audit subsystem, including the number of lost records and the current backlog limit. This step confirms the netlink socket is active and communicating with the kernel.

3. Configure Global Daemon Parameters

Edit the auditd configuration file located at /etc/audit/auditd.conf. Set the max_log_file to 100 and max_log_file_action to rotate.
System Note: Modifying /etc/audit/auditd.conf alters how the user-space daemon handles the incoming payload from the kernel. Setting rotation policies prevents disk exhaustion, which could otherwise lead to a system-wide denial of service or packet-loss of log data.

4. Implement File Integrity Watches

Add a rule to monitor the SSH configuration: auditctl -w /etc/ssh/sshd_config -p wa -k sshd_changes.
System Note: The -w flag sets a watch on a specific file path. The -p wa attribute tracks “write” and “attribute” changes, while the -k flag assigns a unique key to the event. This allows for rapid searching using ausearch -k sshd_changes.

5. Define Persistent System Call Rules

Open /etc/audit/rules.d/audit.rules and append -a always,exit -F arch=b64 -S execve -k execution_trace.
System Note: This rule instructs the kernel to log every execution of a binary on a 64-bit architecture. By targeting the execve system call, the auditor captures the full command line and environment of every process started on the system.

6. Loading and Locking the Configuration

Apply the rules using augenrules –load, then verify with auditctl -l.
System Note: The augenrules script compiles individual rule files into a single audit.rules file and loads them into the kernel. This ensures the configuration is idempotent and survives system reboots.

Section B: Dependency Fault-Lines:

The most significant bottleneck in Auditd performance is the disk subsystem. If the logging throughput exceeds the storage write speed, the kernel’s backlog buffer will fill. Once the backlog_limit is reached, the kernel’s response depends on the panic or halt setting; it may drop events or crash the system to maintain the audit trail integrity. Furthermore, conflicts with Rsyslog or systemd-journald can occur if multiple daemons attempt to parse the same netlink stream without the audispd multiplexer.

THE TROUBLESHOOTING MATRIX

Section C: Logs & Debugging:

All raw audit data is stored in /var/log/audit/audit.log. This file is not standard text; it contains encoded hex strings for certain fields to prevent log injection attacks.

Error: “audit_backlog_limit exceeded”: This indicates the system is generating events faster than auditd can write them to disk. Increase the buffer using auditctl -b 8192.

Error: “audit: audit_lost=NNN”: Events are being discarded. Inspect disk latency* and check if the dispatcher is overloaded.

Visual Cues: Use aureport -s to generate a summary report. If the “Config Changes” count is unexpectedly high, it indicates an unauthorized attempt to modify audit rules.

Verification: Run ausearch -m SYSCALL -ts today to view all system call events triggered in the last 24 hours. Cross-reference the auid (audit user ID) with /etc/passwd to identify the actual user who initiated the action, even if they used sudo.

OPTIMIZATION & HARDENING

Performance Tuning:
To handle high concurrency, increase the num_logs variable in auditd.conf to allow for more granular rotation. Use the priority_boost setting to give the auditd process a higher CPU scheduling priority, reducing the overhead caused by context switching. For systems with high thermal-inertia in their physical components, ensure that cooling is sufficient for the increased CPU load during heavy auditing bursts.

Security Hardening:
Set the file permissions of /var/log/audit/ to 0700 and ensure the owner is root. The most critical hardening step is adding -e 2 as the final line in your rules file. This makes the audit configuration immutable; rules cannot be changed or deleted until the system is rebooted. This prevents an attacker from disabling the audit trail after gaining initial access.

Scaling Logic:
In distributed architectures, local log storage is insufficient. Use the audisp-remote plugin to forward events to a centralized log aggregator. This maintains the signal-attenuation required for clean data delivery over long-distance network segments. Implement a load balancer if multiple aggregators are used to manage the incoming payload from thousands of nodes.

THE ADMIN DESK

Q: How do I find who deleted a specific file?
A: Use ausearch -f /path/to/file -m UNLINK_RENAME. The auid field in the resulting log entry identifies the original user account that executed the deletion command, regardless of current privilege level.

Q: Why are my audit rules disappearing after a reboot?
A: Rules applied via auditctl are reside in volatile kernel memory. You must add them to /etc/audit/rules.d/audit.rules to ensure they are reloaded by augenrules or the auditd service during the boot sequence.

Q: Can Auditd monitor network connections?
A: Yes, by tracking the connect and accept system calls. Use -a always,exit -F arch=b64 -S connect -k network_out to log all outbound socket connections initiated by local processes.

Q: Does Auditd impact system performance significantly?
A: Auditd has minimal impact (usually under 5 percent CPU) when collecting file watches. However, auditing high-frequency system calls like read or write can cause significant latency and should be avoided in production environments.

Q: How can I see a summary of all failed login attempts?
A: Execute aureport -au. This command parses the audit log specifically for authentication events and provides a categorized table of successful and failed attempts, including the terminal and remote host used.