Examining event and endpoint logs is the first step towards building comprehensive customized rulesets. 

Many information security leaders have significant deployments on open-source operating systems based on the Linux kernel, and for good reason. Linux distributions like Debian and Ubuntu have a reputation for visibility and security at a price that's impossible to beat – they're 100% free.  

Even enterprise-ready subscription-based distributions like Rhel typically cost much less than a proprietary solution with similar capabilities. But Linux itself isn't free from security vulnerabilities, and security leaders need to take concrete steps to protect Linux systems from cyberattacks. 

Understand Your Security Posture Through Event and Endpoint Logs 

Linux stores a timeline of events involving its server, kernel, and applications. These logs record activities like logins, password changes, file modifications, and more. Logs contain raw data describing what happened to a specific system at a specific time. If the data surrounding these events indicate a potential security breach, they become incidents that require investigation. 

The strengths and weaknesses of your organization's security posture are reflected in this log data. The problem is that most information security teams generate more log data than they can efficiently process. Security information and event management (SIEM) platforms help security teams prioritize and interpret log data more effectively. 

Most Linux log directories fit into one of four primary categories: 

• Application logs. These logs refer specifically to non-system applications that generate log data. They store information about commands executed by running applications, like error messages and browser identification strings. MySQL database logs and Apache HTTP server logs are examples of application logs that can play a role in incident investigation. 
• Event logs. These logs describe specific system and application events. Linux event logs help analysts identify attempts to access systems, data, applications, files, or networks and provide data on the connection used to perform the task. 

• Service logs. Linux kernel messages are included in the service log. Platform diagnostics and some other services and productivity rules write specifically to Linux's service logs, keeping them separate from other types of log data. 
• System logs. Linux needs these logs in order to work. System logs contain the most significant information about system activities, including daemon logs, authentication logs, and system boot information. 

How to Access Log Data in Linux 

Linux allows users to view log data directly through its command line interface. There is more than one way to view log data through the command line: 

• Access the directory cd/var/log, which stores different log types in subfolders nested in the main log folder. 

• Use the dmesg command to call up kernel-related logs from the system's kernel ring buffer. This presents system log data in chronological order. 
• Use the tail command to display the last lines written to a specific log file. This allows analysts to follow changes to log files as they occur. 
• Use the Linux Audit System (Auditd) to centralize log monitoring and investigation. 

Of these options, Auditd presents the most modern approach for handling security incident investigations. It is native to the Linux kernel and provides visibility into three distinct areas – system calls, file access, and pre-configured auditable events. 

These three event categories enable analysts to audit many different types of activities in Linux, including authentications, abnormal application terminations, program executions, and more. When the audit rules are triggered, the Linux Audit System outputs a record describing the activity. 

This record can provide ample data for investigating security incidents and identifying attacks. Optimizing this capability requires writing custom rules to generate more descriptive logs than the system default. The Linux Auditing System supports more than 200 audit event fields, enabling analysts to describe log events with precision. 

The audit package contains pre-configured rule files based on four different certification standards. Depending on your organization's specific needs, one of these rule files may be an appropriate starting point for building a custom ruleset: 

• Controlled Access Protection Profile (CAPP): capp.rules 
• Labeled Security Protection Profile (LSPP): lspp.rules 
• National Industrial Security Program Operating Manual (NISPOM): nispom.rules 

• Security Technical Implementations Guide (STIG): stig.rules 

Once Linux has an appropriate set of audit rules to follow, viewing the logs those rules generate is simple. By default, Linux stores them in /var/log/audit/audit.log, but the resulting file is too dense to be useful in a time-critical security context. Most users prefer native audit record query commands like ausearch, aureport, and the audisp-remote plugin. 

• Ausearch lets users search for specific criteria in audit log files. These criteria might include command names, event identifiers, hostnames, or other types of event data. 
• Aureport generates a summary report of the audit log file. It shows less detailed information about each event but presents the data in an easy-to-read format. 
• Audisp-remote can send logs to a remote server by writing them to syslog. Outputting the audit logs in a friendly format like JSON or XML may require a bit of extra work. 

Centralized analysis through a log management system or a SIEM platform is also possible. It might be necessary to employ a job scheduling utility to periodically send local audit logs to the platform. Alternately, hosts that generate audit logs could write directly to an event streaming tool like Apache Kafka. 

Frequently Used Linux Log Files 

The following is a quick list of Linux log files Exabeam users frequently use when conducting incident investigations: 

• /var/log/syslog or /var/log/messages. This shows general system activity logs. Analysts use it to detect problems that occur during startup and to isolate application service errors.  
• /var/log/auth.log or /var/log/secure. These are authentication and authorization logs used to investigate failed login attempts.  

• /var/log/kern.log. This folder contains kernel activity logs, including custom kernels. 
• /var/log/faillog. These are failed login attempts, useful for identifying credential compromise attack risks. 
• /var/log/maillog or var/log/mail.log. These logs relate to mail servers, enabling analysts to to track issues like spam emails and suspicious postfix or smtpd usage. 

Security analysts using Exabeam to investigate Linux system events should use these log files to examine user and terminal IDs, login attempts, and system configuration changes. Analysts must have visibility into running executable processes on the machine and be able to view security-related events (like the activation or de-activation of cybersecurity tools) in real-time. 

Have Lumifi Perform a Risk Assessment of Your Linux Systems 

As an experienced managed detection and response provider, Lumif is well-equipped to help enterprise IT leaders determine the level of logging ideal for an optimal security posture. Rely on our expertise to specify which log events should generate security alerts, and what level of priority each alert should have. 

Speak with a Lumifi expert and learn how you can protect Linux systems from cyberattacks.