5 Often-Overlooked Log Sources

All logs are not created equal. Common logs from servers and firewalls are fairly easily ingested and parsed, while DNS or physical security logs are much tougher to manage at scale, and block visibility into the security environment. The challenging logs are more likely to be skipped: According to a 451 Research survey of 150 large enterprises, security information and event management (SIEM) platforms only ingest logs from about 45% of their organizations’ log-producing systems.

But whether logs are a slam-dunk to ingest or demand more time and attention, security teams should consider logging often-overlooked sources that are valuable for threat hunting exercises. Here are five log sources that deserve a second look, along with suggestions for maneuvering around the challenges.

1. Domain Name System (DNS) Logs

Logs from DNS servers provide a wealth of information about which sites users visit, and whether malicious applications are reaching out to command-and-control sites. DNS has also been successfully used as a tunneling protocol for exfiltrating data since firewalls typically allow it out.

However, DNS logs are challenging to work with because of the volume of data and their multi-line format. Consider using Microsoft’s Analytical Event Logging method, which uses a more standard logging format, instead of the old method of turning on debugging and importing the flat file.

Take it one step further: Check out ReliaQuest’s threat hunting use case to learn how to leverage DNS logs for threat hunting. 

2. Cloud Platform Logs

Many cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) don’t have consistent logging formats. For this reason, your security team will need different parsers or methods of logging events from each platform’s various applications.

Cloud Application Security Broker (CASB) solutions, which sit between cloud service users and cloud applications, provide granular auditing capabilities at the application or service level. If your team is using a CASB, the solution needs to have the same logging and monitoring considerations as the full cloud platforms.

3. Database Logs

Database auditing and logging can be a stumbling block, since database administrators often don’t want to enable features that could affect server performance. Also, auditing individual databases and tables is difficult given the large number of database servers in a normal enterprise environment.

To gain visibility into these databases without enabling auditing functions on database tables, you can try these two options:

• If Database Activity Monitoring (DAM) is present, ingest and correlate built-in rules and alerts into the SIEM, since it performs many of the same restrictive functions as a firewall or web application firewall.
• Create stored procedures that watch for specific actions, such as unencrypted PII stored in a field every 30 minutes, and write an event log with the record ID, date, and time to trigger an alert. These scripts are low-impact and can look for specific scenarios, unlike the all-or-nothing approach for table auditing.

4. Web Server Logs

According to vulnerability- and exploit-tracking efforts such as the annual Verizon Data Breach Investigations Report, most breaches trace back to holes in public-facing web applications, such as the log sources into which teams have the least visibility. That’s a problem, because web applications often have access to highly sensitive customer account information.

Parsing web server logs is challenging because they’re often in a multi-line or custom format, and possibly logged in a non-standard way to a text file or database versus the native web server log, such as Internet Information Services (IIS) or Apache. If you’re using standard web server logs, enable all the relevant fields since the default W3C layout in IIS doesn’t capture some critical elements, such as page size and cookie values.

5. Physical Security Logs

Obtaining event logs from physical security applications such as camera systems, biometric/card access readers, or alarm systems is highly valuable for cases involving insider threats – especially when combined with evidence correlated from workstations, firewalls, and remote access devices to pinpoint a person’s location. However, physical security teams and IT security teams often work separately, and many access control systems operate on closed legacy systems.

To work around these challenges, try to limit focus on logs for these events:

• Remote login with corresponding badge entries
• Unauthorized physical access to remote, unmanned facilities
• Audit of authorized employees accessing potentially unauthorized areas of the company
• Excessive biometric or card failures
• Visitor/contractor access to unauthorized areas
• After-hours alarm triggers or excessive open-door time alerts

Ingesting the log sources above is an important step toward improving visibility into the enterprise security environment. Ask your security team to work with data and application owners ahead of time so you can review actionable event types together, and see which elements the source owners might need visibility into as well.

Need more guidance on ingesting these logs? Download the ReliaQuest paper, Top 5 Log Sources You Should Be Ingesting but Probably Aren’t.

Joe Partlow, ReliaQuest CTO, currently oversees all new research and development efforts and new product initiatives. He has been involved with Infosec in some capacity or role for over 20 years, mostly on the defensive side but always impressed by offensive tactics. Current projects and interests include data analytics at scale, forensics, threat, security metrics and automation, red/purple teaming, and artificial intelligence. Outside of Information Security, he has been involved in many other areas of the business including Web Development, Business Intelligence, Database Administration, Project Management, IT, and Operations. He has experience in many different business verticals including retail, healthcare, financial, state/local government, and the Department of Defense. He is also a regular speaker and contributor at security conferences, groups, and associations.