Posted in Logging, Resources

Graylog Homelab/POC: Part 2 – Searching, Streams, Alerts, and Dashboards

Now that you have data coming in (if you don’t, see Part 1), it’s time to do something with it. A good place to start is figuring out what the logs are saying. After looking over a couple of common log types, I think it gets to be relatively intuitive. Most vendors will have some sort of explanation of log structure, or you can refer to the RFC when appropriate. For our POC home lab, these resources give a solid overview of the logs we’re bringing in:

Once you have a general idea of what the logs look like, it’s time to start searching for things that might be interesting.

Searching

Honestly, if you can Google, you can do basic searches in most log management systems. I’ve found Graylog’s search language to be easy to jump into – typical Boolean search structure with an OR default for multiple words. Even though it’s straightforward, taking the time to read through the Graylog documentation on searching would be worth it. According to the Graylog Searching documentation, Graylog is close to Lucene syntax. Lucene is used in a lot of different places, including Azure search – so a good thing to get comfortable with. Microsoft has a guide for using Lucene with Azure that is pretty thorough. (Note – these are good things to notice to help identify transferable skills.)

  • Search for a keyword anywhere – keyword

  • Search for multiple keywords with OR – keyword1 keyword2

  • Search for exact phrase "keyword1 keyword2"

  • Search for a specific type – type:keyword

  • Search for a specific type or another specific type – type:(keyword1 OR keyword2)

You can also pull in regular expressions. If you aren’t familiar with regular expressions, the short Regex course on Cybrary is a demo heavy introduction (lots of examples, not a lot of lecture – I got a lot out of it) and regexr is a fantastic sandbox to practice in.

Graylog does have a time frame selector that defaults to the last 5 minutes. If you forget to adjust this, you may see nothing. And wonder what’s going on. It’s a handy feature though that quickly allows you to narrow things down.

The search page generates a histogram with the search that you can easily add to a dashboard. I like this feature and it was completely intuitive to understand. You can also do some additional analysis that can be popped into dashboards. One of the options I find nice is the “Quick Values” option – this gives you a pie chart and table of the results (again, easily popped into a dashboard). I found it really easy to search, make a visualization, and send it to a dashboard in Graylog. Easy enough that I was able to point and click through to make some dashboards without looking at the documentation. Looking over the documentation opens up a lot of additional possibilities.

Streams

The streams concept in Graylog took gave me pause for a moment, but reading about what the purpose is, made them make sense. Of course the documentation is helpful. I basically figured out what they were initially by trying to create an alert and the web console saying I had to set up a stream first. So the process is straightforward, but good to actual understand the architecture as we work on moving from a POC to a production deployment. I look at streams as a way to look at just messages of a certain type that I want to look at. So you could pull out logs from a specific machine or of a certain type. A couple of important things to note from the documentation is that the streams are in real-time and you can search for complex things using streams because the tag for the stream is applied as the message is processed. These aspects make streams a great way to keep on eye on specific things going on in the environment and quickly find messages of interest.

The easiest way to set up the stream is to set up a rule using the name of the machine sending in the desired logs. You can choose to send messages to multiple streams – for example, having a stream for a specific Windows machine and another tracking all failed logins.

For our homelab POC, we’ll walk through setting up a stream for the Parrot VM. In the Graylog web console, go to Streams then Create Stream.

  • Title: Parrot

  • Description: Logs from Parrot VM

  • Index: Default index set

  • Leave the box for “Remove matches from ‘All messages’ stream” unchecked

  • Save

Now to give the stream a rule to pull in the logs. Click on Manage Rules, select the Syslog UDP input created earlier. (If you have other things coming in on this input, go to the All Messages stream and copy the message ID for one of the Parrot logs to save a little time. Also make note of the index name. ) Pull up a message from the Parrot VM and use that to create a stream rule. (You can select the inverted option to exclude messages matching the rule.)

  • Field: source

  • Type: match exactly

  • Value: parrot

Verify that the message would be routed to the stream, then click I'm done. Once you’re back on the main Streams console, be sure to start the stream (click the Start Stream button).

Alerts

Now that you’ve got a stream setup, you can create an alert to trigger off the stream. The simplest alert is probably setting up an alert to trigger when more than X messages occur in Y amount of time. Alerts can be either unresolved (trigger condition is true) or resolved (trigger condition is no longer true). Graylog also provides the option of setting a grace period so a period of time must pass before an alert is re-triggered. This is a high-level overview, so check out the documentation for more depth. You need to set up conditions to trigger alerts – this process is very similar to setting up a stream. A few alerts to get started with could be triggering when a login failure occurs on Windows (search for event id 4625) or when you get more than 5 alerts at level 3 or lower from your firewall. In a POC environment, playing around with the different streams and alerts is important. Try adjusting the sensitivity of the triggering conditions to the point where you get alerts when something needs attention. If you notice that you aren’t getting much going on in your lab environment, you can either create traffic that would trigger the alerts (an easy way is intentionally entering incorrect login credentials) or adjust the sensitivity to trigger on normal traffic in your environment.

Steps for setting up an alert:

  • Define the condition: Alerts > Conditions > Add New Condition

    • Choose which steam to alert on.

    • Choose condition type:

      • Field Content Alert Condition: At least one message containing the value specified.

        • Somewhat nuclear option – I want to know whenever this happens. For your own sanity, I would use these sparingly.

      • Field Aggregation Alert Condition: Computation on stream is higher/lower than a threshold.

        • Math – Use mean, SD, min, max, or sum to monitor a condition. Good for monitoring performance, or when you want to know when something goes out of range (set conditions for both min and max options).

      • Message Count Alert Condition: More than a specified number of messages in the specified time frame.

        • Frequency count – Let me know when something spikes. Easy to set up for some basic security monitoring.

  • Set up the notification: Alerts > Notifications > Add New Notification

    • Note: The notification will apply to all conditions for a stream.

    • Choose which stream to notify on.

    • Choose notification type (these are the defaults, more can be added):

      • HTTP Alarm Callback: Call an endpoint when the alert is triggered. Sends a POST request to the notification URL.

      • Email Alert Callback: Email, have to modify the server configuration file with the SMTP settings.

      • For the POC, either an SMTP server or a webpage to post alerts to will work. I found these walkthroughs from DigitalOcean useful for setting up a send only SMTP server and for installing the Apache Web Server. I think the easiest option is setting up the webpage, but building an SMTP server would definitely be a great learning exercise.

Dashboards

vehicle gauges
Photo by Craig Adderley on Pexels.com

I’m not going to go into a lot of depth here, but setting up basic dashboards is ridiculously easy. I mentioned earlier being able to point and click through searches to add things to the dashboard.

You will need to set up a dashboard: Dashboards > Create Dashboard. Then you can add searches or other items by clicking on the Add to dashboard button. Once you’ve added widgets, you need to click the Unlock/Edit button on the dashboard to move things around. Learn more about the available widget types in the Graylog Documentation.

Once you’ve set up a dashboard, it’s time to add elements to the dashboard. I’ve seen some amazing dashboards, and there is a learning curve. People often identify themselves as search people, dashboard people, etc. when talking about logging. The goal here is to be able to handle the basics to put yourself in a position where you can develop more in-depth skills as needed.

Some things you might want to track would be various Windows events, such as 4625 (failed login) or 4776 (audit failures), number of messages in a stream (for instance dropping to 0 meaning you probably have a storage shortage), or the quick values for a search (such as the number of password resets done in the last day). The one thing I’ve noticed so far is that I’m not able to do some of the more in-depth analysis in Graylog that I could in Splunk. It’s probably because I haven’t gotten into the search language deep enough.

Roles

A quick note on roles…the default is Admin (because, of course it is). You can easily add other roles. The other available default role is Reader. What you’ll want to do is create specific roles for your environment such as Analyst or Manager, and then assign permissions to view or edit the available streams and dashboards. Being able to give managers or other higher ups a single pane of glass (yeah, I went there) can be really helpful, and by giving them view only permissions, you can make it easy for them to keep an eye on things without having to get in the weeds of managing the system. “Single pane of glass” may be one of those eyeroll eliciting terms, but there are a lot of benefits to making the work you do visible.

What’s Next?

I think one of the most beneficial next steps is building Graylog out yourself, without the training wheels of using the VM. The documentation is fairly thorough, but I have run into some permissions issues getting Elasticsearch setup. Some searching should get that straightened out, but you may need to search related to Graylog, Elasticsearch, or MongoDB. However, to get Graylog beyond a small POC, you’ve got to learn how to build it out yourself.

It would also be helpful to learn some of the other options – Splunk, Elastic, LogRhythm, etc. Elastic is next on my list because of the free SIEM option. I think it’s important to get familiar with a variety of options because the solution that works for you now may not be the best option in the future.

Probably the most important thing you can do is spend time with your logs. Get to know what normal looks like. The event codes and other descriptors will eventually start to make sense without having to look up the code – though making a cheat sheet would be a good idea as well.

Graylog is a great way to learn the basics of centralized logging. In a small lab environment, you should be able to keep the VM running for quite awhile without running out of space. I have found it very quick and easy to get set up, which is important when you are trying to develop skills and want to maximize learning time.

Author:

Lifelong paradox - cyber sec enthusiast - loves to learn

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.