Posted in Blog, Resources, Walkthrough

Graylog Homelab/POC: Part 1 – Initial Setup

Intro – what is Graylog?

Graylog is a centralized logging solution, so similar to Splunk, Elastic, etc. (yes, I’m simplifying). The company emphasizes the speed, scalability, and affordability of the product. The website provides use case info for security, compliance, IT operations, and devops, as well as a section for MSSPs. Graylog started as an opensource project and maintains those roots with a free opensource version and a free tier of the Enterprise option. The Enterprise option offers some (important) add-ons like Views, Offline Log Archival, and User Audit Logs. The opensource version gives you a whole lot, but there are some good incentives for jumping to the Enterprise tier. The comparison breaks the differences down well and provides enough info to make an informed decision.

One of the things that I really like about Graylog is the focus on usability and getting things in place in a functional way. I see way too many things get implemented and then be too complicated to maintain or easily bring people up to speed. I find this typically leads to the product being dropped and the business not seeing a good ROI on the product. I thought the Graylog whitepapers on growing a SIEM deployment and using log management to optimize SIEM were really helpful for developing a plan to implement centralized logging/SIEM. I’ve also found Graylog easy enough to use that I think training people for daily tasks could be done efficiently. Of course this is based on small scale usage, but at least when things are relatively uncomplicated on a small scale there is hope of being relatively uncomplicated on a large scale.

This will be a multi-part series focused on getting Graylog up and running. This portion will focus on just the initial setup using a VM. Part 2 will look at streams, alerts, and dashboards.

Why test?

depth of field photography of brown tree logs
Photo by Khari Hayden on Pexels.com

Graylog is a great option for a home setup because the opensource option doesn’t have a data limit. This makes it really nice for home monitoring since you won’t have to worry about license violations. The Enterprise version has a free version with a 5 GB daily limit, which should be enough for many home options and possibly some smaller businesses. The big caveat I found with the free Enterprise option was that the user agreement indicated that Graylog needed to be able to check-in with the server at any time. I understand the reason (because keeping track of usage for the free tier is understandable), but it makes me hesitate to use the Enterprise option for home lab or POC when I will be shutting off the server regularly. Of course, turning off the log monitoring server does kind of defeat the purpose of centralized monitoring, so in productions situations you wouldn’t be turning the server off. A reasonable approach would be going with the generic free option until you have determined viability, and then move to the free Enterprise tier when you are ready to set up a 24/7 server.

Although Graylog isn’t one of the biggest names in log management right now, working with it in either a VM or setting up the components would develop transferable skills. (I also personally think Graylog is poised for growth – solid platform and pricing seems quite competitive.) Moving beyond the VM setup covered here would be very beneficial because you would be setting up Elasticsearch and MongoDB to support the Graylog server, as well as configuring the Linux-based Graylog server. There are also options for Docker and AWS, so there are a lot of different things to work on and develop needed skills. The documentation is solid, and the user forums are active – so you can probably find troubleshooting help quickly. I’ve also found Graylog to be pretty easy to use once you get oriented to the platform. At some point, I imagine I’m going to end up with a home solution that is up and running some sort of central logging (probably Graylog) and Security Onion so I have stuff to play with.

Quick setup & Tips

I’m going to cover setting up a Graylog OVA VM in a home lab and getting logs in from a few different sources. I think this is one of the quickest ways to get familiar with a centralized logging solution. I’ve done this with Splunk as well, and I’ve found the setup to be a little easier with Graylog because of the VM option. Neither is “hard,” but I do like being able to quickly pop up a VM that I can just blow away when I’m done with it.

scarlet macaw
Photo by Tim Mossholder on Pexels.com

My setup will be the Graylog VM, a ParrotSec VM to connect to the Graylog server, and a Windows machine. The Windows machine could be a VM or an actual machine. You can also just as easily connect to the Graylog web console from Windows as Parrot, but do note that Graylog itself should go on Linux. The documentation gives some additional info (basically, you could install on Windows but it’s likely to be an exercise in frustration). This way you’ll cover getting logs in from Linux and Windows, which are probably the most likely ones to see. If you have a Mac around, I would add those logs as well. Unfortunately Apple is even more of a pain to get into a home lab than Microsoft. At least Microsoft offers trials or development ISOs you can use for home lab/testing/POC. I’ve yet to find a good (and ethical) way to test Apple without buying a device. There are options for getting logs from applications, but I won’t get into that in this post. I’ll also use Syslog to send in router logs, just for fun. I will be using VirtualBox because I find it to be a more feature-rich option than VMWare’s free offering.

Basic steps

  1. Download and import the Graylog appliance – the OVA option is used for VirtualBox.

    1. Note: Make sure you record the admin password the first time you power on the VM. If you don’t, it’s lost to the ether and you’ll need to import the VM again if you didn’t take a snapshot before starting it the first time. But we always take a snapshot of the initial import, right?

    2. Set up the VM to have a static IP. I prefer to do this on the VM for this use case – for a more permanent setup, setting at the router or DHCP server would be preferred. LinuxConfig.org has a solid walkthrough on how to update the netplan config file.

    3. Check the time and date using date. For some reason my VM decided to be 30 minutes off. A restart took care of it, but in case you have more issues, DigitalOcean has a nice write up on managing time sync in Ubuntu.

    4. Note: The Graylog server will orient itself to UTC and if you put the VM in host-only mode, it will take your host OS time to be UTC, and that can cause a major headache when you can’t find your logs. If you decide to go with a host-only adapter for your VM, make sure that you carefully check time zones on the Graylog server and machines sending in logs. Don’t ask me how I know…

  2. Download and import Parrot appliance – I use the Security version, but the home/workstation version should work as well.

  3. Set up appliances to be on the same network.

    1. Since I want to send in my router logs, I’m going to use bridged mode to make life easier.

  4. Set up Graylog using the web console.

    1. Access the appropriate IP through a browser.

    2. Login using the credentials provided when you first started the Graylog VM.

  5. Set up Graylog syslog input. I’m putting the menu options in the code format so they stick out a little more, these are all from the web interface.

    1. From the Graylog web console, select System/Inputs > Inputs.

    2. On the Inputs main page, select Input > Syslog UDP > Launch new input.

    3. Choose the node or select Global.

    4. Give it a title like “Syslog UDP”.

    5. Set the bind address (can be 0.0.0.0) and port (some systems may complain at 514, so you may need to pick something else).

    6. Leave the rest as defaults.

    7. Save.

  6. Set up Parrot to forward syslog to Graylog. These steps are from the terminal.

    1. Choose whether to add a separate Graylog configuration or just add it to the /etc/rsyslog.conf file – for this example, I’m going to add a new config file. This makes the info quick and easy to find for updates.

    2. Move to the rsyslog configuration file directory cd /etc/rsyslog.d.

    3. Create the new configuration file: sudo touch graylog.conf.

    4. Open the file for editing: sudo nano /etc/rsyslog.conf.

    5. Add the entry for the UDP input (See the documentation for more info):

       *.* @<graylogappliance>:<port>;RSYSLOG_SyslogProtocol23Format
       *.* @192.168.1.100:514;RSYSLOG_SyslogProtocol23Format
    6. Restart the syslog service service rsyslog restart.

  7. Verify logs are getting in:

    1. If logs aren’t, some troubleshooting steps…

      1. Verify proper ports are open on Graylog server netstat -tulpn | grep listen.

      2. Open up the firewall ports as needed – check the Ubuntu docs for info. I found this article from IBM on firewall configuration that shows several different ways of managing firewalls. The Graylog VM comes with UFW disabled and iptables set to allow all, so you may want to enable it to practice working with firewalls. I did some quick and dirty configuration just to get some practice…

         ufw allow 9000 #connection to web interface
         ufw allow 514 #Syslog UDP input
         ufw allow https
         ufw allow http

        ETA: A couple more rules that might be important…

         ufw allow ssh #do this before enabling ufw if you are on ssh
         ufw allow 5044 #Windows beats
      3. I had to disable ipv6 on my Graylog appliance because it wasn’t playing nicely. In a production environment, you would want to make sure this wouldn’t cause any errors. For homelab, I’m more interested in getting things dealt with quickly.

         sysctl -w net.ipv6.conf.all.disable_ipv6=1
         sysctl -w net.ipv6.conf.default.disable_ipv6=1
      4. Try restarting the Graylog server service sudo systemctl restart graylog.server.service.

  8. Set up router/firewall to forward syslog to Graylog:

    1. This part will be very device specific, so you’ll need to some research (AKA Googling) to see what you need to do.

    2. For many ISPs, the steps would likely look like this:

      1. Navigate to the web-based control portal for your router.

      2. Find the area that contains logging info (click through until you find it – consider it pentesting practice).

      3. Set up log forwarding using Syslog – you may need to enable Syslog for this to work.

  9. Set up Windows using Sidecar – read through the Graylog documentation before starting. It wasn’t difficult necessarily, but it would have been easier had I looked at it a little more closely the first time through. I’m going to use the Graylog Sidecar because it was easy to use once I got it figured out.

    1. Download the appropriate Sidecar package.

    2. Get the needed API token to set up the Sidecar in the Graylog web console:

      1. System > Authentication

      2. Users

      3. For the graylog-sidecar user, click on "More actions" > "Edit tokens".

      4. Give the token a name.

      5. Click Create Token.

    3. Install the Sidecar, using the Graylog appliance IP and API token when prompted. Be sure to give the install a unique name if you do this on multiple systems.

    4. I’ve also had to open the Command Prompt as an administrator and install/start the Graylog service – go the the folder where Graylog is installed, then open the Sidecar folder and run:

       graylog-sidecar.exe -service install
       graylog-sidecar.exe -service start
  10. Set up Graylog Sidecar collector in the web console:

    1. I’m using Beats on Windows because I’ve found it works well and comes included in the Sidecar

    2. Go to System > Inputs.

    3. Then Beats > Launch new input.

    4. Give the input a name, leave the rest as defaults.

    5. Go to System > Sidecar > Configuration > Create Configuration.

    6. Name the Collector Configuration.

    7. Choose the winlogbeat on Windows option.

    8. In the Configuration area

      1. Make sure you update the host appropriately.

      2. Adjust the path if desired.

    9. Click Create to finish.

    10. Go to the Collector Administration page, click on the Configure menu and select the collector you created.

    11. The side car should now show Running by the winlogbeat.

  11. Verify logs are getting in by checking for messages come from each input and each expected source.

  12. See what you can see. Look at log entries to get familiar with the format. See what you can understand and what you may need to look up.

Wrap Up

Now you should have a Graylog instance stood up and receiving logs from a couple different sources. It’s a pretty easy process.I would take some time to get familiar with the log formats that you are receiving, maybe even look up some documentation to understand what each portion of the log means. Taking the time to understand what’s coming in will make it much easier to set up streams, alerts, and dashboards, which will be covered in part 2.

It would also be nice to add in other log sources if you have things available. Going through the setup steps multiple times helps you solidify the process internally and discover any nuances to the deployment that you may have glossed over going through the first time.

Author:

Lifelong paradox - cyber sec enthusiast - loves to learn

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.