If you are doing much with security logs, you’ve probably heard of Splunk or other SIEMs. While the tools are great, learning them can be easier said than done because of the logistics of getting the system up and running with enough information to give you a realistic picture of how it would function in a job setting. However, you also have to learn the tools of the trade.
Fair warning, this is a really long post. I’ve decided to keep it together because I suspect if you care enough to read the first part, you would want to read the second as well. This was originally presented at a Splunk user’s group, and I’ve crammed/reformatted it into a blog post.
If you want to learn Splunk, but aren’t sure how to go about it, this will lay out a 60 day plan to learn it. You won’t be an expert, but you should know enough to pick up what you need to know for a specific deployment relatively quickly.
If you know some Splunk but need to know more, you should be able to find some good resources to fill in the gaps you have.
If you are an expert Splunker, you might find something, but more than that, this can serve as a resource for you to point others to when they ask you to teach them Splunk. You can minimize your time expense to get them started, and be available to help once they know the basics.
I’ve gone with a 60 days to give you time to spend with Splunk – ideally an hour or so daily-ish. You can rush through things, but time with a tool is one of the best things you can do to learn it. You could compress each 10 days or so into 1, but it would be intense. Completing the actual “activities” doesn’t take too long, so where you should be putting in the time is playing around with the data. This is also geared toward someone learning Splunk independently. If you are learning Splunk for a job and will immediately be working with a deployment, cramming isn’t as bad because you’ll be using it often enough for work that the things that need to stick, will stick.
Note: While most of what’s covered will relate to Splunk in general, there will be a definite cyber security focus.
This is your recon time. See what you are getting yourself into. Make sure you have some time to dedicate to learning the tool so you can take advantage of the 60 day free trial license. There’s a lot you can do with the perpetual free license, but the trial license has some additional functionality that will be helpful as you are working through different activities.
Take some time to familiarize yourself with the available resources. You should have a couple of places to look for help when you get stuck. It’s also a good idea to consider joining some of the Splunk groups so you have actual people to ask for help.
Splunk has several free training courses available online. To sign up, log into your Splunk account and click on the training link. You only have 30 days for each course, so be mindful of that when you decide to start a course.
I recommend starting with the Infrastructure Course so that you understand the basic way Splunk works. This course doesn’t have labs, but you can look around and familiarize yourself with the basic setup. If you are feeling adventurous, input a dataset you are comfortable with and start playing around. You’ll do this later anyways, so why not jump in now?
Now that you have some basic working knowledge of Splunk, it’s time to start digging in. You should take a minute to make sure you understand how Splunk functions.
- Do you know what the different components are?
- Do you know what the different roles are?
- What are indexes?
- What’s the pipeline look like?
- How to you get data in?
- How do you grow your deployment?
If you can’t answer those questions in at least a general sense, it would be a good idea to go back and review the infrastructure course.
During this block, you’ll be working on 2 main things – the Splunk Fundamentals I Course and outside activities that reinforcement what you’ll be using Splunk for.
This is a great intro to actually using Splunk. There are lots of guided labs, a nice dataset to play with, and you’ll be able to access the dataset even after the course ends. You can blow through this course really quickly and just click/type through the labs, but that’s not going to get you where you want to go.
I’d work through a section, do the labs, and then spend a time using what was covered to look at the dataset in different ways. Even if you aren’t familiar with what the things in the dataset are, you can work on manipulating different fields to see what different searches give you and what some output options are. You should also take what you are learning from the outside activities you’re doing and apply them to the class data set. This will help you internalize what you are doing so you don’t have to constantly repeat things.
While you are working through the Fundamentals I Course, you should also work through other things that will help you retain what you are doing. The Splunk Bucketlist is basically a (not so) little CTF for Splunk. What’s nice is that it will likely put resources in front of you that you wouldn’t see on your own.
I’m a big fan of podcasts (you can see some of my favorites here). There was a really good episode of Braking Down Incident Response just on different SIEM usage. Different options were also discussed, but the big takeaway was the discussion of tuning options and how to get the most out of your log management. At this point, a fair amount may go over your head, but being able to hear how other people are using different tools can be very helpful.
The last outside activity to work on while you are doing the Fundamentals I Course is to dig into your specialty. Are there apps that would be helpful? (More about that later.) What blogs are there related to your interest? You don’t have to go chasing too many rabbits at this point, but making a reading list would be a good start.
At this point, you should feel like you have a good idea of what’s going on. You should feel comfortable running searches, even if you aren’t quite sure what they mean or how to get what you want. So it’s time to start exploring. This block also gives you a bit of a breather to process things before jumping into more advanced topics.
For the getting comfortable part, you need to play with what you know. If you haven’t yet added a dataset of familiar data, now is the time to do it. It’s also a good idea to try to mirror an enterprise setup to the extent that you can. You can set up a forwarder and home monitor to see what’s going on in your network.
You should also start watching your licensing at this time. See how much data you are going through with your home setup to determine what you’ll need to adjust when your trial license expires.
The branching out part come from exploring your area of interest. For security, I would start with the threat hunting blog series. You might also benefit from joining/attending a Splunk User’s Group. They are a great way to network and get ideas. I got several new tools to check out by presenting this plan at a user’s group (more on that later). If you are someone who uses Slack, the Braking Down Security Podcast has a SIEM channel on their Slack that is a great resource. And it’s not just Splunk, so if you end up working with another SIEM, you’ll probably be able to find help.
Now it’s time to get serious and immerse yourself in your specific focus. These resources will be security focused, but I’m sure with a little digging you can find the resources for your area. The Security Datasets Project has two sample datasets with walkthroughs to explore the data. There are also resources to help you understand what you are looking at. There is also the Boss of the SOC App. This is a play-at-home version of the Boss of the SOC Workshop. You can play either through the Security Datasets Project or download the app and data to use on your installation.
Taking the time to work through these datasets will set you up to look for threats in your own data. Be sure to look up anything that doesn’t make sense to you. It may mean taking more time, but it’ll be worth it in the long run.
Only 10 days to go – time to get serious. Set up a trial of the Splunk Security sandbox. This will give you seven days and preloaded data. You can see what you can see and get an idea of how Splunk Security is used in practice.
Once you’ve got the Security sandbox setup and have gone through the guided tour, this would be a good time to take the User Behavior Analytics Course. This is a quick course that helps you see how the different Splunk products work together. There doesn’t seem to be a way to get real hands-on experience with User Behavior Analytics, but going through the course while you have the Security sandbox will help it make sense. You’ll specifically learn about how User Behavior Analytics is used to respond to threats and how false positives aren’t always a bad thing.
Finally, make sure you change your license! Trust me, it’s easy to forget to switch it over. And if you forget, you get warnings, and may get put in timeout until your warning level goes back under the threshold.
At this point, you’re probably realizing just how much you don’t know about Splunk. That’s great! Take some time to celebrate what you’ve accomplished over the last 60 days. You’ve done a lot, and it will take time to absorb it. You’ll need to revisit things regularly to keep it fresh. You may want to collapse into a heap for a bit to recover – totally understandable.
I hope this outline has helped you get familiar with Splunk. But more than that, I hope the basic outline can give you an approach you can use again and again as you need to learn new tools or concepts.
Whatever you need to learn, you can
- Do recon,
- Get comfortable with the basics,
- Explore your interests, and
- Use what’s available.
The content will vary, but the basic approach can be used again and again.
A follow-up post (or two) will be coming with more resources to help get comfortable with Splunk. But for now, I think you’ve got plenty to work with.
Good luck and happy Splunking!