Splunk ATOs and Re-Auths

Uh oh. You’ve just found out that your organization needs you to go through an ATO for your Splunk system.  How could this be? What even is an ATO? 

An ATO (Authorization to Operate) is an organization’s approach to verifying that an IT system is secure and meets specific criteria before it is brought into a production environment.  Whether you are just standing up a new instance of Splunk or it’s simply time to re-certify that your system is secure, here is some advice that may help. 

First Steps

Typically, and like most things in IT, this will not be a short process. However, once you know that an ATO is coming up, it’s not too difficult to start preparing for the process.  The first thing you will need is documentation.  Luckily, you should have most of this info already.  If not, you will get a chance to really learn about your Splunk System.  A good place to start is your Monitoring Console (MC).  The MC will give you an overview of the majority of Splunk’s core components.  

Now it’s time to scrape through share drives and SharePoint pages for documentation.  If you don’t already have them, you’ll need to create asset lists, diagrams, and any other policies and procedures that pertain to your system.  Asset lists are essentially just a listing of all the core Splunk devices, interconnected components, and any other relevant information. The main things you will need to supply are numbers (how many of each device), hostnames, and versions (of relevant software/applications).  Try to get complete asset lists with Hostnames and IPs of the core devices to include Indexers, Search Heads, License Managers, Deployment Server, Heavy Forwarders (HFs), Universal Forwarders, syslog servers and any other components.  Generally, auditors like to see the version of both Splunk and the OS for each Splunk component as well as any other software hosted on these machines. Another helpful place to look (if you are using one) is the Deployment Server.  This can specifically help you track all the Universal Forwarders (UFs) in your environment. 

Once you have a good handle on all the Splunk components you manage, make sure that you have accurate diagrams. Try to include the physical and logical location of your machines as well as data flow such as ports and protocols used. This will go a long way in helping the auditors understand your environment’s layout. 

Pre-Audit Meetings

As you get close to starting the ATO, there will typically be some type of architecture review meeting with the auditors so that they can better understand your environment and determine their boundaries.  Auditors can usually get a lot of context from diagrams, but it’s better to provide clarifications rather than let them make assumptions.  It may benefit your team to have a meeting before these discussions to hash out ownership, management, and boundaries. Many organizations divvy up responsibilities to different groups for ease of management. For example, the server team may be managing the OS for in-house equipment.  Although your team might be utilizing those machines, certain configurations and other responsibilities like patching could fall on another team and be evaluated under a separate ATO. This type of scenario has many benefits, allowing SMEs to manage what they are good at and taking some of the workload off your plate. 

Starting the Audit and More Paperwork

Once the Audit starts, remember: the whole reason for going through this process is to ensure your system is secure.  If you are not a security expert, that’s ok; there are standards already in place that tell you what is important.  You should know which standard or standards your system will be evaluated against.  One of the most well-known Standards is NIST 800-53 (NOTE: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-53r5.pdf), which is a list of controls that dictating security requirements such as Access Control, contingency planning, Incident Response, etc. 

Generally, at this part of the audit you’ll receive a questionnaire with a specific number of controls selected.  Each organization is different, but the more critical a system is, the greater the level of scrutiny and number of controls selected. Details such as PII being used, Internet-facing applications, and whether classified data is present are likely to increase the number and types of controls auditors will select. 

You have to answer ALL of the questions, addressing each control to the best of your ability. You will want to be familiar with your organization’s internal policies since some controls may rely on them.  However, the good news is that the more your system leans on internal policies and the more you leverage other groups and technologies (assuming they are doing their due diligence), the more items can be viewed as complete where you respond to the auditors. 

Performing the Assessment

Now the actual hands-on work starts.  The auditors will show up to perform interviews and evaluate your Splunk implementation. This is done using a variety of different scanning tools.  Nessus is often used to scan your Splunk implementation and is especially good at identifying vulnerable software and common misconfigurations. However, there are other checks to see if a system has specific misconfigurations. One of the most common checklists, especially in the DoD space, is DISA’s Security Technical Implementation Guides (STIGs).  Although the STIGs are generally focused more on the OS layer of the system, you can use more targeted STIGs to help secure Splunk. STIGs and other organization-specific security checks should also be considered when first standing up new instances of Splunk as you can’t always go back and enable a feature after the fact. 

Findings, Remediations and Mitigating Controls

Once the onsite assessment is complete, the auditors will present their findings along with associated risk scores for each. Generally, a meeting is held to discuss these findings and ensure that there were not any oversights or misunderstandings.  Don’t worry if you have some deficiencies; you will get a chance to address them either at this meeting or shortly thereafter. Typically, the more vulnerable a finding is, the sooner it needs to be remediated, so make sure the most significant issues are prioritized. There may be times when you cannot completely fix an issue, but there are still steps you can take in these cases. By implementing mitigating controls, you can configure additional factors that can lower the likelihood that an attacker would be able to exploit a vulnerability successfully.  The goal here is to limit risk to an acceptable level if it cannot be removed entirely. 

Final Steps

After all the fixes, mitigating controls, and accepted risks have been completed and signed off, the Splunk implementation is ready for Certification and Accreditation.  This process is generally more meetings and paperwork to make sure that every stakeholder involved is comfortable with the associated risk.  Once that happens, CONGRATULATIONS, your Splunk implementation is ready to go!  However, although your system may now be ATO certified, you’re still not off the hook.  You will have to maintain documentation and keep it all up to date.  After all, you’ll be going through this process again… so save yourself some heartache.