Automating the SOC: Phase 2

Building on Phase 1 of Automating the SOC has been a long process, but the increases in maturity of the FloQast Detection and Incident Response team over the past nine months since our last update has been a leap. If you weren’t keeping up with us back in June of 2023, I’d recommend taking a quick look through Automating the SOC: Phase 1 so that you’re caught up on where we were as a team. If you’re new to Security Operations, I’d highly recommend reading up on the constant battle against Alert Fatigue or our journey to becoming a Data-Driven SOC to round out your knowledge a bit more on us. But if you’re just here because you’ve been patiently waiting to hear about Phase 2, let’s jump into it!

Rip-and-Replace

When writing about Phase 1, we chose to go with a SaaS solution that had promised us a quick, seamless implementation of our SOAR workflows using a handy GUI interface. One thing we didn’t consider with our roadmap for Phase 2 was this solution not panning out how we thought it would. Without pointing fingers at the solution we had decided on, we realized that the product was simply not meeting our needs. As a solution-based organization, we cut our losses early and pivoted to a new solution – one we had previously toyed with but thought it might be too much to undertake.

The solution we decided on? Rip-and-Replace with a Custom solution.

We leveraged existing AWS infrastructure and templates to build a SOAR from scratch. Using AWS Step Functions and Lambdas, we could send alerts from our SIEM over to our new SOAR to automate the initial triage on all of our alerts. This was no easy task to accomplish, as one might imagine. But after its full implementation approximately six months ago, we haven’t looked back and couldn’t be happier with the flexibility this new tool has given us.

The tool has been coined “PopDART” in our organization and has become the frontline soldier in the fight against Alert Fatigue.

Maturity

After the Rip-and-Replace was complete, we found ourselves in a similar place we were months prior with the previous tool running – but feeling much better positioned with PopDART. Thus began our efforts to mature the current solutions we had in place. We had two main solutions we needed to mature: SOAR and SIEM.

SOAR Maturity

SOAR maturity came in the form of optimizing our workflows for each log type and each detection. Our workflows have branching logic by log type first, then branching logic based on the detection following afterward. Maturing these workflows first requires data to understand what is working and what is not fully. So, after our workflows had been running for a few weeks, we began to make tweaks.

For those unfamiliar with the architecture in SOAR workflows, each log type will have default portions, where every alert from that log type will run through. These actions are tasks that will apply to every alert that comes through. Then there is branching logic, which are detection-dependent tasks that may only apply in certain situations. This was an iterative process, but based on our data, we can now say we are in a place where most of our alerts can be triaged with just a glance at our PopDART-enhanced tickets.

SIEM Maturity

SIEM maturity is never-ending. But outside of our custom Detection additions and regular tweaks that we perform weekly, we identified a need to manage our exceptions better. This began our process of centrally managing exceptions, which, as you can guess based on our product selection and previous blog posts, are managed by code. This new process we started was an opportunity for us to ensure that exceptions for detections don’t get lost in logic, to ensure we had a central place to review exceptions, and to allow us reuse based on properties for organizing our exceptions. All net positives. This took some refactoring, but yielded us what we call our FQGlobalHelper, which allows us to import different exceptions into our detections with just a couple of lines of code.

Another addition we made to simplify our triage automation process are our standardized alert contexts. Alert contexts objects carrying the important values of the logs that triggered the alert. We created methods by log type that we call in our detections to standardize the fields that are sent to our SOAR, conforming them to the same model. These alert contexts still have the ability to add additional fields to the object if a detection calls for it, but these new methods allow us to keep the base key, value pairs consistent.

Incident Response Automation

Improving our Incident Response (IR) processes was an item we had on our backlog for some time but kept getting moved back due to more pressing items (i.e. the Rip-and-Replace). However, we finally built out our automation, and its success has been immediately apparent. Previously, during Incident Response Investigations, we had identified a need to decrease the time it takes from reporting a finding, to having the documentation and infrastructure in place to triage and investigate the finding properly. In order to launch an investigation, we needed a private Slack channel, Jira ticket, and Google Drive Folder filled with Documentation Templates. On top of that, naming conventions must be consistent, all these items had to be shared with the same group of people, and all resources to be linked together for easy reference. We solved our problem by utilizing a simple form that could be filled out and leveraging PopDART to handle automation of the workflow, shaving 10-15 valuable minutes off our time to containment on investigations.

Detection Requests

As our organization has continued to scale, we are increasingly getting requests for new detections to be created. So, of course, we decided to automate this process. Are you sensing a theme here? We created a form that allows other members in the organization to submit requests to the Detection and Incident Response Team, following a format that gives us everything we need to engineer that detection. Our automation also places these new requests directly into our backlog, creating a seamless experience for us and allowing the submitter to track the item as they please.

Looking Forward

As always, the Security team is looking forward to what is next. Undoubtedly, it’s Phase 3. And without question, Alert Fatigue and conversations about maturity will still be on the table. But, there are also conversations being had as to how we can continue to utilize cutting edge technologies on our venture to becoming a leader in the Cyber and Information Security space. So, what is going on in Phase 3? We’ll let you know as soon as possible. But hopefully, this time, much, much sooner.

Ryan Cox (Security)

Ryan is a Security Engineer at FloQast with a focus in Detection and Incident Response. Outside of InfoSec, he's an avid golf/basketball/tennis player, weightlifter, guitarist, and loves to travel.