Bespoke Monitoring With Custom AWS Config Rules

At FloQast, we are always looking for new ways to increase visibility into our environments and find opportunities for security improvements. To this end, in June of 2022 we implemented monitoring for AWS EC2 instances using unapproved Amazon machine images. In addition to being able to detect abnormalities, we needed a solution that would allow us to:

• Evaluate configurations with a high degree of flexibility
• Reliably trigger evaluation upon resource changes
• Centrally manage configurations across multiple environments
• Alert on detections
• Flexibility and Reliability

The natural choice for this would be to leverage AWS Config, but will it work? AWS actually provides a default rule option for approved AMIs, but it’s limited to a list of 10 IDs. What if you want to be able to extend the list and evaluate images based on attributes besides ID? Luckily, Config supports Custom Lambda Rules, which allow for evaluation logic to be as sophisticated as needed. Config discovers resources and records configuration items for changes, so this meets our first two requirements.

Central Management

Central management can be achieved by combining AWS Config with AWS Organizations. By designating an account as a delegated admin for Config, rules can be defined and updated in member accounts without any extra effort. Enabling delegated admin permissions only takes a few commands. We can even require that the evaluation lambda resides in the delegated admin account. Doing this means we also need to set up cross-account roles to allow the lambda to query data from the member accounts and return the evaluation result.

This is great because if we need to modify the rule logic, there is only one lambda that needs to be updated. It also gives us confidence that the rule logic can’t be updated by a potentially compromised privileged user in a member account.

Alerting

Alerting on detections is where things get interesting. Our original goal was to integrate alerting into our AWS Vulnerability Management process, which relies on AWS SecurityHub. If it were possible to get everything into SecurityHub, our existing tooling would be able to query SecHub for the relevant findings and handle the rest (Spoiler Alert: AWS now provides this natively, but we’ll get to that later). At first glance, it seems like it would be easy, as Config includes the option to aggregate rule evaluations from organization accounts. However, setting up aggregation doesn’t make the data from other accounts accessible from a single source, except through the console. Being able to click between accounts in the console is great, but the endpoints that enable it aren’t made available to users.

We tried setting up an EventBridge bus to collect Config events for the rules we were interested in, but Config doesn’t actually emit events for evaluations that are aggregated from other accounts. It would be possible to set up an event bus in each account and have them all funnel to the management account, but that gets messy and it conflicts with our central management goal. We also explored using the Config SNS topic, but there isn’t a way to efficiently filter for a specific rule since Config’s SNS message attributes can’t be configured.

Final Architecture

After some trial and error, it became clear that the best solution was to cut out the middleman. The evaluation lambda already receives everything it needs to be able to send an alert, so why not have it write a log to an S3 bucket to be ingested by our SIEM, as well as report back to Config? While we’d miss out on the automatic resource status updates from Config, we would have a reliable way to be notified of issues.

We ended up going with the dual-output lambda approach, but days later, we discovered that AWS added functionality to natively integrate Config and SecHub. It’s nice to have been ahead of the curve, but seeing an announcement like that is definitely bittersweet under the circumstances. SecHub might not be right for every organization, so the lambda approach could still be useful for some applications.

In keeping with informal conventions established at FloQast, this tool was named via backronym as Italic, or “Image Trust Allow-List Inspection Controller.” In the future, we may extend it to add more rules and cover more resource types, but for now, it is effectively fulfilling its intended purpose of monitoring for unauthorized images.

Trevr Fernald

Trevr is a Security Engineer at FloQast with a passion for automation and problem solving. Outside of work, he enjoys reading, playing guitar, and skiing.