Parsing ScoutSuite Results With jq

Cloud Security Audits

Supported by other tools and manual analysis, ScoutSuite provides a solid base to start your Cloud Security audit. Such audits often follow a pattern that is quite familiar to penetration testers:

1. Determine scope and receive testing credentials,
2. Execute scanning (e.g., run ScoutSuite) to determine potential issues,
3. Validate manually,
4. Format results appropriately for the remediation team / environment owner.

ScoutSuite

While ScoutSuite’s native HTML report works well for validation work and is easy to navigate, extraction of cloud object specifiers (e.g., AWS ARNs) for the final report can get quite complicated – especially when dealing with a Cloud Audit scope that is more narrow than the account level. For example, your organization might have multiple applications residing in the same AWS account segmented by VPC, only one of which is in-scope for testing. When this type environment scales up, there can be hundreds of improperly configured objects that need to be picked through and reported (e.g., unencrypted EBS snapshots). 

jq

Thankfully, ScoutSuite builds its HTML report with a well-formatted JSON file that is fairly easy to parse ourselves. In this article, we will be describing a method that can be used to quickly extract details from the JSON file using the ‘jq’ tool. 

“jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.”

Before we get too in-the-weeds discussing technical command details, let’s first investigate the workflow for using ScoutSuite’s HTML and JSON formats. Understanding how the two formats work together will enable you to quickly identify and extract the information you are interested in without getting lost. ScoutSuite’s JSON output is regularly hundreds of thousands of lines long when pretty-formatted, so an understanding of how to navigate between the documents is essential.

ScoutSuite Report Formats & Extraction Workflow

Format the JSON report for easy viewing

Out of the box, the JSON report is not actually a valid document for use with jq. We need to remove the first line from the file. We’ll do that and pipe the output to jq with a single period as the filter argument, which simply causes jq to pretty-print the file, returning every element:

tail -n +2 scoutsuite_results_aws-71592999999.js | jq '.' > scoutsuite_results.pretty.js

After identifying a target finding, extract the Finding Reference from the URL

Most people working with ScoutSuite are going to start by using the HTML report to identify the findings they think are important, and that contain in-scope resources.

While viewing a finding in the HTML report, the HTML anchor hashtag indicates where to look in the JSON report for the finding’s information. E.g., the anchor hashtag “services.iam.findings.iam-inline-role-policy-allows-NotActions” in the following screenshot:

Find the Finding Reference in the JSON report

The JSON report is split into two main sections. The first section defines all of the issues within the report. For each issue, a reference to each problematic cloud resource (“item”) is provided, which specifies where in the JSON schema the item’s configuration data is stored in the second section.

To navigate to the issue in the JSON report, just use your text editor’s find feature, searching for the finding’s name string from the previous step (e.g., “iam-inline-role-policy-allows-NotActions”). We’re interested in both the items and the path below:

Identify the finding’s rule definition

It’s often helpful to view the rule that caused ScoutSuite to flag a finding, even if not doing JSON extraction. Just Google or Github search for the finding name along with “rules”. The conditions section has the information we are looking for:

Re-implement the rule within jq’s filter to extract the data

To re-implement the rule in jq, we need to iterate over the role configuration data and match roles that have inline policies with an “Effect” key set to “Allow” and that have a “NotAction” key with any value, as specified in the above rule file’s conditions statement. To do that, we’ll create a filter that iterates over the path provided in the first (findings) section of the report, reproduced here:

iam.roles.id.inline_policies.id.PolicyDocument.Statement.id

jq Filtering

JSON organizes object elements using arrays, which is a detail that the jq filter parameter needs to have specified for it to parse the schema tree properly. In the above path, each time “.id” is present is a hint that we need to specify an array using square brackets.

So, to iterate through all of the roles, we can provide the following command. For this example, I’ll show the execution of cat as well, but that will be omitted in future commands and assumed:

cat scoutsuite_results.pretty.js | jq '.services.iam.roles[]'

This will return the entire configuration schema for all roles that were scanned.

Keep in mind that the final value we’re trying to extract are the role ARNs, which live directly beneath each role (.services.iam.roles[].arn). But we need to query the inline policy statements that are deeper in the JSON schema…

To do that, we’ll be using the pipe operator. 

The pipe operator works within the jq filter specification much in the same way that it does in the shell. We’ll pass all the roles to a second filter statement using the pipe and then we’ll use the select function to perform the rule’s logic. These pipes are operating inside of the filter’s single-quotes, and not in the shell:

jq '.services.iam.roles[] | select(.inline_policies[].PolicyDocument.Statement[].Effect == "Allow") | select(.inline_policies[].PolicyDocument.Statement[].NotAction != null)'

This will return the full JSON configuration schema of each of the roles that match both of our conditions. If we just want to return the ARNs, which is a property under each role, we can just pipe the filter again to “.arn”:

jq -r '.services.iam.roles[] | select(.inline_policies[].PolicyDocument.Statement[].Effect == "Allow") | select(.inline_policies[].PolicyDocument.Statement[].NotAction != null) | .arn'

Hooray! We’ve retrieved the ARNs we were interested in and can report them now so the issue can be fixed.

Pitfalls & Protips

• The -r flag in the jq commands above specifies raw output, so that we don’t have a bunch of double quotes to remove from each result line.
• Many other functions (e.g., “contains”, “startswith”) are provided by jq. The jq manual is your friend!
• The use of double quotes to enclose jq filter statements is a bad idea. Always use single quotes.
• If you want to pass shell variables to jq filters, you need to do so using the –arg flag and not by using double quotes. For example, the following ugly bash loop first extracts the items from the first section of the report and saves them to a bash variable called “snapname”. It then iterates through all the snapnames, providing them as an argument to the jq filter’s “snapshots” array, which parses the second (configuration) section of the report to extract their ARN:

for snapname in $(cat scoutsuite_results.pretty.js| jq -r '.services.ec2.findings["ec2-ebs-snapshot-not-encrypted"].items[]' | sed 's/.*snapshots.//g' | sed 's/\..*//g');
do
    cat scoutsuite_results.pretty.js| jq -r --arg SNAPNAME "$snapname" '.services.ec2.regions[].snapshots[$SNAPNAME].arn|select(. != null)';
done

More From White Oak Security

White Oak Security is a highly skilled and knowledgeable cyber security and penetration testing company that works hard to get into the minds of opponents to help protect those we serve from malicious threats through expertise, integrity, and passion. 

Read more from White Oak Security’s pentesting team.