Reviewing Automated HTTP Screenshot Tools
Thanks for continuing with us in our Screenshot Tool blog series, where White Oak Security reviews a few of the top HTTP screenshot tools that are currently available for penetration testers or bug bounty hunters. Each part is covering the available features within the highlighted tool, how easy it is to use, and any problems that may occur. At the end of the Screenshot Tool series, we will put all the tools to the test to compare their effectiveness against a set of metrics, to find the most useful tool.
If you’ve read part 1, part 2, or part 3 – then feel free to jump over the criteria section down to the selected tool section, Part 4: Gowitness.
Screenshot Tools
We, at White Oak Security, use lots of tools – be sure to check out some of our other Gowitness related blogs too! We use HTTP screenshotting tools on external, internal, and red team engagements to quickly analyze the footprint of web servers and services exposed across the target environment.
Below are the items that represent the criteria for how I define ‘usefulness’ for a given HTTP screenshot tool. These criteria are skewed towards attack surfacing reconnaissance and penetration testing techniques.
Note: this series was written in early 2022.
SOURCES & CUSTOMIZATION
- Are common network scanner output formats supported as input sources by the tool? (Nmap/Masscan/Nessus)
- Can we append a second scan to the results of the first scan within the Database?
- Can you set a custom User-Agent?
- Can you set custom request headers and/or cookies?
PERFORMANCE
- How accurate is the tool? How many errors does it generate? Does it successfully capture screenshots (no white or blank screenshots)?
- How quickly does the tool perform the task? Can we speed the scan up without degrading the accuracy of the scan?
- Can we easily proxy the network traffic to http and socks proxies?
- Can we provide a navigation timeout against slow assets?
REPORTING
- Does the tool group or categorize similar hosts based on their screenshots?
- Does the tool provide useful output? Can it export to PDF/HTML/JSON/CSV?
- Is the UI/UX functional and enjoyable to use?
GENERAL
- Is the tool actively maintained with updates?
- Any Cross-Platform compatibility?
- How easy is it to build and install the project?
Gowitness
The tool we’ll be reviewing today was built with Golang and contains a promising set of features, the tool is Gowitness. The project’s description is as follows: “gowitness is a website screenshot utility written in Golang, that uses Chrome Headless to generate screenshots of web interfaces using the command line, with a handy report viewer to process results.”
Gowitness Notable Features
Gowitness has several features which make it a standout within the group of HTTP screenshot tools. The most notable feature is that gowitness can perform differential comparison matching across the set of screenshots that are captured using perception hashing. In short, if you capture the screenshots for a large number of similar web assets (e.g., services that return 403 forbidden, IIS/Apache default pages, errors) they can be easily grouped together for a quicker analysis. Gowitness provides the ability to initiate scans from within the CLI and provides a clean HTML UI for analyzing the report.
A snapshot of the CLI arguments can be seen below which provides a window into the tool’s feature set.
Gowitness Sources & Customization
Gowitness supports standard input file formats that meet our goals with the ability to process nmap, flat files, and Nessus files. Gowitness can use custom user agents, network timeouts, navigation delays, and provides additional filtering options within each input source for more fine-grained control.
Gowitness Reporting
Gowitness’s report format is built-in to the web interface. The web UI supports pagination and searching which makes it easy to navigate the screenshots. I personally would like to see some additional filtering in the UI in the future. Gowitness also has the ability to export to JSON via the CLI.
Gowitness General Setup
Gowitness’s setup is very easy if you are familiar with a golang module environment and can be compiled from source from the included Makefile without any significant issues on Debian 11. Under the hood, gowitness is using the chromedp library to drive its HTTP automation via headless chromium. Cross-platform compilation is one of the perks of golang, which means gowitness should easily compile on windows systems from linux systems.
Gowitness Out-Of-The-Box Tests
Now that we’ve covered some of the high-level features of gowitness, it’s time to actually try the utility out. Gowitness’s installation process was a breeze if you have a golang environment already set up. Simply clone the repository, initialize the go modules, and then compile a binary using the makefile.
A sample scan was created and successfully executed.
Similar or duplicate hosts can be grouped together by enabling perception sorting. You’ll notice the abm.com screenshots are grouped together in order.
Overall, gowitness performed with a high degree of accuracy in its out-of-the-box configuration and I really don’t have many complaints about the tool. However, when I was running the performance test I noticed that gowitness would not successfully end its process and would be stuck running due to errors in several web applications it attempted to screenshot. Unlike EyeWitness/WitnessMe, when you kill the gowitness process it will at least clean up the remaining chrome processes that were open.
Gowitness Performance Improvements
Out of the box, gowitness executes its tests with a fairly high degree of accuracy, however, to boost this accuracy, we will need to make some additions to the code base. By default, a lot of these HTTP screenshot tools do not provide ‘enough’ information in the initial HTTP request used to capture the browser viewport and certain websites will deny these requests if headers are missing, or an incorrect user agent is used. If we can customize these preset settings, as well as include additional network timeout and delays, we can probably increase our success rate by a significant measure.
Headers
In an attempt to increase performance of the gowitness scan, we need to add support for custom request headers to ensure that pages can actually be successfully captured. Adding custom headers that a normal browser session typically sets (Accept, Accept-Encoding, Accept-Language) increased the accuracy of the preflight request by a significant measure. A pull request for this feature was created within the GitHub project.
Navigation Timeouts
One of the major issues with screenshot tools is that a page may take a long period of time to load content from external sources. These are typically images, scripts, and other multi-media content that may be loaded from a slow Content Delivery Network (CDN) or external server.
At a high level, any given screenshot tool performs the following actions:
- Navigate to the URL
- Wait for navigation to complete
- If navigation has not completed in under a set period of time, cancel the request (navigation timeout)
- If navigation is complete, optionally sleep for a short period of time to help load visual content (navigation delay)
- Capture the screenshot from the current viewport
This flow is used to provide a timeout for navigation, to prevent slow websites, as well as a simple delay once the page has been loaded to capture any scripts that might load images, or anything else that may occur on page load in order to capture the ‘best’ looking screenshot.
As I went through this blog post series, I identified a problem within this flow. Several websites will take a long period of time (60+ seconds) to complete their navigation page load because they are loading some external resource which delays their complete page load. However, the actual website has loaded a majority of its visual content even though it’s still waiting. This is visualized in a typical browser session via the seemingly endless loading stop icon.
Even though this page is still effectively loading, the main content that we care about has already been loaded so a screenshot should be captured. But most tools in their default configuration (or even with a timeout or navigation delay set) will fail to capture this page. The default logic most tools follow would signal that this website has actually failed to load any content due to network performance.
At the time of writing, the following website, https://www.triumphgroup.com, takes 86 seconds to fully load.
The visual content was properly loaded, but an image was waiting to be fetched from a third-party server. None of the tools in this blog post series were able to successfully capture that screenshot out of the box.
To solve this problem within gowitness, I came up with a solution that actually does handle this edge-case. Instead of immediately canceling the request when a timeout occurs, we will instead take a screenshot of the current viewport. This has a small negative downside where a page could still ‘fail’ and return a white viewport or an empty screenshot. In my opinion, the positives outweigh this small chance which can be filtered out with the perception hashing anyway. A pull request for this can be found here.
As a result, this modification slightly increased the total accuracy against the set of targets in our performance test which suffered from this issue.
This logic should probably be applied to every tool that captures screenshots, as it can cut down on a lot of false-negative timeout exceptions.
The Gowitness Preflight
Another performance gain I found within gowitness was an adjustment to the overall architecture of the main processor goroutine. Most HTTP Screenshot tools follow a similar architecture pattern; however, the concept of a ‘preflight’ check is a bit unique for gowitness. The preflight check exists in order to determine whether a host is “live” and responsive. If that host is live, the preflight check additionally retrieves response data from this connection for additional reporting/quality of life data points.
At the time of writing, gowitness performs the following actions against every target:
- Preflight – An HTTP connection to the target host is established in order to retrieve the response body, response headers, TLS certificate details, and perform some technology fingerprint analysis.
- Store Preflight – The data from the ‘preflight’ stage is saved to the database.
- Screenshot – The application launches chromium headless to capture a screenshot of the viewport.
- Perception Hash – A hash of the screenshot is calculated for similarity comparison analysis
- Write Screenshot – The raw image is saved to disk
However, if any of the stages fail the URL will be marked as ‘dead’ and operation will cease to continue. That means that in the context of gowitness, if a preflight request fails, the screenshot will never be captured. This means that a preflight request is the most important portion of the workflow, when really the emphasis should be on the ‘screenshot’ functionality. Now that we understand one of the pitfalls of this architecture pattern, we can actually easily modify this flow to ensure that the screenshot has a chance to succeed. From my perspective, we don’t always need the response body, but we do always want the screenshot.
So, if the preflight stage is the single most important piece of gowitness, why does it have a chance of failure? Preflight requests are just simple HTTP connections established to the server. If this connection is not the exact same HTTP request as the library driving screenshots (chromedp), then a gap exists between these two stages and they are not equal. The headless chromium driver will set browser headers, user-agents, and perform HTTP/2 transport upgrades automatically because it is emulating the browser. Therefore, we need to either emulate this exact functionality within the preflight request, or come up with a new solution.
Option 1: Completely get rid of the preflight
In a quick test, I removed the entire preflight stage from gowitness but left in all the performance improvements from the other stages. This resulted in the highest total number of successful screenshots that I had seen, which reaffirmed the fact that the preflight request was the biggest blocker for success within gowitness. Out of 1000 targets, it attempted to capture the screenshots of 999 hosts. This does not mean it was accurate against all 999, but at least a screenshot was attempted.
Option 2: Use chromedp for the preflight
Now that we have identified that chromedp has a higher degree of ‘success’ with just screenshot captures without any preflight in the way, what if we used chromedp as the collection agent for the response body, response headers, TLS certificate, and the input for fingerprinting?
Without going on a lengthy tangent of why this is not as easy as it seems, this idea does have promise. But, launching an entire chrome browser session just to get the TLS header may not be as performant as just grabbing the certificate from a HTTP request.
Option 3: Don’t fail on preflights
We know that chromedp can handle capturing screenshots without a high level of failure, but it’s a bit of a pain to use chromed for all the useful information we want from the preflight. So, what if we don’t use the ‘preflight’ stage as an indicator of whether or not a host is live, but instead just use the preflight stage as a collection source for the data points that it is already fairly good at capturing.
If we don’t capture the response body or TLS certificate for a web service, but we do still get a screenshot, in my mind that is still a successful event. In the current build, the opposite is true.
In the above code, we’ve changed the architecture of the processor a bit. First, we’ve added in an additional support for HTTP/2 upgrades within the preflight stage; attempting to navigate to the host with HTTP/2 and then defaulting back to HTTP/1.1. No change is needed for the headless chromedriver as that upgrade happens automatically by the browser (which is why it would have been nice to grab all this information from the chrome driver, but maybe in the future).
Next, you’ll notice that I simply removed the breaking `returns` within the first two preflight functions. The only tradeoff is that the overall “speed” of the run has been increased by a small margin as we are attempting a subsequent HTTP request even if the preflight fails. A pull request for this logic was created here.
Overall, these changes did significantly move the success rate for gowitness. In the future, I would expect the preflight stage of gowitness to be rewritten entirely (perhaps replaced with a faster HTTP client with built in HTTP/2 error handling).
Gowitness Best Performance Setup
The following is the best possible screenshot capture configuration I came up with during my analysis.
Configuration:
- Custom User-Agent
- Custom Headers
- Custom Navigation Context Timeout
- Custom Screenshot Delay
- Custom Preflight Architecture & HTTP/2 Support
- Prevent Screenshot Crashes
Part 4 – Gowitness Conclusion
Overall, gowitness provides an enjoyable experience when capturing HTTP screenshots. It has a ton of features built into the application and decent customization options to fit a wide variety of analysis scenarios. The project strives to do one job and do it well, and I think gowitness achieves this goal. There’s of course always room to grow and new features could be added, so if you use this please support and contribute to these Open-Source projects.
How will gowitness fare in our performance comparison between other major tools? At the end of this blog post series, we will find out!
Author’s Note: This blog series is my personal take on the state of screenshot tools. My network environment and physical setup may differ from yours when using these tools and you may notice your tool is more or less successful. I’ve done the best I can to be platform agnostic to provide the best environment for success. If you run into any errors, remember to check the individual project’s issues page for support.
Our Screenshot Tool Series
Here is Part 1 – EyeWitness blog post.
Check out Part 2 – WitnessMe blog post.
Read Part 3 – Snapback blog post here.
Stay tuned for Part 5 – Aquatone!
MORE FROM WHITE OAK SECURITY
White Oak Security is a highly skilled and knowledgeable cyber security testing company that works hard to get into the minds of opponents to help protect those we serve from malicious threats through expertise, integrity, and passion.