Overview
Vulnerability scanning is a critical component of any cybersecurity program. An organization can identify and remediate security weaknesses in their systems before they can be exploited by malicious actors.
Traditional vulnerability scanning requires the installation and maintenance of software agents on servers. Moreover, supporting infrastructure such as gateways, routing rules and firewall gaps must be introduced to ensure agent traffic reaches the manager.
When thousands of servers spread across multiple cloud accounts are involved, one could see where pain points might arise.
Agentless vulnerability scanning offers an alternative approach by using cloud APIs to copy the storage of the target server and analyze it in a secure environment. This has no impact on the server’s performance, neither does it require any of the above-mentioned infrastructure.
The project
So how do we access the storage of thousands of servers, spread across multiple cloud accounts, and analyze it in a single environment?
We will use a combination of cloud-native services to orchestrate the scanning process:
- AWS Step Functions to orchestrate the workflow
- AWS Lambda to execute custom logic like creating snapshots, copying them and starting the analysis
- Elastic Container Service (ECS) to run the analysis containers
- Amazon S3 to temporarily store the results
- Database to persist the results
1. Start
- We assume a role into the source account and asynchronously create encrypted snapshots of the target servers’ root EBS volumes. These are the volumes where the operating system is installed.
- We share the snapshots to the service account - in cloud terms this means we give the service account permission to use the snapshots in the source account and create volumes from them.
2. Scale
- While the snapshots are being copied, we provision the necessary number of analyzers. These are EC2 instances responsible for mounting the volumes and hosting the vulnerability scanning docker containers.
3. Access data
- Once the analyzers are ready and the snapshots are copied, we create EBS volumes and attach them to the analyzers.
- Once EBS volumes are created, the snapshots in the source account are deleted (as they’re no longer needed) to reduce costs.
4. Analyze
- With the analyzers ready and volumes attached, we provision docker containers on each analyzer to perform the vulnerability scan.
5. Export
- A poller checks the status of the analyzers, and once they’re done, it copies the results to S3.
- Each analyzer instance is terminated once the results are exported.
6. Load
- After all analyzers are terminated, the results are loaded into a database to be presented through a web interface.
The software
Vulnerability scanning
Since we’re only focusing on the pipeline and orchestration, we’ll use existing tools for the vulnerability scanning part.
Wazuh is an open-source security monitoring platform that can perform vulnerability scans (and much more) on endpoints. We create a docker image with Wazuh pre-configured to scan the required volumes. It is important to prevent Wazuh from scanning the host system and only focus on the attached volumes.
Later, this docker image is used as the template for the ECS tasks.
Database
To store the results, we use Neo4j, a graph database, allowing us to create relationships between the vulnerabilities and the servers.
This allows us to query the data in a more natural way and visualize it in a graph format. Additionally, it opens up the possibility of running graph queries to identify patterns or risks.
Hold up…
If you can automatically scan thousands of servers without the grunt work, why isn’t the whole world doing it?
There’s always a catch… In the linux world it’s called a pseudo file system (also known as /proc).
When you copy a server’s storage, you’re not copying the server’s memory. This means that any running processes, open files, network connections, etc. are not included in the snapshot. Your scans only get visibility into the server’s configuration and installed software, but not into its runtime state.
Engineers caught up on this and raised it as a red flag. This is why big players in agentless such as Wiz have now introduced agents.
News
The elephant enters the room…
On Nov 27, 2023 AWS announced the release of Amazon Inspector Agentless, a service that takes snapshots of servers and analyzes them in the background.