“‘ESS-ca-pay’… that’s funny, it’s spelled just like the word ‘escape’!”
-A famous fish with ephemeral memory
Containers are more popular than well-understood. Container escapes are even less understood. This post is intended to solve the latter issue and demystify the heretofore arcane art of container escapology, even for people who feel confused by containers or uneducated about exploitation.
What are containers and why are containers?
An initial stab at defining containers led to some interesting hair-splitting discussions around specific words like “virtualization” and “emulation”. We won’t dwell on those here, so here’s the long-and-short of what you need to know about containers.
The big picture:
- They’re bundles of code dedicated to a specific task within an application
- They are logically isolated and communicate with each other through well-defined channels like APIs
- They share a kernel with their host (unlike VMs)
Why containers matter to businesses:
- Scalability: more efficient to scale applications based on demand
- Portability: they allow devs to test and deploy from any environment
- Speed: because containers need fewer computing resources, they can be spun up and shut down quicker and more efficiently than VMs (this quick spin-up can also mean less prolonged downtime, and therefore less loss of revenue)
The security potential of containers (spoiler alert: these are the assumptions that get broken in container escapes):
- Isolation: Containers are designed to be logically isolated from other services on the same host. Assuming isolation holds true, it provides some security by default (similar to sandboxing) that is seen as an improvement over retrofitting access control post-deployment.
- Immutability: Once containers have been deployed, they’re not meant to be accessed or modified. This is useful defensively because attackers want to gain access to systems and make changes to get the information they’re after, but can’t because container architecture aims to eliminate that possibility.
The reality: “More secure” != “100% secure,” and isolation boundaries aren’t guaranteed.
What are container escapes and why do they matter?
Anytime a system is designed for isolation, the first question from any red-teamer or attacker is, “how do I de-isolate myself from this?” After all, why would you hang out in this ephemeral and isolated space when you could frolic around the much more persistent and access-rich host OS?
The big picture: Container escapes are the attackers’ bus ticket out of Isolation Town.
- Attackers use system weaknesses to break an isolation boundary, like finding a hole in a fence
- For example, getting access to one container, then escaping into the host, grants the attacker access to all of the containers on that host
Why it matters: Even if the place you start doesn’t have sensitive data, the place you end up might.
- For example, a container running a service with penguin GIFs may share the same host as a container that handles sensitive information like social security numbers (SSNs) or passwords
- In isolation, it may not matter if the penguin GIF container is insecure, but if it lets an attacker access the host, even strong security on the SSN-handling container won’t matter
The counter: Container escapes are a multi-step process, presenting the opportunity to detect them at each step.
The lifecycle of a container escape
Container escape techniques involve many stages, and the particulars can vary. It’s not a simple matter of an attacker simply grabbing the nearest container and Leeroy Jenkins-ing their way into its host. They will take steps such as:
- Research the target looking for weaknesses as points of entry
- Execute commands remotely to gain a foothold into the target network
- Gather internal system information to determine next steps
- Exploit a vulnerability in the system to gain greater permissions and defeat the isolation boundary
- Install a method of maintaining persistent access to the network
The reality: Basically, a lot of asking “where am I?” and a lot of Googling.
The following scenario is one type of container escape among many possible escape routes.
An attacker discovers that a webpage’s form field allows them to run commands on the system hosting the site.
The problem: This is not supposed to be doable.
- Even if the web app doesn’t handle particularly juicy information, an attacker can use it to gain access to the broader production environment and find more sensitive data, like passwords or customer contact information
- Side note, this is also an incredibly common class of vulnerability
Execution and initial access
The attacker executes commands on the vulnerable website to gain remote access to the back-end of the application, hoping to access the underlying OS or network capabilities.
What’s happening: This is called an interactive shell.
- The shell allows the attacker to tell the compromised computer what to do from the comfort of their own home
- Remember, containers shouldn’t have anyone accessing them remotely once they’ve been deployed
With a connection established, the attacker tries to gather more information about the system they ended up on in order to determine their next steps.
The analogy: If you’ve ever done an escape room, what’s the first thing you do when you gain access to a new area? You take stock of your environment, trying to find bits of information that will help you identify the next actions to take in your escape attempt.
- Attackers need to go through those same motions: they may have opened a connection, but they won’t always know exactly where they landed.
The process: The attacker wants to find answers to: what kind of system am I on, what level of access do I have, and what weaknesses can I find?
- They figure out that they’re in a container, which is designed to be isolated from other systems, so they need to find a way to break that isolation
- Next, they look for weaknesses in the system, and ways to use them to escape from the container and move around the network
- A Google search tells them that the host OS’s kernel version has publicly-announced vulnerabilities that attackers can exploit (using a tool like Metasploit) to escape a container
This is the moment the attacker has been waiting for: the great escape.
Behind the scenes: When vulns are publicly announced, attackers start building bundles of code to try and exploit them
- Even as a non-privileged user, an attacker can download and execute an exploit to
- give them super-user privileges
- disable built-in security mechanisms
- get themself out of the container and into the host system
PSA: This is why applying updates quickly is so important.
Discovery on the host
Now an attacker will do some more discovery on the host to see where else they can pivot, and how they can exfiltrate data and maintain persistence.
- They will also look for sensitive data in other containers or parts of the host
Depending on the host’s version and vulnerabilities, the attacker has a few ways that they can try to maintain persistent access.
- Linux persistence mechanisms come in a variety of fun flavors like loadable kernel modules, rookits abusing eBPF, runtime backdoors, ptrace hooks, and more
The bottom line: Preserving isolation and immutability is essential to realizing the full benefits of containers.
- These assumptions aren’t guaranteed, and attackers are clever
- Detecting multiple stages of container escape attacks allows users to take swift remediation steps (like restarting a container) and contain an attack’s impact more efficiently
- A Compendium of Container Escapes by our Labs team colleagues Brandon Edwards and Nick Freeman presented at Black Hat USA 2019 talk “A Compendium of Container Escapes”)
- OOMyPod: Nothin’ To CRI-O-bout
- An Exercise in Practical Container Escapology
- Don’t Get Kicked Out! A Tale of Rootkits and Other Backdoors