Meltdown and Spectre are such pervasive issues; they’re news on every major outlet. The security world is simultaneously in awe of the attack and panicking about remediation.What nobody is talking about is detection!
Remediation can be effective, and thanks to increased use of the public cloud, we can expect that applications running in the three major cloud providers are all going to be in good shape as long as they update to new AMI images that use patch kernels; although some unknown subset of applications that will suffer performance-wise due to the page table isolation that Linux is using to remediate.
However, we need detection strategies because there’s a very long tail in remediation. A lot of infrastructure runs on older kernels that are a challenge to upgrade — doing so would result in huge cost and stability risk. Existing mitigations (kernel upgrades and recompiling software) are probably not going to be a priority in many such environments — the risk of a successful attack will be outweighed by the cost of the remediation.
We at Capsule8 don’t think it’s feasible to do generic detection of these attacks at the network level, due to the nature of these attacks. But we’ve already developed practical strategies for detecting them, which we’ve implemented on Linux systems. By practical, we mean:
- Easy to deploy: There is no need to recompile software, or update a kernel.
- Stable: The detection runs in userland, without the need of a kernel module, etc.
- Efficient: The sensors run with minimal CPU overhead.
- Portable: The sensor works for any out-of-the-box version of Linux, dating back to the Linux 2.6 Kernel.
- Effective: There is an extremely low chance of a false negative in the majority of environments. In many environments, we’d expect no false positives. In some environments, there could be, but we’d expect them to be easily manageable.
In this series of blog posts, we’ll introduce our most basic detections, and provide Apache-licensed code you can use, on top of Capsule8’s Open Source Sensor. While these strategies would certainly work for Windows and other operating systems, we leave that as an exercise for the reader.
Today we’ll focus on a simple and effective detection strategy that can detect the most basic Meltdown attack with practically no chance of a false positive. Next week we’ll go through a more generic strategy that can detect both of these speculative execution attacks and more.
The Meltdown Vulnerability
Today we’re only going to concern ourselves with the Meltdown vulnerability, saving Spectre for next week.
The Meltdown vulnerability is the result of speculative execution, which is an optimization mechanism the processor uses to anticipate operations it may need to perform. Speculative execution optimizes execution by pre-emptively computing the results of instructions before necessarily knowing that those instructions should be executed. You can think of this as doing computational work just in case it becomes necessary. For instance, consider the following conditional operations:
IF user orders eggs special: calculate special price of eggs ELSE: calculate the price of normal breakfast
In the case of speculative execution, the processor will calculate both prices before knowing which price is correct based on the user’s order, discarding the results of the incorrect calculation. If the instructions in one of these calculations involve memory read operations, the speculative execution will affect memory mechanisms such as processor caching.
The Meltdown vulnerability specifically is due to the impact speculative execution can have on reading memory contents — not just caching of addresses where instructions are, but also the memory which those instructions access during execution. This occurs because in some cases Intel processors speculatively execute instructions before checking memory access privileges, for instance, checks to ensure that userland instructions do not refer to kernel memory. Thus the speculative execution of privileged memory-read operations affect the processor’s cache even though the instructions might be restricted from actually executing, because execution may not be permitted. This impact on the cache can be timed, and by measuring many successive repetitions of speculative execution, it is possible to conduct a side-channel attack to determine the contents of kernel memory from userland.
For much more detailed information on Meltdown, see the Meltdown paper.
Meltdown Detection Strategy
To understand and implement our first Meltdown detection strategy we are going to use the Linux Tracing facility.
The Linux kernel includes a suite of subsystems to enable performance and diagnostic tracing, including ftrace, tracepoints, uprobes, kprobes, perf, eBPF. These subsystems have been available for many years and have more recently become better understood and more accessible. For an introduction to Linux Tracing, start with this overview by Julia Evans. Then for a deeper dive, check out Brendan Gregg’s Linux Performance page.
Linux Tracing is designed for high-performance, non-intrusive introspection into running systems from user mode software (no kernel module required). It allows hooking of statically defined tracepoints as well as arbitrary symbols in kernel space for introspection purposes (such hooks are called kprobes). This facility is broadly supported in Linux 2.6 and later kernels. All of this makes it an ideal solution to build security monitoring tools and that is why we chose to use it as the basis of Capsule8 Sensor.
The Linux kernel supports an exceptions:page_fault_user tracepoint that generates events for user process page faults. This tracepoint, introduced in Linux kernel 3.13, includes fields for the faulting virtual memory address and an error code indicating the nature of the page fault. If we add a filter on the trace point, we can restrict the trace events generated to those that occur when a kernel memory address is attempted to be read from user mode and generates a protection fault.
Recall that the Meltdown vulnerability involves attempting to read a kernel memory address resulting in a segmentation violation signal being sent to the user process. The malicious process can handle the signal and proceed to retrieve the attempted memory from the cache. In order to perform any meaningful attack, an attacker would need to generate and handle a significant enough number of segmentation violations for kernel memory addresses that their activity is easily discernible from accidental program crashes.
Detecting Meltdown Using Capsule8
The open-source Capsule8 Sensor uses Linux Tracing under the hood to produce behavioral system security telemetry. While the currently supported telemetry events do not surface an event that would indicate exploitation in progress of Meltdown, we can use its lower-level EventMonitor interface to easily tap into a Linux tracepoint that indicates an attempted exploitation of Meltdown.
In order to improve performance of the tracepoint, we also attach a trace event filter for in-kernel evaluation. The filter matches when a user mode process causes a page protection fault trying to access a kernel-space memory address. This is exceedingly rare and has generated minimal false positives in our (albeit limited) testing.
[...] // // Look for segmentation faults trying to read kernel memory addresses // filter := "address > 0xffff000000000000" _, err = monitor.RegisterTracepoint("exceptions/page_fault_user", onPageFaultUser, perf.WithFilter(filter)) [...] [...]
Best of all, the performance impact of our meltdown exploitation detector is negligible since page faults are relatively rare and the kernel-based trace event filtering minimizes the number of events sent to userland for processing. Even on a heavily loaded system this simple Meltdown detector incurs significantly less performance impact than the Linux KAISER patches that mitigate exploitation of both Meltdown and Spectre.
The meltdown detector is designed to emit alerts to server logs and can also easily be packaged in a container and run as a Kubernetes DaemonSet to quickly deploy it across an entire cluster. We’ve supplied a Dockerfile and Kubernetes manifest to assist in this.
To quickly deploy our meltdown detector across your Kubernetes cluster, you can deploy our DaemonSet with the following single command:
$ kubectl apply -f https://raw.githubusercontent.com/capsule8/capsule8/master/examples/meltdown/capsule8-meltdown-detector.yaml
Our meltdown detector can also be run outside of a container by simply running the static binary. The example below shows the output while running one of the public meltdown proofs-of-concept:
$ sudo ./meltdown I0105 16:00:04.112232 1 main.go:46] Starting Capsule8 Meltdown Detector I0105 16:00:04.204312 1 main.go:66] Monitoring for meltdown exploitation attempts I0105 16:00:07.044165 1 main.go:84] pid 19599 kernel address page faults = 1 W0105 16:00:07.044222 1 main.go:87] pid 19599 kernel address page faults = 10 W0105 16:00:07.044892 1 main.go:90] pid 19599 kernel address page faults = 100 E0105 16:00:07.048324 1 main.go:93] pid 19599 kernel address page faults = 1000 E0105 16:00:07.081778 1 main.go:96] pid 19599 kernel address page faults = 10000
Stay tuned, there’s more….
Our simple detector tracks page faults for kernel memory addresses by process ID (PID) and alerts with low, medium, or high severity when a process generates numbers of events that cross defined event count thresholds. These thresholds are all triggered by published proof-of-concept exploits for Meltdown and are exceedingly unlikely to be triggered otherwise. The performance impact for this strategy is also virtually immeasurable, but it definitely will not cover all variations of these speculative execution techniques.
Stay tuned to the Capsule8 Blog — next week, we will go deeper into Meltdown and Spectre detection. We will describe a more general strategy that covers both exploitation of Meltdown and Spectre, including the exception-suppression approach using Intel TSX.
- How to Exploit systemd-journald
- A Basic Guide to Linux Monitoring
- A Practical Container Escape Exercise
- Adoption of Linux Hardening Schemes by 5 Popular Distros
- Intrusion Detection Systems
- Endpoint Security
- Endpoint Detection and Response (EDR)
- File Integrity Monitoring
- Intrusion Prevention Systems (IPS)
- Threat Hunting
Capsule8 is the only company providing high-performance attack protection for Linux production environments – whether containerized, virtualized, or bare-metal. Capsule8 liberates SecOps from managing a high volume of manual tasks, while being safe for even the busiest workloads, on the busiest networks.