Millions of Binaries Later: a Look Into Linux Hardening in the Wild

TL;DR

In this post, we explore the adoption of Linux hardening schemes across five popular distributions by examining their out-of-the-box properties. For each distribution, we analyzed its default kernel configuration, downloaded all its packages, and analyzed the hardening schemes of their enclosed binaries. Our dataset includes the OpenSUSE 12.4, Debian 9, CentOS and RHEL 6.10 & 7 distributions, as well as the Ubuntu 14.04, 12.04, and 18.04 LTS distributions. Our findings confirm that even basic hardening schemes, such as stack canaries and position independent code, are not fully adopted. The situation is even worse when it comes to other compiler protections like stack clash hardening, which recently came into the spotlight due to last month’s systemd vulnerabilities. However, not all is hopeless. A good portion of shipped binaries have basic mitigations in place, and the numbers have improved from version to version. Our experiments indicate that Ubuntu 18.04 shows the largest adoption of OS and application-level mitigations, followed by Debian 9. On the other hand, OpenSUSE 12.4,  CentOS 7 and RHEL 7 also deploy common hardening schemes, and show wider adoption stack-clash mitigations while shipping a much more tight-knit set of packages by default.

Introduction

Shipping quality software is hard. Despite the vast numbers of sophisticated static and dynamic analysis1 toolchains, and the significant compiler and programming language developments of the past years, modern software is still plagued with vulnerabilities that are constantly exploited by attackers for profit. The situation is even worse when it comes to complex and rapidly changing software ecosystems that involve legacy code. In such cases, not only are we faced with the everlasting problem of finding possibly exploitable bugs, but we also are constrained by the hard limits of backwards-compatibility, which often dictate that we are stuck with a piece of code that is known to have limitations, or even worse, be vulnerable or buggy.


This is where hardening techniques come into play. Since we cannot prevent certain types of bugs, we might as well make the attacker’s life harder, and work around the problem by preventing or hindering the exploitation of these bugs. Such software hardening defenses are currently deployed in all modern operating systems, however vary greatly in complexity, effectiveness an overhead: from stack canaries and ASLR to full-fledged CFI and ROP defenses, hardening schemes are not all designed, implemented, or used equally. In this blog post, we will take a deep look into the adoption of such defenses by the most popular Linux distributions, examining their default kernel configurations as well as the properties of the binaries distributed through the package management systems of each distribution.


CVEs vs Security

We all have seen articles bearing titles such as “most vulnerable applications of the year” or “most vulnerable operating systems”. Usually, these articles present statistics on Common Vulnerability and Exposures (CVE) entries, aggregated from sources like NIST’s National Vulnerability Database (NVD) and subsequently rank the compared applications or OSes based on the numbers of CVEs that have been published. Unfortunately, although very useful for keeping track of issues and informing vendors and users alike, CVEs alone do not tell us much about the underlying security properties of software.


To make this clearer, let us examine the total number of CVEs published over the past four years for the Linux Kernel, as well as for five of the most popular server-oriented Linux distributions, namely Ubuntu, Debian, Red Hat Enterprise Linux, and OpenSUSE.

Figure 1

What do we learn from the graph above? Is the number of CVEs per distribution indicative of the fact that one distribution might be more vulnerable than another? The answer is no. For instance, as you will see in this post, empirical evidence suggests that Debian has more hardening mechanisms in place compared to say, OpenSUSE or RedHat Linux, and yet it has the most CVEs. However, these CVEs do not necessarily denote weakened security: even in the presence of a CVE, it is not directly obvious if a vulnerability is exploitable. Severity score assignments do provide an indication of how likely a vulnerability is to be exploited but, at the end of the day, exploitability depends largely on the defenses present in the affected systems, as well as on attackers’ resources and capabilities. Moreover, the absence of CVE reports does not tell us anything about other unreported or unknown vulnerabilities that may be lurking, and differences in numbers may exist not because of differences in software quality but due to other factors, including the resources allocated to testing or the size of the user base. For our Debian example, the higher number of CVEs may simply indicate that Debian ships more software packages.

That said, it goes without saying that the CVE enumeration system provides us with useful information that enables us to build appropriate defenses: the better we understand how software fails, the more likely we are to pinpoint the possible ways attackers might exploit it, and design the appropriate detection and response mechanisms to defend against them. To this end, let us examine, in Figure 2, the categories (in aggregate, across all distributions) for the CVEs of Figure 1 (sourced from here) over the last 4 years. It is immediately obvious that the majority of the CVEs reported over the past years fall into the following categories (as defined by the cvedetails categorization): Denial of Service (DoS), Code Execution, Overflow, Memory Corruption, Information Exfiltration, and Privilege Escalation. Although many of the CVEs are counted multiple times across various categories, we notice that overall, the same core problems persist year-to-year. In the following of this post, we will evaluate the adoption of different hardening schemes that are deployed by mainstream Linux distributions to prevent exploitation of the above vulnerabilities.


Figure 2

Goals

With this survey, we set out to answer the following questions:

  • What are the security properties of different Linux distributions? What hardening defenses are in place when it comes to the kernel, and for user-space applications?
  • How has the adoption of hardening mechanisms changed over time for different distributions?
  • What are the average package and library dependencies for each distribution?
  • What hardening defenses are in place for each binary?

Distribution Selection

It turns out that finding accurate statistics on adoption and deployment of different distributions is not trivial, since in most cases the number of downloads is not indicative of the number of real deployments. That said, UNIX variants represent the majority when it comes to server/infrastructure deployments, with Linux variants having a continuously growing market share2. Thus for our study, we focused on the different Linux distributions that are available out-of-the box in Google’s Cloud Platform (GCP). In particular, the Operating Systems we selected in our study were:


Distribution / Release

Kernel

Build

OpenSUSE 12.4

4.12.14-95.3-default

#1 SMP Wed Dec 5 06:00:48 UTC 2018 (63a8d29)

Debian 9 (stretch)

4.9.0-8-amd64

#1 SMP Debian 4.9.130-2 (2018-10-27)

CentOS 6.10

2.6.32-754.10.1.el6.x86_64

#1 SMP Tue Jan 15 17:07:28 UTC 2019

CentOS 7

3.10.0-957.5.1.el7.x86_64

#1 SMP Fri Feb 1 14:54:57 UTC 2019

Red Hat Enterprise Linux Server 6.10 (Santiago)

2.6.32-754.9.1.el6.x86_64

#1 SMP Wed Nov 21 15:08:21 EST 2018

Red Hat Enterprise Linux Server 7.6 (Maipo)

3.10.0-957.1.3.el7.x86_64

#1 SMP Thu Nov 15 17:36:42 UTC 2018

Ubuntu 14.04 (Trusty Tahr)

4.4.0140-generic

#166~14.04.1-Ubuntu SMP Sat Nov 17 01:52:43 UTC 20

Ubuntu 16.04 (Xenial Xerus)

4.15.01026-gcp

#27~16.04.1-Ubuntu SMP Fri Dec 7 09:59:47 UTC 2018

    Ubuntu 18.04 (Bionic Beaver)

4.15.01026-gcp

#27-Ubuntu SMP Thu Dec 6 18:27:01 UTC 2018

Table 1

Analysis

In order for us to analyze the hardening defenses deployed by the different OSes outlined above, we examine their vanilla kernel configuration, as well as the properties of the packages that are available through each distribution’s package manager out-of-the-box. Thus, we only examined packages available from each distribution’s default mirrors, and omitted packages available from non-stable repositories (e.g., ‘testing’ mirrors in Debian) or third-party packages (e.g., NVIDIA packages available through non-default mirrors). Also, we did not consider custom kernel compilations or configurations that offer increased hardening.

Kernel Configuration Analysis

Using our analysis script based off an open-source kconfig checker, we examined the hardening config options that came out-of-the-box with the above distributions, and compared them against the list published by the Kernel Self Protection Project (KSPP). For each kernel hardening configuration option, Table 2 outlines the desired setting, and has a checkmark for all the distributions that conformed to the KSSP recommendation3.


Kernel Configuration Setting

Desired Value

SLES

Debian 9

CentOS 6.10

CentOS 7

RHEL 6.10

RHEL 7.6

Ubuntu 14.04

Ubuntu 16.04

Ubuntu 18.04

Significance

X86_SMAP

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Critical

STRICT_KERNEL_RWX

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Critical

RANDOMIZE_BASE

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Critical

RANDOMIZE_MEMORY

y

✔️

✔️

✔️

✔️

✔️

✔️

Critical

STACKPROTECTOR_STRONG

y

✔️

✔️

✔️

✔️

✔️

Critical

HARDENED_USERCOPY

y

✔️

✔️

✔️

✔️

✔️

Critical

LOCK_DOWN_KERNEL

y

✔️

✔️

✔️

Critical

STRICT_MODULE_RWX

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Critical

SECURITY

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Critical

SECCOMP

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Critical

STRICT_DEVMEM

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Critical

DEVKMEM

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Critical

X86_INTEL_UMIP

y

✔️

✔️

✔️

✔️

High

VMAP_STACK

y

✔️

✔️

✔️

✔️

High

SLAB_FREELIST_HARDENED

y

✔️

✔️

High

SLAB_FREELIST_RANDOM

y

✔️

✔️

✔️

✔️

High

FORTIFY_SOURCE

y

✔️

✔️

High

BUG_ON_DATA_CORRUPTION

y

High

HARDENED_USERCOPY_FALLBACK

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

SECURITY_DMESG_RESTRICT

y

✔️

✔️

High

SECURITY_YAMA

y

✔️

✔️

✔️

✔️

✔️

✔️

High

SECURITY_SELINUX_DISABLE

not set

✔️

✔️

✔️

High

SECCOMP_FILTER

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

ACPI_CUSTOM_METHOD

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

COMPAT_BRK

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

IO_STRICT_DEVMEM

y

✔️

✔️

✔️

✔️

✔️

✔️

High

LEGACY_VSYSCALL_NONE

y

High

USERFAULTFD

not set

✔️

✔️

High

LIVEPATCH

not set

✔️

✔️

High

BPF_JIT

not set

✔️

✔️

High

PAGE_TABLE_ISOLATION

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

RETPOLINE

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

X86_64

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

DEBUG_WX

y

✔️

✔️

✔️

High

SCHED_STACK_END_CHECK

y

✔️

✔️

✔️

✔️

✔️

High

MODULE_SIG

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

REFCOUNT_FULL

y

High

STATIC_USERMODEHELPER

y

High

COMPAT_VDSO

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

High

BINFMT_MISC

not set

High

PROC_KCORE

not set

High

MODIFY_LDT_SYSCALL

not set

✔️

✔️

✔️

✔️

High

KPROBES

not set

High

UPROBES

not set

✔️

✔️

High

DEBUG_FS

not set

High

BPF_SYSCALL

not set

✔️

✔️

High

USER_NS

not set

High

FTRACE

not set

High

ARCH_MMAP_RND_BITS

32

High

BUG

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Medium

THREAD_INFO_IN_TASK

y

✔️

✔️

✔️

✔️

Medium

MODULE_SIG_ALL

y

✔️

✔️

✔️

✔️

✔️

Medium

PAGE_POISONING

y

✔️

Medium

GCC_PLUGIN_RANDSTRUCT

y

Medium

HIBERNATION

not set

Medium

PROC_VMCORE

not set

Medium

HWPOISON_INJECT

not set

Medium

SLUB_DEBUG

y

✔️

✔️

✔️

✔️

✔️

Medium

SYN_COOKIES

y

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Medium

DEFAULT_MMAP_MIN_ADDR

65536

✔️

✔️

✔️

✔️

✔️

Medium

GCC_PLUGIN_LATENT_ENTROPY

y

Medium

DEBUG_LIST

y

✔️

✔️

✔️

✔️

✔️

Medium

DEBUG_CREDENTIALS

y

Medium

MODULE_SIG_FORCE

y

Medium

GCC_PLUGIN_STACKLEAK

y

Medium

SECURITY_LOADPIN

y

Medium

PAGE_POISONING_NO_SANITY

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Medium

PAGE_POISONING_ZERO

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Medium

SLAB_MERGE_DEFAULT

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Medium

X86_PTDUMP

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Medium

DEBUG_KMEMLEAK

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Medium

KEXEC

not set

Medium

LEGACY_PTYS

not set

✔️

✔️

✔️

✔️

✔️

Medium

IA32_EMULATION

not set

Medium

PROC_PAGE_MONITOR

not set

Medium

GCC_PLUGIN_STRUCTLEAK

y

Medium

GCC_PLUGIN_STRUCTLEAK_BYREF_ALL

y

Medium

DEBUG_SG

y

Medium

SLUB_DEBUG_ON

y

Medium

INET_DIAG

not set

Medium

X86_X32

not set

✔️

✔️

✔️

✔️

✔️

Medium

USELIB

not set

✔️

✔️

✔️

✔️

Medium

CHECKPOINT_RESTORE

not set

✔️

✔️

Medium

MEM_SOFT_DIRTY

not set

✔️

✔️

Medium

MMIOTRACE

not set

✔️

✔️

✔️

✔️

✔️

Medium

KEXEC_FILE

not set

✔️

✔️

Medium

DEBUG_NOTIFIERS

y

Low

ZSMALLOC_STAT

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Low

PAGE_OWNER

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Low

BINFMT_AOUT

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Low

IP_DCCP

not set

✔️

Low

IP_SCTP

not set

Low

DEVPORT

not set

Low

NOTIFIER_ERROR_INJECTION

not set

✔️

✔️

✔️

✔️

✔️

Low

ACPI_TABLE_UPGRADE

not set

✔️

✔️

✔️

✔️

✔️

Low

ACPI_APEI_EINJ

not set

✔️

Low

PROFILING

not set

Low

GCC_PLUGINS

y

Low

MMIOTRACE_TEST

not set

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

✔️

Low

MODULE_SIG_SHA512

y

✔️

✔️

✔️

Low

Total

SLES

Debian 9

CentOS 6.10

CentOS 7

RHEL 6.10

RHEL 7.6

Ubuntu 14.04

Ubuntu 16.04

Ubuntu 18.04

Critical (out of 12)

9

11

4

11

4

11

8

12

12

High (out of 37)

14

16

15

12

15

12

11

18

18

Medium (out of 37)

11

11

15

14

15

14

10

10

10

Low (out of 14)

5

5

6

6

6

6

6

5

5

Total

39

43

40

43

40

43

35

45

45

Table 2


Overall, we notice that newer kernels have stricter settings out-of-the-box. For instance, CentOS 6.10 and RHEL 6.10 running a kernel version 2.6.32 lack most of the critical features implemented in newer kernels such as SMAP, strict RWX permissions, base address randomization, or copy2usr protections. We should note at this point that, although many of the configuration options presented in Table 2 are not present for older kernel versions and are in reality not applicable, we count them as having the mitigation not present, and do not differentiate the two – likewise, if a configuration option is not present for a given kernel version and the desired setting is “not set”, this is still counted as a sane configuration. Another point we need to address when interpreting the above results is that some of the kernel configuration that increase the attack surface for attackers can also be utilized to implement security defenses. Such examples include uprobes and kprobes, kernel modules and BPF/eBPF. Our recommendation is to use the above mechanisms to actually provide defense-in-depth, as they are not trivial to exploit and their abuse already requires malicious actors to have a foothold on the system. However, if enabled, the system administrator must actively monitor for their abuse.

Examining further the entries of Table 2, we observe that modern kernels provide several options that can hinder exploitability of vulnerabilities like information leaks and stack/heap overflows. However, we notice that even the latest commodity distributions do not yet deploy more sophisticated defenses (e.g., shipping with grsecurity patches), or state-of-the art defenses against code reuse attacks (e.g., combining randomization with R^X schemes for code). To make matters worse, even these more advanced defenses do not protect against the full spectrum of attacks. Thus, it is critical for system administrators to complement sane configurations with solutions that offer runtime detection and prevention of exploits.

Application Analysis

It comes as no surprise that different distributions have different package characteristics, compilation options, library dependencies, etc. Not only that, but differences exist even for distributions with close ancestry and packages with small amounts of dependencies (e.g., coreutils in Ubuntu vs. Debian). In order for us to evaluate the differences in the Linux distributions available in GCP, we downloaded all the available packages, extracted their contents, and analyzed their binaries and dependencies. For each package, we kept track of the other packages upon which it depends, and for each binary we kept track of its library dependencies. We outline our findings in this section.

Distribution Packages

In total, we downloaded 361,556 packages across all distributions, only fetching the packages available via the default mirrors. From the available packages of each distribution , we ignored packages that did not contain ELF executables, such as source packages, fonts, etc. After filtering out irrelevant packages, we analyzed a total of 129,569 packages, containing a total of 584,457 binaries. The total packages and binaries analyzed per distribution are presented in Figure 3.

                                                                  Figure 3



We notice that the more modern a distribution is, the more packages and binaries are added, which comes as no surprise. That being said, Ubuntu and Debian packages include many more binaries (both executables as well as dynamic modules and libraries) than those of CentOS, SUSE and RHEL, which is potentially impactful on Ubuntu and Debian’s attack surfaces4. This is particularly important given the fact often packages have dependencies to other packages, and thus, a vulnerability in a binary of a given package can affect many parts of the software ecosystem, similarly to how a vulnerable library might possibly affect all binaries importing it. As a point of reference, we present the distribution of the number of dependencies per package for the different OSes in Figure 4:



  Figure 4



We notice that for almost all distributions, 60% of the packages have at least 10 dependencies. Moreover, we notice that a few packages have large numbers of dependencies (> 100). The same applies to the reverse dependencies of a package: as expected, there are few packages which are used by many packages across the distribution, and therefore vulnerabilities in those select few are of high risk. As an example in the following table, we present the top 20 packages with the most reverse dependencies for SLES, Centos 7, Debian 9, and Ubuntu 18.04 (each cell denotes the package and the number of reverse dependencies).


SLES 12.4

CentOS 7

Debian 9

Ubuntu 18.04

bc (3768)

glibc-0:2.17-260.el7_6.3.x86_64 (7065)

libc6 (23246)

libc6 (21592)

glibc (3667)

glibc-0:2.17-260.el7.i686 (7036)

libstdc++6 (7703)

libstdc++6 (6575)

dconf (1336)

libgcc-0:4.8.5-36.el7.x86_64 (2433)

libgcc1 (7184)

libgcc1 (5683)

at (1118)

libstdc++-0:4.8.5-36.el7.x86_64 (2127)

zlib1g (3416)

libglib2.0-0 (3037)

ed (1053)

bash-0:4.2.46-31.el7.x86_64 (1664)

libglib2.0-0 (3309)

zlib1g (2485)

sed (913)

glib2-0:2.56.1-2.el7.x86_64 (1190)

libgmp10 (2079)

libgmp10 (1926)

file (894)

zlib-0:1.2.7-18.el7.x86_64 (1074)

libx11-6 (1962)

libx11-6 (1576)

rpm (807)

libX11-0:1.6.5-2.el7.x86_64 (638)

libqt5core5a (1751)

libqt5core5a (1486)

pkg-config (746)

cairo-0:1.15.12-3.el7.x86_64 (597)

libcairo2 (1614)

ruby (1315)

coreutils (731)

openssl-libs-1:1.0.2k-16.el7.x86_64 (597)

libgdk-pixbuf2.0-0 (1547)

nodejs (1298)

perl (626)

gdk-pixbuf2-0:2.36.12-3.el7.x86_64 (575)

libpango-1.0-0 (1483)

libqt5gui5 (1123)

gcc (608)

pango-0:1.42.4-1.el7.x86_64 (571)

ruby (1452)

dpkg (1045)

python (542)

atk-0:2.28.1-1.el7.x86_64 (495)

nodejs (1434)

libqt5widgets5 (984)

xli (389)

libxml2-0:2.9.1-6.el7_2.3.x86_64 (484)

libqt5gui5 (1362)

libghc-base-dev-4.9.1.0-d28d6 (898)

tk (375)

perl-4:5.16.3-294.el7_6.x86_64 (463)

libpangocairo-1.0-0 (1295)

libgdk-pixbuf2.0-0 (892)

gd (309)

python-0:2.7.5-76.el7.x86_64 (463)

libatk1.0-0 (1202)

python (874)

bash (290)

freetype-0:2.8-12.el7.x86_64 (417)

libqt5widgets5 (1180)

libgtk2.0-0 (869)

systemd (236)

systemd-0:219-62.el7.x86_64 (401)

libatomic1 (1109)

libgtk-3-0 (795)

info (229)

libgcc-0:4.8.5-36.el7.i686 (393)

libxml2 (1101)

libcairo2 (793)



Table 3



Interestingly, although all the OSes analyzed were built for x86_64 architecture, and the majority of the packages also have a specified architecture of x86_64 and x86, we discovered that the packages often contained binaries for other architectures, as seen in Figure 5.



                                                                           Figure 5



In the following section we will dive deeper into the characteristics of the binaries analyzed.

Binary Hardening Statistics

At absolute minimum, we wanted to examine a basic set of hardening options for the binaries at hand. Several Linux distributions ship with scripts that perform such hardening checks. For instance, Debian/Ubuntu exposes a hardening-check script that provides such useful information. Here’s an example:


$ hardening-check $(which docker)
/usr/bin/docker:
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: no, only unprotected functions found!
 Read-only relocations: yes
 Immediate binding: yes

We notice that the above script checks for the following 5 hardening features:

  • Position Independent Executable (PIE): Indicates if the text section of the program can be relocated in memory, so as to achieve randomization if ASLR is enabled in the kernel.
  • Stack Protected: Whether stack canaries are in place to guard against stack smashing attacks.
  • Fortify Source: Whether unsafe functions (e.g., strcpy) are replaced with their safer counterparts and calls that are verifiable at runtime replace their non-runtime-checked counterparts  (e.g., __memcpy_chk vs memcpy).
  • Read-only relocations (RELRO): Whether relocation table entries are marked as “read-only” if they were resolved before execution begun.
  • Immediate binding: Whether the runtime linker resolves all relocations before starting program execution (this is equivalent to full RELRO).

Are the above hardening mechanisms enough? Unfortunately, not at all. There are known techniques to bypass all the above defenses, but the more hardening defenses are in place, the higher the bar for the attacker. For instance, techniques to bypass RELRO mitigations are hindered if PIE and immediate binding are present. Likewise, full ASLR requires attackers to perform additional work to construct a working exploit. However, for sophisticated attackers, the above mitigations are something to expect — their absence in essence opens the door to your system much faster. Therefore, it is critical that at minimum they are put in place.

For our analysis, we wanted to examine what percentage of the binaries in the distributions under examination had these mitigations in place, as well as the presence of three more mitigations:

  • Non executable bit (NX), which prevents execution on any region that should not be executable such as the stack heap etc.
  • RPATH/RUNPATH, which designates the run-time search path used by the dynamic loader to find the appropriate libraries. The former is a sine qua non for any modern system – its absence allows attackers to arbitrarily write a payload in memory and execute it as is. For the latter, non-sane runtime path configurations are responsible for introducing untrusted code that may lead to series of problems (e.g., privilege escalation as well as other issues).
  • Stack Clash protection, which provides protection against attacks that cause the stack to overlap with other memory regions (e.g., heap). Given the recent exploits abusing stack clash vulnerabilities in systemd, we thought it relevant to include this mitigation in our dataset.

So, without further ado, let us jump into the numbers: Tables 4 and 5 present the summary of our analysis for the executables and libraries of the different distributions, respectively.:

  • We notice that with respect to NX enforcement, it is almost omnipresent, with few exceptions. Particularly, we notice slightly lower adoption in Ubuntu and Debian distributions compared to CentOS, RHEL and OpenSUSE.
  • When it comes to stack canaries, we notice that they are missing from significant portions of the total executables, especially in distributions with older kernels, however there has been progress in latest Centos, RHEL, Debian and Ubuntu distributions.
  • With the exception of Debian and Ubuntu 18.04, PIE support is poor in most distributions.
  • Stack clash mitigations are scarce in OpenSUSE, Centos 7 and RHEL 7 and practically non-existent elsewhere.
  • All distributions with modern kernels have some support for RELRO, with Ubuntu 18.04 leading the way, and Debian at second place.

As mentioned earlier, the metrics presented in this table are averages across all different versions of a binary. Thus, we observe differences in say, PIE statistics compared to examining only the latest version of a binary (see for instance Debian’s progress with PIE packages). Moreover, whereas most distributions generally examine if at least some functions are fortified in a binary when counting the percentage of fortified binaries, for our analysis we measure a true percentage of the fortified functions. Therefore, if for a binary 5 out of 50 fortifiable functions are fortified, we will assign it a score of 0.1, as 10% of the functions are fortified.


OS

Canaries

Stack

Clash

NX

PIE

RELRO (full)

RELRO (partial)

RUNPATH

FORTIFIED

SLES 12.4

55.34

1.15

99.29

6.33

7.30

89.84

85.62

41.04

CentOS 6.10

55.73

0.00

99.66

2.18

2.96

6.26

98.73

41.11

CentOS 7

84.55

0.15

99.83

5.83

11.80

88.00

98.44

41.51

RHEL 6.10

54.32

0.00

99.43

2.20

2.85

6.87

98.78

37.31

RHEL 7.6

82.26

0.13

99.56

5.57

10.93

88.89

98.45

36.66

Debian 9

73.03

0.01

97.78

38.04

35.34

62.15

88.48

31.10

Ubuntu 14.04

53.85

0.01

96.12

4.04

9.53

85.82

99.08

44.12

Ubuntu 16.04

75.58

0.01

95.84

4.76

12.01

85.09

97.54

43.54

Ubuntu 18.04

85.62

0.01

93.43

37.77

56.81

41.46

88.54

41.00

Table 4: Hardening Properties of Fig. 3 Executables
(Percentage of Adoption)



OS

Canaries

Stack

Clash

NX

RELRO (full)

RELRO (partial)

RUNPATH

FORTIFIED

SLES 12.4

49.46

0.37

99.49

6.41

90.41

85.05

35.29

CentOS 6.10

53.15

0.00

99.91

2.72

5.37

98.66

38.07

CentOS 7

83.12

0.06

99.84

10.49

89.39

99.06

38.10

RHEL 6.10

52.28

0.00

99.91

2.51

5.96

98.71

38.06

RHEL 7.6

82.83

0.07

99.89

9.45

90.47

98.88

38.31

Debian 9

74.46

0.01

96.94

33.17

63.88

89.02

33.00

Ubuntu 14.04

43.37

0.01

94.60

9.40

85.00

98.79

38.89

Ubuntu 16.04

67.37

0.01

93.29

12.23

83.76

96.73

35.95

Ubuntu 18.04

81.96

0.01

89.57

32.01

65.97

88.35

33.81

Table 5:Hardening Properties of Fig. 3 Libraries
(Percentage of Adoption)



So are things progressing? They definitely are: this can be seen from statistics on individual distributions (e.g., Debian), as well as from the above tables. As a use case, we present a summary of the hardening mechanism adoption for three consecutive Ubuntu LTS distributions in Figure 5 (we omit stack-clash statistics due to the low adoption). We notice that consistently more binaries have stack canaries from version to version, whereas significantly more binaries in 18.04 are shipped with full RELRO compared to previous versions.



                                    Ubuntu Executables                                          Ubuntu Libraries

Figure 6



Unfortunately, multiple executables are present still in the different distributions, having none of the above mitigations. For instance, taking a quick look at Ubuntu 18.04, we notice the ngetty binary (which is a getty replacement) ships without canaries, NX, PIE and fortified source, and so do the mksh and lksh shells, the picolisp interpreter as well as the packages nvidia-cuda-toolkit (which is a popular package for creating GPU-accelerated applications like machine learning frameworks) and klibc-utils. Similarly, the mandos-client binary (which is an administrative tool allowing unattended reboots with encrypted root filesystems), as well as the rsh-redone-client (a re-implementation of rsh and rlogin), ship without NX and have SUID set :(. Likewise, several suid-binaries lack basic protections like stack canaries (e.g., the Xorg.wrap binary of the Xorg package).

Summary & Closing Remarks

In this post, we highlighted several characteristics of modern Linux distributions when it comes to hardening. Our analysis demonstrated that the latest Ubuntu LTS distribution (18.04) has on average the strongest OS-level and application mitigations amongst the distributions with newer kernels that we examined such as Ubuntu 14.04, 12.04 and Debian 9. The examined CentOS, RHEL and OpenSUSE distributions in our dataset, however, ship a tighter set of packages by default, and in their latest versions (for CentOS and RHEL) have higher numbers of stack clash adoption compared to their debian-based rivals (Debian and Ubuntu). Comparing CentOS and RedHat versions, we notice large improvements in deployment of stack canaries and RELRO from versions 6 to 7, however on average more functions are fortified in CentOS than in RHEL. Overall, we notice that the most progress to be made across all distributions is with regards to adoption of PIE, which, with the exception of Debian 9 and Ubuntu 18.04 is below 10% for the binaries of our dataset.  

Finally we should note that while custom analysis such the above is possible, there is a plethora of security tools (e.g., Lynis, Tiger, Hubble) that may be used for analysis of a system and to prevent insecure OS and application configurations. Unfortunately, even with hardening mitigations and sane configurations, vulnerabilities eventually will lead to exploitation. This is why we strongly believe that it is vital to have solid real-time monitoring and prevention of attacks, by focusing and preventing exploitation patterns. We will explore such exploitation patterns in future posts, as well as how one defend against them. Stay tuned!



Notes

1. Static analysis tools examine code properties without executing the software, by only examining the source code or generated binary. Dynamic techniques require execution of a piece of software and perform runtime analysis.

2. W3techs reports that some UNIX variant is used by 69.2% of all websites, and the percentage is even larger when it comes to supercomputers. A multitude of other sources (e.g., 1, 2, 3) verify this trend, which is especially true for container deployments — even back in 2016, Ubuntu alone was reported to have over 60 million images launched by docker users.

3. For a thorough explanation of the utility of the above settings, the interested reader may refer to the KSPP project or to our kernel configuration glossary. In future posts, we will elaborate more on why many of these came to exist, as well as on possible ways to exploit the system in their absence.

4. We should note that the numbers presented include all binaries across different versions of a software package. For instance, a dynamic module used by Python will be analyzed multiple times, separately for python-2.6, 2.7 etc. Our averages are presented across all packages.