How to Maintain Security for On-Demand Containers & Stateless Environments

On-demand containers and stateless environments are a class of infrastructure components that play a very important part in running the daily business operations. They can be used for business continuity tasks, for backups, batch processing, CI/CD jobs, transactional emails, auto-scaling, asynchronous workloads and many other reasons.

From a security perspective, securing those containers or environments can be a challenge; not because they need to be available at all times, but also to retain their integrity and confidentiality, as well.

Let’s explore the most effective ways to secure comprehensive protection of an increasingly non-homogenous technology stack that includes on-demand containers and stateless environments.

Restricting Storage

Although technically, stateless environments do not have persistent storage, they may require some storage for intermediate buffering. In that case, you should verify what exactly is required for the environment to operate without issues, and if saving something locally is required, you should use a temporary file system like tmpfs.

Running a container with that filesystem should be paired with size and permission quotas; for example:

$ docker run -d -it \
--name example \
--mount type=tmpfs,destination=/app,tmpfs-mode=1770,tmpfs-size=10485760 \

This will provision a 10mb temporary storage with a non world-writable permission.

After each command it's good practice to verify the effects of the operation.

$ docker container inspect example
"Mounts": [
"Type": "tmpfs",
"Target": "/app",
"TmpfsOptions": {
"SizeBytes": 10485760,
"Mode": 1016

Avoid Running as Root

This policy should be enforced for all containers as a general rule. How many times do you forget to change the user when running a container, which effectively breaks the principle of least privilege?

Adding a non-root user is a trivial matter as you just need to include a few lines on each Dockerfile; for example:

RUN groupadd -r user -g 999 && \
useradd -u 999 -r -g user -d /home/user -s /sbin/nologin -c "User" user && \
chown -R user:user /home/user
USER user

That way a lot of common escalation attacks can be mitigated.

Use Container Specific Operating Systems with explicit dependencies

Workloads for stateless environments should contain only the minimal required OS system and libraries to perform the specific workload or job. Some OSs are specifically suitable for containers such as Alpine, Ubuntu Snappy, VMwarePhoton OS, or RancherOS, so they should be given a higher preference.

For example, we could rebuild the previous NGINX image with the following policies:

  • Fixed NGINX version; for example: NGINX-1.17.2
  • Under a thin OS, such as Alpine Linux
  • Removed unstable repositories, and with pinned repositories
  • Recompiled from source using the latest TLS protocols
  • Cleaned unnecessary dev packages after build
  • Cleared the cache after install
  • Run as read-only using a temporary filesystem

If we opt-in to build from source, we may also consider using multi-stage builds, which optimizes the usage of containers and removes unnecessary dependencies from images. For example, we could use a builder container to create the NGINX binary and then move it to a new clean container within a minimal OS.

Handling Secrets Securely

Let’s say we have a CI server that spins up on-demand as part of a pipeline job. This container will need to access some secrets and SSH keys to perform the job. What is the most secure way to make those available?

One of the most common options is to use environmental variables injected just before the container is run. However, this is not really secure by default for several reasons; one of these is that the secrets can be inspected easily from outside the container.

A better alternative is to use temporary mounted volumes that contain only the required secrets or configuration values. If the secret is consumed by the application we can unmount the volume so that the application will keep it in memory.

An even better strategy is to use a custom Secret Store; for example: Vault offers several benefits:

  • Keys can be rotated easily, without compromising credentials.
  • Keys can be controlled via Roles and Permissions, using RBAC controls.
  • Keys are not handled or exposed via docker inspect, as they reside only in memory once the application requests them.
  • Keys are not accidentally committed to the repo or viewed by unauthorised users or via shoulder surfing, as the values are retrieved on-demand and are encrypted.
  • Better for auditing and compliance as reads or writes from the store are logged.

Vulnerability Scanning

Docker images are composed of several immutable layers that they can be identified individually. That makes the process of checking for vulnerabilities more efficient. If a defect is found in one layer, it then propagates to the lower layers and don’t need to be checked again. By utilizing multiple scanners we get a more holistic picture of the security flaws that are discovered on each layer.

Ideally, image scanning should be performed before publishing to a remote registry, to prevent image propagation in case another workload pulls an image locally after the vulnerability is found.

Here are some example tools that can be used for vulnerability scanning:

  • Clair from CoreOS for an API-based scanner that consumes several CVE databases; and offers auditing.
  • Docker Bench – checks for dozens of common best-practices around deploying Docker containers in production.
  • Grafeas – provides a uniform way to audit and govern software resources.
  • OpenSCAP – If you are interested in SCAP compliance.

For each tool there should be an independent risk assessment and evaluation period, to check if they are fit for your specific purpose, or if they have known vulnerabilities themselves. The last thing you want is to use vulnerable tools for security purposes.


Securing on-demand containers and stateless environments should remain a “Day One” requirement to minimize exposure and risk that could prevent the business from operating properly.

Having a carefully prepared list of security controls that cover static image scanning or monitoring options, runtime protection, and secure-by-default policies is essential.

With this approach, a lot of operational risks can be reduced, and System Ops can be confident that their infrastructure components have a good security position.

Capsule8 understands the importance of production uptime as a revenue engine for modern businesses. Designed to avoid costly downtime, overloaded hosts, or stability snafus caused by traditional security tools, modern enterprises depend on Capsule8 to efficiently protect their cloud workloads with protection, detection, and response in any environment. By using Capsule8, security teams have the visibility needed to detect incidents and protect against unwanted behavior, allowing them to mitigate risk, reduce downtime and avoid costly business disruption. If you’re interested in learning more, request a demo today.

Request a Demo of Capsule8 Now