“Containers do not always contain.” Without proper hygiene, containers may not always keep their contents constrained and secure. This is sometimes the unfortunate but fundamental truth of containers. Recently there has been a significant increase in the adoption of containers by our industry. With the advent of Docker and orchestration tools such as Kubernetes, many of us have shifted code deployment platforms to containers. The ‘easy to build, package, and promote’ promise of containers is one primary reason for their rapid adoption. However, as is true with any nascent technology, using containers at scale can come with its own set of security concerns.
Recently a number of CVEs have targeted container infrastructure. It is essential to be aware of these possibilities to help improve security posture. The need is more vital now because platform providers have started to allow third-party code to be hosted using containers. If code is not effectively sanitized, an attacker can more easily plant malicious code in a container and attack other containers or the host kernel itself. This summer I worked as a Cloud Operations Security Researcher Intern at Adobe and one of my primary projects was working to help strengthen security around running untrusted code in containers.
Linux provides various features to help sandbox Docker containers such as namespaces, cgroups, capabilities, SELinux/Apparmor, and Seccomp. All of these security features address different aspects of container behavior such as access control, resource consumption, and kernel interaction. When applied efficiently, they can help provide defense in depth. However, effective utilization of these features requires awareness of the best practices around their use. The developer needs to be aware of detailed nuances of his/her application to come up with rules and profiles. However, it is difficult to expect all developers to know and recognize these nuances. Thus, there is a need for tools which can examine containers and help employ more detailed defense mechanisms.
During my time here at Adobe I developed a tool which generates customized seccomp profiles for containers to try and address these problems. Seccomp decides the syscalls that a container is allowed to make. In Docker, seccomp is implemented as a json file which is provided as a part of the ‘–security-opt’ runtime argument. If the host kernel is built with seccomp, then all docker containers are spun up with a default seccomp profile. This default profile blocks 44 system calls and allows more than 300. In most cases, containers are used for specific tasks such as a web server, a database, etc. and do not require all 300 syscalls. Hence, allowing the container to make these syscalls even when they are not required just adds unnecessarily to the attack surface. The tool I developed generates slimmer seccomp profiles customized according to the container’s use case. This reduces the attack surface to a great extent without affecting its functioning.
Figure 1 – SecComp Profile Tool Flow
The tool generates these profiles by monitoring the syscalls made by the container using diagnostic tools strace and sysdig. These open source tools, unfortunately, do not always capture certain calls which are necessary for a container to start up. Therefore, a custom list of these required syscalls is added in the profiles by default to help increase accuracy and to help avoid any hinderance in the functioning of the containers. This custom list was generated by monitoring auditd logs.
The tool can be implemented across platforms which use Docker containers and can be extended to include capabilities and other Linux based security features. Another possible area of extension is to include IDS/IPS functionality which alerts/blocks a container that is using a vulnerable syscall. The tool also currently requires the container to be run once before generating the custom profile and is currently limited to Linux host OS – all areas being investigated for future improvement.
Furthering my research efforts in container security, I also did a study on gVisor – a new container sandboxing solution recently open-sourced by Google. It is a user-space kernel written in Go which implements a large portion of the Linux API. It includes an OCI runtime called runsc.
Figure 2 – gVisor Components
gVisor uses paravirtualization and acts as an intermediary kernel to isolate the containerized application from the host system. The Go kernel intercepts all the syscalls made by the container application and instead of redirecting them to the host kernel, it executes most of these calls within the userspace. Runsc is split into two processes: 1) Sentry – includes the kernel and is responsible for running user code and handling syscalls and 2) Gofer – the file system proxy that passes host files to Sentry through a 9P connection.
gVisor is trying to hit a middle ground between machine level virtualization (KVM, Xen) and rule-based isolation (seccomp, selinux etc). Each container has its own kernel and network stack. Containers do not have direct access to the host file system and the kernel is written in Go which makes it type and memory safe. In terms of usability, gVisor requires zero modification to the application and integrates with Docker and Kubernetes. However, it is not suitable for syscall heavy workloads and does not yet have complete syscall coverage. Applications which require hardware access are not supported by gVisor. In Docker, gVisor also does not yet support all options, such as –network, –volume, etc. In terms of container security, gVisor seems to be a promising technology which could provide strong isolation with lower resource footprint.
In my time here at Adobe, I have gained useful technical knowledge by working on these interesting projects. However, my internship was much more than just work. I was lucky to be part of a super fun and welcoming team where every day lunch was filled with interesting conversations. I also got the opportunity to be part of meaningful interactions with senior officials including the CEO, CTO, and the CSO – I also attended DefCon – which I think is one of the best security conferences in the world, enjoyed a day at the Santa Cruz boardwalk, and even went to a Giants game with the other interns! All this together made for an amazing summer and I think that is what #AdobeLife truly stands for!
Karthika Brakaspathy
Cloud Operations Security Intern
@karthika1495