Skip to content
Kubernetes Security Best Practices at Every Layer
Jimmy MestaMay 17, 2023 7:21:15 PM6 min read

Kubernetes Security Best Practices at Every Layer

Introduction

Containers are transforming software development. As the new foundation for CI/CD, containers give you a fast, flexible way to deploy apps, APIs, and microservices with the scalability and performance digital success depends on. But containers and container orchestration tools such as Kubernetes are also popular targets for hackers — and if they’re not protected effectively, they can put your whole environment at risk. In this article, we’ll talk about security best practices for every layer of the container stack.

Container security considerations

It’s important to understand the security implications of containers. As an application-layer construct relying on a shared kernel, a container can boot up much faster than a full VM. On the other hand, containers can also be configured much more flexibly than a VM, and can do everything from mounting volumes and directories to disabling security features. In a “container breakout” scenario, when container isolation mechanisms have been bypassed and additional privileges have been obtained on the host, the container can even run as root under the control of a hacker — and then you’re in real trouble.

Here are a few things you can do to keep the bad guys out of your containers.

Layer 0 – The Kernel

Kubernetes is an open source platform built to automate the deployment, scaling, and orchestration of containers, and configuring it properly can help you strengthen security. At the kernel level, you can:

  • Review allowed system calls and remove any that are unnecessary or unwanted
  • Verify your kernel versions are patched and contain no existing vulnerabilities

Layer 1 – The Container

At Rest

Container security at rest focuses on the Docker image you’ll use to build your running container. First, reduce the container’s attack surface by removing unnecessary components, packages, and network utilities — the more stripped-down, the better. Consider using distroless images containing only your application and its runtime dependencies.

Next, make sure to pull your images only from known-good sources, and scan them for vulnerabilities and misconfigurations. Check their integrity throughout your CI/CD pipeline and build process, and verify and approve them before running to make sure hackers haven’t installed any backdoors.

Runtime

Once your image is packed up, it’s time for debugging. Ephemeral containers will let you debug running containers interactively, including distro less or other lightweight images that don’t have their own debugging utilities. Watch for anomalies and suspicious system-level events that might be indicators of compromise, such as an unexpected child process being spawned, a shell running inside a container, or a sensitive file being read unexpectedly. The Cloud Native Computing Foundation‘s  Falco Project open source runtime security tool, and the many Falco rules files that have been created, are hugely useful for this.

Layer 2 – The Workload (Pod)

A pod, the unit of deployment inside Kubernetes, is a collection of containers that can share common security definitions and security-sensitive configurations. Pod Security Standards specify a set of policies with three levels of restriction; privileged, baseline and restrictive. These policies cover the following Kubernetes controls and more:

  • Host Namespaces
  • Privileged Containers
  • HostPath volumes
  • Seccomp
  • Volume Types
  • Running as Non-root

To strengthen basic defense at the pod level, you can use the Pod Security admission controller with the most restrictive policy set.  For more flexibility and granular control over pod security, consider something like an Open Policy Agent (OPA), using the OPA Gatekeeper project.

Layer 3 – Networking

Compromised Workloads

By default, all pods can talk to all the other pods in a cluster without restriction, which makes things very interesting from an attacker’s perspective. If a workload is compromised, the attacker will likely try to probe the network and see what else they might be able to access. The Kubernetes API is also available to access from inside the pod, offering another rich target. And if you see traffic originating from a container in a cluster reaching out to a foreign IP that hasn’t been touched before, it’s not a good sign.

Strict network controls are a critical part of container hardening — pod to pod, cluster to cluster, outside-in, and inside-out. Use built-in Network Policies to isolate workload communication and build granular rulesets. Consider implementing a service mesh to control traffic between workloads as well as ingress/egress, such as by defining namespace-to-namespace traffic.

Application Layer (L7) Attacks – Server-Side Request Forgery (SSRF)

SSRF is consistently in the news when it relates to Kubernetes, and no wonder. With cloud native environments where APIs talk to other APIs, SSRF can be especially hard to stop; customer-supplied webhooks are especially notorious. Once a target has been found, SSRF can be used to escalate privileges and scan the local Kubernetes network and components; hit the cloud metadata endpoint; dump out the data on the Kubernetes metrics endpoint to learn valuable information about the environment — and potentially make it possible to take it over completely.

Application Layer (L7) Attacks – Remote Code Execution (RCE)

RCE is also extremely dangerous in cloud native environments, making it possible to run system-level commands inside a container to grab files, access the Kubernetes API, run image manipulation tools, and compromise the entire machine.

Application Layer (L7) Defenses

The first rule of protection is to adhere to secure coding and architecture practices — that can mitigate the majority of your risk. Beyond that, you can layer on network defenses along both axes: north-south, to monitor and block malicious external traffic to your applications and APIs; and east-west, to monitor traffic from container to container, cluster to cluster, and cloud to cloud to make sure you’re not being victimized by a compromised pod.

Layer 4 – Nodes

Node-level security isn’t quite as exciting as networking, but it’s just as important. To prevent container breakout on a VM or other node, limit external administrative access to nodes as well as the control plane, and watch out for open ports and services. Keep your base operating systems minimal, and harden them using CIS benchmarks. Finally, make sure to scan and patch your nodes just like any other VM.

Layer 5 – Cluster Components

There are all kinds of things going on in a Kubernetes cluster, and there’s no all-in-one tools or strategy to secure it. At a high level, you should focus on:

  • API Server – check your mechanisms for access control and authentication, and perform additional security checks of your dynamic webhooks, Pod Security Policy, and public network access to the Kubernetes API;
  • Access control – use role-based access control (RBAC) to enforce the principle of least privilege for your API server and Kubernetes secrets
  • Service account tokens – to prevent unauthorized access, limit permissions to service accounts as well as to any secrets where service account tokens are stored
  • Audit loggingmake sure this is enabled
  • Third-party components be careful about what you’re bringing into your cluster so you know what’s running there and why
  • Kubernetes versionsKubernetes can have vulnerabilities just like any other system, and has to be updated and patched promptly
  • Kubelet misconfiguration – responsible for container orchestration and interactions with container runtime, Kubelets can be abused and attacked in an attempt to elevate privileges

Conclusion

Kubernetes security can seem daunting, but by working through best practices for each layer of your stack, you can bring your containers to the same high level of protection as the rest of your environment — so you can enjoy the benefits of fast, agile development without putting your environment or your business at risk. Reach out to us for a demo to see how real-time KSPM can help show you the Kubernetes security gaps in your environment and how to fix them. 

 

avatar

Jimmy Mesta

Jimmy Mesta is the Co-Founder and Chief Technology Officer at KSOC and is responsible for the technological vision for the KSOC platform. A veteran security engineering leader focused on building cloud-native security solutions, Jimmy has held various leadership positions with enterprises navigating the growth of cloud services and containerization. At the Web App Firewall innovator Signal Sciences (acquired by Fastly, Inc.), he led offensive and defensive teams across the Security and Engineering organizations while helping build modern, developer-friendly security solutions. Previously, Jimmy was an independent consultant focused on building large-scale cloud security programs, delivering technical security training, producing research and securing some of the largest containerized environments in the world.

RELATED ARTICLES