Intro
Kubernetes admission controllers are an important Kubernetes-native option for enforcing policy on your containers and overall Kubernetes environment. But there are tons of flavors and ways to do it, and candidly, very few actually enforce admission control in their production environments because, well, it can break production. So what are the key factors to look at when evaluating the practical benefit of an admission controller?
Just get the checklist
What is the Kubernetes API server?
The first step to understanding admission controllers is understanding the Kubernetes API server, which is the primary way in which one would interact with Kubernetes objects. In the diagram below, it shows three steps in the process of making a request to the Kubernetes API to change the state of an object; for example, deploying a pod.
In the first step, which has more to do with cloud identity and access management, the server determines whether the presented user is valid. In the second step, which has to do with Kubernetes role based access control, the server determines what actions the user is allowed to take. In the third step, the server determines if there are any specific limitations on the action the user or machine is wanting to take. Etcd is where the data for the cluster is stored, so the state determined by this three step process will be executed and represented in etcd.
Step 1. Is Bob from my cloud IAM account a valid user?
Step 2. If yes, can Bob create a new pod?
Step 3. If yes, can Bob create a new pod with privileges within X namespace?
This would work similarly when substituting Bob, a real user, for a service account, or a machine user. This would work similarly in the case that, instead of the default Kubernetes API, you are trying to perform actions on custom resources, which are basically non-default extensions of the Kubernetes API.
Read more about the container lifecycle, which is necessarily involved with Kubernetes API processes but running on its own schedule.
What is an admission controller in Kubernetes?
Simply put though, admission controllers allow you to set security policies for your Kubernetes clusters. At a glance, admission controllers provide access control to users and applications within a cluster. They allow administrators to set various policies including which users can add or delete objects in a cluster, as well as which applications can be deployed to the clusters.
Admission controllers are typically used to ensure that only safe code and configurations are allowed into the production environment. By intercepting and validating requests to the Kubernetes API before they are allowed to persist in runtime in the cluster, admission controllers mitigate risks associated with unauthorized actions, malicious entities, or erroneous configurations. This preemptive policy enforcement ensures that only safe and compliant workloads are deployed.
In practice, admission controllers wield great versatility, facilitating an array of custom policies tailored to specific organizational needs. They can be leveraged to validate resource requests, evaluate labels and annotations, assess resource quotas, and even trigger webhooks for more advanced validation scenarios. By offering this flexibility, admission controllers significantly enhance the platform's resilience and overall security posture. Kubernetes even provides Kubernetes Pod Security Admission, with three levels of restrictive policies.
Here are some examples of admission control policies:
- A Container starts a shell and attempts to establish a reverse shell.
- Do not generally permit containers to be run with the hostIPC flag set to true. Containers were discovered where spec.template.spec.hostIPC was set to true. Set them to false. (this one is specifically a response to a K8s CVE in the nginx ingress controller)
- Images should only be sourced from allowed registries.
Kubernetes webhook types
Webhooks differ from admission controllers in that they are only triggered when a request is received by the cluster. A webhook enables developers to tailor specific instructions when creating new resources on a Kubernetes cluster. It acts as an interface between your application and the Kubernetes API server, allowing for custom validation rules such as resource quotas and limits, user and client authentication requirements, or admission control policies.
Admission controllers, however, run every time someone attempts to create or modify a resource on the cluster. And to make it even more confusing, there are Kubernetes webhooks that work with admission controllers; namely mutating and validating types. The behavior of these webhooks are determined with the admission controller configuration files.
Mutating Webhook Admission Controller
A mutating webhook admission controller intercepts and processes incoming requests to the Kubernetes API server before they are persisted to the cluster. The primary function of a typical mutating admission controller example in Kubernetes is to modify (mutate) the content of the resource being created or updated.
This is generally an intrusive option that does not allow for close management of what gets deployed to production, as the mutation is automatic and the enforcement of the policy in the first place has the potential to bring down production.
Kubernetes Validating Webhook Admission Controller
A Kubernetes validating admission controller, on the other hand, focuses on validating and potentially rejecting incoming requests before they are persisted to the cluster. Its primary role is to assess whether the request complies with the defined policies and standards of the cluster. This is much less intrusive than a mutating admission controller because it’s results are generally binary; deploy the workload or not.
Deprecation of the Pod Security Policy Admission Controller
The Kubernetes pod security policy, and the usage of the associated admission controller, was deprecated in Kubernetes version 1.21. The PodSecurityPolicy guidelines set a baseline to strive toward, and the admission controller is available to enforce these baseline guardrails. Many are still on applicable Kubernetes versions, but the recommended replacement for anybody using a version more recent than 1.21 is usage of the Kubernetes pod security standards, with options for privileged, baseline and restrictive levels.
Types of Kubernetes admission controllers?
There are multiple platforms available for building kubernetes clusters. The platform you are using influences the way you should approach admission control. Any of these (and KSOC’s admission controller as well) can be installed via AdmissionControl plugins.
- OPA Kubernetes admission controller: OPA (Open Policy Agent) is based on Rego, and is used often by teams that want to amass a large, ever-changing rule-set that can be queried, QA’d, and managed separate from the Kubernetes resources themselves. Policies are not restricted to allow/deny or binary answers, and checks on a policy can result in structured output with lots of context.
- Kyverno admission controller: Kyverno does not use Rego, it relies on YAML, and so it is easier to use than OPA from that perspective. However, managing Kyverno policies at scale is difficult because you cannot decouple the policies from the Kubernetes resources to which they apply.
- JSPolicy admission controller: JSPolicy is a less popular choice than Kyverno and OPA, using JavaScript-based validation for resources being created on a Kubernetes cluster.
You also create a Kubernetes custom admission controller.
See how KSOC compares to OPA
Managed Kubernetes Admission Controllers
Let’s look at another type of Kubernetes admission controller example.
The configuration of admissions controllers differs slightly when used with managed Kubernetes offerings such as EKS and GKE. In these environments, many of the admission controllers are already pre-configured within the platform, making it easier for developers to quickly get their applications up and running on a cluster.
That being said, EKS admission controllers offer support for OPA, Kyverno, Kubewarden, JSPolicy and other similar solutions. GKE admission controllers integrate with OPA and Kyverno to provide access control policies out of the box.
Pros and Cons of Kubernetes Admission Controllers
Admission controllers offer various benefits when it comes to providing access control to your Kubernetes clusters, including real-time validation, authentication enforcement, and policy enforcement. After all, if a resource is created without the proper authorization or validation required by an admission controller, it could create security vulnerabilities within your system.
However, most teams simply don’t turn admission controllers on, even if they have the capability to do so. Many well-meaning teams define the policies and go to enforce them, only to find that when they do, their production environments go down and their development teams are not prepared to understand the consequences of deploying containers that violate admission control policy.
Suffice it to say that using admission controllers can add complexity to the overall system architecture and may require additional resources to be allocated.
The teams best equipped to work with admission controllers in enforcement mode are those that have worked hard, between DevOps and Security, to define the policies, and who have done lots of practice and testing to understand the consequences of enforcement in any environment.
A checklist for evaluating a Kubernetes Admission Controller
Given this context, what do you need in an admission controller in order to be successful? Here is a checklist to help you rate vendors’ admission control options and understand whether you are getting something useful, or something you will pay for but never operationalize.
- How transparent are your admission control policies?
- Does the vendor keep the Rego policy in a black box? Is it labeled as ‘proprietary’? This can be an issue when you need to customize policies for your own environment. If you are able to transparently view - and even change - the Rego policy as you see fit, your engineers will be more comfortable with the policies you are enforcing and your most advanced teammates will feel they are in control of any downtime in production caused by admission control.
- Can you set them in the CI/CD process across multiple clusters?
- One of the issues with using the open source version of OPA (or any other admission controller) is the management and implementation across dozens (or even hundreds) of clusters. Quickly, the duplication and QA process becomes overwhelming to the point that admission control is no longer a practical option. However, if admission control policy creation can be folded directly into the regular GitOps, or CI/CD process, quickly the team will be able to scale across multiple clusters and environments, and the engineering team will have access to the policies themselves, which will reduce back-and-forth between engineering and security.
- Can you do a dry run before you enforce anything?
- Admission control will block a workload before deployment which, in some cases, can break production. It is critical to have an understand of what is going to happen before it happens. Can your vendor get a view to share with engineering of what workloads would be blocked, based on the policies in place? If not, it will be a rocky road getting to actual enforcement.
- How broad is the set of criteria available for enforcement?
- Some admission controllers are based only on one risk factor, such as critical or high-rated image CVEs. The breadth of criteria available for enforcement is something to check on ahead of time; do they take into account Kubernetes misconfigurations, at a minimum?
- How does the policy validation occur?
- Is the vendor requiring you to send data somewhere else in order to make a policy decision? This is worth checking, especially if you have a broad set of criteria you want to enforce. Do they use OPA, do they use Kyverno? This is important to understand, at a minimum in light of understanding the vulnerabilities your admission control will be subject to. For example, there was a recent Kyverno CVE that required a breaking update. You will want to know what third-party risks you are exposed to.
KSOC Admission Controller Configuration with KSOC
At KSOC, we understand the dynamic between the engineers managing Kubernetes and the security teams trying to secure it; in most cases, both teams are focused on security to some extent. KSOC’s admission controller capability is created to help security teams make Kubernetes secure by default in partnership with their engineering colleagues, with:
- Transparent policies: All Rego policies are transparent and visible in-cluster (we are OPA compliant)
- Setting policies across multiple clusters: We integrate with GitHub Actions to set policies for both detection and admission control within the CI/CD pipeline
- Dry-run: Instead of going straight to enforcement, you can get an easy list to share with engineering of policies that you would have blocked
- Criteria: We use a broad range of criteria, across Kubernetes configurations and other Kubernetes components
Our threat vector capability also allows you to immediately triage risk across your clusters, which will help tremendously in deciding which admission control policies to set in the first place.
Get a KSOC Demo
See how you can get real-time, event-based visibility alongside admission control with the KSOC platform.
Conclusion
Admission controllers are powerful in the context of Kubernetes Security, but are difficult to implement from a practical perspective, given team dynamics and sensitive environments. Use this checklist to quickly get to the root of whether or not a vendor’s admission control capability will allow you to operationalize Kubernetes security in the way you need.
