Intro
Kubernetes RBAC, or role based access control, is the broadest authorization mode into different parts of Kubernetes and the Kubernetes API, making it a critical part of your overall Kubernetes security strategy. Security breaches can start with overly permissioned Kubernetes roles, but Kubernetes does not provide easy management for RBAC roles, so practitioners are left to navigate and manage it on their own with a slew of third party tools. By the end of this blog, you should have a better understanding of the meaning, and classic over permission like, for example, assigning Kubernetes cluster roles.
-
RBAC access control: how authentication and authorization work
- RBAC vs ABAC
-
Kubernetes Role-Based Access Control Examples: Over Permissions
- How RBAC and Kubernetes Context Work
-
How to secure RBAC authorization
- Role-based security and zero trust for the cloud and Kubernetes
RBAC access control: how authentication and authorization work
Before we dive deep into the details of RBAC, it’s important to understand its role in overall authentication and authorization into Kubernetes. Below is a diagram of a request flow to the Kubernetes API server. In this article we will be discussing authentication and authorization, as admission control is a relatively larger topic on its own.
The requests to the Kubernetes API server will come from a user (a person), a Service Account, or some other ‘thing’, basically from one API to another. The steps for these requests follow the flow:
- Authentication: Does the user/machine have the right credentials to access the API server? This is usually taken care of by Cloud Identity and Access Management
- Authorization: What is this authenticated user/machine allowed to do?
- Admission control: Is this type of workload ok to be scheduled, based on a set of parameters and policies we want to enforce?
- Etcd is then involved in holding the configuration data, state data, and metadata that would allow for persistence of this set of parameters
Download the Top 5 Tips to Avoid RBAC Over Permissions
RBAC vs ABAC
Of note, there are other ways to authorize a request to the API server; namely Attribute Based Access control (ABAC), Kubelet and Node authorization.
Kubernetes ABAC authorization can be used to confirm users are only able to gain access to resources for which they have permission. Kubelet authorization provides additional protection by only allowing specific operations on the node, such as launching new containers or modifying existing ones. And Kubernetes node authorization is an important part of Kubelet authorization, authorizing API requests made to kubelets. For the purposes of this article, we will stick with the ins and outs of RBAC.
How do you authenticate to a Kubernetes cluster?
If authentication is left to Cloud IAM, then how exactly do you authenticate into a Kubernetes cluster, and how is RBAC involved?
Here we have a flow diagram with the following steps, using the example of Amazon’s managed kubernetes platform, EKS:
- First, the person or the machine (API) will present a Cloud IAM token and authenticate to AWS.
- The person or machine then says, ‘I want to take an action (for example, kubectl get pods all namespaces)’, and present this request to the Kubernetes API server
- The Kubernetes API server will then pass the Cloud IAM token to the aws-iam-authenticator-server, which is installed as a daemonset on the workload cluster
- The aws-iam-authenticator-server is responsible for talking to AWS to determine whether the Cloud IAM token is valid, and the response it gets from AWS is an IAM Identity
- This IAM Identity is then passed to the aws-auth ConfigMap. It’s job is to translate the IAM Identity into a Kubernetes subject. This is an important step, because AWS doesn’t speak native Kubernetes
- The Kubernetes subject is then passed back to the API server, combined with your original request (e.g. kubectl get pods all namespaces)
- RBAC then responds with a yes or no, this person/machine can or cannot take this action
At this point, it should be clear that authentication is performed by Cloud IAM. The difference between RBAC and Cloud IAM is that Cloud IAM is a much broader framework provided by cloud service providers. It governs access to various cloud resources beyond Kubernetes, such as virtual machines, databases, and storage. While Kubernetes authorization specifically focuses on controlling interactions within the cluster, IAM operates at the cloud infrastructure level, allowing administrators to manage access across multiple services and resources within the entire cloud environment.
What is RBAC
As depicted above, Kubernetes RBAC functions as authorization that governs who can do what within your Kubernetes clusters. A policy consists of information specifying a collection of the following:
When you create a service account or a user, you’re defining:
- What Kubernetes resources the user has access to
- What Namespaces the user has access to
- What nodes the user has access to
- What they can do in that resource
The way you do this is with “roles” and “clusterroles” that are allowed specific permissions, and applied to users through rolebindings.
For example, here is a basic (and very insecure) RBAC role that says, “The pod-reader role has wildcard access and get, watch and list permissions for resources in the dev-group-1 namespace.”
RBAC and Kubernetes Security
At this point, after just a short introduction to RBAC, it is possible to see how RBAC could be the golden key to your environment. There is a reason that RBAC over permissions are included in the Kubernetes OWASP Top 10 as one of the top ten Kubernetes risk factors. But how easy is it to get access to your RBAC permissions, or for an attacker to use them to their advantage?
When a user is assigned permissions, those permissions are embedded into the users Kubeconfig, which is essentially the file that sits on your local computer, giving you the capability to authenticate to ‘x’ Kubernetes cluster. The Kubeconfig is not encrypted and it sits in a directory on your computer that is not necessarily hidden.
If you have full-blown admin/root access, what happens when a bad entity gets a hold of that Kubeconfig? A recent software supply chain attack found attackers actively searching for the Kubeconfig file, showing just how aware attackers are regarding this treasure trove.
Securing RBAC (and finding a secure place for your Kubeconfig file if you are an admin) is, in short, one of the top, if not THE top mitigations against possible attacks on your clusters.
Kubernetes Role-Based Access Control Examples: Over Permissions
RBAC policy examples will help to demonstrate what would constitute overly permissive settings.
A ClusterRole is a role that applies cluster-wide. In the policy below, it says that the ClusterRole can use get, watch and list verbs and view secrets for your resources in the dev-group-1 namespace.
Rolebinding, and in this case, ClusteroleBinding is a way to apply role or ClusterRole permissions to a user or machine.
In the example below, the user Jane is now bound to the role named ‘pod-reader’, so she can do anything that is allowed in the pod-reader role. As we learned above, the pod-reader role has wildcard access and get, watch and list permissions for resources in the dev-group-1 namespace.
Let’s now go through the whole process from start to finish, creating an RBAC policy using a serviceaccount.
First, you’ll want to create a service account. As a reminder, a service account is a type of ‘machine’ user, or an abstraction that allows a machine to make an API call to the Kubernetes API server versus a person. This is quite common, as there is a lot of machine to machine communication within Kubernetes. Just like a user that is an actual person, a serviceaccount allows access to specific resources. As such, you can set up a user (like yourself) as a service account or you can set up a service account that’s used for Pod creation.
The service account Manifest below creates a new service account called “mikeuser”.
apiVersion: v1kind: ServiceAccountmetadata: name: mikeuserNext, you have to set up some permissions for the service account. Although the service account isn’t defined in the below Manifest, the below Manifest is creating the permissions that the “mikeuser” service account should have. In this case, you’re giving “mikeuser” authentication to the Pods Kubernetes resource. Then, you’re giving it authorization to Pods to do the following:
- Get
- Watch
- List
This means you’re giving “mikeuser” authentication access to Pods and authorization to perform read-only actions.
kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: name: readerrules:- apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"]After you define the ClusterRole and the permissions that you want the “mikeuser” service account to have, you can bind the service account and the Role together with a ClusterRoleBinding.
apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: read-pod-globalsubjects:- kind: ServiceAccount name: mikeuser apiGroup: "" namespace: defaultroleRef: kind: ClusterRole name: reader apiGroup: rbac.authorization.k8s.ioYou may see other configurations that show “Role” instead of “ClusterRole” and “RoleBinding” instead of “ClusterRoleBinding”. The difference is whenever you see “Cluster” in the Kubernetes resource, the permissions/authentication/authorization you’re giving a service account is cluster-wide, not node specific.
This example highlights a common gotcha; generally, we want to avoid creating clusterroles and clusterrolebindings and instead use more specific, scoped roles and rolebindings. Cluster roles and cluster role bindings will automatically apply to any namespace you are running, from the kubesystem namespace to the default namespace, or anything else.
Now is the chance to test your knowledge. Can you spot anything wrong with the below YAML? For some background, by default, there is a serviceaccount available by default in every namespace, you can grab it from any pod that spins up.
- In this example, we’re looking at the service account located in the default namespace
- The service account is bound to the cluster role named ‘cluster-admin’
- If you were to have a web app running in the default namespace, and you’ve mounted this service account into every pod, if somehow remote code execution happens or somebody gets access to the file system, they are now cluster admin, creating a huge blast radius
How Kubernetes Context and RBAC Work
We often talk about the importance of Kubernetes Role Based Access Control (RBAC) for maintaining the Principle of Least Privilege in your Kubernetes cluster. There are two potential complications to an RBAC implementation that we haven’t discussed, however. First, how do your Kubernetes admins/developers/users assume an RBAC defined role in your cluster? And second, how do you manage such roles across multiple clusters or when users may need multiple roles in the same cluster? In other words, how do you do identity management in Kubernetes?
Luckily, Kubernetes has a single concept that answers both of these questions: “Context.”
Kubernetes is explicitly designed to allow you to interact with multiple clusters. You may have local Kind or K3s clusters on your laptop used for development plus test, staging, and production clusters hosted in the cloud, for example. When you type in a kubectl command it needs to know which cluster to send that command to, and this is where “context” comes into play. Kubernetes context can be thought of as a profile defining a cluster, its location, how to authenticate to it, and what namespace to use. To switch between defined contexts is as simple as running kubectl config use-context context-name.
Using Kubernetes Context and RBAC to Achieve Security Best Practices
At first glance, Kubernetes has what appears to be a surprisingly thin concept of a “user.” Users don’t have a unique file system (i.e., a home directory) like on an operating system or a distinctly managed “account” as in many applications. There is not even a defined API object in Kubernetes for a “user.” This it turns out is an intentional choice to outsource provisioning and authenticating user accounts in deference to whatever system you use already for identity management. You can read more about this in the first part of our series on Kubernetes Security Posture Management.
Once you have authenticated, however, the authentication material becomes part of what is recorded in the Kubernetes context: a certificate to present to the cluster, a user token, etc. The appropriate authentication material is then presented to the Kubernetes API, which validates it as genuine and then grants access in accordance with whatever relevant RBAC policies may apply.
That is because while Kubernetes does not have an object for users, it is aware of user identities and they can be bound, through (Cluster)RoleBindings to particular (Cluster)Roles. This allows the permissions of users to be appropriately scoped and limited based on their access needs and requirements.
As a result of this architecture, it is relatively easy to follow the Kubernetes security best practice of operating day-to-day under a more restricted set of permissions (ideally read-only). If/when the need arises for a user to expand their access, it is quite easy to switch roles by adopting a new context:
kubectl config use-context new-role
This new context just needs to identity a new “user” by supplying a different certificate or token, for example. Once that new authentication material is validated the API will grant access to whatever permissions are specified in the (Cluster)Roles bound to that new user identity.
Wait: Isn’t There Also a Thing Called “Security Context” in Kubernetes?
Yes, yes there is.
Kubernetes security context refers to the permissions attached to the processes running in your pods. It can be used to specify things like the numeric identity of the “user” (in the container) processes run under, assign certain Linux capabilities, or attach SELinux labels. Security context can be assigned at the pod or container level. It has a role to play in the overall permissions architecture of your cluster, but it operates in a somewhat different plane than RBAC, focused on the inner workings of your deployed applications rather than their permissions to access other resources in the cluster.
TLDR: Kubernetes is designed to enable you to access multiple clusters simultaneously. You can also have multiple “identities” in the same cluster. These are all defined as different “contexts” in your Kubernetes configuration. Each of those identities or contexts can be bound to particular sets of permissions (defined in a Role or ClusterRole) in their respective clusters. This enables the application of Role Based Access Control and following best practices around limiting permissions to only those strictly necessary. Because it is easy to switch contexts with a single command, users who sometimes require elevated permissions can still perform day-to-day operations with less privileged access and elevate their permissions only when necessary.
How to secure RBAC authorization
Understanding what RBAC is and why it’s important in Kubernetes is step one. Step two is figuring out how to mitigate as much risk as possible in your Kubernetes cluster with proper RBAC configured. The reality is that you cannot mitigate every single security breach that could occur. It’s simply not possible. However, you can take the proper steps and precautions to mitigate as much as possible.
First, you want to set a standard for your team and organization. Very rarely does technology solve people problems, and it never solves security issues across teams. The first step is to ensure that the team you’re on, the teams you’re working with, and management understand why RBAC is important, why it should be a key focus, and set a standard for everyone across the teams.
Second, you want to ensure that the least privilege is always the first thing that comes to mind when configuring RBAC. Much like every other sector of tech, you don’t want to start by giving everyone full-blown access and take away the access as needed. Instead, you want to give everyone (and every service account) the bare minimum that’s needed. Then, as issues arise, give more permission to suit the needs of whatever is occurring.
Third, ensure you understand what you’re installing. Many Kubernetes platforms and tools have to interact with your Kubernetes cluster. For them to interact with your Kubernetes cluster, 9.999999 times out of 10, the tool/platform is giving you a Kubernetes Manifest or Helm Chart to run on your cluster. This Manifest contains a whole bunch of goodies, including (usually) a service account creation and RBAC bindings. If you look deep enough into what might feel like one million lines of YAML, you’ll notice that a lot of these tools/platforms are giving FULL root/admin access to the service accounts.
This kind of third party access was called out this year in a CubeFS CVE . In Kubernetes, CubeFS is used for data processing services and machine learning. The CVE states that the DaemonSet used for installation relies on a cluster-role called cfs-csi-cluster-role which has the following RBAC configuration, and is too permissive.
So for third party resources, you have to ask yourself “is this actually needed?” and if it is, figure out why and how to remove the need. Sometimes it’s possible, sometimes it’s not. Again, you can’t mitigate all risks, but you can certainly ensure you understand the risk.
RBAC in Kubernetes attacks of 2023
RBAC was a necessary factor to attackers’ success in the new, Kubernetes-targeted attacks in 2023:
- Dero and Monero cryptocurrency miner: In the Dero and Monero cryptocurrency attacks, the attacker relied on a cluster having an RBAC configuration which would allow the creation of pods in that cluster, after gaining access to the Kubernetes API through a misconfiguration.
- RBAC-Buster: As its name suggests, this attack relied heavily on persistence through the creation of used privileged access in the kube-system namespace to create persistence through a new ClusterRole that had admin privileges, bound to a new ServiceAccount to give the ClusterRole’s admin privileges to the ServiceAccount.
Without the RBAC components in these attacks, attackers would not have been able to go beyond the initial stages of the attack.
Role-based security and zero trust in the cloud and Kubernetes
RBAC and cloud IAM are both critical capabilities to incorporate into your zero trust strategy. With KSOC, you can take the first necessary step in understanding the blind spots that have come up as a result of the shared service model, and then apply least privilege access with admission control and remediation guidance down to the manifest code.
User management of KSOC itself is also possible with granular permissions across users in your organization, ensuring that different teams have the access required, and no more.
How KSOC helps with RBAC
With KSOC, you can gain visibility into who has access to which resources using our RBAC Explorer:
Another view shows all the access for a particular resource, in this case a ServiceAccount, with full visibility into rolebindings that might otherwise be invisible in the absence of the associated role:
KSOC goes beyond a view of permissions and who is using what, to show threat vectors that combine an overly permissive RBAC role with a K8s configuration, runtime issue, public cloud misconfiguration and image CVE tied to the same workload. The screenshot below shows a threat vector where a service account with excessive permissions is mounted in a workload exposed to the internet.
Going above and beyond lists of over permissions, KSOC can also show who is accessing your permissions, to identify malicious insiders or attackers utilizing valid credentials. Kubernetes audit logging is an important step in securing your clusters, and can be turned on directly in Kubernetes or from the Kubernetes cloud service. KSOC AccessIQ is a new cloud native identity threat detection capability that uses AI to query API audit logs for actual usage of permissions.
For example, the screenshot below shows forbidden access attempts made on any resource, as well as all the allowed attempts made to exec into pods and the number of times it has happened.
Conclusion
RBAC is easily one of the top risk factors for Kubernetes. The first step to understanding the risk involved is a deeper understanding of how it works.
To get a first preview into KSOC’s new anomaly detection, and detection and response features for RBAC, request a demo today.
