Identity Threat Detection and Response (ITDR) has become indispensable to security teams, especially as identity’s critical role in multiple security breaches continues to make headlines. But today in many organizations, ITDR has a major gap when it comes to cloud native and Kubernetes RBAC.
Three of the four major attacks targeting Kubernetes in 2023 relied on overly permissive RBAC identities. In 2024, for the first time, software supply chain attacks targeted kubeconfig files, and a recent survey showed that 58% of teams using Kubernetes had a security issue in the last 12 months with insufficient access controls in their Kubernetes environment.
In this blog, we will discuss what is required to apply ITDR to cloud native security, with Kubernetes RBAC.
Contents:
What is Identity Threat Detection and Response (ITDR)?
What Are the Unique Challenges of ITDR with Kubernetes RBAC?
Importance of ITDR in mitigating Kubernetes RBAC’s role in cloud native attacks
Posture Management with CIEM and KIEM tools: the legacy approach to securing Kubernetes RBAC
ITDR Best Practices Checklist for Kubernetes RBAC
KSOC’s ITDR Solution for Kubernetes RBAC
Use ITDR to meet compliance requirements for Kubernetes RBAC, including SOX, SOC2 and FedRAMP
User Access Reviews for Kubernetes RBAC
Use KSOC’s Cloud Native ITDR to stop malicious insiders
KSOC Cloud Native ITDR Cost Savings
Download the Cloud Native ITDR Checklist Now
What is Identity Threat Detection and Response (ITDR)?
ITDR tools detects incidents in your identities and provide a way to right-size, delete, block, or otherwise respond in the case that an identity is identified as malicious. The detection of a malicious identity can take many forms, and depends very much on the area of focus. In the case of IAM, the focus would be on malicious logins, perhaps from suspicious locations, or multiple locations all at the same time. In the case of service accounts or machines, one might focus on which user has created the service account, how long it has been in use, or unused. And responses can vary from blocking certain users from logging in, to right-sizing a service account down to least privilege access by removing the verbs get, list and watch if those permissions have gone unused. ITDR capabilities and requirements vary greatly based on the area of focus.
In a zero trust program, ITDR would be the ultimate check against anything that might have gotten through the guardrails and least privilege access policies put in place. ITDR tools would also, ideally, provide strong recommendations, based on malicious activity, to improving identity posture.
Gartner recently labeled ITDR as one of the top security and risk management trends, naming focus areas like Identity and Access Management (IAM) tooling and insider attacks. And all signs point to the prioritization of ITDR as a part of zero trust initiatives in 2024; recent research places identity and access management and cloud infrastructure as the top two areas of focus for CISOs in 2024. Another recent survey showed that identity has moved up to be the second top priority for CISOs, followed closely by cloud infrastructure security, compared to the 8th top priority last year. And analysts forecast an uptick of 10% in the number of organizations that will have a measurable zero trust program in place over the next year. Identity Threat Detection and Response (ITDR) is like having a security guard for your digital identity. Just like how a security guard watches over a building and checks for anyone suspicious, ITDR keeps an eye on your digital information—like your usernames and passwords—to make sure no one's trying to steal or misuse them.
What Are the Unique Challenges of ITDR with Kubernetes RBAC?
The advent of cloud and cloud-native technologies, including containers and Kubernetes, has exponentially increased the number of identities, expanding the scope of security teams' detection and response efforts. Kubernetes RBAC (Role-Based Access Control), in particular, presents a unique challenge due to its complexity and the absence of clear ownership between developers and security teams.
Challenges of ITDR with RBAC include:
- Ambiguous Ownership: To properly secure RBAC, somebody needs to write the policies, audit them, do user access reviews, and be able to investigate in the case of an incident. In general, the only thing teams agree on is that the security teams should be performing the audit and investigation, while user access reviews are generally a shared task as security checks its data with engineering. But who writes the policies is up for grabs, and will differ across various organizations.
- Default toward over-permissions: If you are trying to ship applications fast, it's more practical to purposefully over-permission than under-permission so nobody is blocked.
- Complexity and Kubernetes Security skills gap: The RBAC policies themselves are hard to write, requiring a level of nuanced YAML, and there is an inherent risk involved when they are written by platform teams or somebody without security or least privileges in mind.
- Increased Number of Digital Identities: The volume of digital identities today stretches security teams to their limits. Keeping track of, and securing this myriad of identities becomes a major task.
- Software Supply Chain security in the Kubernetes ecosystem: Many Kubernetes tools require high privileges within RBAC in order to function, as they are deployed in-cluster as pods, daemonsets, and plugins. This expands the attack surface through the hundreds of software supply chain tools in the Kubernetes ecosystem.
- Misconfiguration Risks: Misconfigurations in components like kubelet or out-of-date versons of clusters can open the risk to vulnerabilities and attacks.
- Insecure defaults: There is no built-in RBAC query mechanism in Kubernetes to check for over permissions or help you write policies, and there are many default permissions that can be introduced just be deploying into managed cloud Kubernetes services like AKS, GKE or EKS.
Importance of ITDR in mitigating Kubernetes RBAC’s role in cloud native attacks
ITDR is important because, in the case of a cloud native attack, it is the only solution that could discover or mitigate the large role played by RBAC in cloud native attacks.
Software supply chain attacks: For the first time, in 2023, a software supply chain attack was found to be targeting Kubeconfig files. These files hold the authentication data for a cluster, including users, namespaces, and the other information required to communicate with the Kubernetes API. With this, you could see which role bindings are tied to system:masters group, and find any number of ways to get into a cluster.
Excessive permissions propagated through the software supply chain: Many of the tools used to manage Kubernetes require high permissions. Since many tools are generally deployed via Daemonsets, which deploy a program across multiple nodes, these permissions can be spread widely throughout an environment. A recent report highlighted the tools that require excessive permissions, and how researchers worked with these tools to reduce the risk of these permissions. For example, EKS and AKS ran a Daemonset to update nodes, which required the ‘update nodes’ permission. But without a way to scope down the permission of the Daemonsets by pod, this meant Kubernetes would permit every node in the Daemon set to run ‘update nodes.’ And with this, a malicious pod could take down every other pod but itself, giving itself undue permissions and control.
Dero, Monero cryptocurrency miner Kubernetes attacks: In the Dero and Monero cryptocurrency miner Kubernetes attacks in 2023, the attacker’s ability to run a Daemonset and create it’s own malicious pods had an RBAC pre-requisite. In order to gain access, the attacker first scanned for Kubernetes APIs where authentication was set to --anonymous-auth=true (which allows anyone anonymous access to the Kubernetes API), but for this entry point to work, the cluster would also need RBAC configuration that would allow for the creation of pods in that cluster.
Azurescape: ACI, or Azure Container Instances, is a Container as a Service (CaaS) offering that abstracts away the management of the infrastructure on which containers are deployed. Researchers identified a cross-tenant attack, where an attacker could find its way into somebody else’s containers hosted on ACI. This attack depended on the attackers’ ability to access a privileged Kubernetes Service account token, which they then used to take over the Kubernetes API server and the entire multi-tenant cluster with it.
Privilege escalation in FluentBit for GKE: Fluentbit is the default logging tool in GKE, and is installed on each node in the cluster. Researchers found that, if they were able to find a vulnerability in a FluentBit container, they could take advantage of a misconfiguration to read the projected service account tokens and access the token of any pods on the node.
Before vendors and researchers had found and publicized their fixes and mitigations for these issues, the only thing that could have helped identity an attacker taking advantage of these issues would have been an ITDR solution for Kubernetes RBAC. And in this context, an ITDR solution would need to understand the usage of valid permissions, prioritize risks based on a holistic view of the threat landscape, including runtime events, and in the best case scenario, be able to right-size identities.
Posture Management with CIEM and KIEM tools: the legacy approach to securing Kubernetes RBAC
Historically, RBAC security tools were designed to enhance identity posture and implement hardening strategies. While Cloud Identity and Entitlements Management (CIEM) focuses on Cloud IAM, Kubernetes Identity and Entitlements Management (KIEM) is solely focused on posture for Kubernetes RBAC, with the following capabilities:
- Over Permissions: Managing RBAC involves adhering to security best practices, such as applying the principle of least privilege. Many rules focus on validating the authorizing user (role, role binding, etc.) in Kubernetes resources, e.g. allowing all verbs (*, create, update, etc.) for deployments, jobs, daemonsets, etc. When you create roles to authorize certain Kubernetes resources (deploys, daemonsets, jobs, etc.), mistakes like over permission and misconfigurations are common.
- Misconfigurations related to data breaches: Risks include unauthorized access to sensitive data, such as the ability to read secrets, mount volumes, and get access to tokens with privileges.
- Misconfigurations in control plane and nodes: Misconfigurations in the control plane and nodes components such as kubelet, or using outdated cluster versions can expose vulnerabilities. An example of this includes an admin cluster giving anonymous users excessive permissions.
- Privileged Access: Users may have access to privileged pods and powerful permissions controls allowing them to execute high-level commands,, mount resources, or impersonate other privileged users.
- Advanced controls: To validate advanced kubernetes features such as admissionController, certificateSigningRequest requires sophisticated controls.
But posture is not actual behavior, nor is it a checklist for RBAC security. With ITDR, you can expect more than just laundry lists of permissions, including understanding actual usage of valid permissions, how to right-size, which identities are unused, and which identities are top priority based on their relationship to other contextual factors like runtime events, image CVEs and more.
Difference between Cloud Native ITDR vs KIEM tools
ITDR Best Practices Checklist for Kubernetes RBAC
If you truly want to be able to detect and respond to any compromises of the Kubernetes RBAC identities across your environment, you will need to ensure any tool you use can match the following ITDR best practices checklist for Kubernetes RBAC, and not just Cloud IAM.
Kubernetes RBAC ITDR Checklist
Download the Cloud Native ITDR Best Practices Checklist
KSOC’s ITDR Solution for Kubernetes RBAC
KSOC recently launched cloud native ITDR, introducing attack paths across Cloud IAM and RBAC, as well as AccessIQ, an AI-based query engine to investigate the actual usage of RBAC permissions.
KSOC’s ITDR solution stops Kubernetes RBAC from being part of an incident in your organization, going farther than posture management with an understanding of the active usage of identities and their relationship to other parts of the kill chain. With KSOC, you can truly get a handle on Kubernetes RBAC security by:
- Prioritizing RBAC risk in connection to other risks, including CVEs, runtime alerts, other Kubernetes misconfigurations, network issues and Cloud IAM.
- Understanding the usage of your identities, versus a passive view of over permissions, so you can tackle the largest risks or identities contributing to active incidents first
- Streamlining the implementation of least privilege access with a zero trust right-sizing policy generator
With KSOC, this is easy and automated across your clusters.
Check out KSOC's Cloud Native ITDR capabilities
KSOC ITDR Checklist
KSOC’s ITDR solution addresses Kubernetes RBAC in the following ways:
- Top Risky Identities: With KSOC it is clear which are your riskiest identities, based on their permissions, actual usage, and relationship to other risk factors like image CVEs, runtime events and Kubernetes misconfigurations. The risk score in the identity inventory provides an easy roll-up and filter to prioritize remediation.
KSOC identity inventory with risk ratings
- Risk score: KSOC calculates the relative risk of an identity based on a number of factors, including (but not limited to):
- Relationship to workloads with runtime events or reachable image CVEs
- Failed access attempts
- Usage (or not)
- Privileges and associated misconfigurations
KSOC risk analysis screenshot for service account
- Admission Control Response: With KSOC’s admission controller, you can enforce policies that put guardrails around the usage of RBAC permissions for users and service accounts. In the example below, the policy would apply least privilege scope to service account tokens, and there is a further suggested remediation for RBAC policies.
Example of Admission Control Response Policy for RBAC
- Actual Usage of Identity: With KSOC, it is clear, based on the combination of audit log data with permissions, what identities are actually being used, and how. Below is an example of a service account identity that is not being used, and has not interacted recently with the Kubernetes API, making it a prime candidate for right-sizing or deletion, to enforce a least privilege access framework.
KSOC usage risk analysis
Here are also the audit logs, showing the actual activity details for that service account.
KSOC RBAC identity audit log view
- Namespace/environment filters: In the same identity inventory where you can see your prioritized risk, it is possible to filter across namespaces or identity types.
Filter by type
Filter by source
- Covers service accounts & users: KSOC does not relegate its coverage of Kubernetes RBAC to just service accounts or just users. Cloud native ITDR with KSOC covers both, and can show the relationships across identities taking into account roles and role bindings. The example below shows a service account connected to the cluster role through a cluster rolebinding.
Service account roles and permissions in KSOC
- RBAC right-sizing: KSOC has an automated zero trust policy generator, to streamline the implementation of least privilege access across an environment. In the example below, one of the key recommended changes was to remove the wildcard (*) from the cluster role.
KSOC right-sizing example
Use ITDR to meet compliance requirements for Kubernetes RBAC, including SOX, SOC2 and FedRAMP
Customers and regulators today require that requirements are met for Kubernetes environments. ITDR should provide the necessary detection and response capabilities, along with the least privilege guidelines, to meet these requirements.
User Access Reviews for RBAC
User access reviews involve periodically examining and validating a user’s access rights to ensure an organization is appropriately assigning and adjusting them as the organization changes, all following the principle of least privilege. The end result is a printed report that shows the access review from that quarter, as well as a list of follow-ups for any issues found in the review. Some of the most difficult issues in these reviews, and the areas with the biggest penalties, involve improper offboarding of former employees, as well as unused identities.
KSOC helps to automate this process for Kubernetes RBAC, and is particularly helpful in the area of unused identities, leveraging insight into how identities are or are not being used. User access reviews are a critical part of SOX, SOC2, PCI, and FedRAMP compliance.
Until now, for teams to be able to do this in Kubernetes, even with any of the KIEM or open source RBAC tools available, it would be a manual, lengthy process.
Use KSOC’s Cloud Native ITDR to stop malicious insiders
With KSOC, finding malicious insiders is simple and automated.
Starting on the identity inventory page will tell you which are your riskiest identities, taking into account actual incidents, usage of the identity, and connections to workloads with other kinds of risks that are being exploited.
KSOC identity inventory, filtered by risk level
From there, filtering for those identities that include active runtime alerts and are part of a threat vector will tell you where workloads associated with identities are actively being exploited.
KSOC threat vector and runtime alert filter
Any of those identities that are part of an active incident can then be investigated and right-sized, to shut down any of the exploited permissions that caused the incident in the first place.
KSOC Identity Relationships Map
KSOC zero trust policy generator
KSOC Cloud Native ITDR Cost Savings
KSOC’s Cloud Native ITDR solution is able to assess identity in the context of multiple other components in the environment, with a much broader understanding of holistic risk than traditional Cloud/Kubernetes Identity and Entitlements Management (CIEM/KIEM) solutions.
Just bringing in audit logs, and making them queryable, can take a much longer time than a KSOC installation. For example, for EKS, ingesting audit logs requires a lambda trigger from Cloudwatch or Kinesis to route the logs to save to an S3 bucket or another destination for further processing. Then, a query engine would have to format the logs properly before they can even be queried. Along the way, there might also be a big price tag from a SIEM provider.
Cost and time savings with KSOC Cloud Native ITDR
Conclusion
ITDR is a critical capability in any zero trust strategy, and is also a critical piece of a detection and response strategy, given the consistency of identity in any attack kill chain. This is no different when it comes to identity in Kubernetes, and RBAC security, where software supply chain attacks and other Kubernetes tools can easily spread elevated privileges across an entire multi-cluster environment. It is much more effective, in terms of prioritization and remediation, to work with a solution focused on the actual, malicious usage of valid credentials, and not just laundry lists of over permissions.
Reach out to the KSOC team today to automatically generate zero trust policy recommendations in your environment, as a first step toward adding ITDR to your overall zero trust and incident management strategies.
Request a trial of Cloud Native ITDR
