Intro
In 2023 so far, a wave of new attacks targeting Kubernetes has been reported, from Dero and Monero crypto mining, to Scarleteel and RBAC-Buster. You’ve seen the research reports; but are you exposed? What would it take to protect against these attacks in the practical real world, versus the researcher’s honeypot? Understanding the practical reality of the attacks can demonstrate how you might protect yourself from these, and the next attacks aiming to target Kubernetes.
How KSOC Protects Against the K8s Attacks in 2023
Dero Cryptocurrency Miner
Refresher on the Dero kill chain
Dero is a Layer 1 application platform and cryptocurrency that is popular due to its improved privacy, anonymity and speed over other options like Monero or Z-Cash, making it an ideal cryptocurrency choice for attackers.
Attackers were discovered launching a broad attack with the goal of mining Dero cryptocurrency, in both February and then also March 2023. To gain initial access, the attacker first scans for Kubernetes APIs where authentication is set to --anonymous-auth=true (which allows anyone anonymous access to the Kubernetes API). It is not stated in the research, but for this entry point to work, the cluster would also need RBAC configuration that would allow for the creation of pods in that cluster. With these two conditions met, then the attacker deploys a Daemonset, creating its own pods from malicious images across the entire cluster. These malicious pods then spin up cryptocurrency miners.
What to learn from this attack and how likely are you to be exposed?
The entry point of this attack relies on a misconfiguration that is relatively difficult to make, because Kubernetes defaults to --anonymous-auth=false (versus =true) and managed Kubernetes (e.g. EKS, AKS, GKE) will also not allow anonymous authentication.
Comparatively, the more concerning part of this attack is the creation of the malicious pods, because these can lead to container escape and lateral movement (although, in the case of Dero, the lateral movement did not occur).
In this attack, the most important protective measure is preventing the creation of malicious workloads from Daemonsets; for example, by stopping workloads containing images not from trusted image sources.
There is also a rather obvious case for observability tooling, as many crypto jacking operations are discovered through unexpected spikes in traffic.
Dero TLDR
You are unlikely to be exposed to this attack based on the relatively uncommon point of entry.
If you somehow have been initially compromised, it is relatively simple to set up an admission control policy that prevents the creation of workloads coming from untrusted image sources. Since this attack uses an image to create the malicious pods, this would work. The qualifier is that it would only work where you enforce the policy . . . you would either then have to enforce the policy broadly, or employ a real-time KSPM detection solution (like KSOC) to understand exactly where you are having issues, then use the admission controller surgically as you fix the configurations in code.
Download the Admission Control Checklist
Monero Cryptocurrency Miner
Refresher on the Monero kill chain
Monero is a different kind of cryptocurrency that, like Dero, promises anonymity and is not listed on popular cryptocurrency exchanges like Coinbase. Monero is specifically banned by some governments due to its prolific usage by bad actors.
Monero was discovered at the same time as the Dero attack, primarily because it was piggybacking on Dero activity. Monero has a very similar kill chain to Dero, though it does move laterally and attempt container escape.
The first part of the attack is the same as Dero, with the attacker scanning non-standard ports for Kubernetes API authentication set to --anonymous-auth=true (and RBAC configuration that allows, with this condition, for the creation of pods in the cluster). With access to the Kubernetes API, the attackers delete the Dero pods and deploy their own privileged pod via Daemonset (Dero did not deploy privileged pods, and it used a malicious image versus downloading a payload from a C&C server). The privileged pod then attempts to mount the host directory to escape the container, and downloads a root kit that could hide the miner. Afterward, the attacker is able to install a custom mining service on the host.
What to learn from this attack and how likely are you to be exposed?
As in Dero, the entry point of this attack relies on a misconfiguration that is relatively difficult to make, because Kubernetes defaults to --anonymous-auth=false and managed Kubernetes (e.g. EKS, AKS, GKE) will also not allow anonymous authentication.
Can you see privilege escalation and container escape techniques from a Kubernetes-first perspective (aka in real-time)?
Unlike the Dero attack, this attack involves privilege escalation and container escape techniques, like creating privileged pods and mounting a host directory. Allowing privileged containers is one of the most critical Kubernetes security issues to avoid; it even made the Kubernetes OWASP Top 10 list. Kubernetes disallows privileged pods in its baseline policy for Pod Security Standards, making it less likely this will happen by default. However, if you are running EKS and Kubernetes v1.13 and above, the default pod security policy is privileged. In EKS, you must delete this policy to enable your customer policies; an added step that potentially increases the chances you will allow the creation of privileged pods.
In this attack, there is a lot of runtime activity, from Command and Control traffic downloading a payload to a root kit. But this activity happens after taking advantage of the initial Kubernetes misconfiguration; and locking this down would prevent malicious runtime behavior from spreading to other pods and clusters. So we can still conclude that the most important preventive measures are stopping the well-known, but yet all too common, disallowed host mounted paths and privileged pod misconfigurations. The real question is . . . can you see these exploitation in real time? If you are doing KSPM on polling intervals, you’re missing any attacker activity that happens in between.
Does your runtime tool use a ruleset that incorporates Monero?
Because Monero has been used in multiple previous crypto mining attacks, it is possible there would already be some rules that detect the specific activity occurring at the point of payout. However, at this point, the attacker has already achieved their objective.
Monero TLDR
You are probably not very exposed to this attack, based on its point of entry that relies on an uncommon misconfiguration.
In the case you are exposed, your primary concern should be tamping down the blast radius, as the attack occurs in real-time in Kubernetes, not in runtime.
If your runtime capability includes a rule around Monero crypto mining, you can stop the last step but not the initial phases of the compromise.
Scarleteel
Background
Scarleteel was first reported in February, 2022, and has continued since then with a renewed set of tactics in July, 2023, targeting Kubernetes environments. The goals of Scarleteel threat actors include crypto mining as well as theft of intellectual property. The Scarleteel attack is of serious concern as it demonstrates multiple adversarial advancements that take advantage of nuanced AWS and Kubernetes misconfigurations as well as evasion techniques. This attack uses Kubernetes directly as the entry point to a sophisticated attack that uses open source penetration tests, Pacu (for AWS) and Peirates (for Kubernetes) as a part of its persistence and escalation attempts. Pacu has 3.6 thousand stars on Github and describes itself as an ‘AWS Exploitation Framework,’, while Peirates has one thousand Github stars and describes itself as a Kubernetes penetration testing tool.
The Scarleteel kill chain
In the Scarleteel attack, a Jupyter notebook web application hosted in Kubernetes was the entrypoint. Once the attacker was inside the cluster, it used various scripts and endpoints to try and access AWS credentials. At this point in the attack, they also employed Peirates, a Kubernetes penetration testing tool, to find further avenues of exploit in Kubernetes.
Once successful with the scripts for harvesting AWS credentials (specifically, they got the AWS keys of the node role via instance metadata in EKS), exfiltration of the credentials was done with shell built-ins vs curl to evade detection. With the AWS keys, they installed AWS CLI binary and the open source project Pacu to do further discovery and privilege escalation into the AWS account. At the same time they installed the AWS CLI binary, they deployed the Pandora malware, which would add the compromised pods to the Mirai botnet.
One of their objectives was to access Russian S3 buckets, and do so in a way that would evade audit log creation in CloudTrail. Using the stolen AWS keys, the attackers were able to create new users and access keys for all admins in the account, which would help them accomplish their other objective, which was creating 42 new EC2 instances for crypto mining. When they were caught, they tried unsuccessfully to switch to the other admin accounts they had created.
What to learn from this attack and how likely are you to be exposed?
The Scarleteel attack demonstrates how fluidly attackers work across environments that defenders might see as separate areas; web app in Kubernetes, to Kubernetes hosted environment in cloud, to cloud. Here are some questions to help parse apart your exposure, and key learnings from this complex attack:
Are your developers using Jupyter notebooks?
Vulnerable web applications are not so uncommon, and Jupyter notebooks are a very popular IDE tool for data scientists and machine learning. The last CVE released was recent, in July of 2023 and, like any web app, requires an upgrade to install the patch. If your development team is not using this web app, then you don’t have to worry about the initial entry point.
Are you safe if you are not running Jupyter notebooks?
Even if you are not running Jupyter, you are not necessarily safe. The initial recon scans for Scarleteel could easily be tweaked to find other off the shelf apps similar to Jupyter, and the whole kill chain would look identical.
Do you have EKS-specific guardrails in place?
In the case that your developers do use Jupyter notebooks, the next challenge is to think of Kubernetes and the cloud as inseparable, as the vulnerable web app was running in an EKS environment, a cloud managed Kubernetes service. The attacker’s tactics and successes were specific to EKS, especially at the beginning. The attacker ran many scripts to try and find AWS keys, and they were eventually successful in grabbing the AWS keys for the node role, which basically governs the connection between EKS and the underlying AWS infrastructure. Each node in EKS must have an associated IAM role, and this role can’t be used to create any other clusters. The attackers found the credentials for the node role in the instance metadata, specifically IMDSv1, not IMDSv2.
IMDSv1 is known for its security weaknesses (which in this case, was still only exploited because of a small inconsistency in naming conventions) and AWS launched IMDSv2 in 2019 to accommodate for these. In this case, you want to be able to identify this particular security weakness in your EKS environment.
Once the attacker got the keys via IMDSv1, they launched the AWS CLI, which is a CLI that is just as useful for EKS as other AWS services.
Could you detect any lingering persistence?
Once the attacker started the crypto mining activity, it was easily caught because of the large increase in traffic, and shut down. But the largest potential damage, without doing extensive reverse-engineering of the attack, is the potential for broad lateral movement and persistence via the Pandora malware in the nodes and the knowledge of the broader AWS and EKS environments. It’s unclear exactly whether the attacker used any of the data it got from using Peirates, the Kubernetes penetration testing tool. Suffice it to say, they would have been searching for access to secrets, collecting service account tokens, and more.
Do you have Kubernetes-first, event-based visibility?
The attackers in Scarleteel could have accomplished any of the individual pieces of the attack in a short amount of time. It’s critical to be able to see each of those pieces in real-time, or else lose the context that would help you understand how events are related.
Scarleteel TLDR
If you’re not using Jupyter notebooks in your organization, you might not be susceptible to this attack . . .that being said, there are many web app vulnerabilities out there an attacker could switch in place of the Jupyter issue, so don’t stop your defensive measures if you don’t have that in place.
If you run EKS, you will want to look into places where you have IMDSv1 versus IMDSv2 installed.
And maybe you want to get a blue team to run Peirates and Paco against your environment first before an attacker does.
Runtime capabilities would potentially detect the Pandora malware, but would not connect this to the broader attack and activity happening across the cloud and Kubernetes environments, so it would not stop the entirety of the attack.
RBAC-Buster
Background
RBAC is a constant pain point for engineering and security teams. Kubernetes does not have any kind of permission visualization or default management capabilities, other than the ability to make policies and look at audit logs. As a result, RBAC, can quickly become unmanageable across service accounts, roles and permissions, and ‘overly permissive RBAC configurations’ has its own dedicated spot in the Kubernetes OWASP Top 10.
The RBAC-Buster kill chain
The attacker attempted to gain an initial foothold in a Kubernetes environment by scanning for a misconfigured API server that would allow unauthenticated requests from users with privileges. This sounds the same as the entrypoint for the Dero and Monero attacks, but the Dero and Monero attacks specifically used unauthenticated request permissions (--anonymous-auth=true), whereas in this attack, users are given privileges from the get-go. The privileged access was then used to list secrets and discover the kube-system namespace. Attackers checked for a few other campaigns from other attackers in order to delete them. Then they created a new ClusterRole with admin privileges and a new ServiceAccount in the namespace, binding the two together to give the ClusterRole’s admin privileges to the ServiceAccount. The attacker looked for AWS keys to gain access to the cloud service provider. They then used a Daemonset to deploy malicious pods across the cluster, using a container image to do so (as was done in the Dero attack). The goal of the malicious pods was crypto mining.
What to learn from this attack and how likely are you to be exposed?
The initial step in this attack assumes a scenario where, not only is your Kubernetes API server open, but it is accepting requests that privileged users have. This already signifies an utter and total compromise of the cluster, in and of itself. It is possible to configure K8s api to accept unauthenticated requests with privilege escalation, but that’s a lower level API configuration that is not open by default (see here the Kubernetes guidance for securing the Kubernetes API). It is highly unlikely for any Kubernetes environment, aside from a honeypot, would be exposed to this entry tactic.
The rest of the attack operates with this privileged access, encountering very little bumps along the way, if any.
To examine the RBAC piece for learnings, by binding the ClusterRole that had admin privileges to the ServiceAccount, even if the role was deleted at some point, the ServiceAccount would continue with privileges. To protect against this you would need a way to audit your RBAC permissions.
And to spread laterally, the attackers used the same Daemonset technique as in the Dero campaign; reminding us to prevent the creation of malicious workloads from Daemonsets; for example, by stopping workloads containing images not from trusted image sources.
RBAC-Buster TLDR
You are most likely not susceptible to this attack, given the highly unlikely point of entry.
That being said, you still will want to audit your RBAC permissions.
Other commentary on Kubernetes attacks in 2023
All in all, we are looking at 3 attacks that would be relatively unlikely to take place outside the presence of a honeypot, and 1 attack (Scarleteel) that follows a more common path, starting with a web app vulnerability and moving laterally into the Kubernetes environment from there.
At this point, you may be thinking, “Why would I set my API to such an egregious setting that allows anonymous access?” Most likely, you wouldn’t; but there are other, more realistic, ways that this same entrypoint could be exploited in your organization.
- Malicious Insider or Contractor: Oftentimes, individuals are granted direct access to Kubernetes in some capacity and may plant backdoors or cryptocurrency miners similar to the ones in these attacks.
- Stolen or exposed credential file: A developer may unknowingly check in a Service Account token or .kubeconfig file to a public .git repository that - if found - could leave a cluster open to attack.
There are a few other key points that were made in the research of these attacks that have implications for how you might protect yourself from these and future attacks:
Doing nothing is not an option:
The research team responsible for finding the RBAC-Buster attack said that 60% of exposed clusters they found had an active campaign running. This doesn’t mean that 60% of all clusters are exposed . . . instead the takeaway here is that attackers are actively searching for mistakes, misconfigurations and a way into your Kubernetes environment.
Real-time security for Kubernetes is also not an option:
The same research team mentioned that, “Most of the clusters we encountered were only accessible for a few hours.” This is a hugely important point, because it highlights the ephemeral nature of Kubernetes clusters, and how what today points to an exploitation and exposure might tomorrow be closed off to attackers. For defenders, this means a nightmare in remediation if you are working with polling intervals that can’t show these changes over time.
Conclusion
In general, comparing across the four attacks, the attack chain of Scarleteel has the higher probability of showing up in your environment. he pattern that Scarleteel uses, to find an initial foothold with a web app vulnerability and then move laterally, is the hallmark Kubernetes attack, because web vulnerabilities are so common and a misconfigured Kubernetes environment allows for either broad lateral movement.
That being said, there are realistic ways the initial entry point of the other attacks could be an option in your organization; for example through malicious insiders or a stolen credentials file.
Regardless, if you are relying solely on admission control, or reverse-engineering detection that relies on runtime events, when the next attack comes, you either won’t detect it at all, or you’ll detect it too late. To detect these and other attacks, you need a real-time, combined view of Kubernetes risk. Defense-in-depth is best practice; but if defense-in-depth provides no view of how all the different components work together, you are still one step behind the attacker. In Kubernetes especially, you need to be able to see how each part of the attacker’s campaign works together.
For a detailed analysis of how various protection and detection capabilities like KSPM, runtime, admission controllers and CSPM, would stack up against these attacks, download the full defender’s guide to new Kubernetes attacks.
Get a Demo of KSOC today
See how KSOC can protect you in real-time against both these attacks, as well as the next Kubernetes-targeted attack.
