Getting Started with Kubernetes Audit Logging

Kubernetes Audit logs

Regardless of where your Kubernetes cluster is running, whether it’s in the cloud with a Managed Kubernetes Service, on-prem with something like Kubeadm, or a raw Kubernetes cluster, taking advantage of Kubernetes audit logs will help you understand what is happening inside your cluster. Audit logs give you insights into  unauthorized access, modification, usage and suspicious activity, because Kubernetes audit logs show you every action that occurs on the API. The most important thing to do is get started!

Get the Kubernetes Security Risk Visibility checklist

What do you need in order to have the same visibility for your Kubernetes environment that you may have in other parts of your environment? 

What Is Kubernetes Audit Logging

By definition and per the kubernetes.io website, audit logging “provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself”.

What this essentially means is the audit logger can capture every single piece of information that occurs on the Kubernetes API. If an engineer runs kubectl get pods, the audit logger can pick it up. If a new service account is created, the audit logger can pick it up.

The goal of audit logging is to understand:

• What happened
• Why it happened
• What the destination was
• Where it came from
• Who did it
• Why they did it

Audit logging, however, is not on by default. Because of that, you can retrieve all of the audit logs you’d like, but you must set up a policy configuration.

Example Policy Configuration

Let’s take a look at two example policy configurations.

The first is rather verbose and contains several configurations including:

• Pod logs and status
• ConfigMaps
• Secrets

and even specific policies that state what the engineer may not want to consume.

This type of policy is getting extremely granular in a way that may be needed for an environment. The good thing about a Kubernetes audit policy is you can dive as deep as possible in terms of what you want to collect data for and what you don’t want to collect data for. This can always be updated as well, so if you decide to not consume logs for one policy, but want to change it later, you absolutely can.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None
    nonResourceURLs:
      - /healthz*
      - /logs
      - /metrics
      - /swagger*
      - /version
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: authorization.k8s.io
        resources:
          - subjectaccessreviews
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ''
        resources:
          - pods
        verbs:
          - create
          - patch
          - update
          - delete
  - level: Metadata
    omitStages:
      - RequestReceived

The next audit policy is far simpler to read, yet far more complex. Even though the YAML is shorter, what the policy below states is “consume everything that’s metadata”. As you can imagine, since metadata is data about data, there are going to be a lot of logs that are consumed here.

apiVersion: audit.k8s.io/v1
kind: Policy
rules: 
  - level: None

Both of these work just fine, but it’ll ultimately be up to the security team and the platform team to figure out just how many logs they want to consume and what’s truly needed.

Accessing Audit Logs

In many Managed Kubernetes Service offerings, you can take a look at Audit Logs right after the installation of the Kubernetes cluster. On-prem, you may need to do a few more configurations to ensure that this can occur.

Typically on-prem, you must configure and enable the Audit Logs during the installation of the Kubernetes cluster.

Below is an example command that you would need to run to point the Kubernetes cluster to the audit policy and point to where the logs should be stored.

kube-apiserver --audit-log-path=/var/log/kubernetes/apiserver/audit.log \ --audit-policy-file=/etc/kubernetes/audit-policies/name_of_policy.yaml

If you wanted to run the audit logging on Minikube for example, it would look something like this:

minikube start \
--extra-config=apiserver.audit-policy-file=/etc/audit.yaml \
--extra-config=apiserver.audit-log-path=-

The Minikube command tells Minikube to:

• Start the cluster
• Look at the audit.yaml policy
• Store the logs locally

Because there are many different ways to configure a Kubernetes cluster, there isn’t an exact method/command to run. It’s going to vary based on where you’re installing a Kubernetes cluster. However, the few things that you’ll definitely need are:

• An audit policy
• A place to store the logs

Collecting Audit Logs from a Cloud Provider

Thinking about audit logs from a cloud perspective, the good thing is that a lot of the heavy lifting is done for you.

If you’re running a Kubernetes cluster in Azure using Azure Kubernetes Service (AKS), not only are the audit logs turned on for you automatically, but container logging is enabled so you can start querying audit logs right away.

In AWS if you’re using EKS, it’s a bit different. When you’re creating an EKS cluster, you’ll have to manually set the option to turn on Audit logging. This isn’t necessarily a bad thing, and you should definitely turn on this configuration. For AWS, you can send audit logs to CloudWatch Logs or other destinations.

With GKE you can use GKE Cloud Audit Logs and with Rancher, you can enable rancher logging for both kube-api servers and clusters that are running in your environment. 

Remember, even if you aren’t managing the control plane, you should still understand what’s happening underneath the hood. Just because you aren’t managing the control plane doesn’t mean that attackers can’t breach your system.

Kubernetes Audit Logs: Use Cases

Kubernetes Audit Logs can be used for a variety of use cases, including understanding user activity in your system, tracking potential security threats, and compliance with policies. To better understand, let’s look at a Kubernetes audit log example:

If you have an organization-wide policy that users must authenticate using two-factor authentication before accessing certain resources, you could check the audit logs to ensure everyone is complying. Or if you want to better understand how people are using your clusters or who’s making what requests, you can analyze the audit logs to get that information.

See how KSOC uses audit logs to show how RBAC permissions are used

Get a demo of KSOC's AccessIQ feature for an understanding of the usage of valid permissions, versus just a laundry list of over permissions.

Why Audit Logs are Important

Generally, engineers are primarily concerned with getting the Kubernetes cluster up and running, so audit logging isn’t always the first order of priority. However, one of the easiest security measures an engineer can take  for a security-centric approach is turning on Kubernetes Audit Tools.

Kubernetes Audit Logs - Best Practices

To make sure you’re getting the most out of your Kubernetes Audit Logs, it can be important to adhere to Kubernetes audit policy best practices. Kubernetes audit log best practices can include understanding the audit log format, making sure there’s a process in place to regularly check audit logs in Kubernetes, and setting up alerts so that any suspicious activity can be flagged immediately.  

When using Kubernetes security tools like Audit Logs, it’s important to understand the data they provide. To do so, it can be helpful to understand the Kubernetes audit log format. Each log entry contains a timestamp, an action (e.g., create/delete/modify), the object type (e.g., pod, service, or namespace), and the user that initiated the request (if available). These logs can be used to identify any unauthorized access or suspicious activity occurring within your clusters.

What’s more, Kubernetes Audit Logs can be set up to monitor user activity. If a user attempts an action that violates the organization's security policy, the audit log will alert administrators and help them take corrective action quickly. This can help organizations better ensure that malicious actors are identified and blocked before any damage can be done.

Kubernetes Audit Logs also provide an audit trail for compliance and security audits. Organizations can demonstrate that their cloud environment is secure and compliant with industry regulations by having a detailed log of all activities in the cluster. This can be used to help protect organizations from potential fines or other legal action due to non-compliance. Additionally, it allows administrators to quickly view user activity and identify any anomalies or suspicious behavior.

Using Audit Logs for Kubernetes Performance Insights

Kubernetes Audit Logs also provide organizations with valuable insights into the performance of their operations in the cloud environment. This data can identify areas where resources are being over or under utilized, allowing administrators to adjust their usage accordingly. 

Conclusion

Getting started with Kubernetes audit logs is one of the easiest ways to begin on the path of improving security across your clusters, regardless of whether you are using managed Kubernetes services in the cloud or not. That being said, policy configurations are not wholly simple and getting access to the logs will require you to set up a storage location. These are worthwhile steps getting you started in understanding the actual activity and behavior of your clusters, which is helpful for both performance and security purposes. 

 For more steps to take around Kubernetes security, refer to the ultimate Kubernetes security checklist.