How to run Kubernetes without TLS for Debugging

4 min readAug 15, 2021

While researching Kubernetes components, you might want to check what happens inside the network. Kubernetes, by default, use encrypted communication. In my case, I wanted to see how the objects are being sent, especially between the API server to the kubelet agent in the worker nodes. In this post I will show you I configured my Kubernetes cluster to work without TLS, meaning, all the communication was in plain text. It is important to understand that you can still debug the communication without removing the TLS, which I will discuss at the end of this article. However, I want to show what I can do and see when I disable the TLS communication.

IMPORTANT: this is not recommended. I did it for research purpose and never do it on a production server, only in labs.

NOTE: I did it on Kubernetes version 1.18.0. From version 1.20.0 the insecure-port is deprecated.

We will use the following IP addresses of two EC2 instances:

Master: 172.31.23.181
Worker: 172.31.16.24

Settings on the master node

We will edit the /etc/kubernetes/manifests/kube-apiserver.yaml file like that:

Remove\comment the line with--insecure-port=0 flag. This will enable the default insecure port 8080. If you want to choose different port, edit the flag with a port, for example with 1337, like that--insecure-port=1337.
Add new flag --insecure-bind-address=0.0.0.0. It will expose the insecure port to any address and allow the worker to use the insecure port.

Check that it work by running curl http://<master_ip>:8080, you should get the list of all the API paths.

Settings on the worker node

Edit the API server address in the kubelet configuration file /etc/kubernetes/kubelet.conf with the insecure port (we chose the default 8080) and HTTP instead of HTTPS:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: http://172.31.23.181:8080
  name: default-cluster
...

The next step is to edit the kubelet’s service configuration file (/etc/systemd/system/kubelet.service.d/10-kubeadm.conf) and remove the line --boostrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf from the KUBELET_KUBECONFIG_ARGS argument.

In my case I just commented the line and created the same line below without the --boostrap-kubeconfig:

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
#Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

After saving it you will need to restart the service:

systemctl daemon-reload
service kubelet restart

Check that it work by running curl http://<master_ip>:8080, you should get the list of all the API paths.

Creating a new pod via insecure port and sniff the traffic

Create a YAML pod:

cat <<EOF >./mypod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: super-user-pod2
spec:
  containers:
  - name: redis
    image: busybox:1.28
    securityContext:
      capabilities:
        add: ["SYS_TIME"]
EOF

We are going to use curl so we need to convert it to JSON:

kubectl convert -f mypod.yaml --local -o json >  mypod.json

Before we create the pod we will open a new terminal on the master and sniff the network to a file:

tcpdump -i eth0 -w ./mypod.pcap

All is left to do is to create the pod via the insecure port with our root service account:

curl -k -X POST -H "Content-Type: application/json" http://127.0.0.1:8080/api/v1/namespaces/default/pods -d @mypod.json

Notice that we didn’t need to use any account for the creation because when insecure-port is enabled all the requests bypass the authentication and authorization mechanism.

Unencrypted network traffic between the master to the worker

Open the pcap with wireshark and search for the traffic between the master to the worker with this filter:

ip.src == 172.31.23.181 && ip.dst == 172.31.16.24

Then we right click one interesting packet and choose Follow -> TCP Stream:

We can see all the unencrypted traffic:

Conclusions

Watching the plain text communication can help find interesting stuff, maybe some vulnerabilities, and help understand what happens when an API request is sent.

I mentioned that the insecure-port is deprecated from version 1.20 but you can still do it with a TLS enabled API server. You can use kubectl with verbosity enabled (i.e kubectl get pods -v9 and it’ll show you exactly what the request\response was. You can also do it with burp.