Skip to content

how-to-use-k8s-with-cri-containerd-and-kata.md

Latest commit

 

History

History
258 lines (186 loc) · 7.42 KB

how-to-use-k8s-with-cri-containerd-and-kata.md

File metadata and controls

258 lines (186 loc) · 7.42 KB

How to use Kata Containers and CRI (containerd plugin) with Kubernetes

This document describes how to set up a single-machine Kubernetes (k8s) cluster.

The Kubernetes cluster will use the CRI containerd plugin and Kata Containers to launch untrusted workloads.

Requirements

  • Kubernetes, kubelet, kubeadm
  • cri-containerd
  • Kata Containers

Note: For information about the supported versions of these components, see the Kata Containers versions.yaml file.

Install containerd with CRI plugin enabled

  • Follow the instructions from the CRI installation guide.

  • Check if containerd is now available

    $ command -v containerd

Install Kata Containers

Follow the instructions to install Kata Containers.

Install Kubernetes

  • Follow the instructions for kubeadm installation.

  • Check kubeadm is now available

    $ command -v kubeadm

Configure containerd to use Kata Containers

The CRI containerd plugin supports configuration for two runtime types.

  • Default runtime:

    A runtime that is used by default to run workloads.

  • Untrusted workload runtime:

    A runtime that will be used to run untrusted workloads. This is appropriate for workloads that require a higher degree of security isolation.

Define the Kata runtime as the untrusted workload runtime

Configure containerd to use the Kata runtime to run untrusted workloads by setting the plugins.cri.containerd.untrusted_workload_runtime config option:

$ sudo mkdir -p /etc/containerd/
$ cat << EOT | sudo tee /etc/containerd/config.toml
[plugins]
    [plugins.cri.containerd]
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = "/usr/bin/kata-runtime"
EOT

Note: Unless configured otherwise, the default runtime is set to runc.

Configure Kubelet to use containerd

In order to allow kubelet to use containerd (using the CRI interface), configure the service to point to the containerd socket.

  • Configure Kubernetes to use containerd

    $ sudo mkdir -p  /etc/systemd/system/kubelet.service.d/
$ cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
[Service]                                                 
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF

  • Inform systemd about the new configuration

    $ sudo systemctl daemon-reload

Configure proxy - OPTIONAL

If you are behind a proxy, use the following script to configure your proxy for docker, kubelet, and containerd:

$ services="
kubelet
containerd
docker
"

$ for service in ${services}; do

    service_dir="/etc/systemd/system/${service}.service.d/"
    sudo mkdir -p ${service_dir}

    cat << EOT | sudo tee "${service_dir}/proxy.conf"
[Service]
Environment="HTTP_PROXY=${http_proxy}"
Environment="HTTPS_PROXY=${https_proxy}"
Environment="NO_PROXY=${no_proxy}"
EOT
done

$ sudo systemctl daemon-reload

Start Kubernetes

  • Make sure containerd is up and running

    $ sudo systemctl restart containerd
$ sudo systemctl status containerd

  • Prevent conflicts between docker iptables (packet filtering) rules and k8s pod communication

    If Docker is installed on the node, it is necessary to modify the rule below. See kubernetes/kubernetes#40182 for further details.

    $ sudo iptables -P FORWARD ACCEPT

  • Start cluster using kubeadm

    $ sudo kubeadm init --skip-preflight-checks --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
$ export KUBECONFIG=/etc/kubernetes/admin.conf
$ sudo -E kubectl get nodes
$ sudo -E kubectl get pods

Install a Pod Network

A pod network plugin is needed to allow pods to communicate with each other.

  • Install the flannel plugin by following the Using kubeadm to Create a Cluster guide, starting from the Installing a pod network section.

  • Create a pod network using flannel

    Note: There is no known way to determine programmatically the best version (commit) to use. See flannel-io/flannel#995.

    $ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

  • Wait for the pod network to become available

    # number of seconds to wait for pod network to become available
$ timeout_dns=420

$ while [ "$timeout_dns" -gt 0 ]; do
    if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
        break
    fi

    sleep 1s
    ((timeout_dns--))
 done

  • Check the pod network is running

    $ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )

Allow pods to run in the master node

By default, the cluster will not schedule pods in the master node. To enable master node scheduling:

$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-

Create an unstrusted pod using Kata Containers

By default, all pods are created with the default runtime configured in CRI containerd plugin.

If a pod has the io.kubernetes.cri.untrusted-workload annotation set to "true", the CRI plugin runs the pod with the Kata Containers runtime.

  • Create an untrusted pod configuration

    $ cat << EOT | tee nginx-untrusted.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
  - name: nginx
    image: nginx
    
EOT

  • Create an untrusted pod

    $ sudo -E kubectl apply -f nginx-untrusted.yaml

  • Check pod is running

    $ sudo -E kubectl get pods

  • Check hypervisor is running

    $ ps aux | grep qemu

Delete created pod

$ sudo -E kubectl delete -f nginx-untrusted.yaml