Permission Denied Errors During Kubespray Playbook Execution

[yvesmarinho]

What happened?

Description:
While executing the Kubespray playbook, several tasks failed due to permission denied errors. The errors occurred when attempting to create directories on multiple nodes. This issue prevents the successful completion of the playbook.

Environment:

• Kubespray Version: release-2.27
• Ansible Version: 2.16.14
• Python Version: 3.11
• OS: Debian 12
• Nodes:
  • dsmaster01.vya.digital
  • dsmaster02.vya.digital
  • dsworker01.vya.digital

Steps to Reproduce:

1. Clone the Kubespray repository:

   git clone https://github.com/kubernetes-sigs/kubespray.git
   cd kubespray

2. Install the Kubespray dependencies:

   VENVDIR=kubespray-venv
   KUBESPRAYDIR=kubespray
   python3 -m venv $VENVDIR
   source $VENVDIR/bin/activate
   cd $KUBESPRAYDIR
   pip install -U -r requirements.txt

3. Copy the sample inventory and adjust as needed:

   cp -rfp inventory/sample inventory/mycluster

4. Edit the inventory/mycluster/inventory.ini file to include the master and worker servers.
5. Run the Kubespray playbook to configure the cluster:

   ansible-playbook -i inventory/vyacluster/inventory.ini --become --become-user=archaris cluster.yml

Sudo permissions:

sudo -l
Matching Defaults entries for archaris on dsmaster01:
    env_reset, mail_badpass,
    secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin, use_pty

User archaris may run the following commands on dsmaster01:
    (ALL) NOPASSWD: ALL
    (ALL : ALL) ALL
    (ALL) NOPASSWD: ALL

Groups

archaris@dsmaster01:~/kubespray$ groups archaris
archaris : archaris adm sudo

Folder creation

archaris@dsmaster01:~/kubespray$ sudo mkdir -p /var/lib/kubelet/volumeplugins
archaris@dsmaster01:~/kubespray$ sudo mkdir -p /etc/kubernetes/manifests
archaris@dsmaster01:~/kubespray$ sudo mkdir -p /usr/local/bin/kubernetes-scripts
archaris@dsmaster01:~/kubespray$ ls -ld /var/lib/kubelet /etc/kubernetes /usr/local/bin
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /etc/kubernetes
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /usr/local/bin
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /var/lib/kubelet

Observed Behavior:
The playbook fails with permission denied errors when attempting to create directories on the nodes.

Error Logs:

TASK [bootstrap-os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora)] ***
fatal: [dsmaster01]: FAILED! => {"changed": false, "msg": "Command failed rc=1, out=, err=Could not set pretty hostname: Connection timed out\n"}
fatal: [dsworker01]: FAILED! => {"changed": false, "msg": "Command failed rc=1, out=, err=Could not set pretty hostname: Tempo esgotado para conexão\n"}
fatal: [dsmaster02]: FAILED! => {"changed": false, "msg": "Command failed rc=1, out=, err=Could not set pretty hostname: Connection timed out\n"}

TASK [kubernetes/preinstall : Create kubernetes directories path=/etc/kubernetes, state=directory, owner=kube, mode=0755, _ansible_check_mode=False, _ansible_no_log=False, _ansible_debug=False, _ansible_diff=False, _ansible_verbosity=4, _ansible_version=2.16.14, _ansible_module_name=file, _ansible_syslog_facility=LOG_USER, _ansible_selinux_special_fs=['fuse', 'nfs', 'vboxsf', 'ramfs', '9p', 'vfat'], _ansible_string_conversion_action=warn, _ansible_socket=None, _ansible_shell_executable=/bin/sh, _ansible_keep_remote_files=False, _ansible_tmpdir=None, _ansible_remote_tmp=~/.ansible/tmp] ***
failed: [dsmaster01.vya.digital] (item=/etc/kubernetes) => changed=false
  ansible_loop_var: item
  invocation:
    module_args:
      _diff_peek: null
      _original_basename: null
      access_time: null
      access_time_format: '%Y%m%d%H%M.%S'
      attributes: null
      follow: true
      force: false
      group: null
      mode: '0755'
      modification_time: null
      modification_time_format: '%Y%m%d%H%M.%S'
      owner: kube
      path: /etc/kubernetes
      recurse: false
      selevel: null
      serole: null
      setype: null
      attributes: null
  msg: "There was an issue creating /etc/kubernetes as requested: [Errno 13] Permission denied: b'/etc/kubernetes'"

failed: [dsworker01.vya.digital] (item=/etc/kubernetes/manifests) => changed=false
  msg: "There was an issue creating /etc/kubernetes/manifests as requested: [Errno 13] Permission denied: b'/etc/kubernetes'"

failed: [dsmaster02.vya.digital] (item=/etc/kubernetes/manifests) => changed=false
  msg: "There was an issue creating /etc/kubernetes/manifests as requested: [Errno 13] Permission denied: b'/etc/kubernetes'"

failed: [dsmaster01.vya.digital] (item=/usr/local/bin/kubernetes-scripts) => changed=false
  msg: "There was an issue creating /usr/local/bin/kubernetes-scripts as requested: [Errno 13] Permission denied: b'/usr/local/bin/kubernetes-scripts'"

failed: [dsworker01.vya.digital] (item=/usr/local/bin/kubernetes-scripts) => changed=false
  msg: "There was an issue creating /usr/local/bin/kubernetes-scripts as requested: [Errno 13] Permission denied: b'/usr/local/bin/kubernetes-scripts'"

failed: [dsmaster02.vya.digital] (item=/usr/local/bin/kubernetes-scripts) => changed=false
  msg: "There was an issue creating /usr/local/bin/kubernetes-scripts as requested: [Errno 13] Permission denied: b'/usr/local/bin/kubernetes-scripts'"

failed: [dsmaster01.vya.digital] (item=/var/lib/kubelet/volumeplugins) => changed=false
  msg: "There was an issue creating /var/lib/kubelet as requested: [Errno 13] Permission denied: b'/var/lib/kubelet'"

failed: [dsworker01.vya.digital] (item=/var/lib/kubelet/volumeplugins) => changed=false
  msg: "There was an issue creating /var/lib/kubelet as requested: [Errno 13] Permission denied: b'/var/lib/kubelet'"

failed: [dsmaster02.vya.digital] (item=/var/lib/kubelet/volumeplugins) => changed=false
  msg: "There was an issue creating /var/lib/kubelet as requested: [Errno 13] Permission denied: b'/var/lib/kubelet'"

Playbook Execution Summary:

PLAY RECAP **********************************************************************************************
dsmaster01.vya.digital     : ok=64   changed=3    unreachable=0    failed=1    skipped=41   rescued=0    ignored=0
dsmaster02.vya.digital     : ok=49   changed=3    unreachable=0    failed=1    skipped=35   rescued=0    ignored=0
dsworker01.vya.digital     : ok=47   changed=3    unreachable=0    failed=1    skipped=37   rescued=0    ignored=0
Wednesday 19 March 2025  10:24:46 -0300 (0:00:01.404)       0:00:24.307 ******* 
=============================================================================== 
bootstrap-os : Ensure iproute2 is installed ------------------------------------------------------ 1.78s
/home/archaris/kubespray/roles/bootstrap-os/tasks/debian.yml:71 ----------------------------------------
bootstrap-os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora) --- 1.48s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:46 ------------------------------------------
kubernetes/preinstall : Create kubernetes directories -------------------------------------------- 1.40s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0050-create_directories.yml:2 ---------------
bootstrap-os : Gather facts ---------------------------------------------------------------------- 1.30s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:41 ------------------------------------------
kubernetes/preinstall : Remove swapfile from /etc/fstab ------------------------------------------ 1.11s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0010-swapoff.yml:10 -------------------------
kubernetes/preinstall : Mask swap.target (persist swapoff) --------------------------------------- 0.97s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0010-swapoff.yml:20 -------------------------
Gather necessary facts (hardware) ---------------------------------------------------------------- 0.91s
/home/archaris/kubespray/playbooks/facts.yml:38 --------------------------------------------------------
bootstrap-os : Create remote_tmp for it is used by another module -------------------------------- 0.90s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:35 ------------------------------------------
bootstrap-os : Ensure bash_completion.d folder exists -------------------------------------------- 0.86s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:58 ------------------------------------------
adduser : User | Create User --------------------------------------------------------------------- 0.85s
/home/archaris/kubespray/roles/adduser/tasks/main.yml:7 ------------------------------------------------
adduser : User | Create User Group --------------------------------------------------------------- 0.63s
/home/archaris/kubespray/roles/adduser/tasks/main.yml:2 ------------------------------------------------
kubernetes/preinstall : Check if /etc/fstab exists ----------------------------------------------- 0.62s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0010-swapoff.yml:2 --------------------------
kubernetes/preinstall : Disable swap ------------------------------------------------------------- 0.62s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0010-swapoff.yml:25 -------------------------
kubernetes/preinstall : Fetch resolv.conf -------------------------------------------------------- 0.59s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0020-set_facts.yml:37 -----------------------
Gather necessary facts (network) ----------------------------------------------------------------- 0.57s
/home/archaris/kubespray/playbooks/facts.yml:30 --------------------------------------------------------
Gather minimal facts ----------------------------------------------------------------------------- 0.51s
/home/archaris/kubespray/playbooks/facts.yml:21 --------------------------------------------------------
bootstrap-os : Include vars ---------------------------------------------------------------------- 0.37s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:14 ------------------------------------------
kubernetes/preinstall : Stop if cgroups are not enabled on nodes --------------------------------- 0.36s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0040-verify-settings.yml:103 ----------------
kubernetes/preinstall : Check if kubelet is configured ------------------------------------------- 0.35s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0020-set_facts.yml:69 -----------------------
kubernetes/preinstall : Check systemd-resolved --------------------------------------------------- 0.35s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0020-set_facts.yml:51 -----------------------

Expected Behavior:
The playbook should successfully create the necessary directories on all nodes without permission errors.

What did you expect to happen?

Folder creation

archaris@dsmaster01:~/kubespray$ sudo mkdir -p /var/lib/kubelet/volumeplugins
archaris@dsmaster01:~/kubespray$ sudo mkdir -p /etc/kubernetes/manifests
archaris@dsmaster01:~/kubespray$ sudo mkdir -p /usr/local/bin/kubernetes-scripts
archaris@dsmaster01:~/kubespray$ ls -ld /var/lib/kubelet /etc/kubernetes /usr/local/bin
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /etc/kubernetes
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /usr/local/bin
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /var/lib/kubelet

How can we reproduce it (as minimally and precisely as possible)?

Folder creation

archaris@dsmaster01:~/kubespray$ sudo mkdir -p /var/lib/kubelet/volumeplugins
archaris@dsmaster01:~/kubespray$ sudo mkdir -p /etc/kubernetes/manifests
archaris@dsmaster01:~/kubespray$ sudo mkdir -p /usr/local/bin/kubernetes-scripts
archaris@dsmaster01:~/kubespray$ ls -ld /var/lib/kubelet /etc/kubernetes /usr/local/bin
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /etc/kubernetes
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /usr/local/bin
drwxr-xr-x 3 root root 4096 Mar 19 10:41 /var/lib/kubelet

OS

RHEL 9

Version of Ansible

ansible [core 2.14.18]
config file = /home/archaris/kubespray/ansible.cfg
configured module search path = ['/home/archaris/kubespray/library']
ansible python module location = /usr/lib/python3/dist-packages/ansible
ansible collection location = /home/archaris/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True

Version of Python

Python 3.11.2

Version of Kubespray (commit)

release-2.27

Network plugin used

calico

Full inventory with variables

ansible-vars.txt

Command used to invoke ansible

ansible-playbook -vvvv --become --become-user=archaris --become-method=sudo cluster.yml -i ~/vyacluster/inventory.ini

Output of ansible run

PLAY RECAP **********************************************************************************************
dsmaster01.vya.digital : ok=64 changed=3 unreachable=0 failed=1 skipped=41 rescued=0 ignored=0
dsmaster02.vya.digital : ok=49 changed=3 unreachable=0 failed=1 skipped=35 rescued=0 ignored=0
dsworker01.vya.digital : ok=47 changed=3 unreachable=0 failed=1 skipped=37 rescued=0 ignored=0
Wednesday 19 March 2025 10:24:46 -0300 (0:00:01.404) 0:00:24.307 *******

bootstrap-os : Ensure iproute2 is installed ------------------------------------------------------ 1.78s
/home/archaris/kubespray/roles/bootstrap-os/tasks/debian.yml:71 ----------------------------------------
bootstrap-os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora) --- 1.48s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:46 ------------------------------------------
kubernetes/preinstall : Create kubernetes directories -------------------------------------------- 1.40s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0050-create_directories.yml:2 ---------------
bootstrap-os : Gather facts ---------------------------------------------------------------------- 1.30s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:41 ------------------------------------------
kubernetes/preinstall : Remove swapfile from /etc/fstab ------------------------------------------ 1.11s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0010-swapoff.yml:10 -------------------------
kubernetes/preinstall : Mask swap.target (persist swapoff) --------------------------------------- 0.97s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0010-swapoff.yml:20 -------------------------
Gather necessary facts (hardware) ---------------------------------------------------------------- 0.91s
/home/archaris/kubespray/playbooks/facts.yml:38 --------------------------------------------------------
bootstrap-os : Create remote_tmp for it is used by another module -------------------------------- 0.90s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:35 ------------------------------------------
bootstrap-os : Ensure bash_completion.d folder exists -------------------------------------------- 0.86s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:58 ------------------------------------------
adduser : User | Create User --------------------------------------------------------------------- 0.85s
/home/archaris/kubespray/roles/adduser/tasks/main.yml:7 ------------------------------------------------
adduser : User | Create User Group --------------------------------------------------------------- 0.63s
/home/archaris/kubespray/roles/adduser/tasks/main.yml:2 ------------------------------------------------
kubernetes/preinstall : Check if /etc/fstab exists ----------------------------------------------- 0.62s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0010-swapoff.yml:2 --------------------------
kubernetes/preinstall : Disable swap ------------------------------------------------------------- 0.62s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0010-swapoff.yml:25 -------------------------
kubernetes/preinstall : Fetch resolv.conf -------------------------------------------------------- 0.59s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0020-set_facts.yml:37 -----------------------
Gather necessary facts (network) ----------------------------------------------------------------- 0.57s
/home/archaris/kubespray/playbooks/facts.yml:30 --------------------------------------------------------
Gather minimal facts ----------------------------------------------------------------------------- 0.51s
/home/archaris/kubespray/playbooks/facts.yml:21 --------------------------------------------------------
bootstrap-os : Include vars ---------------------------------------------------------------------- 0.37s
/home/archaris/kubespray/roles/bootstrap-os/tasks/main.yml:14 ------------------------------------------
kubernetes/preinstall : Stop if cgroups are not enabled on nodes --------------------------------- 0.36s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0040-verify-settings.yml:103 ----------------
kubernetes/preinstall : Check if kubelet is configured ------------------------------------------- 0.35s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0020-set_facts.yml:69 -----------------------
kubernetes/preinstall : Check systemd-resolved --------------------------------------------------- 0.35s
/home/archaris/kubespray/roles/kubernetes/preinstall/tasks/0020-set_facts.yml:51 -----------------------

Anything else we need to know

archaris@dsmaster01:~/kubespray$ ansible-playbook -u archaris -e ansible_ssh_user=archaris -b --become-user=archaris -e dir=pwd scripts/collect-info.yaml
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'
[WARNING]: Skipping callback plugin 'ara_default', unable to load

PLAY [Collect debug info] *******************************************************************************
skipping: no hosts matched

PLAY RECAP **********************************************************************************************

Wednesday 19 March 2025 11:04:04 -0300 (0:00:00.002) 0:00:00.002 *******

[vdveldet]

The example you gave is using sudo to go from archaris to root.

archaris@dsmaster01:~/kubespray$ sudo mkdir -p /var/lib/kubelet/volumeplugins

But you define --become-user=archaris while i would expect --become-user=root for comparing.

[yvesmarinho]

The example you gave is using sudo to go from archaris to root.

archaris@dsmaster01:~/kubespray$ sudo mkdir -p /var/lib/kubelet/volumeplugins

But you define --become-user=archaris while i would expect --become-user=root for comparing.

The user is disabled for external connections for security reasons.
The user "archaris" was created and configured to connect without a password and sudo without a password.
What I need to know is which part of the framework is not working so I can analyze the code and solve it.

[VannTen]

Side-note: looks like the auto-label does not work correctly
xref: #11880

[yvesmarinho]

I'm sorry, but I can see this. All I see in the error log is that the folder cannot be created. It doesn't appear to be using sudo at this point.
Is there any way to know what code is being executed when the error occurs?

[VannTen]

The become-user is not expected to have sudo rights, the connecting user (the one you're connecting as) is. `-e ansible_ssh_user=archaris -b --become-user=archaris` -> does not make sense, you don't need to become the same user than the one you're connecting as. Very likely that you just need to drop the become-user and rely on the default (root) /remove-kind bug /kind support /close

[k8s-ci-robot]

@VannTen: Closing this issue.

Details

In response to this:

The become-user is not expected to have sudo rights, the connecting user (the one you're connecting as) is.

-e ansible_ssh_user=archaris -b --become-user=archaris -> does not make sense, you don't need to become the same user than the one you're connecting as.

Very likely that you just need to drop the become-user and rely on the default (root)

/remove-kind bug
/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yvesmarinho]

I am very disappointed with this answer.
all cybersecurity teams processes recommend disabling root.
you are telling me the opposite.
seriously?!
I'm going to abandon this project and manually install Kubernetes.
I was hoping to find a project to contribute to.

The become-user is not expected to have sudo rights, the connecting user (the one you're connecting as) is.

-e ansible_ssh_user=archaris -b --become-user=archaris -> does not make sense, you don't need to become the same user than the one you're connecting as.

Very likely that you just need to drop the become-user and rely on the default (root)

/remove-kind bug
/kind support
/close

[VannTen]

all cybersecurity teams processes recommend disabling root.

You're confusing disabling root login and disabling root. I'm not sure the latter is truly possible on Linux, I'm pretty sure you need uid 0. What's why Ansible usually use become, because you don't connect as root, you use privilege elevation (aka sudo). But you do use uid=0.