Describe the bug
I used centos stream image to create an openshift cluster, the bootstrap process always failed because crio service failed to start.
There is an invalid symlink /opt/cni on bootstrap node, after I removed it and restart crio.service, bootstrap can continue and succeed.
Version
4.21.0-okd-scos.ec.11
Reproducibility
100%
Log Bundle
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: crio.service: Scheduled restart job immediately on client request, restart counter is at 279.
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: Starting crio.service - Container Runtime Interface for OCI (CRI-O)...
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org (crio)[38299]: crio.service: Referenced but unset environment variable evaluates to an empty string: CRIO_CONFIG_OPTIONS, CRIO_METRICS_OPTIONS, CRIO_NETWORK_OPTIONS, CRIO_RUN>
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.412206829Z" level=info msg="Updating config from single file: /etc/crio/crio.conf"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.412241656Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.413357811Z" level=info msg="Updating config from path: /etc/crio/crio.conf.d"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.413411349Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/50-bootstrap-override.conf"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427330688Z" level=info msg="Starting CRI-O, version: 1.34.0, git: unknown(clean)"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427430157Z" level=info msg="Node configuration value for hugetlb cgroup is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427437966Z" level=info msg="Node configuration value for pid cgroup is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427488888Z" level=info msg="Node configuration value for memoryswap cgroup is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427495433Z" level=info msg="Node configuration value for cgroup v2 is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.43780848Z" level=info msg="Node configuration value for systemd AllowedCPUs is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.438366133Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONF>
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.439147226Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CA>
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.440763827Z" level=info msg="Using runtime handler crun version 1.24, commit: 54693209039e5e04cbe3c8b1cd5fe2301219f0a1, rundir: /run/cru>
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445848354Z" level=info msg="Checkpoint/restore support enabled"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445867357Z" level=info msg="Using the internal default seccomp profile"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445874564Z" level=info msg="AppArmor is disabled by the system or at CRI-O build-time"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445880111Z" level=info msg="No blockio config file specified, blockio not configured"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445885407Z" level=info msg="RDT not available in the host system"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.448031865Z" level=info msg="Conmon does support the --sync option"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.448068758Z" level=info msg="Conmon does support the --log-global-size-max option"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.448191062Z" level=fatal msg="validating network config: invalid plugin_dirs entry: mkdir /opt/cni: file exists"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: crio.service: Failed with result 'exit-code'.
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: Failed to start crio.service - Container Runtime Interface for OCI (CRI-O).
Dec 09 08:21:52 test-bootstrap.confidential-cluster.org systemd[1]: crio.service: Scheduled restart job, restart counter is at 280.
Dec 09 08:21:52 test-bootstrap.confidential-cluster.org systemd[1]: Starting crio.service - Container Runtime Interface for OCI (CRI-O)...
Describe the bug
I used centos stream image to create an openshift cluster, the bootstrap process always failed because crio service failed to start.
There is an invalid symlink /opt/cni on bootstrap node, after I removed it and restart crio.service, bootstrap can continue and succeed.
Version
4.21.0-okd-scos.ec.11
Reproducibility
100%
Log Bundle
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: crio.service: Scheduled restart job immediately on client request, restart counter is at 279.
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: Starting crio.service - Container Runtime Interface for OCI (CRI-O)...
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org (crio)[38299]: crio.service: Referenced but unset environment variable evaluates to an empty string: CRIO_CONFIG_OPTIONS, CRIO_METRICS_OPTIONS, CRIO_NETWORK_OPTIONS, CRIO_RUN>
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.412206829Z" level=info msg="Updating config from single file: /etc/crio/crio.conf"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.412241656Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.413357811Z" level=info msg="Updating config from path: /etc/crio/crio.conf.d"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.413411349Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/50-bootstrap-override.conf"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427330688Z" level=info msg="Starting CRI-O, version: 1.34.0, git: unknown(clean)"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427430157Z" level=info msg="Node configuration value for hugetlb cgroup is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427437966Z" level=info msg="Node configuration value for pid cgroup is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427488888Z" level=info msg="Node configuration value for memoryswap cgroup is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.427495433Z" level=info msg="Node configuration value for cgroup v2 is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.43780848Z" level=info msg="Node configuration value for systemd AllowedCPUs is true"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.438366133Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONF>
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.439147226Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CA>
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.440763827Z" level=info msg="Using runtime handler crun version 1.24, commit: 54693209039e5e04cbe3c8b1cd5fe2301219f0a1, rundir: /run/cru>
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445848354Z" level=info msg="Checkpoint/restore support enabled"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445867357Z" level=info msg="Using the internal default seccomp profile"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445874564Z" level=info msg="AppArmor is disabled by the system or at CRI-O build-time"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445880111Z" level=info msg="No blockio config file specified, blockio not configured"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.445885407Z" level=info msg="RDT not available in the host system"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.448031865Z" level=info msg="Conmon does support the --sync option"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.448068758Z" level=info msg="Conmon does support the --log-global-size-max option"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org crio[38299]: time="2025-12-09T08:21:42.448191062Z" level=fatal msg="validating network config: invalid plugin_dirs entry: mkdir /opt/cni: file exists"
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: crio.service: Failed with result 'exit-code'.
Dec 09 08:21:42 test-bootstrap.confidential-cluster.org systemd[1]: Failed to start crio.service - Container Runtime Interface for OCI (CRI-O).
Dec 09 08:21:52 test-bootstrap.confidential-cluster.org systemd[1]: crio.service: Scheduled restart job, restart counter is at 280.
Dec 09 08:21:52 test-bootstrap.confidential-cluster.org systemd[1]: Starting crio.service - Container Runtime Interface for OCI (CRI-O)...