Skip to content

README.md

Latest commit

 

History

History
428 lines (311 loc) · 17.5 KB

README.md

File metadata and controls

428 lines (311 loc) · 17.5 KB
Raw

Pets and core services

Architecture

Physical view

C4Container

Person(admin, "Micro Cloud administrator", "You")
System_Ext(mesh, "Tailscale Mesh VPN", "Network Mesh VPN / SDN.")

Enterprise_Boundary(ring0, "Ring 0 - Management") {
    Container(headnode, "Headnode-0", "ubuntu, incus, tailscale")
}

BiRel(mesh, headnode, "is connected")
BiRel(admin, mesh, "is connected")
Rel(admin, headnode, "has physical access<br> and manages")
Loading

Virtualization view

C4Container

Person(admin, "Micro Cloud administrator", "You")
System_Ext(mesh, "Tailscale Mesh VPN", "Network Mesh VPN / SDN.")

Enterprise_Boundary(ring0, "Ring 0 - Management") {
    Container_Boundary(headnode, "headnode-0") {
        Container(pki, "PKI", "lxc")
        Container(bootstrap, "Bootstrap", "lxc")
        Container(management, "Management", "KVM, Kubernetes")
    }

BiRel(mesh, management, "is connected")
Rel(bootstrap, management, "installs")
}
Loading

Components view

C4Component

Person(admin, "Micro Cloud administrator", "You")
System_Ext(mesh, "Tailscale Mesh VPN", "Network Mesh VPN / SDN.")

Enterprise_Boundary(ring0, "Ring 0") {
    Container_Boundary(headnode, "The first headnode to be deployed") {
        Container_Boundary(bootstrap, "Bootstrap instance") {
            Component(netboot, "Netboot services", "kea, matchbox", "Provides netboot and bootstrap facilities.")
            Component(machinecfg, "Configuration generator", "machinecfg", "Imports DCIM data into the bootstrap facility.")
        }

        Container_Boundary(pki, "PKI instance") {
            Component(openbao, "PKI", "cfssl, openbao", "Provides certificates through webservice.")
        }

        Container_Boundary(management, "Management instance") {
            Component(tailscale, "Tailscale operator", "helm, tailscale-operator")
            Component(issuer, "Provides certificates", "helm, cert-manager, vault-issuer")
            Component(id, "ID Provider", "helm, authentik")
            Component(dcim, "CMDB", "helm, netbox")
            Component(deployment, "Platform deployer", "helm, kamaji, kamaji")
        }
    }
}

Rel(openbao, issuer, "Provides certificates")
Rel(tailscale, id, "Connects to the SDN")
Rel(tailscale, dcim, "Connects to the SDN")
BiRel(mesh, tailscale, "Is connected")

UpdateLayoutConfig($c4ShapeInRow="2", $c4BoundaryInRow="2")
Loading

Sequencing

This is the bootstrap's order to create the core services of the platform.

graph
    subgraph offline["Hands on the headnode"]
        uefi["UEFI setup"]
        subgraph prepare["Preparing the headnode"]
            Tailscale["Mesh VPN client"] -->
            Incus["LXC & KVM platform"]
        end
    end

    subgraph pki["Preparing the PKI"]
        init["Install the instance"]       -- The required tools are present -->
        bootstrap_pki["Bootstrap the PKI"] -- The root and intermediate CA have been created -->
        certificates["Create the first certificates"]
    end

    subgraph netboot["Netboot services"]
        cert["Push/pull certificates"] -- The certificates are available inside the instance -->
        bootstrap_netboot["Install the instance"]
    end

    subgraph management["Management node"]
        bootstrap_management["Install the instance"] -- The cluster is UP --> 
        deploy["Deploy services"]
    end

    uefi        -- The machine is properly configured (CPU settings, Disks…) -->
    prepare     -- The machine is online on the VPN, ready to be controlled remotely, the networking setup has been done -->
    pki         -- The PKI is ready to deliver certificates -->
    netboot     -- The netboot service is able to bootstrap the management node -->
    management
Loading

These are the management services, hosted by the Kubernetes-based management node.

flowchart LR
    subgraph system["System Kubernetes addons"]
        storage["Local storage provisioner"] -- PVC can be created        --> cnpg["PostgreSQL DBaaS operator"]
        cert-manager["Certificates manager"] -- The cert-manager is ready --> cert-issuer["Certificates issuer using the PKI"]
        eso["External Secrets Operator"] -- Syncs secrets from OpenBao --> k8s-secrets["Kubernetes Secrets"]
    end

    subgraph middlewares["Management middlewares"]
        authentik["Identity Provider"] -- A users' database is ready to be used --> netbox["CMDB + DCIM + IPAM"]
        forge["Forge"] -- Artifacts can be build from sources --> tinkerbell
        zot["OCI registry"] -- OCI images can be imported --> kamaji["Kubernetes Controlplane as a Service"]
        zot -- OCI images can be imported --> tinkerbell["Baremetal manager"] -- Tinkerbell has been installed --> clusterapi["Kubernetes Cluster API"] -- The CAPT and CAPK providers have been installed --> kaas["Baremetal Kubernetes as a Service"]
        tailscale["Mesh VPN operator"] -- Services can be published on the tailnet --> zot
        tailscale["Mesh VPN operator"] -- Services can be published on the tailnet -->  kamaji
    end

    system --> middlewares
Loading

Offline task: Preparing the headnode

These actions must be done in front of the headnode, using a KVM (Keyboard, Video, Mouse).
Let's make the machine join the mesh VPN and install the hosting tooling.

# A one-time script to execute on the machine
headnode$ wget https://raw.githubusercontent.com/mgrzybek/micro-cloud/refs/heads/main/ring0/scripts/init-headnode.sh

# Starting the init script
headnode$ bash ./init-headnode.sh

After this task, you should be able to connect against the headnode using Tailscale.

Activating the remote Incus access

# On the headnode, create a token
headnode$ incus config trust add tailscale

# On your workstation, declare the remote node
workstation$ incus remote add headnode-0 headnode-0
workstation$ incus remote switch headnode-0

Bootstrapping the PKI

These tasks are executed on your workstation, inside the git repository root path.

# Some attributes are required to create the root CA
export PKI_COUNTRY="FR"
export PKI_LOCATION="Paris"
export PKI_ORG="My Cloud"
export PKI_ORG_UNIT="CA Services"
export PKI_STATE="IDF"

cd ./ring0
task intermediate-fullchain

# The intermediate CA bundle should be present.
cat dist/bundle.crt

Bootstrapping the netboot services

First, we need to create the bootstrap instance and to configure it.
The task bootstrap command will set up the bootstrap instance with the network bridge, VLAN, and physical interface specified, and prepare the Talos artifacts required for deployment. Talos is a Kubernetes-native, minimal OS for managing bare metal clusters (https://talos.dev).

export BRIDGE_BOOTSTRAP_NAME=bootstrapbr0         # Depending on your incus configuration
export BRIDGE_BOOTSTRAP_VLAN=2                    # Depending on your network fabric
export PHYS_IFACE=enp2s0                          # Depending on your machine's configuration
export IFACE_BOOTSTRAP_IPADDR_CIDR=192.168.2.2/24 # Depending on your network fabric

# Talos attributes used to download the artifacts
export TALOS_FACTORY_URL=factory.talos.dev
export TALOS_FACTORY_UUID=11b09aacd770c0df15510e4d0815853404ada5d251a7461a21ab3ef73e1808ca
export TALOS_VERSION=v1.13.0

task bootstrap

Bootstrapping the forge

Let's deployment the forge. It will be used to create custom artifacts later on.

task forge

Bootstrapping the management node

Let's deploy the management instance. Some variables can be changed if required.

# In addition to the previous variables, some must be added.
export TS_AUTHKEY=xxxxxx
export TS_OPERATOR_CLIENT_ID=xxxxxx
export TS_OPERATOR_CLIENT_SECRET=xxxxxx

export BRIDGE_SERVICES_NAME=services0
export INSTANCE_MANAGEMENT_BOOTSTRAP_IPADDR_CIDR=192.168.2.3/24

export BRIDGE_SERVICES_NAME=services0
export INSTANCE_MANAGEMENT_SERVICES_IPADDR_CIDR=192.168.3.3/24

export BMAAS_NAMESPACE=bmaas-system

task management

Bootstrapping Flux (GitOps)

After task management, bootstrap FluxCD Operator so it takes over all platform workloads. The script creates the pre-requisite Secrets and the cluster-config ConfigMap, then applies the FluxInstance.

# Required variables (add to your variables.sh / environment)
export TS_OPERATOR_CLIENT_ID=xxxxxx
export TS_OPERATOR_CLIENT_SECRET=xxxxxx
export PKI_ENDPOINT=https://pki.<ts_suffix>
export PKI_ORG="My Cloud"
export DNS_IP=192.168.3.7
export HOOKOS_IP=192.168.3.6
export REGISTRY_IP=192.168.3.4
export TINKERBELL_IP=192.168.3.5
export ARTIFACTS_FILE_SERVER=http://<bootstrap-ip>/
export DHCP_BIND_INTERFACE=eth0
export BOOTSTRAP_ENDPOINT=http://<bootstrap-ip>/assets/tinkerbell
export ANNOUNCEMENTS_IFACE=eth1   # interface on management node facing the services VLAN
export GITHUB_ACTOR=<your GitHub username>
export GHCR_TOKEN=<GitHub PAT with read:packages>

task flux

From this point Flux manages the following components from the OCI artifact (ghcr.io/mgrzybek/micro-cloud):

Flux path Component Namespace
infrastructure/01-cilium Cilium CNI kube-system
infrastructure/01-cert-manager cert-manager cert-manager
infrastructure/02-cnpg-operator CloudNative PG operator cnpg-system
infrastructure/02-pg-cluster PostgreSQL cluster platform-management
infrastructure/02-tailscale-operator Tailscale Operator tailscale
apps/03-idp Authentik (IDP) platform-management
apps/04-cmdb Netbox (CMDB) platform-management
apps/04-eso External Secrets Operator external-secrets
apps/05-bmaas/zot Zot OCI registry tinkerbell-system
apps/05-bmaas/kamaji Kamaji kamaji-system
apps/05-bmaas/tinkerbell Tinkerbell tinkerbell-system

Monitor reconciliation with:

flux get all -A

Post-Flux setup

Exposing the IDP service

Once Flux has deployed Authentik, expose it on the tailnet:

task idp

Now you are ready to populate your directory as needed. Please note that Netbox uses two groups by default: staff and superusers. You have to add some users to these groups to be able to manage Netbox.

If you want to use Authentik's API to provision resources, you should create a token using the admin account at https://idp.your-tailscale-suffix/if/admin/#/core/tokens)..

Exposing the CMDB service

The official documentation on how to integrate the SSO mechanism between Authentik and Netbox is described here.

However, some care must be provided concerning the OAuth2/OpenID Provider section. The signing key should be set to the default one but the encryption key must be unselected. The Python module used by Netbox returns errors when trying to decrypt JWT tokens.

export APPLICATION_SLUG=netbox-cmdb
export CLIENT_ID=xxxxx
export CLIENT_SECRET=xxxxx

task cmdb

Warning

Installing Netbox can be quite long because of the database initialization.

Configuring Tinkerbell

Some pre-configuration is needed to make CoreDNS use Netbox as an IPAM. You must create a coredns service account able to read IP addresses from the IPAM section.

export COREDNS_NETBOX_TOKEN="A 40-character long token"
export REGISTRY_IP=192.168.3.4
export TINKERBELL_IP=192.168.3.5
export HOOKOS_IP=192.168.3.6
export DNS_IP=192.168.3.7

task bmaas

Releasing a new version

Push a semver tag to trigger the GitHub Actions workflow that publishes the Flux manifests as an OCI artifact:

git tag v1.2.3
git push origin v1.2.3

Flux picks up the new tag automatically (or pin a specific version in flux/clusters/management/flux-system/flux-instance.yaml).

Day-2: Adding a physical worker node

A physical machine can be added to the management cluster as a worker node via PXE boot through Matchbox.

Each worker requires a dedicated patch file in the gitops repository at:

$GITOPS_ROOT/ring0/workers/<hostname>/patch.yaml

This file contains all the static Talos configuration for the machine (network interfaces, VLANs, disk layout, Tailscale key). Two values are injected at runtime via envsubst:

  • ${INSTALL_IMAGE} — Talos factory installer URL (built from TALOS_FACTORY_URL, WORKER_TALOS_FACTORY_UUID, and the cluster's current Talos version)
  • ${MGMT_TS_IP} — management node Tailscale IP (resolved via tailscale status)

See core-services/management/talos/README.md for the expected file format.

# Identity
export WORKER_HOSTNAME=worker1           # Must match the patch file name in gitops
export WORKER_MAC=aa:bb:cc:dd:ee:ff     # MAC address of the NIC on the bootstrap VLAN

# Gitops repository containing the per-machine patch
export GITOPS_ROOT=/path/to/your-gitops-repo

# Talos factory image for this worker
export TALOS_FACTORY_URL=factory.talos.dev
export WORKER_TALOS_FACTORY_UUID=a78ca499dd99112bd2c2730b1b8a50375d8fa3af36f1a10b30a2fa83cc8c0d35

task add-worker

The task will:

  1. Query the management node's Talos version and Tailscale IP automatically
  2. Render the per-machine patch (envsubst on $GITOPS_ROOT/ring0/workers/$WORKER_HOSTNAME/patch.yaml)
  3. Merge the rendered patch on top of dist/worker.yaml via talosctl machineconfig patch
  4. Validate the resulting config with talosctl validate --mode metal
  5. Download the Talos PXE kernel and initramfs to the bootstrap instance (idempotent)
  6. Push the machine config and Matchbox profile/group files to the bootstrap instance
  7. Wait for the node to appear as Ready in the cluster

Power on the physical machine after launching the task; it will PXE boot and join the cluster automatically.

Note

dist/worker.yaml and dist/talosconfig must exist (generated by task management). The management node's Tailscale IP is resolved automatically at runtime (tailscale status --json). A static /etc/hosts entry is injected into the worker's machineconfig so the kubelet can reach the API server before Tailscale DNS is fully operational on first boot. yq must be installed (brew install yq / apt install yq) — it is used to strip the default HostnameConfig: auto: stable generated by Talos before applying the per-machine patch.

Troubleshooting

Here are some common issues and tips:

  • Unable to connect to headnode via Tailscale after init: Ensure the init-headnode.sh script completed successfully and that your Tailscale auth key is valid and not expired. Check network connectivity and firewall rules. Try pinging your tailnet members using tailscale ping command.

  • Bootstrap instance creation fails: Verify your network bridge and VLAN settings are correct and that the physical interface specified is up. Check Incus logs for errors.

  • Certificates not generated during PKI bootstrap: Confirm that make dist/intermediate-fullchain.pem runs without error. Verify the presence of the root CA and intermediate CA CSR files in the pets/ring0/core-services/pki/files directory.

  • Management services fail to deploy: Ensure that the bootstrap instance is fully operational and accessible. Check that the management instance joins the tailnet using tailscale ping management. Check Kubernetes cluster status and the logs of Helm deployments (cert-manager, authentik, netbox, etc.).

  • SSO login fails using Netbox: Check the signing and encryption keys used in the provider section in Authentik.

  • talosctl validate reports 'auto' and 'hostname' cannot be set at the same time: Talos v1.13+ generates a HostnameConfig: auto: stable document by default in talosctl gen config output. Both bootstrap-instance.sh and add-worker.sh strip this document with yq before applying per-machine patches. If the error appears, verify that yq is installed and that the stripping step ran (look for yq 'select(.kind != "HostnameConfig")' in the script output).

  • talosctl validate reports UserVolumeConfig min size is greater than max size: This collision occurs when a UserVolumeConfig is present in both the global dist/patch.yaml and a per-machine patch. The global patch must contain only TrustedRootsConfig (CA bundle). UserVolumeConfig belongs in dist/cp-patch.yaml (management node, via patch.yaml.j2) and in each worker's per-machine patch file.

  • talosctl validate reports static hostname is already set in v1alpha1 config: The machine config contains both a HostnameConfig document and a machine.network.hostname field. Since Talos v1.13+, hostname must be set exclusively via HostnameConfig. Remove any machine.network.hostname entry from patches and use kind: HostnameConfig instead.

  • add-worker task fails with Per-machine patch not found: Create $GITOPS_ROOT/ring0/workers/$WORKER_HOSTNAME/patch.yaml in your gitops repository. See core-services/management/talos/README.md for the expected structure.

  • General logs and debugging: Use journalctl on the bootstrap and pki instances to inspect system services. Use incus commands with verbose flags (-v) for detailed output. Use incus console management to see the console output of the management instance, especially during the boot process.

For more detailed help, visit the documentation and communities of the respective tools: