- A complete guide for building a virtualized Kubernetes homelab
- Main concepts
- Intended audience
- Goal of this guide
- Software used
- You just need a capable enough computer
- The reference hardware setup
- Why this hardware setup?
- References
- A procedure to install Proxmox VE in limited consumer hardware
- System Requirements
- Installation procedure
- After the installation
- Connecting remotely
- References
- Proxmox VE 9.0 runs on Debian 13 "trixie"
- Editing the apt repository sources
- Update your system
- Installing useful extra tools
- Relevant system paths
- References
- Any server must be always connected to an UPS unit
- Connecting your UPS with your PVE node using NUT
- Checking the NUT logs
- Executing instant commands on your UPS unit
- Other possibilities with NUT
- Relevant system paths
- References
- Your Proxmox VE server's storage needs to be reorganized
- Initial filesystem configuration (PVE web console)
- Initial filesystem configuration (shell as root)
- Configuring the unused storage drives
- LVM rearrangement in the main storage drive
- References
- About the Proxmox subscription warning
- Removing the subscription warning
- Reverting the changes
- Change executed in just one command line
- Final note
- Relevant system paths
- References
- Enable Two Factor Authentication in your PVE system
- Enabling TFA for SSH access
- Enforcing TFA TOTP for accessing the Proxmox VE web console
- Enforcing TFA TOTP as a default requirement for the
pamrealm - Incompatibility of PVE web console login with TFA enforced local shell access
- Relevant system paths
- References
- Avoid using the root user
- Understanding the Proxmox VE user management and the realms
- Creating a new system administrator user for a Proxmox VE node
- Relevant system paths
- References
- Harden your SSH connections with key pairs
- Generating SSH key pairs
- Hardening the
sshdservice - Relevant system paths
- References
- Harden your setup against intrusions with Fail2Ban
- Installing Fail2Ban
- Configuring Fail2Ban
- Considerations regarding Fail2Ban
- Relevant system paths
- References
- Reduce your Proxmox VE server's exposed surface
- Checking currently running services
- Configuring the
pveproxyservice - Disabling RPC services
- Disabling
zfsandceph - Disabling the SPICE proxy
- Disabling cluster and high availability related services
- Considerations
- Relevant system paths
- References
- Harden your PVE's networking with a
sysctlconfiguration - About
sysctl - TCP/IP stack hardening with
sysctl - Relevant system paths
- References
- CPUs also have security vulnerabilities
- Discovering your CPU's vulnerabilities
- Your Proxmox VE system already has the correct microcode package applied
- Relevant system paths
- References
- [Enabling your PVE's firewall is a must]
- Proxmox VE firewall uses iptables
- Zones in the Proxmox VE firewall
- Situation at this point
- Enabling the firewall at the Datacenter tier
- Firewalling with ebtables
- Firewall fine tuning
- Firewall logging
- Connection tracking tool
- Relevant system paths
- References
- Tune your Proxmox VE system's
sysctlfiles to improve performance - First go the
sysctldirectory - Network optimizations
- Memory optimizations
- Kernel optimizations
- Reboot the system
- Final considerations
- Relevant system paths
- References
- Understanding the transparent hugepages
- Status of the transparent hugepages in your host
- Disabling the transparent hugepages
- Relevant system paths
- References
- Preparing your virtual network for Kubernetes
- Current virtual network setup
- Target network scenario
- Creating an isolated Linux bridge
- Bridges management
- Relevant system paths
- References
- Gearing up for your K3s cluster
- Requirements for the K3s cluster and the services to deploy in it
- Arrangement of VMs and services
- References
- Identifying your storage needs and current setup
- Storage organization model
- Creating the logical volumes (LVs)
- Enabling the LVs for Proxmox VE
- Configuration file
- Relevant system paths
- References
- You can start creating VMs in your Proxmox VE server
- Preparing the Debian ISO image
- Building a Debian virtual machine
- Note about the VM's
Boot Orderoption - Relevant system paths
- References
- You have to configure your new Debian VM
- Suggestion about the IP organization within your LAN
- Adding the
aptsources for non-free packages - Installing extra packages
- The QEMU guest agent comes enabled in Debian
- Hardening the VM's access
- Hardening the
sshdservice - Configuring Fail2Ban for SSH connections
- Disabling the
rootuser login - Configuring the VM with
sysctl - Reboot the VM
- Disabling transparent hugepages on the VM
- Regarding the microcode
aptpackages for CPU vulnerabilities - Relevant system paths
- References
- Make your VMs aware of your UPS unit with NUT
- Reconfiguring the NUT server on your Proxmox VE host
- Configuring the NUT client on your Debian VM
- Checking the connection between the VM NUT client and the PVE node NUT server
- Testing a Forced ShutDown sequence (
FSD) with NUT - Relevant system paths
- References
- Turn your Debian VM into a VM template
- Steps for transforming your Debian VM into a VM template
- VM template's backup
- Other considerations regarding VM templates
- References
- You need a more specialized VM template for building K3s nodes
- Reasons for a new VM template
- Creating a new VM based on the Debian VM template
- Setting an static IP for the main network device (
net0) - Setting a proper hostname string
- Disabling the swap volume
- Changing the VG's name
- Setting up the second network card
- Setting up sysctl kernel parameters for K3s nodes
- Turning the VM into a VM template
- Relevant system paths
- References
- Build your virtualized K3s cluster
- Criteria for the VMs' IPs and hostnames
- Creation of VMs based on the K3s node VM template
- Preparing the VMs for K3s
- Firewall setup for the K3s cluster
- Considerations before installing the K3s software
- K3s Server node setup
- K3s Agent nodes setup
- Enabling bash autocompletion for
kubectl - Regular K3s logs are journaled
- Rotating the
containerd.logfile - K3s relevant paths
- Starting up and shutting down the K3s cluster nodes
- Relevant system paths
- References
- Never handle your Kubernetes cluster directly from the server nodes
- Description of the
kubectlclient system - Getting the right version of
kubectl - Installing
kubectlon your client system - Getting the configuration for accessing the K3s cluster
- Opening the
6443port in the K3s server node - Enabling bash autocompletion for
kubectl - Validate Kubernetes configuration files with
kubeconform - Relevant system paths
- References
- Considerations before deploying MetalLB
- Choosing the IP ranges for MetalLB
- Deploying MetalLB on your K3s cluster
- MetalLB's Kustomize project attached to this guide
- Relevant system paths
- References
- Deploy a metric-server service that you can fully configure
- Checking the metrics-server's manifest
- Deployment of metrics-server
- Checking the metrics-server service
- Metrics-server's Kustomize project attached to this guide
- Relevant system paths
- References
- Use cert-manager to handle certificates in your cluster
- Deploying cert-manager
- Setting up self-signed CAs for your cluster
- Deploying the self-signed CAs
- Checking your certificates with the cert-manager command line tool
- Cert-manager's Kustomize project attached to this guide
- Relevant system paths
- References
- Traefik is the embedded ingress controller of K3s
- Steps to enable access to the Traefik dashboard
- Kustomize project's folder structure
- Traefik dashboard user
- Traefik dashboard Middleware
- Traefik dashboard IngressRoute
- Traefik dashboard Kustomize project
- Deploying the Traefik dashboard Kustomize project
- Getting into the Traefik dashboard
- What to do if Traefik's dashboard has bad performance
- Traefik dashboard's Kustomize project attached to this guide
- Relevant system paths
- References
- Headlamp is an alternative to the Kubernetes Dashboard
- Components required for deploying Headlamp
- Kustomize project's folder structure
- Headlamp ServiceAccount
- Headlamp ClusterRoleBinding
- Headlamp TLS certificate
- Headlamp IngressRoute
- Kustomize manifest for the Headlamp project
- Deploying Headlamp
- Getting the administrator user's service account token
- Testing Headlamp
- Headlamp's Kustomize project attached to this guide
- Relevant system paths
- References
- Upcoming chapters are about deploying services in your K3s cluster
- Be watchful of your system's resources usage
- Do not fill your cluster up to the brim
- Beginning with Ghost
- Outlining Ghost's setup
- Choosing the K3s agent node for running Ghost
- Setting up new storage drives in the K3s agent node
- Relevant system paths
- References
- You can use Valkey instead of Redis as caching server for Ghost
- Kustomize project folders for Ghost and Valkey
- Valkey configuration file
- Valkey secrets
- Valkey persistent storage claim
- Valkey StatefulSet
- Valkey Service
- Valkey Kustomize project
- Do not deploy this Valkey project on its own
- Relevant system paths
- References
- You can use MariaDB instead of MySQL as database server for Ghost
- MariaDB Kustomize subproject's folders
- MariaDB configuration files
- MariaDB passwords
- MariaDB persistent storage claim
- MariaDB StatefulSet
- MariaDB Service
- MariaDB Kustomize project
- Do not deploy this MariaDB project on its own
- Relevant system paths
- References
- Deploy the Ghost server just like another component
- Considerations about the Ghost server
- Ghost server Kustomize subproject's folders
- Ghost server configuration file
- Ghost server environment variables
- Ghost server persistent storage claim
- Ghost server StatefulSet
- Ghost server Service
- Ghost server Kustomize project
- Do not deploy this Ghost server project on its own
- Relevant system paths
- References
- Putting together the whole Ghost platform
- Create a folder for the pending Ghost platform resources
- Ghost platform's persistent volumes
- Ghost platform's TLS certificate
- Traefik IngressRoute for enabling HTTPS access to the Ghost platform
- Ghost Namespace
- Main Kustomize project for the Ghost platform
- Deploying the main Kustomize project in the cluster
- Start using Ghost
- Security considerations in Ghost
- Ghost platform's Kustomize project attached to this guide
- Relevant system paths
- References
- Deploy Forgejo like you deployed Ghost
- Outlining Forgejo's setup
- Setting up new storage drives in the K3s agent
- Relevant system paths
- References
- Valkey can be the cache server of Forgejo
- Kustomize project folders for Forgejo and Valkey
- Valkey configuration file
- Valkey secrets
- Valkey persistent storage claim
- Valkey StatefulSet
- Valkey Service
- Valkey Kustomize project
- Do not deploy this Valkey project on its own
- Relevant system paths
- References
- Forgejo can use PostgreSQL as database
- PostgreSQL Kustomize project's folders
- PostgreSQL configuration files
- PostgreSQL passwords
- PostgreSQL persistent storage claim
- PostgreSQL StatefulSet
- PostgreSQL Service
- PostgreSQL Kustomize project
- Do not deploy this PostgreSQL project on its own
- Relevant system paths
- References
- Considerations about the Forgejo server
- Forgejo server Kustomize project's folders
- Forgejo server configuration with environment variables
- Forgejo server persistent storage claims
- Forgejo server StatefulSet
- Forgejo server Service
- Forgejo Kustomize project
- Do not deploy this Forgejo server project on its own
- Relevant system paths
- References
- Finishing up the complete Forgejo platform
- Create a folder for the missing Forgejo platform resources
- Forgejo platform's persistent volumes
- Forgejo platform's TLS certificate
- Traefik IngressRoute for enabling HTTPS access to the Forgejo platform
- Forgejo Namespace resource
- Main Kustomize project for the Forgejo platform
- Deploying the main Kustomize project in the cluster
- Finishing Forgejo installation
- Security considerations in Forgejo
- Forgejo platform's Kustomize project attached to this guide
- Relevant system paths
- References
- Improve your K3s cluster's observability with a Prometheus-based monitoring stack
- Outlining your monitoring stack setup
- Setting up new storage drives in the K3s agents
- Relevant system paths
- References
- Start by deploying the Kube State Metrics service
- Kustomize project folders for your monitoring stack and Kube State Metrics
- Kube State Metrics ServiceAccount
- Kube State Metrics ClusterRole
- Kube State Metrics ClusterRoleBinding
- Kube State Metrics Deployment
- Kube State Metrics Service
- Kube State Metrics Kustomize project
- Do not deploy this Kube State Metrics project on its own
- Relevant system paths
- References
- The Prometheus Node Exporter is simpler to deploy
- Kustomize project folders for Prometheus Node Exporter
- Prometheus Node Exporter DaemonSet
- Prometheus Node Exporter Service
- Prometheus Node Exporter Kustomize project
- Do not deploy this Prometheus Node Exporter project on its own
- Relevant system paths
- References
- Prometheus is the core of your monitoring stack
- Kustomize project folders for Prometheus server
- Prometheus configuration files
- Prometheus server persistent storage claim
- Prometheus server ServiceAccount
- Prometheus server ClusterRole
- Prometheus server ClusterRoleBinding
- Prometheus server StatefulSet
- Prometheus server Service
- Prometheus server's Kustomize project
- Do not deploy this Prometheus server project on its own
- Relevant system paths
- References
- Grafana is your monitoring dashboard
- Kustomize project folders for Grafana
- Grafana server persistent storage claim
- Grafana server StatefulSet
- Grafana server Service
- Grafana server Kustomize project
- Do not deploy this Grafana server project on its own
- Relevant system paths
- References
- Completing your monitoring stack
- Create a folder to hold the missing monitoring stack components
- Monitoring stack persistent volumes
- Monitoring stack TLS certificate
- Traefik IngressRoute for enabling HTTPS access to the monitoring stack's Prometheus and Grafana
- Monitoring stack Namespace
- Main Kustomize project for the monitoring stack
- Deploying the main Kustomize project in the cluster
- Checking on Prometheus
- Finishing Grafana's setup
- Monitoring stack Kustomize project attached to this guide
- Relevant system paths
- References
- Monitor your homelab setup with its own tools
- Monitoring resources usage
- Checking the logs
- Shell access into your containers
- Metrics from the monitoring stack
- References
- What to backup. Identifying your data concerns
- How to backup. Backup tools
- Where to store the backups. Backup storage
- When to do the backups. Backup scheduling
- References
- Host backups are filesystem images
- What gets inside a host backup
- Why doing this backup
- How it affects the host platform
- When to do the backup
- How to backup with Clonezilla
- How to restore with Clonezilla
- Final considerations
- References
- Backup your VMs in Proxmox VE
- What gets covered with the backup job
- Why scheduling a backup job
- How it affects the K3s Kubernetes cluster
- When to do the backup job
- Scheduling the backup job in Proxmox VE
- Restoring a backup in Proxmox VE
- Location of the backup files in the Proxmox VE system
- Relevant system paths
- References
- Use UrBackup to preserve specific directories
- Setting up a new VM for the UrBackup server
- Deploying UrBackup
- Firewall configuration on Proxmox VE
- Adjusting the UrBackup server configuration
- UrBackup server log file
- About backing up the UrBackup server VM
- Relevant system paths
- References
- Install UrBackup clients in your K3s node VMs
- Deploying the UrBackup client program
- UrBackup client log file
- UrBackup client uninstaller
- Configuring file backup paths on a client
- Backups on the UrBackup server
- Restoration from file backups
- Relevant system paths
- References
- Updating this guide's homelab setup is not a straightforward task
- What to update. Identifying your system's software layers
- How to update. Update procedures
- When to apply the updates
- Update order
- Updating the VMs means updating their OS
- Examining your VMs
- Updating Debian on your VMs
- Updating the UrBackup software
- References
- Use
kubectlto help you when updating your K3s cluster - Examining your K3s Kubernetes cluster
- Updating apps and K3s
- References
- Save storage space by cleaning your system up
- Checking your storage status
- Cleaning procedures
- Reminder about the
aptupdates - Relevant system paths
- References
- These instructions could be valid for other remote terminal clients
- Generating a
.ppkfile from a private key - Configuring the connection to the PVE node
- References
- VM hard disks are disk images
- Installing the
libguestfs-toolspackage - Locating and checking a VM or VM template's hard disk volume
- Relevant system paths
- References
- Resize a VM's root LVM volume if you find it too small
- Resizing the storage drive on Proxmox VE
- Extending the root LVM filesystem on a live VM
- Final note
- References
- Real Kubernetes clusters have more than one server node
- Add a new VM to act as the second server node
- Adapt the Proxmox VE firewall setup
- Setup of the FIRST K3s server node
- Setup of the SECOND K3s server node
- Regarding the K3s agent nodes
- References
- Review the status of your K3s cluster's Kubernetes API endpoints with
kubectl - Check out your K8s cluster API endpoints readiness
- See if your K8s cluster's API endpoints are healthy
- Verify if the K8s cluster's API endpoints are live
- Details to notice from the
kubectlcommands - Deprecated component status command
- References
- Upgrading a containerized MariaDB to a new major version is easy
- Concerns
- Enabling the update procedure
- References