Kube360 Cluster Upgrade

This page documents the steps required to perform Kubernetes (K8s) control plane and data plane node upgrades.

Important Notes:

  • Upgrade Order: The control plane must be upgraded before the data plane nodes.
  • Data Plane Considerations: Data plane nodes host Postgres (CloudNativePG) databases, so special care must be taken before upgrading them to ensure data integrity and availability.

Version Skew Policy

Before upgrading, review the official Kubernetes version skew policy regarding the maximum version skew supported between K8s components.

TL;DR: Versions are represented as MAJOR.MINOR.PATCH. Ensure that when upgrading, your components remain within one minor version of each other.

Control Plane Upgrade Steps

Currently, upgrades are performed using the config-tool from within a local Docker container.

  1. Enter the local running Docker container:

    just exec

  2. Export your Amber secret environment variable:

    export AMBER_SECRET="REDACTED"

  3. Perform the upgrade using the k3s-config-tool, specifying the node name and target Kubernetes version:

    amber exec -- /usr/bin/k3s-config-tool upgrade-node \
  --node-name control-plane-replica-one \
  --kubernetes-version v1.34.5+k3s1

Data Plane Upgrade Steps

Upgrading data plane nodes requires additional steps to safely handle PostgreSQL clusters managed by CloudNativePG. This process needs to be repeated for each data plane node.

1. Prepare PostgreSQL Clusters

For CloudNativePG, you must modify the Postgres cluster resources to disable Pod Disruption Budgets (PDB) and place them in node maintenance mode.

Update the cluster specification for all your application databases, as well as Kube360's internal k3dash database:

enablePDB: false

2. Drain the Node

Before upgrading, safely evict all workloads from the node:

# Syntax
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Example
kubectl drain postgres-two --ignore-daemonsets --delete-emptydir-data

3. Perform the Upgrade

Repeat the same upgrade command used for the control plane, but target the data plane node:

amber exec -- /usr/bin/k3s-config-tool upgrade-node \
  --node-name <node-name> \
  --kubernetes-version v1.34.5+k3s1

4. Uncordon the Node

Once the upgrade is complete and the node is ready, mark it as schedulable again:

# Syntax
kubectl uncordon <node-name>

# Example
kubectl uncordon postgres-two

5. Restore PostgreSQL Clusters

Finally, undo the enablePDB: false changes made in Step 1 to re-enable Pod Disruption Budgets for all PostgreSQL clusters.

References