Kube360 Cluster

Introduction

This document provides details on the Kube360 Kubernetes cluster, which is built on bare metal servers from OVHcloud. It covers the cluster's architecture, access procedures, and associated resources.

Rationale

We chose to build our own bare metal cluster for a few key reasons:

  • Performance: To leverage the high performance of bare metal servers.
  • Cost: To achieve better price-to-performance ratio compared to managed cloud providers.
  • Future Goals: To build experience with offering our own infrastructure for future client services.

Please note that this cluster is currently experimental.

Architecture

The cluster is built on the k3s Kubernetes distribution and uses Traefik as its ingress controller. For a detailed overview of the setup, please refer to this blog post.

Nodes

The cluster is currently comprised of 5 nodes:

❯ kubectl get nodes
NAME           STATUS   ROLES                       AGE    VERSION
ns5019222      Ready    control-plane,etcd,master   29d    v1.33.6+k3s1
ns577110       Ready    control-plane,etcd,master   29d    v1.33.6+k3s1
postgres-one   Ready    <none>                      3d     v1.33.6+k3s1
postgres-two   Ready    <none>                      2d5h   v1.33.6+k3s1
replica-1      Ready    control-plane,etcd,master   29d    v1.33.6+k3s1

The two nodes prefixed with postgres- are dedicated to PostgreSQL workloads. They are configured with appropriate node affinity and tolerations to ensure that no other pods are scheduled on them.

Accessing the Cluster

There are two levels of access to the cluster: Users and Admins.

Users

Users are individuals who deploy applications to the cluster. The primary tool for users is ArgoCD:

If you need access to ArgoCD, please contact Sibi.

Currently, all non-admin users have read-only permissions on the cluster.

Admins

Admins are responsible for managing the cluster, including upgrades and software installation. In addition to ArgoCD, admins have access to the Kube360 dashboard:

Note: Access to the Kube360 dashboard and the k3-cli is currently unstable. Manual intervention may be required. Improving the initial onboarding experience is a priority.

The current admins are:

  • Sibi
  • Michael Snoyman
  • Emanuel Borsboom

Code Repositories

The following repositories are associated with this project:

  • fpco/ovhcloud-terraform
    • Contains Terraform code for OVHcloud, Cloudflare, and Azure.
    • Includes various manifests deployed to the Kubernetes cluster.
  • fpco/k3dash-hs
    • The admin dashboard for the cluster.
  • fpco/k3-cli
    • A command-line interface for accessing the cluster.