Clustering

Introduction

This guide explains how to effectively use a multi-node Kubernetes cluster to achieve high availability with the ESB3027 AgileTV CDN Manager. The manager runs on a self-hosted K3s cluster, offering a lightweight yet scalable platform for deploying essential services with high availability. The cluster consists of one or more nodes which may be geographically distributed between multiple datacenters for additional redundancy.

The Kubernetes cluster consists of a series of nodes with either the role “Server” or “Agent”. Server nodes comprise the Kubernetes control-plane, and provide the necessary services and embedded etcd datastore to manage the state of the cluster. Agent nodes are responsible solely for running workloads. When deploying the Manager service for High Availability, a minimum of three server nodes must be deployed, and the total number of server nodes must always be odd. Additional cluster capacity can be achieved by adding zero or more agent nodes to the cluster. It is highly recommended that server nodes be geographically distributed across different datacenters if multiple datacenters are available. By default, workloads will be assigned to any node in the cluster with available capacity, however it is possible to apply a taint to one or more server nodes to prevent non-critical workloads from being assigned there, effectively creating an arbiter node. When installing the Manager service, the first node deployed will adapt the server role. Additional server and agent nodes can be added at any time.

Persistent storage volumes are provided by Longhorn, a distributed storage driver which ensures persistent, reliable storage, across the cluster. Longhorn creates a cluster of replicated storage volumes, accessible from any physical node, while persisting data in local storage on the node. This decouples the workload from the storage, allowing the workloads to run on any node in the cluster.

Before considering adding additional nodes to the cluster, ensure that any required ports have been opened according to the documentation in the Networking Guide.

Expanding the Cluster

Before expanding the cluster, it is required that the user be familiar with the standard installation procedure as described in the Installation Guide. At a minimum, on the primary node, the install command must have been performed to initialize the cluster.

Before continuing, a K3s token must be obtained from any server node in the cluster. This token is used by the K3s installer to authenticate additional nodes. The token can be found on any installed server node at /var/lib/rancher/k3s/server/node-token.

For each additional node, mount the ESB3027 AgileTV CDN Manager ISO, and execute either the join-server or join-agent command from the root of the ISO. It is critical that each node has a unique hostname. If that is not the case, set a unique name for each node in the K3S_NODE_NAME environment variable before running this command.

Both commands take two arguments, the first being the URL to any one of the server nodes, this is of the form https://node:6443, and the second is the K3s token obtained earlier.

To add an additional “Server” node use the following:

/mnt/acd-manager/join-server https://k3s-server:6443 abcdef0123456...987654321

To add an additional “Agent” node use the following:

/mnt/acd-manager/join-agent https://k3s-server:6443 abcdef0123456...987654321

After the command completes, the additional node should appear in the node list as Ready. From any server node, execute the following:

kubectl get nodes

Configuring Longhorn

The default configuration for Longhorn makes several assumptions about how the persistent volume configuration should be managed. It is recommended to update the Longhorn configuration to suit the environment prior to deploying the Manager’s helm charts, since some settings can not be changed once data exists on a volume.

Longhorn provides a frontend UI which is not exposed by default. This is an intentional security precaution. In order to access the frontend UI, a Kubernetes port forward must be used. The command for setting up the port forward is listed below:

kubectl port-forward -n longhorn-system --address 0.0.0.0 svc/longhorn-frontend 8888:80

This will forward traffic from any node in the cluster on port 8888 to the Longhorn frontend UI service. Open a browser to http://k3s-server:8888 to visit the UI and adjust settings as necessary. After finishing with the Longhorn UI, pressing Ctrl+C on the port forward command will close the tunnel.

Some settings which should be considered include setting up “Default Data Locality”, the “Default Data Path”, the “Default Replica Count”, and the Backup settings.