AgileTV CDN Manager (esb3027)

• 1: Getting Started
• 2: System Requirements Guide
• 3: Networking Guide
• 3.1: Shared Interface Setup
• 3.2: Configuring Segregated Networks
• 4: Architecture Guide
• 5: Installation Guide
• 5.1: Quickstart
• 5.2: Inventory
• 5.3: Site Settings
• 5.4: TLS Certificates
• 5.5: MaxMind GeoIP Databases
• 5.6: Application Settings
• 5.7: Running deploy
• 5.8: Changing the Cluster
• 5.9: Air-Gapped Installation
• 5.10: Upgrade Guide
• 5.11: Installation Troubleshooting
• 5.12: Next Steps
• 6: Configuration Guide
• 7: Performance Tuning Guide
• 8: Operations Guide
• 9: Metrics & Monitoring Guide
• 9.1: Grafana Authentication & Roles
• 9.2: Grafana Dashboards
• 9.3: Alerts & Alarms
• 10: API Guide
• 10.1: Authentication API
• 10.2: Health API
• 10.3: Selection Input API
• 10.4: Data Store API
• 10.5: Subnets API
• 10.6: Routing API
• 10.7: Discovery API
• 10.8: Metrics API
• 10.9: Configuration API
• 10.10: Operator UI API
• 10.11: OpenAPI Specification
• 11: Troubleshooting Guide
• 12: Command Line Reference
• 12.1: deploy
• 12.2: acd-admin
• 12.3: acd-support-dump
• 13: Glossary

1 - Getting Started

Overview

The AgileTV CDN Manager (product code ESB3027) is a cloud-native control plane for managing CDN deployments. It provides centralized orchestration for authentication, configuration, routing, and metrics collection across CDN infrastructure.

Before You Start:

• Deployment type: Lab (single-node) or Production (multi-node)? See Installation Guide
• Hardware: Nodes meeting specifications for your deployment type
• OS: RHEL 9.4 or later, or a compatible clone (Oracle Linux, AlmaLinux, Rocky Linux)
• Software: Installation ISO from AgileTV customer portal
• Network: A default route on every node and, optional but recommended, a backplane network for node-to-node traffic, separate from the public network. See the Networking Guide

Deployment Models

Functionality remains consistent across deployment models.

Prerequisites

• Installation ISO: Obtain esb3027-cdn-manager-X.Y.Z.iso from AgileTV customer portal
• OS: RHEL 9.4 or later, or a compatible clone (Oracle Linux, AlmaLinux, Rocky Linux)
• Kubernetes familiarity: Basic understanding of pods and deployments helps with troubleshooting

For detailed hardware, network, and operating system requirements, see the System Requirements Guide.

Installation

Ready to install? The Quickstart takes three freshly installed machines to a working cluster, step by step, with the deploy command on the installation ISO. The Installation Guide covers the details and variations, including single-node labs.

Accessing the System

Following successful deployment, the following interfaces are available:

All services are accessed via https://<cluster-host><path>.

Important: <cluster-host> must be the main manager hostname, the first entry of manager_hostnames in the site settings. Pages may load under another name, but signing in only works using the configured hostname. To reach the system by IP address, configure that IP address at installation time.

Note: Without a TLS certificate of your own, a temporary self-signed certificate is served, and browsers show a warning. For production deployments, install a certificate from a trusted certificate authority before exposing the system to users. See TLS Certificates.

Initial user configuration is performed through Zitadel. See Next Steps for securing the administrator account and adding users. For detailed guidance on managing users, roles, and permissions in the Zitadel Console, see Zitadel’s User Management Documentation.

Documentation Navigation

The following guides provide detailed information for specific operational tasks:

2 - System Requirements Guide

Overview

This document specifies the hardware, operating system, and networking requirements for deploying the AgileTV CDN Manager (ESB3027). Requirements vary based on deployment type and node role within the cluster.

Cluster Sizing

Production Deployments

Production deployments require a minimum of three nodes to achieve high availability. The cluster architecture employs distinct node roles:

Server nodes can be deployed in either Control Plane Only or Combined role configurations. The choice depends on your deployment requirements:

• Control Plane Only: Dedicated control plane nodes with lower resource requirements; requires separate Agent nodes for workloads
• Combined: Server nodes run both control plane and workloads; minimum 3 nodes required for HA

Why Use Control Plane Only Nodes?

Dedicated Control Plane Only nodes provide several benefits for larger deployments:

• Resource Isolation: Control plane components (etcd, API server, scheduler) run on dedicated hardware without competing with application workloads for CPU and memory
• Stability: Application workload spikes or misbehaving pods cannot impact control plane performance
• Security: Smaller attack surface on control plane nodes; fewer containers and services running
• Predictable Performance: Control plane responsiveness remains consistent regardless of application load
• Flexible Sizing: Control Plane Only nodes can use lower-specification hardware (2 cores, 4 GiB) since they don’t run application workloads

For most small to medium deployments, Combined role servers are simpler and more cost-effective. Control Plane Only nodes are recommended for larger deployments with significant workload requirements or where control plane stability is critical.

High Availability Considerations

Production deployments require 3 nodes running control plane (etcd) and 3 nodes capable of running workloads. These can be the same nodes (Combined role) or separate nodes (CP-Only + Agent).

Node Role Combinations:

Any combination works as long as you have 3 control plane nodes and 3 workload-capable nodes.

Note: Regardless of the deployment configuration, a minimum of 3 nodes capable of running workloads is required for production deployments. This ensures both high availability and sufficient capacity for application pods.

For detailed fault tolerance information and data replication strategies, see the Architecture Guide.

Hardware Requirements

Single-Node Lab Deployment

Lab deployments are intended for acceptance testing, demonstrations, and development only. These configurations are not suitable for production workloads.

Production Cluster - Server Node (Control Plane Only)

Server nodes dedicated to control plane functions have modest resource requirements:

These nodes run only control plane components and require separate Agent nodes to run application workloads.

Production Cluster - Server Node (Control Plane + Workloads)

Combined role nodes require resources for both control plane and application workloads:

Production Cluster - Agent Node

Agent nodes execute application workloads and require the following resources:

Swap

Swap may stay on: deploy leaves it as it is, and the cluster runs with or without it. Size memory so that swap is not used in normal operation. Swap in use is a sign of memory pressure.

Storage Notes

* Disk Space: All disk space values must be available in the /var/lib/longhorn partition. It is recommended that /var/lib/longhorn be a separate partition on a fast SSD for optimal performance, though SSD is not strictly required.

Longhorn Capacity: Longhorn storage requires an additional 30% capacity headroom for internal operations and scaling. If less than 30% of the total partition capacity is available, Longhorn may mark volumes as “full” and prevent further writes. Plan disk capacity accordingly.

Storage Performance

For optimal performance, the following storage characteristics are recommended:

• Disk Type: SSD or NVMe storage for Longhorn volumes
• Filesystem: XFS or ext4 with default mount options
• Partition Layout: Dedicated /var/lib/longhorn partition for persistent storage

Virtual machines and bare-metal hardware are both supported. Nested virtualization (running multiple nodes under a single hypervisor) may impact performance and is not recommended for production deployments.

Operating System Requirements

Supported Operating Systems

The CDN Manager supports Red Hat Enterprise Linux and compatible distributions:

Compatible Clones

The following RHEL-compatible distributions are supported at release 9.4 or later:

• Oracle Linux 9
• AlmaLinux 9
• Rocky Linux 9

A minimal installation is enough. Keep firewalld and NetworkManager installed and running: deploy configures the firewall and network through them.

Air-Gapped Deployments

Important: deploy installs a few packages from the operating system’s repositories on every node. For air-gapped deployments (no internet access), make the OS installation media available to dnf on all nodes first. See Air-Gapped Installation.

Oracle Linux UEK Kernel

On Oracle Linux 9 running Unbreakable Enterprise Kernel 8 (UEK 8), install the netfilter kernel modules that K3s and Longhorn need on every node before running the installer:

dnf install kernel-uek-modules-extra-netfilter-$(uname -r)

The package comes from Oracle’s UEK 8 repository, ol9_UEKR8. If dnf cannot find it, check that the repository is enabled, and enable it if not:

dnf repolist
dnf config-manager --enable ol9_UEKR8

Without internet access, that repository is not available. Make the package available to dnf from a local repository first, as for the other packages in Air-Gapped Installation, and check that dnf finds it:

dnf -q repoquery kernel-uek-modules-extra-netfilter-$(uname -r)

SELinux

SELinux is supported in any mode, “Enforcing” included. The installation process installs the SELinux policy for the cluster automatically.

Networking Requirements

Network Interface

Every cluster node needs an IPv4 default route on its public interface, the interface that clients reach it on. The route does not need to lead to the internet: on an isolated network, use your site’s router or any gateway address on that subnet. deploy uses this route to find each machine’s public interface and address, and stops with [ACD-NO-DEFAULT-ROUTE] without one.

A second interface on a backplane network, separate from the public network, is optional but recommended for production. All node-to-node traffic then uses the backplane, and every node must be reachable on it. Other hosts may share it. See Configuring Segregated Networks.

Port Requirements

The cluster requires the following network connectivity:

By default, deploy configures the host firewall for the inter-node and external ports itself. The host firewall does not filter the Kafka ports: restrict them to your CDN Directors on your network’s own firewall. See Kafka Ports. The same applies to the two ingress NodePorts, chosen at installation: see Ingress NodePorts.

Important: Complete port requirements, network ranges, and firewall configuration procedures are provided in the Networking Guide. Do not expose VictoriaMetrics (8428, 8429), Grafana (3000), or PostgreSQL (5432) directly—access these services only through the secure HTTPS ingress (port 443).

Resource Planning

Calculating Cluster Capacity

When planning cluster capacity, consider the following factors:

1. Base Overhead: Kubernetes system components consume approximately 1-2 cores and 2-4 GiB memory per node
2. Application Workloads: Refer to individual component resource requirements in the Architecture Guide
3. Headroom: Maintain 20-30% resource headroom for workload spikes and automatic scaling

Scaling Considerations

The CDN Manager supports horizontal scaling for most components. The Horizontal Pod Autoscaler (HPA) can automatically adjust replica counts based on resource utilization. Detailed scaling guidance is available in the Architecture Guide.

Example Production Deployment

A minimal production deployment with 3 server nodes (combined role) and 2 agent nodes would require:

This configuration provides:

• High availability (survives loss of 1 server node)
• Capacity for application workloads across all nodes
• Headroom for horizontal scaling

Next Steps

After verifying system requirements:

1. Review the Installation Guide for deployment procedures
2. Consult the Networking Guide for firewall configuration
3. Examine the Architecture Guide for component resource requirements

3 - Networking Guide

Overview

deploy configures each machine’s network for the cluster: its firewall, its routing rules for cluster traffic, and the kernel network settings. The site setting backplane_network chooses the layout. A second setting, manage_firewall, chooses whether deploy manages the firewall or you do.

A backplane is optional, but we recommend one wherever the machines can have a second network. The backplane is the network the cluster machines use to reach each other (east-west traffic). Every machine must be reachable on it, and it should be a separate network from the public one. Setting it to the public network’s range works, but then every host on the public network is trusted for all traffic. It does not have to be reserved for the cluster: other hosts may also be on it, but with the firewall managed they can reach every port on the machines, including the cluster datastore (etcd, 2379-2380/tcp), the Kubernetes API (6443/tcp) and the kubelet (10250/tcp).

Interfaces and Addresses

Every machine needs an IPv4 default route on its public interface, the interface that clients reach it on. The route does not need to lead to the internet: on an isolated network, use your site’s router or any gateway address on that subnet. deploy uses this route to find each machine’s public interface and address, and stops with [ACD-NO-DEFAULT-ROUTE] without one.

From that route, deploy works out each machine’s two addresses:

• The public interface is the one holding the default route. Its address is the machine’s public address, which clients reach.
• The cluster address is the address other machines reach it on: its address inside backplane_network when that is set, and otherwise its public address.

Interfaces that are neither the public interface nor the backplane are left as they are.

The cluster also uses two internal address ranges, which must not be in use anywhere on your network:

Firewall

With manage_firewall: true, the default, deploy starts and enables firewalld on every machine and keeps two things as its own:

• The acd-cluster zone, with target ACCEPT, which trusts all traffic from the cluster: the backplane interface (or, without a backplane, the other machines’ addresses), and the pod and service networks.
• The acd-public policy, from the public zone to the machine, which opens the ports clients need:

  Port	Purpose
  80/tcp	HTTP ingress
  443/tcp	HTTPS ingress: the GUI, the API, Zitadel and Grafana
  8086/tcp	Metrics pushed by the CDN Directors (Vector)

It also places each machine’s public interface in the public zone, on its NetworkManager connection. Before the first run, make sure the public zone allows everything else you need on that interface, SSH included. To check, run firewall-cmd --zone=public --list-all on each machine: ssh should be listed under services, or port 22 under ports.

deploy removes anything in acd-cluster and acd-public that it did not put there. Add your own rules to other zones, such as public: deploy never changes their rules.

Ports Between the Machines

Within the cluster, the machines trust each other for all traffic. When you manage the firewall yourself, the ports the cluster needs between machines are:

The pod and service networks must also be in a zone with target ACCEPT.

Kafka Ports

The CDN Directors reach Kafka on these ports, on every machine:

With the default three brokers, the broker ports are 30001-30003, so the full range is 30000-30003. With more brokers, the range grows by one port per broker: six brokers use 30000-30006. In the rest of this documentation, “30000-30003” means this range for the default three brokers. Widen it if your cluster runs more.

deploy neither opens nor checks these ports, and the host firewall does not filter them: as on any host running containers, traffic that Kubernetes forwards to its services passes before the firewall’s zone rules apply. These ports are reachable from every network that can reach the machines.

Important: Restrict 9092/tcp and the whole 30000 range (30000-30003 with three brokers) to your CDN Directors on your network’s own firewall. The same applies to any other port a Kubernetes service publishes on the machines.

Ingress NodePorts

The ingress service (Traefik) also publishes two NodePorts on every machine, which forward to its HTTP and HTTPS ports, 80 and 443. Kubernetes picks them at installation from 30000-32767, so they differ from one cluster to another, and the host firewall does not filter them, as for the Kafka ports. Find them on a server:

kubectl get service traefik -n kube-system

NAME      TYPE           CLUSTER-IP    EXTERNAL-IP                          PORT(S)                      AGE
traefik   LoadBalancer   10.43.12.34   192.0.2.11,192.0.2.12,192.0.2.13     80:31873/TCP,443:30306/TCP   3d

Here they are 31873 (HTTP) and 30306 (HTTPS). Clients need only 80 and 443: block the two NodePorts on your network’s own firewall.

Managing the Firewall Yourself

Set manage_firewall: false to configure firewalld yourself. deploy then only checks that the ports between the machines and the ports clients need are open in the zone their traffic arrives in, and that the pod and service networks are in a zone with target ACCEPT. It stops before changing anything, and shows the firewall-cmd commands to run, if they are not. Only firewalld is checked: if it is not running, the check is skipped, and your own firewall must allow the same traffic.

To hand the firewall of a running cluster over to yourself, on each machine:

1. Move the pod and service networks to a zone with target ACCEPT, such as firewalld’s own trusted. For the default ranges:

   for network in 10.42.0.0/16 10.43.0.0/16; do
       firewall-cmd --permanent --zone=acd-cluster --remove-source="$network"
       firewall-cmd --permanent --zone=trusted --add-source="$network"
       firewall-cmd --zone=trusted --change-source="$network"
   done
   

2. Open the ports in the zone their traffic arrives in, with firewall-cmd --zone=<zone> --add-port=<port>, once with --permanent and once without.

3. On a machine with a backplane, put the backplane interface in the zone you want: nmcli connection modify <its connection> connection.zone <zone>.

4. Set manage_firewall: false in group_vars/all.yaml and run deploy.

deploy then empties the acd-cluster zone and the acd-public policy, and moves any interface still in acd-cluster to firewalld’s default zone, with a warning.

Routing

With manage_routes: true, the default, deploy adds lookup main routing rules at priority 50, so that cluster traffic always uses the main routing table, even on machines that use policy routing. On a machine with a backplane, ip rule then shows:

0:	from all lookup local
50:	from 10.42.0.0/16 lookup main proto static
50:	from all to 10.42.0.0/16 lookup main proto static
50:	from 10.43.0.0/16 lookup main proto static
50:	from all to 10.43.0.0/16 lookup main proto static
50:	from all to 10.10.0.0/24 lookup main proto static
32766:	from all lookup main
32767:	from all lookup default

Keep your own rules at other priorities. See manage_routes for how these rules interact with your own.

Managing Routing Yourself

If your own policy routing rules must sit at priority 50 or below, set manage_routes: false and add the cluster’s rules yourself. Add them before you change the setting on a running cluster: deploy then removes every priority-50 rule from the connections it manages.

Every machine needs five lookup main rules:

Give them a priority other than 50, and a lower number than any of your own rules that send traffic to another table, such as your from <address> lookup <table> source rules. Store them on the NetworkManager connection of the interface the machines reach each other on: the backplane interface, or the public interface without a backplane. For example, at priority 40 on a backplane connection named eth1 (table 254 is main):

nmcli connection modify eth1 +ipv4.routing-rules "priority 40 from 10.42.0.0/16 table 254"
nmcli connection modify eth1 +ipv4.routing-rules "priority 40 to 10.42.0.0/16 table 254"
nmcli connection modify eth1 +ipv4.routing-rules "priority 40 from 10.43.0.0/16 table 254"
nmcli connection modify eth1 +ipv4.routing-rules "priority 40 to 10.43.0.0/16 table 254"
nmcli connection modify eth1 +ipv4.routing-rules "priority 40 to 10.10.0.0/24 table 254"

Stored rules take effect when the connection is next activated. To activate it now, run nmcli connection up eth1 in a maintenance window: it re-applies the whole connection, and briefly interrupts traffic on that interface. Then check the rules with ip rule show.

Whichever way the rules are managed, deploy checks before changing anything that cluster traffic resolves through the main routing table, and stops with [ACD-ROUTING-CONFLICT], naming the rule in the way, if it does not.

Changes Are Checked

After changing a machine’s network, deploy compares the live state with what it planned: the interface zones, the firewall, the routing rules and the kernel settings. It stops before installing anything on the cluster if they differ. This check cannot be skipped. See the [ACD-VERIFY-...] codes in Installation Troubleshooting.

Boot-Time Network Tools

deploy stores its interface zones and routing rules on the machines’ NetworkManager connections. Tools that rewrite those connections at boot, such as cloud-init or a virtualization platform’s contextualization, can undo them after a reboot: an interface back in the wrong zone, or the priority-50 rules gone. Nothing checks for this between deploy runs.

Disable such regeneration of the network connections on the cluster machines. Where you cannot, run deploy again after a reboot: it puts the settings back, prints Host network changed on <machine>: ... naming what it changed, and carries on.

3.1 - Shared Interface Setup

Overview

In a shared layout, each machine has a single network: the cluster’s own traffic and client traffic both use the interface holding the default route. This is the simplest layout, for machines with one network interface.

Security note: Cluster traffic shares the public network, so it is not isolated from clients. For production, a segregated backplane is recommended.

Configuration

Leave backplane_network empty in /etc/acd/deploy/group_vars/all.yaml. This is the default:

backplane_network: ""

Each machine’s cluster address is then its public address. The machines may be on one subnet or on routed subnets, as long as each can reach the others’ addresses.

What deploy Sets Up

On each machine, with manage_firewall: true:

• The public interface is placed in firewalld’s public zone, with the ports clients need opened through the acd-public policy.
• The other machines’ addresses, and the pod and service networks, are trusted in the acd-cluster zone, which accepts all traffic. Hosts other than the cluster machines get only the ports open in public.
• Routing rules keep traffic to the pod and service networks, and to the machine’s own subnet, on the main routing table. If your machines are on different subnets and use policy routing, make sure traffic to the other machines’ subnets also uses the main table.

Verification

After deploy, on any machine:

firewall-cmd --get-active-zones
firewall-cmd --zone=acd-cluster --list-sources

The acd-cluster zone lists the pod and service networks (10.42.0.0/16 and 10.43.0.0/16 by default) and the other cluster machines’ addresses, and the public interface is in the public zone.

For the ports the cluster uses, and for managing the firewall yourself, see the Networking Guide.

3.2 - Configuring Segregated Networks

Overview

In a segregated layout, each machine has two networks:

• A public network, holding the default route, which clients and CDN Directors use to reach the manager.
• A backplane network, separate from the public network, which carries all traffic between the machines (east-west traffic): the Kubernetes API, the cluster datastore, the pod network and storage replication.

This keeps cluster traffic off the public network, and lets the firewall trust the backplane completely while the public interface opens only the ports clients need. A backplane is optional, but it is the recommended layout for production.

The backplane does not have to be reserved for the cluster machines. Other hosts may be on it too, such as your CDN Directors: it is simply the network the cluster machines use to reach each other.

flowchart LR
    clients[Users and CDN Directors] -->|public network| eth0
    subgraph machine[Each cluster machine]
        eth0[eth0: public zone]
        eth1[eth1: acd-cluster zone]
    end
    eth1 <-->|backplane network| peers[Other cluster machines]

Requirements

• The backplane is a separate network from the public network.
• Every machine has exactly one interface with an address in the backplane network, and every machine is reachable from the others on it.
• The default route is on the public interface, not the backplane.
• Only hosts you trust are on the backplane. With manage_firewall: true, every host on the backplane can reach every port on every cluster machine, including the cluster datastore (etcd, 2379-2380/tcp), the Kubernetes API (6443/tcp) and the kubelet (10250/tcp), so connect only hosts you trust to it.
• Both interfaces are managed by NetworkManager, with an active connection.

Configuration

Give the backplane’s address range in /etc/acd/deploy/group_vars/all.yaml, before the first deploy:

backplane_network: "10.10.0.0/24"

Nothing else is needed: no per-machine settings and no firewall commands. deploy finds each machine’s interface in that range and sets up the rest.

Important: backplane_network is locked at the first installation. A cluster cannot be moved on to or off a backplane later without reinstalling it.

What deploy Sets Up

On each machine, with manage_firewall: true:

• Kubernetes uses the backplane: each machine’s cluster address is its backplane address, the pod network (Flannel) runs over the backplane interface, and the public address is advertised as the machine’s external address.
• The backplane interface is placed in the acd-cluster zone, which accepts all traffic, together with the pod and service networks.
• The public interface is placed in firewalld’s public zone, with the ports clients need opened through the acd-public policy.
• Routing rules keep traffic to the pod, service and backplane networks on the main routing table.

Other interfaces are left as they are.

Verification

After deploy, on any machine:

firewall-cmd --get-active-zones

Expected output:

acd-cluster
  interfaces: eth1
  sources: 10.42.0.0/16 10.43.0.0/16
public
  interfaces: eth0

Check the cluster addresses Kubernetes uses:

grep -E 'node-ip|external|flannel' /etc/rancher/k3s/config.yaml.d/20-acd-node.yaml

Expected output, on manager-1:

node-ip: "10.10.0.11"
node-external-ip: "192.0.2.11"
flannel-iface: "eth1"

And on a server:

kubectl get nodes -o wide

INTERNAL-IP shows each machine’s backplane address, and EXTERNAL-IP its public address.

Problems

For every code, see Installation Troubleshooting.

4 - Architecture Guide

Overview

The AgileTV CDN Manager (ESB3027) is a cloud-native Kubernetes application designed for managing CDN operations. This guide provides a detailed description of the system architecture, component interactions, and scaling considerations.

High-Level Architecture

The CDN Manager follows a microservices architecture deployed on Kubernetes. The system is organized into logical layers:

graph LR
    Clients[API Clients] --> Ingress[Ingress Controller]
    Ingress --> Manager[Core Manager]
    Ingress --> Frontend[MIB Frontend]
    Ingress --> Grafana[Grafana]
    Manager --> Redis[(Redis)]
    Manager --> Kafka[(Kafka)]
    Manager --> PostgreSQL[(PostgreSQL)]
    Manager --> Zitadel[Zitadel IAM]
    Manager --> Confd[Configuration Service]
    Grafana --> VM[(VictoriaMetrics)]
    Confd -.-> Gateway[NGinx Gateway]
    Gateway --> Director[CDN Director]

Component Architecture

Ingress Layer

The ingress layer manages all incoming traffic to the cluster:

Traffic flow:

• API clients and Operator UI connect via the Ingress Controller at /api and /gui paths respectively
• Grafana dashboards are accessed via the Ingress Controller at /grafana
• Zitadel authentication console is accessed via the Ingress Controller at /ui/console

Application Services

The application layer contains the core CDN Manager services:

Data Layer

The data layer provides persistent and ephemeral storage:

External Integrations

Detailed Component Descriptions

Core Manager

The Core Manager is the central application server that exposes the REST API. It is implemented in Rust using the Actix-web framework.

Key Responsibilities:

• Authentication and session management via Zitadel
• Configuration document storage and retrieval
• Selection input CRUD operations
• Routing rule evaluation and GeoIP lookups
• Host and namespace discovery
• Operator UI helper endpoints

API Endpoints:

• /api/v1/auth/* - Authentication (login, token, logout)
• /api/v1/configuration - Configuration management
• /api/v1/selection_input/* - Selection input operations
• /api/v2/selection_input/* - Enhanced selection input with list operations
• /api/v1/routing/* - Routing evaluation and validation
• /api/v1/discovery/* - Host and namespace discovery
• /api/v1/metrics - System metrics
• /api/v1/health/* - Liveness and readiness probes
• /api/v1/operator_ui/* - Operator helper endpoints

Runtime Modes: The Core Manager supports multiple runtime modes, each deployed as a separate container:

• http-server - Primary HTTP API server (default)
• metrics-aggregator - Background worker for metrics collection
• selection-input - Background worker for Kafka selection input consumption

MIB Frontend

The MIB Frontend provides a web-based GUI for configuration management.

Key Features:

• Intuitive web interface for CDN configuration
• Real-time configuration validation
• Integration with Zitadel for SSO authentication

Confd (Configuration Service)

Confd provides routing configuration services and synchronizes with the Core Manager application.

Key Responsibilities:

• Hosts the service configuration for routing decisions
• Provides API and CLI for configuration management
• Synchronizes routing configuration with Core Manager
• Maintains configuration state in PostgreSQL

Selection Input Worker

The Selection Input Worker processes selection input events from the Kafka stream.

Key Responsibilities:

• Consumes messages from the selection_input Kafka topic
• Validates and transforms input data
• Updates configuration in the data store
• Maintains message ordering within partitions

Scaling Limitation: The Selection Input Worker cannot be scaled beyond a single consumer per Kafka partition, as message ordering must be preserved.

Metrics Aggregator

The Metrics Aggregator collects and processes metrics from CDN components.

Key Responsibilities:

• Polls metrics from CDN Director instances
• Aggregates usage statistics
• Writes data to VictoriaMetrics (Analytics) for dashboards
• Writes long-term data to VictoriaMetrics (Billing) for compliance

Telegraf

Telegraf is deployed as a DaemonSet to collect host-level metrics.

Key Responsibilities:

• CPU, memory, disk, and network metrics from each node
• Container-level resource usage
• Kubernetes cluster metrics
• Forwards metrics to VictoriaMetrics

Grafana

Grafana provides visualization and dashboard capabilities.

Features:

• Pre-built dashboards for CDN monitoring
• Custom dashboard support
• VictoriaMetrics as data source
• Alerting integration with Alertmanager

Access: https://<host>/grafana

Alertmanager

Alertmanager handles alert routing and notifications.

Key Responsibilities:

• Receives alerts from Grafana and other sources
• Deduplicates and groups alerts
• Routes to notification channels (email, webhook, etc.)
• Manages alert silencing and inhibition

Data Storage

Redis

Redis provides in-memory storage for:

• User sessions and authentication tokens
• Ephemeral configuration cache
• Real-time state synchronization

Deployment: Single instance, with no replicas. Redis is unavailable while its pod restarts or moves to another node.

Kafka

Kafka provides durable event streaming for:

• Selection input events
• Metrics data streams
• Inter-service communication

Deployment: Controller cluster with 3 replicas for production, 1 replica for lab deployments

Node Affinity: Kafka replicas must be scheduled on separate nodes to ensure high availability. The Helm chart configures topology spread constraints to enforce this distribution.

Topics:

• selection_input - Selection input events. One replica, by design: see Data Replication.
• subnets - Subnet changes. Three replicas.

Note: For lab/single-node deployments, the Kafka replica count must be set to 1 in the Helm values. Production deployments require 3 replicas for fault tolerance.

PostgreSQL

PostgreSQL provides persistent storage for:

• Configuration documents
• User and permission data
• System state
• The subnet map

The cluster holds four databases: manager (the manager’s own data, including the subnet map), app (Grafana), zitadel and confsej.

Deployment: 3-node cluster managed by Cloudnative PG (CNPG) operator

High Availability: The CNPG operator manages automatic failover and ensures high availability:

• One primary node handles read/write operations
• Two replica nodes provide redundancy and can be promoted to primary on failure
• Automatic failover occurs within seconds of primary node failure
• Synchronous replication ensures data consistency

Note: The PostgreSQL cluster is deployed and managed automatically by the CNPG operator. Manual intervention is typically not required for normal operations.

VictoriaMetrics

Two VictoriaMetrics instances serve different purposes:

VictoriaMetrics (Analytics):

• Real-time and short-term metrics storage
• Supports Grafana dashboards
• Retention: Configurable (typically 30-90 days)

VictoriaMetrics (Billing):

• Long-term metrics retention
• Billing and license compliance data
• Retention: Minimum 1 year

Authentication and Authorization

Zitadel Integration

Zitadel provides identity and access management:

Authentication Flow:

1. User accesses MIB Frontend or API
2. Redirected to Zitadel for authentication
3. Zitadel validates credentials and issues session token
4. Session token exchanged for access token
5. Access token included in API requests (Bearer authentication)

Default Credentials: See the Glossary for default login credentials.

Access Paths:

• Zitadel Console: /ui/console
• API authentication: /api/v1/auth/*

External Hostname

Zitadel validates the URL that clients connect to against its configured external hostname. deploy sets it from the first entry of manager_hostnames in the site settings, which must be the hostname clients use to reach the system.

Network Architecture

Traffic Flow

graph TB
    External[External Clients] --> Ingress[Ingress Controller]
    External --> Redis[(Redis)]
    External --> Kafka[(Kafka)]
    External --> Telegraf[Telegraf]
    Ingress --> Manager[Core Manager]
    Ingress --> Frontend[MIB Frontend]
    Ingress --> Grafana[Grafana]
    Ingress --> Zitadel[Zitadel]

Note: Certain services (Redis, Kafka, Telegraf) can be accessed directly by external clients without traversing the ingress controller. This is typically used for metrics collection, event streaming, and direct data access scenarios.

Internal Communication

All internal services communicate over the Kubernetes overlay network (Flannel VXLAN). Services discover each other via Kubernetes DNS.

External Communication

• CDN Directors: Accessed via NGinx Gateway for simplified routing.
• MaxMind GeoIP: Local database files (no external calls)

Scaling

Horizontal Pod Autoscaler (HPA)

The following components support automatic horizontal scaling via HPA:

Note: HPA is enabled by default in the Helm chart. The default configuration is tuned for production deployments. Adjust min/max values based on expected load and available cluster capacity.

Manual Scaling

Components can also be scaled manually by setting replica counts in the Helm values:

cdn-manager:
  manager:
    replicaCount: 3
  mibFrontend:
    replicaCount: 2

Important: When manually setting replica counts, you must disable the Horizontal Pod Autoscaler (HPA) for the corresponding component. If HPA remains enabled, it will override manual replica settings. To disable HPA, set autoscaling.hpa.enabled: false for the component in your Helm values.

Components That Do Not Scale

The following components do not support horizontal scaling:

Node Scaling

Additional Agent nodes can be added to the cluster at any time to increase workload capacity. Kubernetes automatically schedules pods to nodes with available resources.

Cluster Balancing

The CDN Manager deployment includes the Kubernetes Descheduler to maintain balanced resource utilization across cluster nodes:

• Automatic Rebalancing: The descheduler periodically analyzes pod distribution and evicts pods from overutilized nodes
• Node Balance: Helps prevent resource hotspots by redistributing workloads across available nodes
• Integration with HPA: Works in conjunction with Horizontal Pod Autoscaler to optimize both pod count and placement

The descheduler runs as a background process and does not require manual intervention under normal operating conditions.

Resource Configuration

For detailed resource preset configurations and planning guidance, see the Configuration Guide.

High Availability

Server Node Redundancy

Production deployments require a minimum of 3 Server nodes:

• Survives loss of 1 server node
• Maintains quorum for etcd and Kafka

For enhanced availability, use 5 Server nodes:

• Survives loss of 2 server nodes
• Recommended for critical production environments

For large-scale deployments, 7 or more Server nodes can be used:

• Survives loss of 3+ server nodes
• Suitable for high-capacity production environments

Pod Distribution

Kubernetes automatically distributes pods across nodes to maximize availability:

• Pods with the same deployment are scheduled on different nodes when possible
• Pod Disruption Budgets (PDB) ensure minimum availability during maintenance

Data Replication

Longhorn Storage: Each volume keeps three replicas on different nodes, so it survives the loss of a node. The three Kafka data volumes keep one replica each, because Kafka copies its topics across its brokers itself. Longhorn tries to keep one replica on the node where the volume’s pod runs, but a pod can run on a node with no replica of its volume.

selection_input has one replica by design. The topic is high-volume and short-lived, so it keeps a single replica, on one broker’s single-replica volume. The cost: selection input is unavailable while that broker restarts or moves to another node, and its messages are lost if that broker’s volume is lost. Every other topic survives the loss of one broker. See Safety before you restart Kafka or drain a node.

Next Steps

After understanding the architecture:

1. Installation Guide - Deploy the CDN Manager
2. Configuration Guide - Configure components for your environment
3. Operations Guide - Day-to-day operational procedures
4. Performance Tuning Guide - Optimize system performance
5. Metrics & Monitoring - Set up monitoring and alerting

5 - Installation Guide

Overview

The CDN Manager is installed with deploy, a command on the installation ISO. You describe your machines and your site in a few files on the first machine, and deploy does the rest over SSH: it configures each machine’s firewall and network, builds the Kubernetes (k3s) cluster, installs the CDN Manager, and checks that it is healthy.

deploy is the only supported way to install the CDN Manager. Running the ISO’s install, join-server and join-agent scripts by hand is no longer supported.

New here? Start with the Quickstart. It walks you through a complete installation on three machines, from minimal operating system installs to signing in to the GUI. The other pages cover the details and the variations.

Upgrading from 1.6 or earlier? Version 1.8 cannot be installed over an earlier release. Install it on a new cluster. See the Upgrade Guide.

How It Works

flowchart LR
    A[Mount the ISO<br>on the first server] --> B[deploy init]
    B --> C[Edit /etc/acd/deploy]
    C --> D[deploy]
    D --> E[Sign in to the GUI]

1. Mount the ISO on the machine that will be the first server.
2. Run deploy init once. It installs Ansible and creates the configuration folder /etc/acd/deploy.
3. Edit the configuration: the machines and their roles, your site settings, and optionally a TLS certificate, MaxMind databases and application settings.
4. Run deploy. It checks everything first, takes a backup, then installs or updates the cluster to match your configuration.

To change anything later, edit the configuration and run deploy again. Every run is safe to repeat.

Guides

Prerequisites

• Machines meeting the System Requirements: three servers for production, optionally with agents for extra capacity. A single server is enough for a lab.
• Operating system: a minimal install of Red Hat Enterprise Linux 9.4 or later, or a compatible rebuild, with firewalld and NetworkManager running.
• Network: a default route on every machine and, optional but recommended, a backplane network, separate from the public network, that all the machines are reachable on. See the Networking Guide.
• Access: passwordless root SSH from the first server to every machine, itself included.
• Software: the installation ISO from the AgileTV customer portal, and dnf access to the operating system’s repositories on every machine.
• Recommended: a TLS certificate from a trusted certificate authority, and MaxMind GeoIP databases from your own MaxMind subscription.

Getting Help

• Installation Troubleshooting lists every failure code deploy reports.
• If the final health check fails, deploy writes /etc/acd/deploy/support-bundle-acd.tar.gz. Send it to AgileTV support.
• The Release Notes list known issues for each version.

5.1 - Quickstart

Overview

This guide takes three freshly installed Enterprise Linux 9 machines and turns them into a production CDN Manager cluster, ending with you signed in to the web GUI. Follow it from top to bottom. Each step shows the commands to run and the output to expect, and links to a page with the details and the alternatives.

The whole installation is driven by one command, deploy, which ships on the installation ISO. You run it on the first machine, and it configures every machine over SSH: the host firewall and network, the Kubernetes cluster, and the CDN Manager itself. You do not run anything by hand on the other machines.

Estimated time: about 30 minutes, of which deploy itself takes 15 to 20.

The example cluster

The commands below use this example. Replace the names and addresses with your own as you go.

• Manager hostname: manager.example.com, the name users and CDN Directors use to reach the manager. Its DNS record points at the public address of one or more of the machines.
• Backplane network: 10.10.0.0/24, the network the machines use to talk to each other. A backplane is optional, but recommended. When you use one, all node-to-node (east-west) traffic stays on it, and every machine must be reachable on it. It should be a separate network from the public one. It does not have to be reserved for the cluster, so other hosts may also be on it.
• CDN Director: director-1 at 192.0.2.201. It runs on its own machine, outside this cluster.

Single network? A dedicated backplane is recommended, but not required. If each machine has only one network interface, follow the guide as written and leave backplane_network empty in Step 6. See the Networking Guide for both layouts.

Before You Begin

Make sure you have:

• Three machines that meet the System Requirements, each with a minimal install of Red Hat Enterprise Linux 9.4 or later, or a compatible rebuild such as Oracle Linux, AlmaLinux or Rocky Linux. Keep the defaults: firewalld and NetworkManager running, and any SELinux mode.
• Final hostnames on every machine, each one unique. A machine cannot be renamed after it joins the cluster.
• A default route on every machine, on the public interface. It does not need to lead to the internet: on an isolated network, use your site’s router or any gateway address on that subnet.
• Root access to every machine, by password or key, to set up SSH in Step 1.
• Package repositories: every machine must be able to install packages with dnf from the operating system’s repositories. deploy installs a few packages, such as ansible-core on the first machine and the storage tools on all of them. Without internet access, see Air-Gapped Installation.
• A DNS name for the manager, such as manager.example.com.
• The installation ISO, esb3027-cdn-manager-1.8.0.iso, from the AgileTV customer portal.
• A TLS certificate for the manager hostname, issued by a certificate authority your users trust, with its private key. You can start without one; see Step 7.
• MaxMind GeoIP databases from your own MaxMind subscription: City, ASN and Anonymous IP. You can start without them; see Step 8.

Run every command in this guide as root on manager-1, unless a step says otherwise.

Step 1: Set Up SSH Keys

deploy connects from manager-1 to every machine as root, manager-1 itself included, and needs to log in without a password prompt.

Create a key on manager-1:

ssh-keygen -t ed25519 -N '' -C "acd-deploy@$(hostname)" -f /root/.ssh/id_ed25519

Expected output:

Generating public/private ed25519 key pair.
Your identification has been saved in /root/.ssh/id_ed25519
Your public key has been saved in /root/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:igFCBNWbW74iWVSPZdMrGhtpKXiPQedvUZEGPmzppsI acd-deploy@manager-1
...

Copy it to all three machines. Enter each machine’s root password when prompted:

for ip in 192.0.2.11 192.0.2.12 192.0.2.13; do
    ssh-copy-id -o StrictHostKeyChecking=accept-new -i /root/.ssh/id_ed25519.pub root@$ip
done

Expected output, once per machine:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.0.2.11's password:

Number of key(s) added: 1

Check that each login now works without a prompt:

for ip in 192.0.2.11 192.0.2.12 192.0.2.13; do
    ssh -o BatchMode=yes root@$ip hostname
done

Expected output:

manager-1
manager-2
manager-3

If a machine asks for a password or prints Permission denied, fix that before going on. Otherwise deploy stops with UNREACHABLE.

Step 2: Check the Machines

Confirm that each machine runs a supported release, has its addresses and default route, and runs firewalld and NetworkManager:

for ip in 192.0.2.11 192.0.2.12 192.0.2.13; do
    echo "===== $ip"
    ssh root@$ip 'cat /etc/redhat-release; ip -br -4 addr; ip route show default; systemctl is-active firewalld NetworkManager'
done

Expected output, for each machine:

===== 192.0.2.11
Red Hat Enterprise Linux release 9.4 (Plow)
lo               UNKNOWN        127.0.0.1/8
eth0             UP             192.0.2.11/24
eth1             UP             10.10.0.11/24
default via 192.0.2.1 dev eth0 proto static
active
active

Check for:

• A release of 9.4 or later.
• The default route on the public interface (eth0).
• The backplane address on another interface (eth1), inside the backplane network.
• active twice: firewalld, then NetworkManager.

deploy checks all of this again before it changes anything, along with processor cores, memory and kernel modules, and stops with a message naming the problem.

Step 3: Mount the Installation ISO

Copy the ISO to manager-1, for example with scp from your workstation:

scp esb3027-cdn-manager-1.8.0.iso root@192.0.2.11:/root/

Then, on manager-1, mount it at /mnt/esb3027:

mkdir -p /mnt/esb3027
mount -o loop,ro /root/esb3027-cdn-manager-1.8.0.iso /mnt/esb3027
ls -1 /mnt/esb3027

Expected output:

ansible
BUILD_ARCH
charts
common.sh
deploy
install
join-agent
join-server
k3s_installer.sh
load-maxmind-databases
openapi.yaml
Packages
rootfs.tar
values-lab.yaml
values-production.yaml
values.yaml
VERSION.txt

The ISO is needed only on manager-1. deploy copies what the other machines need.

Important: Keep the ISO mounted at /mnt/esb3027 whenever you run deploy, now and later. The deployment’s configuration reads its application settings from the ISO, and deploy stops if the ISO is not mounted. The mount does not survive a reboot, so mount it again before the next run.

Step 4: Initialize the Deployment

Initialize manager-1 with the production preset:

/mnt/esb3027/deploy init prod

This installs ansible-core from the operating system’s repositories, installs the Ansible collections bundled on the ISO, and creates the configuration folder /etc/acd/deploy. Expected output (the package list is shortened here):

###############################################################
            ESB3027 AgileTV CDN Manager -- Deployer
 Version: 1.8.0
###############################################################

Installing ansible-core from the OS repositories ...
Dependencies resolved.
================================================================================
 Package               Arch    Version                 Repository          Size
================================================================================
Installing:
 ansible-core          x86_64  1:2.14.18-3.el9         appstream          3.9 M
...
Complete!
Installing bundled Ansible collections ...

Initialization complete, using the production preset. Next:
  1. Edit /etc/acd/deploy/inventory.ini        - your node addresses and roles.
  2. Edit /etc/acd/deploy/group_vars/all.yaml  - at minimum, manager_hostnames.
  3. (Optional) drop a TLS certificate + key into /etc/acd/deploy/certs/
     and the three MaxMind .mmdb files into /etc/acd/deploy/maxmind/.
  4. (Optional) change application settings in /etc/acd/deploy/values.override.yaml.
  5. Run: /mnt/esb3027/deploy

See /etc/acd/deploy/README.md for the full input contract.

WARNING: deploy connects over SSH from this node to every node in
inventory.ini, this node included, as root unless ansible_user says
otherwise. Make sure SSH keys are set up so each of those logins works
without a password prompt before running deploy.

The next four steps follow that list. Everything you configure lives in /etc/acd/deploy:

/etc/acd/deploy/
├── inventory.ini          # Step 5: your machines and their roles
├── group_vars/all.yaml    # Step 6: your site settings
├── certs/                 # Step 7: your TLS certificate and key
├── maxmind/               # Step 8: your MaxMind databases
├── values.override.yaml   # optional application settings, empty to start
├── preset.yaml            # link to the production preset on the ISO; do not edit
└── README.md              # the full reference for this folder

certs/ and maxmind/ each start with a README.txt: short instructions for that folder, which deploy ignores. Leave it there or delete it.

For other options, such as the lab preset, see the deploy reference.

Step 5: List Your Machines

Open /etc/acd/deploy/inventory.ini in an editor. It is a commented example with three sections. Edit the machine lines so that your three machines are listed under [server], with the address deploy reaches each one on over SSH, and the other two sections are empty:

[server]
manager-1 ansible_host=192.0.2.11
manager-2 ansible_host=192.0.2.12
manager-3 ansible_host=192.0.2.13

[agent]

[director]

• [server]: the first server listed must be the machine you run deploy on, here manager-1. The number of servers must be odd: one for a lab, three for production.
• [agent]: lists any additional machines beyond the three servers. It may be left empty, as here.
• [director]: leave it empty. CDN Directors run on their own machines and are listed in the next step.

To add agent machines later, or to set per-machine options, see Inventory.

Step 6: Enter Your Site Settings

Open /etc/acd/deploy/group_vars/all.yaml. Every setting in it is explained in its comments and already holds a working default. For this cluster, change three lines:

manager_hostnames: [manager.example.com]
external_directors: [{name: director-1, address: 192.0.2.201}]
backplane_network: "10.10.0.0/24"

To review your changes, print the settings without their comments:

grep -Ev '^\s*(#|$)' /etc/acd/deploy/group_vars/all.yaml

Expected output:

---
manager_hostnames: [manager.example.com]
manage_firewall: true
manage_routes: true
external_directors: [{name: director-1, address: 192.0.2.201}]
backplane_network: "10.10.0.0/24"
cluster_pod_cidr: 10.42.0.0/16
cluster_svc_cidr: 10.43.0.0/16
firewall_cluster_zone: acd-cluster
firewall_public_policy: acd-public
tls_cert_file: ""
tls_key_file: ""
maxmind_city_db: ""
maxmind_asn_db: ""
maxmind_anon_db: ""
backup_dir: /var/lib/acd/backups

Important: backplane_network, cluster_pod_cidr, cluster_svc_cidr, firewall_cluster_zone and firewall_public_policy are locked at the first installation. Changing one later means reinstalling the cluster, which starts empty: the PostgreSQL databases, Kafka’s messages and everything else stored in the cluster are lost unless you have backed them up. Check them now.

With manage_firewall: true, the default, deploy configures firewalld on every machine for you. The backplane interface is trusted for all traffic, and the public interface stays in firewalld’s public zone with only the ports the manager needs opened. Before you run deploy, make sure the public zone allows everything else you need on the public interface, SSH included: firewall-cmd --zone=public --list-all on each machine should list ssh under services.

For every setting, see Site Settings.

Step 7: Add Your TLS Certificate

Copy your certificate and its private key into /etc/acd/deploy/certs/:

cp manager-fullchain.crt manager.key /etc/acd/deploy/certs/
chmod 600 /etc/acd/deploy/certs/manager.key

• Both files must be PEM (text beginning with -----BEGIN), in two separate files. The file names do not matter.
• Use the full chain: your certificate followed by any intermediate certificates.
• The key must not have a passphrase.
• The certificate must cover the first entry of manager_hostnames, directly or by a wildcard.

Check what the certificate covers:

openssl x509 -in /etc/acd/deploy/certs/manager-fullchain.crt -noout -subject -issuer -dates -ext subjectAltName

Expected output:

subject=CN = manager.example.com
issuer=CN = Example Issuing CA
notBefore=Oct 14 18:57:23 2026 GMT
notAfter=Jan 16 18:57:23 2027 GMT
X509v3 Subject Alternative Name:
    DNS:manager.example.com

deploy checks the certificate and key before it changes anything.

No certificate yet? Add nothing to the folder: its README.txt can stay. deploy then serves a temporary self-signed certificate, which browsers warn about. Add the real one later and run deploy again to replace it. For production, use a certificate from a certificate authority your users already trust. See TLS Certificates.

Step 8: Add Your MaxMind Databases

The manager uses MaxMind GeoIP databases for geographic routing. Copy the three .mmdb files from your MaxMind subscription into /etc/acd/deploy/maxmind/:

cp GeoIP2-City.mmdb GeoLite2-ASN.mmdb GeoIP2-Anonymous-IP.mmdb /etc/acd/deploy/maxmind/
ls -l /etc/acd/deploy/maxmind/

Expected output:

total 126236
-rw-r--r--. 1 root root      4382 Oct 14 18:57 GeoIP2-Anonymous-IP.mmdb
-rw-r--r--. 1 root root 121642973 Oct 14 18:57 GeoIP2-City.mmdb
-rw-r--r--. 1 root root   7609502 Oct 14 18:57 GeoLite2-ASN.mmdb
-rw-r--r--. 1 root root       400 Oct 14 18:57 README.txt

Files are matched by name: one containing City, one ASN and one Anon. Provide all three or none. With none, geographic routing is off until you add them and run deploy again. See MaxMind GeoIP Databases.

Step 9: Run the Deployment

Start the installation:

/mnt/esb3027/deploy

deploy works through fixed stages, and stops at the first problem with a message naming it:

1. Gather inputs: reads every machine and works out the settings. Changes nothing.
2. Preflight: checks the machines and your settings. Changes nothing.
3. Backup: archives each machine’s configuration before any change.
4. Host network: configures the firewall zones, routing rules and kernel network settings on each machine, then checks the result.
5. Cluster: installs Kubernetes (k3s) on manager-1, then joins the other servers one at a time.
6. Platform: installs the CDN Manager, loads the MaxMind databases, and runs a final health check.

The output is long. Shortened, it looks like this:

###############################################################
            ESB3027 AgileTV CDN Manager -- Deployer
 Version: 1.8.0
###############################################################

Running the operator playbook (log: /etc/acd/deploy/deploy.log) ...
[WARNING]: Collection ansible.posix does not support Ansible version 2.14.18

PLAY [Gather inputs - host probes, host network plans and k3s] *****************

TASK [Check that the node can be reached] **************************************
ok: [manager-1]
ok: [manager-3]
ok: [manager-2]
...
PLAY [Join or upgrade the other servers, one at a time] ************************
...
TASK [cluster : Join or upgrade this node with join-server] ********************
changed: [manager-2]
FAILED - RETRYING: [manager-2]: Wait for this server to become Ready (20 retries left).
FAILED - RETRYING: [manager-2]: Wait for this server to become Ready (19 retries left).
...
TASK [cluster : Wait for this server to become Ready] **************************
ok: [manager-2]
...
TASK [acceptance : Report the go verdict] **************************************
ok: [manager-1] => {
    "msg": "Deployment gate PASSED for cdn-platform. GeoIP is serving."
}
...
PLAY RECAP *********************************************************************
manager-1                  : ok=185  changed=28   unreachable=0    failed=0    skipped=106  rescued=0    ignored=0
manager-2                  : ok=116  changed=16   unreachable=0    failed=0    skipped=48   rescued=0    ignored=0
manager-3                  : ok=116  changed=16   unreachable=0    failed=0    skipped=48   rescued=0    ignored=0

The installation succeeded when you see Deployment gate PASSED and the recap shows unreachable=0 and failed=0 for every machine. Your other counts may differ.

Some lines look alarming but are expected:

• FAILED - RETRYING: ... Wait for this server to become Ready: deploy polls each joining server until it is ready. Ten or more of these per server is normal.
• [WARNING]: Collection ansible.posix does not support Ansible version ...: harmless.
• [WARNING]: Could not match supplied host pattern, ignoring: acd_host_network_verify: harmless.

The whole run is also appended to /etc/acd/deploy/deploy.log.

If deploy stops, its message starts with a code in square brackets, such as [ACD-TLS-HOSTNAME]. Look the code up in Installation Troubleshooting, fix the cause, and run deploy again. Running it again is safe: it picks up where the cluster is and re-applies your settings.

Step 10: Verify the Cluster

Check that all three machines are Ready:

kubectl get nodes -o wide

Expected output (the columns right of EXTERNAL-IP are trimmed here):

NAME        STATUS   ROLES                AGE     VERSION        INTERNAL-IP   EXTERNAL-IP
manager-1   Ready    control-plane,etcd   12m     v1.35.0+k3s1   10.10.0.11    192.0.2.11
manager-2   Ready    control-plane,etcd   8m47s   v1.35.0+k3s1   10.10.0.12    192.0.2.12
manager-3   Ready    control-plane,etcd   6m3s    v1.35.0+k3s1   10.10.0.13    192.0.2.13

INTERNAL-IP is each machine’s backplane address, and EXTERNAL-IP its public address.

Check every component of the CDN Manager:

acd-admin status

Every admin target should show ok under HEALTH, and every node yes under READY. acd-admin status exits 0 either way, so read the output. See acd-admin status for what each column means.

Check that the manager answers on its hostname:

curl https://manager.example.com/api/v1/health/ready; echo

Expected output:

{"status":"Ok"}

Note: If your certificate was issued by a private certificate authority that this machine does not trust, add --cacert <your CA certificate> to the curl commands. If you started without a certificate, use -k instead.

If you listed a CDN Director in external_directors, check that the manager reaches it. The trailing slash is required:

curl -s -o /dev/null -w '%{http_code}\n' https://manager.example.com/confd/director-1/config/__active/

Expected output:

200

200 shows that the manager reached the Director and read its configuration. A 5xx status comes from the manager’s gateway when it cannot reach the Director: check the Director’s address in external_directors, that the Director is running, and that the network between them allows the connection. Connecting the Director to the manager is covered in the CDN Director documentation.

Important: Your CDN Directors reach Kafka on ports 9092/tcp and 30000-30003/tcp on every machine (one more port per broker if you run more than the default three). The host firewall does not filter these ports, so they are open to any network that can reach the machines. Restrict them to your CDN Directors on your network’s own firewall. See the Networking Guide.

Step 11: Back Up the Zitadel Master Key

Zitadel, the manager’s identity service, encrypts its stored credentials with a master key. If the key is lost, it cannot be recovered. Print it:

kubectl get secret zitadel-masterkey --namespace default -o jsonpath="{.data.masterkey}" | base64 --decode; echo

Store the output somewhere safe, away from the cluster.

Step 12: Sign In to the GUI

In a browser, open https://manager.example.com/gui/:

Sign in with the default administrator account:

• Username: admin@zitadel.manager.example.com, that is, admin@zitadel. followed by your main manager hostname.
• Password: Password1!

Note: Use the manager hostname you configured, not a machine’s IP address or another name: signing in only works under the main manager hostname. The GUI also needs the full login name above; the email address or admin alone is rejected.

After signing in, the GUI home page appears:

Your CDN Manager cluster is installed.

What’s Next

• Secure the administrator account. The default credentials are published. Create your own administrator, then disable the default account: see Next Steps.
• Add capacity with agent machines: see Inventory.
• Change application settings, such as retention or disk sizes: see Application Settings.
• Change your configuration later: edit the files in /etc/acd/deploy and run deploy again. That is the only way to change what is installed. See Running deploy.

5.2 - Inventory

Overview

The inventory, /etc/acd/deploy/inventory.ini, lists every machine in the cluster and its role. deploy init creates it as a commented example. Edit it before the first deploy, and again whenever you add a machine.

Each machine is one line under the heading for its role:

[server]
manager-1 ansible_host=192.0.2.11
manager-2 ansible_host=192.0.2.12
manager-3 ansible_host=192.0.2.13

[agent]
manager-4 ansible_host=192.0.2.14

[director]

• The first word is the machine’s name. Use its hostname.
• ansible_host is the address deploy reaches the machine on over SSH. It does not decide which network the cluster uses: see Site Settings for that.
• Everything after a # is a comment, and indentation does not matter.

Important: The inventory also has a [director] section. Leave it empty. Installing the CDN Director in Kubernetes is not supported in this release. CDN Directors run on their own machines and are listed in external_directors in Site Settings.

Roles

Servers

Servers run the Kubernetes control plane and the cluster datastore, and by default also run the CDN Manager’s workloads.

• The first server listed is the one you run deploy on. It holds the ISO and the configuration folder.
• The number of servers must be odd, so the control plane can always reach a majority. Use three for production: the cluster then survives losing any one machine. One server is enough for a lab, with no redundancy.
• deploy stops with [ACD-INVENTORY-EVEN-SERVERS] on an even number, and with [ACD-INVENTORY-NO-SERVER] if there is none.

Agents

Agents run workloads only: they add capacity, but take no part in the control plane. List any machines beyond the servers under [agent], or leave the section empty.

To add an agent to a running cluster, add its line under [agent], set up passwordless SSH to it from the first server, and run deploy again. See Changing the Cluster.

A machine cannot be listed under both [server] and [agent] ([ACD-INVENTORY-SERVER-AND-AGENT]). Changing a machine’s role on a running cluster takes a few extra steps: see Changing a Machine’s Role.

How Many of Each

• Three servers are enough for most clusters. The cluster then survives losing any one server. You are unlikely to need more than five: five servers survive losing two, and more add little except overhead.
• Add servers to improve redundancy, two at a time, since the count must stay odd.
• Add agents to expand capacity. Agents take on workloads without adding to the control plane, so they are the way to grow a cluster.

Per-Machine Options

Each option is added to the end of a machine’s line, separated by a space. None is needed for a normal installation.

node_ip and node_external_ip

The machine’s cluster address (the one other machines reach it on) and its public address (the one clients use). Both are detected automatically:

• The public address is the address of the interface holding the default route.
• The cluster address is the machine’s address inside backplane_network if that is set, and otherwise its public address.

Set them only to correct one machine where the wrong address is chosen:

manager-2 ansible_host=192.0.2.12 node_ip=10.10.0.12

Every machine’s cluster address must be on one network that all of them share. deploy stops with [ACD-BACKPLANE-SPLIT] before changing anything if they are not.

Important: These addresses are locked at the first installation. Changing one later means reinstalling the cluster. See Locked at First Install.

node_labels

Your own Kubernetes labels for the machine, as a name-and-value mapping in single quotes:

manager-1 ansible_host=192.0.2.11 node_labels='{"rack": "a1"}'

For example, zone labels for Topology Aware Hints are described in the Performance Tuning Guide.

node_taints

Keeps ordinary workloads off the machine, for a server dedicated to the control plane. The platform’s own cluster services tolerate the taint below and keep running there. The value is a list in single quotes:

manager-1 ansible_host=192.0.2.11 node_taints='["CriticalAddonsOnly=true:NoSchedule"]'

Use it only when there are enough other machines to run the workloads, for example three tainted servers and at least three agents. Never taint every machine.

ansible_user

The account deploy logs in as on this machine. The default is root. Another account needs passwordless sudo:

manager-4 ansible_host=192.0.2.14 ansible_user=operator

To change the account for every machine at once, add it under an [all:vars] heading at the end of the file:

[all:vars]
ansible_user=operator

A setting on a machine’s own line still wins for that machine.

SSH Access

deploy logs in to every machine in the inventory from the first server, the first server included, without a password prompt. For each machine, on the first server:

ssh-copy-id root@<machine address>
ssh -o BatchMode=yes root@<machine address> true

The second command must succeed without a prompt. Otherwise deploy stops with UNREACHABLE for that machine.

5.3 - Site Settings

Overview

Your site settings live in /etc/acd/deploy/group_vars/all.yaml. They are the only deployment settings you set: everything that depends on them, such as the ingress hostnames, the TLS secret, Zitadel’s external domain and the list of CDN Directors, is worked out from them, so related values cannot fall out of step.

Every setting appears once in the file, with comments, and already holds a working default. Change a setting by editing its value in place, then run deploy.

A list can be written on one line or one entry per line. Both mean the same:

manager_hostnames: [manager.example.com, cdn.example.com]

manager_hostnames:
  - manager.example.com
  - cdn.example.com

Settings

manager_hostnames

Required. Every hostname that users and CDN Directors use to reach the manager.

manager_hostnames: [manager.example.com]

• The first entry is the main hostname. Your TLS certificate must cover it, and it is the only name users can sign in under: Zitadel rejects sign-ins made under any other name, including a machine’s IP address.
• Each name needs a DNS record pointing at the public address of one or more of the machines.
• Leave the list empty, [], only for a quick lab try-out. A temporary hostname and a self-signed certificate are then used.

external_directors

The CDN Directors that use this manager. They run on their own machines, outside the cluster: deploy installs nothing on them, and the manager routes to them.

external_directors:
  - {name: director-1, address: 192.0.2.201}
  - {name: director-2, address: 192.0.2.202}

• name is your own name for the Director. The manager publishes its configuration under it, for example at https://<manager hostname>/confd/director-1/config/__active/.
• address must be the Director’s IPv4 address, not a hostname. deploy stops with [ACD-DIRECTOR-ADDRESS] otherwise.
• Leave the list empty, [], if you have none yet, then add your Directors later and run deploy again. Until you do, the preset’s placeholder Directors stay in the configuration: director-1 and director-2 at 192.0.2.1 and 192.0.2.2 with the production preset, router-1 at 192.0.2.1 with the lab preset. Those addresses are reserved for documentation and reach nothing.

The Directors reach Kafka on 9092/tcp and 30000-30003/tcp on every machine, with one more port per broker beyond the default three. The host firewall does not filter these ports: restrict them on your network’s own firewall. See Kafka Ports.

Important: Each CDN Director must also be able to reach the manager’s ingress (HTTPS on 443/tcp) under one of the names in manager_hostnames. The ingress answers on every address of every cluster machine, public and backplane alike. So Directors on the backplane network can reach the manager there, instead of over the public network. In that case:

• Give the Directors a hostname that resolves, for them, to the machines’ backplane addresses, and add it to manager_hostnames, after the main hostname.
• Make sure your TLS certificate covers that name too, as a Subject Alternative Name (SAN). Otherwise the Directors’ certificate checks fail. See TLS Certificates.

manage_firewall

Whether deploy configures firewalld for you. The default is true.

• true: deploy starts and enables firewalld (it must be installed) and configures it on every machine. The machines trust each other for all traffic, the public interface is placed in firewalld’s public zone, and the ports clients need are opened. Before the first run, make sure the public zone allows everything else you need on the public interface, SSH included: firewall-cmd --zone=public --list-all should list ssh under services. See Firewall for exactly what is set up.
• false: you configure the firewall yourself. deploy only checks that the required ports are open and the pod and service networks are trusted, and stops before changing anything if not. Only firewalld is checked: if it is not running, the check is skipped, and your own firewall must allow the same traffic.

To switch a running cluster from true to false, follow Managing the Firewall Yourself first.

manage_routes

Whether deploy keeps cluster traffic on the main routing table. The default is true; leave it on.

With true, deploy adds lookup main routing rules at priority 50 for the pod network, the service network, and the network the machines reach each other on. This matters on machines that use policy routing, where traffic from an address is sent through its own routing table: without these rules, cluster traffic can leave through the wrong interface. On machines without policy routing, the rules are harmless.

• deploy owns priority 50 on the NetworkManager profiles of the backplane and public interfaces, and replaces any rule there at that priority. Keep your own rules at other priorities.
• Your own rules that cluster traffic can match, such as from <address> lookup <table>, must have a priority number of 51 or more. deploy stops with [ACD-ROUTING-CONFLICT], naming the rule, otherwise.
• If your rules must stay at 50 or below, set manage_routes: false and add your own lookup main rules for those three networks, at a priority other than 50 and ahead of your source rules. See Managing Routing Yourself for the rules and the commands.

Either way, deploy checks that cluster traffic uses the main routing table before it changes anything.

Locked at First Install

The settings below are marked as a separate section in the file. Set them before the first deploy and never change them afterwards: the cluster is built around them. Changing one later means uninstalling k3s from every machine and installing again. A reinstall starts with an empty cluster: the PostgreSQL databases, Kafka’s messages and everything else stored in the cluster are lost unless you have backed them up.

The same applies to each machine’s node_ip and node_external_ip in the inventory, and to the machines’ hostnames. On a running cluster, deploy compares these with what the cluster uses, and stops before changing anything if they differ.

backplane_network

Optional, but recommended. The address range of the backplane network, the network the cluster machines use to talk to each other, such as "10.10.0.0/24". Leave it empty, "", if you have no backplane: the machines then talk to each other over the public network.

When it is set, deploy finds each machine’s interface in that range and configures the cluster to use it: all node-to-node (east-west) traffic runs over it, including Kubernetes, its pod network and the cluster datastore.

• The backplane should be a separate network from the public network, the one holding the default route. Setting it to the public network’s range works, but then every host on the public network is trusted for all traffic.
• Every machine needs exactly one interface in the range, and must be reachable from the others on it. deploy stops with [ACD-BACKPLANE-NO-MATCH] or [ACD-BACKPLANE-AMBIGUOUS] otherwise.
• The backplane does not have to be reserved for the cluster: other hosts, such as your CDN Directors, may also be on it. With manage_firewall: true, every host on the backplane can reach every port on every cluster machine, including the cluster datastore (etcd, 2379-2380/tcp), the Kubernetes API (6443/tcp) and the kubelet (10250/tcp), so connect only hosts you trust to it.

See Configuring Segregated Networks.

cluster_pod_cidr and cluster_svc_cidr

The address ranges Kubernetes gives its pods and its services. The defaults are 10.42.0.0/16 and 10.43.0.0/16.

Change them only if your network already uses addresses in these ranges. The two must not overlap each other, or any address or route on any machine. deploy stops with [ACD-CIDR-OVERLAP] or [ACD-CIDR-IN-USE] before changing anything if they do.

firewall_cluster_zone and firewall_public_policy

The names of the firewalld zone and policy that deploy keeps as its own. The defaults are acd-cluster and acd-public. The zone trusts the cluster, and the policy opens the ports reached from outside.

deploy removes anything in them that it did not put there, so pick names nothing else on the machines uses. Never use one of firewalld’s own names, such as public or trusted. The zone name can be up to 17 characters and the policy name up to 18, using letters, digits, - and _. The defaults suit most sites.

File Locations

Most sites leave these empty: the files are picked up from the configuration folder.

5.4 - TLS Certificates

Overview

The manager serves all its interfaces (the GUI, the API, Zitadel and Grafana) over HTTPS on port 443, under the hostnames in manager_hostnames. Give it a certificate by putting the certificate and its key in /etc/acd/deploy/certs/ and running deploy.

We recommend a certificate from a certificate authority (CA) your users and CDN Directors already trust, such as your organization’s own CA or a public one. Without a certificate, deploy serves a temporary self-signed one, which every browser and client warns about.

Requirements

• PEM format: text files beginning with -----BEGIN. Convert DER or PKCS#12 files to PEM first.
• Two files: the certificate in one, the private key in the other. The file names do not matter: deploy identifies them by their contents.
• The full chain: your certificate followed by any intermediate certificates, often supplied as fullchain.
• No passphrase on the private key.
• The right names: the certificate must cover the first entry of manager_hostnames, directly or by a wildcard such as *.example.com. Cover the other entries too, as Subject Alternative Names (SANs), or clients using them fail their certificate check. This includes any name your CDN Directors use to reach the manager, such as a name for the backplane addresses.
• Exactly one certificate and one key in the folder.

Adding or Replacing the Certificate

1. Copy the files into the folder, and keep the key readable only by root:

   cp manager-fullchain.crt manager.key /etc/acd/deploy/certs/
   chmod 600 /etc/acd/deploy/certs/manager.key
   

   When replacing a certificate, remove the old files first, so the folder holds only one pair.

2. Check the names and dates:

   openssl x509 -in /etc/acd/deploy/certs/manager-fullchain.crt -noout -subject -issuer -dates -ext subjectAltName
   

   Expected output:

   subject=CN = manager.example.com
   issuer=CN = Example Issuing CA
   notBefore=Oct 14 18:57:23 2026 GMT
   notAfter=Jan 16 18:57:23 2027 GMT
   X509v3 Subject Alternative Name:
       DNS:manager.example.com
   

3. Run deploy:

   /mnt/esb3027/deploy
   

deploy checks the pair before it changes anything: that both are readable, that they match, that the certificate has not expired, and that it covers the main hostname. It then stores them in the cluster as the ingress certificate for every manager hostname.

The certificate and key are not included in the backups deploy takes. Keep your own copies.

Checking the Served Certificate

From any machine that can reach the manager:

openssl s_client -connect manager.example.com:443 -servername manager.example.com </dev/null 2>/dev/null | grep -E 'subject=|issuer=|Verify return code'

Expected output:

subject=CN = manager.example.com
issuer=CN = Example Issuing CA
Verify return code: 0 (ok)

A verify return code other than 0 means this machine does not trust the issuing CA. With a private CA, add -CAfile <your CA certificate> to the command, and install the CA certificate on every client that connects to the manager, CDN Directors included.

A Certificate for a Test Cluster

For a lab or test cluster without a certificate, you can create a private CA and issue a certificate from it. Clients then trust the manager once they trust the CA. Do not use this for production.

mkdir -p /root/test-ca && cd /root/test-ca

# A private CA, valid for ten years
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
    -keyout test-ca.key -out test-ca.crt -subj "/CN=Test CA"

# A key and certificate request for the manager
openssl req -newkey rsa:2048 -sha256 -nodes \
    -keyout manager.key -out manager.csr -subj "/CN=manager.example.com"

# Sign it, naming every manager hostname
cat > manager.ext <<'EOF'
basicConstraints=CA:FALSE
keyUsage=digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:manager.example.com
EOF
openssl x509 -req -in manager.csr -CA test-ca.crt -CAkey test-ca.key -CAcreateserial \
    -days 825 -sha256 -extfile manager.ext -out manager.crt

# Install the full chain and the key
cat manager.crt test-ca.crt > /etc/acd/deploy/certs/manager-fullchain.crt
install -m 600 manager.key /etc/acd/deploy/certs/manager.key

Then run deploy. Pass --cacert /root/test-ca/test-ca.crt to curl, and import test-ca.crt into your browser, to connect without warnings.

Problems

5.5 - MaxMind GeoIP Databases

Overview

The manager looks up the location, network and anonymity of client addresses in MaxMind GeoIP databases, for geographic routing and the GeoIP API. The databases are not shipped with the CDN Manager: obtain them from your own MaxMind subscription.

Three databases are needed, as .mmdb files:

Provide all three or none. Without them, the CDN Manager installs and runs, but geographic routing is off.

Adding or Updating the Databases

1. Copy the three files into /etc/acd/deploy/maxmind/, replacing any older copies:

   cp GeoIP2-City.mmdb GeoLite2-ASN.mmdb GeoIP2-Anonymous-IP.mmdb /etc/acd/deploy/maxmind/
   ls -l /etc/acd/deploy/maxmind/
   

   Expected output:

   total 126236
   -rw-r--r--. 1 root root      4382 Oct 14 18:57 GeoIP2-Anonymous-IP.mmdb
   -rw-r--r--. 1 root root 121642973 Oct 14 18:57 GeoIP2-City.mmdb
   -rw-r--r--. 1 root root   7609502 Oct 14 18:57 GeoLite2-ASN.mmdb
   -rw-r--r--. 1 root root       400 Oct 14 18:57 README.txt
   

2. Run deploy:

   /mnt/esb3027/deploy
   

deploy loads the databases into the cluster after installing the CDN Manager, and loads them again only when they have changed. MaxMind publishes updates regularly: repeat these steps to apply them.

The databases are not included in the backups deploy takes. Keep your own copies.

Checking the Databases

When the databases are loaded, the final health check of deploy reports:

"msg": "Deployment gate PASSED for cdn-platform. GeoIP is serving."

To look up an address yourself:

curl 'https://manager.example.com/api/v1/routing/geoip?ip=8.8.8.8'; echo

Expected output (shortened):

{"city":{"continent":{"code":"NA",...},"country":{"iso_code":"US",...},...},"asn":{"autonomous_system_number":15169,"autonomous_system_organization":"GOOGLE"},"anon":{}}

See the Routing API for the full response.

Keeping the Files Elsewhere

To use files kept in another folder, set their paths in Site Settings instead of copying them:

maxmind_city_db: "/opt/maxmind/GeoIP2-City.mmdb"
maxmind_asn_db: "/opt/maxmind/GeoLite2-ASN.mmdb"
maxmind_anon_db: "/opt/maxmind/GeoIP2-Anonymous-IP.mmdb"

Problems

5.6 - Application Settings

Overview

The CDN Manager is a Helm release, cdn-platform, configured by Helm values. deploy builds those values from three layers, each overriding the one before it:

deploy merges them into /etc/acd/deploy/values.yaml on every run and installs the release with that one file.

Most sites need nothing beyond the preset and their site settings.

Presets

The preset is the complete starting configuration, chosen once with deploy init and remembered for every later run:

The preset is read from the ISO through the preset.yaml link, never copied. A new release’s ISO therefore brings its own defaults with it. This is also why the ISO must be mounted whenever you run deploy.

Overriding Settings

Write only the settings you want to change in /etc/acd/deploy/values.override.yaml, keeping the same nesting as in the preset. It starts empty. For example, to make the manager log in more detail while you investigate a problem:

cdn-manager:
  manager:
    logLevel: debug

Then run deploy. To undo it, remove the setting and run deploy again.

Warning: Set volume sizes before the first deploy. Helm cannot change the size of a persistent volume claim that already exists, so changing a size or persistence setting in values.override.yaml on an installed cluster does not resize the volume, and can make the upgrade fail. This applies to every component with storage, such as Kafka, VictoriaMetrics, PostgreSQL and Redis. Raising a retention setting also uses more disk, so check the volume has room first.

To find a setting, look it up in the preset on the ISO (/mnt/esb3027/values-production.yaml or /mnt/esb3027/values-lab.yaml), in the full reference /mnt/esb3027/values.yaml, or in the Configuration Guide.

Note: Some settings are derived and override anything in values.override.yaml: every manager hostname, Zitadel’s external domain, the TLS secret, and the list of CDN Directors whenever external_directors is not empty. Set those through manager_hostnames, external_directors and the certs/ folder instead.

The Generated values.yaml

/etc/acd/deploy/values.yaml holds the exact values the cluster received. Read it to see the result of the three layers.

Do not edit it: deploy rewrites it on every run, and stops with [ACD-VALUES-EDITED] if it was changed by hand. Move the change into values.override.yaml or group_vars/all.yaml, then delete values.yaml and run deploy again.

5.7 - Running deploy

Overview

deploy is on the installation ISO, at /mnt/esb3027/deploy. Run it as root on the first server in the inventory, with the ISO mounted at /mnt/esb3027. It reaches every other machine over SSH. For its commands, options, files and exit status, see the deploy reference.

To change what is installed, edit the files in /etc/acd/deploy and run deploy. Do not change the release with helm directly: the next deploy replaces it with what the configuration says.

What a Run Does

/mnt/esb3027/deploy

The first run installs; every later run re-applies your settings and changes only what differs. Running it again is safe, including after a failed run. deploy checks each machine after changing its network, and on a running cluster changes them one at a time, but a wrong network setting can still cut a machine off, and the run then stops with [ACD-NODE-UNREACHABLE].

Each run goes through fixed stages:

deploy backup runs only the first and third stages. --skip-tags preflight skips the second; use it only when you are sure a check is wrong for your site.

When deploy stops on a problem it recognizes, its message starts with a code such as [ACD-TLS-HOSTNAME]. See Installation Troubleshooting.

Backups

Important: These backups hold only the deployment’s configuration, each machine’s configuration and an etcd snapshot of the Kubernetes state. They do not include the data stored in the cluster: not the PostgreSQL databases, not Kafka’s messages, not the contents of any persistent volume, and not the metrics. Restoring one of these backups puts a machine’s configuration back; it cannot recover lost data. Back up the data separately: see Backup Procedures.

Every run of deploy takes a backup first, and deploy backup takes one without changing anything. Each backup is a folder under /var/lib/acd/backups on the first server: see the deploy reference for its files. deploy never deletes a backup: remove old folders yourself. Keep your own copies of the TLS certificate and key and the MaxMind databases, which the backup leaves out.

Important: Treat backups as secret material. A backup of a running cluster holds an etcd snapshot of the whole cluster, which contains every Kubernetes Secret: database passwords, the Zitadel master key and TLS keys among them. Every machine’s archive, servers’ and agents’, also holds the cluster token, and the servers’ hold the cluster-admin kubeconfig, /etc/rancher/k3s/k3s.yaml. Keep backups, and any copies you make, readable only by root, and delete the ones you no longer need.

The etcd snapshot is in two places, and deploy prunes neither:

• the backup folder under /var/lib/acd/backups on the first server, inside one server’s <machine>.tar.gz. Remove an old backup folder with:

  rm -r /var/lib/acd/backups/<folder>
  

• /var/lib/rancher/k3s/server/db/snapshots on that server, named acd-backup-<UTC time>-<machine>-<number>. List and delete old ones on that server with:

  k3s etcd-snapshot list
  k3s etcd-snapshot delete <name>
  

k3s also takes its own etcd snapshots, in the same folder on every server, named etcd-snapshot-<machine>-<time>: one every 12 hours, keeping the latest five, which k3s prunes itself. They hold every Kubernetes Secret too, so treat them, and any copy of them, as secret material.

The backup folder on the first server also holds a copy of /etc/acd/deploy, except the certs/ and maxmind/ folders and deploy.log, which it leaves out. Keep a copy of the latest backup folder off the first server, so that losing it does not lose the configuration deploy needs. Protect the copy as the warning above describes.

Before relying on a backup, check it:

cd /var/lib/acd/backups/<folder>
sha256sum -c SHA256SUMS

Restoring a Machine’s Files

Copy the machine’s archive and checksum file to it, then, as root:

tar --selinux --xattrs --acls -C / -xpzf <machine>.tar.gz
sha256sum -c <machine>.files.sha256

Keep the --selinux --xattrs --acls options: plain tar restores some k3s files with the wrong SELinux label. Then reboot the machine, or restart NetworkManager and k3s.

Rolling back the cluster itself also needs the etcd snapshot, restored on one server with the k3s cluster-reset procedure: stop k3s on every server, restore that server’s archive, run k3s server --cluster-reset --cluster-reset-restore-path=<snapshot path> on it, start k3s there, and rejoin the other servers. See the k3s etcd snapshot documentation for the full procedure.

What to Send Support

When the final health check fails, deploy writes /etc/acd/deploy/support-bundle-acd.tar.gz: send it to AgileTV support. When a run stops before the health check, send deploy.log, decision-record.json and the probe/ folder from /etc/acd/deploy instead.

5.8 - Changing the Cluster

Overview

The machines in the cluster are whatever inventory.ini lists. To change them, edit the inventory and run deploy. This page covers the changes that need more than that.

Before you start, check that the cluster is healthy: every node yes under READY and every admin target ok under HEALTH in acd-admin status.

Adding an Agent

1. Install the new machine as for the others: see Before You Begin. If you use a backplane, give it an address in backplane_network.

2. From the first server, set up SSH to it:

   ssh-copy-id root@192.0.2.14
   ssh -o BatchMode=yes root@192.0.2.14 hostname
   

3. Add it under [agent] in /etc/acd/deploy/inventory.ini:

   [agent]
   manager-4 ansible_host=192.0.2.14
   

4. Run deploy:

   /mnt/esb3027/deploy
   

deploy configures the new machine’s network, joins it to the cluster, and updates every other machine’s firewall to trust it.

Adding Servers

The number of servers must stay odd, so add servers two at a time, for example to go from one to three. Prepare both machines as for an agent, list both under [server], and run deploy once. deploy joins them one at a time.

Changing a Machine’s Role

k3s cannot turn a server into an agent, or an agent into a server, in place: the machine leaves the cluster and joins again in its new role. Because the number of servers must stay odd, a role change always moves two machines: two agents promoted to go from one server to three, two servers demoted to go from three to one, or a server and an agent swapped.

1. Take the first machine out of the cluster, whether it is a server or an agent. From any server, drain and remove it:

   acd-admin drain <machine>
   kubectl delete node <machine>
   

   The drain moves the machine’s pods elsewhere. It can take several minutes while storage moves, and reports what it is waiting for. If it does not finish, see When a Drain Does Not Finish. Removing the node needs kubectl: acd-admin has no command for it. Read Safety first.

   Then, on the machine itself, uninstall k3s: run /usr/local/bin/k3s-uninstall.sh if it was a server, or /usr/local/bin/k3s-agent-uninstall.sh if it was an agent.

2. Only when the first machine is out, do the same for the second. Taking two out at once can leave too few machines for the workloads.

3. List both machines under their new roles in inventory.ini, and run deploy.

With a single server, promoting two agents leaves the server alone to run every workload while both agents are out. Expect pods to stay Pending until the machines rejoin. The deploy that brings them back allows for this, and waits up to 10 minutes for the pods to start.

If a machine is listed under a role other than the one it runs, deploy stops with [ACD-INVENTORY-ROLE-CHANGED] before changing anything.

When a Drain Does Not Finish

acd-admin drain waits for every pod it moves, and has no timeout of its own. Its session ends after 12 hours; running drain again carries on where it stopped. Storage does not hold it up for long: each volume has replicas on other machines, and for the Kafka data volumes, which have a single replica, the storage layer (Longhorn) copies the replica to another machine before it lets the drain go on. While that happens, the drain names the pod whose eviction a disruption budget refuses for now, and keeps retrying.

If it still waits, read what it names:

• A pod that is terminating but not gone, with the finalizer holding it. On a machine that is not Ready, such a pod stays until the machine returns. Once the drain waits only on such pods, press Ctrl-C: the machine stays drained.
• A pod whose eviction is refused for a long time. Check acd-admin status for an admin target that is not ok, and kubectl -n longhorn-system get volumes.longhorn.io for a volume that is not healthy. A disruption budget refuses to evict a pod while its other pods are not Ready. Fix that, and the drain carries on.

See acd-admin drain.

Things You Cannot Change

The cluster is built around a few settings that cannot change once it is installed: backplane_network, cluster_pod_cidr, cluster_svc_cidr, the firewall zone and policy names, each machine’s node_ip and node_external_ip, and the machines’ hostnames. See Locked at First Install.

5.9 - Air-Gapped Installation

Overview

The installation ISO carries everything the CDN Manager needs, container images included, so an installation needs no internet access. What deploy does need is the operating system’s own packages: it installs a few with dnf on every machine: ansible-core on the first server, and on all of them rsync, tar, zstd, bash-completion, the storage tools (iscsi-initiator-utils and nfs-utils), and container-selinux, which the cluster’s SELinux policy needs.

Without internet access, give every machine a local package repository from the operating system’s installation media, then follow the Quickstart as written.

Prerequisites

Set Up a Local Repository on Every Machine

On each machine, mount the operating system ISO and add its two repositories. This example is for Red Hat Enterprise Linux 9; on a rebuild, use its ISO and the signing key file its own documentation names.

mkdir -p /mnt/os
mount -o loop,ro /root/rhel-9.4-x86_64-dvd.iso /mnt/os

cat > /etc/yum.repos.d/os-media.repo <<'REPO'
[os-media-baseos]
name=OS media - BaseOS
baseurl=file:///mnt/os/BaseOS
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

[os-media-appstream]
name=OS media - AppStream
baseurl=file:///mnt/os/AppStream
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
REPO

Disable any repository the machine cannot reach, or dnf fails while trying to read it:

dnf repolist
dnf config-manager --set-disabled <repository id>

Then check that dnf can find the packages:

dnf -q repoquery --latest-limit 1 ansible-core rsync tar zstd bash-completion iscsi-initiator-utils nfs-utils container-selinux

Each package should be listed once, whether or not it is already installed.

Important: Keep the local repository available on every machine whenever you run deploy, not only for the first installation. Every run installs or checks packages with dnf on each machine, and stops if it cannot reach them. Remount the operating system ISO after a reboot, as you do the CDN Manager ISO.

Install

Follow the Quickstart. Copy the MaxMind databases, your TLS certificate and its key, and both ISOs to the machines by whatever means your site allows.

If the first server cannot install ansible-core, deploy init stops with:

Error: could not install ansible-core. Ensure your OS repositories
or mounted install media are available to dnf (air-gap SOP).

Check the repository on that machine and run deploy init again.

5.10 - Upgrade Guide

Upgrading to 1.8 from an Earlier Release

Version 1.8 cannot be installed over 1.6 or any earlier release. The platform’s Helm charts were rearranged, and the cluster is now built and managed by deploy, so an earlier cluster cannot be upgraded in place.

To move to 1.8:

1. Install 1.8 on a new cluster, on new or reinstalled machines, with the Quickstart.
2. Set up its users and configuration. Contact AgileTV support to plan moving your existing configuration.
3. Point your CDN Directors at the new manager, then retire the old cluster.

Upgrading from 1.8 to a Later Release

From 1.8 on, an upgrade is a deploy run with the new release’s ISO: deploy upgrades k3s on each machine, one server at a time, then upgrades the cdn-platform release.

1. Read the Release Notes for the new release, for any step it needs before or after the upgrade.

2. Check that the cluster is healthy:

   acd-admin status
   

3. Replace the ISO at /mnt/esb3027 with the new one, on the first server:

   umount /mnt/esb3027
   mount -o loop,ro /root/esb3027-cdn-manager-<new version>.iso /mnt/esb3027
   

   The preset link in /etc/acd/deploy points into /mnt/esb3027, so mount the new ISO at the same path.

4. Run deploy init from the new ISO. It is safe to run again: it replaces the installed Ansible collections with the new ISO’s versions, installs ansible-core only if it is missing, keeps your preset, and leaves the files you have edited alone.

   /mnt/esb3027/deploy init
   

5. Run deploy:

   /mnt/esb3027/deploy
   

deploy takes a backup of the configuration before it changes anything, including an etcd snapshot of the cluster. That backup does not include the data stored in the cluster, such as the PostgreSQL databases: back that up first, as described in Backup Procedures. See Backups.

5.11 - Installation Troubleshooting

Overview

deploy checks the machines and your settings before it changes anything, and checks its own work as it goes. When it stops on a problem it recognizes, the message starts with a code in square brackets, such as [ACD-TLS-HOSTNAME], and names the machine, file or setting involved. The code stays the same when the wording changes, so search for it on this page, or quote it to support.

After fixing the cause, run deploy again. Re-running is safe: it picks up from the state the cluster is in. “Nothing was changed” in the table below means the run stopped before its first change.

Expected Messages

These appear in a successful run, and need no action:

Problems Without a Code

Failure Codes

Each code below gives the cause, then what to do.

[ACD-ANSIBLE-VERSION]

Cause: The installed ansible-core is older than 2.14.

What to do: Run dnf update ansible-core.

[ACD-HOST-PROBE-IN-DEPLOY]

Cause: A machine has a host_probe setting. It is an input for tests and for replaying a run, and deploy reads every machine itself. Nothing was changed.

What to do: Remove the host_probe setting from inventory.ini and host_vars.

[ACD-HOST-PROBE-SCHEMA]

Cause: A machine’s host_probe setting, used by tests and replays, has a format version this deploy cannot read. Nothing was changed.

What to do: Replace it with that machine’s probe/<machine>.json from a run of this release, or remove the host_probe setting.

[ACD-NO-DEFAULT-ROUTE]

Cause: A machine has no IPv4 default route, so its address cannot be chosen.

What to do: Give the machine a default route.

[ACD-BACKPLANE-AMBIGUOUS]

Cause: backplane_network matches more than one interface on a machine.

What to do: Narrow backplane_network, or set node_ip to the address on the backplane interface, on that machine’s line in inventory.ini.

[ACD-BACKPLANE-NO-MATCH]

Cause: backplane_network matches no interface on a machine.

What to do: Fix backplane_network, or set node_ip to the address on the backplane interface, on that machine’s line in inventory.ini. Every machine needs an address in backplane_network.

[ACD-NODE-ADDRESS-NOT-FOUND]

Cause: No interface on a machine holds the address chosen for it.

What to do: Check node_ip for that machine, or backplane_network.

[ACD-DEFAULT-ADDRESS-NOT-FOUND]

Cause: No interface on a machine holds its default route’s source address.

What to do: Give the default route a source address that one of the machine’s interfaces holds.

[ACD-NM-NOT-RUNNING]

Cause: NetworkManager is not running on a machine, so the deployment cannot set its interfaces’ zones or routing rules.

What to do: Start it with systemctl enable --now NetworkManager and run deploy again.

[ACD-NM-CONNECTION-MISSING]

Cause: An interface the deployment configures has no active NetworkManager connection.

What to do: Let NetworkManager manage it (nmcli device set <interface> managed yes), give it a connection (nmcli device connect <interface>, or nmcli connection add) and run deploy again.

[ACD-PARTIAL-RUN]

Cause: The run was limited to some machines, so it could not plan every machine’s network. What each machine trusts depends on all of them.

What to do: Run deploy without --limit.

[ACD-DIRECTOR-ADDRESS]

Cause: An entry in external_directors has no address, or one that is not an IPv4 address.

What to do: Give each CDN Director’s IPv4 address, such as 192.0.2.201.

[ACD-MAXMIND-AMBIGUOUS]

Cause: More than one file in maxmind/ matches the same MaxMind database.

What to do: Leave one file per database in maxmind/.

[ACD-MAXMIND-INCOMPLETE]

Cause: maxmind/ holds some, but not all three, MaxMind databases.

What to do: Provide all three databases, or none.

[ACD-TLS-COMBINED-FILE]

Cause: One file in certs/ holds both the certificate and the private key.

What to do: Split them into two PEM files in certs/.

[ACD-TLS-TOO-MANY]

Cause: certs/ holds more than one certificate or more than one key.

What to do: Leave exactly one certificate and one key.

[ACD-TLS-UNPAIRED]

Cause: certs/ holds a certificate without a key, or a key without a certificate.

What to do: Provide both, or neither for a temporary self-signed certificate.

[ACD-TLS-NOT-PEM]

Cause: certs/ holds files, but none is a readable PEM certificate or key.

What to do: Convert DER or PKCS#12 files to PEM.

[ACD-TLS-CERT-UNREADABLE]

Cause: The certificate file cannot be read as a PEM certificate.

What to do: Provide a PEM certificate.

[ACD-TLS-KEY-UNREADABLE]

Cause: The private key cannot be read: it is not PEM, or it has a passphrase.

What to do: Remove the passphrase with openssl pkey -in <key> -out <key>.plain.

[ACD-TLS-KEY-MISMATCH]

Cause: The certificate and the private key are not a pair.

What to do: Provide the key the certificate was requested with.

[ACD-TLS-CERT-EXPIRED]

Cause: The certificate has expired.

What to do: Replace it with a current certificate.

[ACD-TLS-HOSTNAME]

Cause: The certificate does not cover the first entry of manager_hostnames.

What to do: Reissue it for that name, or for a wildcard covering it.

[ACD-INVENTORY-NO-SERVER]

Cause: The inventory lists no server machine.

What to do: List at least one; the first server is the control machine.

[ACD-INVENTORY-STRAY-DIRECTOR]

Cause: A machine is listed under [director].

What to do: Leave the [director] section empty. List CDN Directors in external_directors in Site Settings.

[ACD-INVENTORY-SERVER-AND-AGENT]

Cause: A machine is listed under both server and agent.

What to do: List it under only one.

[ACD-INVENTORY-EVEN-SERVERS]

Cause: The inventory lists an even number of servers.

What to do: List an odd number of servers; move the rest under agent.

[ACD-INVENTORY-DUPLICATE-HOSTNAME]

Cause: Two machines share a hostname, ignoring case.

What to do: Give each machine a unique hostname.

[ACD-INVENTORY-ROLE-CHANGED]

Cause: A machine runs k3s as a server but is listed as an agent, or the other way round.

What to do: List it under its current role, or follow Changing a Machine’s Role.

[ACD-NODE-OS]

Cause: A machine does not run Enterprise Linux 9.4 or a later 9 release.

What to do: Update or reinstall it, or remove it from the inventory.

[ACD-NODE-KERNEL-MODULES]

Cause: A machine lacks kernel modules k3s and its storage need.

What to do: Install the kernel modules package for the running kernel.

[ACD-NODE-RESOURCES]

Cause: A machine has fewer processor cores or less memory than its role needs.

What to do: Add cores or memory, or make it an agent.

[ACD-VALUES-EDITED]

Cause: values.yaml was edited by hand.

What to do: Move the changes to values.override.yaml or group_vars/all.yaml, then delete values.yaml.

[ACD-CLUSTER-API-DOWN]

Cause: k3s is installed, but no server’s k3s API answers.

What to do: Get k3s answering on one server. If you are sure the check is wrong, run deploy --skip-tags preflight.

[ACD-CLUSTER-NODE-RENAMED]

Cause: A machine’s hostname changed after it joined the cluster.

What to do: Set the hostname back to the name in the message.

[ACD-CLUSTER-NODE-ADDRESS]

Cause: A machine’s cluster address differs from the one the settings choose.

What to do: Set the address settings back to what the cluster runs with; see Locked at First Install.

[ACD-CLUSTER-POD-CIDR]

Cause: A machine’s pod range lies outside cluster_pod_cidr.

What to do: Set cluster_pod_cidr back to what the cluster runs with.

[ACD-CLUSTER-SERVICE-CIDR]

Cause: The cluster’s service range differs from cluster_svc_cidr.

What to do: Set cluster_svc_cidr back to what the cluster runs with.

[ACD-HEALTH-NODE-NOT-READY]

Cause: A cluster machine is not Ready.

What to do: Check systemctl status k3s or k3s-agent on it, or remove it from the cluster.

[ACD-HEALTH-LIST-FAILED]

Cause: The databases or volumes could not be listed.

What to do: Check the cluster API answers with k3s kubectl get nodes.

[ACD-HEALTH-DATABASE]

Cause: A PostgreSQL database cluster is not healthy: it is still being built, is switching its primary, or more than one of its instances is not ready. One instance being rebuilt is accepted, and so is one instance per machine the run is bringing back after a role change.

What to do: Inspect it with the kubectl describe command in the message.

[ACD-HEALTH-VOLUME-FAULTED]

Cause: A storage volume has no replica with usable data, and not every replica is on a machine the run joins back.

What to do: Recover it from the Longhorn UI or a backup.

[ACD-CIDR-OVERLAP]

Cause: cluster_pod_cidr and cluster_svc_cidr overlap.

What to do: Choose two separate ranges.

[ACD-BACKPLANE-SPLIT]

Cause: The machines’ cluster addresses are not all on one network, so they could not reach each other. Either a machine’s address is outside backplane_network, or, without one, a machine using a second interface for cluster traffic has an address on a different network from the others. Nothing was changed.

What to do: The message names the machine and the network. Put every machine’s cluster address inside backplane_network (correct node_ip in inventory.ini if needed), or set backplane_network to the network the machines actually share.

[ACD-CIDR-IN-USE]

Cause: A machine already uses an address or route in cluster_pod_cidr or cluster_svc_cidr.

What to do: Choose ranges no machine uses.

[ACD-ROUTING-CONFLICT]

Cause: A routing rule sends cluster traffic away from the main routing table.

What to do: Follow the message: give the conflicting rules a priority number above 50, or with manage_routes: false, set it to true or add your own lookup main rules ahead of them.

[ACD-FIREWALL-SOURCE-CONFLICT]

Cause: Another firewalld zone on a machine already holds an address range that overlaps one deploy needs to trust: the pod or service network, backplane_network, or another machine’s address. firewalld allows a range in only one zone. Nothing was changed.

What to do: Run the commands the message shows: they remove the overlapping range from the other zone. If the zone it names is deploy’s own zone under an earlier name, set firewall_cluster_zone back instead: it is locked at first install. Or set manage_firewall: false.

[ACD-FIREWALL-NAME-INVALID]

Cause: firewall_cluster_zone or firewall_public_policy is the name of one of firewalld’s own zones or policies (such as public, trusted or allow-host-ipv6), is empty or not a string, is longer than firewalld allows (17 characters for the zone, 18 for the policy), has characters other than letters, digits, - and _, or both are the same. Nothing was changed.

What to do: Choose other names in group_vars/all.yaml.

[ACD-FIREWALL-NAME-IN-USE]

Cause: A machine already has a firewalld zone or policy named like firewall_cluster_zone or firewall_public_policy, and it holds rules deploy did not create. deploy would delete those rules, so it stops instead. Nothing was changed.

What to do: Choose names in group_vars/all.yaml that nothing on the machines uses, or empty that zone or policy yourself.

[ACD-FIREWALL-HANDOVER-STOPPED]

Cause: With manage_firewall: false, deploy needs to empty its own zone and policy, but firewalld is stopped. Rules saved in a stopped firewalld would come back when it starts. If this appears after the network changes, firewalld stopped during the run, and nothing was installed on the cluster.

What to do: Start firewalld (systemctl enable --now firewalld) and run deploy again. Or run the commands the message shows, then run deploy again.

[ACD-FIREWALL-CLOSED]

Cause: With manage_firewall: false, a machine’s firewall blocks something the cluster needs: a required port is closed, or the pod or service network is not in a zone that accepts all traffic. Nothing was changed.

What to do: Run the firewall-cmd commands the message shows. See Managing the Firewall Yourself. Or set manage_firewall: true.

[ACD-BACKUP-FOLDER-CREATE]

Cause: The folder for this run’s backup could not be created.

What to do: Check that the backup folder’s parent exists and is writable, or set backup_dir in group_vars/all.yaml.

[ACD-BACKUP-FOLDER-EXISTS]

Cause: Another run started in the same second and owns the backup folder.

What to do: Run deploy again; nothing was changed.

[ACD-BACKUP-SNAPSHOT]

Cause: The etcd snapshot for the backup was not found, or more than one was.

What to do: Check the snapshot folder and the k3s log (journalctl -u k3s) on the machine the message names, then run deploy again.

[ACD-NM-PENDING-EDITS]

Cause: A network connection deploy needs to change has edits saved in NetworkManager that are not active yet. Applying deploy’s change would activate them too. Nothing was changed on that machine.

What to do: Check the pending edits with nmcli connection show <uuid>. Either undo them with nmcli connection modify, or activate them in a maintenance window with nmcli connection up <uuid>. Then run deploy again.

[ACD-CLUSTER-ZONE-MISSING]

Cause: A machine already running k3s has no acd-cluster zone or no acd-public policy (or the names you set). Creating them needs a firewalld reload, which would remove the rules k3s depends on, so deploy will not do it on a running machine. Nothing was changed.

What to do: If you changed firewall_cluster_zone or firewall_public_policy, set them back: they are locked at first install. Otherwise, in a maintenance window, one machine at a time, create them (firewall-cmd --permanent --new-zone=acd-cluster, --zone=acd-cluster --set-target=ACCEPT, --new-policy=acd-public, --policy=acd-public --add-ingress-zone=public --add-egress-zone=HOST), run firewall-cmd --reload, and restart k3s. Then run deploy again.

[ACD-NODE-UNREACHABLE]

Cause: A machine stopped answering over SSH while its host network changed, one machine at a time on a running cluster, so the machines after it were left as they were. The change itself may have cut it off.

What to do: Check that machine’s network from its console, then run deploy again.

[ACD-VERIFY-ZONE]

Cause: After changing a machine’s network, an interface is not in the firewalld zone deploy planned for it. Nothing was installed on the cluster.

What to do: Run deploy again. If the message comes back, send it to support.

[ACD-VERIFY-FIREWALL]

Cause: After changing a machine’s network, its firewall still blocks something the cluster needs, or another zone holds an address range deploy needs to trust. Nothing was installed on the cluster.

What to do: Run the firewall-cmd commands the message shows, then run deploy again. If the message comes back, send it to support.

[ACD-VERIFY-LEFTOVER]

Cause: After changing a machine’s network, deploy’s own zone or policy holds a rule it did not create, or is missing one it planned. Nothing was installed on the cluster.

What to do: Run deploy again, which removes the extra rule. Keep your own rules in other zones. If the message names a zone target or priority, run firewall-cmd --reload in a maintenance window and restart k3s (or k3s-agent) on that machine, then run deploy again. If it comes back otherwise, send it to support.

[ACD-VERIFY-ROUTING]

Cause: After changing a machine’s network, its routing rules are not what deploy planned: a cluster rule is missing, a removed rule is still active, a rule was added by hand with ip rule add, or cluster traffic uses a routing table other than main. Nothing was installed on the cluster.

What to do: Run the ip rule del commands the message shows, check with ip rule show, then run deploy again. If the message comes back, send it to support.

[ACD-VERIFY-SYSCTL]

Cause: After the network changes, a kernel network setting on a machine has another value than the deployment set.

What to do: Find what sets it (a file under /etc/sysctl.d, or a tuned profile), then run deploy again. No k3s step ran.

[ACD-JOIN-UNREACHABLE]

Cause: A joining machine cannot reach the server it joins through.

What to do: Check the firewall on both machines, that their node_ip addresses are on one network, and that reverse path filtering (net.ipv4.conf.all.rp_filter) does not drop the replies; see the message.

[ACD-GATE-DATABASE-UNHEALTHY]

Cause: After the install, a PostgreSQL database cluster did not become healthy in time. This message is followed by [ACD-GATE-FAILED], which writes the support bundle.

What to do: Wait until kubectl get clusters.postgresql.cnpg.io -A shows Cluster in healthy state, then run deploy again.

[ACD-GATE-UNHEALTHY-PODS]

Cause: After the install, pods in a critical namespace (default, kube-system or longhorn-system) were still not running after 10 minutes. The message lists each such pod with its namespace. It is followed by [ACD-GATE-FAILED], which writes the support bundle.

What to do: On a server, find the cause, fix it, then run deploy again. How depends on the pod’s namespace:

• default: the platform’s own pods, which acd-admin covers. Most pods belong to an admin target, whose pods acd-admin status <target> lists with their last termination and recent warnings. Find the pod’s target, then read acd-admin logs <target>. For a pod that belongs to no target, such as the Kafka operator’s, use the kubectl commands in the next item.
• kube-system and longhorn-system: the cluster’s own components and the storage layer, which acd-admin does not cover. Use the commands in the message: kubectl describe pod -n <namespace> <pod> and kubectl logs -n <namespace> <pod>.

[ACD-GATE-RELEASE-STATUS]

Cause: After the install, the Helm release is not deployed. This message is followed by [ACD-GATE-FAILED], which writes the support bundle.

What to do: Find the cause with the helm status command in the message and acd-admin status, fix it, then run deploy again.

[ACD-GATE-FAILED]

Cause: The final health check failed. It follows the message that names the failing check.

What to do: Read that earlier message, and send /etc/acd/deploy/support-bundle-acd.tar.gz to support.

Getting Help

When the final health check fails ([ACD-GATE-FAILED]), deploy writes /etc/acd/deploy/support-bundle-acd.tar.gz, with the run’s log, its decision record and what it read from each machine. Send it to AgileTV support.

When a run stops earlier, send support these files from /etc/acd/deploy instead:

• deploy.log
• decision-record.json
• the probe/ folder

5.12 - Next Steps

Overview

With the cluster installed, finish these tasks before you let users on to the system. They are done in the web interfaces, not with deploy.

Every interface is reached under the main manager hostname, the first entry of manager_hostnames. Signing in under any other name, a machine’s IP address included, fails.

Step 1: Sign In to the Zitadel Console

Zitadel is the manager’s identity service: every user account and role lives there.

1. Open https://<manager hostname>/ui/console.
2. Sign in with the default administrator: login name admin@zitadel.<manager hostname>, password Password1!. The console also accepts the account’s email address, admin@agiletv.dev.
3. If you are asked to set up multi-factor authentication (MFA), skip it. MFA is not supported in this release, and setting it up can lock you out of the account.

Step 2: Replace the Default Administrator

The default credentials are published, so replace them before the system is reachable by anyone else.

1. Create your own administrator account, as in Step 4, and give it the administrator roles. See Zitadel’s administrator documentation for the roles.
2. Sign out, and sign in with the new account to check that it works.
3. Disable or delete the default admin account.

Step 3: Configure SMTP (Recommended)

Zitadel sends email to verify addresses and reset passwords. Without an SMTP server, those features do not work.

1. In the Zitadel Console, open Settings > Default Settings.
2. Enter your mail server’s host, port (typically 587 or 465), user name and password, and the sender address, such as noreply@example.com.
3. Save.

Step 4: Add Users

1. In the Zitadel Console, open Users > Add User.

2. Enter a username, first and last name, and email address.

   Known issue: The username must match the local part of the email address, the part before the @. For foo@example.com, the username must be foo. Otherwise Zitadel can block sign-in with the full email address.

3. Without SMTP, tick Email Verified and set an initial password. Otherwise the account is not enabled.

4. Select Create User, then give the user their username, temporary password, and the console address. They are asked to change the password at first sign-in.

5. Assign the user’s roles. For Grafana’s roles, see Grafana Authentication.

Users sign in to the GUI with their login name, <username>@zitadel.<manager hostname>, not their email address. For more on managing users, see Zitadel’s user management documentation.

Step 5: Open Grafana

Open https://<manager hostname>/grafana/ and select Sign in with Zitadel. The dashboards are installed with it. See Metrics and Monitoring.

Step 6: Connect Your CDN Directors

Each CDN Director in external_directors is configured through the manager at https://<manager hostname>/confd/<name>/. Each one is known to Grafana as <name>.external, for example director-1.external. Connecting a Director to the manager is described in the CDN Director documentation.

Make sure your network’s firewall lets the Directors reach Kafka on every machine, and nothing else does: see Kafka Ports.

Step 7: Record Your Installation

Keep a record, off the cluster, of:

• The Zitadel master key, from Step 11 of the Quickstart.
• Your TLS certificate and key, and its expiry date.
• Your MaxMind databases.
• A copy of /etc/acd/deploy.
• The machines, their addresses, and the manager hostnames.

Further Reading

• Configuration Guide: every application setting.
• Operations Guide: day-to-day procedures.
• Performance Tuning Guide: topology labels, resources and database tuning.
• API Guide: the REST API.

6 - Configuration Guide

Overview

The CDN Manager is the Helm release cdn-platform, configured by Helm values. The release is installed and updated only by deploy, which builds the values from three layers:

1. The preset on the installation ISO: values-production.yaml or values-lab.yaml, chosen with deploy init.
2. Your overrides in /etc/acd/deploy/values.override.yaml: only the settings you want to change.
3. Derived settings worked out from your site settings in /etc/acd/deploy/group_vars/all.yaml and the certs/ folder. These always win.

deploy merges them into /etc/acd/deploy/values.yaml, which is what the cluster receives. See Application Settings.

This guide is the reference for the Helm values.

Applying Configuration

To change a value:

1. Write the setting in /etc/acd/deploy/values.override.yaml, with the same nesting as below. For example:

   cdn-manager:
     manager:
       logLevel: debug
   

2. Run deploy:

   /mnt/esb3027/deploy
   

Do not run helm install, helm upgrade or helm rollback yourself: the next deploy replaces the release with what the configuration says. To undo a change, revert it in values.override.yaml and run deploy again.

helm list -A shows six more releases, which the cluster itself runs on. They come from the installation ISO and k3s, not from your values, so leave them alone too:

Set through site settings, not values: the manager hostnames (global.hosts.manager and Zitadel’s ExternalDomain), the CDN Directors (global.hosts.routers, when external_directors is not empty) and the TLS certificate secret are derived by deploy, and override anything in values.override.yaml. Set them with manager_hostnames, external_directors and the certs/ folder.

Reference Files

The installation ISO provides the presets and a complete reference at /mnt/esb3027/:

Hardware requirements: For per-node hardware specifications, refer to the System Requirements Guide. The System Requirements Guide is the authoritative source — the hardware comments in the header of values-production.yaml may not reflect the current requirements.

Configuration Reference

Global Settings

The global section contains cluster-wide settings. The most critical configuration is global.hosts.

global:
  hosts:
    manager:
      - host: manager.local
    routers:
      - name: default
        address: 127.0.0.1
    edns_proxy: []
    geoip: []

Important: deploy sets global.hosts.manager, Zitadel’s ExternalDomain and global.hosts.routers from manager_hostnames and external_directors in Site Settings. Do not set them in values.override.yaml.

Manager Configuration

Core Manager API server settings:

Manager Resources

The chart supports both resource presets and explicit resource specifications:

Note: For production workloads, explicitly set cdn-manager.manager.resources rather than using presets.

Manager Datastore

cdn-manager:
  manager:
    datastore:
      type: redis
      namespace: "cdn_manager_ds"
      default_ttl: ""
      compression: zstd

Manager Database

The manager keeps its own data, including the subnet map, in a manager database in the PostgreSQL cluster. The manager creates the database if it is absent.

cdn-manager:
  manager:
    database:
      name: manager
      host: acd-cluster-postgresql-rw
      port: 5432
      user: app
      provision:
        enabled: true

Manager Discovery

cdn-manager:
  manager:
    discovery: []
    # Example:
    # - namespace: "other"
    #   hosts:
    #     - other-host1
    #     - other-host2
    #   pattern: "other-.*"

Manager Tuning

cdn-manager:
  manager:
    tuning:
      enable_cache_control: true
      cache_control_max_age: "5m"
      cache_control_miss_max_age: ""
      subnet:
        mutation_lock:
          wait: ""
      postgres:
        retry_min: ""
        retry_max: ""
        acquire_timeout: ""
        pool_size: ""

Manager Container Arguments

cdn-manager:
  manager:
    args:
      - --config-file=/etc/manager/config.toml
      - http-server

Gateway Configuration

NGinx Gateway settings:

MIB Frontend Configuration

Web-based configuration GUI settings:

Confd Configuration

Confd settings for configuration management:

VictoriaMetrics Configuration

Time-series database for metrics:

Ingress Configuration

Traffic exposure settings:

Ingress Extra Paths

The chart includes default extra paths for Confd and GeoIP:

cdn-manager:
  ingress:
    extraPaths:
      - path: /confd
        pathType: Prefix
        backend:
          service:
            name: cdn-platform-cdn-manager-gateway
            port:
              name: http
      - path: /geoip
        pathType: Prefix
        backend:
          service:
            name: cdn-platform-cdn-manager-gateway
            port:
              name: http

TLS Certificate Secrets

Note: With deploy, put the certificate and key in /etc/acd/deploy/certs/ instead: deploy creates the secret and wires it to every ingress. See TLS Certificates. The settings below are for reference.

For production TLS certificates:

cdn-manager:
  ingress:
    secrets:
      - name: manager.local-tls
        key: |-
          -----BEGIN RSA PRIVATE KEY-----
          ...
          -----END RSA PRIVATE KEY-----
        certificate: |-
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
    tls: true

Resource Configuration

Resource Presets

Predefined resource configurations for common deployment sizes:

Note: Limits are calculated as requests plus 50% (except for xlarge/2xlarge and ephemeral-storage).

Custom Resources

Override preset with custom values:

cdn-manager:
  manager:
    resources:
      requests:
        cpu: "300m"
        memory: "512Mi"
      limits:
        cpu: "1"
        memory: "1Gi"

Note:

• CPU values use millicores (1000m = 1 core)
• Memory values use binary SI units (1024Mi = 1GiB)
• Requests represent minimum guaranteed resources
• Limits represent maximum consumable resources

Capacity Planning

When sizing resources:

• Requests determine scheduling (node must have available capacity)
• Limits prevent resource starvation
• Maintain 20-30% cluster headroom for scaling
• Total capacity = sum of all requests × replica count + headroom

Security Contexts

Pod Security Context

cdn-manager:
  manager:
    podSecurityContext:
      enabled: true
      fsGroup: 1001
      fsGroupChangePolicy: Always
      sysctls: []
      supplementalGroups: []

Container Security Context

cdn-manager:
  manager:
    containerSecurityContext:
      enabled: true
      runAsUser: 1001
      runAsGroup: 1001
      runAsNonRoot: true
      readOnlyRootFilesystem: true
      privileged: false
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: "RuntimeDefault"

Health Probes

Probe Types

Default Probe Configuration

Liveness Probe

cdn-manager:
  manager:
    livenessProbe:
      enabled: true
      initialDelaySeconds: 5
      periodSeconds: 30
      timeoutSeconds: 10
      failureThreshold: 5
      successThreshold: 1
      httpGet:
        path: /api/v1/health/alive
        port: http

Readiness Probe

cdn-manager:
  manager:
    readinessProbe:
      enabled: true
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds: 7
      failureThreshold: 3
      successThreshold: 1
      httpGet:
        path: /api/v1/health/ready
        port: http

Startup Probe

cdn-manager:
  manager:
    startupProbe:
      enabled: true
      initialDelaySeconds: 0
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 10
      successThreshold: 1
      httpGet:
        path: /api/v1/health/alive
        port: http

Autoscaling Configuration

Horizontal Pod Autoscaler (HPA)

cdn-manager:
  manager:
    autoscaling:
      hpa:
        enabled: true
        minReplicas: 3
        maxReplicas: 8
        targetCPU: 50
        targetMemory: 80

Network Policy

cdn-manager:
  networkPolicy:
    enabled: true
    allowExternal: true
    allowExternalEgress: true
    addExternalClientAccess: true

Pod Affinity and Anti-Affinity

cdn-manager:
  manager:
    podAffinityPreset: ""
    podAntiAffinityPreset: soft
    nodeAffinityPreset:
      type: ""
      key: ""
      values: []
    affinity: {}

Service Configuration

cdn-manager:
  service:
    type: ClusterIP
    ports:
      http: 80
    annotations:
      service.kubernetes.io/topology-mode: Auto
    externalTrafficPolicy: Cluster
    sessionAffinity: None

Persistence Configuration

Warning: Set volume sizes before the first deploy. Helm cannot change the size of a persistent volume claim that already exists, so changing a size or persistence setting in values.override.yaml on an installed cluster does not resize the volume, and can make the upgrade fail. This applies to every component with storage, such as Kafka, VictoriaMetrics, PostgreSQL and Redis. Raising a retention setting also uses more disk, so check the volume has room first.

cdn-manager:
  persistence:
    enabled: false
    mountPath: /agiletv/manager/data
    storageClass: ""
    accessModes:
      - ReadWriteOnce
    size: 8Gi

RBAC and Service Account

cdn-manager:
  rbac:
    create: false
    rules: []

  serviceAccount:
    create: true
    name: ""
    automountServiceAccountToken: true
    annotations: {}

Metrics

cdn-manager:
  metrics:
    enabled: false
    serviceMonitor:
      enabled: false
      namespace: ""
      annotations: {}
      labels: {}
      interval: ""
      scrapeTimeout: ""

Next Steps

After configuration:

1. Installation Guide - Deploy with your configuration
2. Operations Guide - Day-to-day management
3. Performance Tuning Guide - Optimize system performance
4. Architecture Guide - Understand component relationships

7 - Performance Tuning Guide

Overview

This guide provides performance tuning recommendations for the AgileTV CDN Manager (ESB3027). While the default configuration is suitable for most deployments, certain environments may benefit from additional optimizations.

Network Topology Optimization

Topology Aware Hints

The CDN Manager uses Kubernetes Topology Aware Hints to prefer routing pods in the same zone as the source of network traffic. This reduces cross-zone latency and improves overall system responsiveness.

How It Works

When nodes are labeled with topology zones, Kubernetes automatically routes traffic to pods in the same zone when possible. This is particularly beneficial for:

• Low-latency requirements: Keeps traffic local to reduce round-trip time
• Cost optimization: Reduces cross-zone data transfer costs in cloud environments
• Load distribution: Prevents hotspots by distributing load across zones

Configuring Availability Zones

Each node must have zone and region labels applied for Topology Aware Hints to function. Set them with node_labels on each machine’s line in /etc/acd/deploy/inventory.ini, then run deploy, which applies them:

[server]
manager-1 ansible_host=192.0.2.11 node_labels='{"topology.kubernetes.io/zone": "us-east-1a", "topology.kubernetes.io/region": "us-east-1"}'

Adjust the zone and region values to match your deployment geography. Labels set this way are reapplied on every deploy, so they are kept with the rest of your configuration. See Inventory.

Verify Topology Configuration

Verify labels are applied:

kubectl get nodes --show-labels | grep topology.kubernetes.io

Verify EndpointSlices are being generated with hints:

kubectl get endpointslices

Requirements for Topology Aware Hints

For Topology Aware Hints to activate:

• Minimum Nodes: At least one node must be labeled with each zone referenced by endpoints
• Symmetry: The control plane checks for sufficient CPU capacity across zones to balance traffic
• Zone Coverage: All zones with endpoints should have at least one ready node

Integration with Pod Anti-Affinity

Topology labels complement the pod anti-affinity rules already configured in the Helm chart:

• Pod Anti-Affinity: Handles pod-to-node placement to ensure high availability
• Topology Aware Hints: Handles service-to-pod traffic routing to keep requests within the same zone

Together, these features optimize both placement and routing for improved performance.

Fallback Behavior

If zone labels are not configured, the system falls back to random load-balancing across all available pods. This is functionally correct but may result in:

• Increased cross-zone traffic
• Higher latency for some requests
• Less predictable performance characteristics

Kernel Network Tuning (sysctl)

deploy tunes the kernel’s network settings on every machine for you, for high connection rates and large numbers of concurrent connections. There is nothing to configure: it applies these values on every run, whatever the firewall settings:

Connection tracking (net.netfilter.nf_conntrack_*) is not in the list: k3s sets those values itself every time it starts, so values set elsewhere do not last.

Important: Do not set these parameters yourself, in /etc/sysctl.d/, a tuned profile or anywhere else. After changing a machine’s network, deploy checks that each one holds the value above, and stops with [ACD-VERIFY-SYSCTL] before installing anything if another value is in place. If an earlier installation left a file such as /etc/sysctl.d/99-cdn-manager.conf, remove these parameters from it.

To check the values on a machine:

sysctl net.core.somaxconn net.core.netdev_max_backlog net.ipv4.tcp_max_syn_backlog \
    net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse net.core.rmem_max net.core.wmem_max

Monitoring Impact

After applying these settings, monitor:

• Connection establishment rates
• TIME_WAIT socket count: netstat -n | grep TIME_WAIT | wc -l
• Connection tracking table usage: cat /proc/sys/net/netfilter/nf_conntrack_count
• Network buffer utilization via Grafana dashboards

Resource Configuration

Horizontal Pod Autoscaler (HPA)

The default HPA configuration is tuned for production workloads. For environments with variable load, consider adjusting the scale metrics:

For detailed HPA configuration, see the Architecture Guide.

Resource Requests and Limits

Ensure resource requests and limits are appropriately sized for your workload. Under-provisioned resources can cause:

• Pod evictions during high load
• Increased latency due to CPU throttling
• Slow scaling responses

Refer to the Configuration Guide for preset configurations and planning guidance.

Database Optimization

PostgreSQL

The PostgreSQL cluster is managed by the Cloudnative PG operator. For improved performance:

• Connection Pooling: The application uses connection pooling by default
• Replica Usage: Read queries can be offloaded to replicas for read-heavy workloads
• Backup Scheduling: Schedule backups during low-traffic periods to minimize I/O impact

Redis

Redis provides in-memory caching for sessions and ephemeral state:

• Memory Allocation: Ensure sufficient memory for cache hit rates
• Persistence: RDB snapshots are enabled; adjust frequency based on durability needs

Kafka

Kafka handles event streaming for selection input and metrics:

• Partition Count: Default partitions are sized for typical workloads
• Replication Factor: Production deployments use 3 replicas for fault tolerance
• Consumer Groups: The Selection Input Worker is limited to one consumer per partition

Monitoring Performance

Key Metrics to Watch

Monitor the following metrics for performance insights:

• API Response Time: Track via Grafana dashboards
• Pod CPU/Memory Usage: Identify resource bottlenecks
• Kafka Lag: Monitor consumer lag for selection input processing
• Database Connections: Watch for connection pool exhaustion

Grafana Dashboards

Pre-built dashboards are available at https://<manager-host>/grafana:

• System Health: Overall cluster and application health
• CDN Metrics: Routing and usage statistics
• Resource Utilization: CPU, memory, and network usage per component

Troubleshooting Performance Issues

High Latency

1. Check pod distribution across nodes: kubectl get pods -o wide
2. Verify topology labels are applied: kubectl get nodes --show-labels
3. Review network latency between nodes
4. Check for resource contention: kubectl top pods

Slow Scaling

1. Verify HPA is enabled: kubectl get hpa
2. Check cluster capacity for scheduling new pods
3. Review HPA metrics: kubectl describe hpa cdn-platform-cdn-manager

Database Performance

1. Check PostgreSQL cluster status: kubectl get pods -l app=postgresql
2. Review slow query logs (if enabled)
3. Monitor connection pool usage

Next Steps

After reviewing performance tuning:

1. Architecture Guide - Understand component interactions
2. Configuration Guide - Detailed configuration options
3. Metrics & Monitoring Guide - Comprehensive monitoring setup
4. Troubleshooting Guide - Resolve performance issues

8 - Operations Guide

Overview

This guide covers day-to-day operational procedures for managing the AgileTV CDN Manager (ESB3027). Topics include routine maintenance, backup procedures, log management, and common operational tasks.

Prerequisites

Before performing operations, ensure you have:

• kubectl access to the cluster (available as root on every server)
• Access to the first server, where deploy and its configuration in /etc/acd/deploy live
• Appropriate RBAC permissions for administrative tasks

Cluster Access

There are two supported methods for accessing the Kubernetes cluster:

1. SSH to a Server Node (Recommended for operations staff) - SSH into any Server node and run kubectl commands directly
2. Remote kubectl - Install kubectl on your local machine and configure it to connect to the cluster remotely

Method 1: SSH to Server Node (Recommended)

The kubectl command-line tool is pre-configured on all Server nodes and can be used directly without additional setup:

# SSH to any Server node
ssh root@<server-ip>

# Run kubectl commands directly
kubectl get nodes
kubectl get pods

This method is recommended for day-to-day operations as it requires no local configuration and provides direct access to the cluster.

For routine status checks, logs, restarts and node maintenance, acd-admin on a server does the same by admin target name, without kubectl.

Method 2: Remote kubectl from Local Machine

To use kubectl from your local workstation or laptop:

Step 1: Install kubectl

Download and install kubectl for your operating system:

• Official Documentation: Install kubectl
• macOS (Homebrew): brew install kubectl
• Linux: Download from the official Kubernetes release page
• Windows: Download from the official Kubernetes release page

Step 2: Copy kubeconfig from Server Node

# Copy kubeconfig from any Server node
scp root@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Step 3: Update kubeconfig

Edit the kubeconfig file to point to the correct server address:

# Replace localhost with the actual server IP
# macOS/Linux:
sed -i '' 's/127.0.0.1/<server-ip>/g' ~/.kube/config  # macOS
sed -i 's/127.0.0.1/<server-ip>/g' ~/.kube/config    # Linux

# Or manually edit ~/.kube/config and change:
# server: https://127.0.0.1:6443
# to:
# server: https://<server-ip>:6443

Step 4: Verify connectivity

kubectl get nodes

Managing Multiple Clusters

If you manage multiple Kubernetes clusters from the same machine, you can maintain multiple kubeconfig files:

# Set KUBECONFIG environment variable to include multiple config files
export KUBECONFIG=~/.kube/config-prod:~/.kube/config-lab

# View all contexts
kubectl config get-contexts

# Switch between clusters
kubectl config use-context <context-name>

# View current context
kubectl config current-context

For more information, see the official Kubernetes documentation: Organizing Cluster Access

Helm Commands

Helm releases are managed cluster-wide. Besides cdn-platform, the cluster runs six releases of its own: see Configuration.

# List all releases
helm list -A

# View release history
helm history cdn-platform

# Get deployed values
helm get values cdn-platform -o yaml

# Get deployed manifest
helm get manifest cdn-platform