AgileTV CDN Manager (esb3027)

• 1: Getting Started
• 2: System Requirements Guide
• 3: Networking Guide
• 3.1: Shared Interface Network Setup
• 3.2: Configuring Segregated Networks
• 4: Architecture Guide
• 5: Installation Guide
• 5.1: Installation Checklist
• 5.2: Single-Node Installation
• 5.3: Multi-Node Installation
• 5.4: Air-Gapped Deployment
• 5.5: Helm Chart Installation
• 5.6: Upgrade Guide
• 5.7: Next Steps
• 6: Configuration Guide
• 7: Performance Tuning Guide
• 8: Operations Guide
• 9: Metrics & Monitoring Guide
• 9.1: Grafana Authentication & Roles
• 9.2: Grafana Dashboards
• 9.3: Alerts & Alarms
• 10: API Guide
• 10.1: Authentication API
• 10.2: Health API
• 10.3: Selection Input API
• 10.4: Data Store API
• 10.5: Subnets API
• 10.6: Routing API
• 10.7: Discovery API
• 10.8: Metrics API
• 10.9: Configuration API
• 10.10: Operator UI API
• 10.11: OpenAPI Specification
• 11: Troubleshooting Guide
• 12: Glossary

1 - Getting Started

Overview

The AgileTV CDN Manager (product code ESB3027) is a cloud-native control plane for managing CDN deployments. It provides centralized orchestration for authentication, configuration, routing, and metrics collection across CDN infrastructure.

Before You Start:

• Deployment type: Lab (single-node) or Production (multi-node)? See Installation Guide
• Hardware: Nodes meeting specifications for your deployment type
• OS: RHEL 9 or compatible clone (Oracle Linux, AlmaLinux, Rocky Linux)
• Software: Installation ISO from AgileTV customer portal; Extras ISO for air-gapped
• Network: Firewall ports configured per Networking Guide

Deployment Models

Functionality remains consistent across deployment models.

Prerequisites

• Installation ISO: Obtain esb3027-acd-manager-X.Y.Z.iso from AgileTV customer portal
• Extras ISO (air-gapped): Obtain esb3027-acd-manager-extras-X.Y.Z.iso for offline installations
• OS: RHEL 9 or compatible clone (Oracle Linux, AlmaLinux, Rocky Linux)
• Kubernetes familiarity: Basic understanding of pods, deployments, and Helm charts

For detailed hardware, network, and operating system requirements, see the System Requirements Guide.

Installation

Ready to install? The Installation Guide provides step-by-step procedures for both lab and production deployments:

• Lab/Single-Node: Quick deployment for testing and demonstrations
• Production/Multi-Node: High-availability cluster with 3+ nodes

See the Installation Guide to get started.

Accessing the System

Following successful deployment, the following interfaces are available:

All services are accessed via https://<cluster-host><path>.

Note: A self-signed SSL certificate is deployed by default. When accessing services through a browser, you will need to accept the self-signed certificate warning. For production deployments, configure a valid SSL certificate before exposing the system to users.

Initial user configuration is performed through Zitadel. Refer to the Configuration Guide for authentication setup procedures. For detailed guidance on managing users, roles, and permissions in the Zitadel Console, see Zitadel’s User Management Documentation.

Documentation Navigation

The following guides provide detailed information for specific operational tasks:

2 - System Requirements Guide

Overview

This document specifies the hardware, operating system, and networking requirements for deploying the AgileTV CDN Manager (ESB3027). Requirements vary based on deployment type and node role within the cluster.

Cluster Sizing

Production Deployments

Production deployments require a minimum of three nodes to achieve high availability. The cluster architecture employs distinct node roles:

Server nodes can be deployed in either Control Plane Only or Combined role configurations. The choice depends on your deployment requirements:

• Control Plane Only: Dedicated control plane nodes with lower resource requirements; requires separate Agent nodes for workloads
• Combined: Server nodes run both control plane and workloads; minimum 3 nodes required for HA

Why Use Control Plane Only Nodes?

Dedicated Control Plane Only nodes provide several benefits for larger deployments:

• Resource Isolation: Control plane components (etcd, API server, scheduler) run on dedicated hardware without competing with application workloads for CPU and memory
• Stability: Application workload spikes or misbehaving pods cannot impact control plane performance
• Security: Smaller attack surface on control plane nodes; fewer containers and services running
• Predictable Performance: Control plane responsiveness remains consistent regardless of application load
• Flexible Sizing: Control Plane Only nodes can use lower-specification hardware (2 cores, 4 GiB) since they don’t run application workloads

For most small to medium deployments, Combined role servers are simpler and more cost-effective. Control Plane Only nodes are recommended for larger deployments with significant workload requirements or where control plane stability is critical.

High Availability Considerations

Production deployments require 3 nodes running control plane (etcd) and 3 nodes capable of running workloads. These can be the same nodes (Combined role) or separate nodes (CP-Only + Agent).

Node Role Combinations:

Any combination works as long as you have 3 control plane nodes and 3 workload-capable nodes.

Note: Regardless of the deployment configuration, a minimum of 3 nodes capable of running workloads is required for production deployments. This ensures both high availability and sufficient capacity for application pods.

For detailed fault tolerance information and data replication strategies, see the Architecture Guide.

Hardware Requirements

Single-Node Lab Deployment

Lab deployments are intended for acceptance testing, demonstrations, and development only. These configurations are not suitable for production workloads.

Production Cluster - Server Node (Control Plane Only)

Server nodes dedicated to control plane functions have modest resource requirements:

These nodes run only control plane components and require separate Agent nodes to run application workloads.

Production Cluster - Server Node (Control Plane + Workloads)

Combined role nodes require resources for both control plane and application workloads:

Production Cluster - Agent Node

Agent nodes execute application workloads and require the following resources:

Storage Notes

* Disk Space: All disk space values must be available in the /var/lib/longhorn partition. It is recommended that /var/lib/longhorn be a separate partition on a fast SSD for optimal performance, though SSD is not strictly required.

Longhorn Capacity: Longhorn storage requires an additional 30% capacity headroom for internal operations and scaling. If less than 30% of the total partition capacity is available, Longhorn may mark volumes as “full” and prevent further writes. Plan disk capacity accordingly.

Storage Performance

For optimal performance, the following storage characteristics are recommended:

• Disk Type: SSD or NVMe storage for Longhorn volumes
• Filesystem: XFS or ext4 with default mount options
• Partition Layout: Dedicated /var/lib/longhorn partition for persistent storage

Virtual machines and bare-metal hardware are both supported. Nested virtualization (running multiple nodes under a single hypervisor) may impact performance and is not recommended for production deployments.

Operating System Requirements

Supported Operating Systems

The CDN Manager supports Red Hat Enterprise Linux and compatible distributions:

Compatible Clones

The following RHEL-compatible distributions are supported when major version requirements are satisfied:

• Oracle Linux 9
• AlmaLinux 9
• Rocky Linux 9

Air-Gapped Deployments

Important: For air-gapped deployments (no internet access), the OS installation ISO must be mounted on all nodes before running the installer or join commands. The installer needs to install one or more packages from the distribution’s repository.

Oracle Linux UEK Kernel

Note: For Oracle Linux 9.7 and later using the Unbreakable Enterprise Kernel (UEK), you must install the kernel-uek-modules-extra-netfilter-$(uname -r) package before running the installer:

# Mount OS ISO first (required for air-gapped)
mount -o loop /path/to/oracle-linux-9.iso /mnt/iso

# Install required kernel modules
dnf install kernel-uek-modules-extra-netfilter-$(uname -r)

This package provides netfilter kernel modules required by K3s and Longhorn.

SELinux

SELinux is supported when installed in “Enforcing” mode. The installation process will configure appropriate SELinux policies automatically.

Networking Requirements

Network Interface

Each cluster node must have at least one network interface card (NIC) configured as the default gateway. If the node lacks a pre-configured default route, one must be established prior to installation.

Port Requirements

The cluster requires the following network connectivity:

Important: Complete port requirements, network ranges, and firewall configuration procedures are provided in the Networking Guide. Do not expose VictoriaMetrics (8428, 8429), Grafana (3000), or PostgreSQL (5432) directly—access these services only through the secure HTTPS ingress (port 443).

Resource Planning

Calculating Cluster Capacity

When planning cluster capacity, consider the following factors:

1. Base Overhead: Kubernetes system components consume approximately 1-2 cores and 2-4 GiB memory per node
2. Application Workloads: Refer to individual component resource requirements in the Architecture Guide
3. Headroom: Maintain 20-30% resource headroom for workload spikes and automatic scaling

Scaling Considerations

The CDN Manager supports horizontal scaling for most components. The Horizontal Pod Autoscaler (HPA) can automatically adjust replica counts based on resource utilization. Detailed scaling guidance is available in the Architecture Guide.

Example Production Deployment

A minimal production deployment with 3 server nodes (combined role) and 2 agent nodes would require:

This configuration provides:

• High availability (survives loss of 1 server node)
• Capacity for application workloads across all nodes
• Headroom for horizontal scaling

Next Steps

After verifying system requirements:

1. Review the Installation Guide for deployment procedures
2. Consult the Networking Guide for firewall configuration
3. Examine the Architecture Guide for component resource requirements

3 - Networking Guide

Network Architecture

Physical Network

Each cluster node must have at least one network interface card (NIC) configured as the default gateway. If the node lacks a pre-configured default route, it must be established prior to installation.

K3s requires a default route to auto-detect the node’s primary IP and for kube-proxy ClusterIP routing to function properly. If no default route exists, create a dummy interface as a workaround:

ip link add dummy0 type dummy
ip link set dummy0 up
ip addr add 203.0.113.254/31 dev dummy0
ip route add default via 203.0.113.255 dev dummy0 metric 1000

Overlay Network

Kubernetes creates virtual network interfaces for pods that are typically not associated with any specific firewalld zone. The cluster uses the following network ranges:

Firewall regulations should target the primary physical interface. The overlay network traffic is handled by Flannel VXLAN.

Port Requirements

Inter-Node Communication

The following ports must be permitted between all cluster nodes for Kubernetes and cluster infrastructure:

Application Services Ports

The following ports must be accessible for application services within the cluster:

External Access Ports

The following ports must be accessible from external clients to cluster nodes:

Network Configuration Guides

Deployment Type

Choose the guide that matches your deployment architecture:

Not sure which to use? If you have explicitly separate interfaces for cluster communication and external access, start with Configuring Segregated Networks. Only use the shared interface guide if your hardware is limited to a single NIC.

3.1 - Shared Interface Network Setup

Overview

This guide covers network configuration for standard single-NIC deployments. In this architecture, all traffic—including internal cluster communication (East-West) and external internet access (North-South)—is routed through a single network interface.

Security Warning: Because all traffic shares the same interface and firewall zone, there is no physical or logical isolation between cluster management traffic and public-facing service traffic. For production environments requiring security isolation, see Configuring Segregated Networks.

Note: The installer script automatically detects if firewalld is enabled. If so, it will verify that the required inter-node ports are open through the firewall in the default zone before proceeding. If any required ports are missing, the installer will report an error and exit. Application service ports (such as Kafka, VictoriaMetrics, and Vector) are not checked by the installer as they are configurable.

For network architecture, port requirements, and general information, see the Network Architecture Overview section in the main Networking Guide.

firewall Configuration

Assign Interface to Default Zone

Assign your primary network interface to the default zone:

firewall-cmd --permanent --zone=public --change-interface=<interface>
firewall-cmd --reload

Replace <interface> with your actual interface name (e.g., eth0).

Configure Firewall Rules

In a shared interface setup, you must manually configure firewall rules for both internal cluster traffic and external access, as K3s does not automatically manage the public zone.

# 1. Allow pod and service networks (Internal CIDRs)
firewall-cmd --permanent --zone=public --add-source=10.42.0.0/16
firewall-cmd --permanent --zone=public --add-source=10.43.0.0/16

# 2. Kubernetes and Cluster Infrastructure (East-West Traffic)
# These ports must be opened manually for the cluster to function on a single interface.
firewall-cmd --permanent --zone=public --add-port=2379-2380/tcp
firewall-cmd --permanent --zone=public --add-port=6443/tcp
firewall-cmd --permanent --zone=public --add-port=8472/udp
firewall-cmd --permanent --zone=public --add-port=10250/tcp
firewall-cmd --permanent --zone=public --add-port=5001/tcp
firewall-cmd --permanent --zone=public --add-port=9500-9503/tcp
firewall-cmd --permanent --zone=public --add-port=8500-8504/tcp
firewall-cmd --permanent --zone=public --add-port=10000-30000/tcp
firewall-cmd --permanent --zone=public --add-port=3260/tcp
firewall-cmd --permanent --zone=public --add-port=2049/tcp

# 3. External Access Ports (North-South Traffic)
firewall-cmd --permanent --zone=public --add-port=80/tcp
firewall-cmd --permanent --zone=public --add-port=443/tcp
firewall-cmd --permanent --zone=public --add-port=9095/tcp
firewall-cmd --permanent --zone=public --add-port=6379/tcp
firewall-cmd --permanent --zone=public --add-port=8086/tcp

# Apply changes
firewall-cmd --reload

Verification

Verify all port rules are applied:

firewall-cmd --zone=public --list-all

Expected output:

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 10.42.0.0/16 10.43.0.0/16
  services: dhcpv6-client ssh
  ports: 80/tcp 443/tcp 9095/tcp 6379/tcp 8086/tcp
  protocols: 2379-2380/tcp 6443/tcp 8472/udp 10250/tcp 5001/tcp 9500-9503/tcp 8500-8504/tcp 10000-30000/tcp 3260/tcp 2049/tcp
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich-rules:

Note: Additional interfaces may appear in the zone (e.g., eth0 eth1) if firewalld auto-assigned them based on network configuration. This is expected and does not affect functionality.

Verify the interface is correctly assigned to the public zone:

firewall-cmd --get-active-zones

Expected output will show eth0 listed under the public zone:

public (active)
  interfaces: eth0

Troubleshooting

Expected output will show eth0 listed under the public zone:

public (active)
  interfaces: eth0

Troubleshooting

Nodes Cannot Communicate

Verify firewall rules allow inter-node traffic in the public zone:

firewall-cmd --list-all

Test basic connectivity between nodes:

ping <node-ip>

Post-Installation Troubleshooting

Once the cluster is installed, if you encounter issues with pod-to-pod communication or service access, verify the following:

1. Flannel Interface: Ensure the flannel.1 interface is up and has the correct IP addresses.
2. Network Routes: Verify that the pod and service CIDR routes are present in the routing table.
3. Firewall Rules: Ensure all required Kubernetes and cluster ports are allowed in the public zone.

For detailed troubleshooting of Kubernetes-specific components (like Ingress or Pod connectivity), please refer to the Kubernetes Troubleshooting Guide.

3.2 - Configuring Segregated Networks

Overview

This guide covers configuring a cluster with separate interfaces for internal cluster communication and external internet access (also known as segregated or dual-homed deployments). In this setup, eth1 handles the internal cluster traffic (pod-to-pod, control plane) while eth0 provides public internet access.

Security Benefit: This configuration provides physical isolation between East-West (cluster) and North-South (external) traffic. The trusted zone allows unrestricted internal communication, while the public zone handles external access with controlled port exposure.

When configuring segregated networks with K3s, proper interface binding is essential. K3s uses the --flannel-iface flag to ensure pod traffic stays on the private network, and the --node-external-ip flag to advertise the public address for external access. Server nodes additionally require --advertise-address=<ETH1_IP> to ensure the API server advertises its internal/private address; without this flag, k3s promotes the external IP to the advertise address when --node-external-ip is set, causing the kubernetes service ClusterIP endpoint to register as an address that is unreachable from within the cluster.

Important: K3s manages pod masquerading and service routing automatically. You only need to configure firewalld zones correctly and pass the proper flags to the K3s installer.

Complete, step-by-step instructions follow.

Prerequisites

Before starting, ensure:

• Operating system is installed and updated on all nodes
• Network connectivity between nodes is available
• SSH access is configured for all cluster nodes

Configure Firewalld Zones

This guide configures separate zones for internal cluster traffic and external access.

Assign Interfaces to Zones

K3s uses trusted zone for the internal network to allow unrestricted pod-to-pod and control plane traffic:

# Assign eth0 (external/internet) to public zone
firewall-cmd --permanent --zone=public --change-interface=eth0

# Assign eth1 (internal/cluster) to trusted zone
firewall-cmd --permanent --zone=trusted --change-interface=eth1

# Allow pod and service CIDRs in trusted zone (required for pod communication)
firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16
firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16

# Reload firewall
firewall-cmd --reload

Configure Firewall Ports

Open the necessary ports on the public zone for external access:

# External access ports
firewall-cmd --permanent --zone=public --add-port=80/tcp
firewall-cmd --permanent --zone=public --add-port=443/tcp
firewall-cmd --permanent --zone=public --add-port=9095/tcp
firewall-cmd --permanent --zone=public --add-port=6379/tcp
firewall-cmd --permanent --zone=public --add-port=8086/tcp

# Apply changes
firewall-cmd --reload

Note: K3s automatically creates iptables rules for internal cluster ports (6443, 10250, 2379-2380, 8472, 5001, 9500-9503, 8500-8504, 10000-30000, 3260, 2049) when using --flannel-iface=eth1. Pod and service CIDRs (10.42.0.0/16 and 10.43.0.0/16) are already allowed in the trusted zone via the --add-source commands above.

Verify Zone Configuration

firewall-cmd --zone=public --list-all
firewall-cmd --zone=trusted --list-all

Expected output for public zone:

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0 eth2
  sources: 
  services: dhcpv6-client ssh cockpit
  ports: 80/tcp 443/tcp 9095/tcp 6379/tcp 8086/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Expected output for trusted zone:

trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: eth1
  sources: 10.42.0.0/16 10.43.0.0/16
  services: ssh mdns
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Note: Additional interfaces may appear in a zone (e.g., eth0 eth2) if firewalld auto-assigned them based on network configuration. This is expected and does not affect functionality.

Single-NIC Alternative

If you only have a single network interface, see the Shared Interface Setup guide instead. This guide is specifically for multi-NIC deployments with separate interfaces for cluster and external traffic.

Troubleshooting

Verify Zone Configuration

If pods cannot communicate with services, verify the trusted zone has the correct sources configured:

firewall-cmd --zone=trusted --list-all

Expected output:

trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: eth1
  sources: 10.42.0.0/16 10.43.0.0/16
  services: ssh mdns
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Ensure both 10.42.0.0/16 (pod network) and 10.43.0.0/16 (service network) are listed under sources. If missing, re-run:

firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16
firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16
firewall-cmd --reload

4 - Architecture Guide

Overview

The AgileTV CDN Manager (ESB3027) is a cloud-native Kubernetes application designed for managing CDN operations. This guide provides a detailed description of the system architecture, component interactions, and scaling considerations.

High-Level Architecture

The CDN Manager follows a microservices architecture deployed on Kubernetes. The system is organized into logical layers:

graph LR
    Clients[API Clients] --> Ingress[Ingress Controller]
    Ingress --> Manager[Core Manager]
    Ingress --> Frontend[MIB Frontend]
    Ingress --> Grafana[Grafana]
    Manager --> Redis[(Redis)]
    Manager --> Kafka[(Kafka)]
    Manager --> PostgreSQL[(PostgreSQL)]
    Manager --> Zitadel[Zitadel IAM]
    Manager --> Confd[Configuration Service]
    Grafana --> VM[(VictoriaMetrics)]
    Confd -.-> Gateway[NGinx Gateway]
    Gateway --> Director[CDN Director]

Component Architecture

Ingress Layer

The ingress layer manages all incoming traffic to the cluster:

Traffic flow:

• API clients and Operator UI connect via the Ingress Controller at /api and /gui paths respectively
• Grafana dashboards are accessed via the Ingress Controller at /grafana
• Zitadel authentication console is accessed via the Ingress Controller at /ui/console
• MIB Frontend uses NGinx Gateway when communicating with external Confd instances on CDN Director nodes

Application Services

The application layer contains the core CDN Manager services:

Data Layer

The data layer provides persistent and ephemeral storage:

External Integrations

Detailed Component Descriptions

Core Manager

The Core Manager is the central application server that exposes the REST API. It is implemented in Rust using the Actix-web framework.

Key Responsibilities:

• Authentication and session management via Zitadel
• Configuration document storage and retrieval
• Selection input CRUD operations
• Routing rule evaluation and GeoIP lookups
• Service discovery for CDN Directors and edge servers
• Operator UI helper endpoints

API Endpoints:

• /api/v1/auth/* - Authentication (login, token, logout)
• /api/v1/configuration - Configuration management
• /api/v1/selection_input/* - Selection input operations
• /api/v2/selection_input/* - Enhanced selection input with list operations
• /api/v1/routing/* - Routing evaluation and validation
• /api/v1/discovery/* - Host and namespace discovery
• /api/v1/metrics - System metrics
• /api/v1/health/* - Liveness and readiness probes
• /api/v1/operator_ui/* - Operator helper endpoints

Runtime Modes: The Core Manager supports multiple runtime modes, each deployed as a separate container:

• http-server - Primary HTTP API server (default)
• metrics-aggregator - Background worker for metrics collection
• selection-input - Background worker for Kafka selection input consumption

MIB Frontend

The MIB Frontend provides a web-based GUI for configuration management.

Key Features:

• Intuitive web interface for CDN configuration
• Real-time configuration validation
• Integration with Zitadel for SSO authentication
• Uses NGinx Gateway for external Director communication

Confd (Configuration Service)

Confd provides routing configuration services and synchronizes with the Core Manager application.

Key Responsibilities:

• Hosts the service configuration for routing decisions
• Provides API and CLI for configuration management
• Synchronizes routing configuration with Core Manager
• Maintains configuration state in PostgreSQL

Selection Input Worker

The Selection Input Worker processes selection input events from the Kafka stream.

Key Responsibilities:

• Consumes messages from the selection_input Kafka topic
• Validates and transforms input data
• Updates configuration in the data store
• Maintains message ordering within partitions

Scaling Limitation: The Selection Input Worker cannot be scaled beyond a single consumer per Kafka partition, as message ordering must be preserved.

Metrics Aggregator

The Metrics Aggregator collects and processes metrics from CDN components.

Key Responsibilities:

• Polls metrics from Director instances
• Aggregates usage statistics
• Writes data to VictoriaMetrics (Analytics) for dashboards
• Writes long-term data to VictoriaMetrics (Billing) for compliance

Telegraf

Telegraf is deployed as a DaemonSet to collect host-level metrics.

Key Responsibilities:

• CPU, memory, disk, and network metrics from each node
• Container-level resource usage
• Kubernetes cluster metrics
• Forwards metrics to VictoriaMetrics

Grafana

Grafana provides visualization and dashboard capabilities.

Features:

• Pre-built dashboards for CDN monitoring
• Custom dashboard support
• VictoriaMetrics as data source
• Alerting integration with Alertmanager

Access: https://<host>/grafana

Alertmanager

Alertmanager handles alert routing and notifications.

Key Responsibilities:

• Receives alerts from Grafana and other sources
• Deduplicates and groups alerts
• Routes to notification channels (email, webhook, etc.)
• Manages alert silencing and inhibition

Data Storage

Redis

Redis provides in-memory storage for:

• User sessions and authentication tokens
• Ephemeral configuration cache
• Real-time state synchronization

Deployment: Master + read replicas for high availability

Kafka

Kafka provides durable event streaming for:

• Selection input events
• Metrics data streams
• Inter-service communication

Deployment: Controller cluster with 3 replicas for production, 1 replica for lab deployments

Node Affinity: Kafka replicas must be scheduled on separate nodes to ensure high availability. The Helm chart configures pod anti-affinity rules to enforce this distribution.

Topics:

• selection_input - Selection input events
• metrics - Metrics data streams

Note: For lab/single-node deployments, the Kafka replica count must be set to 1 in the Helm values. Production deployments require 3 replicas for fault tolerance.

PostgreSQL

PostgreSQL provides persistent storage for:

• Configuration documents
• User and permission data
• System state

Deployment: 3-node cluster managed by Cloudnative PG (CNPG) operator

High Availability: The CNPG operator manages automatic failover and ensures high availability:

• One primary node handles read/write operations
• Two replica nodes provide redundancy and can be promoted to primary on failure
• Automatic failover occurs within seconds of primary node failure
• Synchronous replication ensures data consistency

Note: The PostgreSQL cluster is deployed and managed automatically by the CNPG operator. Manual intervention is typically not required for normal operations.

VictoriaMetrics

Two VictoriaMetrics instances serve different purposes:

VictoriaMetrics (Analytics):

• Real-time and short-term metrics storage
• Supports Grafana dashboards
• Retention: Configurable (typically 30-90 days)

VictoriaMetrics (Billing):

• Long-term metrics retention
• Billing and license compliance data
• Retention: Minimum 1 year

Authentication and Authorization

Zitadel Integration

Zitadel provides identity and access management:

Authentication Flow:

1. User accesses MIB Frontend or API
2. Redirected to Zitadel for authentication
3. Zitadel validates credentials and issues session token
4. Session token exchanged for access token
5. Access token included in API requests (Bearer authentication)

Default Credentials: See the Glossary for default login credentials.

Access Paths:

• Zitadel Console: /ui/console
• API authentication: /api/v1/auth/*

CORS Configuration

Zitadel enforces Cross-Origin Resource Sharing (CORS) policies. The external hostname configured in Zitadel must match the first entry in global.hosts.manager in the Helm values.

Network Architecture

Traffic Flow

graph TB
    External[External Clients] --> Ingress[Ingress Controller]
    External --> Redis[(Redis)]
    External --> Kafka[(Kafka)]
    External --> Telegraf[Telegraf]
    Ingress --> Manager[Core Manager]
    Ingress --> Frontend[MIB Frontend]
    Ingress --> Grafana[Grafana]
    Ingress --> Zitadel[Zitadel]

Note: Certain services (Redis, Kafka, Telegraf) can be accessed directly by external clients without traversing the ingress controller. This is typically used for metrics collection, event streaming, and direct data access scenarios.

Internal Communication

All internal services communicate over the Kubernetes overlay network (Flannel VXLAN). Services discover each other via Kubernetes DNS.

External Communication

• CDN Directors: Accessed via NGinx Gateway for simplified routing
• MaxMind GeoIP: Local database files (no external calls)

Scaling

Horizontal Pod Autoscaler (HPA)

The following components support automatic horizontal scaling via HPA:

Note: HPA is enabled by default in the Helm chart. The default configuration is tuned for production deployments. Adjust min/max values based on expected load and available cluster capacity.

Manual Scaling

Components can also be scaled manually by setting replica counts in the Helm values:

manager:
  replicaCount: 3
mib-frontend:
  replicaCount: 2

Important: When manually setting replica counts, you must disable the Horizontal Pod Autoscaler (HPA) for the corresponding component. If HPA remains enabled, it will override manual replica settings. To disable HPA, set autoscaling.hpa.enabled: false for the component in your Helm values.

Components That Do Not Scale

The following components do not support horizontal scaling:

Node Scaling

Additional Agent nodes can be added to the cluster at any time to increase workload capacity. Kubernetes automatically schedules pods to nodes with available resources.

Cluster Balancing

The CDN Manager deployment includes the Kubernetes Descheduler to maintain balanced resource utilization across cluster nodes:

• Automatic Rebalancing: The descheduler periodically analyzes pod distribution and evicts pods from overutilized nodes
• Node Balance: Helps prevent resource hotspots by redistributing workloads across available nodes
• Integration with HPA: Works in conjunction with Horizontal Pod Autoscaler to optimize both pod count and placement

The descheduler runs as a background process and does not require manual intervention under normal operating conditions.

Resource Configuration

For detailed resource preset configurations and planning guidance, see the Configuration Guide.

High Availability

Server Node Redundancy

Production deployments require a minimum of 3 Server nodes:

• Survives loss of 1 server node
• Maintains quorum for etcd and Kafka

For enhanced availability, use 5 Server nodes:

• Survives loss of 2 server nodes
• Recommended for critical production environments

For large-scale deployments, 7 or more Server nodes can be used:

• Survives loss of 3+ server nodes
• Suitable for high-capacity production environments

Pod Distribution

Kubernetes automatically distributes pods across nodes to maximize availability:

• Pods with the same deployment are scheduled on different nodes when possible
• Pod Disruption Budgets (PDB) ensure minimum availability during maintenance

Data Replication

Longhorn Storage: Longhorn volumes are configured with a single replica by default. Pod scheduling is configured with node affinity to prefer scheduling pods on the same node as their persistent volume data. This approach optimizes I/O performance while maintaining data locality.

Next Steps

After understanding the architecture:

1. Installation Guide - Deploy the CDN Manager
2. Configuration Guide - Configure components for your environment
3. Operations Guide - Day-to-day operational procedures
4. Performance Tuning Guide - Optimize system performance
5. Metrics & Monitoring - Set up monitoring and alerting

5 - Installation Guide

Overview

This guide provides detailed instructions for installing the AgileTV CDN Manager (ESB3027) in various deployment scenarios. The installation process varies depending on the target environment and desired configuration.

Estimated Installation Time:

Actual installation time may vary depending on hardware performance, network speed, and whether air-gapped procedures are required.

Note: These estimates assume the operating system is already installed on all nodes. OS installation is outside the scope of this guide.

Installation Types

Installation Process Summary

The installation follows a sequential process:

1. Prepare the host system - Verify requirements and mount the installation ISO
2. Install the Kubernetes cluster - Deploy K3s, Longhorn storage, and PostgreSQL
3. Join additional nodes (production only) - Expand the cluster for HA or capacity
4. Deploy the Manager application - Install the CDN Manager Helm chart
5. Post-installation configuration - Configure authentication, networking, and users

Quick Links

Prerequisites

Before beginning installation, ensure the following requirements are met:

• Hardware: Nodes meeting the System Requirements including CPU, memory, and disk specifications
• Operating System: RHEL 9 or compatible clone (details); air-gapped deployments require the OS ISO mounted on all nodes
• Network: Proper firewall configuration between nodes (port requirements, firewall configuration)
• Software: Installation ISO obtained from AgileTV; air-gapped deployments also require the Extras ISO
• Kernel Tuning: For production deployments, apply recommended sysctl settings (Performance Tuning Guide)

We recommend using the Installation Checklist to track your progress through the installation process.

Getting Help

If you encounter issues during installation:

• Review the Troubleshooting Guide for common issues
• Check the System Requirements to verify your environment
• Consult the Release Notes for version-specific known issues

5.1 - Installation Checklist

Overview

Use this checklist to track your installation progress. Print this page or keep it open during your installation to ensure all steps are completed correctly.

Pre-Installation

Hardware and Software

• Verify hardware meets System Requirements
• Confirm operating system is supported (RHEL 9 or compatible clone)
• Configure firewall rules between nodes (details)
• Apply recommended sysctl settings (details)
• Obtain installation ISO (esb3027-acd-manager-X.Y.Z.iso)

Air-Gapped Deployments

• Obtain Extras ISO (esb3027-acd-manager-extras-X.Y.Z.iso)
• Mount OS ISO on all nodes before installation
• Verify OS packages are accessible from mounted ISO

Special Requirements

• Oracle Linux UEK: Install kernel-uek-modules-extra-netfilter-$(uname -r) package
• Control Plane Only nodes: Set SKIP_REQUIREMENTS_CHECK=1 if below lab minimums
• SELinux: Set to “Enforcing” mode before running installer (cannot enable after)

Cluster Installation

Single-Node Deployment

Follow the Single-Node Installation Guide.

• Mount installation ISO (Step 1)
• Install the base cluster (Step 2)
• Verify cluster status (Step 3)
• Air-gapped only: Load container images (Step 4)
• Create configuration file (Step 5)
• Optional: Load MaxMind GeoIP databases (Step 6)
• Deploy the Manager Helm chart (Step 7)
• Verify deployment (Step 8)

Multi-Node Deployment

Follow the Multi-Node Installation Guide.

Primary Server Node

• Mount installation ISO (Step 1)
• Install the base cluster (Step 2)
• Verify system pods are running (Step 2)
• Retrieve the node token (Step 3)

Additional Server Nodes

• Mount installation ISO (Step 5)
• Join the cluster (Step 5)
• Verify each node joins (Step 5)
• Optional: Taint Control Plane Only nodes (Step 5b)

Agent Nodes (Optional)

• Mount installation ISO (Step 6)
• Join the cluster as an agent (Step 6)
• Verify each agent joins (Step 6)

Cluster Verification

• Verify all nodes are ready (Step 7)
• Verify system pods running on all nodes (Step 7)
• Air-gapped only: Load container images on each node (Step 9)

Application Deployment

• Create configuration file (Step 10)
• Optional: Load MaxMind GeoIP databases (Step 11)
• Optional: Configure TLS certificates from trusted CA (Step 12)
• Deploy the Manager Helm chart (Step 13)
• Verify all pods are running and distributed (Step 14)
• Configure DNS records for manager hostname (Step 15)

Post-Installation

Initial Access

• Access the system via HTTPS
• Accept self-signed certificate warning (if using default certificate)
• Log in with default credentials (see Glossary)

Security Configuration

• Create new administrator account in Zitadel
• Delete or secure the default admin account
• Configure additional users and permissions
• Review Zitadel Administrator Documentation for role assignments

Monitoring and Operations

• Access Grafana dashboards at /grafana
• Review pre-built monitoring dashboards
• Configure alerting rules (optional)
• Set up notification channels (optional)

Next Steps

• Review Next Steps Guide for additional configuration
• Configure CDN routing rules
• Set up GeoIP-based routing (if using MaxMind databases)
• Review Operations Guide for day-to-day procedures

Troubleshooting

If you encounter issues during installation:

1. Check pod status: kubectl describe pod <pod-name>
2. Review logs: kubectl logs <pod-name>
3. Check cluster events: kubectl get events --sort-by='.lastTimestamp'
4. Review the Troubleshooting Guide for common issues

5.2 - Single-Node Installation

Warning: Single-node deployments are for lab environments, acceptance testing, and demonstrations only. This configuration is not suitable for production workloads. For production deployments, see the Multi-Node Installation Guide, which requires a minimum of 3 server nodes for high availability.

Air-Gapped Deployment? This guide assumes internet connectivity. For air-gapped deployments, see the Air-Gapped Deployment Guide for additional requirements and procedures.

Overview

This guide describes the installation of the AgileTV CDN Manager on a single node. This configuration is intended for lab environments, acceptance testing, and demonstrations only. It is not suitable for production workloads.

Prerequisites

Hardware Requirements

Refer to the System Requirements Guide for hardware specifications. Single-node deployments require the “Single-Node (Lab)” configuration.

Operating System

Refer to the System Requirements Guide for supported operating systems.

Software Access

• Installation ISO: esb3027-acd-manager-X.Y.Z.iso
• Extras ISO (air-gapped only): esb3027-acd-manager-extras-X.Y.Z.iso

Network Configuration

Ensure that required firewall ports are configured before installation. See the Networking Guide for complete firewall configuration requirements.

SELinux

If SELinux is to be used, it must be set to “Enforcing” mode before running the installer script. The installer will configure appropriate SELinux policies automatically. SELinux cannot be enabled after installation.

Installation Steps

Step 1: Mount the ISO

Create a mount point and mount the installation ISO:

mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

Replace X.Y.Z with the actual version number.

Step 2: Install the Base Cluster

Run the installer to set up the K3s Kubernetes cluster:

/mnt/esb3027/install

This installs:

• K3s Kubernetes distribution
• Longhorn distributed storage
• Cloudnative PG operator for PostgreSQL
• Base system dependencies

The installer will configure the node as both a server and agent node.

Step 3: Verify Cluster Status

After the installer completes, verify that all components are operational before proceeding. This verification serves as an important checkpoint to confirm the installation is progressing correctly.

1. Verify the node is ready:

kubectl get nodes

Expected output:

NAME         STATUS   ROLES                       AGE   VERSION
k3s-server   Ready    control-plane,etcd,master   2m    v1.33.4+k3s1

2. Verify system pods in both namespaces are running:

# Check kube-system namespace (Kubernetes core components)
kubectl get pods -n kube-system

# Check longhorn-system namespace (distributed storage)
kubectl get pods -n longhorn-system

All pods should show Running status. If any pods are still Pending or ContainerCreating, wait until they are ready. Proceeding with incomplete system pods can cause subsequent steps to fail in unpredictable ways.

This verification confirms:

• K3s cluster is operational
• Longhorn distributed storage is running
• Cloudnative PG operator is deployed
• All core components are healthy before continuing

Step 4: Air-Gapped Deployments (If Applicable)

If deploying in an air-gapped environment, load container images from the extras ISO:

mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras
/mnt/esb3027-extras/load-images

Step 5: Deploy the Manager Helm Chart

For complete instructions on deploying the CDN Manager Helm chart, including configuration file setup, MaxMind GeoIP database loading, TLS certificate configuration, deployment commands, and verification steps, see the Helm Chart Installation Guide.

This guide covers the common deployment steps that apply to all installation types. After completing the helm chart installation steps, proceed to Post-Installation below.

Post-Installation

After installation completes, proceed to the Next Steps guide for:

• Initial user configuration
• Accessing the web interfaces
• Configuring authentication
• Setting up monitoring

Accessing the System

Refer to the Accessing the System section in the Getting Started guide for service URLs and default credentials.

Note: A self-signed SSL certificate is deployed by default. You will need to accept the certificate warning in your browser.

Troubleshooting

If pods fail to start:

1. Check pod status: kubectl describe pod <pod-name>
2. Review logs: kubectl logs <pod-name>
3. Verify resources: kubectl top pods

See the Troubleshooting Guide for additional assistance.

Next Steps

After successful installation:

1. Next Steps Guide - Post-installation configuration
2. Configuration Guide - System configuration
3. Operations Guide - Day-to-day operations

Appendix: Lab Configuration File

The installation ISO includes a pre-built lab configuration at /mnt/esb3027/values-lab.yaml, designed specifically for single-node deployments. It handles single-replica settings for Kafka and Zitadel, resource sizing, and TLS configuration automatically.

Copy it as your starting point:

cp /mnt/esb3027/values-lab.yaml ~/values.yaml

At minimum, update these two fields to match your environment before deploying:

global:
  hosts:
    manager:
      - host: manager.local   # Replace with your hostname or IP

zitadel:
  zitadel:
    configmapConfig:
      ExternalDomain: manager.local   # Must match global.hosts.manager[0].host

These two values must match exactly or authentication will fail. For a full description of all available options in values-lab.yaml, see the Configuration Guide.

5.3 - Multi-Node Installation

Overview

This guide describes the installation of the AgileTV CDN Manager across multiple nodes for production deployments. This configuration provides high availability and horizontal scaling capabilities.

Air-Gapped Deployment? This guide assumes internet connectivity. For air-gapped deployments, see the Air-Gapped Deployment Guide for additional requirements and procedures.

Prerequisites

Hardware Requirements

Refer to the System Requirements Guide for hardware specifications. Production deployments require:

• Minimum 3 Server nodes (Control Plane Only or Combined role)
• Optional Agent nodes for additional workload capacity

Operating System

Refer to the System Requirements Guide for supported operating systems.

Software Access

• Installation ISO: esb3027-acd-manager-X.Y.Z.iso (for each node)
• Extras ISO (air-gapped only): esb3027-acd-manager-extras-X.Y.Z.iso

Network Configuration

Ensure that required firewall ports are configured between all nodes before installation. See the Configuring Segregated Networks guide for the standard firewall configuration.

Note: When using segregated networks, the K3s API server on the primary node will be reachable via its internal/private interface. Consequently, when joining additional nodes, the <primary-server-ip> provided to the join script must be the internal/private IP address of the primary node to ensure the join request is routed correctly through the private network.

Single-NIC Deployments: If your nodes have only a single network interface, see the Shared Interface Setup guide instead. This guide assumes segregated networks with separate interfaces for cluster traffic (eth1) and external access (eth0).

Segregated Network Configuration

If your nodes have multiple network interfaces and you want to use a separate interface for cluster traffic (not the default route interface), configure the INSTALL_K3S_EXEC environment variable before installing the cluster or joining nodes.

For segregated networks (private cluster network on eth1 + public external access on eth0), set all three K3s flags:

# For server nodes
export INSTALL_K3S_EXEC="server --node-ip=<ETH1_IP> --node-external-ip=<ETH0_IP> --flannel-iface=eth1 --advertise-address=<ETH1_IP>"

# For agent nodes  
export INSTALL_K3S_EXEC="agent --node-ip=<ETH1_IP> --node-external-ip=<ETH0_IP> --flannel-iface=eth1"

Where:

• Mode: Use server for the primary node establishing the cluster, or for additional server nodes. Use agent for agent nodes joining the cluster.
• --node-ip=<ETH1_IP>: The internal/private IP address of eth1 for cluster communication
• --node-external-ip=<ETH0_IP>: The public IP address of eth0 for external access (LoadBalancer services, ingress)
• --flannel-iface=eth1: The network interface name for Flannel VXLAN overlay traffic
• --advertise-address=<ETH1_IP>: The address the API server uses to advertise itself to cluster members. Must be set to the internal/private IP address in a segregated-network deployment; without this flag, k3s defaults to the external IP when --node-external-ip is set, causing the kubernetes service endpoint to register as an unreachable address. This flag is required for server nodes only; agent nodes do not run an API server.

Set this variable on each node before running the install or join scripts.

SELinux

If SELinux is to be used, it must be set to “Enforcing” mode before running the installer script. The installer will configure appropriate SELinux policies automatically. SELinux cannot be enabled after installation.

Installation Steps

Step 1: Prepare the Primary Server Node

Mount the installation ISO on the primary server node:

mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

Replace X.Y.Z with the actual version number.

Step 2: Install the Base Cluster on Primary Server

Segregated Networks: If your node has multiple network interfaces, set the INSTALL_K3S_EXEC environment variable with the complete segregated network configuration before running the installer (see Segregated Network Configuration):

export INSTALL_K3S_EXEC="server --node-ip=<ETH1_IP> --node-external-ip=<ETH0_IP> --flannel-iface=eth1 --advertise-address=<ETH1_IP>"

Replace <ETH1_IP> with the internal/private IP address and <ETH0_IP> with the public IP address.

If your node has only a single network interface, do not set INSTALL_K3S_EXEC. K3s will use the default interface automatically.

Run the installer to set up the K3s Kubernetes cluster:

/mnt/esb3027/install

This installs:

• K3s Kubernetes distribution
• Longhorn distributed storage
• Cloudnative PG operator for PostgreSQL
• Base system dependencies

Important: After the installer completes, verify that all system pods in both namespaces are in the Running state before proceeding:

# Check kube-system namespace (Kubernetes core components)
kubectl get pods -n kube-system

# Check longhorn-system namespace (distributed storage)
kubectl get pods -n longhorn-system

All pods should show Running status. If any pods are still Pending or ContainerCreating, wait until they are ready. Proceeding with incomplete system pods can cause subsequent steps to fail in unpredictable ways.

This verification confirms:

• K3s cluster is operational
• Longhorn distributed storage is running
• Cloudnative PG operator is deployed
• All core components are healthy before continuing

Step 3: Retrieve the Node Token

Retrieve the node token for joining additional nodes:

cat /var/lib/rancher/k3s/server/node-token

Save this token for use on additional nodes. Also note the IP address of the primary server node.

Step 4: Server vs Agent Node Roles

Before joining additional nodes, determine which nodes will serve as Server nodes vs Agent nodes:

Guidance:

• Combined role (default): Server nodes run both control plane and workloads; minimum 3 nodes required for HA
• Control Plane Only: Dedicate nodes to control plane functions; requires at least 3 Server nodes plus 3+ Agent nodes for workloads
• Agent nodes are required if using Control Plane Only servers; optional if using Combined role servers
• For most deployments, 3 Server nodes (Combined role) with no Agent nodes is sufficient
• Add Agent nodes to scale workload capacity without affecting control plane quorum

Proceed to Step 5 to join Server nodes. Agent nodes are joined after all Server nodes are ready.

Step 5: Join Additional Server Nodes

On each additional server node:

1. Mount the ISO:

   mkdir -p /mnt/esb3027
   mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027
   

2. Join the cluster:

Segregated Networks: If your node has multiple network interfaces, set the INSTALL_K3S_EXEC environment variable with the complete segregated network configuration before running the join script (see Segregated Network Configuration):

export INSTALL_K3S_EXEC="server --node-ip=<ETH1_IP> --node-external-ip=<ETH0_IP> --flannel-iface=eth1 --advertise-address=<ETH1_IP>"

Replace <ETH1_IP> with the internal/private IP address and <ETH0_IP> with the public IP address.

If your node has only a single network interface, do not set INSTALL_K3S_EXEC. K3s will use the default interface automatically.

Note for Segregated Networks: When joining nodes in a segregated network environment, ensure the <primary-server-ip> used in the join command is the internal/private IP address (the eth1 address) of the primary server. Using the external IP will cause the join attempt to fail as the service will be listening on the private interface.

Run the join script:

/mnt/esb3027/join-server https://<primary-server-ip>:6443 <node-token>

Replace <primary-server-ip> with the IP address of the primary server and <node-token> with the token retrieved in Step 3.

3. Verify the node joined successfully:

kubectl get nodes

Repeat for each server node. A minimum of 3 server nodes is required for high availability.

Step 5b: Taint Control Plane Only Nodes (Optional)

If you are using dedicated Control Plane Only nodes (not Combined role), apply taints to prevent workload scheduling:

kubectl taint nodes <node-name> CriticalAddonsOnly=true:NoSchedule

Apply this taint to each Control Plane Only node. Verify taints are applied:

kubectl describe nodes | grep -A 5 "Taints"

Note: This step is only required if you want dedicated control plane nodes. For Combined role deployments, do not apply taints.

Important: Control Plane Only Server nodes can be deployed with lower hardware specifications (2 cores, 4 GiB, 64 GiB) than the installer’s default minimum requirements. If your Control Plane Only Server nodes do not meet the Single-Node Lab configuration minimums (8 cores, 16 GiB, 128 GiB), you must set the SKIP_REQUIREMENTS_CHECK environment variable before running the installer or join command:

# For the primary server node
export SKIP_REQUIREMENTS_CHECK=1
/mnt/esb3027/install

# For additional Control Plane Only Server nodes
export SKIP_REQUIREMENTS_CHECK=1
/mnt/esb3027/join-server https://<primary-server-ip>:6443 <node-token>

Note: This applies to Server nodes only. Agent nodes have separate minimum requirements.

Step 6: Join Agent Nodes (Optional)

On each agent node:

1. Mount the ISO:

   mkdir -p /mnt/esb3027
   mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027
   

2. Join the cluster as an agent:

Segregated Networks: If your node has multiple network interfaces, set the INSTALL_K3S_EXEC environment variable with the complete segregated network configuration before running the join script (see Segregated Network Configuration):

export INSTALL_K3S_EXEC="agent --node-ip=<ETH1_IP> --node-external-ip=<ETH0_IP> --flannel-iface=eth1"

Replace <ETH1_IP> with the internal/private IP address and <ETH0_IP> with the public IP address.

If your node has only a single network interface, do not set INSTALL_K3S_EXEC. K3s will use the default interface automatically.

Run the join script:

/mnt/esb3027/join-agent https://<primary-server-ip>:6443 <node-token>

Note for Segregated Networks: When joining nodes in a segregated network environment, ensure the <primary-server-ip> used in the join command is the internal/private IP address (the eth1 address) of the primary server. Using the external IP will cause the join attempt to fail as the service will be listening on the private interface.

3. Verify the node joined successfully from an existing server node:

   kubectl get nodes
   

Agent nodes provide additional workload capacity but do not participate in the control plane quorum.

Step 7: Verify Cluster Status

After all nodes are joined, verify the cluster is operational:

1. Verify all nodes are ready:

kubectl get nodes

Expected output:

NAME                 STATUS   ROLES                       AGE   VERSION
k3s-server-0         Ready    control-plane,etcd,master   5m    v1.33.4+k3s1
k3s-server-1         Ready    control-plane,etcd,master   3m    v1.33.4+k3s1
k3s-server-2         Ready    control-plane,etcd,master   2m    v1.33.4+k3s1
k3s-agent-1          Ready    <none>                      1m    v1.33.4+k3s1
k3s-agent-2          Ready    <none>                      1m    v1.33.4+k3s1

2. Verify system pods in both namespaces are running:

# Check kube-system namespace (Kubernetes core components)
kubectl get pods -n kube-system

# Check longhorn-system namespace (distributed storage)
kubectl get pods -n longhorn-system

All pods should show Running status. If any pods are still Pending or ContainerCreating, wait until they are ready.

This verification confirms:

• K3s cluster is operational across all nodes
• Longhorn distributed storage is running
• Cloudnative PG operator is deployed
• All core components are healthy before proceeding to application deployment

Step 9: Air-Gapped Deployments (If Applicable)

If deploying in an air-gapped environment, on each node:

mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras
/mnt/esb3027-extras/load-images

Step 10: Deploy the Manager Helm Chart

For complete instructions on deploying the CDN Manager Helm chart, including configuration file setup, MaxMind GeoIP database loading, TLS certificate configuration, deployment commands, and verification steps, see the Helm Chart Installation Guide.

This guide covers the common deployment steps that apply to all installation types. After completing the helm chart installation steps, proceed to Post-Installation below.

Step 15: Configure DNS (Optional)

Add DNS records for the manager hostname. For high availability, configure multiple A records pointing to different server nodes:

manager.example.com.  IN  A  <server-1-ip>
manager.example.com.  IN  A  <server-2-ip>
manager.example.com.  IN  A  <server-3-ip>

Alternatively, configure a load balancer to distribute traffic across nodes.

Post-Installation

After installation completes, proceed to the Next Steps guide for:

• Initial user configuration
• Accessing the web interfaces
• Configuring authentication
• Setting up monitoring

Accessing the System

Refer to the Accessing the System section in the Getting Started guide for service URLs and default credentials.

Note: A self-signed SSL certificate is deployed by default. For production deployments, configure a valid SSL certificate before exposing the system to users.

High Availability Considerations

Pod Distribution

The Helm chart configures pod anti-affinity rules to ensure:

• Kafka controllers are scheduled on separate nodes
• PostgreSQL cluster members are distributed across nodes
• Application pods are spread across available nodes

Data Replication and Failure Tolerance

For detailed information on data replication strategies and failure scenario tolerance, refer to the Architecture Guide and System Requirements Guide.

Troubleshooting

If pods fail to start or nodes fail to join:

1. Check node status: kubectl get nodes
2. Describe problematic pods: kubectl describe pod <pod-name>
3. Review logs: kubectl logs <pod-name>
4. Check cluster events: kubectl get events --sort-by='.lastTimestamp'

Nodes Ready but Workloads Cannot Reach the API Server (Segregated Networks)

Symptom: All nodes show Ready status, but cluster components (kubelet, controller-manager, scheduler) or workloads fail to communicate with the API server. Pods in kube-system or longhorn-system may fail to start or remain in a crash loop.

Cause: This is caused by omitting --advertise-address from the server-node INSTALL_K3S_EXEC. When --node-external-ip is set without --advertise-address, k3s defaults the API server’s advertise address to the external IP (eth0). In a segregated-network topology where nodes are not routable to each other over eth0, the kubernetes service ClusterIP endpoint registers as an unreachable address.

Diagnostic check:

kubectl get endpoints kubernetes -n default

If the IP shown is the eth0 (external) address rather than the eth1 (internal) address, the cluster was installed without --advertise-address.

Remediation: The kubernetes service endpoint cannot be corrected by reconfiguration alone. K3s must be reinstalled on all server nodes with the correct flags:

export INSTALL_K3S_EXEC="server --node-ip=<ETH1_IP> --node-external-ip=<ETH0_IP> --flannel-iface=eth1 --advertise-address=<ETH1_IP>"

After reinstallation, re-run the diagnostic check to confirm the endpoint IP is now the eth1 (internal) address.

See the Troubleshooting Guide for additional assistance.

Next Steps

After successful installation:

1. Next Steps Guide - Post-installation configuration
2. Configuration Guide - System configuration
3. Operations Guide - Day-to-day operations

5.4 - Air-Gapped Deployment

Overview

This guide describes the installation of the AgileTV CDN Manager in air-gapped environments (no internet access). Air-gapped deployments require additional preparation compared to connected deployments.

Key differences from connected deployments:

• Both Installation ISO and Extras ISO are required on all nodes
• OS installation ISO must be mounted on all nodes for package access
• Container images must be loaded from the Extras ISO on each node
• Additional firewall considerations for OS package repositories

Prerequisites

Required ISOs

Before beginning installation, obtain the following:

Hardware Requirements

Refer to the System Requirements Guide for hardware specifications.

• Single-Node (Lab): Minimum 8 cores, 16 GiB RAM, 128 GiB disk
• Multi-Node (Production): Minimum 3 Server nodes for high availability

Network Configuration

Air-gapped environments may have internal network mirrors for OS packages. If no internal mirror exists, the OS installation ISO must be mounted on each node to provide packages during installation.

Ensure that required firewall ports are configured before installation. See the Networking Guide for complete firewall configuration requirements.

SELinux

If SELinux is to be used, it must be set to “Enforcing” mode before running the installer script. The installer will configure appropriate SELinux policies automatically. SELinux cannot be enabled after installation.

Installation Steps

Step 1: Prepare All Nodes

On each node (primary server, additional servers, and agents):

1. Mount the OS installation ISO:

mkdir -p /mnt/os
mount -o loop,ro /path/to/rhel-9.iso /mnt/os

2. Configure local repository (if no internal mirror):

cat > /etc/yum.repos.d/local.repo <<EOF
[local]
name=Local OS Repository
baseurl=file:///mnt/os/BaseOS
enabled=1
gpgcheck=0
EOF

# Also configure AppStream if needed
cat >> /etc/yum.repos.d/local.repo <<EOF

[appstream]
name=AppStream Repository
baseurl=file:///mnt/os/AppStream
enabled=1
gpgcheck=0
EOF

3. Verify repository is accessible:

dnf repolist
dnf makecache

Step 2: Prepare the Primary Server Node

Mount the installation ISOs on the primary server node:

# Mount Installation ISO
mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

# Mount Extras ISO
mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras

Step 3: Install the Base Cluster on Primary Server

Run the installer to set up the K3s Kubernetes cluster:

/mnt/esb3027/install

This installs:

• K3s Kubernetes distribution
• Longhorn distributed storage
• Cloudnative PG operator for PostgreSQL
• Base system dependencies

Important: After the installer completes, verify that all system pods in both namespaces are in the Running state before proceeding:

# Check kube-system namespace (Kubernetes core components)
kubectl get pods -n kube-system

# Check longhorn-system namespace (distributed storage)
kubectl get pods -n longhorn-system

All pods should show Running status. If any pods are still Pending or ContainerCreating, wait until they are ready. Proceeding with incomplete system pods can cause subsequent steps to fail in unpredictable ways.

This verification confirms:

• K3s cluster is operational
• Longhorn distributed storage is running
• Cloudnative PG operator is deployed
• All core components are healthy before continuing

Step 4: Retrieve the Node Token

Retrieve the node token for joining additional nodes:

cat /var/lib/rancher/k3s/server/node-token

Save this token for use on additional nodes. Also note the IP address of the primary server node.

Step 5: Join Additional Server Nodes (Multi-Node Only)

On each additional server node:

1. Mount the OS ISO:

mkdir -p /mnt/os
mount -o loop,ro /path/to/rhel-9.iso /mnt/os

# Configure local repository
cat > /etc/yum.repos.d/local.repo <<EOF
[local]
name=Local OS Repository
baseurl=file:///mnt/os/BaseOS
enabled=1
gpgcheck=0

[appstream]
name=AppStream Repository
baseurl=file:///mnt/os/AppStream
enabled=1
gpgcheck=0
EOF

dnf makecache

2. Mount the Installation ISOs:

# Mount Installation ISO
mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

# Mount Extras ISO
mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras

3. Join the cluster:

Run the join script:

/mnt/esb3027/join-server https://<primary-server-ip>:6443 <node-token>

Replace <primary-server-ip> with the IP address of the primary server and <node-token> with the token retrieved in Step 4.

4. Verify the node joined successfully:

kubectl get nodes

Repeat for each server node. A minimum of 3 server nodes is required for high availability.

Step 6: Join Agent Nodes (Optional)

On each agent node:

1. Mount the OS ISO:

mkdir -p /mnt/os
mount -o loop,ro /path/to/rhel-9.iso /mnt/os

# Configure local repository
cat > /etc/yum.repos.d/local.repo <<EOF
[local]
name=Local OS Repository
baseurl=file:///mnt/os/BaseOS
enabled=1
gpgcheck=0

[appstream]
name=AppStream Repository
baseurl=file:///mnt/os/AppStream
enabled=1
gpgcheck=0
EOF

dnf makecache

2. Mount the Installation ISOs:

# Mount Installation ISO
mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

# Mount Extras ISO
mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras

3. Join the cluster as an agent:

Run the join script:

/mnt/esb3027/join-agent https://<primary-server-ip>:6443 <node-token>

4. Verify the node joined successfully from an existing server node:

kubectl get nodes

Agent nodes provide additional workload capacity but do not participate in the control plane quorum.

Step 7: Load Container Images

On each node in the cluster:

/mnt/esb3027-extras/load-images

This script loads all container images from the Extras ISO into the local container runtime.

Important: This step must be performed on every node (primary server, additional servers, and agents) before deploying the Manager application.

Step 8: Verify Cluster Status

After all nodes are joined and images are loaded, verify the cluster is operational:

1. Verify all nodes are ready:

kubectl get nodes

Expected output:

NAME                 STATUS   ROLES                       AGE   VERSION
k3s-server-0         Ready    control-plane,etcd,master   5m    v1.33.4+k3s1
k3s-server-1         Ready    control-plane,etcd,master   3m    v1.33.4+k3s1
k3s-server-2         Ready    control-plane,etcd,master   2m    v1.33.4+k3s1
k3s-agent-1          Ready    <none>                      1m    v1.33.4+k3s1

2. Verify system pods in both namespaces are running:

# Check kube-system namespace (Kubernetes core components)
kubectl get pods -n kube-system

# Check longhorn-system namespace (distributed storage)
kubectl get pods -n longhorn-system

All pods should show Running status.

3. Verify container images are loaded:

crictl images | grep acd-manager

Step 9: Deploy the Manager Helm Chart

For complete instructions on deploying the CDN Manager Helm chart, including configuration file setup, MaxMind GeoIP database loading, TLS certificate configuration, deployment commands, and verification steps, see the Helm Chart Installation Guide.

This guide covers the common deployment steps that apply to all installation types. After completing the helm chart installation steps, proceed to Post-Installation below.

Post-Installation

After installation completes, proceed to the Next Steps guide for:

• Initial user configuration
• Accessing the web interfaces
• Configuring authentication
• Setting up monitoring

Accessing the System

Refer to the Accessing the System section in the Getting Started guide for service URLs and default credentials.

Note: A self-signed SSL certificate is deployed by default. You will need to accept the certificate warning in your browser.

Updating MaxMind GeoIP Databases

If using GeoIP-based routing, load the MaxMind databases:

/mnt/esb3027/generate-maxmind-volume

The utility will prompt for the database file locations and volume name. Reference the volume in your values.yaml:

manager:
  maxmindDbVolume: maxmind-geoip-2026-04

See the Operations Guide for database update procedures.

Troubleshooting

Image Pull Errors

If pods fail with image pull errors:

1. Verify the load-images script completed successfully on all nodes
2. Check container runtime image list:

   crictl images | grep <image-name>
   

3. Ensure image tags in Helm chart match tags on the Extras ISO

OS Package Errors

If the installer reports missing OS packages:

1. Verify OS ISO is mounted on the affected node
2. Check repository configuration:

   dnf repolist
   dnf info <package-name>
   

3. Ensure the ISO matches the installed OS version

Longhorn Volume Issues

If Longhorn volumes fail to mount:

1. Verify all nodes have the load-images script completed
2. Check Longhorn system pods:

   kubectl get pods -n longhorn-system
   

3. Review Longhorn UI via port-forward:

   kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80
   

Next Steps

After successful installation:

1. Next Steps Guide - Post-installation configuration
2. Configuration Guide - System configuration
3. Operations Guide - Day-to-day operational procedures
4. Troubleshooting Guide - Common issues and resolution

5.5 - Helm Chart Installation

Overview

This guide covers the common steps for deploying the CDN Manager Helm chart. These steps apply to all deployment types (single-node, multi-node, and air-gapped) after the Kubernetes cluster is fully operational.

Prerequisites: This guide assumes the Kubernetes cluster is already installed and all system pods are running. If you haven’t installed the cluster yet, refer to:

• Single-Node Installation for lab environments
• Multi-Node Installation for production deployments
• Air-Gapped Deployment for air-gapped environments

Prerequisites

Before proceeding, verify the following:

• Cluster operational: All nodes show Ready status
• System pods running: All pods in kube-system and longhorn-system namespaces are Running
• ISO mounted: Installation ISO is mounted at /mnt/esb3027
• Extras ISO mounted (air-gapped only): Extras ISO is mounted at /mnt/esb3027-extras and images are loaded on all nodes

Step 1: Create Configuration File

The installation ISO includes environment-specific configuration files as the recommended starting points. Choose the file that matches your deployment type:

After copying, edit ~/values.yaml for your environment. The two fields that must be updated in either file are:

global:
  hosts:
    manager:
      - host: manager.example.com   # Your manager hostname

zitadel:
  zitadel:
    configmapConfig:
      ExternalDomain: manager.example.com   # Must match global.hosts.manager[0].host exactly

Important: global.hosts.manager[0].host and zitadel.zitadel.configmapConfig.ExternalDomain must match exactly or authentication will fail due to CORS policy violations.

For a full description of what each file configures and the complete list of required changes per environment, see the Configuration Guide.

Complete reference: /mnt/esb3027/values.yaml documents every available option with its default value. Use this as a reference, not as a starting point for your configuration.

Split configuration files: For better organisation, split your configuration into multiple files and specify them with repeated --values flags. Later files override earlier files:

helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml \
  --values ~/values-tls.yaml

Step 2: Load MaxMind GeoIP Databases (Optional)

If you plan to use GeoIP-based routing or validation features, load the MaxMind GeoIP databases. The following databases are used by the manager:

• GeoIP2-City.mmdb - The City Database
• GeoLite2-ASN.mmdb - The ASN Database
• GeoIP2-Anonymous-IP.mmdb - The VPN and Anonymous IP Database

Create the Kubernetes volume using the helper utility:

/mnt/esb3027/generate-maxmind-volume

The utility will prompt for:

1. Location of GeoIP2-City.mmdb
2. Location of GeoLite2-ASN.mmdb
3. Location of GeoIP2-Anonymous-IP.mmdb
4. Name of the volume

After running this command, reference the volume in your configuration file:

manager:
  maxmindDbVolume: maxmind-db-volume

Replace maxmind-db-volume with the volume name you specified when running the utility.

Tip: When naming the volume, include a revision number or date (e.g., maxmind-db-volume-2026-04 or maxmind-db-volume-v2). This simplifies future updates: create a new volume with an updated name, update the values.yaml to reference the new volume, and delete the old volume after verification.

Step 3: Configure TLS Certificates (Optional)

For production deployments, configure a valid TLS certificate from a trusted Certificate Authority (CA). A self-signed certificate is deployed by default if no certificate is provided.

Method 1: Create TLS Secret Manually

Create a Kubernetes TLS secret with your certificate and key:

kubectl create secret tls acd-manager-tls --cert=tls.crt --key=tls.key

Method 2: Helm-Managed Secret

Add the certificate directly to your values.yaml:

ingress:
  secrets:
    acd-manager-tls: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
  tls:
    - hosts:
        - manager.example.com
      secretName: acd-manager-tls

Configuring All Ingress Controllers

All ingress controllers must be configured with the same certificate secret and hostname:

ingress:
  hostname: manager.example.com
  tls: true
  secretName: acd-manager-tls

zitadel:
  ingress:
    tls:
      - hosts:
          - manager.example.com
        secretName: acd-manager-tls

confd:
  ingress:
    hostname: manager.example.com
    tls: true
    secretName: acd-manager-tls

mib-frontend:
  ingress:
    hostname: manager.example.com
    tls: true
    secretName: acd-manager-tls

Important: The hostname must match the first entry in global.hosts.manager for Zitadel CORS compatibility. The secret name has a maximum length of 53 characters.

Step 4: Deploy the Manager Helm Chart

Deploy the CDN Manager application:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml

Real-time output: By default, helm install runs silently until completion. To see real-time output during deployment, add the --debug flag:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml --debug

Monitor deployment:

kubectl get pods --watch

Wait for all pods to show Running status before proceeding.

Timeout handling: The default Helm timeout is 5 minutes. If the installation fails due to a rollout timeout, retry with a larger timeout value:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml --timeout 10m

Retry failed installation: If a previous installation attempt failed and you receive an error that the release name is already in use, uninstall the previous release before retrying:

helm uninstall acd-manager
helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml

Step 5: Verify Deployment

Verify all application pods are running:

kubectl get pods

Expected Output: Single-Node

NAME                                              READY   STATUS      RESTARTS   AGE
acd-manager-5b98d569d9-abc12                      1/1     Running     0          3m
acd-manager-confd-6fb78548c4-xnrh4                1/1     Running     0          3m
acd-manager-gateway-8bc8446fc-chs26               1/1     Running     0          3m
acd-manager-kafka-controller-0                    2/2     Running     0          3m
acd-manager-metrics-aggregator-76d96c4964-lwdcj   1/1     Running     0          3m
acd-manager-mib-frontend-7bdb69684b-6qxn8         1/1     Running     0          3m
acd-manager-postgresql-0                          1/1     Running     0          3m
acd-manager-redis-master-0                        2/2     Running     0          3m
acd-manager-redis-replicas-0                      2/2     Running     0          3m
acd-manager-selection-input-5fb694b857-qxt67      1/1     Running     0          3m
acd-manager-zitadel-8448b4c4fc-2pkd8              1/1     Running     0          3m
acd-manager-zitadel-init-hh6j7                    0/1     Completed   0          4m
acd-manager-zitadel-setup-nwp8k                   0/2     Completed   0          4m
alertmanager-0                                    1/1     Running     0          3m
grafana-6d948cfdc6-77ggk                          1/1     Running     0          3m
victoria-metrics-agent-dc87df588-tn8wv            1/1     Running     0          3m
victoria-metrics-alert-757c44c58f-kk9lp           1/1     Running     0          3m
victoria-metrics-longterm-server-0                1/1     Running     0          3m
victoria-metrics-server-0                         1/1     Running     0          3m

Expected Output: Multi-Node

NAME                                              READY   STATUS      RESTARTS   AGE
acd-cluster-postgresql-1                          1/1     Running     0               11m
acd-cluster-postgresql-2                          1/1     Running     0               11m
acd-cluster-postgresql-3                          1/1     Running     0               10m
acd-manager-5b98d569d9-2pbph                      1/1     Running     0               3m
acd-manager-5b98d569d9-m54f9                      1/1     Running     0               3m
acd-manager-5b98d569d9-pq26f                      1/1     Running     0               3m
acd-manager-confd-6fb78548c4-xnrh4                1/1     Running     0               3m
acd-manager-gateway-8bc8446fc-chs26               1/1     Running     0               3m
acd-manager-gateway-8bc8446fc-wzrml               1/1     Running     0               3m
acd-manager-kafka-controller-0                    2/2     Running     0               3m
acd-manager-kafka-controller-1                    2/2     Running     0               3m
acd-manager-kafka-controller-2                    2/2     Running     0               3m
acd-manager-metrics-aggregator-76d96c4964-lwdcj   1/1     Running     2               3m
acd-manager-mib-frontend-7bdb69684b-6qxn8         1/1     Running     0               3m
acd-manager-mib-frontend-7bdb69684b-pkjrw         1/1     Running     0               3m
acd-manager-redis-master-0                        2/2     Running     0               3m
acd-manager-redis-replicas-0                      2/2     Running     0               3m
acd-manager-selection-input-5fb694b857-qxt67      1/1     Running     2               3m
acd-manager-zitadel-8448b4c4fc-2pkd8              1/1     Running     0               3m
acd-manager-zitadel-8448b4c4fc-vchp9              1/1     Running     0               3m
acd-manager-zitadel-init-hh6j7                    0/1     Completed   0               4m
acd-manager-zitadel-setup-nwp8k                   0/2     Completed   0               4m
alertmanager-0                                    1/1     Running     0               3m
grafana-6d948cfdc6-77ggk                          1/1     Running     0               3m
telegraf-54779f5f46-2jfj5                         1/1     Running     0               3m
victoria-metrics-agent-dc87df588-tn8wv            1/1     Running     0               3m
victoria-metrics-alert-757c44c58f-kk9lp           1/1     Running     0               3m
victoria-metrics-longterm-server-0                1/1     Running     0               3m
victoria-metrics-server-0                         1/1     Running     0               3m

Pod Distribution Verification

Verify pods are distributed across nodes:

kubectl get pods -o wide

Expected Behavior

• Init pods (such as zitadel-init and zitadel-setup) will show Completed status after successful initialization. This is expected behavior.
• Multi-node deployments: Some pods may enter CrashLoopBackoff state during initial deployment depending on the timing of other containers starting up. This is expected behavior as some services wait for dependencies (such as databases or Kafka) to become available. The deployment should stabilize automatically after a few minutes.
• Restart counts: Some pods may show restart counts as they wait for dependencies to become available. This is normal during initial deployment.

Next Steps

After successful deployment:

1. Next Steps Guide - Post-installation configuration
2. Getting Started Guide - Accessing the system
3. Configuration Guide - System configuration
4. Operations Guide - Day-to-day operations

5.6 - Upgrade Guide

Overview

This guide describes the procedure for upgrading the AgileTV CDN Manager (ESB3027) to a newer version. The upgrade process involves updating the Kubernetes cluster components and redeploying the Helm chart with the new version.

Prerequisites

Backup Requirements

Before beginning any upgrade, ensure you have:

• PostgreSQL Backup: Verify recent backups are available via the Cloudnative PG operator
• Configuration Backup: Save your current values.yaml file(s)
• TLS Certificates: Ensure certificate files are backed up
• MaxMind Volumes: Note the current volume names if using GeoIP databases

Version Compatibility

Review the Release Notes for the target version to check for:

• Breaking changes requiring manual intervention
• Required intermediate upgrade steps
• New configuration options that should be set

Cluster Health

Verify the cluster is healthy before upgrading:

kubectl get nodes
kubectl get pods
kubectl get pvc

All nodes should show Ready status and all pods should be Running (or Completed for job pods).

Upgrade Methods

There are three upgrade methods available. Choose the one that best fits your situation:

Method Selection Guidance:

• Rolling Upgrade (Method 1) is the default choice for most upgrades. Use this for patch releases (e.g., 1.6.0 → 1.6.1) and even minor version upgrades (e.g., 1.4.0 → 1.6.0) where no breaking changes are documented. This method preserves all existing resources and performs an in-place update. Note: This method supports Helm’s automatic rollback (helm rollback) if the upgrade fails, allowing quick recovery to the previous state.

• Clean Upgrade (Method 2) is recommended for major version upgrades (e.g., 1.x → 2.x) or when the release notes indicate significant component changes. This method ensures all resources are recreated with the new version, avoiding potential issues with stale configurations. Also use this method when troubleshooting upgrade failures from Method 1.

• Full Reinstall (Method 3) should only be used when a completely clean cluster state is required. This includes troubleshooting persistent cluster-level issues, recovering from failed upgrades that cannot be rolled back, or when migrating between significantly different deployment configurations. This method requires verified backups and should be planned for extended downtime.

Upgrade Steps

Method 1: Rolling Upgrade (Recommended)

This method performs an in-place rolling upgrade with minimal downtime. All upgrade commands are executed from the primary server node.

Step 1: Obtain the New Installation ISO

Unmount the old ISO (if mounted) and mount the new installation ISO:

umount /mnt/esb3027 2>/dev/null || true
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

Replace X.Y.Z with the target version number.

Step 2: Update Containers and Cluster Software

Run the installation script to update the container images and cluster software:

/mnt/esb3027/install

Wait for the script to complete.

Step 2b: Air-Gapped Environments (If Applicable)

If deploying in an air-gapped environment, also mount and load the extras ISO:

# Mount the Extras ISO
mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras

# Load container images from the extras ISO
/mnt/esb3027-extras/load-images

Replace X.Y.Z with the target version number.

Step 4: Review and Update Configuration

Compare the default values.yaml from the new ISO with your current configuration:

diff /mnt/esb3027/values.yaml ~/values.yaml

Update your configuration file to include any new required settings. Common updates include:

# ~/values.yaml
global:
  hosts:
    manager:
      - host: manager.example.com
    routers:
      - name: director-1
        address: 192.0.2.1

zitadel:
  zitadel:
    ExternalDomain: manager.example.com

# Add any new required settings for the target version

Important: Do not modify settings unrelated to the upgrade unless specifically documented in the release notes.

Step 5: Update MaxMind GeoIP Volumes (If Applicable)

If you use MaxMind GeoIP databases, use the utility from the new ISO to create an updated volume:

/mnt/esb3027/generate-maxmind-volume

Update your values.yaml to reference the new volume name:

manager:
  maxmindDbVolume: maxmind-geoip-2026-04

Tip: Using dated or versioned volume names (e.g., maxmind-geoip-2026-04) allows you to create new volumes during upgrades and delete old ones after verification.

Step 6: Update TLS Certificates (If Needed)

If your TLS certificates need renewal or the new version requires certificate updates, create or update the secret:

kubectl create secret tls acd-manager-tls --cert=tls.crt --key=tls.key --dry-run=client -o yaml | kubectl apply -f -

Step 7: Upgrade the Helm Release

Perform a Helm upgrade with the new chart:

helm upgrade acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml

Note: The upgrade performs a rolling update of each deployment in the chart. Deployments are upgraded one at a time, with pods being terminated and recreated sequentially. StatefulSets (PostgreSQL, Kafka, Redis) roll out one pod at a time to maintain data availability.

Monitor the upgrade progress:

kubectl get pods --watch

Wait for all pods to stabilize and show Running status before considering the upgrade complete. Some pods may temporarily enter CrashLoopBackoff during the transition as they wait for dependencies to become available.

Step 8: Verify the Upgrade

Check the deployed version:

helm list
kubectl get deployments -o wide

Verify application functionality:

• Access the MIB Frontend and confirm it loads
• Test API connectivity
• Verify Grafana dashboards are accessible
• Check that Zitadel authentication is working

Step 9: Clean Up

After confirming the upgrade is successful:

1. Unmount the old ISO (if still mounted):

   umount /mnt/esb3027
   

2. Delete old MaxMind volumes (if replaced):

   kubectl get pvc
   kubectl delete pvc <old-volume-name>
   

3. Remove old configuration files if no longer needed.

Method 2: Clean Upgrade (Helm Uninstall/Install)

This method removes the existing Helm release before installing the new version. This is useful for major version upgrades or when troubleshooting upgrade issues. All upgrade commands are executed from the primary server node.

Warning: This method causes brief downtime as all resources are deleted before reinstallation.

Step 1: Obtain the New Installation ISO

Mount the new installation ISO:

umount /mnt/esb3027 2>/dev/null || true
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

Step 2: Backup Configuration

Save your current Helm values:

helm get values acd-manager -o yaml > ~/values-backup.yaml

Step 3: Uninstall the Existing Release

Remove the existing Helm release:

helm uninstall acd-manager

Wait for pods to terminate:

kubectl get pods --watch

Note: Helm uninstall does not remove PersistentVolumes (PVs) or PersistentVolumeClaims (PVCs). All data stored in PostgreSQL, Kafka, Redis, and Longhorn volumes is preserved during the uninstall process. When the new version is installed, it will reattach to the existing PVCs and restore data automatically.

Step 4: Review and Update Configuration

Compare the default values.yaml from the new ISO with your configuration:

diff /mnt/esb3027/values.yaml ~/values.yaml

Update your configuration file as needed.

Step 5: Install the New Release

Install the new version:

helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml

Monitor the deployment:

kubectl get pods --watch

Wait for all pods to stabilize before proceeding.

Step 6: Verify the Upgrade

Verify the upgrade as described in Method 1, Step 8.

Method 3: Full Reinstall (Cluster Rebuild)

This method completely removes Kubernetes and reinstalls from scratch. Use only for cluster rebuilds or when other upgrade methods fail.

Warning: This method causes extended downtime and permanent data loss. The K3s uninstall process destroys all Longhorn PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). All data stored in PostgreSQL, Kafka, Redis, and application volumes will be permanently lost. Verified backups are required before proceeding.

Warning: This method should only be used when necessary. Ensure you have verified backups before proceeding.

Step 1: Stop Kubernetes Services

On all nodes (server and agent), stop the K3s service:

systemctl stop k3s

Step 2: Uninstall K3s (Server Nodes Only)

On the primary server node first, then each additional server node:

/usr/local/bin/k3s-uninstall.sh

Step 3: Clean Up Residual State (All Nodes)

On all nodes, remove residual state:

/usr/local/bin/k3s-kill-all.sh
rm -rf /var/lib/rancher/k3s/*

Warning: This removes all cluster data including Longhorn PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). All data stored in PostgreSQL, Kafka, Redis, and application volumes will be permanently lost. Ensure verified backups are available before proceeding.

Step 4: Reinstall K3s Cluster and Deploy Manager

Follow the installation procedure in the Installation Guide to reinstall the cluster and deploy the Helm chart. At this point, you are in the same state as a fresh installation:

• Primary server installation
• Additional server joins (if applicable)
• Agent joins (if applicable)
• Helm chart deployment

Note: The K3s node token is regenerated during reinstallation. Retrieve the new token from /var/lib/rancher/k3s/server/node-token on the primary server after installation if you need to join additional nodes.

Rollback Procedure

Rollback procedures vary by upgrade method:

Method 1 (Rolling Upgrade)

Use Helm’s built-in rollback command:

helm rollback acd-manager

This reverts to the previous Helm release revision automatically.

Or manually redeploy the previous version:

helm upgrade acd-manager /mnt/esb3027-old/helm/charts/acd-manager \
  --values ~/values.yaml

Note: If you use multiple --values files for organization, ensure they are specified in the same order as the original installation.

Method 2 (Clean Upgrade)

Reinstall the previous version:

helm uninstall acd-manager
helm install acd-manager /mnt/esb3027-old/helm/charts/acd-manager \
  --values ~/values-backup.yaml

Method 3 (Full Reinstall)

Rollback requires repeating the full cluster reinstall procedure using the old installation ISO. Follow Method 3 steps with the previous version’s ISO. Ensure verified backups are available before attempting.

Troubleshooting

Pods Fail to Start

1. Check pod status and events:

   kubectl describe pod <pod-name>
   kubectl get events --sort-by='.lastTimestamp'
   

2. Review pod logs:

   kubectl logs <pod-name>
   kubectl logs <pod-name> -p  # Previous instance logs
   

Database Migration Issues

If PostgreSQL migrations fail:

1. Check Cloudnative PG cluster status:

   kubectl get clusters
   kubectl describe cluster <cluster-name>
   

2. Review migration job logs:

   kubectl get jobs
   kubectl logs job/<migration-job-name>
   

Helm Upgrade Fails

If helm upgrade fails:

1. Check Helm release status:

   helm status acd-manager
   helm history acd-manager
   

2. Review the error message for specific failures

3. Attempt rollback if necessary

Post-Upgrade

After a successful upgrade:

1. Review the Release Notes for any post-upgrade tasks
2. Update monitoring dashboards if new metrics are available
3. Test all critical functionality
4. Document the upgrade in your change management system

Next Steps

After completing the upgrade:

1. Next Steps Guide - Review post-installation tasks
2. Operations Guide - Day-to-day operational procedures
3. Release Notes - Review new features and changes

5.7 - Next Steps

Overview

After completing the installation of the AgileTV CDN Manager (ESB3027), several post-installation configuration tasks must be performed before the system is ready for production use. This guide walks you through the essential next steps.

Prerequisites

Before proceeding, ensure:

• The CDN Manager Helm chart is successfully deployed
• All pods are in Running status
• You have network access to the cluster hostname or IP
• You have the default credentials available

Step 1: Access Zitadel Console

The first step is to configure user authentication through Zitadel Identity and Access Management (IAM).

1. Navigate to the Zitadel Console:

   https://<manager-host>/ui/console
   

   Replace <manager-host> with your configured hostname (e.g., manager.local or manager.example.com).

   Important: The <manager-host> must match the first entry in global.hosts.manager from your Helm values exactly. Zitadel uses name-based virtual hosting and CORS validation. If the hostname does not match, authentication will fail.

2. Log in with the default administrator credentials (also listed in the Glossary):

   • Username: admin@agiletv.dev
   • Password: Password1!

3. Important: If prompted to configure Multi-Factor Authentication (MFA), you must skip this step for now. MFA is not currently supported. Attempting to configure MFA may lock you out of the administrator account.

4. Security Recommendation: After logging in, create a new administrator account with proper roles. Once verified, disable or delete the default admin@agiletv.dev account. For details on required roles and administrator permissions, see Zitadel’s Administrator Documentation.

Step 2: Configure SMTP Settings (Recommended)

Zitadel requires an SMTP server to send email notifications and perform email validations.

1. In the Zitadel Console, navigate to Settings > Default Settings

2. Configure the SMTP settings:

   • SMTP Host: Your mail server hostname
   • SMTP Port: Typically 587 (TLS) or 465 (SSL)
   • SMTP Username: Mail account username
   • SMTP Password: Mail account password
   • Sender Address: Email address for outgoing mail (e.g., noreply@example.com)

3. Save the configuration

Note: Without SMTP configuration, email-based user validation and password recovery features will not function.

Step 3: Create Additional User Accounts

Create user accounts for operators and administrators:

Tip: For detailed guidance on managing users, roles, and permissions in the Zitadel Console, see Zitadel’s User Management Documentation.

1. In the Zitadel Console, navigate to Users > Add User

2. Fill in the user details:

   • Username: Unique username
   • First Name: User’s first name
   • Last Name: User’s last name
   • Email: User’s email address (this is their login username)

   Known Issue: Due to a limitation in this release of Zitadel, the username must match the local part (the portion before the @) of the email address. For example, if the email is foo@example.com, the username must be foo.

   If these do not match, Zitadel may allow login with the mismatched local part while blocking the full email address. For instance, if username is foo but email is foo.bar@example.com, login with foo@example.com may succeed while foo.bar@example.com is blocked.

   Workaround: Always ensure the username matches the email local part exactly.

3. Important: The following options must be configured:

   • Email Verified: Check this box to skip email verification
   • Set Initial Password: Enter a temporary password for the user

   Note: If you configured SMTP settings in Step 2, the user will receive an email asking to verify their address and set their initial password. If SMTP is not configured, you must check the “Email Verified” box and set an initial password manually, otherwise the user account will not be enabled.

4. Click Create User

5. Provide the user with:

   • Their username
   • The temporary password (if set manually)
   • The Zitadel Console URL

6. Instruct the user to change their password on first login

Step 4: Configure User Roles and Permissions

Zitadel manages roles and permissions for accessing the CDN Manager:

1. In the Zitadel Console, navigate to Roles

2. Assign appropriate roles to users:

   • Admin: Full administrative access
   • Operator: Operational access without administrative functions
   • Viewer: Read-only access

3. To assign a role:

   • Select the user
   • Click Add Role
   • Select the appropriate role
   • Save the assignment

Step 5: Access the MIB Frontend

The MIB Frontend is the web-based configuration GUI for CDN operators:

1. Navigate to the MIB Frontend:

   https://<manager-host>/gui
   

2. Log in using your Zitadel credentials

3. Verify you can access the configuration interface

Step 6: Verify API Access

Test API connectivity to ensure the system is functioning:

curl -k https://<manager-host>/api/v1/health/ready

Expected response:

{
  "status": "ready"
}

See the API Guide for detailed API documentation.

Step 7: Configure TLS Certificates (If Not Done During Installation)

For production deployments, a valid TLS certificate from a trusted Certificate Authority should be configured. If you did not configure TLS certificates during installation, refer to Step 12: Configure TLS Certificates in the Installation Guide.

Step 8: Set Up Monitoring and Alerting

Configure monitoring dashboards and alerting:

1. Access Grafana:

   • Navigate to https://<manager-host>/grafana
   • Log in with default credentials (also listed in the Glossary):
     • Username: admin
     • Password: edgeware

2. Review Pre-built Dashboards:

   • System health dashboards are included by default
   • CDN metrics dashboards show routing and usage statistics

   Note: CDN Director instances automatically have DNS names configured for use in Grafana dashboards. The DNS name is derived from the name field in global.hosts.routers with .external appended. For example, a router named my-router-1 will have the DNS name my-router-1.external in Grafana configuration.

Step 9: Verify Kafka and PostgreSQL Health

Ensure the data layer components are healthy:

kubectl get pods

Verify the following pods are running:

All pods should show Running status with no restarts.

Step 10: Configure Availability Zones (Optional)

For improved network performance, configure availability zones to enable Topology Aware Hints. This optimizes service-to-pod routing by keeping traffic within the same zone when possible.

See the Performance Tuning Guide for detailed instructions on:

• Labeling nodes with zone and region topology
• Verifying topology configuration
• Requirements for Topology Aware Hints to activate
• Integration with pod anti-affinity rules

Note: This step is optional. If zone labels are not configured, the system will fall back to random load-balancing.

Step 11: Review System Configuration

Verify the initial configuration:

1. Review Helm Values:

   helm get values acd-manager -o yaml
   

2. Check Ingress Configuration:

   kubectl get ingress
   

3. Verify Service Endpoints:

   kubectl get endpoints
   

Step 12: Document Your Deployment

Maintain documentation for your deployment:

• Cluster hostname and IP addresses
• Configuration file locations
• User accounts and roles created
• TLS certificate expiration dates
• Backup procedures and schedules
• Monitoring and alerting contacts

Next Steps

After completing post-installation configuration:

1. Configuration Guide - Detailed system configuration options
2. Operations Guide - Day-to-day operational procedures
3. Metrics & Monitoring Guide - Comprehensive monitoring setup
4. API Guide - REST API reference and integration examples

Troubleshooting

Cannot Access Zitadel Console

• Verify DNS resolution or hosts file configuration
• Check that Traefik ingress is running: kubectl get pods -n kube-system | grep traefik
• Review Traefik logs: kubectl logs -n kube-system -l app.kubernetes.io/name=traefik

Authentication Failures

• Verify Zitadel pods are healthy: kubectl get pods | grep zitadel
• Check Zitadel logs: kubectl logs <zitadel-pod-name>
• Ensure the external domain matches your hostname in Zitadel configuration

MIB Frontend Not Loading

• Verify MIB Frontend pods are running: kubectl get pods | grep mib-frontend
• Check for connectivity issues to Confd and API services
• Review browser console for JavaScript errors

API Returns 401 Unauthorized

• Verify you have a valid bearer token
• Check token expiration
• Ensure Zitadel authentication is functioning

For additional troubleshooting assistance, refer to the Troubleshooting Guide.

6 - Configuration Guide

Overview

The CDN Manager is deployed via Helm chart with configuration supplied through values.yaml files. This guide explains the configuration structure, how to apply changes, and provides a reference for all configurable options.

Configuration Files

The installation ISO provides three configuration files at /mnt/esb3027/:

You only need to specify fields that differ from the defaults. Helm applies configuration hierarchically — values from your file override the chart’s built-in defaults, and any key you omit retains its default value.

Lab Configuration (values-lab.yaml)

values-lab.yaml is the recommended starting point for single-node lab, acceptance testing, and demonstration deployments. It pre-configures settings appropriate for a constrained single-node environment:

• Single Kafka controller replica (the default 3 replicas require 3 separate nodes to satisfy pod anti-affinity rules)
• Single Zitadel replica
• Self-signed TLS by default, with real certificate configuration commented out for reference
• Minimal resource requests suited to a single node

Copy the file to a writable location and edit it before deploying:

cp /mnt/esb3027/values-lab.yaml ~/values.yaml

The minimum required changes are:

1. Set global.hosts.manager[0].host to your node’s hostname or IP address
2. Set zitadel.zitadel.configmapConfig.ExternalDomain to the same value

These two values must match exactly or authentication will fail due to CORS policy violations. See Global Settings for details.

Production Configuration (values-production.yaml)

values-production.yaml is the recommended starting point for multi-node production deployments across a minimum three-node cluster. It pre-configures settings appropriate for a high-availability environment:

• Three Zitadel replicas spread across nodes for HA
• Production-grade resource requests and limits for all major components
• Kafka with a dedicated single-replica StorageClass (avoiding unnecessary triple-redundancy on top of Kafka’s own quorum)
• Manager HPA configured to scale between 3 and 8 replicas
• TLS certificate configuration with clearly marked placeholders

Copy the file to a writable location and edit it before deploying:

cp /mnt/esb3027/values-production.yaml ~/values.yaml

The minimum required changes before deploying are:

1. Set global.hosts.manager[0].host to your primary manager hostname
2. Set zitadel.zitadel.configmapConfig.ExternalDomain to the same hostname
3. Replace the placeholder TLS certificate and key in the ingress.secrets section, and update the secretName values in mibFrontend.ingress.extraTls and zitadel.ingress.tls to match
4. Update global.hosts.routers with your CDN Director instances

See TLS Configuration and Global Settings for full details.

Hardware requirements: For per-node hardware specifications, refer to the System Requirements Guide. The System Requirements Guide is the authoritative source — the hardware comments in the header of values-production.yaml may not reflect the current requirements.

Complete Reference (values.yaml)

The full default values file at /mnt/esb3027/values.yaml documents every configurable option with its default value and inline comments. Use this as a reference when looking up available settings or understanding what the environment-specific files override.

Note: values.yaml is not intended to be used directly as your deployment configuration. Use values-lab.yaml or values-production.yaml as your starting point instead.

Configuration Merging

Helm merges configuration files from left to right, with later files overriding earlier values. This allows you to split your configuration into multiple files — for example, keeping TLS certificates separate from the main configuration:

# Multiple files merged left-to-right
helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml \
  --values ~/values-tls.yaml

Individual Value Overrides

For temporary changes, you can override individual values with --set:

helm upgrade acd-manager /mnt/esb3027/helm/charts/acd-manager \
  --values ~/values.yaml \
  --set manager.logLevel=debug

Note: Using --set is discouraged for permanent changes, as the same arguments must be specified for every Helm operation.

Applying Configuration

Initial Installation

helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml

Updating Configuration

helm upgrade acd-manager /mnt/esb3027/helm/charts/acd-manager \
  --values ~/values.yaml

Dry Run

Before applying changes, validate the configuration with a dry run:

helm upgrade acd-manager /mnt/esb3027/helm/charts/acd-manager \
  --values ~/values.yaml \
  --dry-run

Rollback

If an upgrade fails, rollback to the previous revision:

# View revision history
helm history acd-manager

# Rollback to previous revision
helm rollback acd-manager

# Rollback to specific revision
helm rollback acd-manager <revision_number>

Note: Rollback reverts the Helm release but does not modify your values.yaml file. You must manually revert configuration file changes.

Force Reinstall

If an upgrade fails and rollback is not sufficient, you can perform a clean reinstall:

helm uninstall acd-manager
helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml

Warning: This is service-affecting as all pods will be destroyed and recreated.

Configuration Reference

Global Settings

The global section contains cluster-wide settings. The most critical configuration is global.hosts.

global:
  hosts:
    manager:
      - host: manager.local
    routers:
      - name: default
        address: 127.0.0.1
    edns_proxy: []
    geoip: []

Important: The first entry in global.hosts.manager must match zitadel.zitadel.ExternalDomain exactly. Zitadel enforces CORS protection, and authentication will fail if these do not match.

Manager Configuration

Core Manager API server settings:

Manager Resources

The chart supports both resource presets and explicit resource specifications:

Note: For production workloads, explicitly set manager.resources rather than using presets.

Manager Datastore

manager:
  datastore:
    type: redis
    namespace: "cdn_manager_ds"
    default_ttl: ""
    compression: zstd

Manager Discovery

manager:
  discovery: []
  # Example:
  # - namespace: "other"
  #   hosts:
  #     - other-host1
  #     - other-host2
  #   pattern: "other-.*"

Manager Tuning

manager:
  tuning:
    enable_cache_control: true
    cache_control_max_age: "5m"
    cache_control_miss_max_age: ""

Manager Container Arguments

manager:
  args:
    - --config-file=/etc/manager/config.toml
    - http-server

Gateway Configuration

NGinx Gateway settings for external Director communication:

MIB Frontend Configuration

Web-based configuration GUI settings:

Confd Configuration

Confd settings for configuration management:

VictoriaMetrics Configuration

Time-series database for metrics:

Ingress Configuration

Traffic exposure settings:

Ingress Extra Paths

The chart includes default extra paths for Confd and GeoIP:

ingress:
  extraPaths:
    - path: /confd
      pathType: Prefix
      backend:
        service:
          name: acd-manager-gateway
          port:
            name: http
    - path: /geoip
      pathType: Prefix
      backend:
        service:
          name: acd-manager-gateway
          port:
            name: http

TLS Certificate Secrets

For production TLS certificates:

ingress:
  secrets:
    - name: manager.local-tls
      key: |-
        -----BEGIN RSA PRIVATE KEY-----
        ...
        -----END RSA PRIVATE KEY-----
      certificate: |-
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
  tls: true

Resource Configuration

Resource Presets

Predefined resource configurations for common deployment sizes:

Note: Limits are calculated as requests plus 50% (except for xlarge/2xlarge and ephemeral-storage).

Custom Resources

Override preset with custom values:

manager:
  resources:
    requests:
      cpu: "300m"
      memory: "512Mi"
    limits:
      cpu: "1"
      memory: "1Gi"

Note:

• CPU values use millicores (1000m = 1 core)
• Memory values use binary SI units (1024Mi = 1GiB)
• Requests represent minimum guaranteed resources
• Limits represent maximum consumable resources

Capacity Planning

When sizing resources:

• Requests determine scheduling (node must have available capacity)
• Limits prevent resource starvation
• Maintain 20-30% cluster headroom for scaling
• Total capacity = sum of all requests × replica count + headroom

Security Contexts

Pod Security Context

manager:
  podSecurityContext:
    enabled: true
    fsGroup: 1001
    fsGroupChangePolicy: Always
    sysctls: []
    supplementalGroups: []

Container Security Context

manager:
  containerSecurityContext:
    enabled: true
    runAsUser: 1001
    runAsGroup: 1001
    runAsNonRoot: true
    readOnlyRootFilesystem: true
    privileged: false
    allowPrivilegeEscalation: false
    capabilities:
      drop: ["ALL"]
    seccompProfile:
      type: "RuntimeDefault"

Health Probes

Probe Types

Default Probe Configuration

Liveness Probe

manager:
  livenessProbe:
    enabled: true
    initialDelaySeconds: 5
    periodSeconds: 30
    timeoutSeconds: 10
    failureThreshold: 5
    successThreshold: 1
    httpGet:
      path: /api/v1/health/alive
      port: http

Readiness Probe

manager:
  readinessProbe:
    enabled: true
    initialDelaySeconds: 5
    periodSeconds: 10
    timeoutSeconds: 7
    failureThreshold: 3
    successThreshold: 1
    httpGet:
      path: /api/v1/health/ready
      port: http

Startup Probe

manager:
  startupProbe:
    enabled: true
    initialDelaySeconds: 0
    periodSeconds: 5
    timeoutSeconds: 3
    failureThreshold: 10
    successThreshold: 1
    httpGet:
      path: /api/v1/health/alive
      port: http

Autoscaling Configuration

Horizontal Pod Autoscaler (HPA)

manager:
  autoscaling:
    hpa:
      enabled: true
      minReplicas: 3
      maxReplicas: 8
      targetCPU: 50
      targetMemory: 80

Network Policy

networkPolicy:
  enabled: true
  allowExternal: true
  allowExternalEgress: true
  addExternalClientAccess: true

Pod Affinity and Anti-Affinity

manager:
  podAffinityPreset: ""
  podAntiAffinityPreset: soft
  nodeAffinityPreset:
    type: ""
    key: ""
    values: []
  affinity: {}

Service Configuration

service:
  type: ClusterIP
  ports:
    http: 80
  annotations:
    service.kubernetes.io/topology-mode: Auto
  externalTrafficPolicy: Cluster
  sessionAffinity: None

Persistence Configuration

persistence:
  enabled: false
  mountPath: /agiletv/manager/data
  storageClass: ""
  accessModes:
    - ReadWriteOnce
  size: 8Gi

RBAC and Service Account

rbac:
  create: false
  rules: []

serviceAccount:
  create: true
  name: ""
  automountServiceAccountToken: true
  annotations: {}

Metrics

metrics:
  enabled: false
  serviceMonitor:
    enabled: false
    namespace: ""
    annotations: {}
    labels: {}
    interval: ""
    scrapeTimeout: ""

Next Steps

After configuration:

1. Installation Guide - Deploy with your configuration
2. Operations Guide - Day-to-day management
3. Performance Tuning Guide - Optimize system performance
4. Architecture Guide - Understand component relationships

7 - Performance Tuning Guide

Overview

This guide provides performance tuning recommendations for the AgileTV CDN Manager (ESB3027). While the default configuration is suitable for most deployments, certain environments may benefit from additional optimizations.

Network Topology Optimization

Topology Aware Hints

The CDN Manager uses Kubernetes Topology Aware Hints to prefer routing pods in the same zone as the source of network traffic. This reduces cross-zone latency and improves overall system responsiveness.

How It Works

When nodes are labeled with topology zones, Kubernetes automatically routes traffic to pods in the same zone when possible. This is particularly beneficial for:

• Low-latency requirements: Keeps traffic local to reduce round-trip time
• Cost optimization: Reduces cross-zone data transfer costs in cloud environments
• Load distribution: Prevents hotspots by distributing load across zones

Configuring Availability Zones

Each node must have zone and region labels applied for Topology Aware Hints to function:

# Label a node with a zone
kubectl label nodes <node-name> topology.kubernetes.io/zone=us-east-1a

# Label a node with a region
kubectl label nodes <node-name> topology.kubernetes.io/region=us-east-1

Replace <node-name> with your actual node names and adjust the zone/region values to match your deployment geography.

Note: Labels applied via kubectl label are automatically persistent and will survive node restarts.

Verify Topology Configuration

Verify labels are applied:

kubectl get nodes --show-labels | grep topology.kubernetes.io

Verify EndpointSlices are being generated with hints:

kubectl get endpointslices

Requirements for Topology Aware Hints

For Topology Aware Hints to activate:

• Minimum Nodes: At least one node must be labeled with each zone referenced by endpoints
• Symmetry: The control plane checks for sufficient CPU capacity across zones to balance traffic
• Zone Coverage: All zones with endpoints should have at least one ready node

Integration with Pod Anti-Affinity

Topology labels complement the pod anti-affinity rules already configured in the Helm chart:

• Pod Anti-Affinity: Handles pod-to-node placement to ensure high availability
• Topology Aware Hints: Handles service-to-pod traffic routing to keep requests within the same zone

Together, these features optimize both placement and routing for improved performance.

Fallback Behavior

If zone labels are not configured, the system falls back to random load-balancing across all available pods. This is functionally correct but may result in:

• Increased cross-zone traffic
• Higher latency for some requests
• Less predictable performance characteristics

Kernel Network Tuning (sysctl)

For high-throughput deployments, tuning Linux kernel network parameters can significantly improve connection handling and overall system performance. These settings are particularly beneficial for environments with high connection rates or large numbers of concurrent connections.

Recommended sysctl Settings

Apply the following settings to optimize network performance:

# Networking
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2048
net.ipv4.tcp_max_syn_backlog = 2048

# Connection Tracking
net.netfilter.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_tcp_timeout_established = 1200

# Port Reuse
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1

# Memory Buffers
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608

Setting Descriptions

Applying Settings

Temporary (Until Reboot)

Apply settings immediately but they will be lost on reboot:

sudo sysctl -w net.core.somaxconn=1024
sudo sysctl -w net.core.netdev_max_backlog=2048
# ... repeat for each parameter

Persistent (Across Reboots)

Add settings to /etc/sysctl.conf or a file in /etc/sysctl.d/:

# Create a dedicated config file
cat <<EOF | sudo tee /etc/sysctl.d/99-cdn-manager.conf
# CDN Manager Network Tuning
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2048
net.ipv4.tcp_max_syn_backlog = 2048
net.netfilter.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_tcp_timeout_established = 1200
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
EOF

# Apply all settings
sudo sysctl -p /etc/sysctl.d/99-cdn-manager.conf

Kubernetes Considerations

For Kubernetes deployments, these sysctl settings can be applied via:

1. Node-level configuration: Use DaemonSets or node provisioning scripts
2. Pod-level safe sysctls: Some sysctls can be set per-pod via securityContext.sysctls
3. Container runtime configuration: Configure via container runtime options

Note that some sysctls require privileged containers or node-level configuration.

Monitoring Impact

After applying these settings, monitor:

• Connection establishment rates
• TIME_WAIT socket count: netstat -n | grep TIME_WAIT | wc -l
• Connection tracking table usage: cat /proc/sys/net/netfilter/nf_conntrack_count
• Network buffer utilization via Grafana dashboards

Resource Configuration

Horizontal Pod Autoscaler (HPA)

The default HPA configuration is tuned for production workloads. For environments with variable load, consider adjusting the scale metrics:

For detailed HPA configuration, see the Architecture Guide.

Resource Requests and Limits

Ensure resource requests and limits are appropriately sized for your workload. Under-provisioned resources can cause:

• Pod evictions during high load
• Increased latency due to CPU throttling
• Slow scaling responses

Refer to the Configuration Guide for preset configurations and planning guidance.

Database Optimization

PostgreSQL

The PostgreSQL cluster is managed by the Cloudnative PG operator. For improved performance:

• Connection Pooling: The application uses connection pooling by default
• Replica Usage: Read queries can be offloaded to replicas for read-heavy workloads
• Backup Scheduling: Schedule backups during low-traffic periods to minimize I/O impact

Redis

Redis provides in-memory caching for sessions and ephemeral state:

• Memory Allocation: Ensure sufficient memory for cache hit rates
• Persistence: RDB snapshots are enabled; adjust frequency based on durability needs

Kafka

Kafka handles event streaming for selection input and metrics:

• Partition Count: Default partitions are sized for typical workloads
• Replication Factor: Production deployments use 3 replicas for fault tolerance
• Consumer Groups: The Selection Input Worker is limited to one consumer per partition

Monitoring Performance

Key Metrics to Watch

Monitor the following metrics for performance insights:

• API Response Time: Track via Grafana dashboards
• Pod CPU/Memory Usage: Identify resource bottlenecks
• Kafka Lag: Monitor consumer lag for selection input processing
• Database Connections: Watch for connection pool exhaustion

Grafana Dashboards

Pre-built dashboards are available at https://<manager-host>/grafana:

• System Health: Overall cluster and application health
• CDN Metrics: Routing and usage statistics
• Resource Utilization: CPU, memory, and network usage per component

Troubleshooting Performance Issues

High Latency

1. Check pod distribution across nodes: kubectl get pods -o wide
2. Verify topology labels are applied: kubectl get nodes --show-labels
3. Review network latency between nodes
4. Check for resource contention: kubectl top pods

Slow Scaling

1. Verify HPA is enabled: kubectl get hpa
2. Check cluster capacity for scheduling new pods
3. Review HPA metrics: kubectl describe hpa acd-manager

Database Performance

1. Check PostgreSQL cluster status: kubectl get pods -l app=postgresql
2. Review slow query logs (if enabled)
3. Monitor connection pool usage

Next Steps

After reviewing performance tuning:

1. Architecture Guide - Understand component interactions
2. Configuration Guide - Detailed configuration options
3. Metrics & Monitoring Guide - Comprehensive monitoring setup
4. Troubleshooting Guide - Resolve performance issues

8 - Operations Guide

Overview

This guide covers day-to-day operational procedures for managing the AgileTV CDN Manager (ESB3027). Topics include routine maintenance, backup procedures, log management, and common operational tasks.

Prerequisites

Before performing operations, ensure you have:

• kubectl access to the cluster
• helm CLI installed
• Access to the node where values.yaml is stored
• Appropriate RBAC permissions for administrative tasks

Cluster Access

There are two supported methods for accessing the Kubernetes cluster:

1. SSH to a Server Node (Recommended for operations staff) - SSH into any Server node and run kubectl commands directly
2. Remote kubectl - Install kubectl on your local machine and configure it to connect to the cluster remotely

Method 1: SSH to Server Node (Recommended)

The kubectl command-line tool is pre-configured on all Server nodes and can be used directly without additional setup:

# SSH to any Server node
ssh root@<server-ip>

# Run kubectl commands directly
kubectl get nodes
kubectl get pods

This method is recommended for day-to-day operations as it requires no local configuration and provides direct access to the cluster.

Method 2: Remote kubectl from Local Machine

To use kubectl from your local workstation or laptop:

Step 1: Install kubectl

Download and install kubectl for your operating system:

• Official Documentation: Install kubectl
• macOS (Homebrew): brew install kubectl
• Linux: Download from the official Kubernetes release page
• Windows: Download from the official Kubernetes release page

Step 2: Copy kubeconfig from Server Node

# Copy kubeconfig from any Server node
scp root@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Step 3: Update kubeconfig

Edit the kubeconfig file to point to the correct server address:

# Replace localhost with the actual server IP
# macOS/Linux:
sed -i '' 's/127.0.0.1/<server-ip>/g' ~/.kube/config  # macOS
sed -i 's/127.0.0.1/<server-ip>/g' ~/.kube/config    # Linux

# Or manually edit ~/.kube/config and change:
# server: https://127.0.0.1:6443
# to:
# server: https://<server-ip>:6443

Step 4: Verify connectivity

kubectl get nodes

Managing Multiple Clusters

If you manage multiple Kubernetes clusters from the same machine, you can maintain multiple kubeconfig files:

# Set KUBECONFIG environment variable to include multiple config files
export KUBECONFIG=~/.kube/config-prod:~/.kube/config-lab

# View all contexts
kubectl config get-contexts

# Switch between clusters
kubectl config use-context <context-name>

# View current context
kubectl config current-context

For more information, see the official Kubernetes documentation: Organizing Cluster Access

Helm Commands

Helm releases are managed cluster-wide:

# List all releases
helm list

# View release history
helm history acd-manager

# Get deployed values
helm get values acd-manager -o yaml

# Get deployed manifest
helm get manifest acd-manager

Note: If using remote kubectl, ensure helm is installed on your local machine. See Helm Installation for instructions.

Helm Commands

Helm releases are managed cluster-wide:

# List all releases
helm list

# View release history
helm history acd-manager

# Get deployed values
helm get values acd-manager -o yaml

# Get deployed manifest
helm get manifest acd-manager

Backup Procedures

PostgreSQL Backup

PostgreSQL is managed by the Cloudnative PG operator, which provides continuous backup capabilities.

# Check backup status
kubectl get backup

# Create manual backup
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: manual-backup-$(date +%Y%m%d-%H%M%S)
spec:
  cluster:
    name: acd-cluster-postgresql
EOF

# List available backups
kubectl get backup -o wide

# Restore from backup (requires downtime)
# See Upgrade Guide for restore procedures

Longhorn Volume Backups

Longhorn provides snapshot and backup capabilities for persistent volumes:

# List all volumes
kubectl get volumes -n longhorn-system

# Create snapshot via Longhorn UI
# Port-forward to Longhorn UI (do not expose via ingress)
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

# Access: http://localhost:8080
# WARNING: Longhorn UI grants access to sensitive storage information
# and should never be exposed through the ingress controller

Accessing Internal Services

For debugging and troubleshooting, you may need direct access to internal services.

PostgreSQL

PostgreSQL is managed by the Cloudnative PG operator. Connection details are stored in the acd-cluster-postgresql-app Secret:

# View connection details
kubectl describe secret acd-cluster-postgresql-app

# Extract individual fields
PG_HOST=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.host}' | base64 -d)
PG_USER=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.username}' | base64 -d)
PG_PASS=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.password}' | base64 -d)
PG_DB=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.dbname}' | base64 -d)

# Connect via psql
kubectl exec -it acd-cluster-postgresql-0 -- psql -U $PG_USER -d $PG_DB

Secret fields: The CNPG operator populates the following fields: username, password, host, port, dbname, uri, jdbc-uri, fqdn-uri, fqdn-jdbc-uri, pgpass.

Redis

Redis runs on port 6379 with no authentication:

# Connect via redis-cli
kubectl exec -it acd-manager-redis-master-0 -- redis-cli

# Or connect from another pod
kubectl run redis-test --rm -it --image=redis -- redis-cli -h acd-manager-redis-master

Kafka

kafka-topics.sh –bootstrap-server :9095 –list

The selection_input topic is pre-configured for selection input events.

Kubernetes Port Forwarding

For accessing internal Kubernetes services that are not exposed via ingress or services, use kubectl port-forward to create a secure tunnel from your local machine to the service.

Basic Port Forwarding

# Forward local port to a service
kubectl port-forward -n <namespace> svc/<service-name> <local-port>:<service-port>

# Example: Forward local port 8080 to Grafana (port 3000)
kubectl port-forward -n default svc/acd-manager-grafana 8080:3000

Note: “Local” refers to the machine where you run kubectl. This can be:

• A Server node in the cluster (common for administrative tasks)
• A remote machine with kubectl configured to access the cluster

Accessing the Forwarded Service

Once the port-forward is established, access the service at http://localhost:<local-port> from the machine where you ran kubectl port-forward.

If running on a Server node: To access the forwarded port from your local workstation:

• Ensure the firewall on the Server node allows traffic on the forwarded port from your network
• Use the Server node’s IP address instead of localhost from your workstation

# From your workstation (if firewall allows)
curl http://<server-node-ip>:<local-port>

For simplicity, consider running port-forward from your local machine (if kubectl is configured for remote cluster access) rather than from a Server node.

Background Port Forwarding

To run port-forward in the background:

kubectl port-forward -n <namespace> svc/<service-name> <local-port>:<service-port> &

Security Considerations

Port forwarding is recommended for:

• Administrative interfaces (e.g., Longhorn UI) that should not be publicly exposed
• Debugging and troubleshooting internal services
• Temporary access to services without modifying ingress configuration

The port-forward tunnel remains active only while the kubectl port-forward command is running. Press Ctrl+C to terminate the tunnel.

Example: The Longhorn storage UI is intentionally not exposed via ingress due to security risks. Access it via port-forward:

kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

Then navigate to http://localhost:8080 in your browser.

Longhorn Storage

Longhorn is a distributed block storage system for Kubernetes that provides persistent volumes for stateful applications such as PostgreSQL and Kafka.

Architecture

Longhorn deploys controller and replica engines on each node, forming a distributed storage system. When a volume is created, Longhorn replicates data across multiple nodes to ensure durability even in the event of node failures.

Storage Protocols:

• iSCSI: Used for standard Read-Write-Once (RWO) volumes
• NFS: Used for Read-Write-Many (RWX) volumes that can be mounted by multiple pods simultaneously

Configuration

The CDN Manager deploys Longhorn with a single replica configuration, which differs from the Longhorn default of 3 replicas. This configuration is optimized for the cluster architecture where:

• Pod-node affinity is configured to schedule pods on the same node as their persistent volume data
• This optimizes I/O performance by reducing network traffic
• Data locality is maintained while still providing volume portability

Capacity Planning

Longhorn storage requires an additional 30% capacity headroom for internal operations and scaling. If less than 30% of the total partition capacity is available, Longhorn may mark volumes as “full” and prevent further writes.

For detailed storage requirements and disk partitioning guidance, see the System Requirements Guide.

Configuration Backup

Always backup your Helm values before making changes:

# Export current values
helm get values acd-manager -o yaml > ~/values-backup-$(date +%Y%m%d).yaml

# Backup custom values files
cp ~/values.yaml ~/values-backup-$(date +%Y%m%d).yaml

Backup Schedule Recommendations

Updating MaxMind GeoIP Databases

The MaxMind GeoIP databases (GeoIP2-City, GeoLite2-ASN, GeoIP2-Anonymous-IP) are used for GeoIP-based routing and validation features. These databases should be updated periodically to ensure accurate IP geolocation data.

Prerequisites

• Updated MaxMind database files (.mmdb format) obtained from MaxMind
• Access to the cluster via kubectl
• Helm CLI installed

Update Procedure

Step 1: Create New Volume with Updated Databases

Run the volume generation utility with a unique volume name that includes a revision identifier:

# Mount the installation ISO if not already mounted
mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

# Generate new volume with updated databases
/mnt/esb3027/generate-maxmind-volume

When prompted:

1. Provide the paths to the three database files:
   • GeoIP2-City.mmdb
   • GeoLite2-ASN.mmdb
   • GeoIP2-Anonymous-IP.mmdb
2. Enter a unique volume name with a revision number or date, for example:
   • maxmind-geoip-2026-04
   • maxmind-geoip-v2

Tip: Using a revision-based naming convention simplifies rollback if needed.

Step 2: Update Helm Configuration

Edit your values.yaml file to reference the new volume:

manager:
  maxmindDbVolume: maxmind-geoip-2026-04

Replace maxmind-geoip-2026-04 with the volume name you specified in Step 1.

Step 3: Apply Configuration Update

Upgrade the Helm release with the updated configuration: