AgileTV CDN Manager (esb3027)

• 1: Getting Started
• 2: System Requirements Guide
• 3: Networking Guide
• 4: Architecture Guide
• 5: Installation Guide
• 5.1: Installation Checklist
• 5.2: Single-Node Installation
• 5.3: Multi-Node Installation
• 5.4: Air-Gapped Deployment Guide
• 5.5: Upgrade Guide
• 5.6: Next Steps
• 6: Configuration Guide
• 7: Performance Tuning Guide
• 8: Operations Guide
• 9: Metrics & Monitoring Guide
• 10: API Guide
• 10.1: Authentication API
• 10.2: Health API
• 10.3: Selection Input API
• 10.4: Data Store API
• 10.5: Subnets API
• 10.6: Routing API
• 10.7: Discovery API
• 10.8: Metrics API
• 10.9: Configuration API
• 10.10: Operator UI API
• 10.11: OpenAPI Specification
• 11: Troubleshooting Guide
• 12: Glossary

1 - Getting Started

Overview

The AgileTV CDN Manager (product code ESB3027) is a cloud-native control plane for managing CDN deployments. It provides centralized orchestration for authentication, configuration, routing, and metrics collection across CDN infrastructure.

Before You Start:

• Deployment type: Lab (single-node) or Production (multi-node)? See Installation Guide
• Hardware: Nodes meeting specifications for your deployment type
• OS: RHEL 9 or compatible clone (Oracle Linux, AlmaLinux, Rocky Linux)
• Software: Installation ISO from AgileTV customer portal; Extras ISO for air-gapped
• Network: Firewall ports configured per Networking Guide

Deployment Models

Functionality remains consistent across deployment models.

Prerequisites

• Installation ISO: Obtain esb3027-acd-manager-X.Y.Z.iso from AgileTV customer portal
• Extras ISO (air-gapped): Obtain esb3027-acd-manager-extras-X.Y.Z.iso for offline installations
• OS: RHEL 9 or compatible clone (Oracle Linux, AlmaLinux, Rocky Linux)
• Kubernetes familiarity: Basic understanding of pods, deployments, and Helm charts

For detailed hardware, network, and operating system requirements, see the System Requirements Guide.

Installation

Ready to install? The Installation Guide provides step-by-step procedures for both lab and production deployments:

• Lab/Single-Node: Quick deployment for testing and demonstrations
• Production/Multi-Node: High-availability cluster with 3+ nodes

See the Installation Guide to get started.

Accessing the System

Following successful deployment, the following interfaces are available:

All services are accessed via https://<cluster-host><path>.

Note: A self-signed SSL certificate is deployed by default. When accessing services through a browser, you will need to accept the self-signed certificate warning. For production deployments, configure a valid SSL certificate before exposing the system to users.

Initial user configuration is performed through Zitadel. Refer to the Configuration Guide for authentication setup procedures. For detailed guidance on managing users, roles, and permissions in the Zitadel Console, see Zitadel’s User Management Documentation.

Documentation Navigation

The following guides provide detailed information for specific operational tasks:

2 - System Requirements Guide

Overview

This document specifies the hardware, operating system, and networking requirements for deploying the AgileTV CDN Manager (ESB3027). Requirements vary based on deployment type and node role within the cluster.

Cluster Sizing

Production Deployments

Production deployments require a minimum of three nodes to achieve high availability. The cluster architecture employs distinct node roles:

Server nodes can be deployed in either Control Plane Only or Combined role configurations. The choice depends on your deployment requirements:

• Control Plane Only: Dedicated control plane nodes with lower resource requirements; requires separate Agent nodes for workloads
• Combined: Server nodes run both control plane and workloads; minimum 3 nodes required for HA

Why Use Control Plane Only Nodes?

Dedicated Control Plane Only nodes provide several benefits for larger deployments:

• Resource Isolation: Control plane components (etcd, API server, scheduler) run on dedicated hardware without competing with application workloads for CPU and memory
• Stability: Application workload spikes or misbehaving pods cannot impact control plane performance
• Security: Smaller attack surface on control plane nodes; fewer containers and services running
• Predictable Performance: Control plane responsiveness remains consistent regardless of application load
• Flexible Sizing: Control Plane Only nodes can use lower-specification hardware (2 cores, 4 GiB) since they don’t run application workloads

For most small to medium deployments, Combined role servers are simpler and more cost-effective. Control Plane Only nodes are recommended for larger deployments with significant workload requirements or where control plane stability is critical.

High Availability Considerations

Production deployments require 3 nodes running control plane (etcd) and 3 nodes capable of running workloads. These can be the same nodes (Combined role) or separate nodes (CP-Only + Agent).

Node Role Combinations:

Any combination works as long as you have 3 control plane nodes and 3 workload-capable nodes.

Note: Regardless of the deployment configuration, a minimum of 3 nodes capable of running workloads is required for production deployments. This ensures both high availability and sufficient capacity for application pods.

For detailed fault tolerance information and data replication strategies, see the Architecture Guide.

Hardware Requirements

Single-Node Lab Deployment

Lab deployments are intended for acceptance testing, demonstrations, and development only. These configurations are not suitable for production workloads.

Production Cluster - Server Node (Control Plane Only)

Server nodes dedicated to control plane functions have modest resource requirements:

These nodes run only control plane components and require separate Agent nodes to run application workloads.

Production Cluster - Server Node (Control Plane + Workloads)

Combined role nodes require resources for both control plane and application workloads:

Production Cluster - Agent Node

Agent nodes execute application workloads and require the following resources:

Storage Notes

* Disk Space: All disk space values must be available in the /var/lib/longhorn partition. It is recommended that /var/lib/longhorn be a separate partition on a fast SSD for optimal performance, though SSD is not strictly required.

Longhorn Capacity: Longhorn storage requires an additional 30% capacity headroom for internal operations and scaling. If less than 30% of the total partition capacity is available, Longhorn may mark volumes as “full” and prevent further writes. Plan disk capacity accordingly.

Storage Performance

For optimal performance, the following storage characteristics are recommended:

• Disk Type: SSD or NVMe storage for Longhorn volumes
• Filesystem: XFS or ext4 with default mount options
• Partition Layout: Dedicated /var/lib/longhorn partition for persistent storage

Virtual machines and bare-metal hardware are both supported. Nested virtualization (running multiple nodes under a single hypervisor) may impact performance and is not recommended for production deployments.

Operating System Requirements

Supported Operating Systems

The CDN Manager supports Red Hat Enterprise Linux and compatible distributions:

Compatible Clones

The following RHEL-compatible distributions are supported when major version requirements are satisfied:

• Oracle Linux 9
• AlmaLinux 9
• Rocky Linux 9

Air-Gapped Deployments

Important: For air-gapped deployments (no internet access), the OS installation ISO must be mounted on all nodes before running the installer or join commands. The installer needs to install one or more packages from the distribution’s repository.

Oracle Linux UEK Kernel

Note: For Oracle Linux 9.7 and later using the Unbreakable Enterprise Kernel (UEK), you must install the kernel-uek-modules-extra-netfilter-$(uname -r) package before running the installer:

# Mount OS ISO first (required for air-gapped)
mount -o loop /path/to/oracle-linux-9.iso /mnt/iso

# Install required kernel modules
dnf install kernel-uek-modules-extra-netfilter-$(uname -r)

This package provides netfilter kernel modules required by K3s and Longhorn.

SELinux

SELinux is supported when installed in “Enforcing” mode. The installation process will configure appropriate SELinux policies automatically.

Networking Requirements

Network Interface

Each cluster node must have at least one network interface card (NIC) configured as the default gateway. If the node lacks a pre-configured default route, one must be established prior to installation.

Port Requirements

The cluster requires the following network connectivity:

Important: Complete port requirements, network ranges, and firewall configuration procedures are provided in the Networking Guide. Do not expose VictoriaMetrics (8428, 8429), Grafana (3000), or PostgreSQL (5432) directly—access these services only through the secure HTTPS ingress (port 443).

Resource Planning

Calculating Cluster Capacity

When planning cluster capacity, consider the following factors:

1. Base Overhead: Kubernetes system components consume approximately 1-2 cores and 2-4 GiB memory per node
2. Application Workloads: Refer to individual component resource requirements in the Architecture Guide
3. Headroom: Maintain 20-30% resource headroom for workload spikes and automatic scaling

Scaling Considerations

The CDN Manager supports horizontal scaling for most components. The Horizontal Pod Autoscaler (HPA) can automatically adjust replica counts based on resource utilization. Detailed scaling guidance is available in the Architecture Guide.

Example Production Deployment

A minimal production deployment with 3 server nodes (combined role) and 2 agent nodes would require:

This configuration provides:

• High availability (survives loss of 1 server node)
• Capacity for application workloads across all nodes
• Headroom for horizontal scaling

Next Steps

After verifying system requirements:

1. Review the Installation Guide for deployment procedures
2. Consult the Networking Guide for firewall configuration
3. Examine the Architecture Guide for component resource requirements

3 - Networking Guide

Overview

This guide describes the network architecture and firewall configuration requirements for the AgileTV CDN Manager (ESB3027). Proper network configuration is essential for cluster communication and external access to services.

Note: The installer script automatically detects if firewalld is enabled. If so, it will verify that the required inter-node ports are open through the firewall in the default zone before proceeding. If any required ports are missing, the installer will report an error and exit. Application service ports (such as Kafka, VictoriaMetrics, and Telegraf) are not checked by the installer as they are configurable.

Network Architecture

Physical Network

Each cluster node must have at least one network interface card (NIC) configured as the default gateway. If the node lacks a pre-configured default route, one must be established prior to installation.

Overlay Network

Kubernetes creates virtual network interfaces for pods that are typically not associated with any specific firewalld zone. The cluster uses the following network ranges:

Firewall regulations should target the primary physical interface. The overlay network traffic is handled by Flannel VXLAN.

IP Routing

Proper IP routing is critical for cluster communication. Ensure your network infrastructure allows routing between all subnets used by the cluster.

Port Requirements

Inter-Node Communication

The following ports must be permitted between all cluster nodes for Kubernetes and cluster infrastructure:

Application Services Ports

The following ports must be accessible for application services within the cluster:

External Access Ports

The following ports must be accessible from external clients to cluster nodes:

Firewall Configuration

firewalld Configuration

firewalld Configuration

For systems using firewalld, it is recommended to use separate zones for internal cluster traffic and external public access. This ensures that sensitive inter-node communication is restricted to the internal network.

1. Assign Interfaces to Zones: First, assign your network interfaces to the appropriate zones. For example, if eth0 is your public interface and eth1 is your internal cluster interface:

   firewall-cmd --permanent --zone=public --add-interface=eth0
   firewall-cmd --permanent --zone=internal --add-interface=eth1
   

2. Configure Firewall Rules: The following commands configure the minimum required firewall rules.

   # Inter-node communication (Internal Zone)
   firewall-cmd --permanent --zone=internal --add-port=2379-2380/tcp
   firewall-cmd --permanent --zone=internal --add-port=6443/tcp
   firewall-cmd --permanent --zone=internal --add-port=8472/udp
   firewall-cmd --permanent --zone=internal --add-port=10250/tcp
   firewall-cmd --permanent --zone=internal --add-port=5001/tcp
   firewall-cmd --permanent --zone=internal --add-port=9500-9503/tcp
   firewall-cmd --permanent --zone=internal --add-port=8500-8504/tcp
   firewall-cmd --permanent --zone=internal --add-port=10000-30000/tcp
   firewall-cmd --permanent --zone=internal --add-port=3260/tcp
   firewall-cmd --permanent --zone=internal --add-port=2049/tcp
   
   # Pod and service networks (Internal Zone)
   firewall-cmd --permanent --zone=internal --add-source=10.42.0.0/16
   firewall-cmd --permanent --zone=internal --add-source=10.43.0.0/16
   
   # External access (Public Zone)
   firewall-cmd --permanent --zone=public --add-port=80/tcp
   firewall-cmd --permanent --zone=public --add-port=443/tcp
   firewall-cmd --permanent --zone=public --add-port=9095/tcp
   firewall-cmd --permanent --zone=public --add-port=6379/tcp
   firewall-cmd --permanent --zone=public --add-port=8125/tcp
   firewall-cmd --permanent --zone=public --add-port=8125/udp
   
   # Apply changes
   firewall-cmd --reload
   

   For more restrictive configurations, you can scope rules to specific source subnets using --add-source=<subnet> within the internal zone.

Internal Application Ports (Optional)

For internal cluster communication, the following ports may be opened if direct application access is required:

firewall-cmd --permanent --add-port=9092/tcp

Note: This port is used for internal Kafka cluster communication only.

Security Warning: Do not expose VictoriaMetrics (8428, 8429), or PostgreSQL (5432) directly. These services require authentication and their direct ports do not use TLS connections, creating a security risk. Always access these services through the secure HTTPS ingress (port 443).

Externally Accessible Application Ports: The following application ports are safe for external access and are already configured in the External Access section:

Verification

Verify firewall rules are applied:

firewall-cmd --list-all

Verify ports are accessible between nodes:

# From one node, test connectivity to another
nc -zv <node-ip> 6443
nc -zv <node-ip> 8472

Kubernetes Port Forwarding

For accessing internal Kubernetes services that are not exposed via ingress or services, use kubectl port-forward to create a secure tunnel from your local machine to the service.

Basic Port Forwarding

# Forward local port to a service
kubectl port-forward -n <namespace> svc/<service-name> <local-port>:<service-port>

# Example: Forward local port 8080 to Grafana (port 3000)
kubectl port-forward -n default svc/acd-manager-grafana 8080:3000

Note: “Local” refers to the machine where you run kubectl. This can be:

• A Server node in the cluster (common for administrative tasks)
• A remote machine with kubectl configured to access the cluster

Accessing the Forwarded Service

Once the port-forward is established, access the service at http://localhost:<local-port> from the machine where you ran kubectl port-forward.

If running on a Server node: To access the forwarded port from your local workstation, you need to:

1. Ensure the firewall on the Server node allows traffic on the forwarded port from your network
2. Use the Server node’s IP address instead of localhost from your workstation

# From your workstation (if firewall allows)
curl http://<server-node-ip>:<local-port>

For simplicity, consider running port-forward from your local machine (if kubectl is configured for remote cluster access) rather than from a Server node.

Background Port Forwarding

To run port-forward in the background:

kubectl port-forward -n <namespace> svc/<service-name> <local-port>:<service-port> &

Security Considerations

Port forwarding is recommended for:

• Administrative interfaces (e.g., Longhorn UI) that should not be publicly exposed
• Debugging and troubleshooting internal services
• Temporary access to services without modifying ingress configuration

The port-forward tunnel remains active only while the kubectl port-forward command is running. Press Ctrl+C to terminate the tunnel.

Example: The Longhorn storage UI is intentionally not exposed via ingress due to security risks. Access it via port-forward:

kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

Then navigate to http://localhost:8080 in your browser.

Network Security Considerations

Network Segmentation

For production deployments, consider network segmentation:

• Management Network: Dedicated network for Kubernetes control plane traffic
• Application Network: Separate network for application service traffic
• External Network: Public-facing network for ingress traffic

Traffic Encryption

• All external traffic uses HTTPS (TLS 1.2 or higher)
• Internal cluster traffic uses Flannel VXLAN encryption (if enabled)
• Database connections (PostgreSQL, Redis) are internal to the cluster

Access Control

• External access is limited to ports 80 and 443 by default
• Application service ports should not be exposed externally
• Use Kubernetes NetworkPolicies for fine-grained pod-to-pod traffic control

Troubleshooting

Nodes Cannot Communicate

1. Verify firewall rules allow inter-node traffic:

   firewall-cmd --list-all
   

2. Test connectivity between nodes:

   ping <node-ip>
   nc -zv <node-ip> 6443
   

3. Check network routing:

   ip route
   

Pods Cannot Reach Services

1. Verify Flannel is running:

   kubectl get pods -n kube-system | grep flannel
   

2. Check VXLAN interface:

   ip link show flannel.1
   

3. Verify pod network routes:

   ip route | grep 10.42
   

External Access Fails

1. Verify ingress controller is running:

   kubectl get pods -n kube-system | grep traefik
   

2. Check ingress configuration:

   kubectl get ingress
   

3. Verify external firewall allows ports 80 and 443

Next Steps

After configuring networking:

1. Installation Guide - Proceed with cluster installation
2. System Requirements - Review hardware and OS requirements
3. Architecture Guide - Understand component communication patterns

4 - Architecture Guide

Overview

The AgileTV CDN Manager (ESB3027) is a cloud-native Kubernetes application designed for managing CDN operations. This guide provides a detailed description of the system architecture, component interactions, and scaling considerations.

High-Level Architecture

The CDN Manager follows a microservices architecture deployed on Kubernetes. The system is organized into logical layers:

graph LR
    Clients[API Clients] --> Ingress[Ingress Controller]
    Ingress --> Manager[Core Manager]
    Ingress --> Frontend[MIB Frontend]
    Ingress --> Grafana[Grafana]
    Manager --> Redis[(Redis)]
    Manager --> Kafka[(Kafka)]
    Manager --> PostgreSQL[(PostgreSQL)]
    Manager --> Zitadel[Zitadel IAM]
    Manager --> Confd[Configuration Service]
    Grafana --> VM[(VictoriaMetrics)]
    Confd -.-> Gateway[NGinx Gateway]
    Gateway --> Director[CDN Director]

Component Architecture

Ingress Layer

The ingress layer manages all incoming traffic to the cluster:

Traffic flow:

• API clients and Operator UI connect via the Ingress Controller at /api and /gui paths respectively
• Grafana dashboards are accessed via the Ingress Controller at /grafana
• Zitadel authentication console is accessed via the Ingress Controller at /ui/console
• MIB Frontend uses NGinx Gateway when communicating with external Confd instances on CDN Director nodes

Application Services

The application layer contains the core CDN Manager services:

Data Layer

The data layer provides persistent and ephemeral storage:

External Integrations

Detailed Component Descriptions

Core Manager

The Core Manager is the central application server that exposes the REST API. It is implemented in Rust using the Actix-web framework.

Key Responsibilities:

• Authentication and session management via Zitadel
• Configuration document storage and retrieval
• Selection input CRUD operations
• Routing rule evaluation and GeoIP lookups
• Service discovery for CDN Directors and edge servers
• Operator UI helper endpoints

API Endpoints:

• /api/v1/auth/* - Authentication (login, token, logout)
• /api/v1/configuration - Configuration management
• /api/v1/selection_input/* - Selection input operations
• /api/v2/selection_input/* - Enhanced selection input with list operations
• /api/v1/routing/* - Routing evaluation and validation
• /api/v1/discovery/* - Host and namespace discovery
• /api/v1/metrics - System metrics
• /api/v1/health/* - Liveness and readiness probes
• /api/v1/operator_ui/* - Operator helper endpoints

Runtime Modes: The Core Manager supports multiple runtime modes, each deployed as a separate container:

• http-server - Primary HTTP API server (default)
• metrics-aggregator - Background worker for metrics collection
• selection-input - Background worker for Kafka selection input consumption

MIB Frontend

The MIB Frontend provides a web-based GUI for configuration management.

Key Features:

• Intuitive web interface for CDN configuration
• Real-time configuration validation
• Integration with Zitadel for SSO authentication
• Uses NGinx Gateway for external Director communication

Confd (Configuration Service)

Confd provides routing configuration services and synchronizes with the Core Manager application.

Key Responsibilities:

• Hosts the service configuration for routing decisions
• Provides API and CLI for configuration management
• Synchronizes routing configuration with Core Manager
• Maintains configuration state in PostgreSQL

Selection Input Worker

The Selection Input Worker processes selection input events from the Kafka stream.

Key Responsibilities:

• Consumes messages from the selection_input Kafka topic
• Validates and transforms input data
• Updates configuration in the data store
• Maintains message ordering within partitions

Scaling Limitation: The Selection Input Worker cannot be scaled beyond a single consumer per Kafka partition, as message ordering must be preserved.

Metrics Aggregator

The Metrics Aggregator collects and processes metrics from CDN components.

Key Responsibilities:

• Polls metrics from Director instances
• Aggregates usage statistics
• Writes data to VictoriaMetrics (Analytics) for dashboards
• Writes long-term data to VictoriaMetrics (Billing) for compliance

Telegraf

Telegraf is deployed as a DaemonSet to collect host-level metrics.

Key Responsibilities:

• CPU, memory, disk, and network metrics from each node
• Container-level resource usage
• Kubernetes cluster metrics
• Forwards metrics to VictoriaMetrics

Grafana

Grafana provides visualization and dashboard capabilities.

Features:

• Pre-built dashboards for CDN monitoring
• Custom dashboard support
• VictoriaMetrics as data source
• Alerting integration with Alertmanager

Access: https://<host>/grafana

Alertmanager

Alertmanager handles alert routing and notifications.

Key Responsibilities:

• Receives alerts from Grafana and other sources
• Deduplicates and groups alerts
• Routes to notification channels (email, webhook, etc.)
• Manages alert silencing and inhibition

Data Storage

Redis

Redis provides in-memory storage for:

• User sessions and authentication tokens
• Ephemeral configuration cache
• Real-time state synchronization

Deployment: Master + read replicas for high availability

Kafka

Kafka provides durable event streaming for:

• Selection input events
• Metrics data streams
• Inter-service communication

Deployment: Controller cluster with 3 replicas for production, 1 replica for lab deployments

Node Affinity: Kafka replicas must be scheduled on separate nodes to ensure high availability. The Helm chart configures pod anti-affinity rules to enforce this distribution.

Topics:

• selection_input - Selection input events
• metrics - Metrics data streams

Note: For lab/single-node deployments, the Kafka replica count must be set to 1 in the Helm values. Production deployments require 3 replicas for fault tolerance.

PostgreSQL

PostgreSQL provides persistent storage for:

• Configuration documents
• User and permission data
• System state

Deployment: 3-node cluster managed by Cloudnative PG (CNPG) operator

High Availability: The CNPG operator manages automatic failover and ensures high availability:

• One primary node handles read/write operations
• Two replica nodes provide redundancy and can be promoted to primary on failure
• Automatic failover occurs within seconds of primary node failure
• Synchronous replication ensures data consistency

Note: The PostgreSQL cluster is deployed and managed automatically by the CNPG operator. Manual intervention is typically not required for normal operations.

VictoriaMetrics

Two VictoriaMetrics instances serve different purposes:

VictoriaMetrics (Analytics):

• Real-time and short-term metrics storage
• Supports Grafana dashboards
• Retention: Configurable (typically 30-90 days)

VictoriaMetrics (Billing):

• Long-term metrics retention
• Billing and license compliance data
• Retention: Minimum 1 year

Authentication and Authorization

Zitadel Integration

Zitadel provides identity and access management:

Authentication Flow:

1. User accesses MIB Frontend or API
2. Redirected to Zitadel for authentication
3. Zitadel validates credentials and issues session token
4. Session token exchanged for access token
5. Access token included in API requests (Bearer authentication)

Default Credentials: See the Glossary for default login credentials.

Access Paths:

• Zitadel Console: /ui/console
• API authentication: /api/v1/auth/*

CORS Configuration

Zitadel enforces Cross-Origin Resource Sharing (CORS) policies. The external hostname configured in Zitadel must match the first entry in global.hosts.manager in the Helm values.

Network Architecture

Traffic Flow

graph TB
    External[External Clients] --> Ingress[Ingress Controller]
    External --> Redis[(Redis)]
    External --> Kafka[(Kafka)]
    External --> Telegraf[Telegraf]
    Ingress --> Manager[Core Manager]
    Ingress --> Frontend[MIB Frontend]
    Ingress --> Grafana[Grafana]
    Ingress --> Zitadel[Zitadel]

Note: Certain services (Redis, Kafka, Telegraf) can be accessed directly by external clients without traversing the ingress controller. This is typically used for metrics collection, event streaming, and direct data access scenarios.

Internal Communication

All internal services communicate over the Kubernetes overlay network (Flannel VXLAN). Services discover each other via Kubernetes DNS.

External Communication

• CDN Directors: Accessed via NGinx Gateway for simplified routing
• MaxMind GeoIP: Local database files (no external calls)

Scaling

Horizontal Pod Autoscaler (HPA)

The following components support automatic horizontal scaling via HPA:

Note: HPA is enabled by default in the Helm chart. The default configuration is tuned for production deployments. Adjust min/max values based on expected load and available cluster capacity.

Manual Scaling

Components can also be scaled manually by setting replica counts in the Helm values:

manager:
  replicaCount: 3
mib-frontend:
  replicaCount: 2

Important: When manually setting replica counts, you must disable the Horizontal Pod Autoscaler (HPA) for the corresponding component. If HPA remains enabled, it will override manual replica settings. To disable HPA, set autoscaling.hpa.enabled: false for the component in your Helm values.

Components That Do Not Scale

The following components do not support horizontal scaling:

Node Scaling

Additional Agent nodes can be added to the cluster at any time to increase workload capacity. Kubernetes automatically schedules pods to nodes with available resources.

Cluster Balancing

The CDN Manager deployment includes the Kubernetes Descheduler to maintain balanced resource utilization across cluster nodes:

• Automatic Rebalancing: The descheduler periodically analyzes pod distribution and evicts pods from overutilized nodes
• Node Balance: Helps prevent resource hotspots by redistributing workloads across available nodes
• Integration with HPA: Works in conjunction with Horizontal Pod Autoscaler to optimize both pod count and placement

The descheduler runs as a background process and does not require manual intervention under normal operating conditions.

Resource Configuration

For detailed resource preset configurations and planning guidance, see the Configuration Guide.

High Availability

Server Node Redundancy

Production deployments require a minimum of 3 Server nodes:

• Survives loss of 1 server node
• Maintains quorum for etcd and Kafka

For enhanced availability, use 5 Server nodes:

• Survives loss of 2 server nodes
• Recommended for critical production environments

For large-scale deployments, 7 or more Server nodes can be used:

• Survives loss of 3+ server nodes
• Suitable for high-capacity production environments

Pod Distribution

Kubernetes automatically distributes pods across nodes to maximize availability:

• Pods with the same deployment are scheduled on different nodes when possible
• Pod Disruption Budgets (PDB) ensure minimum availability during maintenance

Data Replication

Longhorn Storage: Longhorn volumes are configured with a single replica by default. Pod scheduling is configured with node affinity to prefer scheduling pods on the same node as their persistent volume data. This approach optimizes I/O performance while maintaining data locality.

Next Steps

After understanding the architecture:

1. Installation Guide - Deploy the CDN Manager
2. Configuration Guide - Configure components for your environment
3. Operations Guide - Day-to-day operational procedures
4. Performance Tuning Guide - Optimize system performance
5. Metrics & Monitoring - Set up monitoring and alerting

5 - Installation Guide

Overview

This guide provides detailed instructions for installing the AgileTV CDN Manager (ESB3027) in various deployment scenarios. The installation process varies depending on the target environment and desired configuration.

Estimated Installation Time:

Actual installation time may vary depending on hardware performance, network speed, and whether air-gapped procedures are required.

Note: These estimates assume the operating system is already installed on all nodes. OS installation is outside the scope of this guide.

Installation Types

Installation Process Summary

The installation follows a sequential process:

1. Prepare the host system - Verify requirements and mount the installation ISO
2. Install the Kubernetes cluster - Deploy K3s, Longhorn storage, and PostgreSQL
3. Join additional nodes (production only) - Expand the cluster for HA or capacity
4. Deploy the Manager application - Install the CDN Manager Helm chart
5. Post-installation configuration - Configure authentication, networking, and users

Quick Links

Prerequisites

Before beginning installation, ensure the following requirements are met:

• Hardware: Nodes meeting the System Requirements including CPU, memory, and disk specifications
• Operating System: RHEL 9 or compatible clone (details); air-gapped deployments require the OS ISO mounted on all nodes
• Network: Proper firewall configuration between nodes (port requirements, firewall configuration)
• Software: Installation ISO obtained from AgileTV; air-gapped deployments also require the Extras ISO
• Kernel Tuning: For production deployments, apply recommended sysctl settings (Performance Tuning Guide)

We recommend using the Installation Checklist to track your progress through the installation process.

Getting Help

If you encounter issues during installation:

• Review the Troubleshooting Guide for common issues
• Check the System Requirements to verify your environment
• Consult the Release Notes for version-specific known issues

5.1 - Installation Checklist

Overview

Use this checklist to track your installation progress. Print this page or keep it open during your installation to ensure all steps are completed correctly.

Pre-Installation

Hardware and Software

• Verify hardware meets System Requirements
• Confirm operating system is supported (RHEL 9 or compatible clone)
• Configure firewall rules between nodes (details)
• Apply recommended sysctl settings (details)
• Obtain installation ISO (esb3027-acd-manager-X.Y.Z.iso)

Air-Gapped Deployments

• Obtain Extras ISO (esb3027-acd-manager-extras-X.Y.Z.iso)
• Mount OS ISO on all nodes before installation
• Verify OS packages are accessible from mounted ISO

Special Requirements

• Oracle Linux UEK: Install kernel-uek-modules-extra-netfilter-$(uname -r) package
• Control Plane Only nodes: Set SKIP_REQUIREMENTS_CHECK=1 if below lab minimums
• SELinux: Set to “Enforcing” mode before running installer (cannot enable after)

Cluster Installation

Single-Node Deployment

Follow the Single-Node Installation Guide.

• Mount installation ISO (Step 1)
• Install the base cluster (Step 2)
• Verify cluster status (Step 3)
• Air-gapped only: Load container images (Step 4)
• Create configuration file (Step 5)
• Optional: Load MaxMind GeoIP databases (Step 6)
• Deploy the Manager Helm chart (Step 7)
• Verify deployment (Step 8)

Multi-Node Deployment

Follow the Multi-Node Installation Guide.

Primary Server Node

• Mount installation ISO (Step 1)
• Install the base cluster (Step 2)
• Verify system pods are running (Step 2)
• Retrieve the node token (Step 3)

Additional Server Nodes

• Mount installation ISO (Step 5)
• Join the cluster (Step 5)
• Verify each node joins (Step 5)
• Optional: Taint Control Plane Only nodes (Step 5b)

Agent Nodes (Optional)

• Mount installation ISO (Step 6)
• Join the cluster as an agent (Step 6)
• Verify each agent joins (Step 6)

Cluster Verification

• Verify all nodes are ready (Step 7)
• Verify system pods running on all nodes (Step 7)
• Air-gapped only: Load container images on each node (Step 9)

Application Deployment

• Create configuration file (Step 10)
• Optional: Load MaxMind GeoIP databases (Step 11)
• Optional: Configure TLS certificates from trusted CA (Step 12)
• Deploy the Manager Helm chart (Step 13)
• Verify all pods are running and distributed (Step 14)
• Configure DNS records for manager hostname (Step 15)

Post-Installation

Initial Access

• Access the system via HTTPS
• Accept self-signed certificate warning (if using default certificate)
• Log in with default credentials (see Glossary)

Security Configuration

• Create new administrator account in Zitadel
• Delete or secure the default admin account
• Configure additional users and permissions
• Review Zitadel Administrator Documentation for role assignments

Monitoring and Operations

• Access Grafana dashboards at /grafana
• Review pre-built monitoring dashboards
• Configure alerting rules (optional)
• Set up notification channels (optional)

Next Steps

• Review Next Steps Guide for additional configuration
• Configure CDN routing rules
• Set up GeoIP-based routing (if using MaxMind databases)
• Review Operations Guide for day-to-day procedures

Troubleshooting

If you encounter issues during installation:

1. Check pod status: kubectl describe pod <pod-name>
2. Review logs: kubectl logs <pod-name>
3. Check cluster events: kubectl get events --sort-by='.lastTimestamp'
4. Review the Troubleshooting Guide for common issues

5.2 - Single-Node Installation

Warning: Single-node deployments are for lab environments, acceptance testing, and demonstrations only. This configuration is not suitable for production workloads. For production deployments, see the Multi-Node Installation Guide, which requires a minimum of 3 server nodes for high availability.

Air-Gapped Deployment? This guide assumes internet connectivity. For air-gapped deployments, see the Air-Gapped Deployment Guide for additional requirements and procedures.

Overview

This guide describes the installation of the AgileTV CDN Manager on a single node. This configuration is intended for lab environments, acceptance testing, and demonstrations only. It is not suitable for production workloads.

Prerequisites

Hardware Requirements

Refer to the System Requirements Guide for hardware specifications. Single-node deployments require the “Single-Node (Lab)” configuration.

Operating System

Refer to the System Requirements Guide for supported operating systems.

Software Access

• Installation ISO: esb3027-acd-manager-X.Y.Z.iso
• Extras ISO (air-gapped only): esb3027-acd-manager-extras-X.Y.Z.iso

Network Configuration

Ensure that required firewall ports are configured before installation. See the Networking Guide for complete firewall configuration requirements.

SELinux

If SELinux is to be used, it must be set to “Enforcing” mode before running the installer script. The installer will configure appropriate SELinux policies automatically. SELinux cannot be enabled after installation.

Installation Steps

Step 1: Mount the ISO

Create a mount point and mount the installation ISO:

mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

Replace X.Y.Z with the actual version number.

Step 2: Install the Base Cluster

Run the installer to set up the K3s Kubernetes cluster:

/mnt/esb3027/install

This installs:

• K3s Kubernetes distribution
• Longhorn distributed storage
• Cloudnative PG operator for PostgreSQL
• Base system dependencies

The installer will configure the node as both a server and agent node.

Step 3: Verify Cluster Status

After the installer completes, verify that all components are operational before proceeding. This verification serves as an important checkpoint to confirm the installation is progressing correctly.

1. Verify the node is ready:

kubectl get nodes

Expected output:

NAME         STATUS   ROLES                       AGE   VERSION
k3s-server   Ready    control-plane,etcd,master   2m    v1.33.4+k3s1

2. Verify system pods in both namespaces are running:

# Check kube-system namespace (Kubernetes core components)
kubectl get pods -n kube-system

# Check longhorn-system namespace (distributed storage)
kubectl get pods -n longhorn-system

All pods should show Running status. If any pods are still Pending or ContainerCreating, wait until they are ready. Proceeding with incomplete system pods can cause subsequent steps to fail in unpredictable ways.

This verification confirms:

• K3s cluster is operational
• Longhorn distributed storage is running
• Cloudnative PG operator is deployed
• All core components are healthy before continuing

Step 4: Air-Gapped Deployments (If Applicable)

If deploying in an air-gapped environment, load container images from the extras ISO:

mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras
/mnt/esb3027-extras/load-images

Step 5: Create Configuration File

Create a Helm values file for your deployment. At minimum, configure the manager hostname and at least one router:

# ~/values.yaml
global:
  hosts:
    manager:
      - host: manager.local
    routers:
      - name: default
        address: 127.0.0.1

The routers configuration specifies CDN Director instances. For lab deployments, a placeholder entry is sufficient. For production, specify the actual Director hostnames or IP addresses.

For single-node deployments, you must also disable Kafka replication:

kafka:
  replicaCount: 1
  controller:
    replicaCount: 1

Step 6: Load MaxMind GeoIP Databases (Optional)

If you plan to use GeoIP-based routing or validation features, load the MaxMind GeoIP databases. The following databases are used by the manager:

• GeoIP2-City.mmdb - The City Database
• GeoLite2-ASN.mmdb - The ASN Database
• GeoIP2-Anonymous-IP.mmdb - The VPN and Anonymous IP Database

A helper utility is provided on the ISO to create the Kubernetes volume:

/mnt/esb3027/generate-maxmind-volume

The utility will prompt for the locations of the three database files and the name of the volume. After running this command, reference the volume in your configuration file:

manager:
  maxmindDbVolume: maxmind-db-volume

Replace maxmind-db-volume with the volume name you specified when running the utility.

Tip: When naming the volume, include a revision number or date (e.g., maxmind-db-volume-2026-04 or maxmind-db-volume-v2). This simplifies future updates: create a new volume with an updated name, update the values.yaml to reference the new volume, and delete the old volume after verification.

Step 7: Deploy the Manager Helm Chart

Deploy the CDN Manager application:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml

Monitor the deployment progress:

kubectl get pods

Wait for all pods to show Running status before proceeding.

Note: The default Helm timeout is 5 minutes. If the installation fails due to a rollout timeout, retry with a larger timeout value:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml --timeout 10m

If a previous installation attempt failed and you receive an error that the release name is already in use, uninstall the previous release before retrying:

helm uninstall acd-manager
helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml

Step 8: Verify Deployment

Verify all application pods are running:

kubectl get pods

Expected output for a single-node deployment (pod names will vary):

NAME                                              READY   STATUS      RESTARTS   AGE
acd-manager-5b98d569d9-abc12                      1/1     Running     0          3m
acd-manager-confd-6fb78548c4-xnrh4                1/1     Running     0          3m
acd-manager-gateway-8bc8446fc-chs26               1/1     Running     0          3m
acd-manager-kafka-controller-0                    2/2     Running     0          3m
acd-manager-metrics-aggregator-76d96c4964-lwdcj   1/1     Running     0          3m
acd-manager-mib-frontend-7bdb69684b-6qxn8         1/1     Running     0          3m
acd-manager-postgresql-0                          1/1     Running     0          3m
acd-manager-redis-master-0                        2/2     Running     0          3m
acd-manager-redis-replicas-0                      2/2     Running     0          3m
acd-manager-selection-input-5fb694b857-qxt67      1/1     Running     0          3m
acd-manager-zitadel-8448b4c4fc-2pkd8              1/1     Running     0          3m
acd-manager-zitadel-init-hh6j7                    0/1     Completed   0          4m
acd-manager-zitadel-setup-nwp8k                   0/2     Completed   0          4m
alertmanager-0                                    1/1     Running     0          3m
grafana-6d948cfdc6-77ggk                          1/1     Running     0          3m
victoria-metrics-agent-dc87df588-tn8wv            1/1     Running     0          3m
victoria-metrics-alert-757c44c58f-kk9lp           1/1     Running     0          3m
victoria-metrics-longterm-server-0                1/1     Running     0          3m
victoria-metrics-server-0                         1/1     Running     0          3m

Note: Init pods (such as zitadel-init and zitadel-setup) will show Completed status after successful initialization. This is expected behavior.

Post-Installation

After installation completes, proceed to the Next Steps guide for:

• Initial user configuration
• Accessing the web interfaces
• Configuring authentication
• Setting up monitoring

Accessing the System

Refer to the Accessing the System section in the Getting Started guide for service URLs and default credentials.

Note: A self-signed SSL certificate is deployed by default. You will need to accept the certificate warning in your browser.

Troubleshooting

If pods fail to start:

1. Check pod status: kubectl describe pod <pod-name>
2. Review logs: kubectl logs <pod-name>
3. Verify resources: kubectl top pods

See the Troubleshooting Guide for additional assistance.

Next Steps

After successful installation:

1. Next Steps Guide - Post-installation configuration
2. Configuration Guide - System configuration
3. Operations Guide - Day-to-day operations

Appendix: Example Configuration

The following values.yaml provides a minimal working configuration for lab deployments:

# Minimal lab configuration for single-node deployment
global:
  hosts:
    manager:
      - host: manager.local
    routers:
      - name: default
        address: 127.0.0.1

# Single-node: Disable Kafka replication
kafka:
  replicaCount: 1
  controller:
    replicaCount: 1

Customization notes:

• Replace manager.local with your desired hostname
• The routers entry specifies CDN Director instances. The placeholder 127.0.0.1 may be used if a Director instance isn’t available, or specify actual Director hostnames for production testing
• For air-gapped deployments, see Step 4: Air-Gapped Deployments

5.3 - Multi-Node Installation

Overview

This guide describes the installation of the AgileTV CDN Manager across multiple nodes for production deployments. This configuration provides high availability and horizontal scaling capabilities.

Air-Gapped Deployment? This guide assumes internet connectivity. For air-gapped deployments, see the Air-Gapped Deployment Guide for additional requirements and procedures.

Prerequisites

Hardware Requirements

Refer to the System Requirements Guide for hardware specifications. Production deployments require:

• Minimum 3 Server nodes (Control Plane Only or Combined role)
• Optional Agent nodes for additional workload capacity

Operating System

Refer to the System Requirements Guide for supported operating systems.

Software Access

• Installation ISO: esb3027-acd-manager-X.Y.Z.iso (for each node)
• Extras ISO (air-gapped only): esb3027-acd-manager-extras-X.Y.Z.iso

Network Configuration

Ensure that required firewall ports are configured between all nodes before installation. See the Networking Guide for complete firewall configuration requirements.

Multiple Network Interfaces

If your nodes have multiple network interfaces and you want to use a separate interface for cluster traffic (not the default route interface), configure the INSTALL_K3S_EXEC environment variable before installing the cluster or joining nodes.

For example, if bond0 has the default route but you want cluster traffic on bond1:

# For server nodes
export INSTALL_K3S_EXEC="server --node-ip 10.0.0.10 --flannel-iface=bond1"

# For agent nodes
export INSTALL_K3S_EXEC="agent --node-ip 10.0.0.20 --flannel-iface=bond1"

Where:

• Mode: Use server for the primary node establishing the cluster, or for additional server nodes. Use agent for agent nodes joining the cluster.
• --node-ip: The IP address of the interface to use for cluster traffic
• --flannel-iface: The network interface name for Flannel VXLAN overlay traffic

Set this variable on each node before running the install or join scripts.

SELinux

If SELinux is to be used, it must be set to “Enforcing” mode before running the installer script. The installer will configure appropriate SELinux policies automatically. SELinux cannot be enabled after installation.

Installation Steps

Step 1: Prepare the Primary Server Node

Mount the installation ISO on the primary server node:

mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

Replace X.Y.Z with the actual version number.

Step 2: Install the Base Cluster on Primary Server

If your node has multiple network interfaces and you need to specify a separate interface for cluster traffic, set the INSTALL_K3S_EXEC environment variable before running the installer (see Multiple Network Interfaces):

export INSTALL_K3S_EXEC="server --node-ip <node-ip> --flannel-iface=<interface>"

Run the installer to set up the K3s Kubernetes cluster:

/mnt/esb3027/install

This installs:

• K3s Kubernetes distribution
• Longhorn distributed storage
• Cloudnative PG operator for PostgreSQL
• Base system dependencies

Important: After the installer completes, verify that all system pods in both namespaces are in the Running state before proceeding:

# Check kube-system namespace (Kubernetes core components)
kubectl get pods -n kube-system

# Check longhorn-system namespace (distributed storage)
kubectl get pods -n longhorn-system

All pods should show Running status. If any pods are still Pending or ContainerCreating, wait until they are ready. Proceeding with incomplete system pods can cause subsequent steps to fail in unpredictable ways.

This verification confirms:

• K3s cluster is operational
• Longhorn distributed storage is running
• Cloudnative PG operator is deployed
• All core components are healthy before continuing

Step 3: Retrieve the Node Token

Retrieve the node token for joining additional nodes:

cat /var/lib/rancher/k3s/server/node-token

Save this token for use on additional nodes. Also note the IP address of the primary server node.

Step 4: Server vs Agent Node Roles

Before joining additional nodes, determine which nodes will serve as Server nodes vs Agent nodes:

Guidance:

• Combined role (default): Server nodes run both control plane and workloads; minimum 3 nodes required for HA
• Control Plane Only: Dedicate nodes to control plane functions; requires at least 3 Server nodes plus 3+ Agent nodes for workloads
• Agent nodes are required if using Control Plane Only servers; optional if using Combined role servers
• For most deployments, 3 Server nodes (Combined role) with no Agent nodes is sufficient
• Add Agent nodes to scale workload capacity without affecting control plane quorum

Proceed to Step 5 to join Server nodes. Agent nodes are joined after all Server nodes are ready.

Step 5: Join Additional Server Nodes

On each additional server node:

1. Mount the ISO:

   mkdir -p /mnt/esb3027
   mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027
   

2. Join the cluster:

If your node has multiple network interfaces, set the INSTALL_K3S_EXEC environment variable with the server mode before running the join script (see Multiple Network Interfaces):

export INSTALL_K3S_EXEC="server --node-ip <node-ip> --flannel-iface=<interface>"

Run the join script:

/mnt/esb3027/join-server https://<primary-server-ip>:6443 <node-token>

Replace <primary-server-ip> with the IP address of the primary server and <node-token> with the token retrieved in Step 3.

3. Verify the node joined successfully:

   kubectl get nodes
   

Repeat for each server node. A minimum of 3 server nodes is required for high availability.

Step 5b: Taint Control Plane Only Nodes (Optional)

If you are using dedicated Control Plane Only nodes (not Combined role), apply taints to prevent workload scheduling:

kubectl taint nodes <node-name> CriticalAddonsOnly=true:NoSchedule

Apply this taint to each Control Plane Only node. Verify taints are applied:

kubectl describe nodes | grep -A 5 "Taints"

Note: This step is only required if you want dedicated control plane nodes. For Combined role deployments, do not apply taints.

Important: Control Plane Only Server nodes can be deployed with lower hardware specifications (2 cores, 4 GiB, 64 GiB) than the installer’s default minimum requirements. If your Control Plane Only Server nodes do not meet the Single-Node Lab configuration minimums (8 cores, 16 GiB, 128 GiB), you must set the SKIP_REQUIREMENTS_CHECK environment variable before running the installer or join command:

# For the primary server node
export SKIP_REQUIREMENTS_CHECK=1
/mnt/esb3027/install

# For additional Control Plane Only Server nodes
export SKIP_REQUIREMENTS_CHECK=1
/mnt/esb3027/join-server https://<primary-server-ip>:6443 <node-token>

Note: This applies to Server nodes only. Agent nodes have separate minimum requirements.

Step 6: Join Agent Nodes (Optional)

On each agent node:

1. Mount the ISO:

   mkdir -p /mnt/esb3027
   mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027
   

2. Join the cluster as an agent:

If your node has multiple network interfaces, set the INSTALL_K3S_EXEC environment variable with the agent mode before running the join script (see Multiple Network Interfaces):

export INSTALL_K3S_EXEC="agent --node-ip <node-ip> --flannel-iface=<interface>"

Run the join script:

/mnt/esb3027/join-agent https://<primary-server-ip>:6443 <node-token>

3. Verify the node joined successfully from an existing server node:

   kubectl get nodes
   

Agent nodes provide additional workload capacity but do not participate in the control plane quorum.

Step 7: Verify Cluster Status

After all nodes are joined, verify the cluster is operational:

1. Verify all nodes are ready:

kubectl get nodes

Expected output:

NAME                 STATUS   ROLES                       AGE   VERSION
k3s-server-0         Ready    control-plane,etcd,master   5m    v1.33.4+k3s1
k3s-server-1         Ready    control-plane,etcd,master   3m    v1.33.4+k3s1
k3s-server-2         Ready    control-plane,etcd,master   2m    v1.33.4+k3s1
k3s-agent-1          Ready    <none>                      1m    v1.33.4+k3s1
k3s-agent-2          Ready    <none>                      1m    v1.33.4+k3s1

2. Verify system pods in both namespaces are running:

# Check kube-system namespace (Kubernetes core components)
kubectl get pods -n kube-system

# Check longhorn-system namespace (distributed storage)
kubectl get pods -n longhorn-system

All pods should show Running status. If any pods are still Pending or ContainerCreating, wait until they are ready.

This verification confirms:

• K3s cluster is operational across all nodes
• Longhorn distributed storage is running
• Cloudnative PG operator is deployed
• All core components are healthy before proceeding to application deployment

Step 9: Air-Gapped Deployments (If Applicable)

If deploying in an air-gapped environment, on each node:

mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras
/mnt/esb3027-extras/load-images

Step 10: Create Configuration File

Create a Helm values file for your deployment. At minimum, configure the manager hostnames, Zitadel external domain, and at least one router:

# ~/values.yaml
global:
  hosts:
    manager:
      - host: manager.example.com
      - host: manager-backup.example.com
    routers:
      - name: director-1
        address: 192.0.2.1
      - name: director-2
        address: 192.0.2.2

zitadel:
  zitadel:
    ExternalDomain: manager.example.com

Tip: A complete default values.yaml file is available on the installation ISO at /mnt/esb3027/values.yaml. Copy this file to use as a starting point for your configuration.

Important: The zitadel.zitadel.ExternalDomain must match the first entry in global.hosts.manager or authentication will fail due to CORS policy violations.

Important: For multi-node deployments, Kafka replication is enabled by default with 3 replicas. Do not modify the kafka.replicaCount or kafka.controller.replicaCount settings unless you understand the implications for data durability.

Step 11: Load MaxMind GeoIP Databases (Optional)

If you plan to use GeoIP-based routing or validation features, load the MaxMind GeoIP databases. The following databases are used by the manager:

• GeoIP2-City.mmdb - The City Database
• GeoLite2-ASN.mmdb - The ASN Database
• GeoIP2-Anonymous-IP.mmdb - The VPN and Anonymous IP Database

A helper utility is provided on the ISO to create the Kubernetes volume:

/mnt/esb3027/generate-maxmind-volume

The utility will prompt for the locations of the three database files and the name of the volume. After running this command, reference the volume in your configuration file:

manager:
  maxmindDbVolume: maxmind-db-volume

Replace maxmind-db-volume with the volume name you specified when running the utility.

Tip: When naming the volume, include a revision number or date (e.g., maxmind-db-volume-2026-04 or maxmind-db-volume-v2). This simplifies future updates: create a new volume with an updated name, update the values.yaml to reference the new volume, and delete the old volume after verification.

Step 12: Configure TLS Certificates (Optional)

For production deployments, configure a valid TLS certificate from a trusted Certificate Authority (CA). A self-signed certificate is deployed by default if no certificate is provided.

Method 1: Create TLS Secret Manually

Create a Kubernetes TLS secret with your certificate and key:

kubectl create secret tls acd-manager-tls --cert=tls.crt --key=tls.key

Method 2: Helm-Managed Secret

Add the certificate directly to your values.yaml:

ingress:
  secrets:
    acd-manager-tls: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
  tls:
    - hosts:
        - manager.example.com
      secretName: acd-manager-tls

Configuring All Ingress Controllers

All ingress controllers must be configured with the same certificate secret and hostname:

ingress:
  hostname: manager.example.com
  tls: true
  secretName: acd-manager-tls

zitadel:
  ingress:
    tls:
      - hosts:
          - manager.example.com
        secretName: acd-manager-tls

confd:
  ingress:
    hostname: manager.example.com
    tls: true
    secretName: acd-manager-tls

mib-frontend:
  ingress:
    hostname: manager.example.com
    tls: true
    secretName: acd-manager-tls

Important: The hostname must match the first entry in global.hosts.manager for Zitadel CORS compatibility. The secret name has a maximum length of 53 characters.

Step 13: Deploy the Manager Helm Chart

Deploy the CDN Manager application:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml

Note: By default, helm install runs silently until completion. To see real-time output during deployment, add the --debug flag:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml --debug

Tip: For better organization, split your configuration into multiple files and specify them with repeated --values flags:

helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values-base.yaml \
  --values ~/values-tls.yaml \
  --values ~/values-autoscaling.yaml

Later files override earlier files, allowing you to maintain a base configuration with environment-specific overrides.

Monitor the deployment progress:

kubectl get pods

Wait for all pods to show Running status before proceeding.

Note: The default Helm timeout is 5 minutes. If the installation fails due to a rollout timeout, retry with a larger timeout value:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml --timeout 10m

If a previous installation attempt failed and you receive an error that the release name is already in use, uninstall the previous release before retrying:

helm uninstall acd-manager
helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml

Step 14: Verify Deployment

Verify all application pods are running:

kubectl get pods

Note: During the initial deployment, several pods may enter a CrashLoopBackoff state depending on the timing of other containers starting up. This is expected behavior as some services wait for dependencies (such as databases or Kafka) to become available. The deployment should stabilize automatically after a few minutes.

Verify pods are distributed across nodes:

kubectl get pods -o wide

Expected output for a 3-node cluster (pod names will vary):

NAME                                              READY   STATUS      RESTARTS        AGE
acd-cluster-postgresql-1                          1/1     Running     0               11m
acd-cluster-postgresql-2                          1/1     Running     0               11m
acd-cluster-postgresql-3                          1/1     Running     0               10m
acd-manager-5b98d569d9-2pbph                      1/1     Running     0               3m
acd-manager-5b98d569d9-m54f9                      1/1     Running     0               3m
acd-manager-5b98d569d9-pq26f                      1/1     Running     0               3m
acd-manager-confd-6fb78548c4-xnrh4                1/1     Running     0               3m
acd-manager-gateway-8bc8446fc-chs26               1/1     Running     0               3m
acd-manager-gateway-8bc8446fc-wzrml               1/1     Running     0               3m
acd-manager-kafka-controller-0                    2/2     Running     0               3m
acd-manager-kafka-controller-1                    2/2     Running     0               3m
acd-manager-kafka-controller-2                    2/2     Running     0               3m
acd-manager-metrics-aggregator-76d96c4964-lwdcj   1/1     Running     2               3m
acd-manager-mib-frontend-7bdb69684b-6qxn8         1/1     Running     0               3m
acd-manager-mib-frontend-7bdb69684b-pkjrw         1/1     Running     0               3m
acd-manager-redis-master-0                        2/2     Running     0               3m
acd-manager-redis-replicas-0                      2/2     Running     0               3m
acd-manager-selection-input-5fb694b857-qxt67      1/1     Running     2               3m
acd-manager-zitadel-8448b4c4fc-2pkd8              1/1     Running     0               3m
acd-manager-zitadel-8448b4c4fc-vchp9              1/1     Running     0               3m
acd-manager-zitadel-init-hh6j7                    0/1     Completed   0               4m
acd-manager-zitadel-setup-nwp8k                   0/2     Completed   0               4m
alertmanager-0                                    1/1     Running     0               3m
grafana-6d948cfdc6-77ggk                          1/1     Running     0               3m
telegraf-54779f5f46-2jfj5                         1/1     Running     0               3m
victoria-metrics-agent-dc87df588-tn8wv            1/1     Running     0               3m
victoria-metrics-alert-757c44c58f-kk9lp           1/1     Running     0               3m
victoria-metrics-longterm-server-0                1/1     Running     0               3m
victoria-metrics-server-0                         1/1     Running     0               3m

Note: Init pods (such as zitadel-init and zitadel-setup) will show Completed status after successful initialization. This is expected behavior. Some pods may show restart counts as they wait for dependencies to become available.

Step 15: Configure DNS (Optional)

Add DNS records for the manager hostname. For high availability, configure multiple A records pointing to different server nodes:

manager.example.com.  IN  A  <server-1-ip>
manager.example.com.  IN  A  <server-2-ip>
manager.example.com.  IN  A  <server-3-ip>

Alternatively, configure a load balancer to distribute traffic across nodes.

Post-Installation

After installation completes, proceed to the Next Steps guide for:

• Initial user configuration
• Accessing the web interfaces
• Configuring authentication
• Setting up monitoring

Accessing the System

Refer to the Accessing the System section in the Getting Started guide for service URLs and default credentials.

Note: A self-signed SSL certificate is deployed by default. For production deployments, configure a valid SSL certificate before exposing the system to users.

High Availability Considerations

Pod Distribution

The Helm chart configures pod anti-affinity rules to ensure:

• Kafka controllers are scheduled on separate nodes
• PostgreSQL cluster members are distributed across nodes
• Application pods are spread across available nodes

Data Replication and Failure Tolerance

For detailed information on data replication strategies and failure scenario tolerance, refer to the Architecture Guide and System Requirements Guide.

Troubleshooting

If pods fail to start or nodes fail to join:

1. Check node status: kubectl get nodes
2. Describe problematic pods: kubectl describe pod <pod-name>
3. Review logs: kubectl logs <pod-name>
4. Check cluster events: kubectl get events --sort-by='.lastTimestamp'

See the Troubleshooting Guide for additional assistance.

Next Steps

After successful installation:

1. Next Steps Guide - Post-installation configuration
2. Configuration Guide - System configuration
3. Operations Guide - Day-to-day operations

5.4 - Air-Gapped Deployment Guide

Overview

This guide describes the installation of the AgileTV CDN Manager in air-gapped environments (no internet access). Air-gapped deployments require additional preparation compared to connected deployments.

Key differences from connected deployments:

• Both Installation ISO and Extras ISO are required
• OS installation ISO must be mounted on all nodes
• Container images must be loaded from the Extras ISO on each node
• Additional firewall considerations for OS package repositories

Prerequisites

Required ISOs

Before beginning installation, obtain the following:

Single-Node vs Multi-Node

Air-gapped procedures apply to both deployment types:

• Lab/Single-Node: Follow Single-Node Installation with additional air-gapped steps below
• Production/Multi-Node: Follow Multi-Node Installation with additional air-gapped steps below

Network Configuration

Air-gapped environments may have internal network mirrors for OS packages. If no internal mirror exists, the OS installation ISO must be mounted on each node to provide packages during installation.

Air-Gapped Installation Steps

Step 1: Prepare All Nodes

On each node (primary server, additional servers, and agents):

1. Mount the OS installation ISO:

   mkdir -p /mnt/os
   mount -o loop,ro /path/to/rhel-9.iso /mnt/os
   

2. Configure local repository (if no internal mirror):

   cat > /etc/yum.repos.d/local.repo <<EOF
   [local]
   name=Local OS Repository
   baseurl=file:///mnt/os/BaseOS
   enabled=1
   gpgcheck=0
   EOF
   

3. Verify repository is accessible:

   dnf repolist
   

Step 2: Mount Installation ISOs

On the primary server node first, then each additional node:

# Mount Installation ISO
mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

# Mount Extras ISO
mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras

Step 3: Install Kubernetes Cluster

Primary Server Node

/mnt/esb3027/install

Wait for the installer to complete and verify system pods are running:

kubectl get nodes
kubectl get pods -n kube-system
kubectl get pods -n longhorn-system

Additional Server Nodes (Multi-Node Only)

On each additional server node:

/mnt/esb3027/join-server https://<primary-server-ip>:6443 <node-token>

Agent Nodes (Optional)

On each agent node:

/mnt/esb3027/join-agent https://<primary-server-ip>:6443 <node-token>

Step 4: Load Container Images

On each node in the cluster:

/mnt/esb3027-extras/load-images

This script loads all container images from the Extras ISO into the local container runtime.

Important: This step must be performed on every node (primary server, additional servers, and agents) before deploying the Manager application.

Step 5: Create Configuration File

Create a Helm values file for your deployment. At minimum, configure the manager hostname and router addresses:

# ~/values.yaml
global:
  hosts:
    manager:
      - host: manager.local
    routers:
      - name: default
        address: 127.0.0.1

# Single-node: Disable Kafka replication
kafka:
  replicaCount: 1
  controller:
    replicaCount: 1

For multi-node deployments, see the Multi-Node Installation Guide for complete configuration requirements.

Step 6: Deploy the Manager

Deploy the CDN Manager Helm chart:

helm install acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml

Monitor the deployment progress:

kubectl get pods --watch

Wait for all pods to show Running status before proceeding.

Step 7: Verify Deployment

Verify all application pods are running:

kubectl get pods

All pods should show Running status (except init pods which show Completed).

Post-Installation

After installation completes:

1. Access the system via HTTPS at https://<manager-host>
2. Configure authentication via Zitadel at https://<manager-host>/ui/console
3. Set up monitoring via Grafana at https://<manager-host>/grafana

See the Next Steps Guide for detailed post-installation configuration.

Updating MaxMind GeoIP Databases

If using GeoIP-based routing, load the MaxMind databases:

/mnt/esb3027/generate-maxmind-volume

The utility will prompt for the database file locations and volume name. Reference the volume in your values.yaml:

manager:
  maxmindDbVolume: maxmind-geoip-2026-04

See the Operations Guide for database update procedures.

Troubleshooting

Image Pull Errors

If pods fail with image pull errors:

1. Verify the load-images script completed successfully on all nodes
2. Check container runtime image list:

   crictl images | grep <image-name>
   

3. Ensure image tags in Helm chart match tags on the Extras ISO

OS Package Errors

If the installer reports missing OS packages:

1. Verify OS ISO is mounted on the affected node
2. Check repository configuration:

   dnf repolist
   dnf info <package-name>
   

3. Ensure the ISO matches the installed OS version

Longhorn Volume Issues

If Longhorn volumes fail to mount:

1. Verify all nodes have the load-images script completed
2. Check Longhorn system pods:

   kubectl get pods -n longhorn-system
   

3. Review Longhorn UI via port-forward:

   kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80
   

Next Steps

After successful installation:

1. Next Steps Guide - Post-installation configuration
2. Operations Guide - Day-to-day operational procedures
3. Troubleshooting Guide - Common issues and resolution

5.5 - Upgrade Guide

Overview

This guide describes the procedure for upgrading the AgileTV CDN Manager (ESB3027) to a newer version. The upgrade process involves updating the Kubernetes cluster components and redeploying the Helm chart with the new version.

Prerequisites

Backup Requirements

Before beginning any upgrade, ensure you have:

• PostgreSQL Backup: Verify recent backups are available via the Cloudnative PG operator
• Configuration Backup: Save your current values.yaml file(s)
• TLS Certificates: Ensure certificate files are backed up
• MaxMind Volumes: Note the current volume names if using GeoIP databases

Version Compatibility

Review the Release Notes for the target version to check for:

• Breaking changes requiring manual intervention
• Required intermediate upgrade steps
• New configuration options that should be set

Cluster Health

Verify the cluster is healthy before upgrading:

kubectl get nodes
kubectl get pods
kubectl get pvc

All nodes should show Ready status and all pods should be Running (or Completed for job pods).

Upgrade Methods

There are three upgrade methods available. Choose the one that best fits your situation:

Method Selection Guidance:

• Rolling Upgrade (Method 1) is the default choice for most upgrades. Use this for patch releases (e.g., 1.6.0 → 1.6.1) and even minor version upgrades (e.g., 1.4.0 → 1.6.0) where no breaking changes are documented. This method preserves all existing resources and performs an in-place update. Note: This method supports Helm’s automatic rollback (helm rollback) if the upgrade fails, allowing quick recovery to the previous state.

• Clean Upgrade (Method 2) is recommended for major version upgrades (e.g., 1.x → 2.x) or when the release notes indicate significant component changes. This method ensures all resources are recreated with the new version, avoiding potential issues with stale configurations. Also use this method when troubleshooting upgrade failures from Method 1.

• Full Reinstall (Method 3) should only be used when a completely clean cluster state is required. This includes troubleshooting persistent cluster-level issues, recovering from failed upgrades that cannot be rolled back, or when migrating between significantly different deployment configurations. This method requires verified backups and should be planned for extended downtime.

Upgrade Steps

Method 1: Rolling Upgrade (Recommended)

This method performs an in-place rolling upgrade with minimal downtime. All upgrade commands are executed from the primary server node.

Step 1: Obtain the New Installation ISO

Unmount the old ISO (if mounted) and mount the new installation ISO:

umount /mnt/esb3027 2>/dev/null || true
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

Replace X.Y.Z with the target version number.

Step 2: Update Containers and Cluster Software

Run the installation script to update the container images and cluster software:

/mnt/esb3027/install

Wait for the script to complete.

Step 2b: Air-Gapped Environments (If Applicable)

If deploying in an air-gapped environment, also mount and load the extras ISO:

# Mount the Extras ISO
mkdir -p /mnt/esb3027-extras
mount -o loop,ro esb3027-acd-manager-extras-X.Y.Z.iso /mnt/esb3027-extras

# Load container images from the extras ISO
/mnt/esb3027-extras/load-images

Replace X.Y.Z with the target version number.

Step 4: Review and Update Configuration

Compare the default values.yaml from the new ISO with your current configuration:

diff /mnt/esb3027/values.yaml ~/values.yaml

Update your configuration file to include any new required settings. Common updates include:

# ~/values.yaml
global:
  hosts:
    manager:
      - host: manager.example.com
    routers:
      - name: director-1
        address: 192.0.2.1

zitadel:
  zitadel:
    ExternalDomain: manager.example.com

# Add any new required settings for the target version

Important: Do not modify settings unrelated to the upgrade unless specifically documented in the release notes.

Step 5: Update MaxMind GeoIP Volumes (If Applicable)

If you use MaxMind GeoIP databases, use the utility from the new ISO to create an updated volume:

/mnt/esb3027/generate-maxmind-volume

Update your values.yaml to reference the new volume name:

manager:
  maxmindDbVolume: maxmind-geoip-2026-04

Tip: Using dated or versioned volume names (e.g., maxmind-geoip-2026-04) allows you to create new volumes during upgrades and delete old ones after verification.

Step 6: Update TLS Certificates (If Needed)

If your TLS certificates need renewal or the new version requires certificate updates, create or update the secret:

kubectl create secret tls acd-manager-tls --cert=tls.crt --key=tls.key --dry-run=client -o yaml | kubectl apply -f -

Step 7: Upgrade the Helm Release

Perform a Helm upgrade with the new chart:

helm upgrade acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml

Note: The upgrade performs a rolling update of each deployment in the chart. Deployments are upgraded one at a time, with pods being terminated and recreated sequentially. StatefulSets (PostgreSQL, Kafka, Redis) roll out one pod at a time to maintain data availability.

Monitor the upgrade progress:

kubectl get pods --watch

Wait for all pods to stabilize and show Running status before considering the upgrade complete. Some pods may temporarily enter CrashLoopBackoff during the transition as they wait for dependencies to become available.

Step 8: Verify the Upgrade

Check the deployed version:

helm list
kubectl get deployments -o wide

Verify application functionality:

• Access the MIB Frontend and confirm it loads
• Test API connectivity
• Verify Grafana dashboards are accessible
• Check that Zitadel authentication is working

Step 9: Clean Up

After confirming the upgrade is successful:

1. Unmount the old ISO (if still mounted):

   umount /mnt/esb3027
   

2. Delete old MaxMind volumes (if replaced):

   kubectl get pvc
   kubectl delete pvc <old-volume-name>
   

3. Remove old configuration files if no longer needed.

Method 2: Clean Upgrade (Helm Uninstall/Install)

This method removes the existing Helm release before installing the new version. This is useful for major version upgrades or when troubleshooting upgrade issues. All upgrade commands are executed from the primary server node.

Warning: This method causes brief downtime as all resources are deleted before reinstallation.

Step 1: Obtain the New Installation ISO

Mount the new installation ISO:

umount /mnt/esb3027 2>/dev/null || true
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

Step 2: Backup Configuration

Save your current Helm values:

helm get values acd-manager -o yaml > ~/values-backup.yaml

Step 3: Uninstall the Existing Release

Remove the existing Helm release:

helm uninstall acd-manager

Wait for pods to terminate:

kubectl get pods --watch

Note: Helm uninstall does not remove PersistentVolumes (PVs) or PersistentVolumeClaims (PVCs). All data stored in PostgreSQL, Kafka, Redis, and Longhorn volumes is preserved during the uninstall process. When the new version is installed, it will reattach to the existing PVCs and restore data automatically.

Step 4: Review and Update Configuration

Compare the default values.yaml from the new ISO with your configuration:

diff /mnt/esb3027/values.yaml ~/values.yaml

Update your configuration file as needed.

Step 5: Install the New Release

Install the new version:

helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml

Monitor the deployment:

kubectl get pods --watch

Wait for all pods to stabilize before proceeding.

Step 6: Verify the Upgrade

Verify the upgrade as described in Method 1, Step 8.

Method 3: Full Reinstall (Cluster Rebuild)

This method completely removes Kubernetes and reinstalls from scratch. Use only for cluster rebuilds or when other upgrade methods fail.

Warning: This method causes extended downtime and permanent data loss. The K3s uninstall process destroys all Longhorn PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). All data stored in PostgreSQL, Kafka, Redis, and application volumes will be permanently lost. Verified backups are required before proceeding.

Warning: This method should only be used when necessary. Ensure you have verified backups before proceeding.

Step 1: Stop Kubernetes Services

On all nodes (server and agent), stop the K3s service:

systemctl stop k3s

Step 2: Uninstall K3s (Server Nodes Only)

On the primary server node first, then each additional server node:

/usr/local/bin/k3s-uninstall.sh

Step 3: Clean Up Residual State (All Nodes)

On all nodes, remove residual state:

/usr/local/bin/k3s-kill-all.sh
rm -rf /var/lib/rancher/k3s/*

Warning: This removes all cluster data including Longhorn PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). All data stored in PostgreSQL, Kafka, Redis, and application volumes will be permanently lost. Ensure verified backups are available before proceeding.

Step 4: Reinstall K3s Cluster and Deploy Manager

Follow the installation procedure in the Installation Guide to reinstall the cluster and deploy the Helm chart. At this point, you are in the same state as a fresh installation:

• Primary server installation
• Additional server joins (if applicable)
• Agent joins (if applicable)
• Helm chart deployment

Note: The K3s node token is regenerated during reinstallation. Retrieve the new token from /var/lib/rancher/k3s/server/node-token on the primary server after installation if you need to join additional nodes.

Rollback Procedure

Rollback procedures vary by upgrade method:

Method 1 (Rolling Upgrade)

Use Helm’s built-in rollback command:

helm rollback acd-manager

This reverts to the previous Helm release revision automatically.

Or manually redeploy the previous version:

helm upgrade acd-manager /mnt/esb3027-old/helm/charts/acd-manager \
  --values ~/values.yaml

Note: If you use multiple --values files for organization, ensure they are specified in the same order as the original installation.

Method 2 (Clean Upgrade)

Reinstall the previous version:

helm uninstall acd-manager
helm install acd-manager /mnt/esb3027-old/helm/charts/acd-manager \
  --values ~/values-backup.yaml

Method 3 (Full Reinstall)

Rollback requires repeating the full cluster reinstall procedure using the old installation ISO. Follow Method 3 steps with the previous version’s ISO. Ensure verified backups are available before attempting.

Troubleshooting

Pods Fail to Start

1. Check pod status and events:

   kubectl describe pod <pod-name>
   kubectl get events --sort-by='.lastTimestamp'
   

2. Review pod logs:

   kubectl logs <pod-name>
   kubectl logs <pod-name> -p  # Previous instance logs
   

Database Migration Issues

If PostgreSQL migrations fail:

1. Check Cloudnative PG cluster status:

   kubectl get clusters
   kubectl describe cluster <cluster-name>
   

2. Review migration job logs:

   kubectl get jobs
   kubectl logs job/<migration-job-name>
   

Helm Upgrade Fails

If helm upgrade fails:

1. Check Helm release status:

   helm status acd-manager
   helm history acd-manager
   

2. Review the error message for specific failures

3. Attempt rollback if necessary

Post-Upgrade

After a successful upgrade:

1. Review the Release Notes for any post-upgrade tasks
2. Update monitoring dashboards if new metrics are available
3. Test all critical functionality
4. Document the upgrade in your change management system

Next Steps

After completing the upgrade:

1. Next Steps Guide - Review post-installation tasks
2. Operations Guide - Day-to-day operational procedures
3. Release Notes - Review new features and changes

5.6 - Next Steps

Overview

After completing the installation of the AgileTV CDN Manager (ESB3027), several post-installation configuration tasks must be performed before the system is ready for production use. This guide walks you through the essential next steps.

Prerequisites

Before proceeding, ensure:

• The CDN Manager Helm chart is successfully deployed
• All pods are in Running status
• You have network access to the cluster hostname or IP
• You have the default credentials available

Step 1: Access Zitadel Console

The first step is to configure user authentication through Zitadel Identity and Access Management (IAM).

1. Navigate to the Zitadel Console:

   https://<manager-host>/ui/console
   

   Replace <manager-host> with your configured hostname (e.g., manager.local or manager.example.com).

   Important: The <manager-host> must match the first entry in global.hosts.manager from your Helm values exactly. Zitadel uses name-based virtual hosting and CORS validation. If the hostname does not match, authentication will fail.

2. Log in with the default administrator credentials (also listed in the Glossary):

   • Username: admin@agiletv.dev
   • Password: Password1!

3. Important: If prompted to configure Multi-Factor Authentication (MFA), you must skip this step for now. MFA is not currently supported. Attempting to configure MFA may lock you out of the administrator account.

4. Security Recommendation: After logging in, create a new administrator account with proper roles. Once verified, disable or delete the default admin@agiletv.dev account. For details on required roles and administrator permissions, see Zitadel’s Administrator Documentation.

Step 2: Configure SMTP Settings (Recommended)

Zitadel requires an SMTP server to send email notifications and perform email validations.

1. In the Zitadel Console, navigate to Settings > Default Settings

2. Configure the SMTP settings:

   • SMTP Host: Your mail server hostname
   • SMTP Port: Typically 587 (TLS) or 465 (SSL)
   • SMTP Username: Mail account username
   • SMTP Password: Mail account password
   • Sender Address: Email address for outgoing mail (e.g., noreply@example.com)

3. Save the configuration

Note: Without SMTP configuration, email-based user validation and password recovery features will not function.

Step 3: Create Additional User Accounts

Create user accounts for operators and administrators:

Tip: For detailed guidance on managing users, roles, and permissions in the Zitadel Console, see Zitadel’s User Management Documentation.

1. In the Zitadel Console, navigate to Users > Add User

2. Fill in the user details:

   • Username: Unique username
   • First Name: User’s first name
   • Last Name: User’s last name
   • Email: User’s email address (this is their login username)

   Known Issue: Due to a limitation in this release of Zitadel, the username must match the local part (the portion before the @) of the email address. For example, if the email is foo@example.com, the username must be foo.

   If these do not match, Zitadel may allow login with the mismatched local part while blocking the full email address. For instance, if username is foo but email is foo.bar@example.com, login with foo@example.com may succeed while foo.bar@example.com is blocked.

   Workaround: Always ensure the username matches the email local part exactly.

3. Important: The following options must be configured:

   • Email Verified: Check this box to skip email verification
   • Set Initial Password: Enter a temporary password for the user

   Note: If you configured SMTP settings in Step 2, the user will receive an email asking to verify their address and set their initial password. If SMTP is not configured, you must check the “Email Verified” box and set an initial password manually, otherwise the user account will not be enabled.

4. Click Create User

5. Provide the user with:

   • Their username
   • The temporary password (if set manually)
   • The Zitadel Console URL

6. Instruct the user to change their password on first login

Step 4: Configure User Roles and Permissions

Zitadel manages roles and permissions for accessing the CDN Manager:

1. In the Zitadel Console, navigate to Roles

2. Assign appropriate roles to users:

   • Admin: Full administrative access
   • Operator: Operational access without administrative functions
   • Viewer: Read-only access

3. To assign a role:

   • Select the user
   • Click Add Role
   • Select the appropriate role
   • Save the assignment

Step 5: Access the MIB Frontend

The MIB Frontend is the web-based configuration GUI for CDN operators:

1. Navigate to the MIB Frontend:

   https://<manager-host>/gui
   

2. Log in using your Zitadel credentials

3. Verify you can access the configuration interface

Step 6: Verify API Access

Test API connectivity to ensure the system is functioning:

curl -k https://<manager-host>/api/v1/health/ready

Expected response:

{
  "status": "ready"
}

See the API Guide for detailed API documentation.

Step 7: Configure TLS Certificates (If Not Done During Installation)

For production deployments, a valid TLS certificate from a trusted Certificate Authority should be configured. If you did not configure TLS certificates during installation, refer to Step 12: Configure TLS Certificates in the Installation Guide.

Step 8: Set Up Monitoring and Alerting

Configure monitoring dashboards and alerting:

1. Access Grafana:

   • Navigate to https://<manager-host>/grafana
   • Log in with default credentials (also listed in the Glossary):
     • Username: admin
     • Password: edgeware

2. Review Pre-built Dashboards:

   • System health dashboards are included by default
   • CDN metrics dashboards show routing and usage statistics

   Note: CDN Director instances automatically have DNS names configured for use in Grafana dashboards. The DNS name is derived from the name field in global.hosts.routers with .external appended. For example, a router named my-router-1 will have the DNS name my-router-1.external in Grafana configuration.

Step 9: Verify Kafka and PostgreSQL Health

Ensure the data layer components are healthy:

kubectl get pods

Verify the following pods are running:

All pods should show Running status with no restarts.

Step 10: Configure Availability Zones (Optional)

For improved network performance, configure availability zones to enable Topology Aware Hints. This optimizes service-to-pod routing by keeping traffic within the same zone when possible.

See the Performance Tuning Guide for detailed instructions on:

• Labeling nodes with zone and region topology
• Verifying topology configuration
• Requirements for Topology Aware Hints to activate
• Integration with pod anti-affinity rules

Note: This step is optional. If zone labels are not configured, the system will fall back to random load-balancing.

Step 11: Review System Configuration

Verify the initial configuration:

1. Review Helm Values:

   helm get values acd-manager -o yaml
   

2. Check Ingress Configuration:

   kubectl get ingress
   

3. Verify Service Endpoints:

   kubectl get endpoints
   

Step 12: Document Your Deployment

Maintain documentation for your deployment:

• Cluster hostname and IP addresses
• Configuration file locations
• User accounts and roles created
• TLS certificate expiration dates
• Backup procedures and schedules
• Monitoring and alerting contacts

Next Steps

After completing post-installation configuration:

1. Configuration Guide - Detailed system configuration options
2. Operations Guide - Day-to-day operational procedures
3. Metrics & Monitoring Guide - Comprehensive monitoring setup
4. API Guide - REST API reference and integration examples

Troubleshooting

Cannot Access Zitadel Console

• Verify DNS resolution or hosts file configuration
• Check that Traefik ingress is running: kubectl get pods -n kube-system | grep traefik
• Review Traefik logs: kubectl logs -n kube-system -l app.kubernetes.io/name=traefik

Authentication Failures

• Verify Zitadel pods are healthy: kubectl get pods | grep zitadel
• Check Zitadel logs: kubectl logs <zitadel-pod-name>
• Ensure the external domain matches your hostname in Zitadel configuration

MIB Frontend Not Loading

• Verify MIB Frontend pods are running: kubectl get pods | grep mib-frontend
• Check for connectivity issues to Confd and API services
• Review browser console for JavaScript errors

API Returns 401 Unauthorized

• Verify you have a valid bearer token
• Check token expiration
• Ensure Zitadel authentication is functioning

For additional troubleshooting assistance, refer to the Troubleshooting Guide.

6 - Configuration Guide

Overview

The CDN Manager is deployed via Helm chart with configuration supplied through values.yaml files. This guide explains the configuration structure, how to apply changes, and provides a reference for all configurable options.

Configuration Files

Default Configuration

The default values.yaml file is located on the installation ISO at /mnt/esb3027/values.yaml. This file contains all default values and should be copied to a writable location for modification:

cp /mnt/esb3027/values.yaml ~/values.yaml

Important: You only need to specify fields in your custom values.yaml that differ from the default. Helm applies configuration hierarchically:

1. Default values from the Helm chart itself
2. Values from the default values.yaml on the ISO
3. Values from your custom values.yaml file(s)

For example, if you only need to change the manager hostname and router addresses, your custom values.yaml might contain only:

global:
  hosts:
    manager:
      - host: manager.example.com
    routers:
      - name: default
        address: 192.0.2.1

All other configuration values will be inherited from the default values.yaml on the ISO. This approach simplifies upgrades, as you only maintain your customizations.

Configuration Merging

Helm merges configuration files from left to right, with later files overriding earlier values. This allows you to:

• Maintain a base configuration with common settings
• Create environment-specific override files
• Keep the default chart values for unchanged settings

# Multiple files merged left-to-right
helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values-base.yaml \
  --values ~/values-production.yaml \
  --values ~/values-tls.yaml

Individual Value Overrides

For temporary changes, you can override individual values with --set:

helm upgrade acd-manager /mnt/esb3027/helm/charts/acd-manager \
  --values ~/values.yaml \
  --set manager.logLevel=debug

Note: Using --set is discouraged for permanent changes, as the same arguments must be specified for every Helm operation.

Applying Configuration

Initial Installation

helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml

Updating Configuration

helm upgrade acd-manager /mnt/esb3027/helm/charts/acd-manager \
  --values ~/values.yaml

Dry Run

Before applying changes, validate the configuration with a dry run:

helm upgrade acd-manager /mnt/esb3027/helm/charts/acd-manager \
  --values ~/values.yaml \
  --dry-run

Rollback

If an upgrade fails, rollback to the previous revision:

# View revision history
helm history acd-manager

# Rollback to previous revision
helm rollback acd-manager

# Rollback to specific revision
helm rollback acd-manager <revision_number>

Note: Rollback reverts the Helm release but does not modify your values.yaml file. You must manually revert configuration file changes.

Force Reinstall

If an upgrade fails and rollback is not sufficient, you can perform a clean reinstall:

helm uninstall acd-manager
helm install acd-manager /mnt/esb3027/charts/acd-manager \
  --values ~/values.yaml

Warning: This is service-affecting as all pods will be destroyed and recreated.

Configuration Reference

Global Settings

The global section contains cluster-wide settings. The most critical configuration is global.hosts.

global:
  hosts:
    manager:
      - host: manager.local
    routers:
      - name: default
        address: 127.0.0.1
    edns_proxy: []
    geoip: []

Important: The first entry in global.hosts.manager must match zitadel.zitadel.ExternalDomain exactly. Zitadel enforces CORS protection, and authentication will fail if these do not match.

Manager Configuration

Core Manager API server settings:

Manager Resources

The chart supports both resource presets and explicit resource specifications:

Note: For production workloads, explicitly set manager.resources rather than using presets.

Manager Datastore

manager:
  datastore:
    type: redis
    namespace: "cdn_manager_ds"
    default_ttl: ""
    compression: zstd

Manager Discovery

manager:
  discovery: []
  # Example:
  # - namespace: "other"
  #   hosts:
  #     - other-host1
  #     - other-host2
  #   pattern: "other-.*"

Manager Tuning

manager:
  tuning:
    enable_cache_control: true
    cache_control_max_age: "5m"
    cache_control_miss_max_age: ""

Manager Container Arguments

manager:
  args:
    - --config-file=/etc/manager/config.toml
    - http-server

Gateway Configuration

NGinx Gateway settings for external Director communication:

MIB Frontend Configuration

Web-based configuration GUI settings:

Confd Configuration

Confd settings for configuration management:

VictoriaMetrics Configuration

Time-series database for metrics:

Ingress Configuration

Traffic exposure settings:

Ingress Extra Paths

The chart includes default extra paths for Confd and GeoIP:

ingress:
  extraPaths:
    - path: /confd
      pathType: Prefix
      backend:
        service:
          name: acd-manager-gateway
          port:
            name: http
    - path: /geoip
      pathType: Prefix
      backend:
        service:
          name: acd-manager-gateway
          port:
            name: http

TLS Certificate Secrets

For production TLS certificates:

ingress:
  secrets:
    - name: manager.local-tls
      key: |-
        -----BEGIN RSA PRIVATE KEY-----
        ...
        -----END RSA PRIVATE KEY-----
      certificate: |-
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
  tls: true

Resource Configuration

Resource Presets

Predefined resource configurations for common deployment sizes:

Note: Limits are calculated as requests plus 50% (except for xlarge/2xlarge and ephemeral-storage).

Custom Resources

Override preset with custom values:

manager:
  resources:
    requests:
      cpu: "300m"
      memory: "512Mi"
    limits:
      cpu: "1"
      memory: "1Gi"

Note:

• CPU values use millicores (1000m = 1 core)
• Memory values use binary SI units (1024Mi = 1GiB)
• Requests represent minimum guaranteed resources
• Limits represent maximum consumable resources

Capacity Planning

When sizing resources:

• Requests determine scheduling (node must have available capacity)
• Limits prevent resource starvation
• Maintain 20-30% cluster headroom for scaling
• Total capacity = sum of all requests × replica count + headroom

Security Contexts

Pod Security Context

manager:
  podSecurityContext:
    enabled: true
    fsGroup: 1001
    fsGroupChangePolicy: Always
    sysctls: []
    supplementalGroups: []

Container Security Context

manager:
  containerSecurityContext:
    enabled: true
    runAsUser: 1001
    runAsGroup: 1001
    runAsNonRoot: true
    readOnlyRootFilesystem: true
    privileged: false
    allowPrivilegeEscalation: false
    capabilities:
      drop: ["ALL"]
    seccompProfile:
      type: "RuntimeDefault"

Health Probes

Probe Types

Default Probe Configuration

Liveness Probe

manager:
  livenessProbe:
    enabled: true
    initialDelaySeconds: 5
    periodSeconds: 30
    timeoutSeconds: 10
    failureThreshold: 5
    successThreshold: 1
    httpGet:
      path: /api/v1/health/alive
      port: http

Readiness Probe

manager:
  readinessProbe:
    enabled: true
    initialDelaySeconds: 5
    periodSeconds: 10
    timeoutSeconds: 7
    failureThreshold: 3
    successThreshold: 1
    httpGet:
      path: /api/v1/health/ready
      port: http

Startup Probe

manager:
  startupProbe:
    enabled: true
    initialDelaySeconds: 0
    periodSeconds: 5
    timeoutSeconds: 3
    failureThreshold: 10
    successThreshold: 1
    httpGet:
      path: /api/v1/health/alive
      port: http

Autoscaling Configuration

Horizontal Pod Autoscaler (HPA)

manager:
  autoscaling:
    hpa:
      enabled: true
      minReplicas: 3
      maxReplicas: 8
      targetCPU: 50
      targetMemory: 80

Network Policy

networkPolicy:
  enabled: true
  allowExternal: true
  allowExternalEgress: true
  addExternalClientAccess: true

Pod Affinity and Anti-Affinity

manager:
  podAffinityPreset: ""
  podAntiAffinityPreset: soft
  nodeAffinityPreset:
    type: ""
    key: ""
    values: []
  affinity: {}

Service Configuration

service:
  type: ClusterIP
  ports:
    http: 80
  annotations:
    service.kubernetes.io/topology-mode: Auto
  externalTrafficPolicy: Cluster
  sessionAffinity: None

Persistence Configuration

persistence:
  enabled: false
  mountPath: /agiletv/manager/data
  storageClass: ""
  accessModes:
    - ReadWriteOnce
  size: 8Gi

RBAC and Service Account

rbac:
  create: false
  rules: []

serviceAccount:
  create: true
  name: ""
  automountServiceAccountToken: true
  annotations: {}

Metrics

metrics:
  enabled: false
  serviceMonitor:
    enabled: false
    namespace: ""
    annotations: {}
    labels: {}
    interval: ""
    scrapeTimeout: ""

Next Steps

After configuration:

1. Installation Guide - Deploy with your configuration
2. Operations Guide - Day-to-day management
3. Performance Tuning Guide - Optimize system performance
4. Architecture Guide - Understand component relationships

7 - Performance Tuning Guide

Overview

This guide provides performance tuning recommendations for the AgileTV CDN Manager (ESB3027). While the default configuration is suitable for most deployments, certain environments may benefit from additional optimizations.

Network Topology Optimization

Topology Aware Hints

The CDN Manager uses Kubernetes Topology Aware Hints to prefer routing pods in the same zone as the source of network traffic. This reduces cross-zone latency and improves overall system responsiveness.

How It Works

When nodes are labeled with topology zones, Kubernetes automatically routes traffic to pods in the same zone when possible. This is particularly beneficial for:

• Low-latency requirements: Keeps traffic local to reduce round-trip time
• Cost optimization: Reduces cross-zone data transfer costs in cloud environments
• Load distribution: Prevents hotspots by distributing load across zones

Configuring Availability Zones

Each node must have zone and region labels applied for Topology Aware Hints to function:

# Label a node with a zone
kubectl label nodes <node-name> topology.kubernetes.io/zone=us-east-1a

# Label a node with a region
kubectl label nodes <node-name> topology.kubernetes.io/region=us-east-1

Replace <node-name> with your actual node names and adjust the zone/region values to match your deployment geography.

Note: Labels applied via kubectl label are automatically persistent and will survive node restarts.

Verify Topology Configuration

Verify labels are applied:

kubectl get nodes --show-labels | grep topology.kubernetes.io

Verify EndpointSlices are being generated with hints:

kubectl get endpointslices

Requirements for Topology Aware Hints

For Topology Aware Hints to activate:

• Minimum Nodes: At least one node must be labeled with each zone referenced by endpoints
• Symmetry: The control plane checks for sufficient CPU capacity across zones to balance traffic
• Zone Coverage: All zones with endpoints should have at least one ready node

Integration with Pod Anti-Affinity

Topology labels complement the pod anti-affinity rules already configured in the Helm chart:

• Pod Anti-Affinity: Handles pod-to-node placement to ensure high availability
• Topology Aware Hints: Handles service-to-pod traffic routing to keep requests within the same zone

Together, these features optimize both placement and routing for improved performance.

Fallback Behavior

If zone labels are not configured, the system falls back to random load-balancing across all available pods. This is functionally correct but may result in:

• Increased cross-zone traffic
• Higher latency for some requests
• Less predictable performance characteristics

Kernel Network Tuning (sysctl)

For high-throughput deployments, tuning Linux kernel network parameters can significantly improve connection handling and overall system performance. These settings are particularly beneficial for environments with high connection rates or large numbers of concurrent connections.

Recommended sysctl Settings

Apply the following settings to optimize network performance:

# Networking
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2048
net.ipv4.tcp_max_syn_backlog = 2048

# Connection Tracking
net.netfilter.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_tcp_timeout_established = 1200

# Port Reuse
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1

# Memory Buffers
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608

Setting Descriptions

Applying Settings

Temporary (Until Reboot)

Apply settings immediately but they will be lost on reboot:

sudo sysctl -w net.core.somaxconn=1024
sudo sysctl -w net.core.netdev_max_backlog=2048
# ... repeat for each parameter

Persistent (Across Reboots)

Add settings to /etc/sysctl.conf or a file in /etc/sysctl.d/:

# Create a dedicated config file
cat <<EOF | sudo tee /etc/sysctl.d/99-cdn-manager.conf
# CDN Manager Network Tuning
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2048
net.ipv4.tcp_max_syn_backlog = 2048
net.netfilter.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_tcp_timeout_established = 1200
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
EOF

# Apply all settings
sudo sysctl -p /etc/sysctl.d/99-cdn-manager.conf

Kubernetes Considerations

For Kubernetes deployments, these sysctl settings can be applied via:

1. Node-level configuration: Use DaemonSets or node provisioning scripts
2. Pod-level safe sysctls: Some sysctls can be set per-pod via securityContext.sysctls
3. Container runtime configuration: Configure via container runtime options

Note that some sysctls require privileged containers or node-level configuration.

Monitoring Impact

After applying these settings, monitor:

• Connection establishment rates
• TIME_WAIT socket count: netstat -n | grep TIME_WAIT | wc -l
• Connection tracking table usage: cat /proc/sys/net/netfilter/nf_conntrack_count
• Network buffer utilization via Grafana dashboards

Resource Configuration

Horizontal Pod Autoscaler (HPA)

The default HPA configuration is tuned for production workloads. For environments with variable load, consider adjusting the scale metrics:

For detailed HPA configuration, see the Architecture Guide.

Resource Requests and Limits

Ensure resource requests and limits are appropriately sized for your workload. Under-provisioned resources can cause:

• Pod evictions during high load
• Increased latency due to CPU throttling
• Slow scaling responses

Refer to the Configuration Guide for preset configurations and planning guidance.

Database Optimization

PostgreSQL

The PostgreSQL cluster is managed by the Cloudnative PG operator. For improved performance:

• Connection Pooling: The application uses connection pooling by default
• Replica Usage: Read queries can be offloaded to replicas for read-heavy workloads
• Backup Scheduling: Schedule backups during low-traffic periods to minimize I/O impact

Redis

Redis provides in-memory caching for sessions and ephemeral state:

• Memory Allocation: Ensure sufficient memory for cache hit rates
• Persistence: RDB snapshots are enabled; adjust frequency based on durability needs

Kafka

Kafka handles event streaming for selection input and metrics:

• Partition Count: Default partitions are sized for typical workloads
• Replication Factor: Production deployments use 3 replicas for fault tolerance
• Consumer Groups: The Selection Input Worker is limited to one consumer per partition

Monitoring Performance

Key Metrics to Watch

Monitor the following metrics for performance insights:

• API Response Time: Track via Grafana dashboards
• Pod CPU/Memory Usage: Identify resource bottlenecks
• Kafka Lag: Monitor consumer lag for selection input processing
• Database Connections: Watch for connection pool exhaustion

Grafana Dashboards

Pre-built dashboards are available at https://<manager-host>/grafana:

• System Health: Overall cluster and application health
• CDN Metrics: Routing and usage statistics
• Resource Utilization: CPU, memory, and network usage per component

Troubleshooting Performance Issues

High Latency

1. Check pod distribution across nodes: kubectl get pods -o wide
2. Verify topology labels are applied: kubectl get nodes --show-labels
3. Review network latency between nodes
4. Check for resource contention: kubectl top pods

Slow Scaling

1. Verify HPA is enabled: kubectl get hpa
2. Check cluster capacity for scheduling new pods
3. Review HPA metrics: kubectl describe hpa acd-manager

Database Performance

1. Check PostgreSQL cluster status: kubectl get pods -l app=postgresql
2. Review slow query logs (if enabled)
3. Monitor connection pool usage

Next Steps

After reviewing performance tuning:

1. Architecture Guide - Understand component interactions
2. Configuration Guide - Detailed configuration options
3. Metrics & Monitoring Guide - Comprehensive monitoring setup
4. Troubleshooting Guide - Resolve performance issues

8 - Operations Guide

Overview

This guide covers day-to-day operational procedures for managing the AgileTV CDN Manager (ESB3027). Topics include routine maintenance, backup procedures, log management, and common operational tasks.

Prerequisites

Before performing operations, ensure you have:

• kubectl access to the cluster
• helm CLI installed
• Access to the node where values.yaml is stored
• Appropriate RBAC permissions for administrative tasks

Cluster Access

There are two supported methods for accessing the Kubernetes cluster:

1. SSH to a Server Node (Recommended for operations staff) - SSH into any Server node and run kubectl commands directly
2. Remote kubectl - Install kubectl on your local machine and configure it to connect to the cluster remotely

Method 1: SSH to Server Node (Recommended)

The kubectl command-line tool is pre-configured on all Server nodes and can be used directly without additional setup:

# SSH to any Server node
ssh root@<server-ip>

# Run kubectl commands directly
kubectl get nodes
kubectl get pods

This method is recommended for day-to-day operations as it requires no local configuration and provides direct access to the cluster.

Method 2: Remote kubectl from Local Machine

To use kubectl from your local workstation or laptop:

Step 1: Install kubectl

Download and install kubectl for your operating system:

• Official Documentation: Install kubectl
• macOS (Homebrew): brew install kubectl
• Linux: Download from the official Kubernetes release page
• Windows: Download from the official Kubernetes release page

Step 2: Copy kubeconfig from Server Node

# Copy kubeconfig from any Server node
scp root@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Step 3: Update kubeconfig

Edit the kubeconfig file to point to the correct server address:

# Replace localhost with the actual server IP
# macOS/Linux:
sed -i '' 's/127.0.0.1/<server-ip>/g' ~/.kube/config  # macOS
sed -i 's/127.0.0.1/<server-ip>/g' ~/.kube/config    # Linux

# Or manually edit ~/.kube/config and change:
# server: https://127.0.0.1:6443
# to:
# server: https://<server-ip>:6443

Step 4: Verify connectivity

kubectl get nodes

Managing Multiple Clusters

If you manage multiple Kubernetes clusters from the same machine, you can maintain multiple kubeconfig files:

# Set KUBECONFIG environment variable to include multiple config files
export KUBECONFIG=~/.kube/config-prod:~/.kube/config-lab

# View all contexts
kubectl config get-contexts

# Switch between clusters
kubectl config use-context <context-name>

# View current context
kubectl config current-context

For more information, see the official Kubernetes documentation: Organizing Cluster Access

Helm Commands

Helm releases are managed cluster-wide:

# List all releases
helm list

# View release history
helm history acd-manager

# Get deployed values
helm get values acd-manager -o yaml

# Get deployed manifest
helm get manifest acd-manager

Note: If using remote kubectl, ensure helm is installed on your local machine. See Helm Installation for instructions.

Helm Commands

Helm releases are managed cluster-wide:

# List all releases
helm list

# View release history
helm history acd-manager

# Get deployed values
helm get values acd-manager -o yaml

# Get deployed manifest
helm get manifest acd-manager

Backup Procedures

PostgreSQL Backup

PostgreSQL is managed by the Cloudnative PG operator, which provides continuous backup capabilities.

# Check backup status
kubectl get backup

# Create manual backup
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: manual-backup-$(date +%Y%m%d-%H%M%S)
spec:
  cluster:
    name: acd-cluster-postgresql
EOF

# List available backups
kubectl get backup -o wide

# Restore from backup (requires downtime)
# See Upgrade Guide for restore procedures

Longhorn Volume Backups

Longhorn provides snapshot and backup capabilities for persistent volumes:

# List all volumes
kubectl get volumes -n longhorn-system

# Create snapshot via Longhorn UI
# Port-forward to Longhorn UI (do not expose via ingress)
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

# Access: http://localhost:8080
# WARNING: Longhorn UI grants access to sensitive storage information
# and should never be exposed through the ingress controller

Accessing Internal Services

For debugging and troubleshooting, you may need direct access to internal services.

PostgreSQL

PostgreSQL is managed by the Cloudnative PG operator. Connection details are stored in the acd-cluster-postgresql-app Secret:

# View connection details
kubectl describe secret acd-cluster-postgresql-app

# Extract individual fields
PG_HOST=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.host}' | base64 -d)
PG_USER=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.username}' | base64 -d)
PG_PASS=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.password}' | base64 -d)
PG_DB=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.dbname}' | base64 -d)

# Connect via psql
kubectl exec -it acd-cluster-postgresql-0 -- psql -U $PG_USER -d $PG_DB

Secret fields: The CNPG operator populates the following fields: username, password, host, port, dbname, uri, jdbc-uri, fqdn-uri, fqdn-jdbc-uri, pgpass.

Redis

Redis runs on port 6379 with no authentication:

# Connect via redis-cli
kubectl exec -it acd-manager-redis-master-0 -- redis-cli

# Or connect from another pod
kubectl run redis-test --rm -it --image=redis -- redis-cli -h acd-manager-redis-master

Kafka

Kafka is accessible on port 9095 from any cluster node:

# Connect from within cluster
kubectl exec -it acd-manager-kafka-controller-0 -- kafka-topics.sh --bootstrap-server localhost:9092 --list

# Connect from external (via any node IP)
kafka-topics.sh --bootstrap-server <node-ip>:9095 --list

The selection_input topic is pre-configured for selection input events.

Longhorn Storage

Longhorn is a distributed block storage system for Kubernetes that provides persistent volumes for stateful applications such as PostgreSQL and Kafka.

Architecture

Longhorn deploys controller and replica engines on each node, forming a distributed storage system. When a volume is created, Longhorn replicates data across multiple nodes to ensure durability even in the event of node failures.

Storage Protocols:

• iSCSI: Used for standard Read-Write-Once (RWO) volumes
• NFS: Used for Read-Write-Many (RWX) volumes that can be mounted by multiple pods simultaneously

Configuration

The CDN Manager deploys Longhorn with a single replica configuration, which differs from the Longhorn default of 3 replicas. This configuration is optimized for the cluster architecture where:

• Pod-node affinity is configured to schedule pods on the same node as their persistent volume data
• This optimizes I/O performance by reducing network traffic
• Data locality is maintained while still providing volume portability

Capacity Planning

Longhorn storage requires an additional 30% capacity headroom for internal operations and scaling. If less than 30% of the total partition capacity is available, Longhorn may mark volumes as “full” and prevent further writes.

For detailed storage requirements and disk partitioning guidance, see the System Requirements Guide.

Configuration Backup

Always backup your Helm values before making changes:

# Export current values
helm get values acd-manager -o yaml > ~/values-backup-$(date +%Y%m%d).yaml

# Backup custom values files
cp ~/values.yaml ~/values-backup-$(date +%Y%m%d).yaml

Backup Schedule Recommendations

Updating MaxMind GeoIP Databases

The MaxMind GeoIP databases (GeoIP2-City, GeoLite2-ASN, GeoIP2-Anonymous-IP) are used for GeoIP-based routing and validation features. These databases should be updated periodically to ensure accurate IP geolocation data.

Prerequisites

• Updated MaxMind database files (.mmdb format) obtained from MaxMind
• Access to the cluster via kubectl
• Helm CLI installed

Update Procedure

Step 1: Create New Volume with Updated Databases

Run the volume generation utility with a unique volume name that includes a revision identifier:

# Mount the installation ISO if not already mounted
mkdir -p /mnt/esb3027
mount -o loop,ro esb3027-acd-manager-X.Y.Z.iso /mnt/esb3027

# Generate new volume with updated databases
/mnt/esb3027/generate-maxmind-volume

When prompted:

1. Provide the paths to the three database files:
   • GeoIP2-City.mmdb
   • GeoLite2-ASN.mmdb
   • GeoIP2-Anonymous-IP.mmdb
2. Enter a unique volume name with a revision number or date, for example:
   • maxmind-geoip-2026-04
   • maxmind-geoip-v2

Tip: Using a revision-based naming convention simplifies rollback if needed.

Step 2: Update Helm Configuration

Edit your values.yaml file to reference the new volume:

manager:
  maxmindDbVolume: maxmind-geoip-2026-04

Replace maxmind-geoip-2026-04 with the volume name you specified in Step 1.

Step 3: Apply Configuration Update

Upgrade the Helm release with the updated configuration:

helm upgrade acd-manager /mnt/esb3027/charts/acd-manager --values ~/values.yaml

Step 4: Rolling Restart (Optional)

To ensure all pods immediately use the new database files, perform a rolling restart of the manager deployment:

kubectl rollout restart deployment acd-manager

Monitor the rollout status:

kubectl rollout status deployment acd-manager

Step 5: Verify Update

Verify the pods are running with the new volume:

kubectl get pods
kubectl describe pod -l app.kubernetes.io/component=manager | grep -A 5 "Volumes"

Step 6: Clean Up Old Volume (Optional)

After verifying the new databases are working correctly, you can delete the old persistent volume:

# List persistent volumes to find the old one
kubectl get pv

# Delete the old volume
kubectl delete pv <old-volume-name>

Caution: Ensure the new volume is functioning correctly before deleting the old volume. Keep the old volume for at least 24-48 hours as a rollback option.

Rollback Procedure

If issues occur after updating the databases:

1. Revert the maxmindDbVolume value in your values.yaml to the previous volume name
2. Run helm upgrade with the reverted configuration
3. Optionally restart the deployment: kubectl rollout restart deployment acd-manager

Update Frequency Recommendations

MaxMind releases database updates on a regular schedule. Subscribe to MaxMind notifications to stay informed of new releases.

Log Management

Application Logs

# View manager logs
kubectl logs -l app.kubernetes.io/component=manager

# Follow logs in real-time
kubectl logs -l app.kubernetes.io/component=manager -f

# View logs from specific pod
kubectl logs <pod-name>

# View previous instance logs (after crash)
kubectl logs <pod-name> -p

# View logs with timestamps
kubectl logs <pod-name> --timestamps

# View logs from all containers in pod
kubectl logs <pod-name> --all-containers

Component-Specific Logs

# Zitadel logs
kubectl logs -l app.kubernetes.io/name=zitadel

# Gateway logs
kubectl logs -l app.kubernetes.io/component=gateway

# Confd logs
kubectl logs -l app.kubernetes.io/component=confd

# MIB Frontend logs
kubectl logs -l app.kubernetes.io/component=mib-frontend

# PostgreSQL logs
kubectl logs -l app.kubernetes.io/name=postgresql

# Kafka logs
kubectl logs -l app.kubernetes.io/name=kafka

# Redis logs
kubectl logs -l app.kubernetes.io/name=redis

Log Aggregation

Logs are collected by Telegraf and sent to VictoriaMetrics:

# Access Grafana for log visualization
# https://<manager-host>/grafana

# Query logs via Grafana Explore
# Select VictoriaMetrics datasource and use log queries