Operations Guide

Overview

This guide covers day-to-day operational procedures for managing the AgileTV CDN Manager (ESB3027). Topics include routine maintenance, backup procedures, log management, and common operational tasks.

Prerequisites

Before performing operations, ensure you have:

• kubectl access to the cluster (available as root on every server)
• Access to the first server, where deploy and its configuration in /etc/acd/deploy live
• Appropriate RBAC permissions for administrative tasks

Cluster Access

There are two supported methods for accessing the Kubernetes cluster:

1. SSH to a Server Node (Recommended for operations staff) - SSH into any Server node and run kubectl commands directly
2. Remote kubectl - Install kubectl on your local machine and configure it to connect to the cluster remotely

Method 1: SSH to Server Node (Recommended)

The kubectl command-line tool is pre-configured on all Server nodes and can be used directly without additional setup:

# SSH to any Server node
ssh root@<server-ip>

# Run kubectl commands directly
kubectl get nodes
kubectl get pods

This method is recommended for day-to-day operations as it requires no local configuration and provides direct access to the cluster.

For routine status checks, logs, restarts and node maintenance, acd-admin on a server does the same by admin target name, without kubectl.

Method 2: Remote kubectl from Local Machine

To use kubectl from your local workstation or laptop:

Step 1: Install kubectl

Download and install kubectl for your operating system:

• Official Documentation: Install kubectl
• macOS (Homebrew): brew install kubectl
• Linux: Download from the official Kubernetes release page
• Windows: Download from the official Kubernetes release page

Step 2: Copy kubeconfig from Server Node

# Copy kubeconfig from any Server node
scp root@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Step 3: Update kubeconfig

Edit the kubeconfig file to point to the correct server address:

# Replace localhost with the actual server IP
# macOS/Linux:
sed -i '' 's/127.0.0.1/<server-ip>/g' ~/.kube/config  # macOS
sed -i 's/127.0.0.1/<server-ip>/g' ~/.kube/config    # Linux

# Or manually edit ~/.kube/config and change:
# server: https://127.0.0.1:6443
# to:
# server: https://<server-ip>:6443

Step 4: Verify connectivity

kubectl get nodes

Managing Multiple Clusters

If you manage multiple Kubernetes clusters from the same machine, you can maintain multiple kubeconfig files:

# Set KUBECONFIG environment variable to include multiple config files
export KUBECONFIG=~/.kube/config-prod:~/.kube/config-lab

# View all contexts
kubectl config get-contexts

# Switch between clusters
kubectl config use-context <context-name>

# View current context
kubectl config current-context

For more information, see the official Kubernetes documentation: Organizing Cluster Access

Helm Commands

Helm releases are managed cluster-wide. Besides cdn-platform, the cluster runs six releases of its own: see Configuration.

# List all releases
helm list -A

# View release history
helm history cdn-platform

# Get deployed values
helm get values cdn-platform -o yaml

# Get deployed manifest
helm get manifest cdn-platform

These commands are read-only. Change the release only through deploy: see Changing Application Settings.

Backup Procedures

PostgreSQL Backup

PostgreSQL is managed by the Cloudnative PG operator, which provides continuous backup capabilities.

A backup of the cluster covers all of its databases: manager (the manager’s own data, including the subnet map), app (Grafana), zitadel and confsej. The subnet map is not in Redis, so back up PostgreSQL to keep it.

# Check backup status
kubectl get backup

# Create manual backup
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: manual-backup-$(date +%Y%m%d-%H%M%S)
spec:
  cluster:
    name: acd-cluster-postgresql
EOF

# List available backups
kubectl get backup -o wide

# Restore from backup (requires downtime)
# See Upgrade Guide for restore procedures

Longhorn Volume Backups

Longhorn provides snapshot and backup capabilities for persistent volumes:

# List all volumes
kubectl get volumes -n longhorn-system

# Create snapshot via Longhorn UI
# Port-forward to Longhorn UI (do not expose via ingress)
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

# Access: http://localhost:8080
# WARNING: Longhorn UI grants access to sensitive storage information
# and should never be exposed through the ingress controller

Accessing Internal Services

For debugging and troubleshooting, you may need direct access to internal services.

PostgreSQL

PostgreSQL is managed by the Cloudnative PG operator. Connection details are stored in the acd-cluster-postgresql-app Secret:

# View connection details
kubectl describe secret acd-cluster-postgresql-app

# Extract individual fields
PG_HOST=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.host}' | base64 -d)
PG_USER=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.username}' | base64 -d)
PG_PASS=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.password}' | base64 -d)
PG_DB=$(kubectl get secret acd-cluster-postgresql-app -o jsonpath='{.data.dbname}' | base64 -d)

# Connect via psql
kubectl exec -it acd-cluster-postgresql-1 -- psql -U $PG_USER -d $PG_DB

Secret fields: The CNPG operator populates the following fields: username, password, host, port, dbname, uri, jdbc-uri, fqdn-uri, fqdn-jdbc-uri, pgpass.

Redis

Redis runs on port 6379 with no authentication:

# Connect via redis-cli
kubectl exec -it cdn-platform-redis-master-0 -- redis-cli

# Or connect from another pod
kubectl run redis-test --rm -it --image=redis -- redis-cli -h cdn-platform-redis-master

Kafka

Kafka’s bootstrap listens on the default port, 9092, on every machine. See Kafka Ports. To list the topics from a host with the Kafka command-line tools:

kafka-topics.sh --bootstrap-server <node-ip>:9092 --list

The selection_input topic is pre-configured for selection input events, with a single replica by design: see Data Replication.

Kubernetes Port Forwarding

For accessing internal Kubernetes services that are not exposed via ingress or services, use kubectl port-forward to create a secure tunnel from your local machine to the service.

Basic Port Forwarding

# Forward local port to a service
kubectl port-forward -n <namespace> svc/<service-name> <local-port>:<service-port>

# Example: Forward local port 8080 to Grafana (port 3000)
kubectl port-forward -n default svc/grafana 8080:3000

Note: “Local” refers to the machine where you run kubectl. This can be:

• A Server node in the cluster (common for administrative tasks)
• A remote machine with kubectl configured to access the cluster

Accessing the Forwarded Service

Once the port-forward is established, access the service at http://localhost:<local-port> from the machine where you ran kubectl port-forward.

If running on a Server node: kubectl port-forward listens only on the node’s own loopback address, 127.0.0.1, so the node’s IP address does not reach it from your workstation. Carry the port over SSH instead, from your workstation:

# From your workstation: forward <local-port> to the same port on the Server node
ssh -L <local-port>:127.0.0.1:<local-port> root@<server-node>

Keep that SSH session open, and open http://localhost:<local-port> on your workstation.

For simplicity, consider running port-forward from your local machine (if kubectl is configured for remote cluster access) rather than from a Server node.

Background Port Forwarding

To run port-forward in the background:

kubectl port-forward -n <namespace> svc/<service-name> <local-port>:<service-port> &

Security Considerations

Port forwarding is recommended for:

• Administrative interfaces (e.g., Longhorn UI) that should not be publicly exposed
• Debugging and troubleshooting internal services
• Temporary access to services without modifying ingress configuration

The port-forward tunnel remains active only while the kubectl port-forward command is running. Press Ctrl+C to terminate the tunnel.

Example: The Longhorn storage UI is intentionally not exposed via ingress due to security risks. Access it via port-forward:

kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

Then navigate to http://localhost:8080 in your browser.

Longhorn Storage

Longhorn is a distributed block storage system for Kubernetes that provides persistent volumes for stateful applications such as PostgreSQL and Kafka.

Architecture

Longhorn deploys controller and replica engines on each node, forming a distributed storage system. When a volume is created, Longhorn replicates data across multiple nodes to ensure durability even in the event of node failures.

Storage Protocols:

• iSCSI: Used for standard Read-Write-Once (RWO) volumes
• NFS: Used for Read-Write-Many (RWX) volumes that can be mounted by multiple pods simultaneously

Configuration

Each volume keeps three replicas on different nodes, the Longhorn default. The three Kafka data volumes are the exception, with one replica each, because Kafka copies its topics across its brokers itself. Longhorn tries to keep one replica on the node where the volume’s pod runs, but a pod can run on a node with no replica of its volume.

When a node is drained, its replicas stop. Volumes with three replicas stay degraded while the node is away; on a cluster with a spare node, Longhorn rebuilds the missing replica there after about 10 minutes. After a short drain, the replicas catch up within about a minute of the node’s return. Before the drain of a node that holds a Kafka volume’s only replica finishes, Longhorn copies that replica to another node.

Capacity Planning

Longhorn storage requires an additional 30% capacity headroom for internal operations and scaling. If less than 30% of the total partition capacity is available, Longhorn may mark volumes as “full” and prevent further writes.

For detailed storage requirements and disk partitioning guidance, see the System Requirements Guide.

Configuration Backup

Every run of deploy backs up each machine’s configuration, and the configuration folder /etc/acd/deploy, before it changes anything. It does not back up the data stored in the cluster: use the PostgreSQL procedure above for the databases. To take a backup without changing anything:

/mnt/esb3027/deploy backup

The backup leaves out your TLS certificate and key and the MaxMind databases: keep your own copies. A backup of a running cluster holds an etcd snapshot with every Kubernetes Secret, and the cluster token: protect and prune backups as secret material. See Backups.

Backup Schedule Recommendations

Updating MaxMind GeoIP Databases

The MaxMind GeoIP databases (GeoIP2-City, GeoLite2-ASN, GeoIP2-Anonymous-IP) are used for GeoIP-based routing and validation features. Update them periodically to keep IP geolocation accurate.

To update them, copy the new .mmdb files into /etc/acd/deploy/maxmind/, replacing the old ones, and run deploy. It loads the databases only when they have changed. See MaxMind GeoIP Databases.

Update Frequency Recommendations

MaxMind releases database updates on a regular schedule. Subscribe to MaxMind notifications to stay informed of new releases.

Log Management

acd-admin logs <target> shows the logs of every pod of a component at once: see acd-admin logs.

Application Logs

# View manager logs
kubectl logs -l app.kubernetes.io/component=manager

# Follow logs in real-time
kubectl logs -l app.kubernetes.io/component=manager -f

# View logs from specific pod
kubectl logs <pod-name>

# View previous instance logs (after crash)
kubectl logs <pod-name> -p

# View logs with timestamps
kubectl logs <pod-name> --timestamps

# View logs from all containers in pod
kubectl logs <pod-name> --all-containers

Component-Specific Logs

# Zitadel logs
kubectl logs -l app.kubernetes.io/name=zitadel

# Gateway logs
kubectl logs -l app.kubernetes.io/component=gateway

# Confd logs
kubectl logs -l app.kubernetes.io/component=confd

# MIB Frontend logs
kubectl logs -l app.kubernetes.io/component=mib-frontend

# PostgreSQL logs
kubectl logs -l app.kubernetes.io/name=postgresql

# Kafka logs
kubectl logs -l app.kubernetes.io/name=kafka

# Redis logs
kubectl logs -l app.kubernetes.io/name=redis

Log Aggregation

Logs are collected by Telegraf and sent to VictoriaMetrics:

# Access Grafana for log visualization
# https://<manager-host>/grafana

# Query logs via Grafana Explore
# Select VictoriaMetrics datasource and use log queries

Log Rotation

Container logs are automatically rotated by Kubernetes:

• Default max size: 10MB per container
• Default max files: 5 rotated files
• Total per pod: ~50MB maximum

Scaling Operations

Manual Scaling

Note: If HPA (Horizontal Pod Autoscaler) is enabled for a deployment, manual scaling changes will be overridden by the HPA. To manually scale, you must first disable the HPA.

# Check if HPA is enabled
kubectl get hpa

# Disable HPA before manual scaling
kubectl patch hpa cdn-platform-cdn-manager -p '{"spec": {"minReplicas": null, "maxReplicas": null}}'

# Or delete the HPA entirely
kubectl delete hpa cdn-platform-cdn-manager

# Scale manager replicas
kubectl scale deployment cdn-platform-cdn-manager --replicas=3

# Scale gateway replicas
kubectl scale deployment cdn-platform-cdn-manager-gateway --replicas=2

# Scale MIB frontend replicas
kubectl scale deployment cdn-platform-cdn-manager-mib-frontend --replicas=2

HPA Configuration

# View HPA status
kubectl get hpa

# Describe HPA details
kubectl describe hpa cdn-platform-cdn-manager

# Edit HPA configuration
kubectl edit hpa cdn-platform-cdn-manager

Configuration Updates

Changing Application Settings

Application settings are changed through deploy, never with helm directly: the next deploy would undo the change.

# Write only the settings you want to change
vi /etc/acd/deploy/values.override.yaml

# Apply them
/mnt/esb3027/deploy

# Check every component
acd-admin status

See Application Settings.

Rolling Back Changes

To undo a change, revert it in values.override.yaml or group_vars/all.yaml and run deploy again. To see the release’s revisions:

helm history cdn-platform

Certificate Management

Checking Certificate Expiration

# Check TLS secret expiration
kubectl get secret <tls-secret> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

# Check via Grafana dashboard
# Certificate expiration metrics are available in Grafana

Renewing Certificates

Replace the certificate and key in /etc/acd/deploy/certs/ and run deploy. See TLS Certificates.

Health Checks

acd-admin status shows the health of every component and node in one view: see acd-admin status.

Component Health

# Check all pods
kubectl get pods

# Check specific component
kubectl get pods -l app.kubernetes.io/component=manager

# Check persistent volumes
kubectl get pvc

# Check cluster status
kubectl get nodes

# Check ingress
kubectl get ingress

API Health Endpoints

# Liveness check
curl -k https://<manager-host>/api/v1/health/alive

# Readiness check
curl -k https://<manager-host>/api/v1/health/ready

Database Health

# PostgreSQL cluster status
kubectl get clusters -n default

# Check PostgreSQL pods
kubectl get pods -l app.kubernetes.io/name=postgresql

# Kafka cluster status
kubectl get pods -l app.kubernetes.io/name=kafka

# Redis status
kubectl get pods -l app.kubernetes.io/name=redis

Maintenance Windows

Planned Maintenance

Before performing maintenance:

1. Notify users of potential service impact
2. Verify backups are current
3. Document the maintenance procedure
4. Prepare rollback plan

Node Maintenance

# Take the node out of service: cordon it and evict its pods
acd-admin drain <node-name>

# Perform maintenance

# Return the node to service
acd-admin uncordon <node-name>

Before you drain or restart anything, read Safety: drain one server at a time, check which broker holds selection_input, and wait for every admin target ok and every storage volume healthy before the next drain. Cordoning or draining the node of the PostgreSQL primary switches the primary to another instance.

A drained node keeps running its DaemonSet pods (host metrics, ingress and storage) and any acd-admin session. Everything else moves to other nodes. Longhorn’s node components keep running, but its storage replicas on the node stop: see Longhorn Storage for what that means for volumes.

acd-admin drain refuses a node that runs a pod that no controller recreates until you add --force. Ctrl-C stops only the waiting, and pods still waiting to move are not retried until drain runs again, which carries on. The node stays drained through reboots and deploy runs until acd-admin uncordon; kubectl uncordon returns the node to service but leaves its drained label. See acd-admin drain and Drained Nodes.

Uncordoning a node does not move pods back to it: it takes only new pods. After a round of drains, the last node returned can sit almost empty while the others carry its load. The descheduler gradually moves pods without persistent storage from busy nodes to underused ones. To spread a target’s pods sooner, restart it with acd-admin restart <target>, starting with the targets that have the most pods.

Cluster Upgrades

See the Upgrade Guide for cluster upgrade procedures.

Troubleshooting Quick Reference

Common Commands

# Describe problematic pod
kubectl describe pod <pod-name>

# View pod events
kubectl get events --sort-by='.lastTimestamp'

# Check resource usage
kubectl top pods
kubectl top nodes

# Exec into container
kubectl exec -it <pod-name> -- /bin/sh

# Check network policies
kubectl get networkpolicies

# Check service endpoints
kubectl get endpoints

Restarting Components

acd-admin restart <target> restarts an admin target the safe way for its kind and waits for it, including kafka and postgresql through their operators: see acd-admin restart.

Important: Never restart Kafka or PostgreSQL with the kubectl commands below. Their operators, Strimzi and CloudNativePG, own their pods: a kubectl restart or pod deletion bypasses the broker-by-broker roll and the PostgreSQL switchover. Use acd-admin restart kafka and acd-admin restart postgresql.

For other workloads:

# Restart deployment
kubectl rollout restart deployment/<deployment-name>

# Restart statefulset
kubectl rollout restart statefulset/<statefulset-name>

# Delete pod (auto-recreated)
kubectl delete pod <pod-name>

Security Operations

Rotating Service Account Tokens

# Delete service account secret (auto-regenerated)
kubectl delete secret <service-account-token-secret>

# Tokens are automatically regenerated

Updating RBAC Permissions

# View current roles
kubectl get roles
kubectl get clusterroles

# View role bindings
kubectl get rolebindings
kubectl get clusterrolebindings

# Edit role
kubectl edit role <role-name>

Disaster Recovery

Pod Recovery

Pods are automatically recreated if they fail:

# Check pod status
kubectl get pods

# If pod is stuck in Terminating
kubectl delete pod <pod-name> --force --grace-period=0

# If pod is stuck in Pending, check resources
kubectl describe pod <pod-name>
kubectl get events --sort-by='.lastTimestamp'

Node Failure Recovery

When a node fails:

1. Automatic: Pods are rescheduled on healthy nodes (after timeout)
2. Manual: Force delete stuck pods

# Force delete pods on failed node
kubectl delete pod --all --force --grace-period=0 \
  --field-selector spec.nodeName=<failed-node>

Data Recovery

For data recovery scenarios, refer to:

• PostgreSQL: Cloudnative PG backup/restore procedures
• Longhorn: Volume snapshot restoration
• Kafka: Topics other than selection_input survive the loss of one broker. selection_input has a single replica: see Data Replication

Routine Maintenance Checklist

Daily

• Review Grafana dashboards for anomalies
• Check alert notifications
• Verify backup completion

Weekly

• Review pod restart counts
• Check certificate expiration dates
• Review log storage usage
• Verify HPA is functioning correctly

Monthly

• Test backup restoration procedure
• Review and rotate credentials if needed
• Update documentation if configuration changed
• Review resource utilization trends

Next Steps

After mastering operations:

1. Troubleshooting Guide - Deep dive into problem resolution
2. Performance Tuning Guide - Optimize system performance
3. Metrics & Monitoring Guide - Comprehensive monitoring setup
4. API Guide - REST API reference and automation