Troubleshooting Guide

Common issues and resolution procedures
You're viewing a development version of manager, the latest released version is 1.6.2
Go to the latest released version

Overview

This guide provides troubleshooting procedures for common issues encountered when operating the AgileTV CDN Manager (ESB3027). Use the diagnostic commands and resolution steps to identify and resolve problems.

Diagnostic Tools

On a server, acd-admin covers the most common checks and fixes by admin target name: acd-admin status, acd-admin logs <target> and acd-admin restart <target>.

Cluster Status

# Check node status
kubectl get nodes

# Check all pods
kubectl get pods -A

# Check events sorted by time
kubectl get events --sort-by='.lastTimestamp'

# Check resource usage
kubectl top nodes
kubectl top pods

Component Status

# Check deployments
kubectl get deployments

# Check statefulsets
kubectl get statefulsets

# Check persistent volumes
kubectl get pvc
kubectl get pv

# Check services
kubectl get services

# Check ingress
kubectl get ingress

Common Issues

Pods Stuck in Pending State

Symptoms: Pods remain in Pending state indefinitely.

Causes:

  • Insufficient cluster resources (CPU/memory)
  • No nodes match scheduling constraints
  • PersistentVolume not available

Diagnosis:

# Describe the pending pod
kubectl describe pod <pod-name>

# Check events for scheduling failures
kubectl get events --field-selector reason=FailedScheduling

# Check node capacity
kubectl describe nodes | grep -A 5 "Allocated resources"

# Check available PVs
kubectl get pv

Resolution:

# Free up resources by scaling down non-critical workloads
kubectl scale deployment <deployment> --replicas=0

# Or add additional nodes to the cluster

# If PV is stuck, delete and recreate
kubectl delete pvc <pvc-name>
kubectl delete pod <pod-name>

Pods Stuck in ContainerCreating

Symptoms: Pods remain in ContainerCreating state.

Causes:

  • Image pull failures
  • Volume mount issues
  • Network configuration problems

Diagnosis:

kubectl describe pod <pod-name>

# Check for image pull errors
kubectl get events | grep -i "failed to pull"

# Check volume mount status
kubectl get events | grep -i "mount"

Resolution:

# For image pull issues, verify image exists and credentials
kubectl get secret <pull-secret-name> -o yaml

# For volume issues, check Longhorn volume status
kubectl get volumes -n longhorn-system

# Delete stuck pod to trigger recreation
kubectl delete pod <pod-name> --force --grace-period=0

Persistent Volume Mount Failures

Symptoms: Pod fails to start with error “AttachVolume.Attach failed for volume… is not ready for workloads” or similar volume attachment errors.

Causes:

  • Longhorn volume created but unable to be successfully mounted
  • Network connectivity issues between nodes (Longhorn requires iSCSI and NFS traffic)
  • Longhorn service unhealthy
  • Incorrect storage class configuration

Diagnosis:

# Describe the failing pod to see the error
kubectl describe pod <pod-name>

# Check Longhorn volumes status
kubectl get volumes -n longhorn-system

# Check Longhorn UI for detailed volume status
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80
# Access: http://localhost:8080

Resolution:

# Longhorn traffic runs over the pod network, which deploy trusts between nodes.
# Check that each node's firewall still matches what deploy set up: run deploy
# again, and it stops with an [ACD-VERIFY-...] code if a node's network differs.

# Check Longhorn is healthy
kubectl get pods -n longhorn-system

# If volume is stuck, delete PVC and pod to trigger recreation
kubectl delete pvc <pvc-name>
kubectl delete pod <pod-name>

Pods in CrashLoopBackOff

Symptoms: Pods repeatedly crash and restart.

Causes:

  • Application configuration errors
  • Missing dependencies (database not ready)
  • Resource limits too low
  • Liveness probe failures

Diagnosis:

# View current logs
kubectl logs <pod-name>

# View previous instance logs
kubectl logs <pod-name> -p

# Describe pod for restart reasons
kubectl describe pod <pod-name>

# Check if dependencies are healthy
kubectl get pods | grep -E "(postgres|kafka|redis)"

Resolution:

# For dependency issues, wait for dependencies to be ready
kubectl wait --for=condition=Ready pod/<dependency-pod> --timeout=300s

# For resource issues, increase limits
kubectl edit deployment <deployment-name>

# For configuration issues, check ConfigMaps and Secrets
kubectl get configmap <configmap-name> -o yaml
kubectl get secret <secret-name> -o yaml

# Restart the deployment
kubectl rollout restart deployment/<deployment-name>

Pods in Terminating State

Symptoms: Pods stuck in Terminating state indefinitely.

Causes:

  • Volume detachment issues
  • Node communication problems
  • Finalizer blocking deletion

Diagnosis:

kubectl describe pod <pod-name>

# Check if node is reachable
kubectl get nodes

# Check finalizers
kubectl get pod <pod-name> -o jsonpath='{.metadata.finalizers}'

Resolution:

# Force delete the pod
kubectl delete pod <pod-name> --force --grace-period=0

# If node is unreachable, drain and remove from cluster
acd-admin drain <node-name>
# Add --force only if drain refuses and you accept losing the pods it names.
# Pods on an unreachable node never finish terminating, so once drain is
# waiting only on those, press Ctrl-C. The node stays drained.
kubectl delete node <node-name>

acd-admin drain names each pod it is waiting on, with when its deletion started and the finalizer holding it, if any.

Leftover acd-admin Pods

Symptoms: acd-admin-* pods remain in the acd-system namespace after an SSH session was lost.

Cause: Each acd-admin command runs in its own pod, which is removed when the command ends, Ctrl-C included. A session lost so abruptly that the command cannot clean up leaves its pod behind, holding its resources. Its command may still be running, such as a drain or the wait of a restart. A drain never evicts these pods. A leftover pod ends on its own at its session deadline, which for a drain, shell or exec is 12 hours and for logs -f 24 hours: see acd-admin.

Diagnosis:

# The age tells a leftover from a command someone is running now
kubectl get pods -n acd-system

Resolution:

kubectl delete pod -n acd-system <pod-name>

Deleting the pod stops its command. A drain stopped this way leaves the node as Ctrl-C does: drained, with the pods already moved where they went. A restart already started carries on in the cluster.

Service Unreachable

Symptoms: Service endpoints not accessible.

Causes:

  • No ready pods backing the service
  • Network policy blocking traffic
  • Service port mismatch

Diagnosis:

# Check service endpoints
kubectl get endpoints <service-name>

# Check if pods are ready
kubectl get pods -l app=<label>

# Check network policies
kubectl get networkpolicies

# Test connectivity from within cluster
kubectl run test --rm -it --image=busybox -- wget -O- <service-name>:<port>

Resolution:

# Ensure pods are ready and matching service selector
kubectl get pods --show-labels

# Check service selector matches pod labels
kubectl get service <service-name> -o jsonpath='{.spec.selector}'

# Temporarily disable network policy for testing
kubectl edit networkpolicy <policy-name>

Ingress Not Working

Symptoms: External access via ingress fails.

Causes:

  • Traefik ingress controller not running
  • Ingress configuration errors
  • TLS certificate issues
  • DNS resolution problems

Diagnosis:

# Check Traefik pods
kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik

# Check ingress resources
kubectl get ingress

# Describe ingress for errors
kubectl describe ingress <ingress-name>

# Check Traefik logs
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik

# Test DNS resolution
nslookup <hostname>

Resolution:

# Restart Traefik, which runs as a DaemonSet: one pod per node
kubectl rollout restart daemonset -n kube-system traefik

# Fix ingress configuration
kubectl edit ingress <ingress-name>

# Renew or recreate TLS secret
kubectl create secret tls <secret-name> --cert=tls.crt --key=tls.key \
  --dry-run=client -o yaml | kubectl apply -f -

# Verify hostname matches certificate
openssl x509 -in tls.crt -noout -subject -issuer

Database Connection Failures

Symptoms: Application cannot connect to PostgreSQL.

Causes:

  • PostgreSQL cluster not ready
  • Connection pool exhausted
  • Network connectivity issues
  • Authentication failures

Diagnosis:

# Check PostgreSQL cluster status
kubectl get clusters

# Check PostgreSQL pods
kubectl get pods -l app.kubernetes.io/name=postgresql

# Check PostgreSQL logs
kubectl logs -l app.kubernetes.io/name=postgresql

# Test connectivity
kubectl exec -it <app-pod> -- psql -h <postgres-service> -U <user> -d <database>

Resolution:

# Wait for PostgreSQL to be ready
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=postgresql --timeout=300s

# Check connection string in application config
kubectl get secret <secret-name> -o jsonpath='{.data}' | base64 -d

# Restart application pods
kubectl rollout restart deployment/<deployment-name>

Kafka Connection Issues

Symptoms: Application cannot connect to Kafka.

Causes:

  • Kafka controllers not ready
  • Topic not created
  • Network connectivity issues

Diagnosis:

# Check Kafka pods
kubectl get pods -l app.kubernetes.io/name=kafka

# Check Kafka logs
kubectl logs -l app.kubernetes.io/name=kafka

# List topics
kubectl exec -it <kafka-pod> -- kafka-topics.sh --bootstrap-server localhost:9092 --list

Resolution:

# Wait for Kafka controllers to be ready
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=kafka --timeout=300s

# Create missing topic
kubectl exec -it <kafka-pod> -- kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic <topic-name> --partitions 3 --replication-factor 3

# Restart application to reconnect
kubectl rollout restart deployment/<deployment-name>

Redis Connection Issues

Symptoms: Application cannot connect to Redis.

Diagnosis:

# Check Redis pods
kubectl get pods -l app.kubernetes.io/name=redis

# Check Redis logs
kubectl logs -l app.kubernetes.io/name=redis

# Test connectivity
kubectl exec -it <redis-pod> -- redis-cli ping

Resolution:

# Wait for Redis to be ready
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=redis --timeout=300s

# Restart application
kubectl rollout restart deployment/<deployment-name>

High Memory Usage

Symptoms: Pods approaching or hitting memory limits.

Diagnosis:

# Check memory usage
kubectl top pods

# Check OOMKilled pods
kubectl get pods --field-selector=status.phase=Failed

# Check for memory leaks in logs
kubectl logs <pod-name> | grep -i "memory\|oom"

Resolution:

# Temporarily increase memory limit
kubectl edit deployment <deployment-name>

# Or scale horizontally if HPA is enabled
kubectl scale deployment <deployment-name> --replicas=<n>

# Long-term: set the resources in /etc/acd/deploy/values.override.yaml and run deploy

High CPU Usage

Symptoms: Pods consistently using high CPU.

Diagnosis:

# Check CPU usage
kubectl top pods

# Check for runaway processes
kubectl top pods --sort-by=cpu

Resolution:

# Scale horizontally if HPA is enabled
kubectl scale deployment <deployment-name> --replicas=<n>

# Or increase CPU limits
kubectl edit deployment <deployment-name>

Persistent Volume Issues

Symptoms: PVC not binding or volume errors.

Diagnosis:

# Check PVC status
kubectl get pvc

# Check PV status
kubectl get pv

# Check Longhorn volumes
kubectl get volumes -n longhorn-system

# Check Longhorn UI for details
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

Resolution:

# For stuck PVC, delete and recreate
kubectl delete pvc <pvc-name>
kubectl delete pod <pod-name>

# For Longhorn issues, check Longhorn UI
# Access via http://localhost:8080

# Recreate Longhorn volume if necessary

Zitadel Authentication Failures

Symptoms: Users cannot authenticate via Zitadel.

Causes:

  • CORS configuration mismatch
  • External domain misconfigured
  • Zitadel pods not healthy

Diagnosis:

# Check Zitadel pods
kubectl get pods -l app.kubernetes.io/name=zitadel

# Check Zitadel logs
kubectl logs -l app.kubernetes.io/name=zitadel

# Verify external domain configuration
grep -A 5 ExternalDomain /etc/acd/deploy/values.yaml

Resolution: Browse to the main manager hostname, the first entry of manager_hostnames in /etc/acd/deploy/group_vars/all.yaml. deploy sets Zitadel’s external domain from it. Sign-ins under any other name, including a machine’s IP address, fail.

# Restart Zitadel
kubectl rollout restart deployment -l app.kubernetes.io/name=zitadel

Certificate Errors

Symptoms: TLS/SSL errors in browser or API calls.

Diagnosis:

# Check certificate expiration
kubectl get secret <tls-secret> -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -dates

# Check certificate subject
kubectl get secret <tls-secret> -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -subject -issuer

Resolution: Replace the certificate and key in /etc/acd/deploy/certs/ and run deploy. See TLS Certificates.

/mnt/esb3027/deploy

Log Collection

The AgileTV CDN Manager ships with acd-support-dump, a single command that captures a point-in-time diagnostic snapshot of the whole cluster into one compressed, redacted tarball suitable for attaching to a support ticket. It is preinstalled on every control-plane node and works in air-gapped environments with no additional setup or downloads.

Running a support dump

Run the command from a control-plane (server) node:

acd-support-dump

The tool writes a tarball named support-dump-<hostname>-<timestamp>.tar.gz to the current directory and prints the path when it finishes. Collection usually takes a minute or two.

To choose the output location, pass a path as the first argument:

acd-support-dump /tmp/support-dump.tar.gz

See the acd-support-dump reference for its options and exit status.

What the dump contains

A support dump captures the management-plane state needed to diagnose the majority of field issues without direct access to the cluster:

  • Pod logs from the Manager workloads, covering up to the last 72 hours.
  • Cluster resources and events, including OOMKilled, restart reasons, and eviction events.
  • Node system logs (journald) — kernel messages such as the out-of-memory killer, plus k3s, containerd, and systemd unit output.
  • Database health for PostgreSQL and Redis — connectivity, version, sizes, and replication/keyspace statistics. This is health information only; application data is never collected.
  • Longhorn volume and replica health.
  • Metric history for the last 72 hours (node CPU, memory, and disk; per-container memory; container restarts; and volume usage) from the monitoring stack.
  • Automated analyzers that flag common failure modes — out-of-memory kills, disk exhaustion, and unhealthy or crash-looping pods — as a quick triage summary.

Kubernetes Secret values are excluded, and the tool’s built-in redactors run over the collected data before the tarball is written.

Note: A support dump captures Manager (management-plane) state only; it does not collect data-plane state. It also captures database health, not database contents — for data, use the backup procedures in the Operations Guide.

Sharing a support dump

Attach the generated .tar.gz to your support ticket. Because it contains log output and cluster configuration, treat it as potentially sensitive and share it only through approved support channels.

Manual Log Collection

If acd-support-dump is unavailable — for example, on an older installation — collect the essential logs by hand:

# Capture timestamp once to ensure consistency
TS=$(date +%Y%m%d-%H%M%S)

# Create log collection directory
mkdir -p ~/cdn-logs-$TS
cd ~/cdn-logs-$TS

# Collect pod logs
for pod in $(kubectl get pods -o name); do
  kubectl logs $pod > ${pod#pod/}.log 2>&1
  kubectl logs $pod -p > ${pod#pod/}.previous.log 2>&1 || true
done

# Collect cluster events
kubectl get events --sort-by='.lastTimestamp' > events.log

# Collect pod descriptions
for pod in $(kubectl get pods -o name); do
  kubectl describe $pod > ${pod#pod/}.describe.txt
done

# Compress for transfer
tar czf cdn-logs-$TS.tar.gz *.log *.txt

Emergency Procedures

Complete Cluster Recovery

If the cluster is completely down:

  1. Assess node status:

    kubectl get nodes

  2. Restart k3s on every machine. Start it on all the servers at once, without waiting for one to finish before the next, then on the agents. The unit is k3s on servers and k3s-agent on agents:

    # On every server, at the same time
systemctl restart --no-block k3s

# Then on every agent
systemctl restart --no-block k3s-agent

    The cluster datastore needs a majority of the servers running, so a server’s k3s does not finish starting until enough other servers have started too. With three servers, the Kubernetes API returns once two are up. Restart servers one at a time only during maintenance, while the rest of the cluster is running: see Safety.

  3. Restore from backup if necessary: see Backups.

The first server holds /etc/acd/deploy and the backups. Keep a copy of the latest backup folder off it, so that you can recover the configuration if the first server is lost. The backup folder holds every Kubernetes Secret: protect the copy as Backups describes.

Data Recovery

For data recovery scenarios:

  • PostgreSQL: Use Cloudnative PG backup/restore
  • Longhorn: Restore from volume snapshots
  • Kafka: Topics other than selection_input survive the loss of one broker. selection_input has a single replica: see Data Replication

Getting Help

If issues persist:

  1. Generate a support dump with acd-support-dump (see Log Collection), or collect logs manually if the tool is unavailable
  2. Check release notes for known issues
  3. Contact support with the support dump and a description of the issue

Next Steps

After resolving issues:

  1. Operations Guide - Preventive maintenance procedures
  2. Configuration Guide - Verify configuration is correct
  3. Architecture Guide - Understand component dependencies