Performance Tuning Guide

Overview

This guide provides performance tuning recommendations for the AgileTV CDN Manager (ESB3027). While the default configuration is suitable for most deployments, certain environments may benefit from additional optimizations.

Network Topology Optimization

Topology Aware Hints

The CDN Manager uses Kubernetes Topology Aware Hints to prefer routing pods in the same zone as the source of network traffic. This reduces cross-zone latency and improves overall system responsiveness.

How It Works

When nodes are labeled with topology zones, Kubernetes automatically routes traffic to pods in the same zone when possible. This is particularly beneficial for:

• Low-latency requirements: Keeps traffic local to reduce round-trip time
• Cost optimization: Reduces cross-zone data transfer costs in cloud environments
• Load distribution: Prevents hotspots by distributing load across zones

Configuring Availability Zones

Each node must have zone and region labels applied for Topology Aware Hints to function. Set them with node_labels on each machine’s line in /etc/acd/deploy/inventory.ini, then run deploy, which applies them:

[server]
manager-1 ansible_host=192.0.2.11 node_labels='{"topology.kubernetes.io/zone": "us-east-1a", "topology.kubernetes.io/region": "us-east-1"}'

Adjust the zone and region values to match your deployment geography. Labels set this way are reapplied on every deploy, so they are kept with the rest of your configuration. See Inventory.

Verify Topology Configuration

Verify labels are applied:

kubectl get nodes --show-labels | grep topology.kubernetes.io

Verify EndpointSlices are being generated with hints:

kubectl get endpointslices

Requirements for Topology Aware Hints

For Topology Aware Hints to activate:

• Minimum Nodes: At least one node must be labeled with each zone referenced by endpoints
• Symmetry: The control plane checks for sufficient CPU capacity across zones to balance traffic
• Zone Coverage: All zones with endpoints should have at least one ready node

Integration with Pod Anti-Affinity

Topology labels complement the pod anti-affinity rules already configured in the Helm chart:

• Pod Anti-Affinity: Handles pod-to-node placement to ensure high availability
• Topology Aware Hints: Handles service-to-pod traffic routing to keep requests within the same zone

Together, these features optimize both placement and routing for improved performance.

Fallback Behavior

If zone labels are not configured, the system falls back to random load-balancing across all available pods. This is functionally correct but may result in:

• Increased cross-zone traffic
• Higher latency for some requests
• Less predictable performance characteristics

Kernel Network Tuning (sysctl)

deploy tunes the kernel’s network settings on every machine for you, for high connection rates and large numbers of concurrent connections. There is nothing to configure: it applies these values on every run, whatever the firewall settings:

Connection tracking (net.netfilter.nf_conntrack_*) is not in the list: k3s sets those values itself every time it starts, so values set elsewhere do not last.

Important: Do not set these parameters yourself, in /etc/sysctl.d/, a tuned profile or anywhere else. After changing a machine’s network, deploy checks that each one holds the value above, and stops with [ACD-VERIFY-SYSCTL] before installing anything if another value is in place. If an earlier installation left a file such as /etc/sysctl.d/99-cdn-manager.conf, remove these parameters from it.

To check the values on a machine:

sysctl net.core.somaxconn net.core.netdev_max_backlog net.ipv4.tcp_max_syn_backlog \
    net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse net.core.rmem_max net.core.wmem_max

Monitoring Impact

After applying these settings, monitor:

• Connection establishment rates
• TIME_WAIT socket count: netstat -n | grep TIME_WAIT | wc -l
• Connection tracking table usage: cat /proc/sys/net/netfilter/nf_conntrack_count
• Network buffer utilization via Grafana dashboards

Resource Configuration

Horizontal Pod Autoscaler (HPA)

The default HPA configuration is tuned for production workloads. For environments with variable load, consider adjusting the scale metrics:

For detailed HPA configuration, see the Architecture Guide.

Resource Requests and Limits

Ensure resource requests and limits are appropriately sized for your workload. Under-provisioned resources can cause:

• Pod evictions during high load
• Increased latency due to CPU throttling
• Slow scaling responses

Refer to the Configuration Guide for preset configurations and planning guidance.

Database Optimization

PostgreSQL

The PostgreSQL cluster is managed by the Cloudnative PG operator. For improved performance:

• Connection Pooling: The application uses connection pooling by default
• Replica Usage: Read queries can be offloaded to replicas for read-heavy workloads
• Backup Scheduling: Schedule backups during low-traffic periods to minimize I/O impact

Redis

Redis provides in-memory caching for sessions and ephemeral state:

• Memory Allocation: Ensure sufficient memory for cache hit rates
• Persistence: RDB snapshots are enabled; adjust frequency based on durability needs

Kafka

Kafka handles event streaming for selection input and metrics:

• Partition Count: Default partitions are sized for typical workloads
• Replication Factor: Production deployments use 3 replicas for fault tolerance
• Consumer Groups: The Selection Input Worker is limited to one consumer per partition

Monitoring Performance

Key Metrics to Watch

Monitor the following metrics for performance insights:

• API Response Time: Track via Grafana dashboards
• Pod CPU/Memory Usage: Identify resource bottlenecks
• Kafka Lag: Monitor consumer lag for selection input processing
• Database Connections: Watch for connection pool exhaustion

Grafana Dashboards

Pre-built dashboards are available at https://<manager-host>/grafana:

• System Health: Overall cluster and application health
• CDN Metrics: Routing and usage statistics
• Resource Utilization: CPU, memory, and network usage per component

Troubleshooting Performance Issues

High Latency

1. Check pod distribution across nodes: kubectl get pods -o wide
2. Verify topology labels are applied: kubectl get nodes --show-labels
3. Review network latency between nodes
4. Check for resource contention: kubectl top pods

Slow Scaling

1. Verify HPA is enabled: kubectl get hpa
2. Check cluster capacity for scheduling new pods
3. Review HPA metrics: kubectl describe hpa cdn-platform-cdn-manager

Database Performance

1. Check PostgreSQL cluster status: kubectl get pods -l app=postgresql
2. Review slow query logs (if enabled)
3. Monitor connection pool usage

Next Steps

After reviewing performance tuning:

1. Architecture Guide - Understand component interactions
2. Configuration Guide - Detailed configuration options
3. Metrics & Monitoring Guide - Comprehensive monitoring setup
4. Troubleshooting Guide - Resolve performance issues