Networking Guide

Overview

deploy configures each machine’s network for the cluster: its firewall, its routing rules for cluster traffic, and the kernel network settings. The site setting backplane_network chooses the layout. A second setting, manage_firewall, chooses whether deploy manages the firewall or you do.

A backplane is optional, but we recommend one wherever the machines can have a second network. The backplane is the network the cluster machines use to reach each other (east-west traffic). Every machine must be reachable on it, and it should be a separate network from the public one. Setting it to the public network’s range works, but then every host on the public network is trusted for all traffic. It does not have to be reserved for the cluster: other hosts may also be on it, but with the firewall managed they can reach every port on the machines, including the cluster datastore (etcd, 2379-2380/tcp), the Kubernetes API (6443/tcp) and the kubelet (10250/tcp).

Interfaces and Addresses

Every machine needs an IPv4 default route on its public interface, the interface that clients reach it on. The route does not need to lead to the internet: on an isolated network, use your site’s router or any gateway address on that subnet. deploy uses this route to find each machine’s public interface and address, and stops with [ACD-NO-DEFAULT-ROUTE] without one.

From that route, deploy works out each machine’s two addresses:

• The public interface is the one holding the default route. Its address is the machine’s public address, which clients reach.
• The cluster address is the address other machines reach it on: its address inside backplane_network when that is set, and otherwise its public address.

Interfaces that are neither the public interface nor the backplane are left as they are.

The cluster also uses two internal address ranges, which must not be in use anywhere on your network:

Firewall

With manage_firewall: true, the default, deploy starts and enables firewalld on every machine and keeps two things as its own:

• The acd-cluster zone, with target ACCEPT, which trusts all traffic from the cluster: the backplane interface (or, without a backplane, the other machines’ addresses), and the pod and service networks.
• The acd-public policy, from the public zone to the machine, which opens the ports clients need:

  Port	Purpose
  80/tcp	HTTP ingress
  443/tcp	HTTPS ingress: the GUI, the API, Zitadel and Grafana
  8086/tcp	Metrics pushed by the CDN Directors (Vector)

It also places each machine’s public interface in the public zone, on its NetworkManager connection. Before the first run, make sure the public zone allows everything else you need on that interface, SSH included. To check, run firewall-cmd --zone=public --list-all on each machine: ssh should be listed under services, or port 22 under ports.

deploy removes anything in acd-cluster and acd-public that it did not put there. Add your own rules to other zones, such as public: deploy never changes their rules.

Ports Between the Machines

Within the cluster, the machines trust each other for all traffic. When you manage the firewall yourself, the ports the cluster needs between machines are:

The pod and service networks must also be in a zone with target ACCEPT.

Kafka Ports

The CDN Directors reach Kafka on these ports, on every machine:

With the default three brokers, the broker ports are 30001-30003, so the full range is 30000-30003. With more brokers, the range grows by one port per broker: six brokers use 30000-30006. In the rest of this documentation, “30000-30003” means this range for the default three brokers. Widen it if your cluster runs more.

deploy neither opens nor checks these ports, and the host firewall does not filter them: as on any host running containers, traffic that Kubernetes forwards to its services passes before the firewall’s zone rules apply. These ports are reachable from every network that can reach the machines.

Important: Restrict 9092/tcp and the whole 30000 range (30000-30003 with three brokers) to your CDN Directors on your network’s own firewall. The same applies to any other port a Kubernetes service publishes on the machines.

Ingress NodePorts

The ingress service (Traefik) also publishes two NodePorts on every machine, which forward to its HTTP and HTTPS ports, 80 and 443. Kubernetes picks them at installation from 30000-32767, so they differ from one cluster to another, and the host firewall does not filter them, as for the Kafka ports. Find them on a server:

kubectl get service traefik -n kube-system

NAME      TYPE           CLUSTER-IP    EXTERNAL-IP                          PORT(S)                      AGE
traefik   LoadBalancer   10.43.12.34   192.0.2.11,192.0.2.12,192.0.2.13     80:31873/TCP,443:30306/TCP   3d

Here they are 31873 (HTTP) and 30306 (HTTPS). Clients need only 80 and 443: block the two NodePorts on your network’s own firewall.

Managing the Firewall Yourself

Set manage_firewall: false to configure firewalld yourself. deploy then only checks that the ports between the machines and the ports clients need are open in the zone their traffic arrives in, and that the pod and service networks are in a zone with target ACCEPT. It stops before changing anything, and shows the firewall-cmd commands to run, if they are not. Only firewalld is checked: if it is not running, the check is skipped, and your own firewall must allow the same traffic.

To hand the firewall of a running cluster over to yourself, on each machine:

1. Move the pod and service networks to a zone with target ACCEPT, such as firewalld’s own trusted. For the default ranges:

   for network in 10.42.0.0/16 10.43.0.0/16; do
       firewall-cmd --permanent --zone=acd-cluster --remove-source="$network"
       firewall-cmd --permanent --zone=trusted --add-source="$network"
       firewall-cmd --zone=trusted --change-source="$network"
   done
   

2. Open the ports in the zone their traffic arrives in, with firewall-cmd --zone=<zone> --add-port=<port>, once with --permanent and once without.

3. On a machine with a backplane, put the backplane interface in the zone you want: nmcli connection modify <its connection> connection.zone <zone>.

4. Set manage_firewall: false in group_vars/all.yaml and run deploy.

deploy then empties the acd-cluster zone and the acd-public policy, and moves any interface still in acd-cluster to firewalld’s default zone, with a warning.

Routing

With manage_routes: true, the default, deploy adds lookup main routing rules at priority 50, so that cluster traffic always uses the main routing table, even on machines that use policy routing. On a machine with a backplane, ip rule then shows:

0:	from all lookup local
50:	from 10.42.0.0/16 lookup main proto static
50:	from all to 10.42.0.0/16 lookup main proto static
50:	from 10.43.0.0/16 lookup main proto static
50:	from all to 10.43.0.0/16 lookup main proto static
50:	from all to 10.10.0.0/24 lookup main proto static
32766:	from all lookup main
32767:	from all lookup default

Keep your own rules at other priorities. See manage_routes for how these rules interact with your own.

Managing Routing Yourself

If your own policy routing rules must sit at priority 50 or below, set manage_routes: false and add the cluster’s rules yourself. Add them before you change the setting on a running cluster: deploy then removes every priority-50 rule from the connections it manages.

Every machine needs five lookup main rules:

Give them a priority other than 50, and a lower number than any of your own rules that send traffic to another table, such as your from <address> lookup <table> source rules. Store them on the NetworkManager connection of the interface the machines reach each other on: the backplane interface, or the public interface without a backplane. For example, at priority 40 on a backplane connection named eth1 (table 254 is main):

nmcli connection modify eth1 +ipv4.routing-rules "priority 40 from 10.42.0.0/16 table 254"
nmcli connection modify eth1 +ipv4.routing-rules "priority 40 to 10.42.0.0/16 table 254"
nmcli connection modify eth1 +ipv4.routing-rules "priority 40 from 10.43.0.0/16 table 254"
nmcli connection modify eth1 +ipv4.routing-rules "priority 40 to 10.43.0.0/16 table 254"
nmcli connection modify eth1 +ipv4.routing-rules "priority 40 to 10.10.0.0/24 table 254"

Stored rules take effect when the connection is next activated. To activate it now, run nmcli connection up eth1 in a maintenance window: it re-applies the whole connection, and briefly interrupts traffic on that interface. Then check the rules with ip rule show.

Whichever way the rules are managed, deploy checks before changing anything that cluster traffic resolves through the main routing table, and stops with [ACD-ROUTING-CONFLICT], naming the rule in the way, if it does not.

Changes Are Checked

After changing a machine’s network, deploy compares the live state with what it planned: the interface zones, the firewall, the routing rules and the kernel settings. It stops before installing anything on the cluster if they differ. This check cannot be skipped. See the [ACD-VERIFY-...] codes in Installation Troubleshooting.

Boot-Time Network Tools

deploy stores its interface zones and routing rules on the machines’ NetworkManager connections. Tools that rewrite those connections at boot, such as cloud-init or a virtualization platform’s contextualization, can undo them after a reboot: an interface back in the wrong zone, or the priority-50 rules gone. Nothing checks for this between deploy runs.

Disable such regeneration of the network connections on the cluster machines. Where you cannot, run deploy again after a reboot: it puts the settings back, prints Host network changed on <machine>: ... naming what it changed, and carries on.