Configuring Segregated Networks

Overview

In a segregated layout, each machine has two networks:

• A public network, holding the default route, which clients and CDN Directors use to reach the manager.
• A backplane network, separate from the public network, which carries all traffic between the machines (east-west traffic): the Kubernetes API, the cluster datastore, the pod network and storage replication.

This keeps cluster traffic off the public network, and lets the firewall trust the backplane completely while the public interface opens only the ports clients need. A backplane is optional, but it is the recommended layout for production.

The backplane does not have to be reserved for the cluster machines. Other hosts may be on it too, such as your CDN Directors: it is simply the network the cluster machines use to reach each other.

flowchart LR
    clients[Users and CDN Directors] -->|public network| eth0
    subgraph machine[Each cluster machine]
        eth0[eth0: public zone]
        eth1[eth1: acd-cluster zone]
    end
    eth1 <-->|backplane network| peers[Other cluster machines]

Requirements

• The backplane is a separate network from the public network.
• Every machine has exactly one interface with an address in the backplane network, and every machine is reachable from the others on it.
• The default route is on the public interface, not the backplane.
• Only hosts you trust are on the backplane. With manage_firewall: true, every host on the backplane can reach every port on every cluster machine, including the cluster datastore (etcd, 2379-2380/tcp), the Kubernetes API (6443/tcp) and the kubelet (10250/tcp), so connect only hosts you trust to it.
• Both interfaces are managed by NetworkManager, with an active connection.

Configuration

Give the backplane’s address range in /etc/acd/deploy/group_vars/all.yaml, before the first deploy:

backplane_network: "10.10.0.0/24"

Nothing else is needed: no per-machine settings and no firewall commands. deploy finds each machine’s interface in that range and sets up the rest.

Important: backplane_network is locked at the first installation. A cluster cannot be moved on to or off a backplane later without reinstalling it.

What deploy Sets Up

On each machine, with manage_firewall: true:

• Kubernetes uses the backplane: each machine’s cluster address is its backplane address, the pod network (Flannel) runs over the backplane interface, and the public address is advertised as the machine’s external address.
• The backplane interface is placed in the acd-cluster zone, which accepts all traffic, together with the pod and service networks.
• The public interface is placed in firewalld’s public zone, with the ports clients need opened through the acd-public policy.
• Routing rules keep traffic to the pod, service and backplane networks on the main routing table.

Other interfaces are left as they are.

Verification

After deploy, on any machine:

firewall-cmd --get-active-zones

Expected output:

acd-cluster
  interfaces: eth1
  sources: 10.42.0.0/16 10.43.0.0/16
public
  interfaces: eth0

Check the cluster addresses Kubernetes uses:

grep -E 'node-ip|external|flannel' /etc/rancher/k3s/config.yaml.d/20-acd-node.yaml

Expected output, on manager-1:

node-ip: "10.10.0.11"
node-external-ip: "192.0.2.11"
flannel-iface: "eth1"

And on a server:

kubectl get nodes -o wide

INTERNAL-IP shows each machine’s backplane address, and EXTERNAL-IP its public address.

Problems

For every code, see Installation Troubleshooting.