The Director serves as a versatile network service designed to redirect incoming HTTP(s) requests to the optimal host or Content Delivery Network (CDN) by evaluating various request properties through a set of rules. Although requests can be generic, the primary focus centers around audio-video content delivery. The rule engine allows users to construct routing configurations using predefined blocks, providing for the creation of intricate routing logic. This modular approach allows the users to tailor and streamline the content delivery process to meet their specific needs. The Director’s flexible rule engine takes into account factors such as geographical location, server load, content type, and other metadata from external sources to intelligently route incoming requests. It supports dynamic adjustments to seamlessly adapt to changing network conditions, ensuring efficient and reliable content delivery. The Director improves the overall user experience by delivering content from the most suitable and responsive sources, thereby reducing latency and enhancing performance.

Requirements

Hardware

The Director is designed to be installed and operated on commodity hardware, ensuring accessibility for a broad range of users. The minimum hardware specifications are as follows:

CPU: x86-64 AMD or Intel with at least 2 cores.
Memory: At least 2 GB free at runtime.

Operating System Compatibility

The Director is officially supported on Red Hat Enterprise Linux 8 (RHEL8) or any compatible operating system. While it is possible to install and run on Red Hat Enterprise Linux 7 (RHEL7) or compatible OS, this is provided as a convenience, and users should note the limited management capabilities on this platform.

External Internet access is necessary during the installation process for the installer to download and install additional dependencies. This ensures a seamless setup and optimal functionality of the Director on Red Hat Enterprise Linux 8. It’s worth noting that, due to the unique workings of the DNF package manager in Red Hat Enterprise Linux 8 with rolling package streams, an air-gapped installation process is not available.

Firewall Recommendations

See Firewall.

Installation

See Installation.

Operations

See Operations.

Configuration Process

Once the router is operational, it requires a valid configuration before it can route incoming requests.

There are currently three methods available for configuring the router, each catering to different levels of complexity. The first is a Web UI, suitable for the most common use-cases, providing an intuitive interface for configuration. The second involves utilizing a confd REST service, complemented by an optional command line tool, confcli, suitable for all but the most advanced scenarios. The third method involves leveraging an internal REST API, ideal for the most intricate cases where using confd proves to be less flexible. It’s essential to note that as the configuration method advances through these levels, both flexibility and complexity increase, providing users with tailored options based on their specific needs and expertise.

API Key Management

Regardless of the method used to configure the system, a unique API key is crucial for safeguarding the router’s configuration and preventing unauthorized access to the API. This key must be supplied when interacting with the API. During the router software installation, an automatically generated API key is created and can be located on the installed system at /opt/edgeware/acd/router/cache/rest-api-key.json. The structure of this file is as follows:

{"api_key": "abc123"}

When accessing the internal configuration API, the key must be included in the X-API-key header of the request, as shown below:

curl -v -k -H "X-API-Key: abc123" https://<router-host.example>:5001/v2/configuration

Modification to the authentication key and behavior can be done through the /v2/rest_api_key endpoint. To change the key, a PUT request with a JSON body of the same structure can be sent to the endpoint:

curl -v -k -X PUT -T new-key.json -H "X-API-Key: abc123" \
-H "Content-Type: application/json" https://<router-host.example>:5001/v2/rest_api_key

Additionally, key authentication can be disabled completely by sending a DELETE request to the endpoint:

curl -v -k -X DELETE -H "X-API-Key: abc123" \
https://<router-host.example>:5001/v2/rest_api_key

In the event of a lost or forgotten authentication key, it can always be retrieved at /opt/edgeware/acd/router/cache/rest-api-key.json on the machine running the router. It is critical to emphasize that the API key should remain private to prevent unauthorized access to the internal API, as it grants full access to the router’s configuration.

Configuration Basics

Upon completing the installation process and configuring the API keys, the subsequent section will provide guidance on configuring the router to route all incoming requests to a single host. For straightforward CDN Offload use cases, there is a web based user interface described here.

For further details on configuring the router using confd and confcli, please consult the Confd documentation.

The initial step involves defining the target host group. In this illustration, a singular group named all will be established, comprising two hosts.

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: 2
  Adding a 'host' element
    hostGroup : {
      name (default: ): all
      type (default: host):
      httpPort (default: 80):
      httpsPort (default: 443):
      hosts : [
        host : {
          name (default: ): host1.example.com
          hostname (default: ): host1.example.com
          ipv6_address (default: ):
        }
        Add another 'host' element to array 'hosts'? [y/N]: y
        host : {
          name (default: ): host2.example.com
          hostname (default: ): host2.example.com
          ipv6_address (default: ):
        }
        Add another 'host' element to array 'hosts'? [y/N]: n
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: n
]
Generated config:
{
  "hostGroups": [
    {
      "name": "all",
      "type": "host",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
        {
          "name": "host1.example.com",
          "hostname": "host1.example.com",
          "ipv6_address": ""
        },
        {
          "name": "host2.example.com",
          "hostname": "host2.example.com",
          "ipv6_address": ""
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]:

After defining the host group, the next step is to establish a rule that directs incoming requests to the designated host. In this example, a sole rule named random will be generated, ensuring that all incoming requests are consistently routed to the previously defined host.

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: 6
  Adding a 'random' element
    rule : {
      name (default: ): random
      type (default: random):
      targets : [
        target (default: ): host1.example.com
        Add another 'target' element to array 'targets'? [y/N]: y
        target (default: ): host2.example.com
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "random",
      "type": "random",
      "targets": [
        "host1.example.com",
        "host2.example.com"
      ]
    }
  ]
}
Merge and apply the config? [y/n]:

The last essential step involves instructing the router on which rule should serve as the entry point into the routing tree. In this example, we designate the rule random as the entrypoint for the routing process.

$ confcli services.routing.entrypoint random
services.routing.entrypoint = 'random'

Once this configuration is defined, all incoming requests will initiate their traversal through the routing rules, starting with the rule named random. This rule is designed to consistently match for every incoming request, effectively load balancing evenly between host1.example.com and host2.example.com on port 80 or 443, depending on whether the initial request was made using HTTP or HTTPS.

Integration with Convoy

The router is equipped with the capability to synchronize specific configuration metadata with a separate Convoy installation through the integrated convoy-bridge service. However, this service necessitates additional setup and configuration, and you can find comprehensive details on the process here..

Additional Resources

Additional documentation resources are included with the Director and can be accessed at the following directory: /opt/edgeware/acd/documentation/. This directory contains supplementary materials to provide users with comprehensive information and guidance for optimizing their experience with the Director.

Ready for Production

Once the Director software is completely installed and configured, there are a few additional considerations before moving to a full production environment. See the section Ready for Production for additional information.

1.1.2 - Installation

How to install and upgrade ESB3024 Router

1.1.2.1 - Installing release 1.10.x

How to install and upgrade to ESB3024 Router release 1.10.x

To install ESB3024 Router, one first needs to copy the installation ISO image to the target node where the router will be run. Due to the way the installer operates, it is necessary that the host is reachable by password-less SSH from itself for the user account that will perform the installation, and that this user has sudo access.

Prerequisites:

Ensure that the current user has sudo access.
```
sudo -l
```
If the above command fails, you may need to add the user to the /etc/sudoers file.
Ensure that the installer has password-less SSH access to localhost.
If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.
The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:
```
mkdir -m 0700 -p ~/.ssh
ssh-keyscan localhost >> ~/.ssh/authorized_keys
```
Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice its recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.
Disable SELinux.
The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.
To check if SELinux is enabled:
```
getenforce
```
This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.
```
setenforce 0
```
This disables SELinux, but does not make the change persistent across reboots. To do that, edit the /etc/selinux/config file and set the SELINUX property to disabled.
It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

Mount the installation ISO image under /mnt/acd.
Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.
```
mkdir -p /mnt/acd
mount esb3024-acd-router-esb3024-1.10.1.iso /mnt/acd
```
Run the installer script.
```
/mnt/acd/installer
```

Upgrade from ESB3024 Router release 1.8.0

The following steps can be used to upgrade the router from release 1.8.0 to 1.10.x. If upgrading from an earlier release it is recommended to perform the upgrade in multiple steps; for instance when upgrading from release 1.6.0 to 1.10.x, it is recommended to first upgrade to 1.8.0 and then to 1.10.x.

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new release of the router, and applying the saved configuration.

With the router running, save a backup of the configuration.
The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.
Extracting the configuration using confd is the recommend approach where available.
```
confcli | tee config_backup.json
```
To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.
```
curl --insecure https://localhost:5001/v2/configuration \
  | tee config_backup.json
```
If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.
```
curl --insecure -H "X-API-Key: 1234abcd" \
  https://localhost:5001/v2/configuration \
  | tee config_backup.json
```
Mount the new installation ISO under /mnt/acd.
Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.
```
mkdir -p /mnt/acd
mount esb3024-acd-router-esb3024-1.10.1.iso /mnt/acd
```
Stop the router and all associated services.
Before upgrading the router it needs to be stopped, which can be done by typing this:
```
systemctl stop 'acd-*'
```
Run the installer script.
```
/mnt/acd/installer
```
Migrate the configuration.
Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.
The confd configuration used in version 1.8.0 is not directly compatible with 1.10.x, and may need to be converted. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes.
To determine if the configuration needs to be converted, confcli can be run like below. If it prints error messages, the configuration needs to be converted. If no error messages are printed, the configuration is valid and no further updates are necessary.
```
confcli | head -n5
[2024-04-02 14:48:37,155] [ERROR] Missing configuration key /integration
[2024-04-02 14:48:37,162] [ERROR] Missing configuration key /services/routing/settings/qoeTracking
[2024-04-02 14:48:37,222] [ERROR] Missing configuration key /services/routing/hostGroups/convoy-rr/hosts/convoy-rr-1/healthChecks
[2024-04-02 14:48:37,222] [ERROR] Missing configuration key /services/routing/hostGroups/convoy-rr/hosts/convoy-rr-2/healthChecks
[2024-04-02 14:48:37,242] [ERROR] Missing configuration key /services/routing/hostGroups/e-dns/hosts/linton-dns-1/healthChecks
{
   "integration": {
      "convoy": {
            "bridge": {
               "accounts": {
```
If error messages are printed, the configuration needs to be converted. If the configuration was saved in the file config_backup.json, the conversion can be done by typing this at the command line:
```
sed -E -e '/"hosts":/,/]/ s/([[:space:]]+)("hostname":.*)/\1\2\n\1"healthChecks": [],/' -e '/"apiKey":/ d' config_backup.json | \
curl -s -X PUT -T - -H 'Content-Type: application/json' http://localhost:5000/config/__active/
```
```
systemctl restart acd-router
```
This adds empty healthChecks sections to all hosts and removes the apiKey configuration. After that, acd-confd is restarted. See Configuration changes between 1.8.0 and 1.10.x for more details about the configuration changes.
Remove the Account Monitor container
Older versions of the router installed the Account Monitor tool. This was removed in release 1.8.0, but if it is still present and unused, it can be removed by typing:
```
podman rm account-monitor
```

Troubleshooting

If there is a problem running the installer, additional debug information can be output by adding -v or -vv or -vvv to the installer command, the more “v” characters, the more detailed output.

1.1.2.1.1 - Configuration changes between 1.8.0 and 1.10.x

This describes the configuration changes between ESB3024 Router version 1.8.0 and 1.10.x.

Confd configuration changes

Below are the major changes to the confd configuration between version 1.8.0 and 1.10.x listed.

Added `integration.convoy` section

An integration.convoy section has been added to the configuration. It is currently used for configuring the Convoy Bridge service.

Removed `services.routing.apiKey` configuration

The services.routing.apiKey configuration key has been removed. This was an obsolete way of giving the configuration access to the router. The key has to be removed from the configuration when upgrading, otherwise the configuration will not be accepted.

Added `services.routing.settings.qoeTracking`

A services.routing.settings.qoeTracking section has been added to the configuration.

Added `healthChecks` sections to the hosts

The hosts in the hostGroup entries have been extended with a healthChecks key, which is a list of functions that determine if a host is in good health.

For example, a redirecting host might look like this after the configuration has been updated:

{
    "services": {
        "routing": {
            "hostGroups": [
                {
                    "name": "convoy-rr",
                    "type": "redirecting",
                    "httpPort": 80,
                    "httpsPort": 443,
                    "forwardHostHeader": true,
                    "hosts": [
                        {
                            "name": "convoy-rr-1",
                            "hostname": "convoy-rr-1",
                            "ipv6_address": "",
                            "healthChecks": [
                                "health_check('convoy-rr-1')"
                            ]
                        }
                    ]
                }
            ],

1.1.2.2 - Installing release 1.8.0

How to install and upgrade to ESB3024 Router release 1.8.0

Prerequisites:

Ensure that the current user has sudo access.
```
sudo -l
```
If the above command fails, you may need to add the user to the /etc/sudoers file.
Ensure that the installer has password-less SSH access to localhost.
If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.
The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:
```
mkdir -m 0700 -p ~/.ssh
ssh-keyscan localhost >> ~/.ssh/authorized_keys
```
Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice its recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.
Disable SELinux.
The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.
To check if SELinux is enabled:
```
getenforce
```
This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.
```
setenforce 0
```
It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

Mount the installation ISO image under /mnt/acd.
Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.
```
mkdir -p /mnt/acd
mount esb3024-acd-router-esb3024-1.8.0.iso /mnt/acd
```
Run the installer script.
```
/mnt/acd/installer
```

Upgrade

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new version of the router, and applying the saved configuration.

With the router running, save a backup of the configuration.
The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.
Extracting the configuration using confd is the recommend approach where available.
```
confcli | tee config_backup.json
```
To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.
```
curl --insecure https://localhost:5001/v2/configuration \
  | tee config_backup.json
```
If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.
```
curl --insecure -H "X-API-Key: 1234abcd" \
  https://localhost:5001/v2/configuration \
  | tee config_backup.json
```
Mount the new installation ISO under /mnt/acd.
Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.
```
mkdir -p /mnt/acd
mount esb3024-acd-router-1.2.0.iso /mnt/acd
```
Stop the router and all associated services.
Before upgrading the router it needs to be stopped, which can be done by typing this:
```
systemctl stop 'acd-*'
```
Run the installer script.
```
/mnt/acd/installer
```
Migrate the configuration.
Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.
The confd configuration used in version 1.6.0 is not directly compatible with 1.8.0, and may need to have a few minor manual updates in order to be valid. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes.
To determine if the configuration needs to be manually updated, confcli can be run like below. If it prints error messages, the configuration needs to be updated. If no error messages are printed, the configuration is valid and no further updates are necessary.
```
confcli services.routing | head
[2024-02-01 19:05:10,769] [ERROR] Missing configuration key /services/routing/hostGroups/convoy-rr/forwardHostHeader
[2024-02-01 19:05:10,779] [ERROR] Missing configuration key /services/routing/hostGroups/e-dns/forwardHostHeader
[2024-02-01 19:05:10,861] [ERROR] 'forwardHostHeader'
```
If error messages are printed, a forwardHostHeader configuration needs to be added to the hostGroups configuration. This can be done by running this at the command line:
```
curl -s http://localhost:5000/config/__active/ | \
sed -E 's/([[:space:]]+)"type": "(host|redirecting|dns)"(,?)/\1"type": "\2",\n\1"forwardHostHeader": false\3/' | \
curl -s -X PUT -T - -H 'Content-Type: application/json' http://localhost:5000/config/__active/
```
This reads the active configuration from the router, adds the “forwardHostHeader” configuration to all host groups, and then sends the updated configuration back to the router.
See Configuration changes between 1.6.0 and 1.8.0 for more details about the configuration changes.
Remove the Account Monitor container
Previous versions of the router installed the Account Monitor tool. This is no longer included, but since the previous version installed, there will be a stopped Account Monitor container. If it is not used, the container can be removed by typing:
```
podman rm account-monitor
```

Troubleshooting

1.1.2.2.1 - Configuration changes between 1.6.0 and 1.8.0

This describes the configuration changes between ESB3024 Router version 1.6.0 and 1.8.0.

Confd configuration changes

Below are some of the configuration changes between version 1.4.0 and 1.6.0 listed. The list only contains the changes that might affect already existing configuration, enirely new items are not listed. Normally nothing needs to be done about this since they will be upgraded automatically, but they are listed here for reference.

Added `enabled` to `contentPopularity`

An enabled key has been added to services.routing.settings.contentPopularity. After the key has been added, the configuration looks like this:

{
    "services": {
        "routing": {
            "settings": {
                "contentPopularity": {
                    "enabled": true,
                    "algorithm": "score_based",
                    "sessionGroupNames": []
                },
                ...

Added `selectionInputItemLimit` to `tuning`

A selectionInputItemLimit key has been added to services.routing.tuning.general. After the key has been added, the configuration looks like this:

{
    "services": {
        "routing": {
            "tuning": {
                "general": {
                    ...
                    "selectionInputItemLimit": 10000,
                    ...

Added `forwardHostHeader` to `hostGroups`

All three hostGroup types (host, redirecting and dns) have been extended with a forwardHostHeader key. For example, a redirecting host might look like this after the change:

{
    "services": {
        "routing": {
            "hostGroups": [
                {
                    "name": "convoy-rr",
                    "type": "redirecting",
                    "httpPort": 80,
                    "httpsPort": 443,
                    "forwardHostHeader": true,
                    "hosts": [
                        {
                            "name": "convoy-rr-1",
                            "hostname": "convoy-rr-1",
                            "ipv6_address": ""
                        }
                    ]
                }
            ],
            ...

REST API configuration changes

The following items have been added to the REST API configuration. They will not need to be manually updated, the router will add the new keys with default values. Note that this is not a complete list of all changes, it only contains the changes that will be automatically added when upgrading the router.

If the router is configured via confd and confcli, these changes will be applied by them. This section is only relevant if the router is configured via the v2/configuration API.

Added the session_translation_function key.
Added the tuning.selection_input_item_limit key.

1.1.2.3 - Installing release 1.6.0

How to install and upgrade to ESB3024 Router release 1.6.0

Prerequisites:

Ensure that the current user has sudo access.
```
sudo -l
```
If the above command fails, you may need to add the user to the /etc/sudoers file.
Ensure that the installer has password-less SSH access to localhost.
If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.
The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:
```
mkdir -m 0700 -p ~/.ssh
ssh-keyscan localhost >> ~/.ssh/authorized_keys
```
Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice its recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.
Disable SELinux.
The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.
To check if SELinux is enabled:
```
getenforce
```
This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.
```
setenforce 0
```
It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

Mount the installation ISO image under /mnt/acd.
Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.
```
mkdir -p /mnt/acd
mount esb3024-acd-router-esb3024-1.8.0.iso /mnt/acd
```
Run the installer script.
```
/mnt/acd/installer
```

Upgrade

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new version of the router, and applying the saved configuration.

With the router running, save a backup of the configuration.
The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.
Extracting the configuration using confd is the recommend approach where available.
```
confcli | tee config_backup.json
```
To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.
```
curl --insecure https://localhost:5001/v2/configuration \
  | tee config_backup.json
```
If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.
```
curl --insecure -H "X-API-Key: 1234abcd" \
  https://localhost:5001/v2/configuration \
  | tee config_backup.json
```
Mount the new installation ISO under /mnt/acd.
Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.
```
mkdir -p /mnt/acd
mount esb3024-acd-router-1.2.0.iso /mnt/acd
```
Stop the router and all associated services.
Before upgrading the router it needs to be stopped, which can be done by typing this:
```
systemctl stop 'acd-*'
```
Run the installer script.
```
/mnt/acd/installer
```
Migrate the configuration.
Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.
See Configuration changes between 1.4.0 and 1.6.0 for instructions on how to migrate the configuration to release 1.6.0.

Troubleshooting

1.1.2.3.1 - Configuration changes between 1.4.0 and 1.6.0

This describes the configuration changes between ESB3024 Router version 1.4.0 and 1.6.0.

confd configuration

The confd configuration used in version 1.4.0 is not directly compatible with 1.6.0, and will need to have a few minor updates in order to be valid. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes. Running confcli will cause error messages and an empty default configuration to be printed.

$ confcli services.routing.
[2023-12-12 16:08:07,120] [ERROR] Missing configuration key /services/routing/translationFunctions
[2023-12-12 16:08:07,122] [ERROR] Missing configuration key /services/routing/settings/instream/dashManifestRewrite/sessionGroupNames
[2023-12-12 16:08:07,122] [ERROR] Missing configuration key /services/routing/settings/instream/hlsManifestRewrite/sessionGroupNames
[2023-12-12 16:08:07,123] [ERROR] Missing configuration key /services/routing/settings/managedSessions
[2023-12-12 16:08:07,123] [ERROR] Missing configuration key /services/routing/tuning/target/recentDurationMilliseconds
{
    "routing": {
        "apiKey": "",
        "settings": {
            "allowedProxies": [],
            "contentPopularity": {
                "algorithm": "score_based",
                "sessionGroupNames": []
            },
            "extendedContentIdentifier": {
            ...

The first thing that needs to be done is to rename the keys sessionGroupIds to sessionGroupNames. If the configuration was backed up to the file config_backup.json before upgrading, the keys can be renamed and the updated configuration can be applied by typing this:

sed 's/"sessionGroupIds"/"sessionGroupNames"/' config_backup.json | confcli -i
[2023-12-19 12:33:17,725] [ERROR] Missing configuration key /services/routing/translationFunctions
[2023-12-19 12:33:17,726] [ERROR] Missing configuration key /services/routing/settings/instream/dashManifestRewrite/sessionGroupNames
[2023-12-19 12:33:17,727] [ERROR] Missing configuration key /services/routing/settings/instream/hlsManifestRewrite/sessionGroupNames
[2023-12-19 12:33:17,727] [ERROR] Missing configuration key /services/routing/settings/managedSessions
[2023-12-19 12:33:17,727] [ERROR] Missing configuration key /services/routing/tuning/target/recentDurationMilliseconds

The configuration has not yet been converted, so the error messages are still printed. The configuration will be converted when the acd-confd service is restarted.

systemctl restart acd-confd

This concludes the conversion of the configuration and the router is ready to be used.

Configuration changes

Below are all configuration changes between version 1.4.0 and 1.6.0 listed. Normally nothing needs to be done about this since they will be upgraded automatically, but they are listed here for reference.

Added `translationFunctions` block

services.routing.translationFunctions has been added. It can be added as a map with two empty strings as values, to make the top of the configuration look like this:

{
    "services": {
        "routing": {
            "translationFunctions": {
                "request": "",
                "response": ""
            },
            ...

Renamed `sessionGroupIds` to `sessionGroupNames`

The keys services.routing.settings.instream.dashManifestRewrite.sessionGroupIds and services.routing.settings.instream.hlsManifestRewrite.sessionGroupIds have been renamed to services.routing.settings.instream.dashManifestRewrite.sessionGroupNames and services.routing.settings.instream.hlsManifestRewrite.sessionGroupNames respectively. Any session group IDs need to be manually converted to session group names.

After the conversion, the head of the configuration file might look like this:

{
    "services": {
        "routing": {
            "apiKey": "",
            "settings": {
                "allowedProxies": [],
                "contentPopularity": {
                    "algorithm": "score_based",
                    "sessionGroupNames": []
                },
                "extendedContentIdentifier": {
                    "enabled": false,
                    "includedQueryParams": []
                },
                "instream": {
                    "dashManifestRewrite": {
                        "enabled": false,
                        "sessionGroupNames": []
                    },
                    "hlsManifestRewrite": {
                        "enabled": false,
                        "sessionGroupNames": []
                    },
                    "reversedFilenameComparison": false
                },
                ...

Added `managedSessions` block

A services.routing.settings.managedSessions block has been added. After adding the block, the configuration might look like this:

{
    "services": {
        "routing": {
            "apiKey": "",
            "settings": {
                "allowedProxies": [],
                "contentPopularity": {
                    "algorithm": "score_based",
                    "sessionGroupNames": []
                },
                ...
                "managedSessions": {
                    "fraction": 0.0,
                    "maxActive": 100000,
                    "sessionTypes": []
                },
                "usageLog": {
                    "enabled": false,
                    "logInterval": 3600000
                }
            },
            ...

Added `recentDurationMilliseconds`

A services.routing.tuning.target.recentDurationMilliseconds key has been added to the configuration file, with a default value of 500. After adding the key, the configuration might look like this:

{
    "services": {
        "routing": {
            "apiKey": "",
            ...
            "tuning": {
                "target": {
                    ...
                    "recentDurationMilliseconds": 500,
                    ...

Storing the updated configuration

After all these changes have been done to the configuration file, it can be applied to the router using confcli.

confcli will still display error messages because the stored configuration is not valid. They will not be displayed anymore after the valid configuration has been applied.

$ confcli -i < updated_config.json
[2023-12-12 18:52:05,500] [ERROR] Missing configuration key /services/routing/translationFunctions
[2023-12-12 18:52:05,502] [ERROR] Missing configuration key /services/routing/settings/instream/dashManifestRewrite/sessionGroupNames
[2023-12-12 18:52:05,502] [ERROR] Missing configuration key /services/routing/settings/instream/hlsManifestRewrite/sessionGroupNames
[2023-12-12 18:52:05,503] [ERROR] Missing configuration key /services/routing/settings/managedSessions
[2023-12-12 18:52:05,511] [ERROR] Missing configuration key /services/routing/tuning/target/recentDurationMilliseconds

Raw configuration

The following changes have been made to the raw configuration. If the router is configured via confd and confcli, these changes will be applied by them. This section is only relevant if the router is configured via the v2/configuration API.

Simple changes

The following keys were added or removed. They will not need to be manually updated, the router will add the new keys with default values.

Removed the tuning.repeated_session_start_threshold_seconds key.
Removed the lua_paths key.
Added the tuning.target_recent_duration_milliseconds key.

EDNS proxy changes

If the router has been configured to use an EDNS server, the following has to be changed for the configuration to work.

The hosts.proxy_address key has been renamed to hosts.proxy_url and now accepts a port that is used when connecting to the proxy.

The cdns.http_port and cdns.https_port keys now configure the port that is used for connecting to the EDNS server, before they configured the port that is used for connecting to the proxy.

1.1.3 - Firewall

Firewall Configuration

For security reasons, the ESB3024 Installer does not automatically configure the local firewall to allow incoming traffic. It is the responsibility of the operations person to ensure that the system is protected from external access by placing it behind a suitable firewall solution. The following table describes the set of ports required for operation of the router.

Application	Port	Protocol	Direction	Source	Description
Prometheus Alert Manager	9093	TCP	IN	internal	Monitoring Services
Confd	5000	TCP	IN	internal	Configuration Services
Router	80	TCP	IN	public	Incoming HTTP Requests
Router	443	TCP	IN	public	Incoming HTTPS Requests
Router	5001	TCP	IN	localhost	Access to router’s REST API
Router	8000	TCP	IN	localhost	Internal monitoring port
EDNS-Proxy	8888	TCP	IN	localhost	Proxy EDNS Requests
Grafana	3000	TCP	IN	internal	Monitoring Services
Grafana-Loki	3100	TCP	IN	internal	Log monitoring daemon
Prometheus	9090	TCP	IN	internal	Monitoring Service

The “Direction” column represents the direction in which the connection is established.

IN - The connection is originated from an outside server
OUT - The connection is established from the host to an external server.

Once a connection is established through the firewall, bidirectional traffic must be allowed using the established connection.

For the “Source” column, the following terms are used.

internal - Any host or network which is allowed to monitor or operate the system.
public - Any host or subnet that can access the router. This includes any customer network that will be making routing requests.
localhost - Access can be limited to local connections only.
any - All traffic from any source or to any destination.

Additional Ports

Convoy bridge integration

The optional convoy-bridge service needs the ability to access the Convoy MariaDB service, which by default runs on port 3306 on all of the Convoy Management servers. To allow this integration to run, port 3306/tcp must be allowed from the router to the configured Convoy Management node.

1.1.4 - API Overview

A brief description of the API:s served by ESB3024 Router

ESB3024 Router provides two different types of API:s:

A content request API that is used by video clients to ask for content, normally using port 80 for HTTP and port 443 for HTTPS.
A few REST API:s used by administrators to configure and monitor the router installation, using port 5001 over HTTPS by default.

The content API won’t be described further in this document, since it’s a simple HTTP interface serving content as regular files or redirect responses.

Configuration – `/v2/configuration`

Used to check and update the configuration of ESB3024 Router. Change the configuration by PUTing a JSON payload to the endpoint, and view the current configuration by GETing the endpoint without payload. See Configuration for more information about the configuration

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`application/json`
`PUT`	`application/json`	Success	204 No Content	<N/A>
`PUT`	`application/json`	Failure	400 Bad Request	`application/json`¹

Example request (`GET`)

Request the currently applied configuration using a simple GET request:

$ curl -i https://router.example:5001/v2/configuration
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 1234
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "id": "router.example-5fc78d",
  "version": "v2",
  "image_tag": "20221117T190005Z-2a71714b6",
  "tuning": {
    "log_level": 3,
    [...]
  },
  "rest_api_server": {
    "enable": true,
    "port": 5001
  },
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "routing": {
    [...]
  },
  [...]
}

Example request (`PUT`)

Updating only a part of the configuration is possible by sending a JSON snippet with the fields to change:

$ curl -i -X PUT \
    -d '{"tuning" {"log_level": 3}}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v2/configuration
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Validate Configuration – `/v2/validate_configuration`

Used to determine if a JSON payload is correctly formatted without actually applying its configuration. A successful return status does not guarantee that the applied configuration will work, it only validates the JSON structure.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`PUT`	`application/json`	Success	204 No Content	<N/A>
`PUT`	`application/json`	Failure	400 Bad Request	`application/json`¹

Example request

When an expected field is missing from the payload, the validation will show which one and return an appropriate error message in its payload:

$ curl -i -X PUT \
    -d '{"routing": {"log_level": 3}}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v2/validate_configuration
HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Content-Length: 132
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

"Configuration validation: Configuration parsing failed. \
  Exception: [json.exception.out_of_range.403] (/routing) key 'id' not found"

Selection Input – `/v1/selection_input`

Selection input API can be used to inject external key:value data into the routing engine, making the data available when making routing decisions. An arbitrary JSON structure can be pushed to the endpoint. When performing GET or DELETE requests, specific selection input values can be accessed or deleted by including a path to the request. Note that not specifying a path will select all selection input values.

One use case for selection input is to provide data on cache availability. E.g. If you send {"edge-streamer-2-online": true} to the selection input API, you can create a routing condition eq('edge-streamer-online', true) to ensure that no traffic gets routed to the streamer if it’s offline. Note that sending the same key:value data to the selection input API will overwrite the previous value.

There is a configurable limit to how many key:value items that can be injected into the router, see tuning parameter selection_input_item_limit under tuning configuration.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`PUT`	`application/json`	Success	204 No Content	<N/A>
`PUT`	`application/json`	Failure	400 Bad Request	`application/json`
`GET`	<N/A>	Success	200 OK	`application/json`
`DELETE`	<N/A>	Success	204 No Content	<N/A>
`DELETE`	<N/A>	Failure	404 Not Found	<N/A>

Example successful request (`PUT`)

$ curl -i -X PUT \
    -d '{"host1_bitrate": 13000, "host1_capacity": 50000}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v1/selection_input
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Example unsuccessful request (`PUT`)

$ curl -i -X PUT \
    -d '{"cdn-status": {"session-count": 12345, "load-percent" 98}}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v1/selection_input
HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Content-Length: 169
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "error": "[json.exception.parse_error.101] parse error at line 1, column 57: \
    syntax error while parsing object separator - \
    unexpected number literal; expected ':'"
}

Example successful request (`GET`)

curl -i https://router.example:5001/v1/selection_input
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 129
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "host1_bitrate": 13000,
  "host1_capacity": 50000
}

Example successful specific value request (`GET`)

curl -i https://router.example:5001/v1/selection_input/path/to/value
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 129
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

1

Example successful request (`DELETE`)

curl -i -X DELETE https://router.example:5001/v1/selection_input
HTTP/1.1 204 OK
Access-Control-Allow-Origin: *
Content-Length: 129
X-Service-Identity: router.example-5fc78d

Example successful specific value request (`DELETE`)

curl -i -X DELETE  https://router.example:5001/v1/selection_input/value/to/delete
HTTP/1.1 204 OK
Access-Control-Allow-Origin: *
Content-Length: 129
X-Service-Identity: router.example-5fc78d

Example unsuccessful request (`DELETE`)

curl -i -X DELETE  https://router.example:5001/v1/selection_input/non/existent/value
HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Content-Length: 129
X-Service-Identity: router.example-5fc78d

Subnets – `/v1/subnets`

An API for managing named subnets that can be used for routing and block lists. See Subnets for more details.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`PUT`	`application/json`	Success	204 No Content	<N/A>
`PUT`	`application/json`	Failure	400 Bad Request	`application/json`

Example request

$ curl -i -X PUT \
    -d '{"255.255.255.255/24": "test"}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v1/subnets
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Subrunner Resource Usage – `/v1/usage`

Used to monitor the load on subrunners, the processes performing those tasks that are possible to run in parallel.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`application/json`

Example request

$ curl -i https://router.example:5001/v1/usage
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 1234
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "total_usage": {
    "content": {
      "lru": 0,
      "newest": "-",
      "oldest": "-",
      "total": 0
    },
    "sessions": 0,
    "subrunner_usage": {
      [...]
    }
  },
  "usage_per_subrunner": [
    {
      "subrunner_usage": {
        [...]
      }
    },
    [...]
  ]
}

Metrics – `/m1/v1/metrics`

An interface intended to be scraped by Prometheus. It is possible to scrape it manually to see current values, but doing so will reset some counters and cause actual Prometheus data to become faulty.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`text/plain`

Example request

$ curl -i https://router.example:5001/m1/v1/metrics
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 1234
Content-Type: text/plain
X-Service-Identity: router.example-5fc78d

# TYPE num_configuration_changes counter
num_configuration_changes 12
# TYPE num_log_errors_total counter
num_log_errors_total 0
# TYPE num_log_warnings_total counter
num_log_warnings_total{category=""} 123
# TYPE num_log_warnings_total counter
num_log_warnings_total{category="cdn"} 0
# TYPE num_log_warnings_total counter
num_log_warnings_total{category="content"} 0
# TYPE num_log_warnings_total counter
num_log_warnings_total{category="generic"} 10
# TYPE num_log_warnings_total counter
num_log_warnings_total{category="repeated_session"} 0
# TYPE num_ssl_errors_total counter
[...]

Node Visit Counters – `/v1/node_visits`

Used to gather statistics about the number of visits to each node in the routing tree. The returned value is a JSON object containing node ID names and their corresponding counter values.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`application/json`

See Routing Rule Evaluation Metrics for more details.

Example request

$ curl -i https://router.example:5001/v1/node_visits
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 73
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "cache1.tv": "99900",
  "offload": "100"
  "routingtable": "100000"
}

Node Visit Graph – `/v1/node_visits_graph`

Creates a GraphML representation of the node visitation data that can be rendered into an image to make it easier to understand the data.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`application/xml`

See Routing Rule Evaluation Metrics for more details.

Example request

> curl -i -k https://router.example:5001/v1/node_visits_graph
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 731
Content-Type: application/xml
X-Service-Identity: router.example-5fc78d

<?xml version="1.0"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
  <key id="visits" for="node" attr.name="visits" attr.type="string" />
  <graph id="G" edgedefault="directed">
    <node id="routingtable">
      <data key="visits">100000</data>
    </node>
    <node id="cache1.tv">
      <data key="visits">99900</data>
    </node>
    <node id="offload">
      <data key="visits">100</data>
    </node>
    <edge id="e0" source="routingtable" target="cache1.tv" />
    <edge id="e1" source="routingtable" target="offload" />
  </graph>
</graphml>

Session list - `/v1/sessions`

Used to monitor the load on subrunners, the processes performing those tasks that are possible to run in parallel.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`application/json`

Example request

$ curl -k -i https://router.example:5001/v1/sessions
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 12345
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "sessions": [
    {
      "age_seconds": 103,
      "cdn": "edgeware",
      "cdn_is_redirecting": false,
      "client_ip": "1.2.3.4",
      "host": "cdn.example:80",
      "id": "router.example-5fc78d-00000001",
      "idle_seconds": 103,
      "last_request_time": "2022-12-02T14:05:05Z",
      "latest_request_path": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
      "no_of_requests": 1,
      "requested_bytes": 0,
      "requests_redirected": 0,
      "requests_served": 0,
      "session_groups": [
        "all"
      ],
      "session_groups_generation": 2,
      "session_path": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
      "start_time": "2022-12-02T14:05:05Z",
      "type": "instream",
      "user_agent": "libmpv"
    },
    [...]
  ]
}

Session details - `/v1/sessions/<id: str>`

Used to get details about a specific session from the above session list. The id part of the URL corresponds to the id field in one of the returned session entries in the above response.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`application/json`
`GET`	<N/A>	Failure	404 Not Found	`application/json`

Example request

$ curl -k -i https://router.example:5001/v1/sessions/router.example-5fc78d-00000001
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 763
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "age_seconds": 183,
  "cdn": "edgeware",
  "cdn_is_redirecting": false,
  "client_ip": "1.2.3.4",
  "host": "cdn.example:80",
  "id": "router.example-5fc78d-00000001",
  "idle_seconds": 183,
  "last_request_time": "2022-12-02T14:05:05Z",
  "latest_request_path": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
  "no_of_requests": 1,
  "requested_bytes": 0,
  "requests_redirected": 0,
  "requests_served": 0,
  "session_groups": [
    "all"
  ],
  "session_groups_generation": 2,
  "session_path": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
  "start_time": "2022-12-02T14:05:05Z",
  "type": "instream",
  "user_agent": "libmpv"
}

Content List - `/v1/content`

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`application/json`

Example request

$ curl -k -i https://router.example:5001/v1/content
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 572
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "content": [
    [
      "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
      {
        "cached_count": 0,
        "content_requested": false,
        "content_set": false,
        "expiration_time": "2022-12-02T14:05:05Z",
        "key": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
        "listeners": 0,
        "manifest": "",
        "request_count": 4,
        "state": "HLS:MANIFEST-PENDING",
        "wait_count": 0
      }
    ]
  ]
}

Lua scripts – `/v1/lua/<path str>.lua`

Used to upload, retrieve and delete custom named Lua scripts on the router. Global functions in uploaded scripts automatically become available to Lua code in the configuration (which effectively may be viewed as hooks). Upload a script by PUTing a application/x-lua to the endpoint, and retrieve it by GETing the endpoint without payload.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`PUT`	`application/x-lua`	Success	204 No Content	<N/A>
`PUT`	`application/x-lua`	Failure	400 Bad Request	`application/json`
`GET`	<N/A>	Success	200 OK	`application/x-lua`
`GET`	<N/A>	Failure	404 Not Found	`application/json`
`DELETE`	<N/A>	Success	204 No Content	<N/A>
`DELETE`	<N/A>	Failure	400 Bad Request	`application/json`
`DELETE`	<N/A>	Failure	404 Not Found	`application/json`

Example request (`PUT`)

Save a Lua script under the name advanced_functions/f1.lua:

$ curl -i -X PUT \
    -d 'function fun1() return 1 end' \
    -H "Content-Type: application/x-lua" \
    https://router.example:5001/v1/lua/advanced_functions/f1.lua
HTTP/1.1 204 Successfully saved Lua file
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Example request (`PUT`, from file)

Upload an entire Lua file under the name advanced_functions/f1.lua:

First put your code in a file.

$ cat f1.lua
function fun1()
    return 1
end

Then upload it using the --data-binary flag to preserve newlines

$ curl -i -X PUT \
    --data-binary @f1.lua \
    -H "Content-Type: application/x-lua" \
    https://router.example:5001/v1/lua/advanced_functions/f1.lua
HTTP/1.1 204 Successfully saved Lua file
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Example request (`GET`)

Request the Lua script named advanced_functions/f1.lua using a GET request:

$ curl -i https://router.example:5001/v1/lua/advanced_functions/f1.lua
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 28
Content-Type: application/x-lua
X-Service-Identity: router.example-5fc78d

function fun1() return 1 end

Example request (`DELETE`)

Delete the Lua script named advanced_functions/f1.lua using a DELETE request:

$ curl -i -X DELETE \
    https://router.example:5001/v1/lua/advanced_functions/f1.lua
HTTP/1.1 204 Successfully removed Lua file
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

List Lua scripts – `/v1/lua`

Used to list previously uploaded custom Lua scripts on the router, retrieving their respective paths and file checksums.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`GET`	<N/A>	Success	200 OK	`application/json`

Example request

$ curl -k -i https://router.example:5001/v1/lua
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 108
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

[
  {
    "file_checksum": "d41d8cd98f00b204e9800998ecf8427e",
    "path": "advanced_functions/f1.lua"
  }
]

Debug a Lua expression – `/v1/lua/debug`

Used to debug an arbitrary Lua expression on the router in a “sandbox” (with no visible side effects to the state of the router), and inspect the result.

The Lua expression in the body is evaluated inside an isolated copy of the internal Lua environment including selection input. The stdout field of the resulting JSON body is populated with a concatenation of every string provided as argument to the Lua print() function during the course of evaluation. Upon a successful evaluation, as indicated by the success flag, return.value and return.lua_type_name capture the resulting Lua value. Otherwise, if valuation was aborted (e.g. due to a Lua exception), error_msg reflects any error description arising from the Lua environment.

REQUEST Method	Content-Type	RESPONSE Result	Status Code	Content-Type
`POST`	`application/x-lua`	Success	200 OK	`application/json`

Example successful request

$ curl -i -X POST \
    -d 'fun1()' \
    -H "Content-Type: application/x-lua" \
    https://router.example:5001/v1/lua/debug
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 123
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "error_msg": "",
  "return": {
    "lua_type_name": "number",
    "value": 1.0
  },
  "stdout": "",
  "success": true
}

Example unsuccessful request

(attempt to invoke unknown function)

$ curl -i -X POST \
    -d 'fun5()' \
    -H "Content-Type: application/x-lua" \
    https://router.example:5001/v1/lua/debug
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 123
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "error_msg": "[string \"function f0() ...\"]:2: attempt to call global 'fun5' (a nil value)",
  "return": {
    "lua_type_name": "",
    "value": null
  },
  "stdout": "",
  "success": false
}

Access logging

When the configuration parameter tuning.access_log is set to true, HTTP requests are logged in the common log format (see https://en.wikipedia.org/wiki/Common_Log_Format) and tagged with the matchable field "ACCESSLOG=1" to be retrieved with journalctl:

journalctl SYSLOG_IDENTIFIER=router ACCESSLOG=1

Example output

-- Logs begin at Tue 2023-05-23 07:57:16 UTC, end at Mon 2023-05-29 07:25:59 UTC. --
May 29 07:20:00 router[52236]: ::1 - - [29/May/2023:07:20:00 +0000] "GET /vod/batman.m3u8 HTTP/1.1" 302 0 "-" "curl/7.61.1"

Some useful optional arguments to journalctl are:

journalctl SYSLOG_IDENTIFIER=router ACCESSLOG=1 [--follow] [-o json-pretty] [--since 2022-06-06] [--until 2022-06-10]
[--grep="HTTP/1.1\" 404"]

See the journalctl documentation for further reading.

Footnotes

The content type of the response is set to “application/json” but the payload is actually a regular string without JSON syntax. ↩︎ ↩︎

1.1.5 - Configuration

How to write and deploy configuration for ESB3024 Router

1.1.5.1 - Configuration JSON Overview

A brief overview of the configuration JSON and its components

Configuration of ESB3024 Router is done by PUT:ing a JSON object to a REST API located at https://<router-host>:5001/v2/configuration by default.

This document will go through the JSON object, briefly describe its fields and link to more in-depth documentation where appropriate. For information on the JSON language syntax, please refer to the official JSON documentation.

Top-level fields

Here we’ll go through all the fields that may exist at the first level of the JSON object. In the following sections we’ll explore some of them in more detail while some others are complex enough to warrant their own pages.

"cdns" (array, objects)
A list of all CDNs associated with this router, described further down.
"content_server" (object)
Parameters for the content HTTP interface, described further down.
"custom_lua" (string)
Container-specific path to the parent folder under which the /v1/lua/<path str>.lua PUT endpoint will cache the uploaded Lua script files with relative paths as specified by the <path str>.lua argument. Uploaded Lua scripts are then loaded into the Lua environment available in request/response translation functions as well as routing weight functions. The /v1/lua/<path str>.lua PUT endpoint will attempt to create the folder and any required subdirectories on the specified path, should they not exist. Full read-write access for the router need to be maintained to the parent folder, so that the configured folder can be created by the router, to the configured folder itself, its children and every contained .lua script.
"hosts" (array, objects)
A list of all hosts available in the CDNs listed above, described further down.
"id" (string)
Always "ACD", read only.
"image_tag" (string)
A string unique to the version of software installed, read only.
"managed_sessions" (object)
Settings for how to configure managed sessions, described further down.
"metadata"(object)
Contains metadata about the configuration, described further down.
"request_translation_function" (string)
A Lua function body for changing incoming HTTP requests, described further down.
"response_translation_function" (string)
A Lua function body for changing outgoing HTTP responses, described further down.
"rest_api_server" (object)
Settings for the API endpoint used for e.g. PUT:ing the configuration, described further down.
"routing" (object)
Client routing rules, described further down.
"session_groups" (array, objects)
Classification rules for incoming requests, described further down.
"settings" (object)
Settings for functionality that can be enabled or disabled, described further down.
"standard_lua" (string)
Container-specific path to a folder containing pre-installed Lua script files that are loaded into the Lua environment available in request/response translation functions as well as routing weight functions.
"tuning" (object)
Adjustments of timings, log level, thresholds etc, described further down.
"version" (string)
Always "v2", read only.

`"cdns"`

A list containing CDN definitions. Each object contains the following data:

"disabled_hosts" (array, strings)
Hostnames temporarily taken out of routing, e.g. due to maintenance or because they are known to be broken. For redirecting CDNs, these hosts are added to an Exclude-Host header telling the CDN not to use them.
"http_port" (integer)
The HTTP port the hosts in the CDN listen to.
"https_port" (integer)
The HTTPS port the hosts in the CDN listen to.
"id" (string, unique)
Unique internal name used to reference the CDN within the router.
"manifest_availability_check" (object)
Specifies if a CDN should be checked for manifest availability, and if so for which session groups.
- "enabled" (boolean)
  Toggle availability check.
- "session_group_ids" (array, integers)
  List of the session groups for which to perform check.
"redirecting" (boolean)
Setting to true makes the router request a redirect URL from the CDN to forward to the client, otherwise the router constructs its own redirect response with the selected host in the Location header.
"response_code_rules" (array, objects)
List of response actions per matching response code from redirecting CDN. Each object contains the following data:
- "pattern" (string)
  Sequence of digits and / or wildcards in the form of the letter x. Example: "4xx" which matches response codes 400-499.
- "action" (string)
  Specifies which action to take when the response code from a redirecting CDN matches the pattern. Valid values are "default", which lets the router proceed as-if no pattern was matching, "continue", to proceed with selecting the next host in the CDN, "redirect" to let the router redirect the request to the URL returned by the redirecting CDN, and "proxy" which returns the response from the redirecting CDN back to the client as-if the response originated from the router itself.
- "new_response_code" (integer)
  The overridden response code to send back to the client irrespective of "action". It replaces the response code in the otherwise verbatim response for a selected "proxy" action.
"timeout_action" (string)
Specifies which action to take when the request to the redirecting CDN times out without detecting a response. Valid values are "abort" which stops selection, and "continue" which continues with selecting the next host in the CDN.

`"content_server"`

Settings for the routing interface that clients request content through.

"http_enable" (boolean)
Toggles if HTTP interface is available or not.
"http_port" (integer)
Which port the HTTP interface should listen on.
"https_enable" (boolean)
Toggles if encrypted HTTPS interface is available or not.
"https_port" (integer)
Which port the HTTPS interface should listen on.

`"hosts"`

A list containing CDN host definitions. Each object contains the following data:

"cdn_id" (string)
A reference to a CDN in the cdns array.
"host" (string)
The IP or hostname for the host.
"ipv6_address" (string)
If the client connects via IPv6 or has put an IPv6 address in the X-Forwarded-For header, then the router will connect to the CDN using the address from ipv6_address. If ipv6_address is not given, the connection will be made using the address from host.
"id" (string)
A unique identifier used in the routing rule tree.
"proxy_url" (string)
When using EDNS routing, the incoming request is sent to a proxy server located at proxy_url that creates and sends an EDNS request based on the client’s HTTP request. A default installation of ESB3024 includes an EDNS proxy located at http://localhost:8888.

`"managed_sessions"`

Settings for managed client sessions.

"fraction" (number)
The fraction of sessions to be managed, between 0.0 and 1.0.
"max_active" (integer)
The maximum number of active managed sessions.
"session_types" (array, objects)
A list of the various types of managed sessions and their respective settings. Each object contains the following data:
- "enabled" (boolean)
  description: Toggle this session type.
- "max_active" (integer)
  The maximum number of active sessions belonging to this session type.
- "session_group_ids" (array, integers)
  The session groups allowed to belong to this session type, or empty if any.
- "type" (string)
  The type of managed session. The only valid value is "instream".

`"metadata"`

Metadata mostly generated from the configuration.

"etag" (string)
The MD5 hash of the configuration’s size in memory.
"extra_info" (object)
This field can store any arbitrary key-value pairs, e.g. to tag or name a configuration. The only configurable field in "metadata".
"source_ip" (string)
The IP address from which the configuration was sent.
"timestamp"(string)
The ISO-8601 formatted timestamp of when the configuration was applied.

`"request_translation_function"`

The content of this field is a Lua function body that can be used to modify an incoming request before passing it on to be classified and routed. See Lua Hooks for details.

`"response_translation_function"`

The content of this field is a Lua function body that can be used to modify an outgoing response before it is sent back to the requesting client. See Lua Hooks for details.

`"rest_api_server"`

Settings for the configuration REST endpoint itself.

"enable" (boolean)
Determines whether the REST API is turned on or not.
"port" (integer)
HTTPS port to listen to.

`"routing"`

A tree structure of objects setting up a routing algorithm. See Routing Engine for more details.

`"session_groups"`

A list containing session group definitions. See Session Groups and Classification for details.

`"settings"`

Various behavioral settings related to content requests and CDN requirements.

"reversed_filename_comparison" (object)
Compares segment filenames from the end when trying to match requests with content. This allows the router to be compatible with CDNs that put e.g. a timestamp first in the filename.
- "enabled" (boolean)
  Toggle Akamai support on or off.
"allowed_clients" (array, strings)
If non-empty, this array contains all IP:s proxies that are allowed to use the X-Forwarded-For HTTP header.
"dash_manifest_rewrite" (object)
Rewrite DASH manifests to use absolute URLs for clients that don’t support HTTP redirect properly.
- "enabled" (boolean)
  Enable or disable DASH manifest rewrite.
- "session_group_ids" (array, integers)
  A list of session groups for which to perform manifest rewrite so it only has to be done for e.g. a set of user agents known to be faulty.
- "extended_content_identifier" (object)
  Sets a list of query parameters that may be used to identify a specific content along with the path, used for caching.
  - "enabled" (boolean)
    Turns on or off query parameter content identifiers.
  - "included_query_params" (array, strings)
    The list of query parameter names to allow.
"hls_manifest_rewrite" (object)
Rewrite HLS manifests to use absolute URLs for clients that don’t support HTTP redirect properly.
- "enabled" (boolean)
  Enable or disable DASH manifest rewrite.
- "session_group_ids" (array, integers)
  A list of session groups for which to perform manifest rewrite so it only has to be done for e.g. a set of user agents known to be faulty.
"usage_log" (object)
Usage logging settings. By default, usage logs such as the number of redirected requests handled during a time period are written to. "/opt/edgeware/acd/router/log/usage.log" unless the environment variable USAGE_LOGGING_DIR has been set to something else.
- "enabled" (boolean)
  Enables usage logging.
- "log_interval" (integer)
  Number of milliseconds between each usage log write.

`"tuning"`

Settings for tuning the behavior of the router.

"access_log" (boolean)
Enable or disable access logging.
"content_cache_size_full_manifests" (integer)
Size of cache for storing full manifests and associated metadata, in number of manifests.
"content_cache_size_light_manifests" (integer)
Size of cache for storing simplified manifest metadata, in number of manifests.
"content_light_cache_time_milliseconds" (integer)
Number of milliseconds before a light manifest content expires.
"content_live_cache_time_milliseconds" (integer)
Number of milliseconds before a full manifest live content expires.
"content_vod_cache_time_milliseconds" (integer)
Number of milliseconds before a full manifest VOD content expires.
"cout_flush_rate_milliseconds" (integer)
How often to flush logs to stdout even if buffer isn’t full to make sure logs arrive at a timely fashion.
"cpu_load_window_size" (integer)
Window size used to calculate a running average of the CPU load.
"eager_cdn_switching" (boolean)
Tells the router to evaluate the routing tree for every request of a session, even if there are no known issues for the client.
"http_pipelining_enable" (boolean)
Allows the router to send requests to the CDN while waiting for responses to earlier requests.
"log_level" (integer)
The verbosity of log printouts. Higher value means more logs: 0 -> Critical, 1 -> Error, 2 -> Warnings, 3 -> Info, 4 -> Debug, 5 -> Trace.
"max_connections_per_host" (integer)
The number of connections the router is allowed to have to a single CDN host.
"overload_threshold" (integer)
How long the task queue may be before router is considered overloaded.
"ready_threshold" (integer)
How short the task queue must become before an overloaded router is no longer considered overloaded.
"redirecting_cdn_manifest_download_retries" (integer)
Number of times the router can try requesting a manifest or redirect location from a CDN before giving up.
"repeated_session_start_threshold_seconds" (integer)
How long time a session must wait between requests to avoid being flagged for repeating requests too often, possibly leading to lowered quality of service.
"selection_input_item_limit" (integer)
The maximum number of key:value items that can be stored on the router using the /v1/selection_input API. Only the leaf values in the input JSON object count towards this limit.
"selection_input_metrics_timeout_seconds" (integer)
How many seconds a value send to the /v1/selection_input API remains before it’s considered obsolete.
"session_idle_deactivate_timeout_milliseconds" (integer)
Number of milliseconds between requests before a session is considered idle.
"session_idle_delete_timeout_milliseconds" (integer)
Number of milliseconds an idle session stays alive to be resumed before it is purged.
"target_connect_timeout_seconds" (integer)
Number of seconds allowed to establish a connection to a CDN.
"target_max_idle_time_seconds" (integer)
Number of seconds a connection to a CDN is kept alive even if no new requests occur.
"target_request_attempts" (integer)
Number of times the router will make requests to different hosts, when it gets 5xx error responses, before giving up and sending an error response to the client.
"target_response_timeout_seconds" (integer)
Number of seconds before a request to a CDN times out.
"target_retry_connect_timeout_seconds" (integer)
Number of seconds allowed when retrying to establish a connection to a CDN.
"target_retry_response_timeout_seconds" (integer)
Number of seconds before a retry request to a CDN times out.

A Full Configuration Example

{
  "cdns": [
    {
      "disabled_hosts": [],
      "http_port": 80,
      "https_port": 443,
      "id": "allowed-cdn",
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      },
      "redirecting": false
    },
    {
      "disabled_hosts": [],
      "http_port": 80,
      "https_port": 443,
      "id": "offload-cdn",
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      },
      "redirecting": false
    }
  ],
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "custom_lua": "/tmp/custom_lua",
  "hosts": [
    {
      "cdn_id": "allowed-cdn",
      "host": "allowed-host4.example",
      "id": "allowed-host-4"
    },
    {
      "cdn_id": "allowed-cdn",
      "host": "allowed-host6.example",
      "id": "allowed-host-6"
    },
    {
      "cdn_id": "offload-cdn",
      "host": "offload-host.example",
      "id": "offload-host"
    }
  ],
  "managed_sessions": {
    "fraction": 0.0,
    "max_active": 0,
    "session_types": [
      {
        "enabled": false,
        "max_active": 0,
        "session_group_ids": [],
        "type": "instream"
      }
    ]
  },
  "metadata": {
    "extra_info": {
      "field_1": "value_1",
      "config_name": "large_config",
      "purpose": "demonstration",
      "arbitrary": "value"
    }
  },
  "request_translation_function": "",
  "response_translation_function": "",
  "rest_api_server": {
    "enable": true,
    "port": 5001
  },
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "test-subnet-name",
        "member_order": "weighted",
        "members": [],
        "url_rewrite_function": "",
        "weight_function": "print(tostring(request.subnet)); return 0"
      },
      {
        "id": "test-host-4",
        "member_order": "weighted",
        "members": [],
        "url_rewrite_function": "",
        "weight_function": "retval = request.subnet == 'test_net_4' and 1 or 0;\
            if retval == 1 then print('### ipv4') end; return 0"
      },
      {
        "id": "test-host-6",
        "member_order": "weighted",
        "members": [],
        "url_rewrite_function": "",
        "weight_function": "retval = request.subnet == 'test_net_6' and 1 or 0;\
            if retval == 1 then print('### ipv6') end; return 0"
      },
      {
        "id": "test-offload",
        "member_order": "weighted",
        "members": [],
        "url_rewrite_function": "",
        "weight_function": "retval = ((request.subnet == 'test_net_6')\
            or (request.subnet == 'test_net_4')) and 1 or 0;\
            if retval == 0 then print('### offload') end; return 0"
      },
      {
        "id": "allowed-host-4",
        "member_order": "weighted",
        "members": [],
        "url_rewrite_function": "",
        "weight_function": "return request.subnet == 'test_net_4' and 1 or 0"
      },
      {
        "id": "allowed-host-6",
        "member_order": "weighted",
        "members": [],
        "url_rewrite_function": "",
        "weight_function": "return request.subnet == 'test_net_6' and 1 or 0"
      },
      {
        "id": "offlaod-host",
        "member_order": "weighted",
        "members": [],
        "url_rewrite_function": "",
        "weight_function": "return 1"
      }
    ],
    "url_rewrite_function": "",
    "weight_function": "return 100"
  },
  "session_groups": [
    {
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "hls",
            "rule": {
              "pattern": "/([^\\.]+)\\.news_reports_\\d+/\\.example\\.com.*",
              "rule_type": "regex_rule",
              "source": "session/content_url_path"
            }
          }
        ]
      ],
      "id": 1,
      "name": "hls"
    },
    {
      "classifiers": [
        [
          {
            "id": 2,
            "inverted": false,
            "name": "vod",
            "rule": {
              "pattern": "*vod*",
              "rule_type": "string_match_rule",
              "source": "session/content_url_path"
            }
          }
        ]
      ],
      "id": 2,
      "name": "vod"
    },
    {
      "classifiers": [
        [
          {
            "id": 5,
            "inverted": true,
            "name": "offload_no_peering_subnet",
            "rule": {
              "ip_ranges": [
                "158.174.0.0/16",
                "95.192.0.0/12"
              ],
              "rule_type": "ip_ranges_rule",
              "source": "session/client_ip"
            }
          }
        ]
      ],
      "id": 5,
      "name": "offload_no_peering_subnet"
    },
    {
      "classifiers": [
        [
          {
            "id": 6,
            "inverted": true,
            "name": "offload_country_name",
            "rule": {
              "country": "Sweden",
              "rule_type": "geoip_rule",
              "source": "session/client_ip"
            }
          }
        ]
      ],
      "id": 6,
      "name": "offload_country_name"
    },
    {
      "classifiers": [
        [
          {
            "id": 7,
            "inverted": false,
            "name": "onload_city_name",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip"
            }
          }
        ]
      ],
      "id": 7,
      "name": "onload_city_name"
    },
    {
      "classifiers": [
        [
          {
            "id": 8,
            "inverted": false,
            "name": "offload_no_peering_asn_name",
            "rule": {
              "asn": "Tele*",
              "rule_type": "geoip_rule",
              "source": "session/client_ip"
            }
          }
        ]
      ],
      "id": 8,
      "name": "offload_no_peering_asn_name"
    }
  ],
  "settings": {
    "reversed_filename_comparison": {
      "enabled": false
    },
    "dash_manifest_rewrite": {
      "enabled": false,
      "session_group_ids": []
    },
    "extended_content_identifier": {
      "enabled": false,
      "included_query_params": []
    },
    "hls_manifest_rewrite": {
      "enabled": false,
      "session_group_ids": []
    }
  },
  "standard_lua": "/tmp/standard_lua",
  "tuning": {
    "access_log": false,
    "content_cache_size_full_manifests": 1000,
    "content_cache_size_light_manifests": 10000,
    "content_light_cache_time_milliseconds": 86400000,
    "content_live_cache_time_milliseconds": 5000,
    "content_vod_cache_time_milliseconds": 10000,
    "cout_flush_rate_milliseconds": 1000,
    "cpu_load_window_size": 10,
    "eager_cdn_switching": false,
    "http_pipelining_enable": false,
    "log_level": 5,
    "max_connections_per_host": 5,
    "overload_threshold": 32,
    "ready_threshold": 8,
    "redirecting_cdn_manifest_download_retries": 2,
    "repeated_session_start_threshold_seconds": 30,
    "selection_input_metrics_timeout_seconds": 86400,
    "session_idle_deactivate_timeout_milliseconds": 20000,
    "session_idle_delete_timeout_milliseconds": 1800000,
    "target_connect_timeout_seconds": 1,
    "target_max_idle_time_seconds": 30,
    "target_request_attempts": 2,
    "target_response_timeout_seconds": 1,
    "target_retry_connect_timeout_seconds": 2,
    "target_retry_response_timeout_seconds": 2
  }
}

1.1.5.2 - WebUI Configuration

How to use the web user interface for configuration.

The web based user interface is installed as a separate component and can be used to configure many common use cases. After navigating to the UI, a login screen will be presented.

Enter your credentials and log in. In the top left corner is a menu to select what section of the configuration to change. The configuration that will be active on the router is added in the Routing Workflow view. However, basic elements such as classification rules and routing targets, etc must be added first. Hence the following main steps are required to produce a proper configuration:

Create classifiers serving as basic elements to create session groups.
Create session groups which, using the classifiers, tag requests/clients for later use in the routing logic. of the incoming traffic.
Define offload rules.
Define rules to control behavior of internal traffic.
Define backup rules to be used if the routing targets in the above step are unavailable.
Finally, create the desired routing workflow using the elements defined in the previous steps.

A simplified concrete example of the above steps could be:

Create two classifiers “smartphone” and “off-net”.
Create a session group “mobile off-net”.
Offload off-net traffic from mobile phones to a public CDN.
Route other traffic to a private CDN.
If the private CDN has an outage, use the public CDN for all traffic.

Hence, to start with, define the classifiers you will need. Those are based on information in the incoming request, optionally in combination with GeoIP databases or subnet information configured via the Subnet API. Here we show how to set up a GeoIP classifier. Note that the Director ships with a compatible snapshot of the GeoIP database, but for a production system a licensed and updated database is required.

GeoIP Classifier

Click the plus sign indicated in the picture above to create a new GeoIP classifier. You will be presented with the following view:

GeoIP Classifier Create

Here you can enter the geographical data on which to match, or check the “Inverted” check box to match anything except the entered geographical data.

The other kinds of classifiers are configured in a similar way.

After having added all the classifiers you need, it is time to create the session groups. Those are named filters that group incoming requests, typically video playback sessions in a video streaming CDN, and are defined with the help of the classifiers. For example, a session group “off-net mobile devices” could be composed of the classifiers “off-net traffic” and “mobile devices”.

Open the Session Groups view from the menu and hit the plus sign to add a new session group.

Session Groups Session Group Create

Define the new sessions groups by combining the previously created classifiers. It is often convenient to define an “All” session group that matches any incoming request.

Next go the “CDN Offload” view:

CDN Offload

Here you define conditions for CDN offload. Each row defines a rule for offloading a specified session group. The rule makes use of the Selection Input API. This is an integration API that provides a way to supply additional data for use in the routing decision. Common examples are current bitrates or availability status. The selection input variables to use must be defined in the “Selection Input Types” view in the “Administration” section of the menu:

Selection Input Types

Reach out to the solution engineers from Agile Content in order to perform this integration in the best way. If no external data is required, such that the offload rule can be based solely based on session groups, this is not necessary and the condition field can be set to “Always” or “Disabled”.

When clicking the plus sign to add a new CDN Offload rule, the following view is presented:

CDN Offload Create

The selection input rule is phrased in terms of a variable being above or below a threshold, but also a state such as “available” taking values 0 or 1 can be supported by for instance checking if “available” is below 1.

Moving on, if an incoming request is not offloaded, it will be handled by the Primary CDN section of the routing configuration.

Primary CDN

Add all hosts in your primary CDN, together with a weight. A row in this table will be selected by random weighted load balancing. If each weight is the same, each row will be selected with the same probability. Another example would be three rows with weights 100, 100 and 200 which would randomly balance 50% of the load on the last row and the remaining load on the first two rows, i.e. 25% on each of the first and second row. If a Primary CDN host is unavailable, that host will not take part in the random selection.

If all hosts are unavailable, as a final resort the routing evaluation will go to the final Backup CDN step:

Backup CDN

Here you can define what to do when all else fail. If not all requests are covered, for example with an “All” session group, then the request will fail with 403 Forbidden.

Now you have defined the basic elements and it is time to define the routing workflow. Select “Routing Workflow” from the menu, as pictured below. Here you can combine the elements previously created to achieve the desired routing behavior.

Routing Workflow

When everything seems correct, open the “Publish Routing” view from the menu:

Publish Routing

Hit “Publish All Changes” and verify that you get a successful result.

1.1.5.3 - Confd and Confcli

Using the command line tool confcli to set up routing rules

Configuration of a complex routing tree can be difficult. The command line interface tool called confcli has been developed to make it simpler. It combines building blocks, representing simple routing decisions, into complex routing trees capable of satisfying almost any routing requirements.

These blocks are translated into an ESB3024 Router configuration which is automatically sent to the router, overwriting existing routing rules, CDN list and host list.

Installation and Usage

The confcli tools are installed alongside ESB3024 Router, on the same host, and the confcli command line tool itself is made available on the host machine.

Simply type confcli in a shell on the host to see the current routing configuration:

$ confcli
{
    "services": {
        "routing": {
            "settings": {
                "allowedProxies": [],
                "contentPopularity": {
                    "algorithm": "score_based",
                    "sessionGroupNames": []
                },
                "extendedContentIdentifier": {
                    "enabled": false,
                    "includedQueryParams": []
                },
                "instream": {
                    "dashManifestRewrite": {
                        "enabled": false,
                        "sessionGroupNames": []
                    },
                    "hlsManifestRewrite": {
                        "enabled": false,
                        "sessionGroupNames": []
                    },
                    "reversedFilenameComparison": false
                },
                "usageLog": {
                    "enabled": false,
                    "logInterval": 3600000
                }
            },
            "tuning": {
                "content": {
                    "cacheSizeFullManifests": 1000,
                    "cacheSizeLightManifests": 10000,
                    "lightCacheTimeMilliseconds": 86400000,
                    "liveCacheTimeMilliseconds": 100,
                    "vodCacheTimeMilliseconds": 10000
                },
                "general": {
                    "accessLog": false,
                    "coutFlushRateMilliseconds": 1000,
                    "cpuLoadWindowSize": 10,
                    "eagerCdnSwitching": false,
                    "httpPipeliningEnable": false,
                    "logLevel": 3,
                    "maxConnectionsPerHost": 5,
                    "overloadThreshold": 32,
                    "readyThreshold": 8,
                    "redirectingCdnManifestDownloadRetries": 2,
                    "repeatedSessionStartThresholdSeconds": 30,
                    "selectionInputMetricsTimeoutSeconds": 30
                },
                "session": {
                    "idleDeactivateTimeoutMilliseconds": 20000,
                    "idleDeleteTimeoutMilliseconds": 1800000
                },
                "target": {
                    "responseTimeoutSeconds": 5,
                    "retryConnectTimeoutSeconds": 2,
                    "retryResponseTimeoutSeconds": 2,
                    "connectTimeoutSeconds": 5,
                    "maxIdleTimeSeconds": 30,
                    "requestAttempts": 3
                }
            },
            "sessionGroups": [],
            "classifiers": [],
            "hostGroups": [],
            "rules": [],
            "entrypoint": "",
            "applyConfig": true
        }
    }
}

The CLI tool can be used to modify, add and delete values by providing it with the “path” to the object to change. The path is constructed by joining the field names leading up to the value with a period between each name, e.g. the path to the entrypoint is services.routing.entrypoint since entrypoint is nested under the routing object, which in turn is under the services root object. Lists use an index number in place of a field name, where 0 indicates the very first element in the list, 1 the second element and so on.

If the list contains objects which have a field with the name name, the index number can be replaced by the unique name of the object of interest.

Tab completion is supported by confcli. Pressing tab once will complete as far as possible, and pressing tab twice will list all available alternatives at the path constructed so far.

Display the values at a specific path:

$ confcli services.routing.hostGroups
{
    "hostGroups": [
        {
            "name": "internal",
            "type": "redirecting",
            "httpPort": 80,
            "httpsPort": 443,
            "hosts": [
                {
                    "name": "rr1",
                    "hostname": "rr1.example.com",
                    "ipv6_address": ""
                }
            ]
        },
        {
            "name": "external",
            "type": "host",
            "httpPort": 80,
            "httpsPort": 443,
            "hosts": [
                {
                    "name": "offload-streamer1",
                    "hostname": "streamer1.example.com",
                    "ipv6_address": ""
                },
                {
                    "name": "offload-streamer2",
                    "hostname": "streamer2.example.com",
                    "ipv6_address": ""
                }
            ]
        }
    ]
}

Display the values in a specific list index:

$ confcli services.routing.hostGroups.1
{
    "1": {
        "name": "external",
        "type": "host",
        "httpPort": 80,
        "httpsPort": 443,
        "hosts": [
            {
                "name": "offload-streamer1",
                "hostname": "streamer1.example.com",
                "ipv6_address": ""
            },
            {
                "name": "offload-streamer2",
                "hostname": "streamer2.example.com",
                "ipv6_address": ""
            }
        ]
    }
}

Display the values in a specific list index using the object’s name:

$ confcli services.routing.hostGroups.1.hosts.offload-streamer2
{
    "offload-streamer2": {
        "name": "offload-streamer2",
        "hostname": "streamer2.example.com",
        "ipv6_address": ""
    }
}

Modify a single value:

confcli services.routing.hostGroups.1.hosts.offload-streamer2.hostname new-streamer.example.com
services.routing.hostGroups.1.hosts.offload-streamer2.hostname = 'new-streamer.example.com'

Delete an entry:

$ confcli services.routing.sessionGroups.Apple.classifiers.
{
    "classifiers": [
        "Apple",
        ""
    ]
}

$ confcli services.routing.sessionGroups.Apple.classifiers.1 -d
http://localhost:5000/config/__active/services/routing/sessionGroups/Apple/classifiers/1 reset to default/deleted

$ confcli services.routing.sessionGroups.Apple.classifiers.
{
    "classifiers": [
        "Apple"
    ]
}

Adding new values in objects and lists is done using a wizard by invoking confcli with a path and the -w argument. This will be shown extensively in the examples further down in this document rather than here.

If you have a JSON file with a previously generated confcli configuration output it can be applied to a system by typing confcli -i <file path>.

Session Classification

In order to perform routing it is necessary to classify incoming sessions according to the relevant parameters. This is done through session groups and their associated classifiers.

There are different ways of classifying a request:

Strings with wildcards: Simple case-insensitive string pattern with support for adding asterisks (’*’) in order to match any value at that point in the pattern.
String with regular expressions: A complex string matching pattern capable of matching more complicated strings than the simple wildcard matching type.

Valid string matching sources are content_url_path, content_url_query_parameters, hostname and user_agent, examples of which will be shown below.

GeoIP: Based on the geographic location of the client, supporting wildcard matching. Geographic location data is provided by MaxMind. The possible values to match with are any combinations of:
- Continent
- Country
- Cities
- ASN
IP range: Based on whether a client’s IP belongs to a set of IP ranges or not.
Subnet: Tests if a client’s IP belongs to a named subnet, see Subnets for more details.
ASN ID list: Checks to see if a client’s IP belongs to any of the specified ASN IDs.

A session group may have more than one classifier. If it does, all the classifiers must match the incoming client request for it to belong to the session group. It is also possible for a request to belong to multiple session groups, or to none.

To send certain clients to a specific host you first need to create a suitable classifier using confcli in wizard mode. The wizard will guide you through the process of creating a new entry, asking you what value to input for each field and helping you by telling you what inputs are allowed for restricted fields such as the string comparison source mentioned above:

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 3
  Adding a 'geoip' element
    classifier : {
      name (default: ): sweden_matcher
      type (default: geoip): ⏎
      inverted (default: False): ⏎
      continent (default: ): ⏎
      country (default: ): sweden
      cities : [
        city (default: ): ⏎
        Add another 'city' element to array 'cities'? [y/N]: ⏎
      ]
      asn (default: ): ⏎
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "sweden_matcher",
      "type": "geoip",
      "inverted": false,
      "continent": "",
      "country": "sweden",
      "cities": [
        ""
      ],
      "asn": ""
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 5
  Adding a 'ipranges' element
    classifier : {
      name (default: ): company_matcher
      type (default: ipranges): ⏎
      inverted (default: False): ⏎
      ipranges : [
        iprange (default: ): 90.128.0.0/12
        Add another 'iprange' element to array 'ipranges'? [y/N]: ⏎
      ]
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "company_matcher",
      "type": "ipranges",
      "inverted": false,
      "ipranges": [
        "90.128.0.0/12"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 7
  Adding a 'stringMatcher' element
    classifier : {
      name (default: ): apple_matcher
      type (default: stringMatcher): ⏎
      inverted (default: False): ⏎
      source (default: content_url_path): user_agent
      pattern (default: ): *apple*
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "apple_matcher",
      "type": "stringMatcher",
      "inverted": false,
      "source": "user_agent",
      "pattern": "*apple*"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 6
  Adding a 'regexMatcher' element
    classifier : {
      name (default: ): content_matcher
      type (default: regexMatcher): ⏎
      inverted (default: False): ⏎
      source (default: content_url_path): ⏎
      pattern (default: ): .*/(live|news_channel)/.*m3u8
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "content_matcher",
      "type": "regexMatcher",
      "inverted": false,
      "source": "content_url_path",
      "pattern": ".*/(live|news_channel)/.*m3u8"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: subnet
  Adding a 'subnet' element
    classifier : {
      name (default: ): company_matcher
      type (default: subnet): ⏎
      inverted (default: False): ⏎
      pattern (default: ): company
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "company_matcher",
      "type": "subnet",
      "inverted": false,
      "pattern": "company"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 4
  Adding a 'hostName' element
    classifier : {
      name (default: ): host_name_classifier
      type (default: hostName): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): ⏎
      pattern (default: ): *live.example*
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: n
]
Generated config:
{
  "classifiers": [
    {
      "name": "host_name_classifier",
      "type": "hostName",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "*live.example*"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 1
  Adding a 'contentUrlPath' element
    classifier : {
      name (default: ): vod_matcher
      type (default: contentUrlPath): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): ⏎
      pattern (default: ): *vod*
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: n
]
Generated config:
{
  "classifiers": [
    {
      "name": "vod_matcher",
      "type": "contentUrlPath",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "*vod*"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 2
  Adding a 'contentUrlQueryParameters' element
    classifier : {
      name (default: ): bitrate_matcher
      type (default: contentUrlQueryParameters): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): regex
      pattern (default: ): .*bitrate=100000.*
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: n
]
Generated config:
{
  "classifiers": [
    {
      "name": "bitrate_matcher",
      "type": "contentUrlQueryParameters",
      "inverted": false,
      "patternType": "regex",
      "pattern": ".*bitrate=100000.*"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 9
  Adding a 'userAgent' element
    classifier : {
      name (default: ): iphone_matcher
      type (default: userAgent): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): regex
      pattern (default: ): i(P|p)hone
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: n
]
Generated config:
{
  "classifiers": [
    {
      "name": "iphone_matcher",
      "type": "userAgent",
      "inverted": false,
      "patternType": "regex",
      "pattern": "i(P|p)hone"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: asnIds
    2: contentUrlPath
    3: contentUrlQueryParameters
    4: geoip
    5: hostName
    6: ipranges
    7: regexMatcher
    8: stringMatcher
    9: subnet
    10: userAgent
  Choose element index or name: asnIds
  Adding a 'asnIds' element
    classifier : {
      name (default: ): asn_matcher
      type (default: asnIds): ⏎
      inverted (default: False): ⏎
      asnIds <The list of ASN IDs to accept. (default: [])>: [
        asnId: 1
        Add another 'asnId' element to array 'asnIds'? [y/N]: y
        asnId: 2
        Add another 'asnId' element to array 'asnIds'? [y/N]: y
        asnId: 3
        Add another 'asnId' element to array 'asnIds'? [y/N]: ⏎
      ]
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "asn_matcher",
      "type": "asnIds",
      "inverted": false,
      "asnIds": [
        1,
        2,
        3
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

These classifiers can now be used to construct session groups and properly classify clients. Using the examples above, let’s create a session group classifying clients from Sweden using an Apple device:

$ confcli services.routing.sessionGroups -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): inSwedenUsingAppleDevice
    classifiers : [
      classifier (default: ): sweden_matcher
      Add another 'classifier' element to array 'classifiers'? [y/N]: y
      classifier (default: ): apple_matcher
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "inSwedenUsingAppleDevice",
      "classifiers": [
        "sweden_matcher",
        "apple_matcher"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

Clients classified by the sweden_matcher and apple_matcher classifiers will now be put in the session group inSwedenUsingAppleDevice. Using session groups in routing will be demonstrated later in this document.

Advanced Classification

The above example will simply apply all classifiers in the list, and as long as they all evaluate to true for a session, that session will be tagged with the session group. For situations where this isn’t enough, classifiers can instead be combined using simple logic statements to form complex rules.

A first simple example can be a session group that accepts any viewers in either ASN 1, 2 or 3 (corresponding to the classifier asn_matcher or living in Sweden. This can be done by creating a session group, and adding the following logic statement:

'sweden_matcher' OR 'asn_matcher'

A slightly more advanced case is where a session group should only contain sessions neither in any of the three ASNs nor in Sweden. This is done by negating the previous example:

NOT ('sweden_matcher' OR 'asn_matcher')

A single classifier can also be negated, rather than the whole statement, for example to accept any Swedish viewers except those in the three ASNs:

'sweden_matcher' AND NOT 'asn_matcher'

Arbitrarily complex statements can be created using classifier names, parentheses, and the keywords AND, OR and NOT.

For example a session group accepting any Swedish viewers except those in the Stockholm region unless they are also Apple users:

'sweden_matcher' AND (NOT 'stockholm_matcher' OR 'apple_matcher')

Note that the classifier names must be enclosed in single quotes when using this syntax.

Applying this kind of complex classifier using confcli is no more difficult than adding a single classifier at a time:

$ confcli services.routing.sessionGroups. -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): complex_group
    classifiers : [
      classifier (default: ): 'sweden_matcher' AND (NOT 'stockholm_matcher' OR 'apple_matcher')
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "complex_group",
      "classifiers": [
        "'sweden_matcher' AND (NOT 'stockholm_matcher' OR 'apple_matcher')"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

CDNs and Hosts

Configuration using confcli has no real concept of CDNs, instead it has groups of hosts that share some common settings such as HTTP(S) port and whether they return a redirection URL, serve content directly or perform a DNS lookup. Of these three variants, the two former share the same parameters, while the DNS variant is slightly different.

Each host belongs to a host group and may itself be an entire CDN using a single public hostname or a single streamer server, all depending on the needs of the user.

Host Health

When creating a host in the confd configuration, you have the option to define a list of health check functions. Each health check function must return true for a host to be selected. This means that the host will only be considered available if all the defined health check functions evaluate to true. If any of the health check functions return false, the host will be considered unavailable and will not be selected for routing. All health check functions are detailed in the section Health Check Functions.

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: 3
  Adding a 'redirecting' element
    hostGroup : {
      name (default: ): edgeware
      type (default: redirecting): ⏎
      httpPort (default: 80): ⏎
      httpsPort (default: 443): ⏎
      hosts : [
        host : {
          name (default: ): rr1
          hostname (default: ): convoy-rr1.example.com
          ipv6_address (default: ): ⏎
          healthChecks : [
            healthCheck (default: always()): basic_health_check()
            Add another 'healthCheck' element to array 'healthChecks'? [y/N]: n
          ]
        }
        Add another 'host' element to array 'hosts'? [y/N]: y
        host : {
          name (default: ): rr2
          hostname (default: ): convoy-rr2.example.com
          ipv6_address (default: ): ⏎
          healthChecks : [
            healthCheck (default: always()): ⏎
            Add another 'healthCheck' element to array 'healthChecks'? [y/N]: n
          ]
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: ⏎
]
Generated config:
{
  "hostGroups": [
    {
      "name": "edgeware",
      "type": "redirecting",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
        {
          "name": "rr1",
          "hostname": "convoy-rr1.example.com",
          "ipv6_address": "",
          "healthChecks": [
            "basic_health_check()"
          ]
        },
        {
          "name": "rr2",
          "hostname": "convoy-rr2.example.com",
          "ipv6_address": "",
          "healthChecks": [
            "always()"
          ]
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: 1
  Adding a 'dns' element
    hostGroup : {
      name (default: ): external-dns
      type (default: dns): ⏎
      hosts : [
        host : {
          name (default: ): dns-host
          hostname (default: ): dns.example.com
          ipv6_address (default: ): ⏎
          healthChecks : [
            healthCheck (default: always()): ⏎
            Add another 'healthCheck' element to array 'healthChecks'? [y/N]: n
          ]
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: ⏎
]
Generated config:
{
  "hostGroups": [
    {
      "name": "external-dns",
      "type": "dns",
      "hosts": [
        {
          "name": "dns-host",
          "hostname": "dns.example.com",
          "ipv6_address": "",
          "healthChecks": [
            "always()"
          ]
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

Rule Blocks

The routing configuration using confcli is done using a combination of logical building blocks, or rules. Each block evaluates the incoming request in some way and sends it on to one or more sub-blocks. If the block is of the host type described above, the client is sent to that host and the evaluation is done.

Existing blocks

Currently supported blocks are:

allow: Incoming requests, for which a given rule function matches, are immediately sent to the provided onMatch target.
consistentHashing: Splits incoming requests randomly between preferred hosts, determined by the proprietary consistent hashing algorithm. The amount of hosts to split between is controlled by the spreadFactor.
contentPopularity: Splits incoming requests into two sub-blocks depending on how popular the requested content is.
deny: Incoming requests, for which a given rule function matches, are immediately denied, and all non-matching requests are sent to the onMiss target.
firstMatch: Incoming requests are matched by an ordered series of rules, where the request will be handled by the first rule for which the condition evaluates to true.
random: Splits incoming requests randomly and equally between a list of target sub-blocks. Useful for simple load balancing.
split: Splits incoming requests between two sub-blocks depending on how the request is evaluated by a provided function. Can be used for sending clients to different hosts depending on e.g. geographical location or client hardware type.
weighted: Randomly splits incoming requests between a list of target sub-blocks, weighted according to each target’s associated weight rule. A higher weight means a higher portion of requests will be routed to a sub-block. Rules can be used to decide whether or not to pick a target.
rawGroup: Contains a raw ESB3024 Router configuration routing tree node, to be inserted as is in the generated configuration. This is only meant to be used in the rare cases when it’s impossible to construct the required routing behavior in any other way.
rawHost: A host reference for use as endpoints in rawGroup trees.

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: allow
  Adding a 'allow' element
    rule : {
      name (default: ): allow
      type (default: allow): ⏎
      condition (default: ): customFunction()
      onMatch (default: ): rr1
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "content",
      "type": "contentPopularity",
      "condition": "customFunction()",
      "onMatch": "rr1"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: consistentHashing
  Adding a 'consistentHashing' element
    rule : {
      name (default: ): consistentHashingRule
      type (default: consistentHashing): 
      spreadFactor (default: 1): 2
      targets : [
        target : {
          target (default: ): rr1
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): rr2
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): rr3
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "consistentHashingRule",
      "type": "consistentHashing",
      "spreadFactor": 2,
      "targets": [
        {
          "target": "rr1",
          "enabled": true
        },
        {
          "target": "rr2",
          "enabled": true
        },
        {
          "target": "rr3",
          "enabled": true
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: contentPopularity
  Adding a 'contentPopularity' element
    rule : {
      name (default: ): content
      type (default: contentPopularity): ⏎
      contentPopularityCutoff (default: 10): 20
      onPopular (default: ): rr1
      onUnpopular (default: ): rr2
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "content",
      "type": "contentPopularity",
      "contentPopularityCutoff": 20.0,
      "onPopular": "rr1",
      "onUnpopular": "rr2"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: deny
  Adding a 'deny' element
    rule : {
      name (default: ): deny
      type (default: deny): ⏎
      condition (default: ): customFunction()
      onMiss (default: ): rr1
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "content",
      "type": "contentPopularity",
      "condition": "customFunction()",
      "onMiss": "rr1"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: firstMatch
  Adding a 'firstMatch' element
    rule : {
      name (default: ): firstMatch
      type (default: firstMatch): ⏎
      targets : [
        target : {
          onMatch (default: ): rr1
          rule (default: ): customFunction()
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          onMatch (default: ): rr2
          rule (default: ): otherCustomFunction()
        }
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "firstMatch",
      "type": "firstMatch",
      "targets": [
        {
          "onMatch": "rr1",
          "condition": "customFunction()"
        },
        {
          "onMatch": "rr2",
          "condition": "otherCustomFunction()"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: random
  Adding a 'random' element
    rule : {
      name (default: ): random
      type (default: random): ⏎
      targets : [
        target (default: ): rr1
        Add another 'target' element to array 'targets'? [y/N]: y
        target (default: ): rr2
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "random",
      "type": "random",
      "targets": [
        "rr1",
        "rr2"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: split
  Adding a 'split' element
    rule : {
      name (default: ): split
      type (default: split): ⏎
      condition (default: ): custom_function()
      onMatch (default: ): rr2
      onMiss (default: ): rr1
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "split",
      "type": "split",
      "condition": "custom_function()",
      "onMatch": "rr2",
      "onMiss": "rr1"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules. -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: weighted
  Adding a 'weighted' element
    rule : {
      name (default: ): weight
      type (default: weighted): ⏎
      targets : [
        target : {
          target (default: ): rr1
          weight (default: 100): ⏎
          condition (default: always()): always()
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): rr2
          weight (default: 100): si('rr2-input-weight')
          condition (default: always()): gt('rr2-bandwidth', 1000000)
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): rr2
          weight (default: 100): custom_func()
          condition (default: always()): always()
        }
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "weight",
      "type": "weighted",
      "targets": [
        {
          "target": "rr1",
          "weight": "100",
          "condition": "always()"
        },
        {
          "target": "rr2",
          "weight": "si('rr2-input-weight')",
          "condition": "gt('rr2-bandwith', 1000000)"
        },
        {
          "target": "rr2",
          "weight": "custom_func()",
          "condition": "always()"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

>> First add a raw host block that refers to a regular host

$ confcli services.routing.rules. -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: rawHost
  Adding a 'rawHost' element
    rule : {
      name (default: ): raw-host
      type (default: rawHost): ⏎
      hostId (default: ): rr1
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "raw-host",
      "type": "rawHost",
      "hostId": "rr1"
    }
  ]
}
Merge and apply the config? [y/n]: y

>> And then add a rule using the host node

$ confcli services.routing.rules. -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: rawGroup
  Adding a 'rawGroup' element
    rule : {
      name (default: ): raw-node
      type (default: rawGroup): ⏎
      memberOrder (default: sequential): ⏎
      members : [
        member : {
          target (default: ): raw-host
          weightFunction (default: ): return 1
        }
        Add another 'member' element to array 'members'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "raw-node",
      "type": "rawGroup",
      "memberOrder": "sequential",
      "members": [
        {
          "target": "raw-host",
          "weightFunction": "return 1"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

Rule Language

Some blocks, such as the split and firstMatch types, have a rule field that contains a small function in a very simple programming language. This field is used to filter any incoming client requests in order to determine how to rule block should react.

In the case of a split block, the rule is evaluated and if it is true the client is sent to the onMatch part of the block, otherwise it is sent to the onMiss part for further evaluation.

In the case of a firstMatch block, the rule for each target will be evaluated top to bottom in order until either a rule evaluates to true or the list is exhausted. If a rule evaluates to true, the client will be sent to the onMatch part of the block, otherwise the next target in the list will be tried. If all targets have been exhausted, then the entire rule evaluation will fail, and the routing tree will be restarted with the firstMatch block effectively removed.

Example of Boolean Functions

Let’s say we have an ESB3024 Router set up with a session group that matches Apple devices (named “Apple”). To route all Apple devices to a specific streamer one would simply create a split block with the following rule:

in_session_group('Apple')

In order to make more complex rules it’s possible to combine several checks like this in the same rule. Let’s extend the hypothetical ESB3024 Router above with a configured subnet with all IP addresses in Europe (named “Europe”). To make a rule that accepts any clients using an Apple device and living outside of Europe, but only as long as the reported load on the streamer (as indicated by the selection input variable “europe_load_mbps”) is less than 1000 megabits per second one could make an offload block with the following rule (without linebreaks):

in_session_group('Apple')
    and not in_subnet('Europe')
    and lt('europe_load_mbps', 1000)

In this example in_session_group('Apple') will be true if the client belongs to the session group named ‘Apple’. The function call in_subnet('Europe') is true if the client’s IP belongs to the subnet named ‘Europe’, but the word not in front of it reverses the value so the entire section ends up being false if the client is in Europe. Finally lt('europe_load_mbps', 1000) is true if there is a selection input variable named “europe_load_mbps” and its value is less than 1000.

Since the three parts are conjoined with the and keyword they must all be true for the entire rule to match. If the keyword or had been used instead it would have been enough for any of the parts to be true for the rule to match.

Example of Numeric Functions

A hypothetical CDN has two streamers with different capacity; Host_1 has roughly twice the capacity of Host_2. A simple random load balancing would put undue stress on the second host since it will receive as much traffic as the more capable Host_1.

This can be solved by using a weighted random distribution rule block with suitable rules for the two hosts:

{
    "targets": [
        {
            "target": "Host_1",
            "condition": "always()",
            "weight": "100"
        }
        {
            "target": "Host_2",
            "condition": "always()",
            "weight": "50"
        },
    ]
}

resulting in Host_1 receiving twice as many requests as Host_2 as its weight function is double that of Host_2.

If the CDN is capable of reporting the free capacity of the hosts, for example by writing to a selection input variable for each host, it’s easy to write a more intelligent load balancing rule by making the weights correspond to the amount of capacity left on each host:

{
    "targets": [
        {
            "target": "Host_1",
            "condition": "always()",
            "weight": "si('free_capacity_host_1')"
        }
        {
            "target": "Host_2",
            "condition": "always()",
            "weight": "si('free_capacity_host_2')"
        },
    ]
}

It is also possible to write custom Lua functions that return suitable weights, perhaps taking the host as an argument:

{
    "targets": [
        {
            "target": "Host_1",
            "condition": "always()",
            "weight": "intelligent_weight_function('Host_1')"
        }
        {
            "target": "Host_2",
            "condition": "always()",
            "weight": "intelligent_weight_function('Host_1')"
        },
    ]
}

These different weight rules can of course be combined in the same rule block, with one target having a hard coded number, another using a dynamically updated selection input variable and yet another having a custom-built function.

Due to limitations in the random number generator used to distribute requests, it’s better to use somewhat large values, around 100–1000 or so, than to use small values near 0.

Built-in Functions

The following built-in functions are available when writing rules:

in_session_group(str name): True if session belongs to session group <name>
in_all_session_groups(str sg_name, ...): True if session belongs to all specified session groups
in_any_session_group(str sg_name, ...): True if session belongs to any specified session group
in_subnet(str subnet_name): True if client IP belongs to the named subnet
gt(str si_var, number value): True if selection_inputs[si_var] > value
gt(str si_var1, str si_var2): True if selection_inputs[si_var1] > selection_inputs[si_var2]
ge(str si_var, number value): True if selection_inputs[si_var] >= value
ge(str si_var1, str si_var2): True if selection_inputs[si_var1] >= selection_inputs[si_var2]
lt(str si_var, number value): True if selection_inputs[si_var] < value
lt(str si_var1, str si_var2): True if selection_inputs[si_var1] < selection_inputs[si_var2]
le(str si_var, number value): True if selection_inputs[si_var] <= value
le(str si_var1, str si_var2): True if selection_inputs[si_var1] <= selection_inputs[si_var2]
eq(str si_var, number value): True if selection_inputs[si_var] == value
eq(str si_var1, str si_var2): True if selection_inputs[si_var1] == selection_inputs[si_var2]
neq(str si_var, number value): True if selection_inputs[si_var] != value
neq(str si_var1, str si_var2): True if selection_inputs[si_var1] != selection_inputs[si_var2]
si(str si_var): Returns the value of selection_inputs[si_var] if it is defined and non-negative, otherwise it returns 0.
always(): Returns true, useful when creating weighted rule blocks.
never(): Returns false, opposite of always().

These functions, as well as custom functions written in Lua and uploaded to the ESB3024 Router, can be combined to make suitably precise rules.

Combining Multiple Boolean Functions

In order to make the rule language easy to work with, it is fairly restricted and simple. One restriction is that it’s only possible to chain multiple function results together using either and or or, but not a combination of both conjunctions.

Statements joined with and or or keywords are evaluated one by one, starting with the left-most statement and moving right. As soon as the end result of the entire expression is certain, the evaluation ends. This means that evaluation ends with the first false statement for and expressions since a single false component means the entire expression must also be false. It also means that evaluation ends with the first true statement for or expressions since only one component must be true for the entire statement to be true as well. This is known as short-circuit or lazy evaluation.

Custom Functions

It is possible to write extremely complex Lua functions that take many parameters or calculations into consideration when evaluating an incoming client request. By writing such functions and making sure that they return only non-negative integer values and uploading them to the router they can be used from the rule language. Simply call them like any of the built-in functions listed above, using strings and numbers as arguments if necessary, and their result will be used to determine the routing path to use.

Formal Syntax

The full syntax of the language can be described in just a few lines of BNF grammar:

<rule>               := <weight_rule> | <match_rule> | <value_rule>
<weight_rule>        := "if" <compound_predicate> "then" <weight> "else" <weight>
<match_rule>         := <compound_predicate>
<value_rule>         := <weight>
<compound_predicate> := <logical_predicate> |
                        <logical_predicate> ["and" <logical_predicate> ...] |
                        <logical_predicate> ["or" <logical_predicate> ...] |
<logical_predicate>  := ["not"] <predicate>
<predicate>          := <function_name> "(" ")" |
                        <function_name> "(" <argument> ["," <argument> ...] ")"
<function_name>      := <letter> [<function_name_tail> ...]
<function_name_tail> := empty | <letter> | <digit> | "_"
<argument>           := <string> | <number>
<weight>             := integer | <predicate>
<number>             := float | integer
<string>             := "'" [<letter> | <digit> | <symbol> ...] "'"

Building a Routing Configuration

This example sets up an entire routing configuration for a system with a ESB3008 Request Router, two streamers and the Apple devices outside of Europe example used earlier in this document. Any clients not matching the criteria will be sent to an offload CDN with two streamers in a simple uniformly randomized load balancing setup.

Set up Session Group

First make a classifier and a session group that uses it:

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 9
  Adding a 'userAgent' element
    classifier : {
      name (default: ): Apple
      type (default: userAgent): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): ⏎
      pattern (default: ): *apple*
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "Apple",
      "type": "userAgent",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "*apple*"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.sessionGroups -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): Apple
    classifiers : [
      classifier (default: ): Apple
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "Apple",
      "classifiers": [
        "Apple"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

Set up Hosts

Create two host groups and add a Request Router to the first and two streamers to the second, which will be used for offload:

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: 3
  Adding a 'redirecting' element
    hostGroup : {
      name (default: ): internal
      type (default: redirecting): ⏎
      httpPort (default: 80): ⏎
      httpsPort (default: 443): ⏎
      hosts : [
        host : {
          name (default: ): rr1
          hostname (default: ): rr1.example.com
          ipv6_address (default: ): ⏎
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: y
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: 2
  Adding a 'host' element
    hostGroup : {
      name (default: ): external
      type (default: host): ⏎
      httpPort (default: 80): ⏎
      httpsPort (default: 443): ⏎
      hosts : [
        host : {
          name (default: ): offload-streamer1
          hostname (default: ): streamer1.example.com
          ipv6_address (default: ): ⏎
        }
        Add another 'host' element to array 'hosts'? [y/N]: y
        host : {
          name (default: ): offload-streamer2
          hostname (default: ): streamer2.example.com
          ipv6_address (default: ): ⏎
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: ⏎
]
Generated config:
{
  "hostGroups": [
    {
      "name": "internal",
      "type": "redirecting",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
        {
          "name": "rr1",
          "hostname": "rr1.example.com",
          "ipv6_address": ""
        }
      ]
    },
    {
      "name": "external",
      "type": "host",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
        {
          "name": "offload-streamer1",
          "hostname": "streamer1.example.com",
          "ipv6_address": ""
        },
        {
          "name": "offload-streamer2",
          "hostname": "streamer2.example.com",
          "ipv6_address": ""
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

Create Load Balancing and Offload Block

Add both offload streamers as targets in a randomgroup block:

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: deny
    2: contentPopularity
    3: firstMatch
    4: random
    5: rawGroup
    6: rawHost
    7: split
    8: weighted
    9: allow
  Choose element index or name: random
  Adding a 'random' element
    rule : {
      name (default: ): balancer
      type (default: random): ⏎
      targets : [
        target (default: ): offload-streamer1
        Add another 'target' element to array 'targets'? [y/N]: y
        target (default: ): offload-streamer2
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "balancer",
      "type": "random",
      "targets": [
        "offload-streamer1",
        "offload-streamer2"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

Then create a split block with the request router and the load balanced CDN as targets:

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: deny
    2: contentPopularity
    3: firstMatch
    4: random
    5: rawGroup
    6: rawHost
    7: split
    8: weighted
    9: allow
  Choose element index or name: split
  Adding a 'split' element
    rule : {
      name (default: ): offload
      type (default: split): ⏎
      rule (default: ): in_session_group('Apple') and not in_subnet('Europe') and lt('europe_load_mbps', 1000)
      onMatch (default: ): rr1
      onMiss (default: ): balancer
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "offload",
      "type": "split",
      "condition": "in_session_group('Apple') and not in_subnet('Europe') and lt('europe_load_mbps', 1000)",
      "onMatch": "rr1",
      "onMiss": "balancer"
    }
  ]
}
Merge and apply the config? [y/n]: y

The last step required is to set the entrypoint of the routing tree so the router knows where to start evaluating:

$ confcli services.routing.entrypoint offload
services.routing.entrypoint = 'offload'

Evaluate

Now that all the rules have been set up properly and the router has been reconfigured. The translated configuration can be read from the router’s configuration API:

$ curl -k https://router-host:5001/v2/configuration  2> /dev/null | jq .routing
{
  "id": "offload",
  "member_order": "sequential",
  "members": [
    {
      "host_id": "rr1",
      "id": "offload.rr1",
      "weight_function": "return ((in_session_group('Apple') ~= 0) and
                          (in_subnet('Europe') == 0) and
                          (lt('europe_load_mbps', 1000) ~= 0) and 1) or 0 "
    },
    {
      "id": "offload.balancer",
      "member_order": "weighted",
      "members": [
        {
          "host_id": "offload-streamer1",
          "id": "offload.balancer.offload-streamer1",
          "weight_function": "return 100"
        },
        {
          "host_id": "offload-streamer2",
          "id": "offload.balancer.offload-streamer2",
          "weight_function": "return 100"
        }
      ],
      "weight_function": "return 1"
    }
  ],
  "weight_function": "return 100"
}

Note that the configuration language code has been translated into its Lua equivalent.

1.1.5.4 - Session Groups and Classification

How to classify clients into session groups and use them in routing

ESB3024 Router provides a flexible classification engine, allowing the assignment of clients into session groups that can then be used to base routing decisions on.

Session Groups

A session group is a subset of clients satisfying certain conditions. Define session groups reflecting the environment and objectives.

A client can be part of multiple session groups.

Classification

The session_groups field of the configuration comprises a list of sessions groups, each with their respective classification criteria specified in the corresponding classifiers field.

Each session group consists of the fields

id (number) - ID of the session group, must be globally unique from all other session group IDs
name (string) - name of the session group
classifiers (array of array of objects) - list of the session group’s classifiers.

Classification Logic

Session groups support combining classifiers using AND and OR semantics. This is achieved by nesting classifiers within evaluation groups, as seen in the pseudo-JSON below:

{
  "session_groups": [
    {
      "id": 1,
      "name": "example_AND_vs_OR",
      "classifiers": [
        [ // All classifiers within evaluation group are AND evaluated
          {
            // classifier 1
          },
          {
            // classifier 2
          }
        ],
        // Separate evaluation groups are OR evaluated
        [
          {
            // classifier 3
          }
        ]
      ]
    }
  ]
}

This particular session group defines an evaluation group containing classifier 1 and classifier 2. Both classifiers have to evaluate to true for the whole classifier group to evaluate to true.

This session group also defines another evaluation group containing classifier 3. Separate evaluation groups are OR evaluated, meaning that either the first or second evaluation group needs to evaluate to true for a client to assigned to the session group.

Logically, this session group’s classification criterion can be visualized as
( classifier 1 AND classifier 2 ) OR classifier 3.

Classifier Structure

Each classifier consists of the fields

id (number): ID of the classifier, must be unique within the same session group
name (string): name of the classifier
inverted (boolean): the resulting classification should be inverted (negated) before evaluation
rule (object): the classifier logic

rule contains different fields defining classifier behaviour. The first mandatory field is rule_type which defines the classification method and can be one of string_match_rule, geoip_rule, asn_ids_rule, regex_rule, subnet_rule and ip_ranges_rule. The second mandatory field is source, defining the data source on which the classification will be evaluated and can be one of session/content_url_path, session/content_url_query_params, session/user_agent, session/client_ip or session/hostname.

Examples of all rule types and sources can be seen in the JSON below.

{
  "session_groups": [
    {
      "id": 1,
      "name": "example_string_match_rule_live_and_subtitle",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "live_wildcard_match",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/content_url_path",
              "pattern": "*live*"
            }
          },
          {
            "id": 2,
            "inverted": false,
            "name": "subtitle_wildcard_match",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/content_url_query_params",
              "pattern": "*subtitle=eng*"
            }
          }
        ]
      ]
    },
    {
      "id": 2,
      "name": "example_string_match_rule_vod_or_hostname",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "live_wildcard_match",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/content_url_path",
              "pattern": "*vod*"
            }
          }
        ],
        [
          {
            "id": 3,
            "inverted": false,
            "name": "hostname_match",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/hostname",
              "pattern": "mycdn.com"
            }
          }
        ]
      ]
    },
    {
      "id": 3,
      "name": "example_string_match_rule_ipv4",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "ipv4_wildcard_match",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/client_ip",
              "pattern": "*.*"
            }
          }
        ]
      ]
    },
    {
      "id": 4,
      "name": "example_string_match_rule_ipv6",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "ipv6_wildcard_match",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/client_ip",
              "pattern": "*:*"
            }
          }
        ]
      ]
    },
    {
      "id": 5,
      "name": "example_geo_location",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "geo_location_specific",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "continent": "Europe",
              "country": "Norway",
              "region": "Innlandet",
              "cities": ["Elverum"],
              "geoname_id": 3144096,
              "asn": "Telenor Norge AS"
            }
          }
        ]
      ]
    },
    {
      "id": 6,
      "name": "example_geo_location_asn_wildcard",
      "classifiers": [
        [
          {
          "id": 1,
          "inverted": false,
          "name": "asn_wildcard_match",
          "rule": {
            "rule_type": "geoip_rule",
            "source": "session/client_ip",
            "asn": "Telia*"
            }
          }
        ]
      ]
    },
    {
      "id": 7,
      "name": "example_regex",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "ios_version_5.1.1",
            "rule": {
              "rule_type": "regex_rule",
              "source": "session/user_agent",
              "pattern": "/OS ((\\d+_?){2,3})\\s/"
            }
          }
        ]
      ]
    },
    {
      "id": 8,
      "name": "example_ip_ranges",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": true,
            "name": "offload_no_peering_subnet",
            "rule": {
              "rule_type": "ip_ranges_rule",
              "source": "session/client_ip",
              "ip_ranges": ["158.174.0.0/16", "95.192.0.0/12"]
            }
          }
        ]
      ]
    },
    {
      "id": 9,
      "name": "example_subnets",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "subnets_match",
            "rule": {
              "rule_type": "subnet_rule",
              "source": "session/client_ip",
              "pattern": "stock*"
            }
          }
        ]
      ]
    },
    {
      "id": 10,
      "name": "example_asn_list",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "asn_list_match",
            "rule": {
              "rule_type": "asn_ids_rule",
              "source": "session/client_ip",
              "asn_ids": [1, 2, 3]
            }
          }
        ]
      ]
    }
  ]
}

As seen in "pattern": "/OS ((\\d+_?){2,3})\\s/", backslash characters must always be escaped in JSON. If a tool was used to construct the JSON code, escaping should be done automatically and it must not be included it manually.

The rule types work as following:

string_match_rule flexible string matching on any source
Tip: Supports wildcard matching
regex_rule - regex matching on the source, based on C++ regex implementation
Tip: For basic IPv4 vs. IPv6 differentiation of client IP addresses, a simple string_match_rule on session/client_ip is more efficient and maintainable than a more expressive regex_rule (which may come with the additional benefit of validating the format of the matched field).
ip_ranges_rule - a list of IP ranges on the form ‘0.0.0.0/24’. Any client with an IP within any of the specified ranges will be matched by the classifier
subnet_rule - string matching of the pattern on all subnets the client’s IP matches. Only supports the source session/client_ip. If a client matches into multiple subnets, only one of them needs to match the pattern.
Tip: Supports wildcard matching
geoip_rule allows geolocation based matching. The client’s IP address is used to poll the MaxMind GeoIP2 database for geolocations. Only supports the source session/client_ip. The possible fields within this rule are
- continent (string) - the desired continent
- country (string) - the desired country
- region (string) - the desired region
- cities (array of strings) - the desired cities
  - at least one city must match for successful classification
- asn (string) - the desired ASN
  Tip: Supports wildcard matching
- geoname_id (number) - MaxMind’s internal id matching
  Note: Using geoname_id requires insight into a MaxMind GeoIP2 database to find correct IDs

All fields within the GeoIP rule are optional. For successful classification, all defined fields must match.

If a client is successfully classified, it will labeled as part of that session group. The session group can be then be accessed in Lua contexts, e.g.

{
  "hosts": [
    {
      "id": "geo-location-host",
      "cdn_id": "basic-cdn",
      "host": "geo-location-host.example"
    },
    {
      "id": "ip-ranges-host",
      "cdn_id": "basic-cdn",
      "host": "ip-ranges-host.example"
    }
  ],
  "routing": {
    "id": "session_group_routing",
    "member_order": "sequential",
    "members": [
      {
        "id": "geo_location_session_group",
        "host_id": "geo-location-host",
        "weight_function": "return session_groups.example_geo_location and 1 or 0"
      },
      {
        "id": "ip_ranges_session_group",
        "host_id": "ip-ranges-host",
        "weight_function": "return session_groups.example_ip_ranges and 1 or 0"
      }
    ]
  }
}

1.1.5.5 - Advanced features

Detailed descriptions and examples of advanced features within ESB3024

1.1.5.5.1 - Content popularity

How to tune content popularity parameters and use it in routing

ESB3024 Router allows routing decisions based on content popularity. All incoming content requests are tracked to continuously update a content popularity ranking list. The popularity ranking algorithm is designed to let popular content quickly rise to the top while unpopular content decays and sinks towards the bottom.

Configuration

All configuration parameters for content popularity reside in the settings object of the configuration, an example of which can be seen below:

{
  "settings": {
    "content_popularity": {
      "algorithm": "scored_based",
      "session_group_names": ["vod_only"],
      "score_based:": {
        "requests_between_popularity_decay": 1000,
        "popularity_list_max_size": 100000,
        "popularity_prediction_factor": 2.5,
        "popularity_decay_fraction": 0.2
      },
      "time_based": {
        "intervals_per_hour": 10
      }
    }
  }
}

The field algorithm dictates which content popularity tracking algorithm to use, can either be score_based or time_based.

The field session_group_names defines the sessions for which content popularity should be tracked. In the example above, session belonging to the vod_only session group will be tracked for content popularity. If left empty, content popularity will be tracked for all sessions.

The remaining configuration parameters are algorithm specific.

Score based algorithm

The field popularity_list_max_size defines the maximum amount of unique contents to track for popularity. This can be used to limit memory growth. A single entry in the popularity ranking list will at most consume 180 B of memory, giving an upper bound memory growth of \(180n_{\text{requests}}\) bytes. E.g. using "popularity_list_max_size": 1000 would consume at most 180⋅1,000 = 180,000 B = 0.18 MB. If the content popularity list is full, a request to unique content would replace the least popular content.

Setting a very high max size won’t impact performance, it will only consume more memory.

The field requests_between_popularity_decay defines the number of requests between each popularity decay update, an integral component of this feature.

The fields popularity_prediction_factor and popularity_decay_fraction tune the behaviour of the content popularity ranking algorithm, explained further below.

Decay update

To allow for popular content to quickly rise in popularity and unpopular content to sink, a dynamic popularity ranking algorithm is used. The goal of the algorithm is to track content popularity in real time, allowing routing decisions based on the requested content’s popularity. The algorithm is applied every decay update.

The algorithm uses current trending content to predict content popularity. The field popularity_prediction_factor regulates how much the algorithm should rely on predicted popularity. A high prediction factor allows rising content to quickly rise to high popularity but can also cause unpopular content with a sudden burst of requests to wrongfully rise to the top. A low prediction factor can cause stagnation in the popularity ranking, not allowing new popular content to rise to the top.

Unpopular content decays in popularity, the magnitude of which is regulated by popularity_decay_fraction. A high value will aggressively decay content popularity every decay update while a low value will bloat the ranking, causing stagnation. Once content decays to a trivially low popularity score, it is pruned from the content popularity list.

When configuring these tuning parameters, the most crucial data to consider is the size of your asset catalog, i.e. the number of unique contents you offer. The recommended values, obtained through testing, are presented in the table below. Note that the field popularity_prediction_factor is the principal factor in controlling the algorithm’s behaviour.

Catalog size \(n\)	`popularity_prediction_factor`	`popularity_decay_fraction`
\(n\) < 1000	2.2	0.2
1000 < \(n\) < 5000	2.3	0.2
5000 < \(n\) < 10000	2.5	0.2
\(n\) > 10000	2.6	0.2

Time based algorithm

The time based algorithm only requires the configuration parameter intervals_per_hour. E.g., the value "intervals_per_hour": 10 would give 10 six minute intervals per hour. During each interval, all unique content requests has an associated counter, increasing by one for each incoming request. After an hour, all intervals have been cycled through. The counters in the first interval will be reset and all incoming content requests will increase the counters in the first interval again. This cycle continues forever.

When determining a single content’s popularity, the sum of each content’s counter in all intervals is used to determine a popularity ranking.

Usage in routing

Content popularity ranking is available in Lua through the simple member session.content_global_popularity, returning the requested content’s popularity ranking. E.g., the routing configuration below will route all requests to content with a popularity ranking lower than 10, i.e. top 10 most popular content, to the host edge-streamer while requests to content outside of the top 10 most popular will be routed to the host offload.

"routing": {
  "id": "routing_table",
  "member_order": "sequential",
  "members": [
    {
      "id": "edge",
      "weight_function": "return session.content_global_popularity < 11 and 1 or 0",
      "host_id": "edge-streamer"
    },
    {
      "id": "offload",
      "weight_function": "return 1",
      "host_id": "offload"
    }
  ]
}

1.1.5.5.2 - Consistent Hashing

Details and configuration considerations for using consistent hashing based routing

Consistent hashing based routing is a feature that can be used to distribute requests to a set of hosts in a cache friendly manner. By using Agile Content’s consistent distributed hash algorithm, the amount of cache redistribution is minimized within a set of hosts. Requests for a content will always be routed to the same set of hosts, the amount of which is configured by the spread factor, allowing high cache usage. When adding or removing hosts, the algorithm minimizes cache redistribution.

Say you have the host group [s1, s2, s3, s4, s5] and have configured spreadFactor = 3. A request for a content asset1 would then be routed to the same three hosts with one of them being selected randomly for each request. Requests for a different content asset2 would also be routed to one of three different hosts, most likely a different combination of hosts than requests for content asset1.

Example routing results with spreadFactor = 3:

Request for asset1 → route to one of [s1, s3, s4].
Request for asset2 → route to one of [s2, s4, s5].
Request for asset3 → route to one of [s1, s2, s5].

Since consistent hashing based routing ensures that requests for a specific content always get routed to the same set of hosts, the risk of cache misses are lowered on the hosts since they will be served the same content requests over and over again.

Note that the maximum value of spreadFactor is 64. Consequently, the highest amount of hosts you can use in a consistentHashing rule block is 64.

Configuration

Configuring consistent hashing based routing is easily done using confcli. Let’s configure the example described above:

confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: consistentHashing
  Adding a 'consistentHashing' element
    rule : {
      name (default: ): consistentHashingRule 
      type (default: consistentHashing): 
      spreadFactor (default: 1): 3
      targets : [
        target : {
          target (default: ): s1
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): s2
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): s3
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): s4
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): s5
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "consistentHashingRule",
      "type": "consistentHashing",
      "spreadFactor": 3,
      "targets": [
        {
          "target": "s1",
          "enabled": true
        },
        {
          "target": "s2",
          "enabled": true
        },
        {
          "target": "s3",
          "enabled": true
        },
        {
          "target": "s4",
          "enabled": true
        },
        {
          "target": "s5",
          "enabled": true
        }
      ]
    }
  ]
}

Adding hosts

Adding a host to the list will give an additional target for the consistent hashing algorithm to route requests to. This will shift content distribution onto the new host.

confcli services.routing.rules.consistentHashingRule.targets -w
Running wizard for resource 'targets'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

targets : [
  target : {
    target (default: ): s6
    enabled (default: True): 
  }
  Add another 'target' element to array 'targets'? [y/N]: n
]
Generated config:
{
  "targets": [
    {
      "target": "s6",
      "enabled": true
    }
  ]
}
Merge and apply the config? [y/n]: y

Removing hosts

There is one very important caveat of using a consistent hashing rule block. As long as you don’t modify the list of hosts, the consistent hashing algorithm will keep routing requests to the same hosts. However, if you remove a host from the block in any position except the last, the consistent hashing algorithm’s behaviour will change and the algorithm cannot maintain a minimum amount of cache redistribution.

If you’re in a situation where you have to remove a host from the routing targets but want to keep the same consistent hashing behaviour, e.g. during very high load, you’ll have to toggle that target’s enabled field to false. E.g., disabling requests to s2 can be accomplished by:

$ confcli services.routing.rules.consistentHashingRule.targets.1.enabled false
services.routing.rules.consistentHashingRule.targets.1.enabled = False
$ confcli services.routing.rules.consistentHashingRule.targets.1
{
    "1": {
        "target": "s2",
        "enabled": false
    }
}

If you modify the list order or remove hosts, it is highly recommended to do so during moments where a higher rate of cache misses are acceptable.

1.1.5.5.3 - Predictive load balancing

Configuring predictive load balancing in ESB3024 Router

Why use predictive load balancing?

Predictive load balancing is a tool that can be used to avoid overloading hosts with traffic. Consider the case where a popular event starts at a certain time, let’s say 12 PM. A spike in traffic will be routed to the hosts that are streaming the content at 12 PM, most of them starting at low bitrates. A host might have sufficient bandwidth left to take on more clients but when the recently connected clients start ramping up in video quality and increase their bitrate, the host can quickly become overloaded, possibly dropping incoming requests or going offline. Predictive load balancing solves this issue by considering how many times a host recently been redirected to.

Basic configuration

The router allows predicting incoming load for individual hosts. Through the use of the selection input API, the routing engine can predict incoming load by tracking recent host selections and make routing decisions accordingly.

What is considered recent is configurable and defaults to 500 milliseconds:

$ confcli services.routing.tuning.target.recentDurationMilliseconds
{
    "recentDurationMilliseconds": 500
}
$ confcli services.routing.tuning.target.recentDurationMilliseconds 1000
services.routing.tuning.target.recentDurationMilliseconds = 1000

{
  "tuning": {
    "target_recent_duration_milliseconds": 500
  }
}

How to use predictive load balancing

The standard Lua library provides four functions for predictive load balancing that can be used when constructing conditions/weight functions: host_bitrate() , host_bitrate_custom(), host_has_bw() and host_has_bw_custom(). All require data to be supplied to the selection input API and apply only to leaf nodes in the routing tree. In order for predictive load balancing to work properly the data must be updated at regular intervals. The data needs to be supplied by the target system.

These functions are suitable to used as host health checks. To configure host health checks, see configuring CDNs and hosts.

Note that host_bitrate() and host_has_bw() rely on data supplied by metrics agents, detailed in Cache hardware metrics: monitoring and routing.

host_bitrate_custom() and host_has_bw_custom() rely on manually supplied selection input data, detailed in selection input API. The bitrate unit depends on the data submitted to the selection input API.

`host_bitrate()`

host_bitrate() returns the predicted bitrate (in megabits per second) of the host after the recently connected clients start ramping up in streaming quality. The function accepts up to four arguments:

interface: The name of the interface to use for bitrate prediction.
Optional avg_bitrate: the average bitrate per client, defaults to 6 megabits per second.
Optional num_routers: the number of routers that can route to this host, defaults to 1. This is important to accurately predict the incoming load if multiple routers are used.
Optional host: The name of the host to use for bitrate prediction. Defaults to the current host if not provided.

Examples of usage:

host_bitrate({interface="eths0"})
host_bitrate({avg_bitrate=1, interface="eths0"})
host_bitrate({num_routers=2, interface="eths0"})
host_bitrate({avg_bitrate=1, num_routers=4, interface="eths0"})
host_bitrate({avg_bitrate=1, num_routers=4, host="custom_host", interface="eths0"})

host_bitrate() calculates the predicted bitrate as:

predicted_host_bitrate = current_host_bitrate + (recent_connections * avg_bitrate * num_routers)

`host_bitrate_custom()`

Same functionality as host_bitrate() but uses a custom selection input variable as bitrate input instead of accessing hardware metrics. The function accepts up to three arguments:

custom_bitrate_var: The name of the selection input variable to be used for accessing current host bitrate.
Optional avg_bitrate: see host_bitrate() documentation above.
Optional num_routers: see host_bitrate() documentation above.

host_bitrate_custom({custom_bitrate_var="host1_current_bitrate"})
host_bitrate_custom({avg_bitrate=1, custom_bitrate_var="host1_current_bitrate"})
host_bitrate_custom({num_routers=4, custom_bitrate_var="host1_current_bitrate"})

`host_has_bw()`

Instead of accessing the predicted bitrate of a host through host_bitrate(), host_has_bw() returns 1 if the host is predicted to have enough bandwidth left to take on more clients after recent connections ramp up in bitrate, otherwise it returns 0. The function accepts up to five arguments:

interface: see host_bitrate() documentation above.
Optional avg_bitrate: see host_bitrate() documentation above.
Optional num_routers: see host_bitrate() documentation above.
Optional host: see host_bitrate() documentation above.
Optional margin: the bitrate (megabits per second) headroom that should be taken into account during calculation, defaults to 0.

host_has_bw() returns whether or not the following statement is true:

predicted_host_bitrate + margin < host_bitrate_capacity

Examples of usage:

host_has_bw({interface="eths0"})
host_has_bw({margin=10, interface="eth0"})
host_has_bw({avg_bitrate=1, interface="eth0"})
host_has_bw({num_routers=4, interface="eth0"})
host_has_bw({host="custom_host", interface="eth0"})

`host_has_bw_custom()`

Same functionality as host_has_bw() but uses a custom selection input variable as bitrate and capacity input. The function accepts up to five arguments:

custom_capacity_var: the name of the selection input variable to be used for accessing host capacity.
custom_bitrate_var: see host_bitrate_custom() documentation
Optional margin: see host_has_bw() documentation above. above.
Optional avg_bitrate: see host_bitrate() documentation above.
Optional num_routers: see host_bitrate() documentation above.

Examples of usage:

host_has_bw_custom({custom_capacity_var="host1_capacity", custom_bitrate_var="host1_current_bitrate"})
host_has_bw_custom({margin=10, custom_capacity_var="host1_capacity", custom_bitrate_var="host1_current_bitrate"})
host_has_bw_custom({avg_bitrate=1, custom_capacity_var="host1_capacity", custom_bitrate_var="host1_current_bitrate"})
host_has_bw_custom({num_routers=4, custom_capacity_var="host1_capacity", custom_bitrate_var="host1_current_bitrate"})

Examples in routing

When using confcli it’s not recommended to use host_bitrate() since it’s not possible to make comparisons using <, > or similar operators. Using host_has_bw() makes it easy to construct conditions utilizing predictive load balancing.

host_has_bw() and host_has_bw_custom() are excellent candidates to be used as health check functions for hosts, see configuring CDNs and hosts for more details.

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: weighted
  Adding a 'weighted' element
    rule : {
      name (default: ): weighted_and_has_bw
      type (default: weighted):
      targets : [
        target : {
          target (default: ): host1
          weight (default: 100):
          condition (default: always()): host_has_bw({interface="eths0"})
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): host2
          weight (default: 100): 100
          condition (default: always()): host_has_bw({margin=1000, interface="eths0"})
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): host3
          weight (default: 100):
          condition (default: always()): host_has_bw({avg_bitrate=1, num_routers=2, margin=5000, interface="eths0"})
        }
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "weighted_and_has_bw",
      "type": "weighted",
      "targets": [
        {
          "target": "host1",
          "weight": "100",
          "condition": "host_has_bw({interface="eths0"})"
        },
        {
          "target": "host2",
          "weight": "100",
          "condition": "host_has_bw({margin=1000, interface="eths0"})"
        },
        {
          "target": "host3",
          "weight": "100",
          "condition": "host_has_bw({avg_bitrate=1, num_routers=2, margin=5000, interface="eths0"})"
        }
      ]
    }
  ]
}

"routing": {
  "id": "weighted_and_has_bw",
  "member_order": "sorted",
  "members": [
    {
      "id": "1",
      "host_id": "host1",
      "weight_function": "return host_bitrate({interface="eths0"}) > 0 and 1 or 0"
    },
    {
      "id": "2",
      "host_id": "host2",
      "weight_function": "return host_bitrate({avg_bitrate=1, interface="eths0"}) > 10000 and 1 or 0"
    },
    {
      "id": "3",
      "host_id": "host3",
      "weight_function": "return host_has_bw({avg_bitrate=1, num_routers=2, margin=5000, interface="eths0"}) and 1 or 0"
    }
  ],
  "weight_function": "return 1"
}

1.1.5.5.4 - Security token verification

Only allow requests that contain a correct security token

The security token verification feature allows for ESB3024 Router to only process requests that contain a correct security token. The token is generated by the client, for example in the portal, using an algorithm that it shares with the router. The router verifies the token and rejects the request if the token is incorrect.

It is beyond the scope of this document to describe how the token is generated, that is described in the Security Tokens application note that is installed with the ESB3024 Router’s extra documentation.

Setting up a routing rule

The token verification is performed by calling the verify_security_token() function from a routing rule. The function returns 1 if the token is correct, otherwise it returns 0. It should typically be called from the first routing rule, to make requests with bad tokens fail as early as possible.

The confcli example assumes that the router already has rules configured, with an entry point named select_cdn. Token verification is enabled by inserting an “allow” rule first in the rule list.

confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: allow
  Adding a 'allow' element
    rule : {
      name (default: ): token_verification
      type (default: allow):
      condition (default: always()): verify_security_token()
      onMatch (default: ): select_cdn
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "token_verification",
      "type": "allow",
      "condition": "verify_security_token()",
      "onMatch": "select_cdn"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.entrypoint token_verification
services.routing.entrypoint = 'token_verification'

"routing": {
  "id": "token_verification",
  "member_order": "sequential",
  "members": [
    {
      "id": "token_verification.0.select_cdn",
      "member_order": "weighted",
      "members": [
        ...
      ],
      "weight_function": "return verify_security_token() ~= 0"
    },
    {
      "id": "token_verification.1.rejected",
      "member_order": "sequential",
      "members": [],
      "weight_function": "return 1"
    }
  ],
  "weight_function": "return 100"
},

Configuring security token options

The secret parameter is not part of the router request, but needs to be configured separately in the router. That can be done with the host-config tool that is installed with the router.

Besides configuring the secret, host-config can also configure floating sessions and a URL prefix. Floating sessions are sessions that are not tied to a specific IP address. When that is enabled, the token verification will not take the IP address into account when verifying the token.

The security token verification is configured per host, where a host is the name of the host that the request was sent to. This makes it possible for a router to support multiple customer accounts, each with their own secret. If no configuration is found for a host, a configuration with the name default is used.

host-config supports three commands: print, set and delete.

Print

The print command prints the current configuration for a host. The following parameters are supported:

host-config print [-n <host-name>]

By default it prints the configuration for all hosts, but if the optional -n flag is given it will print the configuration for a single host.

Set

The set command sets the configuration for a host. The configuration is given as command line parameters. The following parameters are supported:

host-config set
    -n <host-name>
    [-f floating]
    [-p url-prefix]
    [-r <secret-to-remove>]
    [-s <secret-to-add>]

-n <host-name> - The name of the host to configure.
-f floating - A boolean option that specifies if floating sessions are accepted. The parameter accepts the values true and false.
-p url-prefix - A URL prefix that is used for identifying requests that come from a certain account. This is not used when verifying tokens.
-r <secret-to-remove> - A secret that should be removed from the list of secrets.
-s <secret-to-add> - A secret that should be added to the list of secrets.

For example, to set the secret “secret-1” and enable floating sessions for the default host, the following command can be used:

host-config set -n default -s secret-1 -f true

The set command only touches the configuration options that are mentioned on the command line, so the following command line will add a second secret to the default host without changing the floating session setting:

host-config set -n default -s secret-2

It is possible to set multiple secrets per host. This is useful when updating a secret, then both the old and the new secret can be valid during the transition period. After the transition period the old secret can be removed by typing:

host-config set -n default -r secret-1

Delete

The delete command deletes the configuration for a host. It supports the following parameters:

host-config delete -n <host-name>

For example, to delete the configuration for example.com, the following command can be used:

host-config delete -n example.com

Global options

host-config also has a few global options. They are:

-k <security-key> - The security key that is used when communicating with the router. This is normally retrieved automatically.
-h - Print a help message and exit.
-r <router> - The router to connect to. This default to localhost, but can be changed to connect to a remote router.
-v - Verbose output, can be given multiple times.

Debugging security token verification

The security token verification only logs messages when the log level is set to 4 or higher. Then it will only log some errors. It is possible to enable more verbose logging using the security-token-config that is installed together with the router.

When verbose logging is enabled, the router will log information about the token verification, including the configured token secrets, so it needs to be used with care.

The logged lines are prefixed with verify_security_token.

The security-token-config tool supports the commands print and set.

Print

The print command prints the current configuration. If nothing is configured it will not print anything.

Set

The set command sets the configuration. The following parameters are supported:

security-token-config set
    [-d <enabled>]

-d <enabled> - A boolean option that specifies if debug logging should be enabled or not. The parameter accepts the values true and false.

1.1.5.5.5 - Subnets API

How to match clients into named subnets and use them in routing

ESB3024 Router provides utilities to quickly match clients into subnets. Any combination of IPv4 and IPv6 addresses can be used. To begin, a JSON file is needed, defining all subnets, e.g:

{
  "255.255.255.255/24": "area1",
  "255.255.255.255/16": "area2",
  "255.255.255.255/8": "area3",
  "90.90.1.3/16": "area4",
  "5.5.0.4/8": "area5",
  "2a02:2e02:9bc0::/48": "area6",
  "2a02:2e02:9bc0::/32": "area7",
  "2a02:2e02:9bc0::/16": "area8",
  "2a02:2e02:9de0::/44": "combined_area",
  "2a02:2e02:ada0::/44": "combined_area"
}

and PUT it to the endpoint :5001/v1/subnets or :5001/v2/subnets, the API version doesn’t matter for subnets:

curl -k -T subnets.json -H "Content-Type: application/json" https://router-host:5001/v1/subnets

Note that it is possible for several subnet CIDR strings to share the same label, effectively grouping them together.

This will load the subnets into the router which will then be usable within any Lua functions, accessed from the request and session tables respectively, e.g.

{
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "node1",
        "host_id": "host1",
        "weight_function": "return request.subnet == 'area1' and 1 or 0"
      }
    ]
  }
}

Subnet matching will use the client IP fetched from the request or session tables. When possible, the subnet with the longest matching subnet mask will be chosen, i.e. 255.255.255.255/28will be chosen over 255.255.255.255/24. If no matching subnets are found, the boolean value false will be returned.

NOTE: To use booleans in strings in Lua, calling tostring(bool) is required. Therefore, since subnet lookup might return false, the subnet call should be wrapped as: tostring(session.subnet) or tostring(requst.subnet).

Invalid ip-addresses will be omitted during subnet list construction accompanied by a message in the log displaying the invalid IP address.

1.1.5.5.6 - Lua Features

Detailed descriptions and examples of Lua features offered by ESB3024 Router.

1.1.5.5.6.1 - Health Check Functions

Built in Lua health check functions

This section details built in Lua functions that are meant to be used for host health checks. Note that these functions rely on data supplied by metric agents detailed in Cache hardware metrics: monitoring and routing. Make sure cache hardware metrics are supplied to the router before using any of these functions.

`cpu_load_ok()`

Parameters

The function accepts an optional argument table with the following keys:

Optional hostname: The name of the host. Defaults to the hostname of the selected host if not provided.
Optional cpu_load5_limit: The acceptable limit for the 5-minute CPU load. Defaults to 0.9 if not provided.

Returns

The function returns 1 if the five minute CPU load average is below their respective limits, and 0 otherwise.

Usage examples

cpu_load_ok()
cpu_load_ok({hostname = "custom_host"})
cpu_load_ok({cpu_load5_limit = 0.8})
cpu_load_ok({hostname = "custom_host", cpu_load5_limit = 0.8})

`memory_usage_ok()`

Parameters

The function accepts an optional argument table with the following keys:

Optional hostname: The name of the host. Defaults to the hostname of the selected host if not provided.
Optional memory_usage_limit: The acceptable limit for the memory usage. Defaults to 0.9 if not provided.

Returns

The function returns 1 if the memory usage is below the limit, and 0 otherwise.

Usage examples

memory_usage_ok()
memory_usage_ok({hostname = "custom_host"})
memory_usage_ok({memory_usage_limit = 0.7})
memory_usage_ok({hostname = "custom_host", memory_usage_limit = 0.7})

`interfaces_online()`

Parameters

The function accepts an argument table with the following keys:

Required interfaces: A string or a table of strings representing the network interfaces to check.
Optional hostname: The name of the host. Defaults to the hostname of the selected host if not provided.

Returns

The function returns 1 if all the specified interfaces are online, and 0 otherwise.

Usage examples

interfaces_online({interfaces = "eth0"})
interfaces_online({interfaces = {"eth0", "eth1"}})
interfaces_online({hostname = "custom_host", interfaces = "eth0"})
interfaces_online({hostname = "custom_host", interfaces = {"eth0", "eth1"}})

`health_check()`

Parameters

The function accepts an optional argument table with the following keys:

Required interfaces: A string or a table of strings representing the network interfaces to check.
Optional hostname: The name of the host. Defaults to the hostname of the selected host if not provided.
Optional cpu_load5_limit: The acceptable limit for the 5-minute CPU load. Defaults to 0.9 if not provided.
Optional memory_usage_limit: The acceptable limit for the memory usage. Defaults to 0.9 if not provided.

Returns

The function returns 1 if all the specified interfaces are online, and 0 otherwise.

Usage examples

health_check({interfaces = "eths0"})
health_check({hostname = "custom_host", interfaces = "eths0"})
health_check({cpu_load5_limit = 0.7, memory_usage_limit = 0.8, interfaces = "eth0"})
health_check({hostname = "custom_host", cpu_load5_limit = 0.7, memory_usage_limit = 0.8, interfaces = {"eth0", "eth1"}})

1.1.5.5.6.2 - Request Translation Function

Instructions for how to write a function to modify incoming requests before routing decisions are being made.

Specifies the body of a Lua function that inspects every incoming HTTP request and overwrites individual fields before further processing by the router.

Returns nil when nothing is to be changed, or HTTPRequest(t) where t is a table with any of the following optional fields:

Method
- Description: Replaces the HTTP request method in the request being processed.
- Type: string
- Example: 'GET', 'POST'
Path
- Description: Replaces the request path in the request being processed.
- Type: string
- Example: '/mycontent/superman.m3u8'
ClientIp
- Description: Replaces client IP address in the request being processed.
- Type: string
- Example: '172.16.238.128'
Body
- Description: Replaces body in the request being processed.
- Type: string or nil
- Example: '{"foo": "bar"}'
QueryParameters
- Description: Adds, removes or replaces individual query parameters in the request being processed.
- Type: nested table (indexed by number) representing an array of query parameters as {[1]='Name',[2]='Value'} pairs that are added to the request being processed, or overwriting existing query parameters with colliding names. To remove a query parameter from the request, specify nil as value, i.e. QueryParameters={..., {[1]='foo',[2]=nil} ...}. Returning a query parameter with a name but no value, such as a in the request '/index.m3u8?a&b=22' is currently not supported.
Headers
- Description: Adds, removes or replaces individual headers in the request being processed.
- Type: nested table (indexed by number) representing an array of request headers as {[1]='Name',[2]='Value'} pairs that are added to the request being processed, or overwriting existing request headers with colliding names. To remove a header from the request, specify nil as value, i.e. Headers={..., {[1]='foo',[2]=nil} ...}. Duplicate names are supported. A multi-value header such as Foo: bar1,bar2 is defined by specifying Headers={..., {[1]='foo',[2]='bar1'}, {[1]='foo',[2]='bar2'}, ...}.

Example of a request_translation_function body that sets the request path to a hardcoded value and adds the hardcoded query parameter a=b:

-- Statements go here
print('Setting hardcoded Path and QueryParameters')
return HTTPRequest({
  Path = '/content.mpd',
  QueryParameters = {
    {'a','b'}
  }
})

Arguments

The following (iterable) arguments will be known by the function:

`QueryParameters`

Type: nested table (indexed by number).
Description: Array of query parameters as {[1]='Name',[2]='Value'} pairs that were present in the query string of the request. Format identical to the HTTPRequest.QueryParameters-field specified for the return value above.

Example usage:

for _, queryParam in pairs(QueryParameters) do
  print(queryParam[1]..'='..queryParam[2])
end

`Headers`

Type: nested table (indexed by number).
Description: Array of request headers as {[1]='Name',[2]='Value'} pairs that were present in the request. Format identical to the HTTPRequest.Headers-field specified for the return value above. A multi-value header such as Foo: bar1,bar2 is seen in request_translation_function as Headers={..., {[1]='foo',[2]='bar1'}, {[1]='foo',[2]='bar1'}, ...}.

Example usage:

for _, header in pairs(Headers) do
  print(header[1]..'='..header[2])
end

Global metatables

In addition to the arguments above, the following (non-iterable) global metatables will be populated with fields that may be retrieved by the request_translation_function:

Note that metatables may only be accessed by already known keys:

Metatable `request`

request.method
- Description: HTTP request method.
- Type: string
- Example: 'GET', 'POST'
request.body
- Description: HTTP request body string.
- Type: string or nil
- Example: '{"foo": "bar"}'
request.major_version
- Description: Major HTTP version such as x in HTTP/x.1.
- Type: integer
- Example: 1
request.minor_version
- Description: Minor HTTP version such as x in HTTP/1.x.
- Type: integer
- Example: 1
request.protocol
- Description: Transfer protocol variant.
- Type: string
- Example: 'HTTP', 'HTTPS'
request.client_ip
- Description: IP address of the client issuing the request.
- Type: string
- Example: '172.16.238.128'
request.path_with_query_params
- Description: Full request path including query parameters.
- Type: string
- Example: '/mycontent/superman.m3u8?b=y&c=z&a=x'
request.path
- Description: Request path without query parameters.
- Type: string
- Example: '/mycontent/superman.m3u8'
request.query_params
- Description: The query parameter string.
- Type: string
- Example: 'b=y&c=z&a=x'
request.filename
- Description: The part of the path following the final slash, if any.
- Type: string
- Example: 'superman.m3u8'
request.subnet
- Description: Subnet of client_ip.
- Type: string or nil
- Example: 'all'

Metatable `request_query_params`

Contains the query parameters keyed by name.

Example:

print(request_query_params.a)

Metatable `request_headers`

Contains the request headers keyed by name.

Example:

print(request_headers.a)

Multiple values are separated with a comma.

Global tables

In addition to the (non-iterable) metatables and (iterable) arguments above, the following global iterable tables are available from all Lua functions:

Table `selection_input`

Contains arbitrary, custom fields fed into the router by clients. Be careful to document any dependencies between the translation functions and selection inputs.

Example usage:

if selection_input then
    for k, v in pairs(selection_input) do
        print('here is '..'selection_input!')
        print(k..'='..v)
    end
else
    print('selection_input is nil')
end

Upon returning from request_translation_function, values in the request, request_query_params and request_headers metatables will reflect the new current status of the updated HTTP Request.

1.1.5.5.6.3 - Response Translation Function

Instructions for how to write a function to modify outgoing responses after a routing decision has been made.

Specifies the body of a Lua function that inspects every outgoing HTTP response and overwrites individual fields before being sent to the client.

Returns nil when nothing is to be changed, or HTTPResponse(t) where t is a table with any of the following optional fields:

Code
- Description: Replaces status code in the response being sent.
- Type: integer
- Example: 200, 404
Text
- Description: Replaces status text in the response being sent.
- Type: string
- Example: 'OK', 'Not found'
MajorVersion
- Description: Replaces major HTTP version such as x in HTTP/x.1 in the response being sent.
- Type: integer
- Example: 1
MinorVersion
- Description: Replaces minor HTTP version such as x in HTTP/1.x in the response being sent.
- Type: integer
- Example: 1
Protocol
- Description: Replaces protocol in the response being sent.
- Type: string
- Example: 'HTTP', 'HTTPS'
Body
- Description: Replaces body in the response being sent.
- Type: string or nil
- Example: '{"foo": "bar"}'
Headers
- Description: Adds, removes or replaces individual headers in the response being sent.
- Type: nested table (indexed by number) representing an array of response headers as {[1]='Name',[2]='Value'} pairs that are added to the response being sent, or overwriting existing request headers with colliding names. To remove a header from the response, specify nil as value, i.e. Headers={..., {[1]='foo',[2]=nil} ...}. Duplicate names are supported. A multi-value header such as Foo: bar1,bar2 is defined by specifying Headers={..., {[1]='foo',[2]='bar1'}, {[1]='foo',[2]='bar2'}, ...}.

Example of a response_translation_function body that sets the Location header to a hardcoded value:

-- Statements go here
print('Setting hardcoded Location')
return HTTPResponse({
  Headers = {
    {'Location', 'cdn1.com/content.mpd?a=b'}
  }
})

Arguments

The following (iterable) arguments will be known by the function:

`Headers`

Type: nested table (indexed by number).
Description: Array of response headers as {[1]='Name',[2]='Value'} pairs that are present in the response being sent. Format identical to the HTTPResponse.Headers-field specified for the return value above. A multi-value header such as Foo: bar1,bar2 is seen in response_translation_function as Headers={..., {[1]='foo',[2]='bar1'}, {[1]='foo',[2]='bar1'}, ...}.

Example usage:

for _, header in pairs(Headers) do
  print(header[1]..'='..header[2])
end

Global metatables

In addition to the arguments above, the following (non-iterable) global metatables will be populated with fields that may be retrieved by the response_translation_function:

Note that metatables may only be accessed by already known keys.

Metatable `request`

See documentation for request_translation_function. If the request translation function has modified the incoming request, the request metatable will contain those changes.

Metatable `request_query_params`

See documentation for request_translation_function. If the request translation function has modified the incoming request, the request_query_params metatable will contain those changes.

Metatable `session_query_params`

Alias for metatable request_query_params.

Metatable `request_headers`

See documentation for request_translation_function. If the request translation function has modified the incoming request, the request_headers metatable will contain those changes.

Metatable `response`

Contains all the parts of the outgoing response apart from the headers.

response.body
- Description: HTTP response body string.
- Type: string or nil
- Example: '{"foo": "bar"}'
response.code
- Description: HTTP response status code.
- Type: integer
- Example: 200, 404
response.text
- Description: HTTP response status text.
- Type: string
- Example: 'OK', 'Not found'
response.major_version
- Description: Major HTTP version such as x in HTTP/x.1.
- Type: integer
- Example: 1
response.minor_version
- Description: Minor HTTP version such as x in HTTP/1.x.
- Type: integer
- Example: 1
response.protocol
- Description: Transfer protocol variant.
- Type: string
- Example: 'HTTP', 'HTTPS'

Metatable `response_headers`

Contains the response headers keyed by name. Example:

print(response_headers.a)

Multiple values are separated with a comma.

Metatable `session`

See documentation for session_translation_function. If the session translation function has modified the incoming request, the session metatable will contain those changes.

Metatable `session_groups`

See documentation for session_translation_function.

Global tables

In addition to the (non-iterable) metatables and (iterable) arguments above, the following global iterable tables are available from all Lua functions:

Table `selection_input`

See documentation for request_translation_function. Values in the selection_input metatable will remain unchanged since a prior call to a request_translation_function.

1.1.5.5.6.4 - Session Translation Function

Instructions for how to write a function to modify a client session to affect how it is handled by the router.

Specifies the body of a Lua function that inspects a newly created session and may override its suggested type from “initial” to “instream” or vice versa.

Returns nil when the session type is to remain unchanged, or Session(t) where t is a table with a single field:

Type
- Description: New type of the session.
- Type: string
- Example: 'instream', 'initial'

Example of a session_translation_function body that unconditionally makes all sessions 'instream':

-- Statements go here
print('Modifying session type')
return Session({['Type'] = 'instream'})

Arguments

The following (iterable) arguments are passed to the function:

`Type`

Description: The suggested type of the session.
Type: string
Example: 'instream', 'initial'

Example usage:

-- Flip session type
local newType = 'initial'
if Type == 'initial' then
    newType = 'instream'
end
print('Changing session type from ' .. Type .. ' to ' .. newType)
return Session({['Type'] = newType})

Global metatables

In addition to the arguments above, the following (non-iterable) global metatables will be populated with fields that may be retrieved by the response translation function.

Metatable `request`

See documentation for request_translation_function. If the request translation function has modified the incoming request, the request metatable will contain those changes.

Metatable `request_query_params`

See documentation for request_translation_function. If the request translation function has modified the incoming request, the request_query_params metatable will contain those changes.

Metatable `session_query_params`

Alias for metatable request_query_params.

Metatable `request_headers`

See documentation for request_translation_function. If the request translation function has modified the incoming request, the request_headers metatable will contain those changes.

Metatable `session`

session.client_ip
- Description: Alias for request.client_ip. See documentation for request_translation_function.
session.path_with_query_params
- Description: Alias for request.path_with_query_params. See documentation for request_translation_function.
session.path
- Description: Alias for request.path. See documentation for request_translation_function.
session.query_params
- Description: Alias for request.query_params. See documentation for request_translation_function.
session.filename
- Description: Alias for request.filename. See documentation for request_translation_function.
session.subnet
- Description: Alias for request.subnet. See documentation for request_translation_function.
session.host
- Description: ID of the currently selected host for the session.
- Type: string or nil
- Example: 'host1'
session.id
- Description: ID of the session.
- Type: string
- Example: '8eb2c1bdc106-17d2ff-00000000'
session.session_type
- Description: Type of the session.
- Type: string
- Example: 'initial' or 'instream'. Identical to the value of the Type argument of the session translation function.
session.is_managed
- Description: Identifies managed sessions.
- Type: boolean
- Example: true if Type/session.session_type is 'instream'

Metatable `session_groups`

Defines a mapping from session group name to boolean, indicating whether the session belongs to the session group or not.

Example usage:

if session_groups.vod then print('vod') else print('not vod') end

if session_groups['vod'] then print('vod') else print('not vod') end

Global tables

In addition to the (non-iterable) metatables and (iterable) arguments above, the following global iterable tables are available from all Lua functions:

Table `selection_input`

See documentation for See documentation for request_translation_function. Values in the selection_input metatable will remain unchanged since a prior call to a request_translation_function.

1.1.5.6 - Routing Engine

How the routing engine works and how to configure it

The central component of ESB3024 Router is the routing engine. It is what analyses an incoming video request and determines which CDN, if any, to pass it on to based on a variety of factors such as geographic location, content type and current server load.

Like most of the router, the routing engine is configured through the /v2/configuration API, see Configuration for general configuration information. The basic structure looks something like this:

{
  // ...
  "cdns": [
    {
      "id": "cdn1",
      "http_port": 80,
      "https_port": 443,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      },
      "redirecting": false
    }
  ],
  "hosts": [
    {
      "id": "host1"
      "cdn_id": "cdn1",
      "host": "host1.example.com",
      "ipv6_address": "fc00:ed6e:0:201:0:aff:fe10:306b"
    },
    {
      "id": "host2"
      "cdn_id": "cdn1",
      "host": "host2.example.com",
    }
  ],
  // ...
  "routing": {
    "id": "routing_table",
    "member_order": "weighted",
    "members": [
      {
        "id": "node1",
        "host_id": "host1",
        "weight_function": "return 100"
      },
      {
        "id": "node2",
        "host_id": "host2",
        "weight_function": "return 100"
      }
    ],
    "weight_function": "return 1"
  },
  // ...
}

In the above example the routing engine configuration is contained in the "routing" field of the JSON structure. The "cdns" and "hosts" fields are included to provide a context and show where the "host_id" values come from.

Tree components

The "routing" part of the configuration is a tree of routing nodes. These nodes can be either of two types: A leaf node or a branch node. Leaf nodes contain references to CDN hosts while branch nodes contain a list of member nodes but do not reference any hosts itself.

All nodes, regardless of type, have a unique ID as well as a weight function. The ID is used to identify the node in e.g. rule evaluation metrics and the weight function is used to dynamically determine if a node should be evaluated for the incoming request.

Let us go through the two structures one by one and look at their fields and possible values.

Branch node

Branch nodes do not refer to any hosts by themselves, but contain member nodes that may be traversed in order to find a suitable host.

{
  "id": "branch_node_example",
  "members": [],
  "member_order": "weighted",
  "weight_function": "return 1"
}

"id": The unique identifier for a node. Its value cannot be used on any other node, not even a leaf node.
"weight_function": A Lua function body returning an integer value. Exactly how the return value is used depends on the parent node’s member_order field, but a higher value generally indicates higher likelihood of the node being traversed. The value of 0 or less means that this node should not be used.
"members": A list of child nodes that will be evaluated if this node is selected by the routing engine. How they are evaluated depends on the value of the member_order field.
"member_order: A string determining how this node is to be evaluated. Allowed values are "sequential", "sorted" and "weighted".

Member Order Evaluation

As mentioned above, there are three different member orders and they affect how children of a branch node are traversed.

"sequential" means that all children are evaluated in the order they are listed in the configuration. If a child’s weight function returns a positive value, it will be picked for traversal even if one of its siblings has a higher weight. In order to skip a child its weight function must return 0.

If a child is picked for traversal but none of the leaf nodes in its subtree return a positive weight the routing engine will move on to the next sibling and attempt to find a leaf node there instead. This will continue until a leaf node is found or the entire tree is exhausted. If no valid leaf is found an error will be sent to the client.

"sorted" will cause the route engine to compute the weight of each child node, using each child’s "weight_function" Lua code, sort them from highest to lowest and pick the first node.

If no leaf node within that child’s subtree is accepted, i.e. they all return 0 or less, then the next child will be traversed and so on until a suitable host is found or there are no more subtrees left to evaluate. If no host is found, an error message will be sent to the client.

"weighted" tells the routing engine to evaluate the weight functions for all children, and then pick one of them randomly with a probability equal to the child’s weight divided by the sum of the weights of all immediate children.

Just as with the other two variants, if no suitable leaf node is found within the child’s subtree one of its siblings is picked instead. The same random method is used to pick subsequent children until a leaf node is accepted or the entire tree is exhausted. An error message will be sent to the client if no host is found.

Leaf node

Leaf nodes contain a reference to a host object rather than more child nodes.

{
  "id": "leaf_node_example",
  "host_id": "cdn_host_id",
  "weight_function": "return 100"
}

"id": The unique identifier for a node. Its value cannot be used on any other node, not even a branch node.
"host_id": The ID of one of the hosts in the "hosts" list. Represents the host the client will be routed through if this leaf node is selected.
"weight_function": A Lua function body that returns the weight of this particular node. The return value is used by the parent node to determine which leaf node to select for the current request.

A Note on Weight Functions

While the primary and mandatory purpose of the weight function is to return a value to let the routing engine determine which leaf node is most suitable for a client request, it can do more things. It is, however, vital that a weight function always returns an integer value.

There is a global Lua context for the functions to use to set and read global states, third party software can be used to keep track of external states and values through the selection_input API. It is even possible to set up a dummy node that prints out debug information but always returns 0, in effect having nothing to do with the actual routing.

Due to the global nature of variables in Lua it is important to remember to declare any variables used for temporary calculations as local or risk cross-contamination between client requests. It is generally recommended to avoid using global variables at all in the router unless absolutely certain what the consequences may be.

A Practical Example

Imagine we have a system for a Swedish market, with capacity enough for most of the time, but with rented offload capacity in the third party CDN for peak hour traffic as well as for any clients connecting from outside of Sweden. The private CDN is separated in a host for live content and one for VOD, but the offload CDN has a single host capable of handling anything. The private CDN in this example will also report its remaining capacity in per cent to the selection_input variable capacity_percent and as long as at least 10 % is unused it should accept new requests.

The router configuration will have to set up a few hosts, classifiers for incoming requests and finally a routing tree which we will go through in detail further down:

{
  "cdns": [
    {
      "http_port": 80,
      "https_port": 443,
      "id": "offload",
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      },
      "redirecting": false
    },
    {
      "http_port": 80,
      "https_port": 443,
      "id": "cdn",
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      },
      "redirecting": false
    }
  ],
  "hosts": [
    {
      "cdn_id": "offload",
      "host": "offload.example",
      "ipv6_address": "fc00:ed6e:0:201:0:aff:fe10:306b",
      "id": "offload"
    },
    {
      "cdn_id": "cdn",
      "host": "vod.cdn.example",
      "id": "cdn_vod"
    },
    {
      "cdn_id": "cdn",
      "host": "live.cdn.example",
      "id": "cdn_live"
    }
  ],
  "session_groups": [
    {
      "id": 1,
      "name": "Not Sweden",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": true,
            "name": "Not Sweden",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "country": "Sweden"
            }
          }
        ]
      ]
    },
    {
      "id": 2,
      "name": "IsLive",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "Is live content",
            "rule": {
              "pattern": "*/live/*",
              "rule_type": "string_match_rule",
              "source": "session/content_url_path"
            }
          }
        ]
      ]
    }
  ],
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "Offload if not Sweden",
        "host_id": "offload",
        "weight_function": "local retval = 0; if session_groups[\"Not Sweden\"] \
          then retval = 1; end; return retval"
      },
      {
        "id": "Private CDN",
        "member_order": "sorted",
        "members": [
          {
            "id": "live",
            "host_id": "cdn_live",
            "weight_function": "local retval = 0; if (selection_input.capacity_percent \
              or 0) > 10 and session_groups.IsLive then retval = 1; end; return retval"
          },
          {
            "id": "vod",
            "host_id": "cdn_vod",
            "weight_function": "local retval = 1; if (selection_input.capacity_percent \
              or 0) < 10 or session_groups.IsLive then retval = 0; end; return retval"
          }
        ],
        "weight_function": "return 1"
      },
      {
        "id": "Offload if no match",
        "host_id": "offload",
        "weight_function": "return 1"
      }
    ],
    "weight_function": "return 1"
  }
}

In order to understand this configuration, let us pretend some content is requested from a client:

Client IP: 94.127.35.102
Content path: "/live/news.m3u8"

The first thing that will happen is that the session group classifiers will analyse the request.

The IP is geolocated to Stockholm, Sweden. This means that the classifier named "Not Sweden" will be negative¹, which in turn means that the session group named "Not Sweden" will also be negative.

The content path contains the string "/live/" which causes the classifier named "Is live content" to be positive and the corresponding session group "IsLive" will also become positive as a result.

With the classifiers done, it is time to traverse the routing tree.²

First the root node is evaluated. Its weight function returns a positive value, so its child nodes will be traversed.³ The member order is set to "sequential" so the routing engine goes through them one by one in the order they are written in the configuration and picks the first one to return a positive value.

"Offload if not Sweden" comes first, so its weight function is called. Since the session group "Not Sweden" is false, the weight function will return 0 and the engine moves on to the next sibling.

"Private CDN" has two children and the member order "sorted" meaning that they will both be evaluated, and the one with the highest weight will be picked. Assuming that the CDN capacity is sufficient, the child "live" will return 1 and "vod" will return 0, meaning that the host object with ID "cdn_live" will be selected and the client redirected to live.cdn.example/live/news.m3u8.

However, if the capacity is insufficient, neither child will return a positive value and the engine continues on to "Offload if no match" which always returns 1 and is therefore guaranteed to match and cause any unclaimed requests to end up at offload.example/live/news.m3u8.

The classifier rule actually evaluates positively against IP:s located in Sweden, but since it is marked as inverted, the result is flipped and the whole thing ends up being negative instead. This is much simpler than trying to write rules that match against everything except Sweden. ↩︎
Note that some of the weight functions are too long to fit on a single line. They have been split into multiple lines with a backslash indicating the break point. These backslashes and line breaks must be removed for the configuration to be valid and accepted by the router. ↩︎
This is not strictly necessary. By default any node without an explicit weight function will have "return 100" instead. Explicit functions are used in this example for clarity. ↩︎

1.1.6 - Operations

Operators Guide

This guide describes how to perform day-to-day operations of the ACD Router and its associated services, collectively known as the Director.

Component Overview

To effectively operate the Director software, it is important to understand the composition of the various software components and how they are deployed.

Each Director instance functions as an independent system, comprising multiple containerized services. These containers are managed by a standard container runtime and are seamlessly integrated with the host’s operating system to enhance the overall operator experience.

For Red Hat Enterprise Linux 8 based operating systems, the Podman container runtime is used for container management. In the case of older Red Hat releases, Docker serves as the container runtime. Although both

runtimes

utilize the same OCI compliant container images and offer equivalent levels of abstraction, performance, and security, the key distinction lies in Podman’s ability to operate without an additional daemon service running on the host. Unlike Docker, Podman manages each container as a separate process, eliminating the reliance on a shared daemon and mitigating the risk of a single-point-of-failure scenario.

Although several distinct services make up the Director, the primary component is the router. The router is responsible for listening for incoming requests, processing the request, and redirecting the client to the appropriate host, or CDN to deliver the requested content.

Two additional containers are responsible for configuration management. Those are confd and confd-transformer. The former manages a local database of configuration metadata and provides a REST API for managing the configuration. The confd-transformer simply listens for configuration changes from confd and adapts that configuration to a format suitable for the router to ingest. For additional information about setting up and using confd see here..

The next two components, the edns-proxy and the convoy-bridge allow the router to communicate with an EDNS server for EDNS-based routing, and with synchronization with Convoy respectively. Additional information about the EDNS-Proxy is available here.. For the Convoy Bridge service see here..

The remaining containers are useful for metrics, monitoring, and alerting. These include prometheus and grafana for monitoring and analytics, fluentbit and alertmanager for monitoring and alarms, and grafana-loki for log aggregation.

1.1.6.1 - Services

Starting / Stopping / Monitoring Services

Each container shipped with the Director is fully-integrated with the systemd service on the host, enabling easy management using standard systemd commands. The logs for each container are also full-integrated with journald to simplify troubleshooting.

In order to integrate the Podman containers with systemd, a common prefix of acd- has been applied to each service name. For example the router container is managed by the service acd-router, and the confd container is managed by the service acd-confd. These same prefixed names apply while fetching logs via journald. This common prefix aids in grouping the related services as well as provides simpler filtering for tab-completion.

Starting / Stopping Services

Standard systemd commands should be used to start and stop the services.

systemctl start acd-router - Starts the router container.
systemctl stop acd-router - Stops the router container.
systemctl status acd-router - Displays the status of the router container.

Due to the limitation of needing the acd- prefix, it provides the ability to work with all ACD services in a group. For example:

systemctl status 'acd-*' - Display the status of all installed ACD components.
systemctl start 'acd-*' - Start all ACD components.

Logging

Each ACD component corresponds to a journal entry with the same unit name, with the acd- prefix. Standard journald commands can be used to view and manage the logging.

journalctl -u acd-router - Display the logs for the router container

Access Log

Access logs can be obtained from journald by using the following command:

journalctl SYSLOG_IDENTIFIER=router ACCESSLOG=1

This will output the router’s access logs in a standard format.

Troubleshooting

Some additional logging may be available in the filesystem, the paths of which can be determined by executing the ew-sysinfo command. See Diagnostics. for additional details.

1.1.6.2 - Docker

Special operating instructions for Red Hat Enterprise Linux 7

Support for Red Hat Enterprise Linux 7 and compatible Operating Systems is considered best-effort and the instructions included here are only maintained for archival purposes.

Since Red Hat Enterprise Linux 7 uses the Docker runtime, there are several key differences which are noted below.

Starting and Stopping Services

Since the containers are not integrated into systemd under Docker, standard Docker container management commands must be used to start and stop the services.

docker ps - Displays all running containers along with their relevant metadata.
docker stop router - Stops the router container.
docker start router - Starts the router container.

Obtaining container logs

The standard docker logs command should be used to view the logs for the various containers.

docker logs router - Display the complete log for the router container.
docker logs -f router - Display and tail the log for the router container.

Router Access Logs are not available under Docker.

1.1.7 - Convoy Bridge

Convoy Bridge Integration

The convoy-bridge is an optional integration service, pre-installed alongside the router which provides two-way communication between the router and a separate Convoy installation.

The convoy-bridge is designed to allow the Convoy account metadata to be available from within the router for such use-cases as inserting the account specific prefixes in the redirect URL and validating per-account internal security tokens. The service works by periodically polling the Convoy server for changes to the configuration, and when detected, the relevant configuration information is pushed to the router.

In addition, the convoy-bridge has the ability to integrate the router with the Convoy analytics service, such that client sessions started by the router are properly collected by Convoy, and are available in the dashboards.

Configuration

The convoy-bridge service is configured using confcli on the router host. All configuration for the convoy-bridge exists under the path integration.convoy.bridge.

{
  "logLevel": "info",
  "accounts": {
    "enabled": true,
    "dbUrl": "mysql://convoy:eith7jee@convoy:3306",
    "dbPollInterval": 60
  },
  "analytics": {
    "enabled": true,
    "brokers": ["broker1:9092", "broker2:9092"],
    "batchInterval": 10,
    "maxBatchSize": 500
  },
  "otherRouters": [
    {
      "url": "https://router2:5001",
      "apiKey": "key1",
      "validateCerts": true
    }
  ]
}

In the above configuration block, there are three main sections. The accounts section enables fetching account metadata from Convoy towards the router. The analytics section controls the integration between the router and the Convoy analytics service. The otherRouters section is used to synchronize additional router instances. The local router instance will always be implicitly included. Additional routers listed in this section will be handled by this instance of the convoy-bridge service.

Logging

For RHEL8 or later, the logs are available in the system journal and can be viewed using:

journalctl -u acd-convoy-bridge

For RHEL7, logs are available from within Docker and can be viewed with the command:

docker logs convoy-bridge

1.1.8 - Monitoring

Monitoring

1.1.8.1 - System troubleshooting

Using ew-sysinfo to monitor and troubleshoot ESB3024

ESB3024 contains the tool ew-sysinfo that gives an overview of how the system is doing. Simply use the command and the tool will output information about the system and the installed ESB3024 services.

The output format can be changed using the --format flag, possible values are human (default) and json, e.g.:

$ ew-sysinfo
system:
   os: ['5.4.17-2136.321.4.el8uek.x86_64', 'Oracle Linux Server 8.8']
   cpu_cores: 2
   cpu_load_average: [0.03, 0.03, 0.0]
   memory_usage: 478 MB
   memory_load_average: [0.03, 0.03, 0.0]
   boot_time: 2023-09-08T08:30:57Z
   uptime: 6 days, 3:43:44.640665
   processes: 122
   open_sockets:
      ipv4: 12
      ipv6: 18
      ip_total: 30
      tcp_over_ipv4: 9
      tcp_over_ipv6: 16
      tcp_total: 25
      udp_over_ipv4: 3
      udp_over_ipv6: 2
      udp_total: 5
      total: 145
system_disk (/):
   total: 33271 MB
   used: 7978 MB (24.00%)
   free: 25293 MB
journal_disk (/run/log/journal):
   total: 1954 MB
   used: 217 MB (11.10%)
   free: 1736 MB
vulnerabilities:
   meltdown: Mitigation: PTI
   spectre_v1: Mitigation: usercopy/swapgs barriers and __user pointer sanitization
   spectre_v2: Mitigation: Retpolines, STIBP: disabled, RSB filling, PBRSB-eIBRS: Not affected
processes:
   orc-re:
      pid: 177199
      status: sleeping
      cpu_usage_percent: 1.0%
      cpu_load_average: 131.11%
      memory_usage: 14 MB (0.38%)
      num_threads: 10
hints:
   get_raw_router_config: cat /opt/edgeware/acd/router/cache/config.json
   get_confd_config: cat /opt/edgeware/acd/confd/store/__active
   get_router_logs: journalctl -u acd-router
   get_edns_proxy_logs: journalctl -u acd-edns-proxy
   check_firewall_status: systemctl status firewalld
   check_firewall_config: iptables -nvL

# For --format=json, it's recommended to pipe the output to a JSON interpreter
# such as jq

$ ew-sysinfo --format=json | jq
{
  "system": {
    "os": [
      "5.4.17-2136.321.4.el8uek.x86_64",
      "Oracle Linux Server 8.8"
    ],
    "cpu_cores": 2,
    "cpu_load_average": [
      0.01,
      0.0,
      0.0
    ],
    "memory_usage": "479 MB",
    "memory_load_average": [
      0.01,
      0.0,
      0.0
    ],
    "boot_time": "2023-09-08 08:30:57",
    "uptime": "6 days, 5:12:24.617114",
    "processes": 123,
    "open_sockets": {
      "ipv4": 13,
      "ipv6": 18,
      "ip_total": 31,
      "tcp_over_ipv4": 10,
      "tcp_over_ipv6": 16,
      "tcp_total": 26,
      "udp_over_ipv4": 3,
      "udp_over_ipv6": 2,
      "udp_total": 5,
      "total": 146
    }
  },
  "system_disk (/)": {
    "total": "33271 MB",
    "used": "7977 MB (24.00%)",
    "free": "25293 MB"
  },
  "journal_disk (/run/log/journal)": {
    "total": "1954 MB",
    "used": "225 MB (11.50%)",
    "free": "1728 MB"
  },
  "vulnerabilities": {
    "meltdown": "Mitigation: PTI",
    "spectre_v1": "Mitigation: usercopy/swapgs barriers and __user pointer sanitization",
    "spectre_v2": "Mitigation: Retpolines, STIBP: disabled, RSB filling, PBRSB-eIBRS: Not affected"
  },
  "processes": {
    "orc-re": {
      "pid": 177199,
      "status": "sleeping",
      "cpu_usage_percent": "0.0%",
      "cpu_load_average": "137.63%",
      "memory_usage": "14 MB (0.38%)",
      "num_threads": 10
    }
  }
}

Note that your system might have different monitored processes and field names.

The field hints is different from the rest. It lists common commands that can be used to further monitor system performance, useful for quickly troubleshooting a faulty system.

1.1.8.2 - Scraping data with Prometheus

Prometheus is a third-party data scraper which is installed as a containerized service in the default installation of ESB3024 Router. It periodically reads metrics data from different services, such as acd-router, aggregates it and makes it available to other services that visualize the data. Those services include Grafana and Alertmanager.

The Prometheus configuration file can be found on the host at /opt/edgeware/acd/prometheus/prometheus.yaml.

Accessing Prometheus

Prometheus has a web interface that is listening for HTTP connections on port 9090. There is no authentication, so anyone who has access to the host that is running Prometheus can access the interface.

Starting / Stopping Prometheus

After the service is configured, it can be managed via systemd, under the service unit acd-prometheus.

systemctl start acd-prometheus

For EL7 systems running under Docker, the container must be started or stopped using the respective Docker start/stop commands. Note that the acd- prefix is not applicable under Docker.

docker start prometheus

Logging

The container logs are automatically published to the system journal, under the same unit descriptor, and can be viewed using journalctl

journalctl -u acd-prometheus

For EL7 systems running under Docker, the container logs can only be viewed using the docker logs command. Note that the acd- prefix is not applicable under Docker.

docker logs prometheus

1.1.8.3 - Visualizing data with Grafana

1.1.8.3.1 - Managing Grafana

Grafana displays graphs based on data from Prometheus. A default deployment of Grafana is running in a container alongside ESB3024 Router.

Grafana’s configuration and runtime files are stored under /opt/edgeware/acd/grafana. It comes with default dashboards that are documented at Grafana dashboards.

Accessing Grafana

Grafana’s web interface is listening for HTTP connections on port 3000. It has two default accounts, edgeware and admin.

The edgeware account can only view graphs, while the admin account can also edit graphs. The accounts with default passwords are shown in the table below.

Account	Default password
`edgeware`	`edgeware`
`admin`	`edgeware`

Starting / Stopping Grafana

Grafana can be managed via systemd, under the service unit acd-grafana.

systemctl start acd-grafana

For EL7 systems running under Docker, the container must be started or stopped using the respective Docker start/stop commands. Note that the acd- prefix is not applicable under Docker.

docker start grafana

Logging

The container logs are automatically published to the system journal, under the same unit descriptor, and can be viewed using journalctl

journalctl -u acd-grafana

For EL7 systems running under Docker, the container logs can only be viewed using the docker logs command. Note that the acd- prefix is not applicable under Docker.

docker logs grafana

1.1.8.3.2 - Grafana Dashboards

Dashboards in default Grafana installation

Grafana will be populated with pre-configured graphs which present some metrics on a time scale. Below is a comprehensive list of those dashboards, along with short descriptions.

Router Monitoring dashboard

This dashboard is by default set as home directory - it’s what user will see after logging in.

Number Of Initial Routing Decisions

HTTP Status Codes

Total number of responses sent back to incoming requests, shown by their status codes. Metric: client-response-status

Incoming HTTP and HTTPS Requests

Total number of incoming requests that were deemed valid, divided into SSL and Unencrypted categories. Metric: num_valid_http_requests

Debugging Information dashboard

Number of Lua Exceptions

Number of exceptions encountered so far while evaluating Lua rules. Metric: lua_num_errors

Number of Lua Contexts

Number of active Lua interpreters, both running and idle. Metric: lua_num_evaluators

Time Spent In Lua

Number of microseconds the Lua interpreters were running. Metric: lua_time_spent

Router Latencies

Histogram-like graph showing how many responses were sent within the given latency interval. Metric: orc_latency_bucket

Internal debugging

A folder that contains dashboards intended for internal use.

ACD: Incoming Internet Connections dashboard

SSL Warnings

Rate of warnings logged during TLS connections Metric: num_ssl_warnings_total

SSL Errors

Rate of errors logged during TLS connections Metric: num_ssl_errors_total

Valid Internet HTTPS Requests

Rate of incoming requests that were deemed valid, HTTPS only. Metric: num_valid_http_requests

Invalid Internet HTTPS Requests

Rate of incoming requests that were deemed invalid, HTTPS only. Metric: num_invalid_http_requests

Valid Internet HTTP Requests

Rate of incoming requests that were deemed valid, HTTP only. Metric: num_valid_http_requests

Invalid Internet HTTP Requests

Rate of incoming requests that were deemed invalid, HTTP only. Metric: num_invalid_http_requests

Prometheus: ACD dashboard

Logged Warnings

Rate of logged warnings since the router has started, divided into CDN-related and CDN-unrelated. Metric: num_log_warnings_total

Logged Errors

Rate of logged errors since the router has started. Metric: num_log_errors_total

HTTP Requests

Rate of responses sent to incoming connections. Metric: orc_latency_count

Number Of Active Sessions

Number of sessions opened on router that are still active. Metric: num_sessions

Total Number Of Sessions

Total number of sessions opened on router. Metric: num_sessions

Session Type Counts (Non-Stacked)

Number of active sessions divided by type; see metric documentation linked below for up-to-date list of types. Metric: num_sessions

Prometheus/ACD: Subrunners

Client Connections

Number of currently open client connections per subrunner. Metric: subrunner_client_conns

Asynchronous Queues (Current)

Number of queued events per subrunner, roughly corresponding to load. Metric: subrunner_async_queue

Used <Send/receive> Data Blocks

Number of send or receive data blocks currently in use per subrunner, as decided by the “Send/receive” drop down box. Metric: subrunner_used_send_data_blocks and subrunner_used_receive_data_blocks

Asynchronous Queues (Max)

Maximum number of events waiting in queue. Metric: subrunner_max_async_queue

Total <Send/receive> Data Blocks

Number of send or receive data blocks allocated per subrunner, as decided by the “Send/receive” drop down box. Metric: subrunner_total_send_data_blocks and subrunner_total_receive_data_blocks

Low Queue (Current)

Number of low priority events queued per subrunner. Metric: subrunner_low_queue

Medium Queue (Current)

Number of medium priority events queued per subrunner. Metric: subrunner_medium_queue

High Queue (Current)

Number of high priority events queued per subrunner. Metric: subrunner_high_queue

Low Queue (Max)

Maximum number of events waiting in low priority queue. Metric: subrunner_max_low_queue

Medium Queue (Max)

Maximum number of events waiting in medium priority queue. Metric: subrunner_max_medium_queue

High Queue (Max)

Maximum number of events waiting in high priority queue. Metric: subrunner_max_high_queue

Wakeups

The number of times a subrunner has been waken up from sleep. Metric: subrunner_io_wakeups

Overloaded

The number of times the number of queued events for a subrunner exceeded its maximum. Metric: subrunner_times_worker_overloaded

Autopause

Number of sockets that have been automatically paused. This happens when the work manager is under heavy load. Metric: subrunner_io_autopause_sockets

1.1.8.4 - Alarms and Alerting

Configuring alarms and alerting

Alerts are generated by the third-party service Prometheus, which sends them to the Alertmanager service. A default containerized instance of Alertmanager is deployed alongside ESB3024 Router. Out of the box, Alertmanager ships with only a sample configuration file, and will require manual configuration prior to enabling the alerting functionality. Due to the many different possible configurations for how alerts are both detected and where they are pushed, the official Alertmanager documentation should be followed for how to configure the service.

The router ships with Alertmanager 0.25, the documentation for which can be found at prometheus.io. The Alertmanager configuration file can be found on the host at /opt/edgeware/acd/alertmanager/alertmanager.yml.

Accessing Alertmanager

Alertmanager has a web interface that is listening for HTTP connections on port 9093. There is no authentication, so anyone who has access to the host that is running Alertmanager can access the interface.

Starting / Stopping Alertmanager

After the service is configured, it can be managed via systemd, under the service unit acd-alertmanager.

systemctl start acd-alertmanager

For EL7 systems running under Docker, the container must be started or stopped using the respective Docker start/stop commands. Note that the acd- prefix is not applicable under Docker.

docker start alertmanager

Logging

The container logs are automatically published to the system journal, under the same unit descriptor, and can be viewed using journalctl

journalctl -u acd-alertmanager

For EL7 systems running under Docker, the container logs can only be viewed using the docker logs command. Note that the acd- prefix is not applicable under Docker.

docker logs alertmanager

1.1.8.5 - Monitoring multiple routers

By default an instance of Prometheus only monitors the ESB3024 Router that is installed on the same host as where Prometheus is installed. It is possible to make it monitor other router instances and visualize all instances on one Grafana instance.

Configuring of Prometheus

This is configured in the scraping configuration of Prometheus, which is found in the file /opt/edgeware/acd/prometheus/prometheus.yaml, which typically looks like this:

global:
  scrape_interval:     15s

rule_files:
  - recording-rules.yaml

# A scrape configuration for router metrics
scrape_configs:
  - job_name: 'router-scraper'
    scheme: https
    tls_config:
      insecure_skip_verify: true
    static_configs:
    - targets:
      - acd-router-1:5001
    metrics_path: /m1/v1/metrics
    honor_timestamps: true
  - job_name: 'edns-proxy-scraper'
    scheme: http
    static_configs:
    - targets:
      - acd-router-1:8888
    metrics_path: /metrics
    honor_timestamps: true

More routers can be added to the scrape configuration by simply adding more routers under targets in the scraper jobs.

For instance, to monitor acd-router-2 and acd-router-3 along acd-router-1, the configuration file needs to be modified like this:

global:
  scrape_interval:     15s

rule_files:
  - recording-rules.yaml

# A scrape configuration for router metrics
scrape_configs:
  - job_name: 'router-scraper'
    scheme: https
    tls_config:
      insecure_skip_verify: true
    static_configs:
    - targets:
      - acd-router-1:5001
      - acd-router-2:5001
      - acd-router-3:5001
    metrics_path: /m1/v1/metrics
    honor_timestamps: true
  - job_name: 'edns-proxy-scraper'
    scheme: http
    static_configs:
    - targets:
      - acd-router-1:8888
      - acd-router-2:8888
      - acd-router-3:8888
    metrics_path: /metrics
    honor_timestamps: true

After the file has been modified, Prometheus needs to be restarted by typing

systemctl restart acd-prometheus

It is possible to use the same configuration on multiple routers, so that all routers in a deployment can monitor each other.

Selecting router in Grafana

In the top left corner the Grafana dashboards have a drop-down menu labeled “ACD Router”, which allows to choose which router to monitor.

1.1.8.6 - Routing Rule Evaluation Metrics

Node Visit counters

ESB3024 Router counts the number of times a node and any of its children is selected in the routing table.

The visit counters can be retrieved with the following end points:

`/v1/node_visits`

Returns visit counters for each node as a flat list of host:counter pairs in JSON.

Example output:

{
  "node1": "1",
  "node2": "1",
  "node3": "1",
  "top": "3"
}

`/v1/node_visits_graph`

Returns a full graph of nodes with their respective visit counters in GraphML.

Example output:

<?xml version="1.0"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns
http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
  <key id="visits" for="node" attr.name="visits" attr.type="string" />
  <graph id="G" edgedefault="directed">
    <node id="routing_table">
      <data key="visits">5</data>
    </node>
    <node id="cdn1">
      <data key="visits">1</data>
    </node>
    <node id="node1">
      <data key="visits">1</data>
    </node>
    <node id="cdn2">
      <data key="visits">2</data>
    </node>
    <node id="node2">
      <data key="visits">2</data>
    </node>
    <node id="cdn3">
      <data key="visits">2</data>
    </node>
    <node id="node3">
      <data key="visits">2</data>
    </node>
    <edge id="e0" source="cdn1" target="node1" />
    <edge id="e1" source="routing_table" target="cdn1" />
    <edge id="e2" source="cdn2" target="node2" />
    <edge id="e3" source="routing_table" target="cdn2" />
    <edge id="e4" source="cdn3" target="node3" />
    <edge id="e5" source="routing_table" target="cdn3" />
  </graph>
</graphml>

To receive the graph as JSON, specify Accept:application/json in the request headers.

Example output:

{
  "edges": [
    {
      "source": "cdn1",
      "target": "node1"
    },
    {
      "source": "routing_table",
      "target": "cdn1"
    },
    {
      "source": "cdn2",
      "target": "node2"
    },
    {
      "source": "routing_table",
      "target": "cdn2"
    },
    {
      "source": "cdn3",
      "target": "node3"
    },
    {
      "source": "routing_table",
      "target": "cdn3"
    }
  ],
  "nodes": [
    {
      "id": "routing_table",
      "visits": "5"
    },
    {
      "id": "cdn1",
      "visits": "1"
    },
    {
      "id": "node1",
      "visits": "1"
    },
    {
      "id": "cdn2",
      "visits": "2"
    },
    {
      "id": "node2",
      "visits": "2"
    },
    {
      "id": "cdn3",
      "visits": "2"
    },
    {
      "id": "node3",
      "visits": "2"
    }
  ]
}

Resetting Visit Counters

A node visit counter with an id not matching any node id of a newly applied routing table is destroyed.

Reset all counters to zero by momentarily applying a configuration with a placeholder routing root node, that has unique id and an empty members list, e.g:

"routing": {
  "id": "empty_routing_table",
  "members": []
}

… and immediately reapply the desired configuration.

1.1.8.7 - Metrics

Metrics endpoint

ESB3024 Router collects a large number of metrics that can give insight into it’s condition at runtime. Those metrics are available in Prometheus’ text-based exposition format at endpoint :5001/m1/v1/metrics.

Below is the description of these metrics along with their labels.

`client_response_status`

Number of responses sent back to incoming requests.

Type: counter

`lua_num_errors`

Number of errors encountered when evaluating Lua rules.

Type: counter

`lua_num_evaluators`

Number of Lua rules evaluators (active interpreters).

Type: gauge

`lua_time_spent`

Time spent by running Lua evaluators, in microseconds.

Type: counter

`num_configuration_changes`

Number of times configuration has been changed since the router has started.

Type: counter

`num_endpoint_requests`

Number of requests redirected per CDN endpoint.

Type: counter
Labels:
- endpoint - CDN endpoint address.
- selector - whether the request was counted during initial or instream selection.

`num_invalid_http_requests`

Number of client requests that either use wrong method or wrong URL path. Also number of all requests that cannot be parsed as HTTP.

Type: counter
Labels:
- source - name of internal filter function that classified request as invalid. Probably not of much use outside debugging.
- type - whether the request was HTTP (Unencrypted) or HTTPS (SSL).

`num_log_errors_total`

Number of logged errors since the router has started.

Type: counter

`num_log_warnings_total`

Number of logged warnings since the router has started.

Type: counter

`num_managed_redirects`

Number of redirects to the router itself, which allows session management.

Type: counter

`num_manifests`

Number of cached manifests.

Type: gauge
Labels:
- count - state of manifest in cache, can be either lru, evicted or total.

`num_qoe_losses`

Number of “lost” QoE decisions per CDN.

Type: counter
Labels:
- cdn_id - ID of CDN that loose QoE battle.
- cdn_name - name of CDN that loose QoE battle.
- selector - whether the decision was taken during initial or instream selection.

`num_qoe_wins`

Number of “won” QoE decisions per CDN.

Type: counter
Labels:
- cdn_id - ID of CDN that won QoE battle.
- cdn_name - name of CDN that won QoE battle.
- selector - whether the decision was taken during initial or instream selection.

`num_rejected_requests`

Deprecated, should always be at 0.

Type: counter
Labels:
- selector - whether the request was counted during initial or instream selection.

`num_requests`

Total number of requests received by the router.

Type: counter
Labels:
- selector - whether the request was counted during initial or instream selection.

`num_sessions`

Number of sessions opened on router.

Type: gauge
Labels:
- state - either active or inactive.
- type - one of: initial, instream, qoe_on, qoe_off, qoe_agent or sp_agent.

`num_ssl_errors_total`

Number of all errors logged during TLS connections, both incoming and outgoing.

Type: counter

`num_ssl_warnings_total`

Number of all warnings logged during TLS connections, both incoming and outgoing.

Type: counter
Labels:
- category - which kind of TLS connection triggered the warning. Can be one of: cdn, content, generic, repeated_session or empty.

`num_unhandled_requests`

Number of requests for which no CDN could be found.

Type: counter
Labels:
- selector - whether the request was counted during initial or instream selection.

`num_unmanaged_redirects`

Number of redirects to “outside” the router - usually to CDN.

Type: counter
Labels:
- cdn_id - ID of CDN picked for redirection.
- cdn_name - name of CDN picked for redirection.
- selector - whether the redirect was result of initial or instream selection.

`num_valid_http_requests`

Number of received requests that were not deemed invalid, see num_invalid_http_requests.

Type: counter
Labels:
- source - name of internal filter function that classified request as invalid. Probably not of much use outside debugging.
- type - whether the request was HTTP (Unencrypted) or HTTPS (SSL).

`orc_latency_bucket`

Total number of responses sorted into “latency buckets” - labels denoting latency interval.

Type: counter
Labels:
- le - latency bucket that given response falls into.
- orc_status_code - HTTP status code of given response.

`orc_latency_count`

Total number of responses.

Type: counter
Labels:
- tls - whether the response was sent via SSL/TLS connection or not.
- orc_status_code - HTTP status code of given response.

`ssl_certificate_days_remaining`

Number of days until a SSL certificate expires.

Type: gauge
Labels:
- domain - the common name of the domain that the certificate authenticates.
- not_valid_after - the expiry time of the certificate.
- not_valid_before - when the certificate starts being valid.
- usable - if the certificate is usable to the router, see the ssl_certificate_usable_count metric for an explanation.

`ssl_certificate_usable_count`

Number of usable SSL certificates. A certificate is usable if it is valid and authenticates a domain name that points to the router.

Type: gauge

1.1.8.7.1 - Internal Metrics

Internal Metrics

A subrunner is an internal module of ESB3024 Router which handles routing requests. The subrunner metrics are technical and mainly of interest for Agile Content. These metrics will be briefly described here.

`subrunner_async_queue`

Number of queued events per subrunner, roughly corresponding to load.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_client_conns`

Number of currently open client connections per subrunner.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_high_queue`

Number of high priority events queued per subrunner.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_io_autopause_sockets`

Number of sockets that have been automatically paused. This happens when the work manager is under heavy load.

Type: counter
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_io_send_data_fast_attempts`

A fast data path was added that in many cases increases the performance of the router. This metric was added to verify that the fast data path is taken.

Type: counter
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_io_wakeups`

The number of times a subrunner has been waken up from sleep.

Type: counter
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_low_queue`

Number of low priority events queued per subrunner.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_max_async_queue`

Maximum number of events waiting in queue.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_max_high_queue`

Maximum number of events waiting in high priority queue.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_max_low_queue`

Maximum number of events waiting in low priority queue.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_max_medium_queue`

Maximum number of events waiting in medium priority queue.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_medium_queue`

Number of medium priority events queued per subrunner.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_times_worker_overloaded`

Number of times when queued events for given subrunner exceeded the tuning.overload_threshold value (defaults to 32).

Type: counter
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_total_receive_data_blocks`

Number of receive data blocks allocated per subrunner.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_total_send_data_blocks`

Number of send data blocks allocated per subrunner.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_used_receive_data_blocks`

Number of receive data blocks currently in use per subrunner. Same as subrunner_total_receive_data_blocks.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

`subrunner_used_send_data_blocks`

Number of send data blocks currently in use per subrunner. Same as subrunner_total_send_data_blocks.

Type: gauge
Labels:
- subrunner_id - ID of given subrunner.

1.1.9 - Releases

ESB3024 Router releases

1.1.9.1 - Release esb3024-1.10.1

Build date

2024-04-18

Release status

Type: production

Breaking changes

If upgrading from a release prior to 1.10.0, the configuration needs to be manually updated after upgrading to 1.10.1. See Installing release 1.10.x for more information.

Change log

NEW: Change predictive load balancing functions to use megabits/s [ESB3024-932]
FIXED: Logic classifier statements can consume all memory [ESB3024-937]

Known Limitations

None

1.1.9.2 - Release esb3024-1.10.0

Build date

2024-04-02

Release status

Type: production

Breaking changes

The configuration needs to be manually updated after upgrading to 1.10.0. See Installing release 1.10.0 for more information.

Change log

NEW: Use metrics from streamers in routing decisions. Added standard library Lua support to use hardware metrics in routing decisions. Added host health checks in the configuration. [ESB3024-154]
NEW: Remove unused field “apiKey” from configuration [ESB3024-426]
NEW: Support integration with Convoy Analytics [ESB3024-694]
NEW: Support combining classifiers using AND/OR in session groups [ESB3024-776]
NEW: Enable access logging by default [ESB3024-816]
NEW: Improved Lua translation function error handling [ESB3024-874]
NEW: Updated predictive load balancing functions to support hardware metrics [ESB3024-887]
NEW: Remove apiKey from documentation [ESB3024-927]
FIXED: Condition with ‘or’ statement sometimes generate faulty Lua [ESB3024-863]

Known Limitations

None

1.1.9.3 - Release esb3024-1.8.0

Build date

2024-02-07

Release status

Type: production

Breaking changes

The configuration needs to be manually updated after upgrading to 1.8.0. See Installing release 1.8.0 for more information.

Change log

NEW: Remove ESB3026 Account Monitor from installer. [ESB3024-354]
NEW: Improve selection input endpoint flexibility and security. See API overview documentation for details. [ESB3024-423]
NEW: Support anonymous geoip rules [ESB3024-699]
NEW: Add ASN IDs list classifiers to confd [ESB3024-778]
NEW: Enable content popularity tracking by default. Added option to enable/disable in confd/confcli. [ESB3024-781]
NEW: Remove dependency on session from security token verification [ESB3024-809]
FIXED: A lot of JSON output on failed routing. HTTP response no longer contains internal routing information. [ESB3024-523]
FIXED: Returning Lua table from Lua debug endpoint can crash router. Selection Input values now support floating point values in a Lua context [ESB3024-691]
FIXED: Floating point selection inputs are truncated to ints when passed to Lua context [ESB3024-710]
FIXED: Race condition between RestApi and Session [ESB3024-753]
FIXED: confd/concli doesn’t support “forward_host_header” on hostGroups [ESB3024-761]
FIXED: Support Lua vector keys in reverse order [ESB3024-780]

Known Limitations

None

1.1.9.4 - Release esb3024-1.6.0

Build date

2023-12-20

Release status

Type: production

Breaking changes

The configuration needs to be manually updated after upgrading to 1.6.0. See configuration changes between 1.4.0 and 1.6.0 for more information.

Change log

NEW: Remove the lua_paths array from the config . Lua scripts are now added using a REST API on the /v1/lua/ endpoint. [ESB3024-204]
NEW: Separate “account-monitor” from installer [ESB3024-238]
NEW: Consistent hashing based routing . Added support for content distribution control for load balancing and cache partitioning [ESB3024-274]
NEW: Predictive load balancing . Account for in-transit traffic to prevent cache overload when there is a sudden burst of new sessions. [ESB3024-275]
NEW: Support Convoy security tokens [ESB3024-386]
NEW: Expose quality, host and session ID in the session object in Lua context [ESB3024-429]
NEW: Support upgrade of system python in installer [ESB3024-442]
NEW: Do not configure selinux and firewalld in installer [ESB3024-493]
NEW: Convoy Distribution/Account integration [ESB3024-503]
NEW: Make eDNS server port configurable . The router configuration hosts.proxy_address has been renamed to hosts.proxy_url and now accepts a port that is used when connecting to the proxy. The cdns.http_port and cdns.https_port configurations now configure the port that is used for connecting to the EDNS server, before they configured the port that was used for connecting to the proxy. [ESB3024-509]
NEW: Expand node table in Lua context . New fields are: node.id, node.visits, host.id, host.recent_selections [ESB3024-630]
FIXED: DNS lookup can fail . DNS lookup can fail when same content requested from both IPv4 and IPv6 clients [ESB3024-427]
FIXED: Failed DNS requests are not retried . Fixed bug where failed eDNS requests were not retried [ESB3024-504]
FIXED: Lua functions are not updated when uploaded [ESB3024-544]
FIXED: Undefined metatable fields evaluate to false rather than nil [ESB3024-642]
FIXED: Evaluator::evaluate() doesn’t support different types of its variadic arguments [ESB3024-687]
FIXED: Segfault when accessing REST api with empty path [ESB3024-752]
FIXED: Container UID/GID may change between versions [ESB3024-755]

1.1.9.5 - Release esb3024-1.4.0

Build date

2023-09-29

Release status

Type: production

Breaking changes

All configuration is now stored under /opt/edgeware/acd, see [ESB3024-425]. Any configuration that is to be kept needs to be manually migrated.
Typically /opt/edgeware/etc/confd/store/store.json needs to be copied to /opt/edgeware/acd/confd/store/store.json, /opt/edgeware/var/lib/acd-router/cached-acd-router-config.json needs to be copied to /opt/edgeware/acd/router/cache/config.json and /opt/edgeware/var/lib/acd-router/cached-router-rest-api-key.json needs to be copied to /opt/edgeware/acd/router/cache/rest-api-key.json. Custom Lua functions need to be migrated from /opt/edgeware/acd/var/lib/custom_lua to /opt/edgeware/acd/router/lib/custom_lua. The Prometheus and Grafana configurations also need to be copied if they have been modified.

The following changes were made to the confcli configuration [ESB3024-455].
The rule fields inside the routing rule items were renamed to condition to avoid confusion with the rules list. This applies to the blocks allow, deny, split and weighted.
The popularityThreshold in the contentPopularity routing rule was renamed to contentPopularityCutoff.

Change log

NEW: 1-Page Status Report . Added command ew-sysinfo that can be used on any machine with an ESB3024 installation. The command outputs various information about the system and installed services which can be used for monitoring and diagnostics. [ESB3024-391]
NEW: Update routing rule property names . Routing rule property names updated for consistency and clarity [ESB3024-455]
FIXED: Deleting confd API array element inside oneOf object fails [ESB3024-355]
FIXED: Container logging not captured by systemd until services are restarted [ESB3024-359]
FIXED: Alertmanager restricts the configuration to a single file [ESB3024-381]
FIXED: Split rules in routing configuration should terminate on error [ESB3024-420]
FIXED: Improve alert configuration in Prometheus [ESB3024-422]
FIXED: Inconsistent storage paths of service configuration and data [ESB3024-425]
FIXED: confd-transformer is not working in el7 [ESB3024-430]

1.1.9.6 - Release acd-router-1.2.3

Build date

2023-08-16

Release status

Type: production

Breaking changes

None

Change log

NEW: Add more classifiers . New classifiers are hostName, contentUrlPath, userAgent, contentUrlQueryParameters [ESB3024-298]
NEW: Add allow- and denylist rule blocks [ESB3024-380]
NEW: Add enhanced validation of scriptable field in routing rules [ESB3024-393]
NEW: Add services to the config tree [ESB3024-410]
NEW: Prohibit unknown configuration properties [ESB3024-416]
FIXED: Duplicate session group IDs are allowed [ESB3024-49]
FIXED: Invalid URL returned for IPv4 requests when using a DNS backend [ESB3024-374]
FIXED: Not possible to set log level in eDNS proxy [ESB3024-378]
FIXED: Instream selection fails when DASH manifest has template paths using “../” [ESB3024-384]

1.1.9.7 - Release acd-router-1.2.0

Build date

2023-06-27

Release status

Type: production

Breaking changes

None

Change log

NEW: Add meta fields to the configuration . The API now allows the meta data fields “created_at”, “source” and “source_checksum” that can be used for the API consumer to track who did what change when.
NEW: Control routing behavior based on backend response code . This gives control over when to return backend response codes to the end user and when to trigger a failover to another CDN or host.
NEW: Manage Lua scripts via API
NEW: Support popularity-based routing . Content can be ordered in multiple groups with descending popularity. Popularity can also be tracked per session group.
NEW: Improved support for IPv6 routing . It is now possible to select backend depending on the IP protocol version.
NEW: Add DNS backend support . This allows delegating routing decisions to an EDNS0 server.
NEW: Support HMAC with SHA256 in Lua scripts
NEW: Add alarm support . The alarms are handled by Prometheus and Alertmanager.
NEW: Support saving Grafana Dashboards
NEW: Add simplified configuration API and CLI tool . A new configuration API with an easier to use model has been added. The “confcli” tool present in many other Edgeware products is now supported.
NEW: Add authentication to the REST API
FIXED: Host headers not forwarded to Request Router when ‘redirecting: true’ is enabled
FIXED: IP range classifier 0.0.0.0/0 does not work in session groups

1.1.9.8 - Release acd-router-1.0.0

First production

Build date

2022-11-22

Release status

Type: First production release

Known Limitations

The setting “allowed_clients” should not be used since the functionality does not work as expected.

Change log

Flexible routing rule engine with support for Lua plugins. Support many use cases, including CDN Offload and CDN Selection.
Advanced client classification mechanisms for routing based on group memberships (device type, content type, etc).
Geobased routing including dedicated high-performing API for subnet matching, associating an incoming request with a region.
Integration API to feed the service with arbitrary variables to use for routing decisions. Can be used to get streaming bitrate in public CDNs, status from network probes, etc.
Flexible request/response translation manipulation on the client facing interface. Can be used for URL manipulation, encoding/decoding tokens or adapting the interface to e.g. the PowerDNS backend protocol.
Metrics API that can be monitored with standard monitoring software. Out-of-the-box integration with Prometheus and Grafana.
Robust deployment with each service instance running independently, and allowing the service to stay in operational state even when backends become temporarily unavailable.
RHEL 7/8 support.
Online documentation at https://docs.agilecontent.com/

1.1.10 - Glossary

ESB3024 Router definitions of commonly used terms

ACD: Agile CDN Director. See “Director”.
Confd: A backend service that hosts the service configuration. Comes with an API, a CLI and a GUI.
Classifier: A filter that associate a request with a tag that can be used to define session groups.
Director: The Agile Delivery OTT router and related services.
ESB: A software bundle that can be separately installed and upgraded, and is released as one entity with one change log. Each ESB is identified with a number. Over time, features and functions within an ESB can change.
Lua: A widely available scripting language that is often used to extend the capabilities of a piece of software.
Router: Unless otherwise specified, an HTTP router that manages an OTT session using HTTP redirect. There are also ways to use DNS instead of HTTP.
Selection Input API: Data posted to this API can be accessed by the routing rules and hence influence the routing decisions.
Subnet API: An API to define mappings between subnets and names (typically regions) for those subnets. Routing rules can then refer to the names rather than the subnets.
Session Group: A handle on a group of requests, defined via classifiers.

1.2 - ESB3032 ACD Aggregator

Aggregates CDN statistics

1.2.1 - Getting Started

Getting started with the Account Aggregator

The account aggregator is a service responsible for monitoring various input streams, compiling and aggregating statistics, and selectively reporting to one or more output streams. It acts primarily as a centralized collector of metrics which may have various aggregations applied before being published to one or more endpoints.

Modes of operation

There are two primary modes of operation, a live-monitoring mode, as well as a reporting mode. The live-monitoring mode measures the account records in real-time, filters, and aggregates the data to the various outputs in real-time. In this mode, only the most recent data will be considered, and any historical context upon startup may be skipped. In the reporting mode, the account record data will be consumed and processed in the order in which they were published to Kafka, and the service will guarantee that all records, still available within the Kafka topic will be processed and reported upon.

Activating the various modes of operation is performed by way of the set of input and output blocks within the configuration file. The file may contain one or more input blocks which specify where the data is sourced, e.g. account records from Kafka, and one or more output blocks which determine how and where the aggregated statistics are published.

While it is possible to specify multiple input and output blocks within a single configuration file, it is highly recommended to separate each pairing of input and output blocks into separate instances running on different nodes. This will yield the best performance and provide for better load balancing, since each instance will be responsible for a single mode of operation.

Real-time account monitoring

In the real-time account monitoring mode, account records, which are sent from each streaming server through the Kafka message broker, are processed by the account aggregator, and current real-time throughput metrics are updated in a Redis database. These metrics, which are constantly being updated, reflect the most current state of the CDN, and can be used by the Convoy Request Router to make real-time routing decisions.

PCX Reporting

In the PCX collector mode, account records are consumed in such a way that past throughput and session statistics can be aggregated to produce billing related reports. These reports are not considered real-time metrics, but represent usage statistics over fixed time intervals. This mode of operation requires a PCX API compatible reporting endpoint. See Appendix B for additional information regarding the PCX reporting format.

Installation

Prerequisites

The account aggregator is shipped as a compressed OCI formatted container image and as such, it requires a supported container runtime such as one of the following:

Docker
Podman
Kubernetes

Any runtime capable of running a Linux container should work the same. For simplicity, the following installation instructions assume that Docker is being used, and that Docker is already configured and running on the target system.

To test that Docker is setup and running, and that the current user has the required privileges to create a container, you may execute the following command.

$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

If you get a permission denied error, ensure that the current user is a member of the docker group or execute all Docker commands under sudo.

Loading the container image

The container image is delivered as a compressed OCI formatted image, which can be loaded directly via the docker load command. The following assumes that the image is in /tmp/esb3032-acd-aggregator-0.0.0.gz

docker load --input /tmp/esb3032-acd-aggregator-0.0.0.gz

You will now be able to verify that the image was loaded successfully by executing the following and looking for the image name in the output.

$ docker images | grep acd-aggregator

images.edgeware.tv/esb3032-acd-aggregator latest   4bbe28b444d3 1 day ago  2.08GB

Creating the configuration file

The configuration file may be located anywhere on the filesystem, however it is recommended to keep everything under the /opt/edgeware/acd/aggregator folder to be consistent with other products under the ACD product family. If that folder doesn’t already exist, you may create the folder with the following command.

mkdir -p /opt/edgeware/acd/aggregator

If using a different location, you will need to map the folder to the container while creating the Docker container. Additional information describing how to map the volume is available in the section “Creating and starting the container” below.

The configuration file for the account aggregator is divided into several sections, input, output and tuning. One or more input blocks may be specified to configure from where the data should be sourced. One or more output blocks may be configured which determine to where the resulting aggregated data is published. Finally the tuning block configures various global settings for how the account aggregator operates, such as the global log_level.

Configuring the input source

As of the current version of the account aggregator, there is only a single type of input source supported, and that is account_records. This input source connects to a Kafka message broker, and consumes account records.
Depending on which output types are configured, the Kafka consumer may either start by processing the oldest or most recent records first.

The following configuration block sample will be used as an example in the description below.

Note that the key input is surrounded by double-square-brackets. This is a syntax element to indicate that there may be multiple input sections in the configuration.

[[input]]
type = "account_records"
servers = [
    "kafka://192.0.2.1:9092",
    "kafka://192.0.2.2:9092",
]
group_name = "acd-aggregator"
kafka_max_poll_interval_ms = 30000
kafka_session_timeout_ms = 3000
log_level = "off"

The type property is used to determine the type of input, and the only valid value is account_records.

The servers list must contain at least 1 Kafka URL, prefixed with the URL scheme kafka://. If not specified, the default Kafka port of 9092 will be used. It is recommended but not required to specify all servers here, as the Kafka client library will obtain the full list of endpoints from the server on startup, however, the initial connection will be made to one or more of the provided URLs.

The group_name property identifies to which consumer group the aggregator should belong. Due to the type of data which account records represent, each instance of the aggregator connecting to the same Kafka message broker MUST have a unique group name. If two instances belong to the same group, the data will be partitioned among both instances, and the resulting aggregations may not be correct. If only a single instance of the account aggregator is used, this property is optional and defaults to “acd-aggregator”.

The kafka_* properties, for max_poll_interval and session_timeout are used to tune the connection parameters for the internal Kafka consumer. More details for these properties can be found in the documentation for the rdkafka library. See Kafka documentation for more details.

The log_level property configures the logging level for the Kafka library and supports the values “off”, “trace”, “debug”, “info”, “warn”, and “error”. By default, logging from this library is disabled. This should only be enabled for troubleshooting purposes, as it is extremely verbose, and any warnings or error messages will be repeated in the account aggregator’s log. The logging level for the Kafka library must be higher then the general logging level for the aggregator, as defined in the “tuning” section or the lower-level messages from the Kafka library will be skipped.

Configuring output

The account aggregator currently supports two types of output blocks, depending on the desired mode of operation. For reference purposes, both types will be described within this section, but it is recommended to only use a single type per instance of the account aggregator.

Note that the key output is surrounded by double-square-brackets. This is a syntax element to indicate that there may be multiple output sections in the configuration.

[[output]]
type = "account_monitor"
redis_servers = [
    "redis://192.0.2.7:6379/0",
    "redis://:password@192.0.2.8:6379/1",
]
stale_threshold_s = 12
throughput_correction_mbps = 0
minimum_check_interval_ms = 1000

[[output]]
type = "pcx_collector"
report_url = "https://192.0.2.5:8000/v1/collector"
client_id = "edgeware"
secret = "abc123"
report_timeout_ms = 2000
report_interval_s = 30
report_delay_s = 30

Real-time account monitor output

The first output block has the type account_monitor and represents the live account monitoring functionality, which publishes per-account bandwidth metrics to one or more Redis servers. When this type of output block is configured, the account records will be consumed starting with the most recent messages first, and offsets will not be committed. Stopping or restarting the service may cause account records to be skipped. This type of output is suitable for making real-time routing decisions, but should not be relied upon for critical billing or reporting metrics.

The redis_servers list consists of URLs to Redis instances which shall be updated with the current real-time bandwidth metrics. If the Redis instance requires authentication, the global instance password can be specified as part of the URL as in the second entry in the list. Since Redis does not support usernames, anything before the : in the credentials part of the URL will be ignored. At least 1 Redis URL must be provided.

The stale_threshold_s property determines the maximum timeout in seconds, after which, if no account records have been received for a given host, the host will be considered stale and removed.

The throughput_correction_mbps property can be used to add or subtract a fixed correction factor to the bandwidth reported in Redis. This is specified in megabits per second, and this may be either positive or negative. If the value is negative, and the calculated bandwidth is less than the correction factor, a minimum bandwidth of 0 will be reported.

The minimum_check_interval_ms property is used to throttle how frequently the statistics will be processed. By default, the account aggregator will not recalculate the statistics more than once per second. Setting this value too low will result in potentially higher CPU usage, while setting it too high may result in some account records being missed. The default of 1 second should be adequate for most situations.

PCX Collector output

The pcx_collector type configures the account aggregator as a reporting

agent for the PCX API. Whenever this configuration is present, the account record consumer will be configured to always start at the oldest records retained within the Kafka topic. It then processes the records one at a time, committing the Kafka offset each time a report is successfully received. This mode does not make any guarantees as to how recent the data is on which the reports are made, but does guarantee that every record will be counted in the aggregated report. Stopping or restarting the service will result in the account record consumer resuming processing from the last successful report. This type of reporting is suitable for billing purposes assuming that there are multiple replicated Kafka nodes, and that the service is not stopped for longer than the maximum retention period configured within Kafka. Stopping the service for longer than the retention period will result in messages being unavailable. Because this type of output requires that the Kafka consumer is processed in a specific order, and will not proceed with reading additional messages until all reports have been successfully received, it is not recommended to have both pcx_collector and the account_monitor type output blocks configured within the same instance.

The report_url property is a single HTTP endpoint URL where the PCX API can be reached. This property is required and may be either an HTTP or HTTPS URL. For HTTPS, the validity of the TLS certificate will be enforced, meaning that self-signed certificates will not be considered valid.

The client_id and secret fields are used to authenticate the client with the PCX API via token-based authentication. These fields are both required, however if not used by the specific PCX API instance, the empty string "" may be provided.

The report_timeout_ms field is an optional maximum timeout for the HTTP connection to the PCX API before the connection will fail. Failed reports will be retried indefinitely.

The report_interval_s property represents the interval bucket size for reporting metrics. The timing for this type of output is based solely on the embedded timestamp value of the account records, meaning that this property is not an absolute period on which the reports will be sent, but instead represents the duration between the start and ending timestamps of the report. Especially upon startup, reports may be sent much more frequently than this interval, but will always cover this duration of time.

The report_delay_s property is an optional offset used to account for both clock synchronization between servers as well as propagation delay of the account records through the message broker. The default delay is 30 seconds. This means that the ending timestamp of a given report will be no more recent than this many seconds in the past. It is important to include this delay, as any account records received with a timestamp that would be within period which has already been reported upon, will be dropped.

Tuning the account aggregator

The tuning configuration block represents the global properties for tuning how the account aggregator functions. Currently only one tuning property can be configured, and that is the log_level. The default log_level is “info”, which should be used in normal operation of the account aggregator, however, other possible values in order of verbosity include “trace”, “debug”, “info”, “warn”, “error”, and “off”.

Note that the tuning key is surrounded by single square-brackets. This is TOML syntax meaning that only one instance of tuning is allowed.

[tuning]
log_level = "info"

Example configurations

This section describes some example configuration files which can be used as a starting template depending on which mode of operation is desired.

Real-time account monitoring

This configuration will consume account records from a Kafka server running on 3 hosts, kafka-1, kafka-2, and kafka-3. The account records will be consumed starting with the most recent records. The resulting aggregations will be published to two Redis instances, running on redis-1 and redis-2. The reported bandwidth will have a 2Gb/s correction factor applied.

[[input]]
type = "account_records"
servers = [
    "kafka://kafka-1:9092",
    "kafka://kafka-2:9092",
    "kafka://kafka-3:9092"
]
group_name = "acd-aggregator-live"
# kafka_max_poll_interval_ms = 30000
# kafka_session_timeout_ms = 3000
# log_level = "off"

[[output]]
type = "account_monitor"
redis_servers = [
    "redis://redis-1:6379/0",
    "redis://redis-2:6379/0",
]
# stale_threshold_s = 12
throughput_correction_mbps = 2000
# minimum_check_interval_ms = 1000

[tuning]
log_level = "info"

The keys prefixed by # are commented out, since the default values will be used. They are included in the example for completeness.

PCX collector

This configuration will consume account records starting from the earliest record, calculate aggregated statistics for every 30 seconds, offset with a delay of 30 seconds, and publish the results to https://pcx.example.com/v1/collector.

[[input]]
type = "account_records"
servers = [
    "kafka://kafka-1:9092",
    "kafka://kafka-2:9092",
    "kafka://kafka-3:9092"
]
group_name = "acd-aggregator-pcx"
# kafka_max_poll_interval_ms = 30000
# kafka_session_timeout_ms = 3000
# log_level = "off"

[[output]]
type = "pcx_collector"
report_url = "https://pcx.example.com/v1/collector"
client_id = "edgeware"
secret = "abc123"
# report_timeout_ms = 2000
# report_interval_s = 30
# report_delay_s = 30

[tuning]
log_level = "info"

The keys prefixed by # are commented out, since the default values will be used. They are included in the example for completeness.

Combined PCX collector with real-time account monitoring

While this configuration is possible, it is not recommended, since the pcx_collector output type will force all records to be consumed starting at the earliest record. This will cause the live statistics to be delayed until ALL earlier records have been consumed, and reports have been successfully accepted by the PCX API. This combined role configuration can be used to minimize the number of servers or services running if the above limitations are acceptable.

Note: This is simply the combination of the above two output blocks in the same configuration file.

[[input]]
type = "account_records"
servers = [
    "kafka://kafka-1:9092",
    "kafka://kafka-2:9092",
    "kafka://kafka-3:9092"
]
group_name = "acd-aggregator-combined"
# kafka_max_poll_interval_ms = 30000
# kafka_session_timeout_ms = 3000
# log_level = "off"

[[output]]
type = "account_monitor"
redis_servers = [
    "redis://redis-1:6379/0",
    "redis://redis-2:6379/0",
]
# stale_threshold_s = 12
throughput_correction_mbps = 2000
# minimum_check_interval_ms = 1000

[[output]]
type = "pcx_collector"
report_url = "https://pcx.example.com/v1/collector"
client_id = "edgeware"
secret = "abc123"
# report_timeout_ms = 2000
# report_interval_s = 30
# report_delay_s = 30

[tuning]
log_level = "info"

Upgrading

The upgrade procedure for the aggregator consists of simply stopping the existing container with docker stop acd-aggregator, removing the existing container with docker rm acd-aggregator, and following the steps in “Creating and starting the container” below with the upgraded Docker image.

To roll back to a previous version, simply perform the same steps with the previous image. It is recommended to keep at least one previous image around until such time that you are satisfied with the new version. After which, you may remove the previous image with docker rmi images.edgeware.tv/esb3032-acd-aggregator:1.2.3 where “1.2.3” represents the previous version number.

Creating and starting the container

Now that the configuration file has been created, and the image has been loaded, we will need to create and start the container instance. The following docker run command will create a new container called “acd-aggregator”, start the process, and automatically resume the container once the Docker daemon is loaded at startup.

docker run \
  --name "acd-aggregator" \
  --detach \
  --restart=always \
  -v <PATH_TO_CONFIG_FOLDER>:/opt/edgeware/acd/aggregator:ro \
  <IMAGE NAME>:<VERSION> \
  --config /opt/edgeware/acd/aggregator/aggregator.toml

As an example using version 1.4.0:

docker run \
  --name "acd-aggregator" \
  --detach \
  --restart=always \
  -v /opt/edgeware/acd/aggregator:/opt/edgeware/acd/aggregator:ro \
  images.edgeware.tv/esb3032-acd-aggregator:1.4.0 \
  --config /opt/edgeware/acd/aggregator/aggregator.toml

Note: The image tag in the example is “1.4.0”, you will need to replace that tag with the image tag loaded from the compressed OCI formatted image file, which can be obtained by running docker images and searching for the account aggregator image as described in the step “Loading the container image” above.

If the configuration file saved in the previous step was at a different location from /opt/edgeware/acd/aggregator/aggregator.toml you will need to change both the -v option and the --config option in the above command to represent that location. The -v option mounts the containing folder from the host system on the left to the corresponding path inside the container on the right, and the :ro tells Docker that the volume is mounted read-only. The --config should be the absolute path to the configuration file from INSIDE the container. For example, if you saved the configuration file as /host/path/config.toml on the host, and you need to map that to /container/path/config.toml within the container, the lines should be -v /host/path:/container/path:ro and --config /container/path/config.toml respectively.

The --restart=always line tells Docker to automatically restart the container when the Docker runtime is loaded, and is the equivalent in systemd to “enabling” the service.

Starting and Stopping the container

To view the status of the running container, use the docker ps command. This will give a line of output for the acd-aggregator container if it is currently running. Appending the -a flag, will list the aggregator container if is not running as well.

Execute the following:

docker ps -a

You should see a line for the container with the container name “acd-aggregator” along with the current state of the container. If all is OK, you should see the container process running at this point, but it may show as “exited” if there was a problem.

To start and stop the container the docker start acd-aggregator and docker stop acd-aggregator commands can be used.

Viewing the logs

By default, Docker will maintain the logs of the individual containers within its own internal logging subsystem, which requires the user to use the command

docker logs

to view them. It is possible however to configure the Docker daemon to send logs to the system journal, however configuring that is beyond the scope of this document. Additional details describing how to do that are described here

[https://docs.docker.com/config/containers/logging/journald/].

To view the complete log for the aggregator the following command can be used.

docker logs acd-aggregator

Supplying the -f flag, can be used to “follow” the log until either the process terminates or CTRL+C is pressed.

docker logs -f acd-aggregator

Appendix A: Real-time account monitoring

Redis key value pairs

Each account will have a single key-value stored in Redis with the current throughput with any correction factor applied, which will be updated in real-time every time all hosts for the given account have received a new account record. This should be approximately every 10 seconds, but may vary slightly due to processing time.

The keys are structured in the following format:

bandwidth:<account>:value

and the value is reported in bits-per-second.

For example for accounts foo, bar and baz we may see the following:

bandwidth:foo:value = 123456789
bandwidth:bar:value = 234567890
bandwidth:baz:value = 102400

These values represent the most current throughput for each account, and will be updated periodically. A TTL of 48 hours is added to the keys, such that they will be pruned automatically after 48 hours since the last update. This is to prevent stale keys from remaining in Redis indefinitely. This TTL is not configurable by the end user.

Appendix B: PCX collector reporting

PCX reporting format

The following is an example of the report sent to the PCX HTTP endpoint.

{
    timestamp_begin: 1674165540,
    timestamp_end: 1674165570,
    writer_id: "writer-1",
    traffic: [
        Traffic {
            account_id: "unknown",
            num_ongoing_sessions: 0,
            bytes_transmitted: 0,
            edges: [
                Edge {
                    server: "orbit-1632",
                    num_ongoing_sessions: 0,
                    bytes_transmitted: 0,
                },
            ],
        },
        Traffic {
            account_id: "default",
            num_ongoing_sessions: 747,
            bytes_transmitted: 75326,
            edges: [
                Edge {
                    server: "orbit-1632",
                    num_ongoing_sessions: 747,
                    bytes_transmitted: 75326,
                },
            ],
        },
    ],
}

The report can be broken down into 3 parts. The outer root section includes the starting and stopping timestamps, as well as a writer_id field which is currently unused. For each account a Traffic section contains the aggregated statistics for that account, as well as a detailed breakdown of each Edge. An Edge is the portion of traffic for the account streamed by each server. Within an Edge the num_ongoing_sessions represents the peak ongoing sessions during the reporting interval, while the bytes_transmitted represents the total egress bandwidth in bytes over the entire period. For each outer Traffic section, the num_ongoing_sessions and bytes_transmitted represent the sum of the corresponding entries in all Edges.

Data protection and consistency

The ACD aggregator works by consuming messages from Kafka. Once a report has successfully been submitted, as determined by a 200 OK HTTP status from the reporting endpoint, the position in the Kafka topic will be committed. This means that if the aggregator process stops and is restarted, reporting will resume from the last successful report, and no data will be lost. There is a limitation to this, however, and that has to do with the data retention time of the messages in Kafka and the TTL value specified in the aggregator configuration. Both default to the same value of 24 hours. This means that if the aggregator process is stopped for more than 24 hours, data loss will result since the source account records will have expired from Kafka before they can be reported on by the aggregator.

Upon startup of the aggregator, all records stored in Kafka will be reported on in the order they are read, starting from either the last successful report or the oldest record currently in Kafka. Reports will be sent each time the timestamp in the current record read from Kafka exceeds the reporting interval meaning a large burst of reports will be sent at startup to cover each interval. Once the aggregator has caught up with the backlog of account records, it will send a single report roughly every 30 seconds (configurable).

It is not recommended to have more than a single account aggregator instance reading from Kafka at a time, as this will result in partial reports being sent to the HTTP endpoint which will require the endpoint to reconstruct the data upon receipt. All redundancy in the account aggregator is handled by the redundancy within Kafka itself. With this in mind, it is important to ensure that there are multiple Kafka instances running and that the aggregator is configured to read from all of them.

1.2.2 - Releases

ESB3032 ACD Aggregator releases

1.2.2.1 - Release esb3032-0.2.0

Build date

2022-12-21

Release status

Type: devdrop

Change log

NEW: Use config file instead of command line switches
NEW: Reports are now aligned with wall-clock time
NEW: Reporting time no longer contains gaps in coverage
FIX: Per-account number of sessions only shows largest host

1.2.2.2 - Release esb3032-1.0.0

First official release

Build date

2023-02-14

Release status

Type: production

Change log

NEW: Create user documentation for ACD Aggregator
NEW: Simplify configuration . Changed from YAML to TOML format.
NEW: Handle account records arriving late
FIXED: Aggregator hangs if committing to Kafka delays more than 5 minutes

1.2.2.3 - Release esb3032-1.2.1

Production release

Build date

2023-04-24

Release status

Type: production

Breaking changes

No breaking changes

Change log

NEW: Port Account Monitor functionality for Convoy Request Router
NEW: Aggregator Performance Improvements
FIXED: Reports lost when restarting acd-aggregator

1.2.2.4 - Release esb3032-1.4.0

Build date

2023-09-28

Release status

Type: production

Breaking changes

None

Change log

NEW: Extend aggregator with additional metrics. Per streamer bandwidth and total bandwidth are now updated in Redis. [ESB3032-98]
FIXED: Not all Redis instances are updated after a failure [ESB3032-99]
FIXED: Kafka consumer restarts on Partition EOF [ESB3032-100]

1.3 - ACD Cache

Information about the ACD Cache Orbit version is available in EDGS-103 Orbit TV Server User Guide¹ and information about the ACD Cache SW Streamer version is available in EDGS-171 SW Streamer User Guide¹.

If you don’t have an Edgeware Support account please contact us here. ↩︎ ↩︎

1.4 - ESB3013 BGP Sniffer

Information about the ACD BGP Sniffer is available in EDGS-214 ESB3013 User Guide¹.

If you don’t have an Edgeware Support account please contact us here. ↩︎

1.5 - Edgeware CDN Management System

Information about the classic Edgeware CDN Management System (aka Convoy) is available in EDGS-069 Convoy Management Software User Guide¹.

If you don’t have an Edgeware Support account please contact us here. ↩︎

1.6 - ESB3008 Edgeware CDN Request Router

Information about the convoy based Edgeware CDN Request Router is available in EDGS-197 ESB3008 HTTP Request Router - User Guide¹.

If you don’t have an Edgeware Support account please contact us here. ↩︎

2 - Introduction

Introduction to ACD - Agile Content Delivery.

ACD is a software suite for CDNs (Content Delivery Networks). The services built around the ACD software come into play when a user starts to view video content on a video streaming portal. The focus is on the server side of the delivery. The portal and players or set-to-boxes are not part of ACD, although they are part of Agile Contents product portfolio as a whole. Likewise, video stream repackaging and DRM are not part of ACD (see ACP).

The ACD software can be used to build a complete private CDN and/or to select between CDNs. A typical deployment is a nation-wide TV streaming service operated by a telecom operator or a broadcaster.

At the core of the system sits ESB3024 Router which will process an incoming client request and make an optimized decision from where to serve the user. The router will classify each playback request according to a variety of configured parameters such as device type, geographic region or content class. Based on the classification together with the current state of the network, its streaming capacity and public offload CDNs status, the best streaming location is determined. If desired, business rules for when to use public CDNs can be applied.

The ACD software suite includes the ACD Cache that is installed on servers placed in data centers that will guarantee an efficient and robust distribution of the content. Depending on the core network architecture, a centralized or decentralized approach is best. Agile Content can help find a good system solution.

Behind the client facing interfaces, there are backend services to monitor and measure the system. This information can be used to troubleshoot network problems, analyze viewing patterns or to plan upgrades of capacity.

Returning to the router, it is built around a very flexible routing rule evaluation engine that can route between and within CDNs. The rule evaluation engine weighs together multiple sources of information to optimize system behavior. Apart from the request classification of device type, content class etc, there are also several integration APIs that can be used to extend the product and solve problems that are particular to a specific customer. One of those integration APIs is the Selection Input API which can take any external information such as streaming bitrate of public CDNs, load balancing settings from a GUI slide bar or status from network probes and include it in the routing decision rules. Another is the Subnet API that can efficiently handle thousands of subnets and map each one to a high-level group membership (typically a region) and route based on that. There is also a metrics API to monitor the service. It integrates out-of-the-box with Prometheus and Grafana.

All APIs are documented under the API overview.

In addition to the flexibility provided by the APIs, there is also a possibility to extend functionality via plugins (using Lua). This can be used to support the decoding and encoding of security tokens (digital signatures of the URLs) or advanced routing rules that are not available out-of-the-box.

A modern TV CDN is considered as critical infrastructure and has very high requirements on uptime and robustness. This is in obtained via multiple independent and stand-alone instances of each service (although particular use cases might require sharing data via a redundant database). The ACD software services run on RHEL8 Linux.

Note: Not all products or components that are part of ACD are documented here. Please contact our Sales for a complete overview.

3 - Solutions

ACD Reference Solutions. Blueprints for different families of use cases.

3.1 - Cache hardware metrics: monitoring and routing

How to set up monitoring and make routing decisions based on cache hardware metrics

Observability and monitoring is important in a CDN. To facilitate this, we use the open source application Telegraf for two key roles: as a metrics agent and as a metrics aggregator. Telegraf performs both these roles depending on its configuration.

The image below demonstrates an example architecture when using multiple caches and instances of ESB3024 Router.

Architecture

Metrics agent

All deployments of ESB2001 Orbit TV Server (from version 3.6.0) and ESB3004 SW Streamer (from version 1.36.0) come bundled with an instance of Telegraf running as a metrics agent, collecting information on hardware metrics such as usage of CPU, memory and network interfaces. If you are using other caches in your CDN, Telegraf can be manually installed and configured.

Telegraf is configured via a configuration daemon, telegraf-configd. On the Orbit TV Server is starts automatically, but on the SW Streamer it needs to be started manually by typing this:

systemctl start telegraf-configd

To make it always start during boot, type

systemctl enable telegraf-configd

Once the configuration daemon is running, Telegraf can be configured using confcli:

$ confcli integration.acd.telegraf
{
  "telegraf": {
    "enable": true,
    "exportUrls": [],
    "hostname": ""
  }
}

To enable or disable the Telegraf instance, type

$ confcli integration.acd.telegraf.enable true
integration.acd.telegraf.enable = True

$ confcli integration.acd.telegraf.enable false
integration.acd.telegraf.enable = false

The Telegraf agents need to export their metrics to an aggregator to be useful. The field integration.acd.telegraf.exportUrls is a list of Telegraf aggregator instances to which the metrics will be exported. It is recommended to use at least two Telegraf aggregator instances for redundancy.

As an example, type this to configure a cache to export metrics to http://aggregator-host-1:8086 and http://aggregator-host-2:8086:

$ confcli integration.acd.telegraf.exportUrls -w
Running wizard for resource 'exportUrls'
<A list of aggregator URLs to export metrics to (default: [])>

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

exportUrls <A list of aggregator URLs to export metrics to (default: [])>: [
  exportUrl (default: ): http://aggregator-host-1:8086
  Add another 'exportUrl' element to array 'exportUrls'? [y/N]: y
  exportUrl (default: ): http://aggregator-host-2:8086
  Add another 'exportUrl' element to array 'exportUrls'? [y/N]: n
]
Generated config:
{
  "exportUrls": [
    "http://aggregator-host-1:8086",
    "http://aggregator-host-2:8086"
  ]
}
Merge and apply the config? [y/n]: y

The Telegraf agent tags all metrics it collects with the hostname of the machine it is running on. To change this hostname, the field integration.acd.telegraf.hostname can be modifed:

$ confcli integration.acd.telegraf.hostname 'new-hostname'
integration.acd.telegraf.hostname = 'new-hostname'

Metrics aggregator

Telegraf is not installed together with the router since it is dependent on your system. Either follow these steps or ask an Agile Content integrator to set it up for your system. To set up Telegraf as an aggregator on your desired hosts, two steps need to be performed.

Install Telegraf according to official instructions.
Configure Telegraf to act as an aggregator by adding the following configuration file: /etc/telegraf/telegraf.conf

Due to a limitation of the [[outputs.http]] plugin, multiple output URLs cannot be defined as a list, the plugin only accepts a single URL. This means that the whole [[outputs.http]] and [outputs.http.headers] needs to be repeated for each router host.

Each router section needs to be modified in two places:

Below [[outputs.http]], the url directive has to be made direct to the router’s address.
Below [outputs.http.headers], X-API-Key has to contain the router’s REST API key. The key is stored in /opt/edgeware/acd/router/cache/rest-api-key.json on the router server.

After the configuration file is complete, type systemctl reload telegraf to make Telegraf reload the new configuration.

When the aggregator is succesfully started and configured, the metrics JSON packet is sent to the selection input API with the following structure:

{
  "cache-1": {
    "hardware_metrics": {
      "/": {
        "free": 18113810432,
        "total": 34887954432,
        "used": 16774144000,
        "used_percent": 48.08004445400899
      },
      "cpu_load1": 0.70,
      "cpu_load5": 0.53,
      "cpu_load15": 0.40,
      "mem_available": 3129425920,
      "mem_available_percent": 76.34786936062332,
      "mem_total": 4098904064,
      "mem_used": 519901184,
      "n_cpus": 2
    },
    "per_interface_metrics": {
      "eth0": {
        "bytes_recv": 57585596566,
        "bytes_recv_rate": 939.4, // bytes per second
        "bytes_sent": 10127106702,
        "bytes_sent_rate": 637.9, // bytes per second
        "drop_in": 1800079,
        "drop_in_rate": 0,
        "drop_out": 0,
        "drop_out_rate": 0,
        "err_in": 0,
        "err_in_rate": 0,
        "err_out": 0,
        "err_out_rate": 0,
        "interface_up": true,
        "link": 1,
        "megabits_recv": 460684,
        "megabits_recv_rate": 0.0075152, // megabits per second
        "megabits_sent": 81016,
        "megabits_sent_rate": 0.0051032, // megabits per second
        "speed": 100
      }
    },
    "timestamp": 1709296800
  }
}

Monitoring

All router installations come bundled with a Prometheus instance that can be configured to scrape the aggregator instances, allowing alarms and visualization of all caches’ metrics.

On a router host, modify the configuration file /opt/edgeware/acd/prometheus/prometheus.yaml to include a scrape job for the aggregators in your system:

global:
  scrape_interval: 15s

rule_files:
  - recording-rules.yaml

scrape_configs:
  - job_name: 'router-scraper'
    scheme: https
    tls_config:
      insecure_skip_verify: true
    static_configs:
    - targets:
      - 10.16.48.106:5001
    metrics_path: /m1/v1/metrics
    honor_timestamps: true
  - job_name: 'edns-proxy-scraper'
    scheme: http
    static_configs:
    - targets:
      - 10.16.48.106:8888
    metrics_path: /metrics
    honor_timestamps: true
  - job_name: 'metric-aggregators'
    scheme: http
    static_configs:
    - targets:
      - http://aggregator-host-1:12001
      - http://aggregator-host-2:12001
    metrics_path: /metrics
    honor_timestamps: true

Once the new configuration file is set, restart the Prometheus instance with systemctl restart acd-prometheus. When the Prometheus instance starts scraping the aggregators, all caches’ metrics will be available for visualization in Grafana and alarms in AlertManager.

Using hardware metrics in routing

When the hardware metrics have been succesfully injected into the selection input API, the data is ready to be used for routing decisions. In addition to creating routing conditions, hardware metrics are particularly suited to host health checks.

Host health checks

When the hardware metrics have been succesfully injected into the selection input API, the data is ready to make routing decisions with. In addition to creating routing conditions, hardware metrics are particularly suited to host health checks.

Host health checks

Host health checks are Lua functions that determine whether or not a host is ready to take on more clients. To configure host health checks, see configuring CDNs and hosts. See Health Check Functions for documentation on how to use the built in health functions. Note that these health check functions can also be used for making routing conditions in the routing tree.

Routing based on cache metrics

Instead of using health check functions to incorporate hardware metrics into the routing decisions, regular routing conditions can be used.

As an example, using the health check function cpu_load_ok() in routing can be configured as follows:

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: firstMatch
  Adding a 'firstMatch' element
    rule : {
      name (default: ): dont_overload_cache
      type (default: firstMatch): 
      targets : [
        target : {
          onMatch (default: ): default_host
          condition (default: always()): cpu_load_ok()
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          onMatch (default: ): offload_host
          condition (default: always()): 
        }
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "dont-overload-cache",
      "type": "firstMatch",
      "targets": [
        {
          "onMatch": "default_host",
          "condition": "cpu_load_ok()"
        },
        {
          "onMatch": "offload_host",
          "condition": "always()"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

{
  "routing": {
    "id": "dont_overload_cache",
    "member_order": "sequential",
    "members": [
      {
        "id": "default-node",
        "host_id": "default-host",
        "weight_function": "return cpu_load_ok()"
      }
      {
        "id": "offload-node",
        "host_id": "offload-host",
        "weight_function": "return 1"
      }
    ]
  }
}

3.2 - Private CDN Offload Routing with DNS

Use CoreDNS and ESB3024 Router to offload traffic from a private CDN

This shows a setup to do DNS based routing and 3rd party CDN offload from a private main CDN. The following components are used in the setup:

A private Edgeware CDN with Convoy (ESB3006), Request Router (ESB3008) and SW Streamers (ESB3004)
ESB3024 Router to offload traffic to an external CDN
CoreDNS to support DNS based routing using ESB3024

The following figure shows an overview of the different components and how they interact.

Solution Overview

A client retrieves a DNS name from it’s content portal. The DNS name is then resolved by CoreDNS that asks the router for a suitable host. The router either returns a host from ESB3008 Request Router or a configured offload host.

The following sequence diagram illustrates the flow.

DNS Resolution Sequence Diagram

Follow the links below to configure different use cases of this setup.

3.3 - Monitor ACD with Prometheus, Grafana and Alert Manager

Use Prometheus, Grafana and Alert Manager to monitor ACD

ACD can be monitored with standard monitoring solutions. This allows you to adapt the monitoring to your needs. The following shows a setup with the following components

Prometheus as metrics database
Grafana to create Dashboards
Alert Manager to create alarms from Prometheus alerts

Overview

Prometheus and Grafana can also be installed with ESB3024 Router installer in which case Grafana is also pre-populated with standard routing dashboards as well as the router troubleshooting Dashboards.

4 - Use Cases

Configuration building blocks for constructing full solutions.

4.1 - Route on GeoIP/ASN

How to write ESB3024 Router configurations for GeoIP and ASN-based routing using classifiers and session groups.

This page describes how to write configuration for GeoIP and ASN-based routing. For configuration in general, see Configuration.

For more details on session groups and classifiers, see Session Groups and Classification.

ASN

Routing on ASN is done through a combination of session group classifiers and Lua script weight functions. We need to create ASN classifier(s) and then associate them with a session group:

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 3
  Adding a 'geoip' element
    classifier : {
      name (default: ): allowed_asn_classifier
      type (default: geoip): ⏎
      inverted (default: False): ⏎
      continent (default: ): ⏎
      country (default: ): ⏎
      cities : [
        city (default: ): ⏎
        Add another 'city' element to array 'cities'? [y/N]: ⏎
      ]
      asn (default: ): Agile*ISP
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "allowed_asn_classifier",
      "type": "geoip",
      "inverted": false,
      "continent": "",
      "country": "",
      "cities": [],
      "asn": "Agile*ISP"
    }
  ]
}
Merge and apply the config? [y/n]: y
$ confcli services.routing.sessionGroups -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): allowed_asn
    classifiers : [
      classifier (default: ): allowed_asn_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "allowed_asn",
      "classifiers": [
        "allowed_asn_classifier"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

{
  "session_groups": [
    {
      "id": 1,
      "name": "allowed_asn",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "allowed_asn_classifier",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "asn": "Agile*ISP"
            }
          }
        ]
      ]
    }
  ]
}

The asn value is a string, matching an ISP name or similar. It supports wildcard matching using asterisks and is case insensitive.

Note that each ASN classifier has to be in its own list. This is because classifiers within the same inner list all have to match for the entire list to be true, and that is impossible for classifiers that match against different ASNs. When the classifiers are in their own lists, it’s enough that one of them matches for the outer classifier list to also match.

Simple Example

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: split
  Adding a 'split' element
    rule : {
      name (default: ): asn_split_node
      type (default: split): ⏎
      rule (default: ): return in_session_group("allowed_asn")
      onMatch (default: ): allowed-asn-host
      onMiss (default: ): offload-host
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "asn_split_node",
      "type": "split",
      "condition": "in_session_group('allowed_asn')",
      "onMatch": "allowed-asn-host",
      "onMiss": "offload-host"
    }
  ]
}
Merge and apply the config? [y/n]: y

{
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "session_groups": [
    {
      "id": 1,
      "name": "allowed_asn",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "allowed_asn_classifier_agile",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "asn": "Agile*ISP"
            }
          }
        ],
        [
          {
            "id": 1,
            "inverted": false,
            "name": "allowed_asn_classifier_edgeware",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "asn": "Edgeware *"
            }
          }
        ]
      ]
    }
  ],
  "cdns": [
    {
      "id": "basic-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "offload-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    }
  ],
  "hosts": [
    {
      "id": "default-host",
      "cdn_id": "basic-cdn",
      "address_family": "ipv4",
      "host": "cdn-host.example"
    },
    {
      "id": "offload-host",
      "cdn_id": "offload-cdn",
      "address_family": "ipv4",
      "host": "offload-host.example"
    }
  ],
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "default-node",
        "host-id": "default-host",
        "weight_function": "return in_session_group('allowed_asn')"
      },
      {
        "id": "offload-node",
        "host_id": "offload-host",
        "weight_function": "return 1"
      }
    ]
  }
}

Geographical Location

Routing on geographical location is done in a similar fashion. Once again, we create location classifier(s) and associate them with a session group:

Geographical locations are provided by a MaxMind database.

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 3
  Adding a 'geoip' element
    classifier : {
      name (default: ): allowed_location
      type (default: geoip): ⏎
      inverted (default: False): ⏎
      continent (default: ): ⏎
      country (default: ): ⏎
      cities : [
        city (default: ): stockholm
        Add another 'city' element to array 'cities'? [y/N]: ⏎
      ]
      asn (default: ): ⏎
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "allowed_location",
      "type": "geoip",
      "inverted": false,
      "continent": "",
      "country": "",
      "cities": [
        "stockholm"
      ],
      "asn": ""
    }
  ]
}
Merge and apply the config? [y/n]: y

  "session_groups": [
    {
      "id": 1,
      "name": "allowed_location",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "allowed_location_classifier",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "continent": "",
              "country": "",
              "region": "",
              "cities": ["stockholm"]
            }
          }
        ]
      ]
    }
  ]

At least one of the optional fields continent, country, region and cities are required. The classifier matches a request when all specified fields do (for the cities list, one matching member is sufficient). All values support wildcard matching using asterisks and are case insensitive.

Note that each location classifier has to be in its own list. This is because classifiers within the same inner list all have to match for the entire list to be true, and that is impossible for classifiers that match against different locations. When the classifiers are in their own lists, it’s enough that one of them matches for the outer classifier list to also match.

Simple Example

confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: split
  Adding a 'split' element
    rule : {
      name (default: ): geographic_location_split_node
      type (default: split): ⏎
      rule (default: ): return in_session_group("allowed_location")
      onMatch (default: ): allowed-location-host
      onMiss (default: ): offload-host
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "geographic_location_split_node",
      "type": "split",
      "condition": "in_session_group('allowed_location')",
      "onMatch": "allowed-location-host",
      "onMiss": "offload-host"
    }
  ]
}
Merge and apply the config? [y/n]: y

{
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "session_groups": [
    {
      "id": 1,
      "name": "allowed_location",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "allowed_location_classifier_sweden",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "country": "Sweden"
            }
          }
        ],
        [
          {
            "id": 1,
            "inverted": false,
            "name": "allowed_location_classifier_oslo",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "country": "Norway",
              "cities": ["Oslo"]
            }
          }
        ]
      ]
    }
  ],
  "cdns": [
    {
      "id": "basic-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "offload-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    }
  ],
  "hosts": [
    {
      "id": "default-host",
      "cdn_id": "basic-cdn",
      "address_family": "ipv4",
      "host": "cdn-host.example"
    },
    {
      "id": "offload-host",
      "cdn_id": "offload-cdn",
      "address_family": "ipv4",
      "host": "offload-host.example"
    }
  ],
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "default-node",
        "host_id": "default-host",
        "weight_function": "return in_session_group('allowed_location')"
      }
      {
        "id": "offload-node",
        "host_id": "offload-host",
        "weight_function": "return 1"
      }
    ]
  }
}

4.2 - Route on Subnet

How to write ESB3024 Router configurations for client routing based on subnets, either by using session groups or by using the subnets API

This page describes how to write configuration for subnet-based routing. For configuration in general, see Configuration.

For details on the subnets API, see Subnets API.

Subnet-based routing can be done either by using session groups and classifiers or by using the subnets API, allowing large amounts of named subnets.

Session Group Classifiers

Routing on subnets can be done through a combination of session group classifiers and Lua script weight functions. Configuring session groups and classifiers based on subnets and using them in routing may look like:

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 5
  Adding a 'ipranges' element
    classifier : {
      name (default: ): ip_ranges_classifier
      type (default: ipranges): ⏎
      inverted (default: False): ⏎
      ipranges : [
        iprange (default: ): 10.0.0.0/8
        Add another 'iprange' element to array 'ipranges'? [y/N]: y
        iprange (default: ): 192.168.0.0/16
        Add another 'iprange' element to array 'ipranges'? [y/N]: y
        iprange (default: ): 2001:0db8:85a3::/128
        Add another 'iprange' element to array 'ipranges'? [y/N]: ⏎
      ]
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "ip_ranges_classifier",
      "type": "ipranges",
      "inverted": false,
      "ipranges": [
        "10.0.0.0/8",
        "192.168.0.0/16",
        "2001:0db8:85a3::/128"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.sessionGroups -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): allowed_subnets
    classifiers : [
      classifier (default: ): ip_ranges_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "allowed_subnets",
      "classifiers": [
        "ip_ranges_classifier"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

{
  "session_groups": [
    {
      "id": 1,
      "name": "allowed_subnets",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "allowed_subnet_classifier",
            "rule": {
              "rule_type": "ip_ranges_rule",
              "source": "session/client_ip",
              "ip_ranges": ["10.0.0.0/8", "192.168.0.0/16", "2001:0db8:85a3::/128"]
            }
          }
        ]
      ]
    }
  ]
}

The field ip_ranges is a list of CIDR-notation strings, supporting both IPv4 and IPv6 format. Incoming requests have their IP tested against the listed ranges. The classifier evaluates as true if any of the ranges in the classifier’s list matches the IP address of the request.

These session groups can then be used in routing as follows:

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: split
  Adding a 'split' element
    rule : {
      name (default: ): subnet_split_node
      type (default: split): ⏎
      rule (default: ): return in_session_group('allowed_subnets')
      onMatch (default: ): allowed-host
      onMiss (default: ): offload-host
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "subnet_split_node",
      "type": "split",
      "condition": "in_session_group('allowed_subnets')",
      "onMatch": "allowed-host",
      "onMiss": "offload-host"
    }
  ]
}
Merge and apply the config? [y/n]: y
$ confcli services.routing.entrypoint subnet_split_node
services.routing.entrypoint = 'subnet_split_node'

{
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "cdns": [
    {
      "id": "allowed-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "offload-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    }
  ],
  "hosts": [
    {
      "id": "allowed-host",
      "cdn_id": "allowed-cdn",
      "host": "allowed-host.example"
    },
    {
      "id": "offload-host",
      "cdn_id": "offload-cdn",
      "host": "offload-host.example"
    }
  ],
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "allowed-node",
        "host_id": "allowed-host",
        "weight_function": "return in_session_group('allowed_subnets')"
      },
      {
        "id": "offload-node",
        "host_id": "offload-host",
        "weight_function": "return 1"
      }
    ]
  },
  "session_groups": [
    {
      "id": 1,
      "name": "allowed_subnets",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "allowed_subnet_classifier",
            "rule": {
              "rule_type": "ip_ranges_rule",
              "source": "session/client_ip",
              "ip_ranges": ["10.0.0.0/8", "192.168.0.0/16", "2001:0db8:85a3::/128"]
            }
          }
        ]
      ]
    }
  ]
}

Named Subnets

Named subnets are injected into the router in the form of JSON payloads to the Subnets API. Once the subnet data has been fed to the router, Lua functions can access the subnet name associated with an incoming request and use the result when it performs routing.

Note that confcli cannot be used to inject named subnets into the router.

Assume we have injected the following subnet configuration into the router:

{
  "10.0.0.0/8": "test_net_4",
  "192.168.0.0/16": "test_net_4",
  "2001:0db8:85a3::/128": "test_net_6"
}

A router configuration using this subnet configuration can then be constructed as:

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: split
  Adding a 'split' element
    rule : {
      name (default: ): subnet_split_node
      type (default: split): ⏎
      rule (default: ): return in_subnet("test_net_4")
      onMatch (default: ): allowed-host-4
      onMiss (default: ): offload-host
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "subnet_split_node",
      "type": "split",
      "condition": "in_subnet('test_net_4')",
      "onMatch": "allowed-host-4",
      "onMiss": "offload-host"
    }
  ]
}
Merge and apply the config? [y/n]: y
$ confcli services.routing.entrypoint subnet_split_node
services.routing.entrypoint = 'subnet_split_node'

{
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "cdns": [
    {
      "id": "allowed-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "offload-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    }
  ],
  "hosts": [
    {
      "id": "allowed-host-4",
      "cdn_id": "allowed-cdn",
      "address_family": "ipv4",
      "host": "allowed-host4.example"
    },
    {
      "id": "allowed-host-6",
      "cdn_id": "allowed-cdn",
      "address_family": "ipv6",
      "host": "allowed-host6.example"
    },
    {
      "id": "offload-host",
      "cdn_id": "offload-cdn",
      "address_family": "ipv4",
      "host": "offload-host.example"
    }
  ],
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "allowed-node-4",
        "host_id": "allowed-host-4",
        "weight_function": "return in_subnet('test_net_4')"
      },
      {
        "id": "allowed-node-6",
        "host_id": "allowed-host-6",
        "weight_function": "return in_subnet('test_net_6')"
      },
      {
        "id": "offload-node",
        "host_id": "offload-host",
        "weight_function": "return 1"
      }
    ]
  }
}

Note the absence of session groups in this configuration, instead relying solely on the named subnet API to making routing decisions based on subnets.

4.3 - Route on Content Type

How to write ESB3024 Router configurations for content-based routing

This page describes how to write configuration for content-based routing. For configuration in general, see Configuration.

Two ways to route on content type will be demonstrated: content path based and hostname based using session groups and classifiers. For more details on session groups and classifiers, see Session Groups and Classification.

Content Path

Routing on content path is done through a combination of session group classifiers and Lua script weight functions. Two suitable classifiers for content path matching are string_match_rule and regex_rule.

The pattern is a string matching the path segment of a request URL, including the file name. When selecting the string_match_rule type, use asterisks for wildcard matching. When selecting the regex_rule follow the C++11 std::regex ECMAScript syntax to make valid patterns. Make sure to write patterns that match the entire path segment, not just part of it.

Note that each content classifier normally has to be in its own list. This is because classifiers within the same inner list all have to match for the entire list to be true, and that is impossible for classifiers that match against different content paths. When the classifiers are in their own lists, it’s enough that one of them matches for the outer classifier list to also match.

Simple Example

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 1
  Adding a 'contentUrlPath' element
    classifier : {
      name (default: ): live_content_classifier
      type (default: contentUrlPath): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): ⏎
      pattern (default: ): *live_tv*
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: y
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 1
  Adding a 'contentUrlPath' element
    classifier : {
      name (default: ): news_content_classifier
      type (default: contentUrlPath): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): regex
      pattern (default: ): /([^/]+/)?news_reports_\\d+/.*
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: y
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 1
  Adding a 'contentUrlPath' element
    classifier : {
      name (default: ): hls_content_classifier
      type (default: contentUrlPath): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): ⏎
      pattern (default: ): *.m3u8
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: n
]
Generated config:
{
  "classifiers": [
    {
      "name": "live_content_classifier",
      "type": "contentUrlPath",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "*live_tv*"
    },
    {
      "name": "news_content_classifier",
      "type": "contentUrlPath",
      "inverted": false,
      "patternType": "regex",
      "pattern": "/([^/]+/)?news_reports_\\\\d+/.*"
    },
    {
      "name": "hls_content_classifier",
      "type": "contentUrlPath",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "*.m3u8"
    }
  ]
}
Merge and apply the config? [y/n]: y

confcli services.routing.sessionGroups -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): live_content
    classifiers : [
      classifier (default: ): live_content_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: y
  sessionGroup : {
    name (default: ): news_content
    classifiers : [
      classifier (default: ): news_content_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: y
  sessionGroup : {
    name (default: ): hls_content
    classifiers : [
      classifier (default: ): hls_content_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "live_content",
      "classifiers": [
        "live_content_classifier"
      ]
    },
    {
      "name": "news_content",
      "classifiers": [
        "news_content_classifier"
      ]
    },
    {
      "name": "hls_content",
      "classifiers": [
        "hls_content_classifier"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

{
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "session_groups": [
    {
      "id": 1,
      "name": "live_content",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "live_content_classifier",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/content_url_path",
              "pattern": "*live_tv*"
            }
          }
        ]
      ],
      [
        [
          {
            "id": 2,
            "inverted": false,
            "name": "news_content_classifier",
            "rule": {
              "rule_type": "regex_rule",
              "source": "session/content_url_path",
              "pattern": "/([^/]+/)?news_reports_\\d+/.*"
            }
          }
        ]
      ]
    },
    {
      "id": 2,
      "name": "hls_content",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "hls_content_classifier",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/content_url_path",
              "pattern": "*.m3u8"
            }
          }
        ]
      ]
    }
  ],
  "cdns": [
    {
      "id": "live-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "vod-hls-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "vod-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    }
  ],
  "hosts": [
    {
      "id": "live-host",
      "cdn_id": "live-cdn",
      "address_family": "ipv4",
      "host": "live-host.example"
    },
    {
      "id": "vod-hls-host",
      "cdn_id": "vod-hls-cdn",
      "address_family": "ipv4",
      "host": "vod-hls-host.example"
    },
    {
      "id": "vod-host",
      "cdn_id": "vod-cdn",
      "address_family": "ipv4",
      "host": "vod-host.example"
    }
  ],
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "live-node",
        "host_id": "live-host",
        "weight_function": "return session_groups.live_content and 1 or 0"
      },
      {
        "id": "vod-hls-node",
        "host_id": "vod-hls-host",
        "weight_function": "return session_groups.hls_content and 1 or 0"
      },
      {
        "id": "vod-node",
        "host_id": "vod-host",
        "weight_function": "return 1"
      }
    ]
  }
}

Hostname

In cases where there are separate domain or subdomain names for different content types, it’s simple to make classifiers that filter on those names.

The pattern is a string matching the hostname segment of a request URL, including the file name. When selecting the string_match_rule type, use asterisks for wildcard matching. When selecting the regex_rule follow the C++11 std::regex ECMAScript syntax to make valid patterns. Make sure to write patterns that match the entire hostname segment, not just part of it.

Note that each hostname classifier has to be in its own list. This is because classifiers within the same inner list all have to match for the entire list to be true, and that is impossible for classifiers that match against different hostnames. When the classifiers are in their own lists, it’s enough that one of them matches for the outer classifier list to also match.

Simple example

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 4
  Adding a 'hostName' element
    classifier : {
      name (default: ): live_content_classifier
      type (default: hostName): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): ⏎
      pattern (default: ): live.example.com
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: y
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 4
  Adding a 'hostName' element
    classifier : {
      name (default: ): news_content_classifier
      type (default: hostName): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): regex
      pattern (default: ): /([^\\.]+/)\\.news_reports_\\d+/\\.example\\.com
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "live_content_classifier",
      "type": "hostName",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "live.example.com"
    },
    {
      "name": "news_content_classifier",
      "type": "hostName",
      "inverted": false,
      "patternType": "regex",
      "pattern": "/([^\\\\.]+/)\\\\.news_reports_\\\\d+/\\\\.example\\\\.com"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.sessionGroups -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): live_content
    classifiers : [
      classifier (default: ): live_content_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: y
  sessionGroup : {
    name (default: ): news_content
    classifiers : [
      classifier (default: ): news_content_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: y
  sessionGroup : {
    name (default: ): hls_content
    classifiers : [
      classifier (default: ): hls_content_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "live_content",
      "classifiers": [
        "live_content_classifier"
      ]
    },
    {
      "name": "news_content",
      "classifiers": [
        "news_content_classifier"
      ]
    },
    {
      "name": "hls_content",
      "classifiers": [
        "hls_content_classifier"
      ]
    }
  ]
}

{
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "session_groups": [
    {
      "id": 1,
      "name": "live_content",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": false,
            "name": "live_content_classifier",
            "rule": {
              "rule_type": "string_match_rule",
              "source": "session/hostname",
              "pattern": "live.example.com"
            }
          }
        ],
        [
          {
            "id": 2,
            "inverted": false,
            "name": "news_content_classifier",
            "rule": {
              "rule_type": "regex_rule",
              "source": "session/hostname",
              "pattern": "/([^\\.]+/)\\.news_reports_\\d+/\\.example\\.com"
            }
          }
        ]
      ]
    }
  ],
  "cdns": [
    {
      "id": "live-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "vod-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    }
  ],
  "hosts": [
    {
      "id": "live-host",
      "cdn_id": "live-cdn",
      "address_family": "ipv4",
      "host": "live-host.example"
    },
    {
      "id": "vod-host",
      "cdn_id": "vod-cdn",
      "address_family": "ipv4",
      "host": "vod-host.example"
    }
  ],
  "routing": {
    "id": "routing_table",
    "member_order": "sequential",
    "members": [
      {
        "id": "live-node",
        "host_id": "live-host", 
        "weight_function": "return session_groups.live_content and 1 or 0"
      },
      {
        "id": "vod-node",
        "host_id": "vod-host",  
        "weight_function": "return 1"
      }
    ]
  }
}

Using the session groups in confcli

The constructed session groups can be used in routing by creating a rule that references them:

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: firstMatch
  Adding a 'firstMatch' element
    rule : {
      name (default: ): classifier_routing_rule
      type (default: firstMatch): ⏎
      targets : [
        target : {
          onMatch (default: ): live-host
          rule (default: ): in_all_session_groups('live_content', 'news_content')
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          onMatch (default: ): vod-hls-host
          rule (default: ): in_session_group('hls_content')
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          onMatch (default: ): offload-host
          rule (default: ): always()
        }
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "classifier_routing_rule",
      "type": "firstMatch",
      "targets": [
        {
          "onMatch": "live-host",
          "condition": "in_all_session_groups('live_content', 'news_content')"
        },
        {
          "onMatch": "vod-hls-host",
          "condition": "in_session_group('hls_content')"
        },
        {
          "onMatch": "offload-host",
          "condition": "always()"
        }
      ],
    }
  ]
}
Merge and apply the config? [y/n]: y
$ confcli services.routing.entrypoint classifier_routing_rule
services.routing.entrypoint = classifier_routing_rule

4.4 - Route on Content Popularity

How to write ESB3024 Router configurations for content popularity based routing

This page describes how to write configuration for content popularity routing. For configuration in general, see Configuration.

For more details on content popularity tuning and routing, see

Content Popularity.

Content Popularity

The router tracks content popularity which can be utilized for routing. Using the configuration tool confcli, creating a content popularity based routing configuration for hierarchical and multi-edge scenarios will be demonstrated.

Hierarchical

Consider a CDN setup with edge streamers that has cached popular content and a central streamer where all content is available. You can decide where to route clients based on the requested content’s popularity.

Assuming that appropriate streamer hosts have already been configured, configuration will look like:

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: contentPopularity
  Adding a 'contentPopularity' element
    rule : {
      name (default: ): contentPopularityHierarchy
      type (default: contentPopularity): ⏎
      popularityThreshold (default: 10): 2000
      onPopular (default: ): edgeStreamer
      onUnpopular (default: ): centralStreamer
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "contentPopularityHierarchy",
      "type": "contentPopularity",
      "popularityThreshold": 2000.0,
      "onPopular": "edgeStreamer",
      "onUnpopular": "offloadStreamer"
    }
  ]
}
Merge and apply the config? [y/n]: y

where name is the name of the rule, isPopular is the rule to route to if the content is popular, otherwise the rule isUnpopular is routed to. Lastly, popularityThreshold is the threshold for which content is considered popular. Configuring popularityThreshold = 2001 means that the top 2000 most popular assets will be routed to edgeStreamer.

The rule contentPopularityHierarchy can then be used to construct your routing tree.

Multi-edge

Consider a CDN setup with three edge streamers, edge1, edge2 and edge3, configured to cache very popular content, mildly popular content and unpopular content respectively. Assuming that appropriate streamer hosts have already been configured, the following configuration can be used for correctly routing a request in this scenario:

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: contentPopularity
  Adding a 'contentPopularity' element
    rule : {
      name (default: ): mildlyAndUnpopular
      type (default: contentPopularity): ⏎
      popularityThreshold (default: 10): 2001
      onPopular (default: ): edge2
      onUnpopular (default: ): edge3
    }
  Add another 'rule' element to array 'rules'? [y/N]: y
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: contentPopularity
  Adding a 'contentPopularity' element
    rule : {
      name (default: ): multiLevelPopularity
      type (default: contentPopularity): ⏎
      popularityThreshold (default: 10): 101
      onPopular (default: ): edge1
      onUnpopular (default: ): mildyAndUnpopular
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "mildlyAndUnpopular",
      "type": "contentPopularity",
      "popularityThreshold": 2001.0,
      "onPopular": "edge2",
      "onUnpopular": "edge3"
    },
    {
      "name": "multiLevelPopularity",
      "type": "contentPopularity",
      "popularityThreshold": 101.0,
      "onPopular": "edge1",
      "onUnpopular": "mildyAndUnpopular"
    }
  ]
}
Merge and apply the config? [y/n]: y

By configuring a rule to route to multiLevelPopularity, this configuration will route requests for the top 100 most popular content to edge1, popularity ranking 101-2000 to edge2 and popularity ranking > 2000 to edge3. Since the contentPopularity rule offers a binary routing choice of either isPopular or isUnpopular, multiple contentPopularity rules can be utilized to construct multi-leveled content popularity based routing configurations.

Application

To use any of these two configurations in your installation, you’ll need to configure either a rule or the entrypoint to route to either multiLevelPopularity or contentPopularityHierarchy.

4.5 - Route on Selection Input

How to write ESB3024 Router configurations to route on selection input, using CDN bandwidth and number of ongoing sessions as example.

This page describes how to write configuration for routing on bandwidth usage or number of ongoing sessions in Edgeware CDN:s by using the selection input API. For configuration in general, see Configuration.

For more details on the selection input API, see Selection Input.

Selection Input

The selection input API allows you to inject custom data, that can be used during routing, into the router. For this use case, we will use CDN bandwidth and the number of ongoing sessions to demonstrate the capabilities of the selection input API.

Imagine that we have a monitor constantly polling the bandwidth and ongoing sessions of hosts in a CDN. These values are injected into the selection input API as JSON packets:

{
  "edge-host-available-bps": 1000000000,
  "edge-host-ongoing-sessions": 300
}

These keys will end up in the router Lua context as “selection inputs”, arbitrary values or JSON objects that can be used to calculate routing weights.

The selection input variable edge-host-available-bps will be available through either selection_input.edge-host-available-bps or si('edge-host-available-bps'). The function si(si_var) fetches the selection input value if it exists, otherwise it returns 0. However, when comparing selection input variables with numbers, there are a number of built-in Lua functions that will simplify things, such as gt() and lt(). For a complete list and more details on these functions, see Built-in Functions.

Simple Example

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: firstMatch
  Adding a 'firstMatch' element
    rule : {
      name (default: ): selection-input-routing
      type (default: firstMatch): ⏎
      targets : [
        target : {
          onMatch (default: ): edge-host
          rule (default: ): lt('edge_host_ongoing_sessions', 1000)
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          onMatch (default: ): edge-host
          rule (default: ): gt('edge_host_available_bps', 5000000)
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          onMatch (default: ): offload-host
          rule (default: ): always()
        }
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
      onMiss (default: ): offload-host
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "selection-input-routing",
      "type": "firstMatch",
      "targets": [
        {
          "onMatch": "edge-host",
          "condition": "lt('edge_host_ongoing_sessions', 1000)"
        },
        {
          "onMatch": "edge-host",
          "condition": "gt('edge_host_available_bps', 5000000)"
        },
        {
          "onMatch": "offload-host",
          "condition": "always()"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y
$ confcli services.routing.entrypoint selection-input-routing
services.routing.entrypoint = 'selection-input-routing'

{
  "content_server": {
    "http_enable": true,
    "http_port": 80,
    "https_enable": true,
    "https_port": 443
  },
  "cdns": [
    {
      "id": "edge-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "offload-cdn",
      "http_port": 80,
      "https_port": 443,
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    }
  ],
  "hosts": [
    {
      "id": "edge-host",
      "cdn_id": "edge-cdn",
      "host": "edge-host.example"
    },
    {
      "id": "offload-host",
      "cdn_id": "offload-cdn",
      "host": "offload-host.example"
    }
  ],
  "routing": {
    "id": "routing-table",
    "member_order": "sequential",
    "members": [
      {
        "id": "edge-host-sessions-available",
        "host_id": "edge-host",
        "weight_function": "return (lt('edge_host_ongoing_sessions', 1000) ~= 0) and 1 or 0"
      },
      {
        "id": "edge-host-bandwith-available",
        "host_id": "edge-host",
        "weight_function": "return (gt('edge_host_available_bps', 5000000) ~= 0) and 1 or 0"
      },
      {
        "id": "offload",
        "host_id": "offload-host",
        "weight_function": "return 1"
      }
    ]
  }
}

4.6 - Adapt to multi-CDN

How to write ESB3024 Router configuration to handle multi-CDN setups with tokens.

This page describes how to write configuration for setups that require the URL sent back to the client to be rewritten based on routing results, for example by adding a token to the path or as a query string parameter. For configuration in general, see Configuration.

In order to change the redirect location URL, we need to add a response translation function to the confd configuration. By default, all translation functions are empty:

$ confcli services.routing.translationFunctions
{
    "translationFunctions": {
        "request": "",
        "response": ""
    }
}

For full documentation of the response translation function, please see Lua Hooks.

Simple example

Translation functions tend to become large, which makes it hard to write them inside a single string. Instead, the function can be written in a separate .lua script file that is uploaded to the router and referenced from the confd configuration.

The following example will take this approach, adding a script called "rewrite.lua" to the configured custom Lua folder /tmp/custom_lua on the router. For more information see the Configuration JSON Overview.

Lua script

The following script assumes the existence of the function parse_url that returns an object with URL segments that can be altered and then combined into a URL string using the tostring(obj) function again.

Beware that the router does not validate the URL in the Location header, it is up to the Lua script writer to ensure that only URLs are returned.

function response_rewrite(Headers)
   -- The live server has the token as part of the path, added first before the
   -- path from the request.
   local location = parse_url(response_headers.location)

   if location.host == "live-host.example" then
      location.path = "/live-prefix" .. location.path
      return HTTPResponse(
         {
            Headers = {
               {"location", tostring(location)}
            }
         }
      )
   end

   -- The VOD server expects the token as a query string rather than as part
   -- of the path segment.
   -- Since VOD can be used for offload, check that session_groups.is_vod
   -- is set as well.
   if location.host == "vod-host.example" and session_groups.is_vod then
      location.query["token"] = md5sum(secret_key .. request.path)
      return HTTPResponse(
         {
            Headers = {
               {"location", tostring(location)}
            }
         }
      )
   end

   -- If the request was neither in the is_live session group, nor contained
   -- the path element "vod", we return a nil object to indicate that we don't
   -- want to change anything in the response.
   return nil
end

Configuration

To configure a response translation function using with the custom Lua function response_rewrite() using confcli, run the command:

$ confcli services.routing.translationFunctions.response 'return response_rewrite()'
services.routing.translationFunctions.response = 'return response_rewrite()'

An example confd configuration utilizing this translation function based on session groups would look like this:

$ confcli services.routing
{
  "translationFunctions": {
    "request": "",
    "response": "return response_rewrite()"
  },
  "sessionGroups": [
    {
      "name": "is_live",
      "classifiers": [
          "is_live_classifier"
      ]
    },
    {
      "name": "is_vod",
      "classifiers": [
          "is_vod_classifier"
      ]
    }
  ],
  "classifiers": [
    {
      "name": "is_live_classifier",
      "type": "contentUrlPath",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "*/live/*"
    },
    {
      "name": "is_vod_classifier",
      "type": "contentUrlPath",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "*/vod/*"
    }
  ],
  "hostGroups": [
    {
      "name": "live-cdn",
      "type": "host",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
          {
              "name": "live-host",
              "hostname": "live-host.example",
              "ipv6_address": ""
          }
      ]
    },
    {
      "name": "vod-cdn",
      "type": "host",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
          {
              "name": "vod-host",
              "hostname": "vod-host.example",
              "ipv6_address": ""
          }
      ]
    }
  ],
  "rules": [
    {
      "name": "multiCdnExample",
      "type": "firstMatch",
      "targets": [
        {
          "condition": "in_session_group('is_live')",
          "onMatch": "live-host"
        },
        {
          "condition": "in_session_group('is_vod')",
          "onMatch": "vod-host"
        },
        // Use the vod-host as offload for any content, but don't set any
        // token for requests not classified as vod.
        {
          "condition": "always()",
          "onMatch": "vod-host"
        }
      ]
    }
  ],
  "entrypoint": "multiCdnExample",
  "applyConfig": true
}

4.7 - How to use ACD router for EDNS routing

How to write ACD Route configurations for EDNS routing

This page describes how to write configuration for EDNS routing. For configuration in general, see Configuration.

EDNS routing

In your CDN architecture, you might utilize EDNS routing to redirect clients. Whether it’s your own CDN or a third party CDN, the ESB3024 Router supports creating EDNS requests and sending them to any DNS server.

Configuration

To configure EDNS routing, you first need to create a DNS type host. Using confcli:

[user@acd-router ~]# confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: 1
  Adding a 'dns' element
    hostGroup : {
      name (default: ): edns-cdn
      type (default: dns): ⏎
      hosts : [
        host : {
          name (default: ): edns-host
          hostname (default: ): ednshost.com
          ipv6_address (default: ): ⏎
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: ⏎
]
Generated config:
{
  "hostGroups": [
    {
      "name": "edns-cdn",
      "type": "dns",
      "hosts": [
        {
          "name": "edns-host",
          "hostname": "ednshost.com",
          "ipv6_address": ""
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

Note that the hostname is the name of the DNS server that will receive the EDNS request. You can now use the DNS host when creating the routing tree, e.g. in a random type rule:

[user@router ~]# confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: random
  Adding a 'random' element
    rule : {
      name (default: ): random-rule
      type (default: random): ⏎
      targets : [
        target (default: ): edns-host
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "random-rule",
      "type": "random",
      "targets": [
        "edns-host"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y
[user@router ~]# confcli services.routing.entrypoint random-rule
services.routing.entrypoint = 'random-rule'

Or simply use it as the entrypoint:

confcli services.routing.entrypoint edns-host

4.8 - How to use ESB3024 Router with Edgeware CDN Request Router

How to use ESB3024 Router with ESB3008 Request Router

This page describes how to write configuration for using ESB3024 Router in conjunction with ESB3008 Request Router. It can request a host from an ESB3008 Request Router and forward the response to the client, perform load balancing between several Request Routers and handle fallback to an external CDN in case no Request Routers are able to service the request.

Simple Example

# The tuning parameter `target.requestAttempts` controls the number of attempts
# made to find a host. In this example we want to fallback to the external CDN
# in case both Request Routers are down, so we set it to at least 3.

$ confcli services.routing.tuning.target.requestAttempts 3
services.routing.tuning.target.requestAttempts = 3

# Create a classifier to filter incoming requests from Sweden.

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: geoip
  Adding a 'geoip' element
    classifier : {
      name (default: ): sweden_classifier
      type (default: geoip): ⏎
      inverted (default: False): ⏎
      continent (default: ): ⏎
      country (default: ): sweden
      cities : [
        city (default: ): ⏎
        Add another 'city' element to array 'cities'? [y/N]: ⏎
      ]
      asn (default: ): ⏎
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "sweden_classifier",
      "type": "geoip",
      "inverted": false,
      "continent": "",
      "country": "sweden",
      "cities": [
        ""
      ],
      "asn": ""
    }
  ]
}
Merge and apply the config? [y/n]: y

# Create a session group for the non-Swedish requests, to use in routing rules

$ confcli services.routing.sessionGroups -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): sweden
    classifiers : [
      classifier (default: ): sweden_classifier
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "sweden",
      "classifiers": [
        "sweden_classifier"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

# The internal CDN uses Edgeware Request Routers that return a redirect
# location.  The router takes the content from the client request,
# and makes its own request to the Request Router using that same
# content.  The returned location is then forwarded to the client.
#
# In case the Request Router responds with an error code, or times out,
# the router will continue traversing the routing tree until a
# successful route is found or the entire tree has been evaluated.

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: 3
  Adding a 'redirecting' element
    hostGroup : {
      name (default: ): internal-cdn
      type (default: redirecting): ⏎
      httpPort (default: 80): ⏎
      httpsPort (default: 443): ⏎
      hosts : [
        host : {
          name (default: ): internal-host-1
          hostname (default: ): internal-host-1.example.com
          ipv6_address (default: ): ⏎
        }
        Add another 'host' element to array 'hosts'? [y/N]: y
        host : {
          name (default: ): internal-host-2
          hostname (default: ): internal-host-2.example.com
          ipv6_address (default: ): ⏎
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: ⏎
]
Generated config:
{
  "hostGroups": [
    {
      "name": "internal-cdn",
      "type": "redirecting",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
        {
          "name": "internal-host-1",
          "hostname": "internal-host-1.example.com",
          "ipv6_address": ""
        },
        {
          "name": "internal-host-2",
          "hostname": "internal-host-2.example.com",
          "ipv6_address": ""
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

# The offload CDN is non-redirecting, meaning that router does not
# make any request to its associated hosts to ask for a redirect location
# to forward to the clients, instead it simply returns a location using
# one of the CDN's hosts substituted for the router's host in the
# client request possibly with some kind of respsonse translation to add
# e.g. tokens or path prefixes.

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: 2
  Adding a 'host' element
    hostGroup : {
      name (default: ): offload-cdn
      type (default: host): ⏎
      httpPort (default: 80): ⏎
      httpsPort (default: 443): ⏎
      hosts : [
        host : {
          name (default: ): offload-host
          hostname (default: ): offload-host.example.com
          ipv6_address (default: ): ⏎
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: ⏎
]
Generated config:
{
  "hostGroups": [
    {
      "name": "offload-cdn",
      "type": "host",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
        {
          "name": "offload-host",
          "hostname": "offload-host.example.com",
          "ipv6_address": ""
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

# Begin with adding a load balancing route rule between the two internal
# hosts. Let's make internal-host-2 have twice the capacity of
# internal-host-1 by using a weighted rule.

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: weighted
  Adding a 'weighted' element
    rule : {
      name (default: ): balancer
      type (default: weighted): ⏎
      targets : [
        target : {
          target (default: ): internal-host-1
          weight (default: 100): 1,
          rule (default: always()): always()
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): internal-host-2
          weight (default: 100): 2,
          rule (default: always()): always()
        }
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "balancer",
      "type": "weighted",
      "targets": [
        {
          "target": "internal-host-1",
          "weight": "1",
          "rule:": "always()"
        },
        {
          "target": "internal-host-2",
          "weight": "2",
          "rule:": "always()"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

# Make an offload rule to route any traffic not in Sweden to the
# offload CDN.

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: firstMatch
  Adding a 'firstMatch' element
    rule : {
      name (default: ): offload
      type (default: firstMatch): ⏎
      targets : [
        target : {
          onMatch (default: ): offload-host
          rule (default: ): not in_session_group('sweden')
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          onMatch (default: ): balancer
          rule (default: ): in_session_group('sweden')
        }
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
      onMiss (default: ): offload-host
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "offload",
      "type": "firstMatch",
      "targets": [
        {
          "onMatch": "offload-host",
          "condition": "not in_session_group('sweden')"
        },
        {
          "onMatch": "balancer",
          "condition": "in_session_group('sweden')"
        }
      ],
      "onMiss": "offload-host"
    }
  ]
}
Merge and apply the config? [y/n]: y

{
  "tuning": {
    // "target_request_attempts" controls the number of attempts to find a host.
    // In this example we want to fallback to the external CDN in case both
    // Request Routers are down, so we set it to 3.
    "target_request_attempts": 3
  },
  "session_groups": [
    {
      "id": 1,
      "name": "not_sweden",
      "classifiers": [
        [
          {
            "id": 1,
            "inverted": true,
            "name": "not_sweden_classifier",
            "rule": {
              "rule_type": "geoip_rule",
              "source": "session/client_ip",
              "country": "Sweden"
            }
          }
        ]
      ]
    }
  ],
  "cdns": [
    {
      "id": "internal-cdn",
      "http_port": 80,
      "https_port": 443,
      // The internal CDN uses Edgeware Request Routers that return a redirect
      // location.  The router takes the content from the client request,
      // and makes its own request to the Request Router using that same
      // content.  The returned location is then forwarded to the client.
      //
      // In case the Request Router responds with an error code, or times out,
      // the router will continue traversing the routing tree until a
      // successful route is found or the entire tree has been evaluated.
      "redirecting": true,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    },
    {
      "id": "offload-cdn",
      "http_port": 80,
      "https_port": 443,
      // The offload CDN is non-redirecting, meaning that router does not
      // make any request to its associated hosts to ask for a redirect location
      // to forward to the clients, instead it simply returns a location using
      // one of the CDN's hosts substituted for the router's host in the
      // client request possibly with some kind of respsonse translation to add
      // e.g. tokens or path prefixes.
      "redirecting": false,
      "manifest_availability_check": {
        "enabled": false,
        "session_group_ids": []
      }
    }
  ],
  "hosts": [
    {
      "id": "internal-host-1",
      "cdn_id": "internal-cdn",
      "address_family": "ipv4",
      "host": "internal-host-1.example"
    },
    {
      "id": "internal-host-2",
      "cdn_id": "internal-cdn",
      "address_family": "ipv4",
      "host": "internal-host-2.example"
    },
    {
      "id": "offload-host",
      "cdn_id": "offload-cdn",
      "address_family": "ipv4",
      "host": "offload-host.example"
    }
  ],
  "routing": {
    "id": "routing_table",
    // "sequential" - Go through the rules at this level one by one, and pick
    // the first that returns a positive weight.
    "member_order": "sequential",
    "members": [
      {
        "id": "reject-node",
        "host_id": "offload-host",
        // Send filtered-out clients to an offload host.
        "weight_function": "return session_groups.not_sweden and 1 or 0"
      },
      {
        "id": "internal_routing",
        // Return 1 to make sure this tree is evaluated at all.
        "weight_function": "return 1"
        // "weighted" - Evaluate all the rules at this level, and pick one of
        // them randomly based on their respective weights.
        //
        // In this example host 1 has half the capacity of host 2, so we just
        // randomly route twice as many clients to host 2.
        "member_order": "weighted",
        "members": [
          {
            "id": "internal-node-1",
            "host_id": "internal-host-1",
            // Half the weight of the other host -> half as likely to be picked.
            "weight_function": "return 1"
          },
          {
            "id": "internal-node-2",
            "host_id": "internal-host-2",
            // Twice the weight of the other -> twice as likely to be picked.
            "weight_function": "return 2"
          }
        ]
      },
      {
        // Add a non-redirecting offload host as the last target in case all the
        // internal redirecting hosts are unavailable or respond with some kind
        // of error code.
        "id": "offload-node",
        "host_id": "offload-host",
        "weight_function": "return 1"
      }
    ]
  }
}

4.9 - How to use ESB3024 Router with CoreDNS

How to use ESB3024 Router with CoreDNS

CoreDNS can be configured to work as a DNS server in front of ESB3024 Router. In this mode of operation the client DNS request is terminated by CoreDNS which requests an optimal streaming location from the router.

Contact Edgeware for detailed information on how to set this up, if you don’t already have an Edgeware Support account please contact us here.

4.10 - How to use the Director as a replacement for Edgeware CDN Request Router

How to use the Director as a replacement for ESB3008 Request Router

This guide outlines how to replicate Convoy’s ESB3008 HTTP Request Router using the Director. This is accomplished using the Convoy Bridge, which is an integration service designed to allow the router and an existing Convoy installation to work together seamlessly.

Following the steps outlined below enables the replacement of existing ESB3008 HTTP Request Router instances with the Director. As of the current documentation, the Director is not a direct substitute for the ESB3008 HTTP Request Router. However, most of the functionality is available, and in the majority of cases, the Director can be used without any loss of functionality.

The purpose of this guide is to detail the process of transitioning from an existing Convoy installation, which includes one or more instances of ESB3008 HTTP Request Router, to a configuration utilizing the Director. This transition allows the existing Convoy installation to continue its role in CDN provisioning, content management, and analytics. Simultaneously, the router assumes responsibility for managing HTTP(S) redirects directly from clients to selected streamers.

One functional difference between ESB3008 HTTP Request Router and the Director is in communication with the streamer to determine specific interface assignments. ESB3008 uses a proprietary protocol to communicate with the streamer to help determine the best interface for the client. For the Director, in order to work with both streamers and third-party CDNs, the use of the proprietary protocol is not supported, and the router’s rule engine must be used to accomplish the same result. It is for this reason that it is recommended to configure each streaming interface as a distinct host entry in the router’s configuration. This allows the router to address each interface separately.

Prerequisites

This guide assumes both an existing Convoy installation and a partiality configured instance of the Director. The router should have all the necessary routing configuration in place for hostGroups, sessionGroups and routing rules to direct traffic to the streaming interfaces of the streamers. It is also assumed that the proper firewall rules are allowing the required traffic between the router, the Convoy Bridge integration and the Convoy management nodes.

The minimum compatible version of Convoy which can be used with the analytics synchronization feature is Convoy 3.0.0. Previous versions may be used only if Convoy is not being used for analytics.

Firewall Configuration

Proper firewall configuration is required to allow both the router and Convoy to communicate through bidirectional connections established from the Convoy Bridge integration. The Convoy Bridge does not need to be enabled for each router instance, one instance of the Convoy Bridge can synchronize multiple instances of the router simultaneously. Only the router nodes with the Convoy Bridge enabled will need access through the firewall to establish outgoing connections to the Convoy management nodes, or whichever nodes are running MariaDB and Kafka. Additionally, if multiple router instances are to be synchronized by a single Convoy Bridge instance, the firewall must allow the Convoy Bridge to establish connections to each router instance on port 5001.

The following table outlines the necessary connections to be allowed through the firewall.

Source	Destination	Port	Description
Convoy Bridge	Convoy Management	3306/TCP	Account Configuration
Convoy Bridge	Convoy Management	9092/TCP	Analytics Data
Convoy Bridge	The router	5001/TCP	Rest API

Response Translation Functions

In order to have the router generate a redirect URL which is compatible with the format generated by the ESB3008 HTTP Request Router, a response translation function must be used to modify the redirect URL before it is returned to the client. A predfined function convoy_compatibility_response is available in the standard library to be used for this purpose. This function appends both the session-id and storage-prefix based on the account and distribution information in Convoy.

On the Director, set the response translation function as follows:

> confcli services.routing.translationFunctions.response "return convoy_compatibility_response(Headers, '')"

This function depends on the account synchronization feature of the Convoy bridge to determine the correct storage prefix based on the incoming Host header value. It works by using the Host header to lookup the storage prefix from the account configuration, and then modifying the redirect URL to prepend /session/<session-id>/<prefix>. The streamer can then use this information for both origin selection and session tracking.

Enabling the Convoy Bridge

Before using the Convoy Bridge integration, a valid configuration must first be applied. All configuration for the Convoy Bridge is available in confd under the key integration.convoy.bridge. The configuration is divided into three main sections: The account synchronization feature, the convoy analytics integration, and a list of additional router instances to synchronize.

By default, the Convoy Bridge will include the current local router instance in the synchronization process. The connection details are hard-wired into the container at startup, and do not need to be specified in the configuration. Because one Convoy Bridge can serve multiple instances of the router, it is not necessary to configure the Convoy Bridge on each node separately. However, if multiple Convoy Bridge instances are used, it is recommended to configure them so that one router is served by multiple instances of the Convoy Bridge. This provides high-availability in a production environment.

As an example, if there are 4 router instances, A, B, C, and D, and 4 Convoy Bridge instances, configuring them such that Bridge 1 synchronizes A & B, 2, synchronizes B & C, 3, synchronizes C & D and 4 synchronizes D & A will provide fault tolerance in the event that one bridge is unavailable.

The account synchronization feature of the Convoy Bridge watches the MariaDB database on the Convoy management node for changes to the account and distribution configuration, and if the configuration differs from what is currently known to the router the configuration will be updated. This feature is required by both the router for use with the convoy_compatibility_response response translation function, and within the Convoy Bridge integration for setting the correct account, storage, and distribution fields with the analytics integration feature.

The analytics integration feature monitors monitors the router’s Event API for newly established sessions, and produces session-record messages over the Kafka message broker for use by Convoy. This feature requires that Convoy is both licensed and configured for use with the analytics feature. The integration works by establishing a persistent connection to each router, and listening for new session events. Due to the protocol in use by the Event API to send the events, only one instance of the Convoy Bridge will have the ability to receive each event, and only that instance will produce the session-record to the Kafka message brokers.

Synchrnoizing multiple routers with the same instance of the Convoy Bridge is performed by configuring the otherRouters section of the Convoy Bridge configuration. In addition to the local router, each otherRouter entry will have the same account and distribution information updated simultaneously, and will be used as an event source for the analytics integration. The otherRouters entries require the URL to the router’s Rest API, an optional API key, and a flag to indicate if SSL certificate validation should be enforced.

Enable the account synchronization feature by running the following command:

> confcli integration.convoy.bridge.accounts -w
Running wizard for resource 'accounts'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

accounts : {
  enabled (default: False): true
  dbUrl (default: mysql://user:pass@localhost:3306): mysql://convoy:password@convoy-mgmt1:3306
  cmsUrl (default: http://localhost:5555): http://convoy-mgmt1:5555
  pollInterval (default: 60):
}
Generated config:
{
  "accounts": {
    "enabled": true,
    "dbUrl": "mysql://convoy:password@convoy-mgmt1:3306",
    "cmsUrl": "http://convoy-mgmt1:5555",
    "pollInterval": 60
  }
}
Merge and apply the config? [y/n]:

This will result in the Convoy Bridge continuously polling the Convoy instance running on convoy-mgmt1 every 60 seconds for changes to the account configuration. Only when the account configuration differs from the current configuration on the router will it be updated with the change.

The pollInterval parameter can be used to control how frequently the Convoy Bridge will attempt to poll the database for changes. Setting this value too high will result in a longer delay between changes being made in Convoy and the changes being reflected in the router. Setting this value too low will result in unnecessary load on the Convoy database. The default value of 60 seconds should be sufficient for most use cases.

To enable the analytics integration feature, run the following command:

> confcli integration.convoy.bridge.analytics -w
Running wizard for resource 'analytics'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

analytics : {
  enabled (default: False): true
  brokers : [
    broker (default: ): convoy-mgmt1:9092
    Add another 'broker' element to array 'brokers'? [y/N]: y
    broker (default: ): convoy-mgmt2:9092
    Add another 'broker' element to array 'brokers'? [y/N]: y
    broker (default: ): convoy-mgmt3:9092
    Add another 'broker' element to array 'brokers'? [y/N]: n
  ]
  batchInterval (default: 10):
  maxBatchSize (default: 500):
}
Generated config:
{
  "analytics": {
    "enabled": true,
    "brokers": [
      "convoy-mgmt1:9092",
      "convoy-mgmt2:9092",
      "convoy-mgmt3:9092"
    ],
    "batchInterval": 10,
    "maxBatchSize": 500
  }
}
Merge and apply the config? [y/n]:

This will enable the analytics integration, which will produce session-record messages to the configured Kafka brokers. The brokers configuration list is used to bootstrap the initial set of brokers which will be contacted by the Convoy Bridge to then obtain the current set of active brokers. It is not necessary to include all brokers here, as the initial connection will automatically negotiate the correct broker list.

The batchInterval parameter must be set to a value much less than 60 seconds, since the streamer will send session-sample messages to Kafka every 60 seconds, and if the corresponding session-record message is not received within that time, the sample will be dropped, resulting in potentially inaccurate bandwidth statistics.

> confcli integration.convoy.bridge.otherRouters -w
Running wizard for resource 'otherRouters'
<Other routers which will be kept in sync (default: [])>

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

otherRouters <Other routers which will be kept in sync (default: [])>: [
  otherRouter : {
    url (default: ): https://router-2:5001
    apiKey (default: ): 1234
    validateCerts (default: True): false
  }
  Add another 'otherRouter' element to array 'otherRouters'? [y/N]: y
  otherRouter : {
    url (default: ): https://router-3:5001
    apiKey (default: ):
    validateCerts (default: True):
  }
  Add another 'otherRouter' element to array 'otherRouters'? [y/N]: n
]
Generated config:
{
  "otherRouters": [
    {
      "url": "https://router-2:5001",
      "apiKey": "1234",
      "validateCerts": false
    },
    {
      "url": "https://router-3:5001",
      "apiKey": "",
      "validateCerts": true
    }
  ]
}
Merge and apply the config? [y/n]:

4.11 - Use In-Stream Sessions

How to configure ACD Router to use In-Stream sessions

In-Stream sessions is a routing mode where the router keeps control of the session for the full session lifetime instead of only routing the session to the preferred CDN (or cache node) once. Technically it works by letting the client return to the router for manifest and segments and redirect each segment to the preferred CDN. This enables changing CDN in the middle of a session.

Configure In-Stream Sessions

The following configuration enables In-Stream routing

{
  "managed_sessions": {
    "fraction": 1.0,
    "max_active": 1000000,
    "session_types": [
      {
        "enabled": true,
        "max_active": 1000000,
        "session_group_ids": [],
        "type": "instream"
      }
    ]
  }
}

The fraction of In-Stream sessions and the maximum number of active In-Stream sessions can be controlled. It is also possible to specify that In-Stream sessions should only be used for certain session groups, for example only for specific contents. See here for a description of the available configuration options.

The following example illustrate the request flow for in-stream session

$ curl  -i http://test-acd-router/content/playlist.m3u8
HTTP/1.1 302 Found
Access-Control-Allow-Origin: *
Content-Length: 0
Location: http://test-acd-router/__s/test-acd-router-a4fd86-0000004f_WyIvY2RuL3BsYXlsaXN0Lm0zdTgiLDAsImRldmNkbjEiLCJkZXZjZG4xLnN0cmVhbXBpbG90LnR2Iiw4MCwiL2Nkbi9wbGF5bGlzdC5tM3U4IiwwLDE2ODAyNjQwOTNd_c1e3fc663a47156e0894b30eddf995bb/content/playlist.m3u8
X-Service-Identity:

$ curl -i http://test-acd-router/__s/test-acd-router-a4fd86-0000004f_WyIvY2RuL3BsYXlsaXN0Lm0zdTgiLDAsImRldmNkbjEiLCJkZXZjZG4xLnN0cmVhbXBpbG90LnR2Iiw4MCwiL2Nkbi9wbGF5bGlzdC5tM3U4IiwwLDE2ODAyNjQwOTNd_c1e3fc663a47156e0894b30eddf995bb/content/playlist.m3u8
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length,Content-Range
Connection: keep-alive
Content-Length: 306
Content-Type: application/vnd.apple.mpegurl
Date: Fri, 31 Mar 2023 10:18:42 GMT
ETag: "5d711f08-132"
Last-Modified: Thu, 05 Sep 2019 14:43:20 GMT
Server: nginx/1.14.0 (Ubuntu)
X-Service-Identity:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=500000,RESOLUTION=320x180
abr_500k.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1000000,RESOLUTION=640x360
abr_1000k.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1700000,RESOLUTION=1280x720
abr_1700k.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3300000,RESOLUTION=1280x720
abr_3300k.m3u8

$ curl -i http://test-acd-router/__s/test-acd-router-a4fd86-0000004f_WyIvY2RuL3BsYXlsaXN0Lm0zdTgiLDAsImRldmNkbjEiLCJkZXZjZG4xLnN0cmVhbXBpbG90LnR2Iiw4MCwiL2Nkbi9wbGF5bGlzdC5tM3U4IiwwLDE2ODAyNjQwOTNd_c1e3fc663a47156e0894b30eddf995bb/content/abr_500k.m3u8
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length,Content-Range
Connection: keep-alive
Content-Length: 276
Content-Type: application/vnd.apple.mpegurl
Date: Fri, 31 Mar 2023 12:02:04 GMT
ETag: "6426cbbc-114"
Last-Modified: Fri, 31 Mar 2023 12:02:04 GMT
Server: nginx/1.14.0 (Ubuntu)
X-Service-Identity:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:1
#EXT-X-MEDIA-SEQUENCE:112583846
#EXTINF:0.834444,
abr_500k112583846.ts
#EXTINF:0.834444,
abr_500k112583847.ts
#EXTINF:0.834444,
abr_500k112583848.ts
#EXTINF:0.834444,
abr_500k112583849.ts
#EXTINF:0.834444,
abr_500k112583850.ts

$ curl -i http://test-acd-router/__s/test-acd-router-a4fd86-0000004f_WyIvY2RuL3BsYXlsaXN0Lm0zdTgiLDAsImRldmNkbjEiLCJkZXZjZG4xLnN0cmVhbXBpbG90LnR2Iiw4MCwiL2Nkbi9wbGF5bGlzdC5tM3U4IiwwLDE2ODAyNjQwOTNd_c1e3fc663a47156e0894b30eddf995bb/content/abr_500k112583846.ts
HTTP/1.1 302 Found
Access-Control-Allow-Origin: *
Content-Length: 0
Location: http://testcdn.tv/abr_500k112583846.ts
X-Service-Identity:

A Note on Confcli

In-stream selection sessions can’t be configured in confcli at the moment, but it is possible to configure such sessions while configuring routing etc using confcli.

Begin with configuring routing, classification and session groups in confcli, for example based on content types.

# Create a classifier for live sessions

$ confcli services.routing.classifiers -w
Running wizard for resource 'classifiers'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

classifiers : [
  classifier can be one of
    1: contentUrlPath
    2: contentUrlQueryParameters
    3: geoip
    4: hostName
    5: ipranges
    6: regexMatcher
    7: stringMatcher
    8: subnet
    9: userAgent
  Choose element index or name: 1
  Adding a 'contentUrlPath' element
    classifier : {
      name (default: ): live
      type (default: contentUrlPath): ⏎
      inverted (default: False): ⏎
      patternType (default: stringMatch): ⏎
      pattern (default: ): *live*
    }
  Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
]
Generated config:
{
  "classifiers": [
    {
      "name": "live",
      "type": "contentUrlPath",
      "inverted": false,
      "patternType": "stringMatch",
      "pattern": "*live*"
    }
  ]
}
Merge and apply the config? [y/n]: y

# Create a session group using the classifier

$ confcli services.routing.sessionGroups -w
Running wizard for resource 'sessionGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

sessionGroups : [
  sessionGroup : {
    name (default: ): live
    classifiers : [
      classifier (default: ): live
      Add another 'classifier' element to array 'classifiers'? [y/N]: ⏎
    ]
  }
  Add another 'sessionGroup' element to array 'sessionGroups'? [y/N]: ⏎
]
Generated config:
{
  "sessionGroups": [
    {
      "name": "live",
      "classifiers": [
        "live"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

Then look at the raw configuration to determine what session ID numbers to use for in-stream routing:

// curl -k https://localhost:5001/v2/configuration | jq .session_groups
[
  {
    "classifiers": [
      [
        {
          "id": 0,
          "inverted": false,
          "name": "classifier_live",
          "rule": {
            "pattern": "*live*",
            "rule_type": "string_match_rule",
            "source": "session/content_url_path"
          }
        }
      ]
    ],
    "id": 0,
    "name": "live"
  }
]

The output shows that the session group with the name "live" has the session group ID number 0, meaning that the configuration required to enable in-stream selection sessions for 100 % of live content sessions would look like

{
  "managed_sessions": {
    "fraction": 1.0,
    "max_active": 1000000,
    "session_types": [
      {
        "enabled": true,
        "max_active": 1000000,
        "session_group_ids": [0], // Session group ID from config here
        "type": "instream"
      }
    ]
  }
}

This configuration can be sent on its own to the configuration API using a PUT request without interfering with any of the configuration done using confcli.

Monitoring

If Grafana is installed together with the ESB-3024 installer, the number of started In-Stream sessions as well as currently active In-Stream sessions can be seen in a Dashboard.

Notes on Performance

Be aware that In-Stream sessions puts additional load on the request router since all clients continuously comes back to the router to fetch new manifests and to get segment redirects. The necessary router capacity therefore scaled both with number of started sessions and the average session length.

5 - Ready for Production?

Prepare for production

5.1 - Setup for Redundancy

Configuring Redundancy

The Agile Director is designed to support the coexistence of multiple independent instances, facilitating either high-availability and/or horizontal scaling based on specific use cases and requirements. Each instance of the Director operates as a complete isolated entity, offering multiple solutions for redundancy. The choice between these solutions depends on the particular use case and requirements of the deployment.

A few select use-cases and example deployments are described in the sections that follow. Note however that this list is not exhaustive, and other approaches not listed here may be more applicable to a specific use-case.

Third-party load balancer

In this first scenario, multiple instances of the router are positioned behind an off-the-shelf load balancer. This could range from a simple

Nginx

proxy to a more sophisticated commercial load balancer, depending on the user’s specific needs and requirements.

DNS Round Robin

A straightforward alternative to deploying multiple instances of the router behind a load balancer is to employ round-robin DNS. In this approach, each router instance is associated with a distinct A record in DNS, all sharing the same domain name. Clients querying this DNS server receive multiple entries in response. Typically, clients utilize a round-robin algorithm to resolve the IP address of the router, although some clients might randomly select an address from the available servers.

While this method effectively distributes the load across multiple instances, it has a few drawbacks. First, clients are not guaranteed to use any specific algorithm for selecting the IP address. Second, due to DNS caching in the network and on the client’s device, modifications to the DNS record (e.g., taking a node out of service or adding additional nodes) will experience higher latency compared to some other techniques. Third, there is a dependency on the client being able to recognize this DNS server as authoritative. And fourth, this approach does not provide any means of high-availability. If a node is offline, requests to that node will fail.

VRRP

If horizontal scaling is not required, employing a shared Virtual IP Address through the VRRP protocol may suffice. This can be easily configured on one or more instances of the router by installing and setting up keepalived. While the specific details of keepalived are beyond the scope of this document, it can establish a Primary/Backup node configuration, where heartbeat packets and server priority determine each node’s role. Using VRRP, a Virtual IP address always points to the current primary node, and in the event of a primary node failure, the remaining nodes elect a new primary node based on the predetermined priority.

This solution has some drawbacks compared to others; the VIP may only be assigned to a single node at a time, rendering horizontal scaling impossible using this solution alone. Another drawback is that if the primary node fails, there will be a brief period, usually around one second, before the failure is detected and the VIP is reassigned.

This solution can be more effectively utilized as part of a larger combined solution in which several groups of nodes are employed. In each group, there is a Primary and Backup, and selection between the groups is handled via one of the other scaling solutions.

An Example Combined Approach Scenario

The following represents an example scenario where eight independent router nodes are utilized employing all three of the above approaches.

The scenario consists of four pairs of nodes, with each pair using VRRP to designate one node as Master and one as a Backup. In front of two pairs of pairs would sit two load balancers, each configured with the VIP address of each pair below. The two load balancers would then be placed into Round Robin DNS with the same FQDN.

Incoming requests would resolve the DNS record, yielding two servers. They would then send requests to each in turn. The load balancer at the end of those IP addresses would then select among the two VIP addresses below, and the request would be routed to the current Primary node in the group.

In this scenario, if any one of the primary nodes becomes offline, the VRRP protocol will move the VIP to the backup node. While that process is ongoing, the load balancer will detect that the VIP is unreachable, routing all traffic in the meantime to the VIP of the second pair. The DNS round-robin would be used to evenly distribute the traffic between the two load balancers.

5.2 - HTTPS Certificates

Installation and verification of SSL certificates

Configuration of ESB3024 Router is done through a REST API over HTTPS. While the router installer generates a self-signed certificate in order to enable the interface at all, this is not considered safe and secure so a properly generated certificate should be used instead.

For SSL to work, the router needs to have both an x509 certificate and a key in ASCII armored PEM format:

-----BEGIN PRIVATE KEY-----
[...]
-----END PRIVATE KEY-----

-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----

The files can be either separate .crt and .key files or a combined .pem file.

Simply copy the file(s) generated by your CA service, into the /opt/edgeware/acd/ssl folder on the host machine and they will automatically be used by the router. The filenames must match the associated hostname, and there’s currently no support for wildcard matching or multiple domains per certificate. Several key/crt pairs can be placed in the folder in order to support more than one domain name.

Agile Content Delivery

1 - Components

1.1 - ESB3024 Router

1.1.1 - Getting Started

Requirements

Hardware

Operating System Compatibility

Firewall Recommendations

Installation

Operations

Configuration Process

API Key Management

Configuration Basics

Integration with Convoy

Additional Resources

Ready for Production

1.1.2 - Installation

1.1.2.1 - Installing release 1.10.x

Upgrade from ESB3024 Router release 1.8.0

Troubleshooting

1.1.2.1.1 - Configuration changes between 1.8.0 and 1.10.x

Confd configuration changes

Added integration.convoy section

Removed services.routing.apiKey configuration

Added services.routing.settings.qoeTracking

Added healthChecks sections to the hosts

1.1.2.2 - Installing release 1.8.0

Upgrade

Troubleshooting

1.1.2.2.1 - Configuration changes between 1.6.0 and 1.8.0

Confd configuration changes

Added enabled to contentPopularity

Added selectionInputItemLimit to tuning

Added forwardHostHeader to hostGroups

REST API configuration changes

1.1.2.3 - Installing release 1.6.0

Upgrade

Troubleshooting

1.1.2.3.1 - Configuration changes between 1.4.0 and 1.6.0

confd configuration

Configuration changes

Added translationFunctions block

Renamed sessionGroupIds to sessionGroupNames

Added managedSessions block

Added recentDurationMilliseconds

Storing the updated configuration

Raw configuration

Simple changes

EDNS proxy changes

1.1.3 - Firewall

Additional Ports

Convoy bridge integration

1.1.4 - API Overview

Configuration – /v2/configuration

Example request (GET)

Example request (PUT)

Validate Configuration – /v2/validate_configuration

Example request

Selection Input – /v1/selection_input

Example successful request (PUT)

Example unsuccessful request (PUT)

Example successful request (GET)

Example successful specific value request (GET)

Example successful request (DELETE)

Example successful specific value request (DELETE)

Example unsuccessful request (DELETE)

Subnets – /v1/subnets

Example request

Subrunner Resource Usage – /v1/usage

Example request

Metrics – /m1/v1/metrics

Example request

Node Visit Counters – /v1/node_visits

Example request

Node Visit Graph – /v1/node_visits_graph

Example request

Session list - /v1/sessions

Example request

Session details - /v1/sessions/<id: str>

Example request

Added `integration.convoy` section

Removed `services.routing.apiKey` configuration

Added `services.routing.settings.qoeTracking`

Added `healthChecks` sections to the hosts

Added `enabled` to `contentPopularity`

Added `selectionInputItemLimit` to `tuning`

Added `forwardHostHeader` to `hostGroups`

Added `translationFunctions` block

Renamed `sessionGroupIds` to `sessionGroupNames`

Added `managedSessions` block

Added `recentDurationMilliseconds`

Configuration – `/v2/configuration`

Example request (`GET`)

Example request (`PUT`)

Validate Configuration – `/v2/validate_configuration`

Selection Input – `/v1/selection_input`

Example successful request (`PUT`)

Example unsuccessful request (`PUT`)

Example successful request (`GET`)

Example successful specific value request (`GET`)

Example successful request (`DELETE`)

Example successful specific value request (`DELETE`)

Example unsuccessful request (`DELETE`)

Subnets – `/v1/subnets`

Subrunner Resource Usage – `/v1/usage`

Metrics – `/m1/v1/metrics`

Node Visit Counters – `/v1/node_visits`

Node Visit Graph – `/v1/node_visits_graph`

Session list - `/v1/sessions`

Session details - `/v1/sessions/<id: str>`

Content List - `/v1/content`

Lua scripts – `/v1/lua/<path str>.lua`

Example request (`PUT`)

Example request (`PUT`, from file)

Example request (`GET`)

Example request (`DELETE`)

List Lua scripts – `/v1/lua`

Debug a Lua expression – `/v1/lua/debug`

`"cdns"`

`"content_server"`

`"hosts"`

`"managed_sessions"`

`"metadata"`

`"request_translation_function"`

`"response_translation_function"`

`"rest_api_server"`

`"routing"`

`"session_groups"`

`"settings"`

`"tuning"`