Components

• 1: ESB3024 Router
• 1.1: Getting Started
• 1.2: Installation
• 1.2.1: Installing a 1.16 release
• 1.2.1.1: Configuration changes between 1.14 and 1.16
• 1.2.2: Installing a 1.14 release
• 1.2.2.1: Configuration changes between 1.12.1 and 1.14
• 1.2.3: Installing a 1.12 release
• 1.2.3.1: Configuration changes between 1.10.2 and 1.12
• 1.2.4: Installing release 1.10.x
• 1.2.4.1: Configuration changes between 1.8.0 and 1.10.x
• 1.2.5: Installing release 1.8.0
• 1.2.5.1: Configuration changes between 1.6.0 and 1.8.0
• 1.2.6: Installing release 1.6.0
• 1.2.6.1: Configuration changes between 1.4.0 and 1.6.0
• 1.3: Firewall
• 1.4: API Overview
• 1.5: Configuration
• 1.5.1: WebUI Configuration
• 1.5.2: Confd and Confcli
• 1.5.3: Session Groups and Classification
• 1.5.4: Accounts
• 1.5.5: Advanced features
• 1.5.5.1: Content popularity
• 1.5.5.2: Consistent Hashing
• 1.5.5.3: Security token verification
• 1.5.5.4: Subnets API
• 1.5.5.5: Lua Features
• 1.5.5.5.1: Built-in Lua Functions
• 1.5.5.5.2: Global Lua Tables
• 1.5.5.5.3: Request Translation Function
• 1.5.5.5.4: Session Translation Function
• 1.5.5.5.5: Host Request Translation Function
• 1.5.5.5.6: Response Translation Function
• 1.5.6: Trusted proxies
• 1.5.7: Confd Auto Upgrade Tool
• 1.6: Operations
• 1.6.1: Services
• 1.7: Convoy Bridge
• 1.8: Monitoring
• 1.8.1: Access logging
• 1.8.2: System troubleshooting
• 1.8.3: Scraping data with Prometheus
• 1.8.4: Visualizing data with Grafana
• 1.8.4.1: Managing Grafana
• 1.8.4.2: Grafana Dashboards
• 1.8.5: Alarms and Alerting
• 1.8.6: Monitoring multiple routers
• 1.8.7: Routing Rule Evaluation Metrics
• 1.8.8: Metrics
• 1.8.8.1: Internal Metrics
• 1.9: Releases
• 1.9.1: Release esb3024-1.16.0
• 1.9.2: Release esb3024-1.14.2
• 1.9.3: Release esb3024-1.14.0
• 1.9.4: Release esb3024-1.12.1
• 1.9.5: Release esb3024-1.12.0
• 1.9.6: Release esb3024-1.10.2
• 1.9.7: Release esb3024-1.10.1
• 1.9.8: Release esb3024-1.10.0
• 1.9.9: Release esb3024-1.8.0
• 1.9.10: Release esb3024-1.6.0
• 1.9.11: Release esb3024-1.4.0
• 1.9.12: Release acd-router-1.2.3
• 1.9.13: Release acd-router-1.2.0
• 1.9.14: Release acd-router-1.0.0
• 1.10: Glossary
• 2: ESB3032 ACD Aggregator
• 2.1: Getting Started
• 2.2: Releases
• 2.2.1: Release esb3032-0.2.0
• 2.2.2: Release esb3032-1.0.0
• 2.2.3: Release esb3032-1.2.1
• 2.2.4: Release esb3032-1.4.0
• 3: ACD Cache
• 4: ESB3013 BGP Sniffer
• 5: Edgeware CDN Management System
• 6: ESB3008 Edgeware CDN Request Router

1 - ESB3024 Router

1.1 - Getting Started

The Director serves as a versatile network service designed to redirect incoming HTTP(s) requests to the optimal host or Content Delivery Network (CDN) by evaluating various request properties through a set of rules. Although requests can be generic, the primary focus centers around audio-video content delivery. The rule engine allows users to construct routing configurations using predefined blocks, providing for the creation of intricate routing logic. This modular approach allows the users to tailor and streamline the content delivery process to meet their specific needs. The Director’s flexible rule engine takes into account factors such as geographical location, server load, content type, and other metadata from external sources to intelligently route incoming requests. It supports dynamic adjustments to seamlessly adapt to changing network conditions, ensuring efficient and reliable content delivery. The Director improves the overall user experience by delivering content from the most suitable and responsive sources, thereby reducing latency and enhancing performance.

Requirements

Hardware

The Director is designed to be installed and operated on commodity hardware, ensuring accessibility for a broad range of users. The minimum hardware specifications are as follows:

• CPU: x86-64 AMD or Intel with at least 2 cores.
• Memory: At least 2 GB free at runtime.

Operating System Compatibility

The Director is officially supported on Red Hat Enterprise Linux 8 or 9 or any compatible operating system. In order to run the service, a minimum CPU architecture of x86-64-v2 is required. This can be determined by running the following command. If supported, it will be listed as “(supported)” in the output.

/usr/lib64/ld-linux-x86-64.so.2 --help | grep x86-64-v2

External Internet access is necessary during the installation process for the installer to download and install additional dependencies. This ensures a seamless setup and optimal functionality of the Director on Red Hat Enterprise Linux 8 or 9. It’s worth noting that, due to the unique workings of the DNF package manager in Red Hat Enterprise Linux with rolling package streams, an air-gapped installation process is not available.

Firewall Recommendations

See Firewall.

Installation

See Installation.

Operations

See Operations.

Configuration Process

Once the router is operational, it requires a valid configuration before it can route incoming requests.

There are currently three methods available for configuring the router, each catering to different levels of complexity. The first is a Web UI, suitable for the most common use-cases, providing an intuitive interface for configuration. The second involves utilizing a confd REST service, complemented by an optional command line tool, confcli, suitable for all but the most advanced scenarios. The third method involves leveraging an internal REST API, ideal for the most intricate cases where using confd proves to be less flexible. It’s essential to note that as the configuration method advances through these levels, both flexibility and complexity increase, providing users with tailored options based on their specific needs and expertise.

API Key Management

Regardless of the method used to configure the system, a unique API key is crucial for safeguarding the router’s configuration and preventing unauthorized access to the API. This key must be supplied when interacting with the API. During the router software installation, an automatically generated API key is created and can be located on the installed system at /opt/edgeware/acd/router/cache/rest-api-key.json. The structure of this file is as follows:

{"api_key": "abc123"}

When accessing the internal configuration API, the key must be included in the X-API-key header of the request, as shown below:

curl -v -k -H "X-API-Key: abc123" https://<router-host.example>:5001/v2/configuration

Modification to the authentication key and behavior can be done through the /v2/rest_api_key endpoint. To change the key, a PUT request with a JSON body of the same structure can be sent to the endpoint:

curl -v -k -X PUT -T new-key.json -H "X-API-Key: abc123" \
-H "Content-Type: application/json" https://<router-host.example>:5001/v2/rest_api_key

Additionally, key authentication can be disabled completely by sending a DELETE request to the endpoint:

curl -v -k -X DELETE -H "X-API-Key: abc123" \
https://<router-host.example>:5001/v2/rest_api_key

In the event of a lost or forgotten authentication key, it can always be retrieved at /opt/edgeware/acd/router/cache/rest-api-key.json on the machine running the router. It is critical to emphasize that the API key should remain private to prevent unauthorized access to the internal API, as it grants full access to the router’s configuration.

Configuration Basics

Upon completing the installation process and configuring the API keys, the subsequent section will provide guidance on configuring the router to route all incoming requests to a single host. For straightforward CDN Offload use cases, there is a web based user interface described here.

For further details on configuring the router using confd and confcli, please consult the Confd documentation.

The initial step involves defining the target host group. In this illustration, a singular group named all will be established, comprising two hosts.

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: host
  Adding a 'host' element
    hostGroup : {
      name (default: ): all
      type (default: host):
      httpPort (default: 80):
      httpsPort (default: 443):
      hosts : [
        host : {
          name (default: ): host1.example.com
          hostname (default: ): host1.example.com
          ipv6_address (default: ):
        }
        Add another 'host' element to array 'hosts'? [y/N]: y
        host : {
          name (default: ): host2.example.com
          hostname (default: ): host2.example.com
          ipv6_address (default: ):
        }
        Add another 'host' element to array 'hosts'? [y/N]: n
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: n
]
Generated config:
{
  "hostGroups": [
    {
      "name": "all",
      "type": "host",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
        {
          "name": "host1.example.com",
          "hostname": "host1.example.com",
          "ipv6_address": ""
        },
        {
          "name": "host2.example.com",
          "hostname": "host2.example.com",
          "ipv6_address": ""
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]:

After defining the host group, the next step is to establish a rule that directs incoming requests to the designated host. In this example, a sole rule named random will be generated, ensuring that all incoming requests are consistently routed to the previously defined host.

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: random
  Adding a 'random' element
    rule : {
      name (default: ): random
      type (default: random):
      targets : [
        target (default: ): host1.example.com
        Add another 'target' element to array 'targets'? [y/N]: y
        target (default: ): host2.example.com
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "random",
      "type": "random",
      "targets": [
        "host1.example.com",
        "host2.example.com"
      ]
    }
  ]
}
Merge and apply the config? [y/n]:

The last essential step involves instructing the router on which rule should serve as the entry point into the routing tree. In this example, we designate the rule random as the entrypoint for the routing process.

$ confcli services.routing.entrypoint random
services.routing.entrypoint = 'random'

Once this configuration is defined, all incoming requests will initiate their traversal through the routing rules, starting with the rule named random. This rule is designed to consistently match for every incoming request, effectively load balancing evenly between host1.example.com and host2.example.com on port 80 or 443, depending on whether the initial request was made using HTTP or HTTPS.

Integration with Convoy

The router is equipped with the capability to synchronize specific configuration metadata with a separate Convoy installation through the integrated convoy-bridge service. However, this service necessitates additional setup and configuration, and you can find comprehensive details on the process here..

Additional Resources

Additional documentation resources are included with the Director and can be accessed at the following directory: /opt/edgeware/acd/documentation/. This directory contains supplementary materials to provide users with comprehensive information and guidance for optimizing their experience with the Director.

Ready for Production

Once the Director software is completely installed and configured, there are a few additional considerations before moving to a full production environment. See the section Ready for Production for additional information.

1.2 - Installation

1.2.1 - Installing a 1.16 release

To install ESB3024 Router, one first needs to copy the installation ISO image to the target node where the router will be run. Due to the way the installer operates, it is necessary that the host is reachable by password-less SSH from itself for the user account that will perform the installation, and that this user has sudo access.

Prerequisites:

1. Ensure that the current user has sudo access.

   sudo -l
   

   If the above command fails, you may need to add the user to the /etc/sudoers file.

2. Ensure that the installer has password-less SSH access to localhost.

   If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.

   The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:

   mkdir -m 0700 -p ~/.ssh
   ssh-keyscan localhost >> ~/.ssh/authorized_keys
   

   Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice it is recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.

3. Disable SELinux.

   The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.

   To check if SELinux is enabled:

   getenforce
   

   This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.

   setenforce 0
   

   This disables SELinux, but does not make the change persistent across reboots. To do that, edit the /etc/selinux/config file and set the SELINUX property to disabled.

   It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

1. Mount the installation ISO image under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-1.16.0.iso /mnt/acd
   

2. Run the installer script.

   /mnt/acd/installer
   

Upgrade from and earlier ESB3024 Router release

The following steps can be taken to upgrade the router from a 1.10 or later release to 1.16.0. If upgrading from an earlier release it is recommended to first upgrade to 1.10.1 and then to upgrade to 1.16.0.

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new release of the router, and applying the saved configuration.

1. With the router running, save a backup of the configuration.

   The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.

   Extracting the configuration using confd is the recommend approach where available.

   confcli | tee config_backup.json
   

   To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.

   curl --insecure https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

   If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.

   curl --insecure -H "X-API-Key: 1234abcd" \
     https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

2. Mount the new installation ISO under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-1.16.0.iso /mnt/acd
   

3. Stop the router and all associated services.

   Before upgrading the router it needs to be stopped, which can be done by typing this:

   systemctl stop 'acd-*'
   

4. Run the installer script.

   /mnt/acd/installer
   

5. Migrate the configuration.

   Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.

   The confd configuration used in the previous versions is not directly compatible with 1.16, and may need to be converted. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes.

   The acd-confd-migration tool will automatically apply any necessary schema migrations. Further details about this tool can be found at Confd Auto Upgrade Tool.

   The tool takes as input the old configuration file, either by reading the file directly, or by reading from standard input, applies any necessary migrations between the two specified versions, and outputs a new configuration to standard output which is suitable for being applied to the upgraded system. While the tool has the ability to migrate between multiple versions at a time, the earliest supported version is 1.10.1.

   The example below shows how to upgrade from 1.10.2. If upgrading from 1.14.0, --from 1.10.2 should be replaced with --from 1.14.0.

   The command line required to run the tool is different depending on which esb3024 release it is run on. On 1.16.0 it is run like this:

   cat config_backup.json | \
     podman run -i --rm \
     images.edgeware.tv/acd-confd-migration:1.16.0 \
     --in - --from 1.10.2 --to 1.16.0 \
     | tee config_upgraded.json
   

   After running the above command, apply the new configuration to confd by running cat config_upgraded.json | confcli -i.

Troubleshooting

If there is a problem running the installer, additional debug information can be output by adding -v or -vv or -vvv to the installer command, the more “v” characters, the more detailed output.

1.2.1.1 - Configuration changes between 1.14 and 1.16

Confd configuration changes

Below are the changes to the confd configuration between versions 1.14 and 1.16 listed.

Added region GeoIP classifier

Classifiers of type geoip now have a region property.

Added integration.routing.gui configuration

There is now an integration.routing.gui section which will be used by the GUI.

Added services.routing.accounts configuration

The services.routing.accounts list has been added to the configuration.

1.2.2 - Installing a 1.14 release

To install ESB3024 Router, one first needs to copy the installation ISO image to the target node where the router will be run. Due to the way the installer operates, it is necessary that the host is reachable by password-less SSH from itself for the user account that will perform the installation, and that this user has sudo access.

Prerequisites:

1. Ensure that the current user has sudo access.

   sudo -l
   

   If the above command fails, you may need to add the user to the /etc/sudoers file.

2. Ensure that the installer has password-less SSH access to localhost.

   If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.

   The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:

   mkdir -m 0700 -p ~/.ssh
   ssh-keyscan localhost >> ~/.ssh/authorized_keys
   

   Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice it is recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.

3. Disable SELinux.

   The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.

   To check if SELinux is enabled:

   getenforce
   

   This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.

   setenforce 0
   

   This disables SELinux, but does not make the change persistent across reboots. To do that, edit the /etc/selinux/config file and set the SELINUX property to disabled.

   It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

1. Mount the installation ISO image under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-1.14.0.iso /mnt/acd
   

2. Run the installer script.

   /mnt/acd/installer
   

Upgrade from and earlier ESB3024 Router release

The following steps can be used to upgrade the router from a 1.10 or 1.12 release to 1.14.0. If upgrading from an earlier release it is recommended to first upgrade to 1.10.1 in multiple steps; for instance when upgrading from release 1.8.0 to 1.14.0, it is recommended to first upgrade to 1.10.1 and then to 1.14.0.

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new release of the router, and applying the saved configuration.

1. With the router running, save a backup of the configuration.

   The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.

   Extracting the configuration using confd is the recommend approach where available.

   confcli | tee config_backup.json
   

   To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.

   curl --insecure https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

   If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.

   curl --insecure -H "X-API-Key: 1234abcd" \
     https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

2. Mount the new installation ISO under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-1.14.0.iso /mnt/acd
   

3. Stop the router and all associated services.

   Before upgrading the router it needs to be stopped, which can be done by typing this:

   systemctl stop 'acd-*'
   

4. Run the installer script.

   /mnt/acd/installer
   

5. Migrate the configuration.

   Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.

   The confd configuration used in the previous versions is not directly compatible with 1.14, and may need to be converted. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes.

   The acd-confd-migration tool will automatically apply any necessary schema migrations. Further details about this tool can be found at Confd Auto Upgrade Tool.

   The tool takes as input the old configuration file, either by reading the file directly, or by reading from standard input, applies any necessary migrations between the two specified versions, and outputs a new configuration to standard output which is suitable for being applied to the upgraded system. While the tool has the ability to migrate between multiple versions at a time, the earliest supported version is 1.10.1.

   The example below shows how to upgrade from 1.10.2. If upgrading from 1.12.0, --from 1.10.2 should be replaced with --from 1.12.0.

   The command line required to run the tool is different depending on which esb3024 release it is run on. On 1.14.0 it is run like this:

   cat config_backup.json | \
     podman run -i --rm \
     images.edgeware.tv/acd-confd-migration:1.14.0 \
     --in - --from 1.10.2 --to 1.14.0 \
     | tee config_upgraded.json
   

   After running the above command, apply the new configuration to confd by running cat config_upgraded.json | confcli -i.

Troubleshooting

If there is a problem running the installer, additional debug information can be output by adding -v or -vv or -vvv to the installer command, the more “v” characters, the more detailed output.

1.2.2.1 - Configuration changes between 1.12.1 and 1.14

Confd configuration changes

Below are the changes to the confd configuration between versions 1.12.1 and 1.14.x listed.

Renamed services.routing.settings.allowedProxies

The configuration setting services.routing.settings.allowedProxies has been renamed to services.routing.settings.trustedProxies.

Added services.routing.tuning.general.restApiMaxBodySize

This parameter configures the maximum body size for the REST API. It mainly applies to the configuration, which sometimes has a large payload size.

1.2.3 - Installing a 1.12 release

To install ESB3024 Router, one first needs to copy the installation ISO image to the target node where the router will be run. Due to the way the installer operates, it is necessary that the host is reachable by password-less SSH from itself for the user account that will perform the installation, and that this user has sudo access.

Prerequisites:

1. Ensure that the current user has sudo access.

   sudo -l
   

   If the above command fails, you may need to add the user to the /etc/sudoers file.

2. Ensure that the installer has password-less SSH access to localhost.

   If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.

   The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:

   mkdir -m 0700 -p ~/.ssh
   ssh-keyscan localhost >> ~/.ssh/authorized_keys
   

   Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice its recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.

3. Disable SELinux.

   The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.

   To check if SELinux is enabled:

   getenforce
   

   This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.

   setenforce 0
   

   This disables SELinux, but does not make the change persistent across reboots. To do that, edit the /etc/selinux/config file and set the SELINUX property to disabled.

   It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

1. Mount the installation ISO image under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-esb3024-1.12.1.iso /mnt/acd
   

2. Run the installer script.

   /mnt/acd/installer
   

Upgrade from ESB3024 Router release 1.10

The following steps can be used to upgrade the router from a 1.10 release to 1.12.0 or 1.12.1. If upgrading from an earlier release it is recommended to perform the upgrade in multiple steps; for instance when upgrading from release 1.8.0 to 1.12.1, it is recommended to first upgrade to 1.10.1 or 1.10.2 and then to 1.12.1.

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new release of the router, and applying the saved configuration.

1. With the router running, save a backup of the configuration.

   The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.

   Extracting the configuration using confd is the recommend approach where available.

   confcli | tee config_backup.json
   

   To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.

   curl --insecure https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

   If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.

   curl --insecure -H "X-API-Key: 1234abcd" \
     https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

2. Mount the new installation ISO under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-esb3024-1.12.1.iso /mnt/acd
   

3. Stop the router and all associated services.

   Before upgrading the router it needs to be stopped, which can be done by typing this:

   systemctl stop 'acd-*'
   

4. Run the installer script.

   /mnt/acd/installer
   

5. Migrate the configuration.

   Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.

   The confd configuration used in the 1.10 versions is not directly compatible with 1.12, and may need to be converted. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes.

   To help with migrating the configuration, a new tool has been included in the 1.12.0 release, which will automatically apply any necessary schema migrations. Further details about this tool can be found here. Confd Auto Upgrade.

   The confd-auto-upgrade tool takes as input the old configuration file, either by reading the file directly, or by reading from standard input, applies any necessary migrations between the two specified versions, and outputs a new configuration to standard output which is suitable for being applied to the upgraded system. While the tool has the ability to migrate between multiple versions at a time, the earliest supported version is 1.10.1.

   The example below shows how to upgrade from 1.10.2. If upgrading from 1.10.1, --from 1.10.2 should be replaced with --from 1.10.1.

   The command line required to run the tool is different if it is run on esb3024-1.12.0 or esb3024-1.12.1. On 1.12.1 it is run like this:

   cat config_backup.json | \
     podman run -i --rm \
     images.edgeware.tv/auto-upgrade-esb3024-1.12.1-master:20240702T151205Z-f1b53a98f \
     --in - --from 1.10.2 --to 1.12.1 \
     | tee config_upgraded.json
   

   On esb3024-1.12.0 it is run like this:

   cat config_backup.json | \
     podman run -i --rm \
     images.edgeware.tv/auto-upgrade-esb3024-1.12.0:20240619T154952Z-2b72f7400 \
     --in - --from 1.10.2 --to 1.12.0 \
     | tee config_upgraded.json
   

   After running the above command, apply the new configuration to confd by running cat config_upgraded.json | confcli -i.

Troubleshooting

If there is a problem running the installer, additional debug information can be output by adding -v or -vv or -vvv to the installer command, the more “v” characters, the more detailed output.

1.2.3.1 - Configuration changes between 1.10.2 and 1.12

Confd configuration changes

Below are the major changes to the confd configuration between versions 1.10.2 and 1.12.0/1.12.1. Note that there are no configuration changes between versions 1.12.0 and 1.12.1, so the differences apply to both.

Added services.routing.translationFunctions.hostRequest

A new translation function has been added which will allow custom Lua code to modify requests to backend hosts before they are sent.

Added services.routing.translationFunctions.session

A new translation function has been added which will allow custom Lua code to be executed after the router has made the routing decision but before generating the redirect URL.

An example use case would be for enabling instream sessions, which can be done by setting this value to return set_session_type('instream').

Removed services.routing.settins.managedSessions configuration

This configuration is no longger used.

Added services.routing.tuning.general.maxActiveManagedSessions tuning parameter.

This parameter configures the maximum number of active managed sessions.

1.2.4 - Installing release 1.10.x

To install ESB3024 Router, one first needs to copy the installation ISO image to the target node where the router will be run. Due to the way the installer operates, it is necessary that the host is reachable by password-less SSH from itself for the user account that will perform the installation, and that this user has sudo access.

Prerequisites:

1. Ensure that the current user has sudo access.

   sudo -l
   

   If the above command fails, you may need to add the user to the /etc/sudoers file.

2. Ensure that the installer has password-less SSH access to localhost.

   If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.

   The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:

   mkdir -m 0700 -p ~/.ssh
   ssh-keyscan localhost >> ~/.ssh/authorized_keys
   

   Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice its recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.

3. Disable SELinux.

   The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.

   To check if SELinux is enabled:

   getenforce
   

   This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.

   setenforce 0
   

   This disables SELinux, but does not make the change persistent across reboots. To do that, edit the /etc/selinux/config file and set the SELINUX property to disabled.

   It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

1. Mount the installation ISO image under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-esb3024-1.10.1.iso /mnt/acd
   

2. Run the installer script.

   /mnt/acd/installer
   

Upgrade from ESB3024 Router release 1.8.0

The following steps can be used to upgrade the router from release 1.8.0 to 1.10.x. If upgrading from an earlier release it is recommended to perform the upgrade in multiple steps; for instance when upgrading from release 1.6.0 to 1.10.x, it is recommended to first upgrade to 1.8.0 and then to 1.10.x.

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new release of the router, and applying the saved configuration.

1. With the router running, save a backup of the configuration.

   The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.

   Extracting the configuration using confd is the recommend approach where available.

   confcli | tee config_backup.json
   

   To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.

   curl --insecure https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

   If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.

   curl --insecure -H "X-API-Key: 1234abcd" \
     https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

2. Mount the new installation ISO under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-esb3024-1.10.1.iso /mnt/acd
   

3. Stop the router and all associated services.

   Before upgrading the router it needs to be stopped, which can be done by typing this:

   systemctl stop 'acd-*'
   

4. Run the installer script.

   /mnt/acd/installer
   

5. Migrate the configuration.

   Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.

   The confd configuration used in version 1.8.0 is not directly compatible with 1.10.x, and may need to be converted. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes.

   To determine if the configuration needs to be converted, confcli can be run like below. If it prints error messages, the configuration needs to be converted. If no error messages are printed, the configuration is valid and no further updates are necessary.

   confcli | head -n5
   [2024-04-02 14:48:37,155] [ERROR] Missing configuration key /integration
   [2024-04-02 14:48:37,162] [ERROR] Missing configuration key /services/routing/settings/qoeTracking
   [2024-04-02 14:48:37,222] [ERROR] Missing configuration key /services/routing/hostGroups/convoy-rr/hosts/convoy-rr-1/healthChecks
   [2024-04-02 14:48:37,222] [ERROR] Missing configuration key /services/routing/hostGroups/convoy-rr/hosts/convoy-rr-2/healthChecks
   [2024-04-02 14:48:37,242] [ERROR] Missing configuration key /services/routing/hostGroups/e-dns/hosts/linton-dns-1/healthChecks
   {
      "integration": {
         "convoy": {
               "bridge": {
                  "accounts": {
   

   If error messages are printed, the configuration needs to be converted. If the configuration was saved in the file config_backup.json, the conversion can be done by typing this at the command line:

   sed -E -e '/"hosts":/,/]/ s/([[:space:]]+)("hostname":.*)/\1\2\n\1"healthChecks": [],/' -e '/"apiKey":/ d' config_backup.json | \
   curl -s -X PUT -T - -H 'Content-Type: application/json' http://localhost:5000/config/__active/
   

   systemctl restart acd-confd
   

   This adds empty healthChecks sections to all hosts and removes the apiKey configuration. After that, acd-confd is restarted. See Configuration changes between 1.8.0 and 1.10.x for more details about the configuration changes.

6. Migrating configuration to esb3024-1.10.2

   When upgrading to version 1.10.2, an extra step is required to migrate the consistent hashing configuration. This step is necessary both when upgrading from an earlier 1.10 release and when upgrading from older versions. It is only needed if consistent hashing was configured in the previous version.

   To determine if consistent hashing was configured, execute the following command:

   confcli | head -n2
   [2024-05-31 09:43:55,932] [ERROR] Missing configuration key /services/routing/rules/constantine/hashAlgorithm
   {
       "integration": {
   

   If an error message about a missing configuration key appears, the configuration must be migrated. If no such error message appears, this step should be skipped.

   To migrate the configuration, execute the following command at the command line:

   curl -s http://localhost:5000/config/__active/ | \
   sed -E 's/(.*)("type":.*"consistentHashing")(,?)/\1\2,\n\1"hashAlgorithm": "MD5"\3/' | \
   curl -s -X PUT -T - -H 'Content-Type: application/json' http://localhost:5000/config/__active/
   

   This command will read the current configuration, add the hashAlgorithm configuration key, and write back the updated configuration.

7. Remove the Account Monitor container

   Older versions of the router installed the Account Monitor tool. This was removed in release 1.8.0, but if it is still present and unused, it can be removed by typing:

   podman rm account-monitor
   

8. Remove the confd-transformer.lua file

   After installing or upgrading to 1.10.x, ensure that the confd-transformer.lua script located in /opt/edgeware/acd/router/lib/standard_lua directory is removed.

   This file contains deprecated Lua language definitions which will override newer versions of those functions already present in the ACD Router’s Lua Standard Library. When upgrading beyond 1.10.2, the installer will automatically remove this file, however for this particular release, it requires manual intervention.

   rm -f /opt/edgeware/acd/router/lib/standard_lua/confd-transformer.lua
   

   After removing this file, it will be necessary to restart the router to flush the definitions from the router’s memory:

   systemctl restart acd-router
   

Troubleshooting

If there is a problem running the installer, additional debug information can be output by adding -v or -vv or -vvv to the installer command, the more “v” characters, the more detailed output.

1.2.4.1 - Configuration changes between 1.8.0 and 1.10.x

Confd configuration changes

Below are the major changes to the confd configuration between version 1.8.0 and 1.10.x listed.

Added integration.convoy section

An integration.convoy section has been added to the configuration. It is currently used for configuring the Convoy Bridge service.

Removed services.routing.apiKey configuration

The services.routing.apiKey configuration key has been removed. This was an obsolete way of giving the configuration access to the router. The key has to be removed from the configuration when upgrading, otherwise the configuration will not be accepted.

Added services.routing.settings.qoeTracking

A services.routing.settings.qoeTracking section has been added to the configuration.

Added healthChecks sections to the hosts

The hosts in the hostGroup entries have been extended with a healthChecks key, which is a list of functions that determine if a host is in good health.

For example, a redirecting host might look like this after the configuration has been updated:

{
    "services": {
        "routing": {
            "hostGroups": [
                {
                    "name": "convoy-rr",
                    "type": "redirecting",
                    "httpPort": 80,
                    "httpsPort": 443,
                    "forwardHostHeader": true,
                    "hosts": [
                        {
                            "name": "convoy-rr-1",
                            "hostname": "convoy-rr-1",
                            "ipv6_address": "",
                            "healthChecks": [
                                "health_check('convoy-rr-1')"
                            ]
                        }
                    ]
                }
            ],

Added hashAlgorithm to the consistentHashing rule

In esb3024-1.10.2 the consistentHashing routing rule has been extended with a hashAlgorithm key, which can have the values MD5, SDBM and Murmur. The default value is MD5.

1.2.5 - Installing release 1.8.0

To install ESB3024 Router, one first needs to copy the installation ISO image to the target node where the router will be run. Due to the way the installer operates, it is necessary that the host is reachable by password-less SSH from itself for the user account that will perform the installation, and that this user has sudo access.

Prerequisites:

1. Ensure that the current user has sudo access.

   sudo -l
   

   If the above command fails, you may need to add the user to the /etc/sudoers file.

2. Ensure that the installer has password-less SSH access to localhost.

   If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.

   The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:

   mkdir -m 0700 -p ~/.ssh
   ssh-keyscan localhost >> ~/.ssh/authorized_keys
   

   Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice its recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.

3. Disable SELinux.

   The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.

   To check if SELinux is enabled:

   getenforce
   

   This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.

   setenforce 0
   

   It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

1. Mount the installation ISO image under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-esb3024-1.8.0.iso /mnt/acd
   

2. Run the installer script.

   /mnt/acd/installer
   

Upgrade

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new version of the router, and applying the saved configuration.

1. With the router running, save a backup of the configuration.

   The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.

   Extracting the configuration using confd is the recommend approach where available.

   confcli | tee config_backup.json
   

   To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.

   curl --insecure https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

   If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.

   curl --insecure -H "X-API-Key: 1234abcd" \
     https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

2. Mount the new installation ISO under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-1.2.0.iso /mnt/acd
   

3. Stop the router and all associated services.

   Before upgrading the router it needs to be stopped, which can be done by typing this:

   systemctl stop 'acd-*'
   

4. Run the installer script.

   /mnt/acd/installer
   

5. Migrate the configuration.

   Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.

   The confd configuration used in version 1.6.0 is not directly compatible with 1.8.0, and may need to have a few minor manual updates in order to be valid. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes.

   To determine if the configuration needs to be manually updated, confcli can be run like below. If it prints error messages, the configuration needs to be updated. If no error messages are printed, the configuration is valid and no further updates are necessary.

   confcli services.routing | head
   [2024-02-01 19:05:10,769] [ERROR] Missing configuration key /services/routing/hostGroups/convoy-rr/forwardHostHeader
   [2024-02-01 19:05:10,779] [ERROR] Missing configuration key /services/routing/hostGroups/e-dns/forwardHostHeader
   [2024-02-01 19:05:10,861] [ERROR] 'forwardHostHeader'
   

   If error messages are printed, a forwardHostHeader configuration needs to be added to the hostGroups configuration. This can be done by running this at the command line:

   curl -s http://localhost:5000/config/__active/ | \
   sed -E 's/([[:space:]]+)"type": "(host|redirecting|dns)"(,?)/\1"type": "\2",\n\1"forwardHostHeader": false\3/' | \
   curl -s -X PUT -T - -H 'Content-Type: application/json' http://localhost:5000/config/__active/
   

   This reads the active configuration from the router, adds the “forwardHostHeader” configuration to all host groups, and then sends the updated configuration back to the router.

   See Configuration changes between 1.6.0 and 1.8.0 for more details about the configuration changes.

6. Remove the Account Monitor container

   Previous versions of the router installed the Account Monitor tool. This is no longer included, but since the previous version installed, there will be a stopped Account Monitor container. If it is not used, the container can be removed by typing:

   podman rm account-monitor
   

Troubleshooting

If there is a problem running the installer, additional debug information can be output by adding -v or -vv or -vvv to the installer command, the more “v” characters, the more detailed output.

1.2.5.1 - Configuration changes between 1.6.0 and 1.8.0

Confd configuration changes

Below are some of the configuration changes between version 1.4.0 and 1.6.0 listed. The list only contains the changes that might affect already existing configuration, enirely new items are not listed. Normally nothing needs to be done about this since they will be upgraded automatically, but they are listed here for reference.

Added enabled to contentPopularity

An enabled key has been added to services.routing.settings.contentPopularity. After the key has been added, the configuration looks like this:

{
    "services": {
        "routing": {
            "settings": {
                "contentPopularity": {
                    "enabled": true,
                    "algorithm": "score_based",
                    "sessionGroupNames": []
                },
                ...

Added selectionInputItemLimit to tuning

A selectionInputItemLimit key has been added to services.routing.tuning.general. After the key has been added, the configuration looks like this:

{
    "services": {
        "routing": {
            "tuning": {
                "general": {
                    ...
                    "selectionInputItemLimit": 10000,
                    ...

Added forwardHostHeader to hostGroups

All three hostGroup types (host, redirecting and dns) have been extended with a forwardHostHeader key. For example, a redirecting host might look like this after the change:

{
    "services": {
        "routing": {
            "hostGroups": [
                {
                    "name": "convoy-rr",
                    "type": "redirecting",
                    "httpPort": 80,
                    "httpsPort": 443,
                    "forwardHostHeader": true,
                    "hosts": [
                        {
                            "name": "convoy-rr-1",
                            "hostname": "convoy-rr-1",
                            "ipv6_address": ""
                        }
                    ]
                }
            ],
            ...

REST API configuration changes

The following items have been added to the REST API configuration. They will not need to be manually updated, the router will add the new keys with default values. Note that this is not a complete list of all changes, it only contains the changes that will be automatically added when upgrading the router.

If the router is configured via confd and confcli, these changes will be applied by them. This section is only relevant if the router is configured via the v2/configuration API.

• Added the session_translation_function key.
• Added the tuning.selection_input_item_limit key.

1.2.6 - Installing release 1.6.0

To install ESB3024 Router, one first needs to copy the installation ISO image to the target node where the router will be run. Due to the way the installer operates, it is necessary that the host is reachable by password-less SSH from itself for the user account that will perform the installation, and that this user has sudo access.

Prerequisites:

1. Ensure that the current user has sudo access.

   sudo -l
   

   If the above command fails, you may need to add the user to the /etc/sudoers file.

2. Ensure that the installer has password-less SSH access to localhost.

   If using the root user, the PermitRootLogin property of the /etc/ssh/sshd_config file must be set to ‘yes’.

   The local host key must also be included in the .ssh/authorized_keys file of the user running the installer. That can be done by issuing the following as the intended user:

   mkdir -m 0700 -p ~/.ssh
   ssh-keyscan localhost >> ~/.ssh/authorized_keys
   

   Note! The ssh-keyscan utility will result in the key fingerprint being output on the console. As a security best-practice its recommended to verify that this host-key matches the machine’s true SSH host key. As an alternative, to this ssh-keyscan approach, establishing an SSH connection to localhost and accepting the host key will have the same result.

3. Disable SELinux.

   The Security-Enhanced Linux Project (SELinux) is designed to add an additional layer of security to the operating system by enforcing a set of rules on processes. Unfortunately out of the box the default configuration is not compatible with the way the installer operates. Before proceeding with the installation, it is recommended to disable SELinux. It can be re-enabled after the installation completes, if desired, but will require manual configuration. Refer to the Red Hat Customer Portal for details.

   To check if SELinux is enabled:

   getenforce
   

   This will result in one of 3 states, “Enforcing”, “Permissive” or “Disabled”. If the state is “Enforcing” use the following to disable SELinux. Either “Permissive” or “Disabled” is required to continue.

   setenforce 0
   

   It is recommended to reboot the computer after changing SELinux modes, but the changes should take effect immediately.

Assuming the installation ISO image is in the current working directory, the following steps need to be executed either by root user or with sudo.

1. Mount the installation ISO image under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-esb3024-1.8.0.iso /mnt/acd
   

2. Run the installer script.

   /mnt/acd/installer
   

Upgrade

The upgrade procedure for the router is performed by taking a backup of the configuration, installing the new version of the router, and applying the saved configuration.

1. With the router running, save a backup of the configuration.

   The exact procedure to accomplish this depends on the current method of configuration, e.g. if confd is used, then the configuration should be extracted from confd, but if the REST API is used directly, then the configuration must be saved by fetching the current configuration snapshot using the REST API.

   Extracting the configuration using confd is the recommend approach where available.

   confcli | tee config_backup.json
   

   To extract the configuration from the REST API, the following may be used instead. Depending on the version of the router used, an API-Key may be required to fetch from the REST API.

   curl --insecure https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

   If the API Key is required, it can be found in the file /opt/edgeware/acd/router/cache/rest-api-key.json and can be passed to the API by setting the value of the X-API-Key header.

   curl --insecure -H "X-API-Key: 1234abcd" \
     https://localhost:5001/v2/configuration \
     | tee config_backup.json
   

2. Mount the new installation ISO under /mnt/acd.

   Note: The mount-point may be any accessible path, but /mnt/acd will be used throughout this document.

   mkdir -p /mnt/acd
   mount esb3024-acd-router-1.2.0.iso /mnt/acd
   

3. Stop the router and all associated services.

   Before upgrading the router it needs to be stopped, which can be done by typing this:

   systemctl stop 'acd-*'
   

4. Run the installer script.

   /mnt/acd/installer
   

5. Migrate the configuration.

   Note that this step only applies if the router is configured using confd. If it is configured using the REST API, this step is not necessary.

   See Configuration changes between 1.4.0 and 1.6.0 for instructions on how to migrate the configuration to release 1.6.0.

Troubleshooting

If there is a problem running the installer, additional debug information can be output by adding -v or -vv or -vvv to the installer command, the more “v” characters, the more detailed output.

1.2.6.1 - Configuration changes between 1.4.0 and 1.6.0

confd configuration

The confd configuration used in version 1.4.0 is not directly compatible with 1.6.0, and will need to have a few minor updates in order to be valid. If this is not done, the configuration will not be valid and it will not be possible to make configuration changes. Running confcli will cause error messages and an empty default configuration to be printed.

$ confcli services.routing.
[2023-12-12 16:08:07,120] [ERROR] Missing configuration key /services/routing/translationFunctions
[2023-12-12 16:08:07,122] [ERROR] Missing configuration key /services/routing/settings/instream/dashManifestRewrite/sessionGroupNames
[2023-12-12 16:08:07,122] [ERROR] Missing configuration key /services/routing/settings/instream/hlsManifestRewrite/sessionGroupNames
[2023-12-12 16:08:07,123] [ERROR] Missing configuration key /services/routing/settings/managedSessions
[2023-12-12 16:08:07,123] [ERROR] Missing configuration key /services/routing/tuning/target/recentDurationMilliseconds
{
    "routing": {
        "apiKey": "",
        "settings": {
            "allowedProxies": [],
            "contentPopularity": {
                "algorithm": "score_based",
                "sessionGroupNames": []
            },
            "extendedContentIdentifier": {
            ...

The first thing that needs to be done is to rename the keys sessionGroupIds to sessionGroupNames. If the configuration was backed up to the file config_backup.json before upgrading, the keys can be renamed and the updated configuration can be applied by typing this:

sed 's/"sessionGroupIds"/"sessionGroupNames"/' config_backup.json | confcli -i
[2023-12-19 12:33:17,725] [ERROR] Missing configuration key /services/routing/translationFunctions
[2023-12-19 12:33:17,726] [ERROR] Missing configuration key /services/routing/settings/instream/dashManifestRewrite/sessionGroupNames
[2023-12-19 12:33:17,727] [ERROR] Missing configuration key /services/routing/settings/instream/hlsManifestRewrite/sessionGroupNames
[2023-12-19 12:33:17,727] [ERROR] Missing configuration key /services/routing/settings/managedSessions
[2023-12-19 12:33:17,727] [ERROR] Missing configuration key /services/routing/tuning/target/recentDurationMilliseconds

The configuration has not yet been converted, so the error messages are still printed. The configuration will be converted when the acd-confd service is restarted.

systemctl restart acd-confd

This concludes the conversion of the configuration and the router is ready to be used.

Configuration changes

Below are all configuration changes between version 1.4.0 and 1.6.0 listed. Normally nothing needs to be done about this since they will be upgraded automatically, but they are listed here for reference.

Added translationFunctions block

services.routing.translationFunctions has been added. It can be added as a map with two empty strings as values, to make the top of the configuration look like this:

{
    "services": {
        "routing": {
            "translationFunctions": {
                "request": "",
                "response": ""
            },
            ...

Renamed sessionGroupIds to sessionGroupNames

The keys services.routing.settings.instream.dashManifestRewrite.sessionGroupIds and services.routing.settings.instream.hlsManifestRewrite.sessionGroupIds have been renamed to services.routing.settings.instream.dashManifestRewrite.sessionGroupNames and services.routing.settings.instream.hlsManifestRewrite.sessionGroupNames respectively. Any session group IDs need to be manually converted to session group names.

After the conversion, the head of the configuration file might look like this:

{
    "services": {
        "routing": {
            "apiKey": "",
            "settings": {
                "allowedProxies": [],
                "contentPopularity": {
                    "algorithm": "score_based",
                    "sessionGroupNames": []
                },
                "extendedContentIdentifier": {
                    "enabled": false,
                    "includedQueryParams": []
                },
                "instream": {
                    "dashManifestRewrite": {
                        "enabled": false,
                        "sessionGroupNames": []
                    },
                    "hlsManifestRewrite": {
                        "enabled": false,
                        "sessionGroupNames": []
                    },
                    "reversedFilenameComparison": false
                },
                ...

Added managedSessions block

A services.routing.settings.managedSessions block has been added. After adding the block, the configuration might look like this:

{
    "services": {
        "routing": {
            "apiKey": "",
            "settings": {
                "allowedProxies": [],
                "contentPopularity": {
                    "algorithm": "score_based",
                    "sessionGroupNames": []
                },
                ...
                "managedSessions": {
                    "fraction": 0.0,
                    "maxActive": 100000,
                    "sessionTypes": []
                },
                "usageLog": {
                    "enabled": false,
                    "logInterval": 3600000
                }
            },
            ...

Added recentDurationMilliseconds

A services.routing.tuning.target.recentDurationMilliseconds key has been added to the configuration file, with a default value of 500. After adding the key, the configuration might look like this:

{
    "services": {
        "routing": {
            "apiKey": "",
            ...
            "tuning": {
                "target": {
                    ...
                    "recentDurationMilliseconds": 500,
                    ...

Storing the updated configuration

After all these changes have been done to the configuration file, it can be applied to the router using confcli.

confcli will still display error messages because the stored configuration is not valid. They will not be displayed anymore after the valid configuration has been applied.

$ confcli -i < updated_config.json
[2023-12-12 18:52:05,500] [ERROR] Missing configuration key /services/routing/translationFunctions
[2023-12-12 18:52:05,502] [ERROR] Missing configuration key /services/routing/settings/instream/dashManifestRewrite/sessionGroupNames
[2023-12-12 18:52:05,502] [ERROR] Missing configuration key /services/routing/settings/instream/hlsManifestRewrite/sessionGroupNames
[2023-12-12 18:52:05,503] [ERROR] Missing configuration key /services/routing/settings/managedSessions
[2023-12-12 18:52:05,511] [ERROR] Missing configuration key /services/routing/tuning/target/recentDurationMilliseconds

Raw configuration

The following changes have been made to the raw configuration. If the router is configured via confd and confcli, these changes will be applied by them. This section is only relevant if the router is configured via the v2/configuration API.

Simple changes

The following keys were added or removed. They will not need to be manually updated, the router will add the new keys with default values.

• Removed the tuning.repeated_session_start_threshold_seconds key.
• Removed the lua_paths key.
• Added the tuning.target_recent_duration_milliseconds key.

EDNS proxy changes

If the router has been configured to use an EDNS server, the following has to be changed for the configuration to work.

The hosts.proxy_address key has been renamed to hosts.proxy_url and now accepts a port that is used when connecting to the proxy.

The cdns.http_port and cdns.https_port keys now configure the port that is used for connecting to the EDNS server, before they configured the port that is used for connecting to the proxy.

1.3 - Firewall

For security reasons, the ESB3024 Installer does not automatically configure the local firewall to allow incoming traffic. It is the responsibility of the operations person to ensure that the system is protected from external access by placing it behind a suitable firewall solution. The following table describes the set of ports required for operation of the router.

The “Direction” column represents the direction in which the connection is established.

• IN - The connection is originated from an outside server
• OUT - The connection is established from the host to an external server.

Once a connection is established through the firewall, bidirectional traffic must be allowed using the established connection.

For the “Source” column, the following terms are used.

• internal - Any host or network which is allowed to monitor or operate the system.
• public - Any host or subnet that can access the router. This includes any customer network that will be making routing requests.
• localhost - Access can be limited to local connections only.
• any - All traffic from any source or to any destination.

Additional Ports

Convoy bridge integration

The optional convoy-bridge service needs the ability to access the Convoy MariaDB service, which by default runs on port 3306 on all of the Convoy Management servers. To allow this integration to run, port 3306/tcp must be allowed from the router to the configured Convoy Management node.

1.4 - API Overview

ESB3024 Router provides two different types of API:s:

1. A content request API that is used by video clients to ask for content, normally using port 80 for HTTP and port 443 for HTTPS.
2. A few REST API:s used by administrators to configure and monitor the router installation, using port 5001 over HTTPS by default.

The content API won’t be described further in this document, since it’s a simple HTTP interface serving content as regular files or redirect responses.

Raw configuration – /v2/configuration

Used to check and update the raw configuration of ESB3024 Router. Note that this API is considered an implementation detail and is not documented further.

Validate Configuration – /v2/validate_configuration

Used to determine if a JSON payload is correctly formatted without actually applying its configuration. A successful return status does not guarantee that the applied configuration will work, it only validates the JSON structure.

Example request

When an expected field is missing from the payload, the validation will show which one and return an appropriate error message in its payload:

$ curl -i -X PUT \
    -d '{"routing": {"log_level": 3}}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v2/validate_configuration
HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Content-Length: 132
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

"Configuration validation: Configuration parsing failed. \
  Exception: [json.exception.out_of_range.403] (/routing) key 'id' not found"

Selection Input – /v1/selection_input

Selection input API can be used to inject external key:value data into the routing engine, making the data available when making routing decisions. An arbitrary JSON structure can be pushed to the endpoint. When performing GET or DELETE requests, specific selection input values can be accessed or deleted by including a path to the request. Note that not specifying a path will select all selection input values.

One use case for selection input is to provide data on cache availability. E.g. If you send {"edge-streamer-2-online": true} to the selection input API, you can create a routing condition eq('edge-streamer-online', true) to ensure that no traffic gets routed to the streamer if it’s offline. Note that sending the same key:value data to the selection input API will overwrite the previous value.

There is a configurable limit to how many key:value items that can be injected into the router, see the tuning parameter

$ confcli services.routing.tuning.general.selectionInputItemLimit
{
    "selectionInputItemLimit": 10000
}

Example successful request (PUT)

$ curl -i -X PUT \
    -d '{"host1_bitrate": 13000, "host1_capacity": 50000}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v1/selection_input
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Example unsuccessful request (PUT)

$ curl -i -X PUT \
    -d '{"cdn-status": {"session-count": 12345, "load-percent" 98}}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v1/selection_input
HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Content-Length: 169
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "error": "[json.exception.parse_error.101] parse error at line 1, column 57: \
    syntax error while parsing object separator - \
    unexpected number literal; expected ':'"
}

Example successful request (GET)

curl -i https://router.example:5001/v1/selection_input
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 129
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "host1_bitrate": 13000,
  "host1_capacity": 50000
}

Example successful specific value request (GET)

curl -i https://router.example:5001/v1/selection_input/path/to/value
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 129
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

1

Example successful request (DELETE)

curl -i -X DELETE https://router.example:5001/v1/selection_input
HTTP/1.1 204 OK
Access-Control-Allow-Origin: *
Content-Length: 129
X-Service-Identity: router.example-5fc78d

Example successful specific value request (DELETE)

curl -i -X DELETE  https://router.example:5001/v1/selection_input/value/to/delete
HTTP/1.1 204 OK
Access-Control-Allow-Origin: *
Content-Length: 129
X-Service-Identity: router.example-5fc78d

Example unsuccessful request (DELETE)

curl -i -X DELETE  https://router.example:5001/v1/selection_input/non/existent/value
HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Content-Length: 129
X-Service-Identity: router.example-5fc78d

Subnets – /v1/subnets

An API for managing named subnets that can be used for routing and block lists. See Subnets for more details.

PUT requests inject key value pairs with the form {<subnet>: <value>}, where <subnet> is a valid CIDR string, into ACD, e.g.:

$ curl -i -X PUT \
    -d '{"255.255.255.255/24": "area1", "1.2.3.4/24": "area2"}' \
    -H "Content-Type: application/json" \
    https://router.example:5001/v1/subnets
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

GET requests are used to fetch injected subnets, e.g.:

# Fetch all injected subnets
$ curl -i https://router.example:5001/v1/subnets
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 411
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "1.2.3.4/16": "area2",
  "1.2.3.4/24": "area1",
  "1.2.3.4/8": "area3",
  "255.255.255.255/16": "area2",
  "255.255.255.255/24": "area1",
  "255.255.255.255/8": "area3",
  "2a02:2e02:9bc0::/16": "area8",
  "2a02:2e02:9bc0::/32": "area7",
  "2a02:2e02:9bc0::/48": "area6",
  "2a02:2e02:9de0::/44": "combined_area",
  "2a02:2e02:ada0::/44": "combined_area",
  "5.5.0.4/8": "area5",
  "90.90.1.3/16": "area4"
}

DELETE requests are used to delete injected subnets, e.g.:

# Delete all injected subnets
$ curl -i https://router.example:5001/v1/subnets -X DELETE
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Both GET and DELETE requests can be specified with the paths /byKey/ and /byValue/ to filter which subnets to GET or DELETE.

• GET
• DELETE

# Fetch subnet with the CIDR string 1.2.3.4/8 if it exists
$ curl -i https://router.example:5001/v1/subnets/byKey/1.2.3.4/8
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 26
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "1.2.3.4/8": "area3"
}

# Fetch all subnets whose CIDR string begins with the IP 1.2.3.4
$ curl -i https://router.example:5001/v1/subnets/byKey/1.2.3.4
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 76
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "1.2.3.4/16": "area2",
  "1.2.3.4/24": "area1",
  "1.2.3.4/8": "area3"
}

# Fetch all subnets whose value equals 'area1'
$ curl -i https://router.example:5001/v1/subnets/byValue/area1
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 60
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "1.2.3.4/24": "area1",
  "255.255.255.255/24": "area1"
}
  

# Delete subnet with the CIDR string 1.2.3.4/8 if it exists
$ curl -i https://router.example:5001/v1/subnets/byKey/1.2.3.4/8
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

# Delete all subnets whose CIDR string begins with the IP 1.2.3.4
$ curl -i https://router.example:5001/v1/subnets/byKey/1.2.3.4
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

# Delete all subnets whose value equals 'area1'
$ curl -i https://router.example:5001/v1/subnets/byValue/area1
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d
  

Subrunner Resource Usage – /v1/usage

Used to monitor the load on subrunners, the processes performing those tasks that are possible to run in parallel.

Example request

$ curl -i https://router.example:5001/v1/usage
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 1234
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "total_usage": {
    "content": {
      "lru": 0,
      "newest": "-",
      "oldest": "-",
      "total": 0
    },
    "sessions": 0,
    "subrunner_usage": {
      [...]
    }
  },
  "usage_per_subrunner": [
    {
      "subrunner_usage": {
        [...]
      }
    },
    [...]
  ]
}

Metrics – /m1/v1/metrics

An interface intended to be scraped by Prometheus. It is possible to scrape it manually to see current values, but doing so will reset some counters and cause actual Prometheus data to become faulty.

Example request

$ curl -i https://router.example:5001/m1/v1/metrics
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 1234
Content-Type: text/plain
X-Service-Identity: router.example-5fc78d

# TYPE num_configuration_changes counter
num_configuration_changes 12
# TYPE num_log_errors_total counter
num_log_errors_total 0
# TYPE num_log_warnings_total counter
num_log_warnings_total{category=""} 123
# TYPE num_log_warnings_total counter
num_log_warnings_total{category="cdn"} 0
# TYPE num_log_warnings_total counter
num_log_warnings_total{category="content"} 0
# TYPE num_log_warnings_total counter
num_log_warnings_total{category="generic"} 10
# TYPE num_log_warnings_total counter
num_log_warnings_total{category="repeated_session"} 0
# TYPE num_ssl_errors_total counter
[...]

Node Visit Counters – /v1/node_visits

Used to gather statistics about the number of visits to each node in the routing tree. The returned value is a JSON object containing node ID names and their corresponding counter values.

See Routing Rule Evaluation Metrics for more details.

Example request

$ curl -i https://router.example:5001/v1/node_visits
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 73
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "cache1.tv": "99900",
  "offload": "100"
  "routingtable": "100000"
}

Node Visit Graph – /v1/node_visits_graph

Creates a GraphML representation of the node visitation data that can be rendered into an image to make it easier to understand the data.

See Routing Rule Evaluation Metrics for more details.

Example request

> curl -i -k https://router.example:5001/v1/node_visits_graph
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 731
Content-Type: application/xml
X-Service-Identity: router.example-5fc78d

<?xml version="1.0"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
  <key id="visits" for="node" attr.name="visits" attr.type="string" />
  <graph id="G" edgedefault="directed">
    <node id="routingtable">
      <data key="visits">100000</data>
    </node>
    <node id="cache1.tv">
      <data key="visits">99900</data>
    </node>
    <node id="offload">
      <data key="visits">100</data>
    </node>
    <edge id="e0" source="routingtable" target="cache1.tv" />
    <edge id="e1" source="routingtable" target="offload" />
  </graph>
</graphml>

Session list - /v1/sessions

Used to monitor the load on subrunners, the processes performing those tasks that are possible to run in parallel.

Example request

$ curl -k -i https://router.example:5001/v1/sessions
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 12345
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "sessions": [
    {
      "age_seconds": 103,
      "cdn": "edgeware",
      "cdn_is_redirecting": false,
      "client_ip": "1.2.3.4",
      "host": "cdn.example:80",
      "id": "router.example-5fc78d-00000001",
      "idle_seconds": 103,
      "last_request_time": "2022-12-02T14:05:05Z",
      "latest_request_path": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
      "no_of_requests": 1,
      "requested_bytes": 0,
      "requests_redirected": 0,
      "requests_served": 0,
      "session_groups": [
        "all"
      ],
      "session_groups_generation": 2,
      "session_path": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
      "start_time": "2022-12-02T14:05:05Z",
      "type": "instream",
      "user_agent": "libmpv"
    },
    [...]
  ]
}

Session details - /v1/sessions/<id: str>

Used to get details about a specific session from the above session list. The id part of the URL corresponds to the id field in one of the returned session entries in the above response.

Example request

$ curl -k -i https://router.example:5001/v1/sessions/router.example-5fc78d-00000001
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 763
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "age_seconds": 183,
  "cdn": "edgeware",
  "cdn_is_redirecting": false,
  "client_ip": "1.2.3.4",
  "host": "cdn.example:80",
  "id": "router.example-5fc78d-00000001",
  "idle_seconds": 183,
  "last_request_time": "2022-12-02T14:05:05Z",
  "latest_request_path": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
  "no_of_requests": 1,
  "requested_bytes": 0,
  "requests_redirected": 0,
  "requests_served": 0,
  "session_groups": [
    "all"
  ],
  "session_groups_generation": 2,
  "session_path": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
  "start_time": "2022-12-02T14:05:05Z",
  "type": "instream",
  "user_agent": "libmpv"
}

Content List - /v1/content

Example request

$ curl -k -i https://router.example:5001/v1/content
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 572
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "content": [
    [
      "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
      {
        "cached_count": 0,
        "content_requested": false,
        "content_set": false,
        "expiration_time": "2022-12-02T14:05:05Z",
        "key": "/__cl/s:storage1/__c/v/f/0/5/v_sintel3v_f05a05f07d352e891d79863131ef4df7/__op/hls-default/__f/index.m3u8",
        "listeners": 0,
        "manifest": "",
        "request_count": 4,
        "state": "HLS:MANIFEST-PENDING",
        "wait_count": 0
      }
    ]
  ]
}

Lua scripts – /v1/lua/<path str>.lua

Used to upload, retrieve and delete custom named Lua scripts on the router. Global functions in uploaded scripts automatically become available to Lua code in the configuration (which effectively may be viewed as hooks). Upload a script by PUTing a application/x-lua to the endpoint, and retrieve it by GETing the endpoint without payload.

Example request (PUT)

Save a Lua script under the name advanced_functions/f1.lua:

$ curl -i -X PUT \
    -d 'function fun1() return 1 end' \
    -H "Content-Type: application/x-lua" \
    https://router.example:5001/v1/lua/advanced_functions/f1.lua
HTTP/1.1 204 Successfully saved Lua file
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Example request (PUT, from file)

Upload an entire Lua file under the name advanced_functions/f1.lua:

First put your code in a file.

$ cat f1.lua
function fun1()
    return 1
end

Then upload it using the --data-binary flag to preserve newlines

$ curl -i -X PUT \
    --data-binary @f1.lua \
    -H "Content-Type: application/x-lua" \
    https://router.example:5001/v1/lua/advanced_functions/f1.lua
HTTP/1.1 204 Successfully saved Lua file
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

Example request (GET)

Request the Lua script named advanced_functions/f1.lua using a GET request:

$ curl -i https://router.example:5001/v1/lua/advanced_functions/f1.lua
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 28
Content-Type: application/x-lua
X-Service-Identity: router.example-5fc78d

function fun1() return 1 end

Example request (DELETE)

Delete the Lua script named advanced_functions/f1.lua using a DELETE request:

$ curl -i -X DELETE \
    https://router.example:5001/v1/lua/advanced_functions/f1.lua
HTTP/1.1 204 Successfully removed Lua file
Access-Control-Allow-Origin: *
Content-Length: 0
X-Service-Identity: router.example-5fc78d

List Lua scripts – /v1/lua

Used to list previously uploaded custom Lua scripts on the router, retrieving their respective paths and file checksums.

Example request

$ curl -k -i https://router.example:5001/v1/lua
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 108
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

[
  {
    "file_checksum": "d41d8cd98f00b204e9800998ecf8427e",
    "path": "advanced_functions/f1.lua"
  }
]

Debug a Lua expression – /v1/lua/debug

Used to debug an arbitrary Lua expression on the router in a “sandbox” (with no visible side effects to the state of the router), and inspect the result.

The Lua expression in the body is evaluated inside an isolated copy of the internal Lua environment including selection input. The stdout field of the resulting JSON body is populated with a concatenation of every string provided as argument to the Lua print() function during the course of evaluation. Upon a successful evaluation, as indicated by the success flag, return.value and return.lua_type_name capture the resulting Lua value. Otherwise, if valuation was aborted (e.g. due to a Lua exception), error_msg reflects any error description arising from the Lua environment.

Example successful request

$ curl -i -X POST \
    -d 'fun1()' \
    -H "Content-Type: application/x-lua" \
    https://router.example:5001/v1/lua/debug
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 123
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "error_msg": "",
  "return": {
    "lua_type_name": "number",
    "value": 1.0
  },
  "stdout": "",
  "success": true
}

Example unsuccessful request

(attempt to invoke unknown function)

$ curl -i -X POST \
    -d 'fun5()' \
    -H "Content-Type: application/x-lua" \
    https://router.example:5001/v1/lua/debug
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 123
Content-Type: application/json
X-Service-Identity: router.example-5fc78d

{
  "error_msg": "[string \"function f0() ...\"]:2: attempt to call global 'fun5' (a nil value)",
  "return": {
    "lua_type_name": "",
    "value": null
  },
  "stdout": "",
  "success": false
}

Footnotes

1.5 - Configuration

1.5.1 - WebUI Configuration

The web based user interface is installed as a separate component and can be used to configure many common use cases. After navigating to the UI, a login screen will be presented.

Enter your credentials and log in. In the top left corner is a menu to select what section of the configuration to change. The configuration that will be active on the router is added in the Routing Workflow view. However, basic elements such as classification rules and routing targets, etc must be added first. Hence the following main steps are required to produce a proper configuration:

1. Create classifiers serving as basic elements to create session groups.
2. Create session groups which, using the classifiers, tag requests/clients for later use in the routing logic. of the incoming traffic.
3. Define offload rules.
4. Define rules to control behavior of internal traffic.
5. Define backup rules to be used if the routing targets in the above step are unavailable.
6. Finally, create the desired routing workflow using the elements defined in the previous steps.

A simplified concrete example of the above steps could be:

• Create two classifiers “smartphone” and “off-net”.
• Create a session group “mobile off-net”.
• Offload off-net traffic from mobile phones to a public CDN.
• Route other traffic to a private CDN.
• If the private CDN has an outage, use the public CDN for all traffic.

Hence, to start with, define the classifiers you will need. Those are based on information in the incoming request, optionally in combination with GeoIP databases or subnet information configured via the Subnet API. Here we show how to set up a GeoIP classifier. Note that the Director ships with a compatible snapshot of the GeoIP database, but for a production system a licensed and updated database is required.

Click the plus sign indicated in the picture above to create a new GeoIP classifier. You will be presented with the following view:

Here you can enter the geographical data on which to match, or check the “Inverted” check box to match anything except the entered geographical data.

The other kinds of classifiers are configured in a similar way.

After having added all the classifiers you need, it is time to create the session groups. Those are named filters that group incoming requests, typically video playback sessions in a video streaming CDN, and are defined with the help of the classifiers. For example, a session group “off-net mobile devices” could be composed of the classifiers “off-net traffic” and “mobile devices”.

Open the Session Groups view from the menu and hit the plus sign to add a new session group.

Define the new sessions groups by combining the previously created classifiers. It is often convenient to define an “All” session group that matches any incoming request.

Next go the “CDN Offload” view:

Here you define conditions for CDN offload. Each row defines a rule for offloading a specified session group. The rule makes use of the Selection Input API. This is an integration API that provides a way to supply additional data for use in the routing decision. Common examples are current bitrates or availability status. The selection input variables to use must be defined in the “Selection Input Types” view in the “Administration” section of the menu:

Reach out to the solution engineers from Agile Content in order to perform this integration in the best way. If no external data is required, such that the offload rule can be based solely based on session groups, this is not necessary and the condition field can be set to “Always” or “Disabled”.

When clicking the plus sign to add a new CDN Offload rule, the following view is presented:

The selection input rule is phrased in terms of a variable being above or below a threshold, but also a state such as “available” taking values 0 or 1 can be supported by for instance checking if “available” is below 1.

Moving on, if an incoming request is not offloaded, it will be handled by the Primary CDN section of the routing configuration.

Add all hosts in your primary CDN, together with a weight. A row in this table will be selected by random weighted load balancing. If each weight is the same, each row will be selected with the same probability. Another example would be three rows with weights 100, 100 and 200 which would randomly balance 50% of the load on the last row and the remaining load on the first two rows, i.e. 25% on each of the first and second row. If a Primary CDN host is unavailable, that host will not take part in the random selection.

If all hosts are unavailable, as a final resort the routing evaluation will go to the final Backup CDN step:

Here you can define what to do when all else fail. If not all requests are covered, for example with an “All” session group, then the request will fail with 403 Forbidden.

Now you have defined the basic elements and it is time to define the routing workflow. Select “Routing Workflow” from the menu, as pictured below. Here you can combine the elements previously created to achieve the desired routing behavior.

When everything seems correct, open the “Publish Routing” view from the menu:

Hit “Publish All Changes” and verify that you get a successful result.

1.5.2 - Confd and Confcli

Configuration of a complex routing tree can be difficult. The command line interface tool called confcli has been developed to make it simpler. It combines building blocks, representing simple routing decisions, into complex routing trees capable of satisfying almost any routing requirements.

These blocks are translated into an ESB3024 Router configuration which is automatically sent to the router, overwriting existing routing rules, CDN list and host list.

Installation and Usage

The confcli tools are installed alongside ESB3024 Router, on the same host, and the confcli command line tool itself is made available on the host machine.

Simply type confcli in a shell on the host to see the current routing configuration:

$ confcli
{
    "services": {
        "routing": {
            "settings": {
                "trustedProxies": [],
                "contentPopularity": {
                    "algorithm": "score_based",
                    "sessionGroupNames": []
                },
                "extendedContentIdentifier": {
                    "enabled": false,
                    "includedQueryParams": []
                },
                "instream": {
                    "dashManifestRewrite": {
                        "enabled": false,
                        "sessionGroupNames": []
                    },
                    "hlsManifestRewrite": {
                        "enabled": false,
                        "sessionGroupNames": []
                    },
                    "reversedFilenameComparison": false
                },
                "usageLog": {
                    "enabled": false,
                    "logInterval": 3600000
                }
            },
            "tuning": {
                "content": {
                    "cacheSizeFullManifests": 1000,
                    "cacheSizeLightManifests": 10000,
                    "lightCacheTimeMilliseconds": 86400000,
                    "liveCacheTimeMilliseconds": 100,
                    "vodCacheTimeMilliseconds": 10000
                },
                "general": {
                    "accessLog": false,
                    "coutFlushRateMilliseconds": 1000,
                    "cpuLoadWindowSize": 10,
                    "eagerCdnSwitching": false,
                    "httpPipeliningEnable": false,
                    "logLevel": 3,
                    "maxConnectionsPerHost": 5,
                    "overloadThreshold": 32,
                    "readyThreshold": 8,
                    "redirectingCdnManifestDownloadRetries": 2,
                    "repeatedSessionStartThresholdSeconds": 30,
                    "selectionInputMetricsTimeoutSeconds": 30
                },
                "session": {
                    "idleDeactivateTimeoutMilliseconds": 20000,
                    "idleDeleteTimeoutMilliseconds": 1800000
                },
                "target": {
                    "responseTimeoutSeconds": 5,
                    "retryConnectTimeoutSeconds": 2,
                    "retryResponseTimeoutSeconds": 2,
                    "connectTimeoutSeconds": 5,
                    "maxIdleTimeSeconds": 30,
                    "requestAttempts": 3
                }
            },
            "sessionGroups": [],
            "classifiers": [],
            "hostGroups": [],
            "rules": [],
            "entrypoint": "",
            "applyConfig": true
        }
    }
}

The CLI tool can be used to modify, add and delete values by providing it with the “path” to the object to change. The path is constructed by joining the field names leading up to the value with a period between each name, e.g. the path to the entrypoint is services.routing.entrypoint since entrypoint is nested under the routing object, which in turn is under the services root object. Lists use an index number in place of a field name, where 0 indicates the very first element in the list, 1 the second element and so on.

If the list contains objects which have a field with the name name, the index number can be replaced by the unique name of the object of interest.

Tab completion is supported by confcli. Pressing tab once will complete as far as possible, and pressing tab twice will list all available alternatives at the path constructed so far.

Display the values at a specific path:

$ confcli services.routing.hostGroups
{
    "hostGroups": [
        {
            "name": "internal",
            "type": "redirecting",
            "httpPort": 80,
            "httpsPort": 443,
            "hosts": [
                {
                    "name": "rr1",
                    "hostname": "rr1.example.com",
                    "ipv6_address": ""
                }
            ]
        },
        {
            "name": "external",
            "type": "host",
            "httpPort": 80,
            "httpsPort": 443,
            "hosts": [
                {
                    "name": "offload-streamer1",
                    "hostname": "streamer1.example.com",
                    "ipv6_address": ""
                },
                {
                    "name": "offload-streamer2",
                    "hostname": "streamer2.example.com",
                    "ipv6_address": ""
                }
            ]
        }
    ]
}

Display the values in a specific list index:

$ confcli services.routing.hostGroups.1
{
    "1": {
        "name": "external",
        "type": "host",
        "httpPort": 80,
        "httpsPort": 443,
        "hosts": [
            {
                "name": "offload-streamer1",
                "hostname": "streamer1.example.com",
                "ipv6_address": ""
            },
            {
                "name": "offload-streamer2",
                "hostname": "streamer2.example.com",
                "ipv6_address": ""
            }
        ]
    }
}

Display the values in a specific list index using the object’s name:

$ confcli services.routing.hostGroups.1.hosts.offload-streamer2
{
    "offload-streamer2": {
        "name": "offload-streamer2",
        "hostname": "streamer2.example.com",
        "ipv6_address": ""
    }
}

Modify a single value:

confcli services.routing.hostGroups.1.hosts.offload-streamer2.hostname new-streamer.example.com
services.routing.hostGroups.1.hosts.offload-streamer2.hostname = 'new-streamer.example.com'

Delete an entry:

$ confcli services.routing.sessionGroups.Apple.classifiers.
{
    "classifiers": [
        "Apple",
        ""
    ]
}

$ confcli services.routing.sessionGroups.Apple.classifiers.1 -d
http://localhost:5000/config/__active/services/routing/sessionGroups/Apple/classifiers/1 reset to default/deleted

$ confcli services.routing.sessionGroups.Apple.classifiers.
{
    "classifiers": [
        "Apple"
    ]
}

Adding new values in objects and lists is done using a wizard by invoking confcli with a path and the -w argument. This will be shown extensively in the examples further down in this document rather than here.

If you have a JSON file with a previously generated confcli configuration output it can be applied to a system by typing confcli -i <file path>.

CDNs and Hosts

Configuration using confcli has no real concept of CDNs, instead it has groups of hosts that share some common settings such as HTTP(S) port and whether they return a redirection URL, serve content directly or perform a DNS lookup. Of these three variants, the two former share the same parameters, while the DNS variant is slightly different.

Each host belongs to a host group and may itself be an entire CDN using a single public hostname or a single streamer server, all depending on the needs of the user.

Host Health

When creating a host in the confd configuration, you have the option to define a list of health check functions. Each health check function must return true for a host to be selected. This means that the host will only be considered available if all the defined health check functions evaluate to true. If any of the health check functions return false, the host will be considered unavailable and will not be selected for routing. All health check functions are detailed in the section Built-in Lua functions.

• Redirecting/Host
• DNS

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: redirecting
  Adding a 'redirecting' element
    hostGroup : {
      name (default: ): edgeware
      type (default: redirecting): ⏎
      httpPort (default: 80): ⏎
      httpsPort (default: 443): ⏎
      hosts : [
        host : {
          name (default: ): rr1
          hostname (default: ): convoy-rr1.example.com
          ipv6_address (default: ): ⏎
          healthChecks : [
            healthCheck (default: always()): health_check()
            Add another 'healthCheck' element to array 'healthChecks'? [y/N]: n
          ]
        }
        Add another 'host' element to array 'hosts'? [y/N]: y
        host : {
          name (default: ): rr2
          hostname (default: ): convoy-rr2.example.com
          ipv6_address (default: ): ⏎
          healthChecks : [
            healthCheck (default: always()): ⏎
            Add another 'healthCheck' element to array 'healthChecks'? [y/N]: n
          ]
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: ⏎
]
Generated config:
{
  "hostGroups": [
    {
      "name": "edgeware",
      "type": "redirecting",
      "httpPort": 80,
      "httpsPort": 443,
      "hosts": [
        {
          "name": "rr1",
          "hostname": "convoy-rr1.example.com",
          "ipv6_address": "",
          "healthChecks": [
            "health_check()"
          ]
        },
        {
          "name": "rr2",
          "hostname": "convoy-rr2.example.com",
          "ipv6_address": "",
          "healthChecks": [
            "always()"
          ]
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y
  

$ confcli services.routing.hostGroups -w
Running wizard for resource 'hostGroups'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

hostGroups : [
  hostGroup can be one of
    1: dns
    2: host
    3: redirecting
  Choose element index or name: dns
  Adding a 'dns' element
    hostGroup : {
      name (default: ): external-dns
      type (default: dns): ⏎
      hosts : [
        host : {
          name (default: ): dns-host
          hostname (default: ): dns.example.com
          ipv6_address (default: ): ⏎
          healthChecks : [
            healthCheck (default: always()): ⏎
            Add another 'healthCheck' element to array 'healthChecks'? [y/N]: n
          ]
        }
        Add another 'host' element to array 'hosts'? [y/N]: ⏎
      ]
    }
  Add another 'hostGroup' element to array 'hostGroups'? [y/N]: ⏎
]
Generated config:
{
  "hostGroups": [
    {
      "name": "external-dns",
      "type": "dns",
      "hosts": [
        {
          "name": "dns-host",
          "hostname": "dns.example.com",
          "ipv6_address": "",
          "healthChecks": [
            "always()"
          ]
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y
  

Rule Blocks

The routing configuration using confcli is done using a combination of logical building blocks, or rules. Each block evaluates the incoming request in some way and sends it on to one or more sub-blocks. If the block is of the host type described above, the client is sent to that host and the evaluation is done.

Existing blocks

Currently supported blocks are:

• allow: Incoming requests, for which a given rule function matches, are immediately sent to the provided onMatch target.
• consistentHashing: Splits incoming requests randomly between preferred hosts, determined by the proprietary consistent hashing algorithm. The amount of hosts to split between is controlled by the spreadFactor.
• contentPopularity: Splits incoming requests into two sub-blocks depending on how popular the requested content is.
• deny: Incoming requests, for which a given rule function matches, are immediately denied, and all non-matching requests are sent to the onMiss target.
• firstMatch: Incoming requests are matched by an ordered series of rules, where the request will be handled by the first rule for which the condition evaluates to true.
• random: Splits incoming requests randomly and equally between a list of target sub-blocks. Useful for simple load balancing.
• split: Splits incoming requests between two sub-blocks depending on how the request is evaluated by a provided function. Can be used for sending clients to different hosts depending on e.g. geographical location or client hardware type.
• weighted: Randomly splits incoming requests between a list of target sub-blocks, weighted according to each target’s associated weight rule. A higher weight means a higher portion of requests will be routed to a sub-block. Rules can be used to decide whether or not to pick a target.
• rawGroup: Contains a raw ESB3024 Router configuration routing tree node, to be inserted as is in the generated configuration. This is only meant to be used in the rare cases when it’s impossible to construct the required routing behavior in any other way.
• rawHost: A host reference for use as endpoints in rawGroup trees.

• Allow
• Consistent Hashing
• Content Popularity
• Deny
• First Match
• Random
• Split
• Weighted
• Raw

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: allow
  Adding a 'allow' element
    rule : {
      name (default: ): allow
      type (default: allow): ⏎
      condition (default: ): customFunction()
      onMatch (default: ): rr1
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "content",
      "type": "contentPopularity",
      "condition": "customFunction()",
      "onMatch": "rr1"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: consistentHashing
  Adding a 'consistentHashing' element
    rule : {
      name (default: ): consistentHashingRule
      type (default: consistentHashing): 
      spreadFactor (default: 1): 2
      hashAlgorithm (default: MD5):
      targets : [
        target : {
          target (default: ): rr1
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): rr2
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): rr3
          enabled (default: True): 
        }
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "consistentHashingRule",
      "type": "consistentHashing",
      "spreadFactor": 2,
      "hashAlgorithm": "MD5",
      "targets": [
        {
          "target": "rr1",
          "enabled": true
        },
        {
          "target": "rr2",
          "enabled": true
        },
        {
          "target": "rr3",
          "enabled": true
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: contentPopularity
  Adding a 'contentPopularity' element
    rule : {
      name (default: ): content
      type (default: contentPopularity): ⏎
      contentPopularityCutoff (default: 10): 20
      onPopular (default: ): rr1
      onUnpopular (default: ): rr2
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "content",
      "type": "contentPopularity",
      "contentPopularityCutoff": 20.0,
      "onPopular": "rr1",
      "onUnpopular": "rr2"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: deny
  Adding a 'deny' element
    rule : {
      name (default: ): deny
      type (default: deny): ⏎
      condition (default: ): customFunction()
      onMiss (default: ): rr1
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "content",
      "type": "contentPopularity",
      "condition": "customFunction()",
      "onMiss": "rr1"
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: firstMatch
  Adding a 'firstMatch' element
    rule : {
      name (default: ): firstMatch
      type (default: firstMatch): ⏎
      targets : [
        target : {
          onMatch (default: ): rr1
          rule (default: ): customFunction()
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          onMatch (default: ): rr2
          rule (default: ): otherCustomFunction()
        }
        Add another 'target' element to array 'targets'? [y/N]: n
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: n
]
Generated config:
{
  "rules": [
    {
      "name": "firstMatch",
      "type": "firstMatch",
      "targets": [
        {
          "onMatch": "rr1",
          "condition": "customFunction()"
        },
        {
          "onMatch": "rr2",
          "condition": "otherCustomFunction()"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: random
  Adding a 'random' element
    rule : {
      name (default: ): random
      type (default: random): ⏎
      targets : [
        target (default: ): rr1
        Add another 'target' element to array 'targets'? [y/N]: y
        target (default: ): rr2
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "random",
      "type": "random",
      "targets": [
        "rr1",
        "rr2"
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y
  

$ confcli services.routing.rules -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: split
  Adding a 'split' element
    rule : {
      name (default: ): split
      type (default: split): ⏎
      condition (default: ): custom_function()
      onMatch (default: ): rr2
      onMiss (default: ): rr1
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "split",
      "type": "split",
      "condition": "custom_function()",
      "onMatch": "rr2",
      "onMiss": "rr1"
    }
  ]
}
Merge and apply the config? [y/n]: y
  

$ confcli services.routing.rules. -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: weighted
  Adding a 'weighted' element
    rule : {
      name (default: ): weight
      type (default: weighted): ⏎
      targets : [
        target : {
          target (default: ): rr1
          weight (default: 100): ⏎
          condition (default: always()): always()
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): rr2
          weight (default: 100): si('rr2-input-weight')
          condition (default: always()): gt('rr2-bandwidth', 1000000)
        }
        Add another 'target' element to array 'targets'? [y/N]: y
        target : {
          target (default: ): rr2
          weight (default: 100): custom_func()
          condition (default: always()): always()
        }
        Add another 'target' element to array 'targets'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "weight",
      "type": "weighted",
      "targets": [
        {
          "target": "rr1",
          "weight": "100",
          "condition": "always()"
        },
        {
          "target": "rr2",
          "weight": "si('rr2-input-weight')",
          "condition": "gt('rr2-bandwith', 1000000)"
        },
        {
          "target": "rr2",
          "weight": "custom_func()",
          "condition": "always()"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y
  

>> First add a raw host block that refers to a regular host

$ confcli services.routing.rules. -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: rawHost
  Adding a 'rawHost' element
    rule : {
      name (default: ): raw-host
      type (default: rawHost): ⏎
      hostId (default: ): rr1
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "raw-host",
      "type": "rawHost",
      "hostId": "rr1"
    }
  ]
}
Merge and apply the config? [y/n]: y

>> And then add a rule using the host node

$ confcli services.routing.rules. -w
Running wizard for resource 'rules'

Hint: Hitting return will set a value to its default.
Enter '?' to receive the help string

rules : [
  rule can be one of
    1: allow
    2: consistentHashing
    3: contentPopularity
    4: deny
    5: firstMatch
    6: random
    7: rawGroup
    8: rawHost
    9: split
    10: weighted
  Choose element index or name: rawGroup
  Adding a 'rawGroup' element
    rule : {
      name (default: ): raw-node
      type (default: rawGroup): ⏎
      memberOrder (default: sequential): ⏎
      members : [
        member : {
          target (default: ): raw-host
          weightFunction (default: ): return 1
        }
        Add another 'member' element to array 'members'? [y/N]: ⏎
      ]
    }
  Add another 'rule' element to array 'rules'? [y/N]: ⏎
]
Generated config:
{
  "rules": [
    {
      "name": "raw-node",
      "type": "rawGroup",
      "memberOrder": "sequential",
      "members": [
        {
          "target": "raw-host",
          "weightFunction": "return 1"
        }
      ]
    }
  ]
}
Merge and apply the config? [y/n]: y
  

Rule Language

Some blocks, such as the split and firstMatch types, have a rule field that contains a small function in a very simple programming language. This field is used to filter any incoming client requests in order to determine how to rule block should react.

In the case of a split block, the rule is evaluated and if it is true the client is sent to the onMatch part of the block, otherwise it is sent to the onMiss part for further evaluation.

In the case of a firstMatch block, the rule for each target will be evaluated top to bottom in order until either a rule evaluates to true or the list is exhausted. If a rule evaluates to true, the client will be sent to the onMatch part of the block, otherwise the next target in the list will be tried. If all targets have been exhausted, then the entire rule evaluation will fail, and the routing tree will be restarted with the firstMatch block effectively removed.

Example of Boolean Functions

Let’s say we have an ESB3024 Router set up with a session group that matches Apple devices (named “Apple”). To route all Apple devices to a specific streamer one would simply create a split block with the following rule:

in_session_group('Apple')

In order to make more complex rules it’s possible to combine several checks like this in the same rule. Let’s extend the hypothetical ESB3024 Router above with a configured subnet with all IP addresses in Europe (named “Europe”). To make a rule that accepts any clients using an Apple device and living outside of Europe, but only as long as the reported load on the streamer (as indicated by the selection input variable “europe_load_mbps”) is less than 1000 megabits per second one could make an offload block with the following rule (without linebreaks):

in_session_group('Apple')
    and not in_subnet('Europe')
    and lt('europe_load_mbps', 1000)

In this example in_session_group('Apple') will be true if the client belongs to the session group named ‘Apple’. The function call in_subnet('Europe') is true if the client’s IP belongs to the subnet named ‘Europe’, but the word not in front of it reverses the value so the entire section ends up being false if the client is in Europe. Finally lt('europe_load_mbps', 1000) is true if there is a selection input variable named “europe_load_mbps” and its value is less than 1000.

Since the three parts are conjoined with the and keyword they must all be true for the entire rule to match. If the keyword or had been used instead it would have been enough for any of the parts to be true for the rule to match.

Example of Numeric Functions

A hypothetical CDN has two streamers with different capacity; Host_1 has roughly twice the capacity of Host_2. A simple random load balancing would put undue stress on the second host since it will receive as much traffic as the more capable Host_1.

This can be solved by using a weighted random distribution rule block with suitable rules for the two hosts:

{
    "targets": [
        {
            "target": "Host_1",
            "condition": "always()",
            "weight": "100"
        }
        {
            "target": "Host_2",
            "condition": "always()",
            "weight": "50"
        },
    ]
}

resulting in Host_1 receiving twice as many requests as Host_2 as its weight function is double that of Host_2.

If the CDN is capable of reporting the free capacity of the hosts, for example by writing to a selection input variable for each host, it’s easy to write a more intelligent load balancing rule by making the weights correspond to the amount of capacity left on each host:

{
    "targets": [
        {
            "target": "Host_1",
            "condition": "always()",
            "weight": "si('free_capacity_host_1')"
        }
        {
            "target": "Host_2",
            "condition": "always()",
            "weight": "si('free_capacity_host_2')"
        },
    ]
}

It is also possible to write custom Lua functions that return suitable weights, perhaps taking the host as an argument:

{
    "targets": [
        {
            "target": "Host_1",
            "condition": "always()",
            "weight": "intelligent_weight_function('Host_1')"
        }
        {
            "target": "Host_2",
            "condition": "always()",
            "weight": "intelligent_weight_function('Host_1')"
        },
    ]
}

These different weight rules can of course be combined in the same rule block, with one target having a hard coded number, another using a dynamically updated selection input variable and yet another having a custom-built function.

Due to limitations in the random number generator used to distribute requests, it’s better to use somewhat large values, around 100–1000 or so, than to use small values near 0.

Built-in Functions

The following built-in functions are available when writing rules:

• in_session_group(str name): True if session belongs to session group <name>
• in_all_session_groups(str sg_name, ...): True if session belongs to all specified session groups
• in_any_session_group(str sg_name, ...): True if session belongs to any specified session group
• in_subnet(str subnet_name): True if client IP belongs to the named subnet
• gt(str si_var, number value): True if selection_inputs[si_var] > value
• gt(str si_var1, str si_var2): True if selection_inputs[si_var1] > selection_inputs[si_var2]
• ge(str si_var, number value): True if selection_inputs[si_var] >= value
• ge(str si_var1, str si_var2): True if selection_inputs[si_var1] >= selection_inputs[si_var2]
• lt(str si_var, number value): True if selection_inputs[si_var] < value
• lt(str si_var1, str si_var2): True if selection_inputs[si_var1] < selection_inputs[si_var2]
• le(str si_var, number value): True if selection_inputs[si_var] <= value
• le(str si_var1, str si_var2): True if selection_inputs[si_var1] <= selection_inputs[si_var2]
• eq(str si_var, number value): True if selection_inputs[si_var] == value
• eq(str si_var1, str si_var2): True if selection_inputs[si_var1] == selection_inputs[si_var2]
• neq(str si_var, number value): True if selection_inputs[si_var] != value
• neq(str si_var1, str si_var2): True if selection_inputs[si_var1] != selection_inputs[si_var2]
• si(str si_var): Returns the value of selection_inputs[si_var] if it is defined and non-negative, otherwise it returns 0.
• always(): Returns true, useful when creating weighted rule blocks.
• never(): Returns false, opposite of always().

These functions, as well as custom functions written in Lua and uploaded to the ESB3024 Router, can be combined to make suitably precise rules.

Combining Multiple Boolean Functions

In order to make the rule language easy to work with, it is fairly restricted and simple. One restriction is that it’s only possible to chain multiple function results together using either and or or, but not a combination of both conjunctions.

Statements joined with and or or keywords are evaluated one by one, starting with the left-most statement and moving right. As soon as the end result of the entire expression is certain, the evaluation ends. This means that evaluation ends with the first false statement for and expressions since a single false component means the entire expression must also be false. It also means that evaluation ends with the first true statement for or expressions since only one component must be true for the entire statement to be true as well. This is known as short-circuit or lazy evaluation.