Tutorials and How-To's

• 1: Getting Started
• 2: Using the configuration-and-monitoring GUI
• 3: Viewing the multiview and outputs
• 4: Creating a video mixer control panel in Companion
• 5: Using the audio control GUI
• 6: Setting up multi-view outputs using the REST API
• 7: Security in Agile Live
• 8: Statistics in Agile Live
• 9: Using Media Player metadata in HTML pages

1 - Getting Started

Prerequisites

A working knowledge of the technical basis of the Agile Live solution is beneficial when starting with this tutorial but not required. It is assumed that sufficient hardware has been provided, i.e.:

• at least one server to be used for ingesting media via an SDI capture card or as NDI via a network interface
• at least one server to be used for running the production pipeline(s) including the rendering engine that performs the video and audio mixing, as well as the generic control panels
• one server that will run all management and orchestration software, including the System Controller, the configuration-and-monitoring GUI, the Prometheus database and the associated Grafana instance
• at least one computer that will have control panel hardware connected to it, run the control panel integration software, and receive and display the multiview video stream and the monitor feed media stream

The servers may be in the field, in a datacenter or in a private or public cloud. For cloud VMs, the requirement is that they have Nvidia GPUs with NvEnc+NvDec support. All major cloud providers have VMs with Nvidia T4’s.

In a very minimal system not all of these machines need to be separate; in fact, all the above could run on a single computer if so is desired. Recommended hardware requirements for the servers running the ingest and production pipelines can be found here.

Installing all components

A guide to installing all the components of the system can be found here, start by following it.

Using the configuration-and-monitoring GUI to set up and monitor a production workflow

A guide on how to use the basic configuration-and-monitoring GUI to set up a working system can be found here.

Using the Grafana dashboard example

Viewing the multiview and outputs

The multiview and outputs (low delay monitor feed and high quality program out) are sent from the platform as MPEG TS streams in SRT or UDP. For viewing the streams, please refer to this guide.

Creating a video mixer control panel in Companion

A good starting point for a video mixer control panel is to use Bitfocus Companion with an Elgato Streamdeck device. Some guidelines for how to use those to build a video mixer control panel for Agile Live can be found here

Using the audio control GUI

There is an example audio control panel integration that also includes a GUI. A guide on how to use it can be found here

2 - Using the configuration-and-monitoring GUI

Using the configuration-and-monitoring GUI to set up and monitor a production workflow

When loading the configuration and monitoring GUI in a web browser the first screen you will encounter is the login screen. Log in using the credentials that you set when installing. After logging in you will see the home screen. Make sure that the bottom strip says that the System Controller is online and that the database is online.

Configure sources

On the home screen, click the Inventory Management button. This will take you to the inventory management page. On the left side there is a list of all sources in the system at the moment. It is possible to filter the list based on source type, location, and active state. For each source, it is possible (and recommended) to click the Edit button to open the source configuration editing page. It is recommended to set a descriptive name (e.g. “Front wide camera”), a type, and a location for the source. This view is also used to configure which embedded audio channels to transport up to the production pipelines. Click Save to save your changes and exit this page and return to the inventory management page. Repeat the process for each source. Whenever a new source is added to the system, it is recommended to perform this step on it. Press the Home button to return to the home screen.

Create and start a production configuration

A production configuration is a description of how the system is configured for a specific production workflow, e.g. a TV program. On the home screen, click the Create New button to create a new production configuration. Give it a decriptive name, e.g. “Stockholm local news”, and click the Create button. This will bring you to the configuration page for the new production configuraton. Click Add Source to add a source to this production configuration. This will bring out a list of available sources on the left side of the screen. Again, it is possible to filter the sources the same way as on the inventory management page. Click Add for the sources that you want to have in your production configuration. The order of the sources is the same as the order they will be presented in the multiviewer. To change the order, drag and drop them. For each source, it is possible to set the UMD text by clicking the name above the thumbnail and editing it.

A preset is a collection of all the detailed variables of each part of the production configuration that will be set up. This includes detailed quality settings, network latency settings, etc. To choose a preset for your current production configuration, click Select Preset and select one of the pre-defined presets. Once that has been done, some settings can be edited by clicking the button with the cogwheel icon.

To run a production configuration click the Start button. Once the production is started, click the Home button to return to the home screen.

Use the runtime monitoring

On the top of the screen there will now be a pane for each pipeline that is running, and information on where to find the multiview and output streams. For the production configuration that is currently running, there is a Runtime Monitoring button, that is either green if all is well, or red if there are any error counters that have increased in the last minute. Clicking the button takes you to the runtime monitoring screen.

The runtime monitoring screen is built as an expanding tree, where each level and instance has a status icon that is either green or red. This way it is very easy to trace an error through the tree down to the erroring counter. On the lowest level, the actual counters can have three colors. White means it is an informational counter that cannot indicate an error. Green color means that the counter is able to indicate an error but at the moment it is fine. Red color means that the counter is currently indicating an error. Such a counter is red if it has increased in the last 60 seconds. A red counter will revert to green after 60 seconds without increasing.

Editing the database

To add or edit production presets and new users, the Mongo database can be manually edited.

The GUI tool MongoDB Compass can be used to edit the database, but MongoDB Shell and other MongoDB tools will also work. Since you connect to the MongoDB via the network, Compass can be used on any computer with network access to the GUI server on port 27017.

After installing MongoDB Compass, connect to the database by pressing the “New Connection” button. In the URI field, exchange localhost for the hostname of the GUI server (unless you run Compass on the same server as the GUI, of course). Add agile_live_gui last in the URL-field, to access the GUI database only. Then under Advanced Connection Options, select Authentication and use method Username/Password. Type the <API_USER> and <API_PASSWORD> you put in the mongo-init.js file when setting up the GUI server. The URI field should then look something like:

mongodb://<API_USER>:<API_PASSWORD>@<HOSTNAME>:27017/agile-live-gui?authMechanism=DEFAULT

Add or edit production presets

Once connected you will see the databases to the left. The database agile-live-gui contains the collections used by the GUI. To add a new, or modify an old production preset, open the presets collection under the agile-live-gui database. To create a new preset, the easiest way is to clone an old preset and make your changes to it. Hover your mouse over one of the presets you want to clone, and press the Clone document button on the top right of the preset. In the popup window you can make your edits to the new preset. Most important is to set a new name for it in the "name" field. This is the name that will be displayed in the GUI when selecting which preset to use. Under pipelines the quality settings for the streams from the Ingests to the Pipelines can be set.

Add new users to the system

New users to the GUI can be added to the Mongo database as well. Open the users collection under the agile-live-gui database. Then add a new user by pressing the Clone document button on one of the existing users. In the popup window, set the username of the new user and remove the hashed password, leaving the password field empty (i.e. "password": ""). Press insert to add the new user to the database. Then open the GUI in a web browser. On the sign-in page, write the username of the new user and then type a password to set it as the new password in the database. A new password must have at least 8 characters to be approved.

3 - Viewing the multiview and outputs

Viewing the multiview and outputs

The multiview and outputs (low delay monitor feed and high quality program out) are sent from the platform as MPEG TS streams in SRT or UDP. For viewing the streams, VLC can be used. For the multiview and low delay monitoring feeds it is important that the viewer does not add a lot of buffering delay. Therefore there is a need to configure it not to do so. The setting network-caching can be used to set the buffering. It can be done in the VLC GUI in settings, or on the command line by specifying --network-caching=140 when starting VLC, for example when using Windows:

C:\Program Files (x86)\VideoLAN\VLC\vlc.exe" --network-caching=140 srt://mydomain:9900

4 - Creating a video mixer control panel in Companion

Create buttons in Companion by using the action tcp-udp: Send Command to send control commands that are listed here

An example companion config can be found here

5 - Using the audio control GUI

The audio control GUI is written in Python and should run in all systems that have Python3. However, since it uses tkinter and the native support of tkinter on some platforms is not perfect, there may occur problems. The application is verified to work on Ubuntu 22.04.

It is started with: ./acl_audio_control.py

First connect to an open TCP port on an acl-tcpcontrolpanel by setting the correct IP/hostname and port under “Mixer network address”. Once connected, use the “Add strip” and “Remove strip” buttons to add and remove volume strips. Then map the audio of each strip. Select the “Slot” of the connected (video) source you want to take the audio from. Then select which channel in the source to find the audio (or to find the L channel in case of stereo, the R channel will be assumed to be the next channel) and if the strip should be a mono or stereo strip. The name of the strip can be changed by clicking on it and editing it.

Once everything is set up, the configuration can be saved with the “Save” and “Save as…” buttons. The configuration is stored as a json file.

6 - Setting up multi-view outputs using the REST API

Prerequisites

Knowledge on how to set up a system and create a production using the REST API.

The multi-view generator

The Agile Live system includes a built-in multi-view generator that can be used with any Rendering Engine and produce multiple composited mosaic video steams of selected inputs with a given layout. The multi-view generator can technically be used both in the High Quality and Low Delay pipelines, but in the general case it is only used in the Low Delay pipeline.

Start a new multi-view output stream

To start a new multi-view output stream, use the [POST] /pipelines/{uuid}/multiviews/ endpoint in the REST API, where {uuid} is the UUID of the Pipeline component from which you want to create the multi-view output (generally the Low Delay pipeline). Then provide a JSON payload to describe how the multi-view output should look. For instance:

{
  "layout": {
    "output_height": 1080,
    "output_width": 1920,
    "views": [
      {
        "width": 960,
        "height": 540,
        "x": 0,
        "y": 0
        "input_slot": 1,
        "label": "Camera 1",
      },
      ... // More views here
    ]
  },
  "output": {
    "format": "MPEG-TS-SRT",
    "frame_rate_d": 1,
    "frame_rate_n": 50,
    "ip": "0.0.0.0",
    "port": 4567,
    "video_format": "AVC",
    "video_kilobit_rate": 5000
  }
}

The description contains two parts, the layout part describing how the multi-view output will look and the output part to describe the format the stream should be encoded to. Starting with the output part, this will generate a stream of MPEG-TS over SRT format, with the SRT in server mode (IP address is 0.0.0.0) on port 4567. The frame rate is set to 50 FPS and the encoding format is set to AVC (H264) and 5000 kilobits/second. To playback the stream, use ffplay or VLC.

ffplay srt://<ip-address>:4567

or in VLC, press Media and then Open network stream and write srt://<ip-address>:4567 in the dialog that pops up. Note that the default latency in VLC is quite high (several seconds) compared to ffplay.

Multi-view layout

The layout part of the JSON payload contains a description of the multi-view. The parameters output_height and output_width defines the resolution of the main frame in pixels, i.e. the resolution of the resulting video stream. Within this frame, multiple views can be placed. Each view can display any of the input sources connected to the pipeline, or any of the auxiliary feedback outputs from the Rendering Engine, more on that below. The views are defined as a width and height of the view in pixels and the x and y position of the view inside the outer frame (the position of the top left corner of the view in pixels from the top left corner of the frame, see the illustration below). Then the content of the view is defined using the input_slot parameter and an optional label to display under the view is provided using the label parameter. In case two views overlap, the one first appearing in the view array will be drawn on top of the other one. In case the aspect ratio of the view does not match the aspect ratio of the source, the video image will be resized with its aspect ratio kept, to fit entirely inside the defined view. Areas that are not covered by the video will be filled with black.

The input_slot parameter can be any of the sources connected to an input slot in the Pipeline. Just use the same input slot as used when the source was connected using the [POST] /streams endpoint. The input_slot parameter of the multi-view layout can also be any of the auxiliary feedback outputs of the Rendering Engine, i.e. video streams that are created by the Rendering Engine. These feedback streams also have their own input_slot number. To show which such feedback streams the Rendering Engine is providing, use the [GET] /pipelines/{uuid} endpoint with the UUID of the Pipeline. The result might look something like this:

{
  "uuid": "ab552c6c-4226-a9ec-66b8-65f98753beaa",
  "name": "My LD Pipeline",
  "type": "pipeline",
  "streams": [
    ...
  ],
  "feedback_streams": [
    {
      "input_slot": 1001,
      "name": "Program"
    },
    {
      "input_slot": 1002,
      "name": "Preview"
    }
  ],
  "multiviews": [
    ...
  ]
}

In this case the Rendering Engine provides two feedback streams to the multi-view generator, one called “Program” on input_slot 1001 which is the mixed output stream and one “Preview” on input_slot 1002 which shows what the Rendering Engine is previewing right now. The “Program” output is the same as the video stream you get from the Output component connected to the same pipeline.

Tally borders

Tally borders, i.e. colored borders around the views shown in the multi-view output that tell which views are currently being used in the program output (red frame) and the preview output (green frame) are automatically added to the multi-view outputs. Setting, removing and choosing the colors of the borders are controlled by the Rendering Engine. The Agile Live Rendering engine (acl-renderingengine) will also use borders on views containing the Program and Preview feedback outputs.

Update the layout

Once a multi-view output is created it will turn up when listing multi-view outputs using the [GET] /pipelines/{uuid}/multiviews endpoint, as well as in some of the other Pipeline related endpoints. When a multi-view output is created, the layout of that multi-view stream can be updated, without interrupting the stream. This can be done using the [PUT] /pipelines/{uuid}/multiviews/{viewid} endpoint. The viewid parameter is the id of the multi-view output stream within that Pipeline component. This id is provided in the JSON-blob returned with the HTTP response when creating the multi-view, and can also be retrieved using the [GET] /pipelines/{uuid} endpoint.

In the update request, an array of views, just like the one used when creating the multi-view output, is provided as a JSON payload. It is not possible to change the resolution of the main frame/stream or change any of the values set in the output section when the stream was created, as it would interrupt the stream. In case you want to alter any of these settings, you will need to stop the multi-view output and create a new one with the desired format.

Multiple multi-view outputs

The Pipeline supports outputting multiple multi-view output stream. Each output stream can have its own, unique layout and have different labels on the views compared to other multi-view outputs. The maximum number of multi-view output streams are limited by the hardware the Pipeline is running on.

Remove a multi-view output

A multi-view output stream can be closed by calling the [DELETE] /pipelines/{uuid}/multiviews/{viewid} with the id of the multi-view output to close.

7 - Security in Agile Live

Connections in the system

There are several types of network connections in the Agile Live system that need encryption to protect them, which is especially important when running productions in a public cloud environment or at least transporting parts of a production’s data over public internet. These connections are:

• The HTTP requests to the REST API of the System Controller
• The WebSocket connections between the components in the system and the System Controller
• The SRT/RIST streams transporting the video and audio from the Ingests to the Production Pipelines.
• The TCP control connections used to transport control commands from the Control Panel to the Production Pipeline, and to propagate the commands from the Low Delay pipeline to the High Quality pipeline.

The last two connection types above will always be encrypted. Each stream/connection will be setup with its own uniquely generated encryption key. The REST API and the WebSocket connections between the components and the System Controller, however, require some manual setup by the user.

Enable HTTPS in the System Controller

Turning the HTTP requests of the REST API and the WebSocket connection between the components and the System Controller secure, requires the following steps to be taken:

1. Get a TLS certificate from a certificate authority
2. Point out the cert.pem and key.pem provided by your certificate authority in the System Controller config file under https, attributes certificate_file and private_key_file. Also make sure enabled under https is set to true.
3. Start the System controller and make sure it prints that the API is served using https
4. Try accessing the REST API using HTTPS to make sure it works as expected

WebSocket connections will automatically be encrypted when HTTPS is turned on.

In case a component trying to connect to the System Controller using HTTPS and fails with the following error message: Failed to connect to System Controller, message: set_fail_handler: 8: TLS handshake failed this is likely because the ACL_SYSTEM_CONTROLLER_IP is not set to the same hostname as the “Common Name” field is set to in the certificate. Set ACL_SYSTEM_CONTROLLER_IP to the same hostname as in the System Controller’s certificate to make it work.

Pre-shared key (PSK)

Components connecting to the System Controller must authorize themselves to be accepted and connected. When components connect to the System Controller they present a pre-shared key (PSK) to the System Controller, which must match the PSK set in the System Controller for the component to be allowed to connect. The PSK is set in the System Controller’s config file using the psk attribute before it is started. The PSK must be 32 characters long. All components must then connect using the same PSK, by setting the environment variable ACL_SYSTEM_CONTROLLER_PSK before starting the application. In case the user fails to set the PSK before starting the application, it will immediately quit and print an error message.

Authentication in the REST API

The System Controller REST API is protected with Basic Authentication that will require a username and password to access the API endpoints. To configure this, edit the client_auth part of the System Controller’s config file. Choose which username and password should grant access to the REST API and make sure enabled is set to true to actually activate the authentication check. The configured password is tested for strength on start. The System Controller will not start if the password is judged to be too easy to crack with brute-force.

System Controller HTTP headers

The System Controller serves the REST API and sets a few headers in the responses. Among these headers, the Content-Security-Policy (CSP) can be fine-tuned according to your configuration needs.

The CSP header set in the distributed configuration file is:

Content-Security-Policy: frame-ancestors 'none'

This header plays a crucial role in enhancing security by controlling iframe (inline frame) embedding, specifically for this site. The frame-ancestors directive with the value 'none' signifies that no pages or sites are allowed to embed the current page as an iframe. This helps prevent various attacks such as cross-site scripting (XSS) and code injection by blocking the embedding of this site in others’ iframes.

8 - Statistics in Agile Live

Statistics

Timestamps and measurements

Let’s start by taking a closer look on how the system measures these statistics:

/ingests/{uuid}/streams/{stream_uuid} {
    processing_time_audio,
    processing_time_video,
    audio_encode_duration,
    video_encode_duration
    }

/pipelines/{uuid}/streams/{stream_uuid} {
    time_to_arrival_audio,
    time_to_arrival_video,
    time_to_ready_audio,
    time_to_ready_video,
    audio_decode_duration,
    video_decode_duration
    }

All media that is ingested will be assigned a timestamp as soon as the audio and video is captured, this is the capture timestamp. With this timestamp as a reference the system calculates three more timestamps which all measure the time passed since the capture timestamp was taken.

processing_time_audio & processing_time_video

After the capture step, the audio and video is converted into the format that the user selected, this may involve resizing, deinterlacing and any color format conversion that is needed. Next step will be to encode the audio and video. Once the encoding is done, a new timestamp is taken which then measures the difference between the capture time and the current time after the encoding is done. Note that these values may be negative (and in rare cases the following timestamps), in cases where the system has shifted the capture timestamp into the future to handle drifting.

time_to_arrival_audio & time_to_arrival_video

When the Rendering Engine receives the EFP stream, and has received a full audio or video frame, a new timestamp is taken, which measures the difference between the capture time and the current time after a full frame was received. This duration also includes the network transport time, i.e. the maximum time that the network protocol will use for re-transmissions. This value can be configured with the max_network_latency_ms value when setting up the stream.

time_to_ready_audio & time_to_ready_video

Next step will be to decode the compressed audio and video into raw data. When the media is decoded a new timestamp is taken, which measures the difference between the capture time and the current time after the decoding of a frame is done. This will be when a frame is ready to be delivered to the Rendering Engine. The alignment of the stream between Ingest and Pipeline must be larger than time_to_ready_audio/video statistics, otherwise the frames will be too late and dropped. This is a good value to check if you experience dropped frames, and potentially increase the alignment value if so. You can also experiment with lowering the bitrate or decreasing the max_network_latency_ms setting. If a frame is ready earlier than the aligment time, it will be queued until the system reaches the alignment time. For the video frames there is also a queue of compressed frames before the actual decoding, this queue makes sure that only a maximum of 10 decoded frames are kept in memory after the decoder. Due to this the difference between these timestamps and the alignment value normally is never larger than the duration of 10 video frames.

The sytem also tracks for how long a frame is processed by the encoders and decoders with the following values:

audio_encode_duration & video_encode_duration

These metrics show how long time audio and video frames take to pass through their respective encoder. This duration includes any delay in the encoder as well, measuring the time from when each frame is passed to the encoder until the encoder returns the compressed frame. For video this also includes the time to do any format conversions, resizing and de-interlacing. Note that B-frames will cause the metric to increase to more than one frame-time, as more frames are needed in some cases before the previous ones can be outputted to the stream.

audio_decode_duration & video_decode_duration

These metrics show how long time audio and video frames take to pass through their respective decoder. This duration includes any delay in the decoder as well, measuring the time from when each frame is passed to the decoder until the decoder returns the raw frame. Also here, B-frames will cause this time to be longer than one frame time, because the decoder has to wait for more frames before the decoding can start.

9 - Using Media Player metadata in HTML pages

Subscribing to metadata

HTML pages loaded into the Rendering Engine can subscribe to metadata from Media Players located on other input slots of the Rendering Engine. The metadata can for instance be used to create graphics to aid the production crew.

In order for Media Player metadata to propagate to an HTML page, the page must first subscribe to metadata updates. This is done in the JavaScript code of the HTML page by calling the function window.aclInputSlotMetadataSubscribe() when the page has been loaded. After doing this the global variable aclInputSlotMetadata will become available and updated periodically. In the simplest case:

<body onLoad="window.aclInputSlotMetadataSubscribe()">

Metadata structure

aclInputSlotMetadata is a JSON object with the Media Player information located in the “media_players” section and further organized by which input slot the Media Player is located at. An example with a media file being played at input slot number 2:

var aclInputSlotMetadata = {
   "media_players": {
     "2": {
         "file_name": "/mnt/media/ads/Clip32.mov",
         "is_paused": false,
         "is_looping": true,
         "current_time_ms": 400000,
         "start_time_ms": 300000,
         "input_duration_ms": 1800000,
         "duration_ms": 120000,
         "time_left_ms": 20000
     }
   }
}

HTML example

The file “media-playback.html” located in the Agile Live examples ZIP file provides a full example of how to subscribe to, and use, the metadata from a Media Player. It can be loaded into the Rendering Engine to create a view that displays the metadata of the currently playing file. The HTML page accepts the Media Player’s input slot number as a query parameter. For example, if a Media Player is located at input slot number 4 and media-playback.html is served at localhost:8899 (on the same machine as the Rendering Engine is running) the following commands can be used to load the page into a new input slot:

html create 5 1920 1080
html load http://localhost:8899/media-playback.html?slot=4