Setting up multi-view outputs using the REST API

Prerequisites

Knowledge on how to set up a system and create a production using the REST API.

The multi-view generator

The Agile Live system includes a built-in multi-view generator that can be used with any Rendering Engine and produce multiple composited mosaic video steams of selected inputs with a given layout. The multi-view generator can technically be used both in the High Quality and Low Delay pipelines, but in the general case it is only used in the Low Delay pipeline.

Start a new multi-view output stream

To start a new multi-view output stream, use the [POST] /pipelines/{uuid}/multiviews/ endpoint in the REST API, where {uuid} is the UUID of the Pipeline component from which you want to create the multi-view output (generally the Low Delay pipeline). Then provide a JSON payload to describe how the multi-view output should look. For instance:

{
  "layout": {
    "output_height": 1080,
    "output_width": 1920,
    "views": [
      {
        "width": 960,
        "height": 540,
        "x": 0,
        "y": 0
        "input_slot": 1,
        "label": "Camera 1",
      },
      ... // More views here
    ]
  },
  "output": {
    "format": "MPEG-TS-SRT",
    "frame_rate_d": 1,
    "frame_rate_n": 50,
    "ip": "0.0.0.0",
    "port": 4567,
    "video_format": "AVC",
    "video_kilobit_rate": 5000
  }
}

The description contains two parts, the layout part describing how the multi-view output will look and the output part to describe the format the stream should be encoded to. Starting with the output part, this will generate a stream of MPEG-TS over SRT format, with the SRT in server mode (IP address is 0.0.0.0) on port 4567. The frame rate is set to 50 FPS and the encoding format is set to AVC (H264) and 5000 kilobits/second. To playback the stream, use ffplay or VLC.

ffplay srt://<ip-address>:4567

or in VLC, press Media and then Open network stream and write srt://<ip-address>:4567 in the dialog that pops up. Note that the default latency in VLC is quite high (several seconds) compared to ffplay.

Multi-view layout

The layout part of the JSON payload contains a description of the multi-view. The parameters output_height and output_width defines the resolution of the main frame in pixels, i.e. the resolution of the resulting video stream. Within this frame, multiple views can be placed. Each view can display any of the input sources connected to the pipeline, or any of the auxiliary feedback outputs from the Rendering Engine, more on that below. The views are defined as a width and height of the view in pixels and the x and y position of the view inside the outer frame (the position of the top left corner of the view in pixels from the top left corner of the frame, see the illustration below). Then the content of the view is defined using the input_slot parameter and an optional label to display under the view is provided using the label parameter. In case two views overlap, the one first appearing in the view array will be drawn on top of the other one. In case the aspect ratio of the view does not match the aspect ratio of the source, the video image will be resized with its aspect ratio kept, to fit entirely inside the defined view. Areas that are not covered by the video will be filled with black.

The input_slot parameter can be any of the sources connected to an input slot in the Pipeline. Just use the same input slot as used when the source was connected using the [POST] /streams endpoint. The input_slot parameter of the multi-view layout can also be any of the auxiliary feedback outputs of the Rendering Engine, i.e. video streams that are created by the Rendering Engine. These feedback streams also have their own input_slot number. To show which such feedback streams the Rendering Engine is providing, use the [GET] /pipelines/{uuid} endpoint with the UUID of the Pipeline. The result might look something like this:

{
  "uuid": "ab552c6c-4226-a9ec-66b8-65f98753beaa",
  "name": "My LD Pipeline",
  "type": "pipeline",
  "streams": [
    ...
  ],
  "feedback_streams": [
    {
      "input_slot": 1001,
      "name": "Program"
    },
    {
      "input_slot": 1002,
      "name": "Preview"
    }
  ],
  "multiviews": [
    ...
  ]
}

In this case the Rendering Engine provides two feedback streams to the multi-view generator, one called “Program” on input_slot 1001 which is the mixed output stream and one “Preview” on input_slot 1002 which shows what the Rendering Engine is previewing right now. The “Program” output is the same as the video stream you get from the Output component connected to the same pipeline.

Tally borders

Tally borders, i.e. colored borders around the views shown in the multi-view output that tell which views are currently being used in the program output (red frame) and the preview output (green frame) are automatically added to the multi-view outputs. Setting, removing and choosing the colors of the borders are controlled by the Rendering Engine. The Agile Live Rendering engine (acl-renderingengine) will also use borders on views containing the Program and Preview feedback outputs.

Update the layout

Once a multi-view output is created it will turn up when listing multi-view outputs using the [GET] /pipelines/{uuid}/multiviews endpoint, as well as in some of the other Pipeline related endpoints. When a multi-view output is created, the layout of that multi-view stream can be updated, without interrupting the stream. This can be done using the [PUT] /pipelines/{uuid}/multiviews/{viewid} endpoint. The viewid parameter is the id of the multi-view output stream within that Pipeline component. This id is provided in the JSON-blob returned with the HTTP response when creating the multi-view, and can also be retrieved using the [GET] /pipelines/{uuid} endpoint.

In the update request, an array of views, just like the one used when creating the multi-view output, is provided as a JSON payload. It is not possible to change the resolution of the main frame/stream or change any of the values set in the output section when the stream was created, as it would interrupt the stream. In case you want to alter any of these settings, you will need to stop the multi-view output and create a new one with the desired format.

Multiple multi-view outputs

The Pipeline supports outputting multiple multi-view output stream. Each output stream can have its own, unique layout and have different labels on the views compared to other multi-view outputs. The maximum number of multi-view output streams are limited by the hardware the Pipeline is running on.

Remove a multi-view output

A multi-view output stream can be closed by calling the [DELETE] /pipelines/{uuid}/multiviews/{viewid} with the id of the multi-view output to close.