General

• 1: Monitoring and Logging
• 2: CCMI URLs
• 3: ESF Format
• 4: Licensing

1 - Monitoring and Logging

Health check

You can check the health of the server by making a request to /healthz, which should return a 200 OK response.

Structured Logging

To aid processing, structured logging is available. This makes the logs more coherent, and more suitable for processing by external tools. The log formats journald or consolejson will trigger structured logging.

Log messages related to a specific HTTP request include the log field named request_id which can be used to filter out specific requests.

The log field named topic is always present and is used to separate different areas of logging. The topics are access, messages, licensing and error. It is useful when querying the log.

Log formats

When run from the command line the application can log in one of the formats consolepretty, consolejson or journald depending on the value of the -logformat parameter. Logging can be disabled by using the value discard.

$ ew-vod2cbm -upstream http://127.0.0.1 -logformat consolejson

consolepretty logs to standard output (stdout) in a human-readable format. Each log line starts with the date and the log message followed by the context- dependent log fields:

2022-02-07T11:54:42Z INF Incoming request bytes_out=27592 latency_ms=6.18251 method=GET ...

consolejson logs to stdout using a JSON object for each message:

{level:info,topic:access,time:2022-02-07T12:06:07Z,bytes_out:27592, ...}

When running as a systemd service, it logs directly to journald. In this case field names are upper case whereas the values keep their format.

Here are a few examples of querying the logs:

$ journalctl -u ew-vodjit TOPIC=access -o verbose

To see entries with a specific request ID:

$ journalctl -u ew-vod2cbm REQUEST_ID=host/B1Po0eYz1d-000006 -o verbose

To see the call stack in the event of a crash:

$ journalctl -u ew-manip TOPIC=error -o verbose

Using the verbose output includes all available name-value fields. Without this only the log message is printed. The field named JSON contains the original structured log message as sent by the service. To filter out only this field --output-fields=JSON can be used, however this requires a systemd version of at least 236.

$ journalctl -u ew-vod2cbm --output-fields=JSON TOPIC=access

To continuously monitor the service:

$ journalctl -fu ew-vod2cbm

Log levels

The startup log level is set according to the service configuration. The log level for a running service can be changed by posting a form-field named level to the loglevel endpoint. For example:

$ curl -F level=debug ew-vod2cbm-server:8090/loglevel

The available log levels and how they relate to systemd priority values is listed below:

Log level  Systemd priority
trace      7
debug      7
info       6
warn       4
error      3
fatal      2
panic      0

To e.g. see all error [3] and warning [4] messages using journalctl:

$ journalctl -u ew-vod2cbm PRIORITY=3 PRIORITY=4

Use a GET request to see the current log level:

$ curl ew-vod2cbm-server:8090/loglevel

Messages with higher level than the configured one will not be logged.

Monitoring

The services generate Prometheus data, available at the endpoint: /metrics. The respective metrics for the services are:

ew-vod2cbm

ew_audio_segment_requests_total
ew_audio_segment_request_duration_milliseconds
ew_video_segment_requests_total
ew_video_segment_request_duration_milliseconds
ew_subs_segment_requests_total
ew_subs_segment_request_duration_milliseconds
ew_segment_times_requests_total
ew_segment_times_request_duration_milliseconds
ew_content_info_requests_total
ew_content_info_request_duration_milliseconds

ew-manip

ew_hls_requests_total
ew_hls_request_duration_milliseconds
ew_dash_requests_total
ew_dash_request_duration_milliseconds

ew-vodjit

ew_cmaf_segment_requests_total
ew_cmaf_segment_request_duration_milliseconds
ew_dat_segment_requests_total
ew_dat_segment_request_duration_milliseconds
ew_content_info_requests_total
ew_content_info_request_duration_milliseconds
ew_manifest_requests_total
ew_manifest_request_duration_milliseconds

2 - CCMI URLs

Edgeware’s origin URL format is called CCMI URLs and is the format that StreamBuilder understands. A typical URL looks like:

http://<server>/__cl/<cl>/__c/<content>/__op/<op>/__f/<pres><ext>

where the different parts are indicated by prefixes starting with two underscores, and where the file extension <ext> starts with a period if present.

The most important StreamBuilder fields are:

__cl : content location with parts prefix:name
       + cg:<live> stands for channel group <live> and denote a group
         of live channels configured with the name <live>
       + s:<vod> stands for a storage path for vod assets
__c  : the actual content or asset path (can contain slashes)
__op : output profile `<op>`, defining the encryption/DRM system or segment addressing
       Segments with different output profiles are not compatible
__f : the file path but also the way to choose a presentation.
      A presentation is a manifest variation, for example exluding some variants.
      The segments can be the same between presentation.
      The extensions ".m3u8" maps to HLS, ".mpd" maps to DASH and "Manifest" or
      ".ism/Manifest" map to MSS

The <pres> part is matched to a presentation configured in repackager. The <live>, <vod>, <op> are also defined in the repackager configuration.

For segments, the URLs are further enhanced with a config_id (for live/catchup), a variant, and a segment number (for HLS and DASH with $Number$)

__f/<cfg_id>/<variant>/<nr>.cmfv
__f/<cfg_id>/<variant>/<nr>.ts

or with a time (for DASH and MSS)

__f/<cfg_id>/<variant>/D<time>.cmfv
__f/<cfg_id>/QualityLevels({bitrate})/Fragments(video={start time})

In addition, there are other possible parts like

__dci : DRM Content ID - an id used to lookup the key from license server
__account : Account to distinguish different tenants

Finally, there are query parameters for time intervals

?startTime=
?stopTime=

and query parameters for variant flags mask

?p=<variant_flags_mask>

In addition, there may also be a part ?pre=1 that triggers ESB3020 - ew-manip.

3 - ESF Format

ESF is Edgeware’s CMAF-based storage format for both live and VoD content. It saves media data in CMAF tracks, and has additional metadata files. It supports H.264/AVC and HEVC/H.265 video, AAC and AC-3 audio, and wvtt subtitles.

Asset media data

The media data is stored as CMAF tracks in files with extensions “.cmfv”, “.cmfa”, and “.cmft”, for video, audio, and subtitles, respectively. The subtitle format is “wvtt” which is much more storage-efficient than “stpp”.

Asset metadata

The metadata for the assets are stored in two places:

1. a single content_info.json file describing the tracks
2. segment info (.dat) files describing the segments, one per track

Content info

Content info is stored in a file named content_info.json inside the asset directory. It is a JSON file containing enough information about the tracks of the assets, to be able to fill in all information about media tracks in HLS, DASH, or MSS manifests. This includes codecs, languages, time scales, bitrates etc. However, it does not contain information about individual segments. Such information is stored in segment info files.

Segment info

Metadata about the segments are stored in segment info files with extension .dat. There is exactly one such file for each media file.

These files describe the data of the individual segments of the CMAF tracks with a 32-byte entry for each segment:

For VoD assets, an init segment is stored at the start of the file. Its size is given by the Offset of the first media segment in the segment info file.

Commonalities with DASH OnDemand format

DASH OnDemand stores media data in the same type of CMAF tracks as ESF. However, the metadata is stored in other structures.

To describe the asset and its variants, there is a manifest called Media Presentation Description (MPD) which is an XML file with file extension .mpd.

It is similar to the ESF content_info.json file. It has explicit switching groups called Adaptation Sets, but is lacking some other information compared to ESF like video parameter sets which are needed to generate MSS manifests.

Similar to ESF, there is a second structure that contains the information about the segments. In the case of DASH OnDemand, this information is stored in a sidx box inside the CMAF track itself. Its position is at the beginning of the media file right after the init segment, and before the actual media data.

By generating a DASH MPD file, and inserting a sidx box in the media tracks, it is possible to make VoD assets which are both compatible with DASH OnDemand and the ESF format. The new ESB3031 ew-vodingest tool generates such combined ingested files. For DASH OnDemand, a complete WebVTT file is used instead of wvtt segments. That complete WebVTT file is generated by extracting and concatenating all subtitle cues from the wvtt subtitle tracks used in the ESF format.

Live storage using ESF

The main reason to use ESF instead of DASH OnDemand, is that the latter does not support live content, but requires a static structure. In fact, since the sidx box must be placed before the media, it is not even possible to concatenate media segments and write a sidx box at the end. With ESF we can have the same format for both VoD and live.

The SegmentInfo files are separated from the media files and use 32 bytes per segment. These are therefore easy to seek and also to grow, by just adding another 32 bytes for each segment. In the ESB3003 catchup buffer, we store live content in one-minute files.

4 - Licensing

All Agile Content Processing products require a valid license file to operate. For the server products a few endpoints such as /metrics and /licensing are accessible without a valid license, but the core functionality requires a license.

The license file

The license file must be obtained from Agile Content along with your software. Depending on your agreement the file will contain either an online or offline license. For online licenses, the machine running the service, must have an internet connection, in order to access the license server.

The command-line products, such as ew-vodingest only support offline licenses.

Using the license file

The license must be placed in: /etc/edgeware/<product-name>/license.json e.g: /etc/edgeware/ew-vod2cbm/license.json You must never attempt to edit the license file. When you receive a new license.json, and your product is a service, the service must be restarted.

License expiry

Your product will not work beyond the expiry date. For online as well as offline licenses the server product logs will warn you that the service is about to expire, when the expiry date is closer than 7 days. For automatic expiry monitoring, the /licensing HTTP endpoint is recommended. Please note that there will be no warning before expiry for command-line products.

When the service, under a valid service contract, is about to expire, your license must be renewed:

• For online licenses, Agile Content should renew your service license on a central license server. The licensing status can be checked by calling the /licensing endpoint (see below).
• For offline licenses, you should obtain a new license file.

Online licenses

Less than three minutes after a service is started it starts to query the license server periodically. In case of temporary network or license server outages (up to 2 hours), your service will continue to function.

Checking the license: /licensing endpoint

You may query status of your license by querying your service with the /licensing endpoint. The response, in JSON-format, looks like:

$ curl localhost:8090/licensing
{
    "status": "valid",
    "pending_until": "",
    "expiry": "2022-12-31T23:59:29Z",
    "invalid_reason": "",
    "license_type": "online"
}

The call will return "status": "pending" during temporary network outage and for less than three minutes after the service has been restarted. The response for an expired license looks like:

{
    "status": "invalid",
    "pending_until": "",
    "expiry": "2021-12-31T23:59:29Z",
    "invalid_reason": "expired",
    "license_type": "online"
}

Checking the license: server log

If the service has, or is about to expire, the server log will show the status under "topic":"licensing". For example:

{"level":"warn","request_id":"adam/HLYRsgfrwz-000005","topic":"licensing",
"LicenseExpiryDate":"2022-08-30T23:59:09Z","time":"2022-08-24T11:42:28Z",
"message":"License will expire in less than 7 days, please remember to get a renewed license"}

These log messages only appear when a licensed endpoint e.g. /segment_times.json is called.