NTP

Agile Live NTP instructions

To ensure that all ingest servers and production pipelines are in sync, a clock synchronization protocol called NTP is required. Historically that has been handled by a systemd service service called timesyncd.

Chrony

Starting from version 4.0.0 of Agile Live the required NTP client has switched from systemd-timesyncd to chrony. Chrony is installed by defualt in many Linux distributions since it has a lot of advantages over using systemd-timesyncd. Some of the advantages are listed below.

  • Full NTP support (systemd-timesyncd only supports Simple NTP)
  • Syncing against multiple servers
  • More complex but more accurate
  • Gradually adjusts the system time instead of hard steps

Installation

Install chrony with:

$ sudo apt install chrony

This will replace systemd-timesyncd if installed.

NTP servers

The default configuration for chrony uses a pool of 8 NTP servers to synchronize against. It is recommended to use these default servers hosted by Ubuntu. Verify that the configuration is valid by checking the configuration file after installing.

$ cat /etc/chrony/chrony.conf
...
pool ntp.ubuntu.com        iburst maxsources 4
pool 0.ubuntu.pool.ntp.org iburst maxsources 1
pool 1.ubuntu.pool.ntp.org iburst maxsources 1
pool 2.ubuntu.pool.ntp.org iburst maxsources 2
...

The file should also contain the line below for the correct handling of leap seconds. It is used to set the leap second timezone to get the correct TAI time.

leapsectz right/UTC

Useful Commands

For a simple check to see that the system time is synchronized, run:

$ timedatectl
               Local time: Wed 2023-03-08 08:23:37 CET
           Universal time: Wed 2023-03-08 07:23:37 UTC
                 RTC time: Wed 2023-03-08 07:23:37
                Time zone: Europe/Stockholm (CET, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

To get more details on how the system time is running compared to NTP, use the chrony client chronyc. The subcommand tracking shows information about the current system time and the selected time server used.

$ chronyc tracking
Reference ID    : C23ACA14 (sth1.ntp.netnod.se)
Stratum         : 2
Ref time (UTC)  : Wed Mar 08 07:23:37 2023
System time     : 0.000662344 seconds slow of NTP time
Last offset     : -0.000669114 seconds
RMS offset      : 0.001053432 seconds
Frequency       : 10.476 ppm slow
Residual freq   : +0.001 ppm
Skew            : 0.295 ppm
Root delay      : 0.007340557 seconds
Root dispersion : 0.001045418 seconds
Update interval : 1028.7 seconds
Leap status     : Normal

The sources subcommand to get a list of the other time servers that chrony uses.

$ chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^- prod-ntp-5.ntp4.ps5.cano>     2   6    17     8    +59us[  -11us] +/-   23ms
^- pugot.canonical.com           2   6    17     7  -1881us[-1951us] +/-   48ms
^+ prod-ntp-4.ntp1.ps5.cano>     2   6    17     7  -1872us[-1942us] +/-   24ms
^+ prod-ntp-3.ntp1.ps5.cano>     2   6    17     6   +878us[ +808us] +/-   23ms
^- ntp4.flashdance.cx            2   6    17     7   -107us[ -176us] +/-   34ms
^+ time.cloudflare.com           3   6    17     8   -379us[ -449us] +/- 5230us
^* time.cloudflare.com           3   6    17     8   +247us[ +177us] +/- 5470us
^- ntp5.flashdance.cx            2   6    17     7  -1334us[-1404us] +/-   34ms

Leap seconds

Due to small variations in the Earth’s rotation speed a one second time adjustment is occasionally introduced in UTC (Coordinated Universal Time). This is usually handled by stepping back the clock in the NTP server one second. These hard steps could cause problems in such a time critical product as video production. To avoid this the system uses a clock known as TAI (International Atomic Time), which does not introduce any leap seconds, and is therefore slightly offset from UTC (37 seconds in 2023).

Leap smearing

Another technique for handling the introduction of a new leap second is called leap smearing. Instead of stepping back the clock in on big step, the leap second is introduced in many minor microsecond intervals during a longer period, usually over 12 to 24 hours. This technique is used by most of the larger cloud providers. Most NTP servers that use leap smearing don’t announce the current TAI offset and will thereby cause issues if used in combination with systems that are using TAI time. It is therefore highly recommended to use Ubuntu’s default NTP servers as described above, as none of these use leap smearing. Mixing leap smearing and non-leap smearing time servers will result in that components in the system will have clocks that are off from each other by 37 seconds (as of 2023), as the ones using leap smearing time servers without TAI offset will set the TAI clock to UTC.