Adding trick play to a DASH or HLS stream

Trick play, or trick mode as it is sometimes called, is a feature that gives viewers visual feedback while they are rewinding or fast-forwarding a stream (i.e., 'scrubbing' through it). Adding this feature to your streams can be done in different ways:

• For DASH, a sync-samples only track can be added to enable trick play support (reference: [1]). Alternatively, a track with tiled thumbnails in JPEG format can be added (reference: [2]). Also, for Live use cases a more rudimentary implementation of trick play can be enabled on-the-fly by specifying the --fixed_gop option for a stream.

• For HLS, an I-frame Playlist is required for trick play. Such a playlist can reference either the original video track or a sync-samples only track.

  • For HLS sync-samples only tracks are supported for fMP4 HLS, but not for HLS with Transport Streams (HLS TS).

  • Using a sync-samples only track can make trick play's visual feedback more smooth, as it may offer a higher density of frames compared to an I-frame Playlist that merely references the original video track.

Note

Unified Streaming does not support Roku EXT-X-IMAGE-STREAM-INF HLS tags for Trickplay. However Roku also accepts standard HLS Trickplay tags.

Note

Unified Streaming does not support VTT text tracks for Trickplay.

How to package a sync-samples only MP4

--trickplay

You can package a CMAF compliant MP4 that contains only sync-samples with proper signaling using mp4split with the --trickplay option:

#!/bin/bash

mp4split --trickplay \
  -o tos-1000k-sync-samples.cmfv \
  tears-of-steel-avc1-1000k.cmfv

Note

The --trickplay option is only supported for CMAF output, which is specified using the .cmfv file extension and can only contain one track per file.

How to package a tiled thumbnails MP4

Attention

Unified Remix does not process tiled thumbnails tracks (it will ignore them). Therefore, when you would like to include tiled thumbnails in a remixed stream, follow these steps:

1. Create a remixed MP4 file as you would do normally

2. Create tiled thumbnails track(s) from the remixed MP4

3. Create a server manifest using both the remixed MP4 and the separate tiled thumbnails tracks as input

--trickplay --fourcc=jpeg

You can package a CMAF compliant MP4 that contains tiled thumbnails encoded in JPEG format with proper signaling using mp4split with the --trickplay and --fourcc=jpeg options.

Important

The --trickplay --fourcc=jpeg option combination requires a working media processing setup. This involves the creation of a transcoders file and the choice between running media processing locally or with the support of a remote Media Processing Unit. Please see the Media processing architecture overview to understand what is the right option for you and for configuration instructions.

Once you have successfully set up your Media Processing, if using local transcoding, a tiled thumnails MP4 track can be packaged by providing mp4split with the path or URL to a transcoders file, using the --transcoders_file option:

#!/bin/bash

mp4split --trickplay \
  --fourcc=jpeg \
  --transcoders_file=/etc/config/my_transcoders_file.usp \
  -o tos-1000k-tiled-thumbnails.cmfv \
  tears-of-steel-avc1-1000k.cmfv

When using remote transcoding instead, the same result is achieved by providing mp4split with the URL of a Media Processing Unit, using the --transcode_proxy_pass option:

#!/bin/bash

mp4split --trickplay \
  --fourcc=jpeg \
  --transcode_proxy_pass=http://my-media-processing-unit/my-location/ \
  -o tos-1000k-tiled-thumbnails.cmfv \
  tears-of-steel-avc1-1000k.cmfv

By default, the tiled thumbnails are generated by decoding a frame from the source video track every 5 seconds, tiling the frames into a 3 by 4 grid, and encoding the result as a JPEG picture. These pictures are stored in the CMAF output file, in a track with special signaling for DASH packaging.

Important

• If your Media Processing setup (either local or remote) includes plugins based on the Intel media SDK plugin, be aware that it does not support 4:2:2 subsampling or 10 bits per sample, so anything over the H.264 High Profile should not be used as source to create tiled thumbnails from.

• Only CMAF output is supported, which can only contain one track per file and is specified using the .cmfv file extension.

Options for packaging a tiled thumbnails MP4

The interval between thumbnails, the number of horizontal and vertical tiles, and the JPEG encoding quality used can all be adjusted, using the following options:

--thumbnail_interval

Specifies the interval between thumbnails, as a fraction of seconds. Must be an exact interval at which keyframes occur in the source. Defaults to 5 seconds.

--thumbnail_width, --thumbnail_height

Specify the width and height to resize each thumbnail before they are tiled. An even integer larger than zero must be used. If the integer is odd, it will be rounded to the nearest even integer, and a warning will be emitted. This leads to three scenarios:

• When width and height are not specified, the thumbnails will not be resized and keep the original size from the source video track (default behavior)

• When only width or height is specified but not both, the thumbnails will be rescaled according to the specified width or height by calculating the value that is not specified according to the original image's aspect ratio

• When both width and height are specified, the thumbnails will be resized according to the specified values (please note that this will distort the original image's aspect ratio when the specified values do not match this aspect ratio)

In other words: if you want to rescale the original images when generating tiled thumbnails it is best to specify either width or height but not both.

--thumbnail_htiles, --thumbnail_vtiles

Specify the number of horizontal and vertical tiled thumbnails in a grid, as an integer larger than zero. Defaults to 3 horizontal and 4 vertical tiles.

--thumbnail_quality

Note

Changed in version 1.14.3: The --thumbnail_quality=<value> command line option has been replaced by the quality=<value> attribute in the transcoders file. See here.

Specify the JPEG encoding quality as an integer, where 1 represents the lowest quality and 100 represents the highest. Defaults to 30.

How to add trick play to DASH

For DASH, trick play can be added with a sync-samples only track or with a tiled thumbnails track. The first can be packaged using the --trickplay option, while the second can be packaged using --trickplay --fourcc=jpeg options. How you can use these tracks to add trick play to your DASH stream is explained below.

When using Unified Origin, it is also possible to dynamically generate a trick play track, without creating a sync-samples only track or a tiled thumbnails track beforehand.

With a sync-samples only track

After you have created a sync-samples only file (using --trickplay), you can simply add that file to the other media files that should become part of your stream and, based on these files, create a MPD (Unified Packager) or a server manifest (Unified Origin)

Unified Packager

#!/bin/bash

mp4split --package_mpd \
  -o tos-with-trickplay.mpd \
  tears-of-steel-avc1-1000k.cmfv \
  tos-1000k-sync-samples.cmfv \
  tears-of-steel-aac-128k.cmfa

Unified Origin

#!/bin/bash

mp4split -o tos-with-trickplay.ism \
  tears-of-steel-avc1-1000k.cmfv \
  tos-1000k-sync-samples.cmfv \
  tears-of-steel-aac-128k.cmfa

With a tiled thumbnails track

After you have created a CMAF compliant MP4 that contains tiled thumbnails (using --trickplay --fourcc=jpeg), you can add that file to the other media files that should become part of your stream and, based on these files, create a MPD (Unified Packager) or a server manifest (Unified Origin)

Note

It is currently not possible to add trick play with tiled thumbnails to HLS playlists, as there is no defined signaling for it yet.

Unified Packager

#!/bin/bash

mp4split --package_mpd \
  -o tos-with-tiled-thumbnails.ism \
  tears-of-steel-avc1-1000k.cmfv \
  tos-1000k-tiled-thumbnails.cmfv \
  tears-of-steel-aac-128k.cmfa

Unified Origin

#!/bin/bash

mp4split -o tos-with-tiled-thumbnails.ism \
  tears-of-steel-avc1-1000k.cmfv \
  tos-1000k-tiled-thumbnails.cmfv \
  tears-of-steel-aac-128k.cmfa

Without a sync-samples only or tiled thumbnails track (Live only)

This feature is only supported by Unified Origin for Live. Origin can add trick play functionality to DASH output without using a sync-samples only or tiled thumbnails track. To do so, your stream must have a fixed GOP, which you should then signal to Origin with the fixed_gop option.

--fixed_gop

The fixed_gop option can be used to specify the length of the GOPs in a live stream. As it's a fixed value it should only be used on streams that are encoded with a fixed GOP length. The specified value must be an integer or a fraction.

Use this options when creating a Live server manifest, and Origin will add a trick play AdaptationSet to the MPD with a frameRate equal to '1' divided by the fixed GOP length that was specified. It will also include an EssentialProperty with a schemeIdUri set to http://dashif.org/guidelines/trickmode and a value that is equal to the @id of the video AdaptationSet that it is associated with:

<EssentialProperty
  schemeIdUri="http://dashif.org/guidelines/trickmode"
  value="2">
</EssentialProperty>

How to add trick play to fMP4 HLS

When streaming fMP4 HLS, the I-frame Playlists necessary for adding trick play to HLS need to be generated manually for each video track if you are using Unified Packager. Depending on whether or not you are working with a dedicated sync-samples only track, this process is slightly different.

If you are using Unified Origin, the I-frame Playlists will be generated automatically and refer to the same tracks as the Media Playlists for the video tracks. This is comparable to how trick play functionality works for Unified Packager when not using a sync-samples only track, see below.

Without a sync-samples only track

Unified Packager

Use the --create_iframe_playlist option to create an I-frame Playlist that references the original video track:

#!/bin/bash

mp4split -o iframe.m3u8 \
  --create_iframe_playlist \
  tears-of-steel-avc1-1000k.cmfv
mp4split -o video.m3u8 \
  --fragment_duration=192/24 \
  tears-of-steel-avc1-1000k.cmfv
mp4split -o audio.m3u8 \
  --fragment_duration=384/48 \
  tears-of-steel-aac-128k.cmfa

mp4split -o master_playlist.m3u8 \
  iframe.m3u8 \
  video.m3u8 \
  audio.m3u8

Unified Origin

Unified Origin will automatically generate the necessary I-frame Playlists based on the video tracks, so no specific tracks need to be added when creating the server manifest, although you need to use the hls.fmp4 option to enable fMP4 HLS output.

#!/bin/bash

mp4split -o tos-with-tiled-thumbnails.ism \
  --hls.fmp4 \
  tears-of-steel-avc1-1000k.cmfv \
  tears-of-steel-aac-128k.cmfa

--fixed_gop (Live only)

If you're streaming Live and want Unified Origin to concatenate multiple GOPs from your source into longer segments for your HLS output, it is possible to combine hls.minimum_fragment_length and fixed_gop to achieve this while still having your I-frame Playlists reference the sync-samples at the start of each GOP for optimal trick play functionality.

The fixed_gop option is used to specify the length of the GOPs in a stream. As it's a fixed value it should only be used on streams that are encoded with a fixed GOP length. The specified value must be an integer or a fraction.

With a sync-samples only track

Using a sync-samples only track for trick play functionality can increase efficiency, because requests from a player that is scrubbing the timeline will require reading a much smaller file.

Unified Packager

When a track contains only sync-samples, mp4split automatically detects this and will create an I-frame Playlist for it, instead of a regular Media Playlist. Using the --create_iframe_playlist option is not necessary.

#!/bin/bash

mp4split -o iframe-sync-samples.m3u8 \
  tos-1000k-sync-samples.cmfv
mp4split -o video.m3u8 \
  tears-of-steel-avc1-1000k.cmfv
mp4split -o audio.m3u8 \
  tears-of-steel-aac-128k.cmfa

mp4split -o master_playlist.m3u8 \
  iframe-sync-samples.m3u8 \
  video.m3u8 \
  audio.m3u8

Note

When using a sync-samples only file to add trick play to HLS, a i common error reported by Apple's MediaStreamValidator is the following must fix issue:

Error: Framerate change without discontinuity tag detected

This problem occurs when the source of sync-samples only file has variable GOP size (which is reflected in the sync-samples only file). To solve it, make sure that your source has a constant GOP size.

Unified Origin

The use of a sync-samples only file to add trick play functionality to Origin's fMP4 HLS output is not yet supported. Rely on the I-frame Playlists that Origin automatically generates based on the regular video tracks instead.

How to add trick play to HLS TS

The I-frame Playlists necessary for adding trick play to HLS are automatically generated for each video track in a HLS TS stream. This is true for static as well as dynamic packaging. In the case of the latter, you must specify a client manifest version of 4 or higher in order for Unified Origin to add the necessary I-frame Playlists, using --hls.client_manifest_version.

For Live, using the fixed_gop option as described in trickplay_fmp4_hls is supported as well.

Footnote

[1]

See 'Trick Mode support' in the DASH-IF Interoperability Guidelines 4.2

[2]

See 'Tiles of thumbnail images' in the DASH-IF Interoperability Guidelines 4.2