Tiled Thumbnails with FFmpeg

Introduction

New in version 1.11.12.

Introducing FFmpeg plugins for libfmp4, which enable decoding of AVC/H.264 and HEVC/H.265 video, image resizing, and encoding of JPEG images.

Together, these features enable creation of tiled thumbnail tracks using only FFmpeg-based components, and without using any Intel Media SDK components.

In particular, this should work on any type of Linux or Windows machine, including virtual machines on Amazon EC2 or other cloud providers, and container solutions such as Docker. A GPU is not required.

Note

Encoding AVC/H.264 and HEVC/H.265 is specifically not enabled. This will be achieved using a different system, using a plugin and a separate service. The reason is that we package FFmpeg libraries with only the components available under the GNU LGPL version 2.1, and this does not include any AVC/H.264 or HEVC/H.265 encoding capabilities.

Installation

Install the mp4split and mp4split-ffmpeg-plugins packages. This will automatically install the required dependencies, in particular the ffmpeg-usp package, which contains the FFmpeg libraries.

Alpine and Docker containers

Follow the Add repository on Alpine to setup the repository access (location and public key).

Then install the packages:

#!/bin/sh

apk update
apk add mp4split mp4split-ffmpeg-plugins

Rocky and Red Hat-derived distributions

Follow the Add repository on Rocky Linux, AlmaLinux, or Red Hat Enterprise Linux (RHEL) to setup the repository access (location and public key).

Then install the packages:

#!/bin/bash

yum update
yum install mp4split mp4split-ffmpeg-plugins

Ubuntu and other Debian-derived distributions

Follow the Add repository on Ubuntu / Debian to setup the repository access (location and public key).

Then install the packages:

#!/bin/bash

apt-get update
apt-get install mp4split mp4split-ffmpeg-plugins

Windows

Follow the regular Installation on Windows installation instructions for installing mp4split using the install-usp.ps1 PowerShell script.

Afterwards, also install the ffmpeg-usp component in a similar fashion:

PowerShell -File "C:\Full\Path\Where\You\Downloaded\install-usp.ps1" -version 1.11.12 ffmpeg-usp -targetdir "C:\Program Files\Unified Streaming" -addtopath

Adding a license key

Finally, add a license key for mp4split. Conventionally, the license key is stored in a file /etc/usp-license.key.

Usually, you specify the license file on each invocation of mp4split, using the --license_key= option, but for convenience you can also set up an environment variable:

export UspLicenseKey=/etc/usp-license.key

Setting up a transcoders file

To create tiled thumbnail tracks using the FFmpeg plugins, instead of using the Intel Media SDK plugins, which are the default, we have to use a so-called transcoders file. This is a configuration file for libfmp4's decoding, filtering and encoding plugins. It is in plain text format, and any empty lines or lines starting with a # character are ignored, so you can put comments in them.

Create a file named /etc/ffmpeg-transcoders.usp, containing the following:

# Override default transcoders with FFmpeg-based ones.

video_decoder_avc     avcodec
video_filter_resize   swscale
video_encoder_jpg     avcodec

Creating tiled thumbnail tracks

To create tiled thumbnail tracks with the FFmpeg plugins, use the --transcoders_file= option in addition to the usual options for tiled thumbnails (see --trickplay --fourcc=jpeg).

AVC/H.264 input

For example, the following creates a tiled thumbnail CMAF track named tiled_thumbnails_from_avc.cmfv from an AVC/H.264 input video, with the following properties:

  • Interval between thumbnails: 4 seconds

  • Each thumbnail is resized to 224x100

  • Each tiling contains 4 horizontal and 5 vertical thumbnails

  • The resulting tile is compressed with medium JPEG quality

Note

Changed in version 1.14.3: The --thumbnail_quality=<value> command line option has been replaced by the quality=<value> attribute in the transcoders file. See here.

mp4split \
  --transcoders_file=/etc/ffmpeg-transcoders.usp \
  --trickplay \
  --fourcc=jpeg \
  --thumbnail_interval=4 \
  --thumbnail_width=224 \
  --thumbnail_height=100 \
  --thumbnail_htiles=4 \
  --thumbnail_vtiles=5 \
  --thumbnail_quality=50 \
  -o tiled_thumbnails_from_avc.cmfv \
  https://live.unified-streaming.com/video/tears-of-steel/tears-of-steel-avc1-1000k.cmfv

The resulting console output should look approximately like:

mp4split version=1.11.12 (25476)    Copyright 2007-2021 CodeShop B.V.

I0.170 Manifest file:///tmp/tiled_thumbnails_from_avc.cmfv
I0.170 Track 1:
I0.170 src=tears-of-steel-avc1-1000k.cmfv
I0.170 video bitrate=1001000/1144232 name=video_eng
I0.170 id=1 timescale=12288 lang=en
I0.170 vide/avc1 dref=1 bitrate=1001000/1144232 size=784x350 sar=1:1 dar=56:25 codecs=avc1.4D401F
I4.273 Loaded transcoders file file:///etc/ffmpeg-transcoders.usp
I4.276 Obtained reference to dynamic library /usr/bin/../lib/libfmp4_video_decoder_avc_avcodec.so
I4.276 Keeping reference to dynamic library /usr/bin/../lib/libfmp4_video_decoder_avc_avcodec.so
I4.278 video_decoder_avc_avcodec: switching to sample description index 1: vide/avc1 dref=1 (tears-of-steel-avc1-1000k.cmfv) bitrate=1001000/1144232 size=784x350 sar=1:1 dar=56:25 codecs=avc1.4D401F avc profile=77 compatibility=64 level=31
I4.284 Obtained reference to dynamic library /usr/bin/../lib/libfmp4_video_filter_resize_swscale.so
I4.284 Keeping reference to dynamic library /usr/bin/../lib/libfmp4_video_filter_resize_swscale.so
I4.289 video_filter_resize_swscale: resize from [784,350] to [224,100], crop [224x100], @[0,0]
I4.292 Obtained reference to dynamic library /usr/bin/../lib/libfmp4_video_encoder_jpg_avcodec.so
I4.292 Keeping reference to dynamic library /usr/bin/../lib/libfmp4_video_encoder_jpg_avcodec.so
I6.579 video_encoder_jpg_avcodec: width=896 height=500 sar=1/1 quality=50
I26.577 video_decoder_avc_avcodec: decoded 17616 samples with description index 1
I26.581 video_encoder_jpg_avcodec: encoded 10 samples
I26.583 writing 27 buckets for a total of 276 KB
I26.583 stat: url=https://live.unified-streaming.com/video/tears-of-steel/tears-of-steel-avc1-1000k.cmfv, reads=39, size=148 MB
I26.583 stat: url=file:///etc/ffmpeg-transcoders.usp, reads=1, size=176 bytes
Status: 200 FMP4_OK

The first tiled thumbnail (the first sample in tiled_thumbnails_from_avc.cmfv) should look approximately like:

../../_images/tiled_thumbnails_ffmpeg-example1.jpg