Troubleshooting LIVE Streaming

Determining the issue 

If you experience a problem with your stream, there are a few steps you can take to determine what the exact issue is that you are experiencing. Sometimes, going through these steps will be enough to solve your problem. Or it might become clear that your issue if one of the more common problems that are described further down below, along with their solutions. And otherwise, these steps will give you the necessary info to open a well-defined ticket in our support portal, so that we can help you as quickly as possible:

Check license key
Check publishing point's state (Live only)
Check server's log files
Check encoder settings
Check release notes
Check your setup

All six steps are described in more detail below.

License key

The first thing is to check is whether the license is properly installed: Installation Check. Also check that there are no Error Messages.

Publishing point's state

This step applies to streaming Live only. To check whether a publishing point is properly set up, you can check its state. A working publishing point will return the following from a /state REST call, with the actual state differing based on whether a stream has been ingested or not ((see Retrieving additional publishing point information):

<?xml version="1.0" encoding="utf-8"?>
<!-- Created with Unified Streaming Platform(version=1.x.x) -->
<smil
  xmlns="http://www.w3.org/2001/SMIL20/Language">
  <head>
    <meta
      name="state"
      content="idle">
    </meta>
  </head>
</smil>

If a publishing point doesn't return a state, it might be either that it has not been set up at all, or that there are other problems with your setup (such as access rights). If the state returned by the publishing point is not as expected, the most obvious cause would be a problem with your ingest, either because of problems with the content itself (i.e. encoder settings) or because the webserver isn't correctly set up to handle the ingest.

Log files

The webserver's log files are the most valuable source for determining the exact nature of the problem that you are experiencing with your stream. These logs allow you to determine whether content is properly ingested (in the case of Live) and whether requests for content are handled correctly (both for Live and VOD). For Apache on Ubuntu, three log files are of importance:

The general error log (/var/log/apache2/error.log)

The virtual host access log (configured in the virtual host config)

The virtual host error log (configured in the virtual host config)

In the virtual host's access log, you can check how the ingest and requests for content are being handled. For every problem with either the ingest or a request, an error is written to the virtual host's error log.

Encoder settings

Adaptive bitrate video streaming is a delicate process that requires well formed content that has to adhere to many requirements. If those requirements are not met, a wide variety of issues can occur. Check Recommendations for Live and Recommendations for VOD to determine whether you have configured your encoder correctly. Do note that even with the right settings, there might still be problems with the output of your encoder. Determining whether this is or is not the case requires a detailed inspection of the content itself, with specialized tools (e.g., ffprobe). For streaming Live, make sure that your encoder is supported by checking the overview in our Factsheet.

Release notes

If you are currently not using the latest version of our software, please check the Release notes to determine whether we introduced a fix for your problem in a newer version.

Setup

A detailed check of your installation is advised. Please make sure that your webserver is installed and configured as detailed in the installation documentation that we provide for Apache:

How to Configure (Unified Origin)

FMP4_403 No virtual path specified 

This error is returned when you make a request for the server manifest (.ism(l)), instead of a playout format. Direct requests for the server manifest are prohibited for security reasons. You should add a virtual path to the URL of the server manifest, specifying the playout format. For more information about the URLs of the different playout formats, please consult Player URLs. .. end

FMP4_403 origin: no policy for streaming 

This error typically indicates that your license key is not being loaded correctly. If you are using Apache, you can check if this is the case in the general error log of the webserver, by looking for the following error:

failed to initialize! (Invalid base64 character)

Common causes for this error are the addition of a newline or an erroneous character that has crept in when copying and pasting the license key.

If the error log does not indicate any problems with the license key, your license might not cover the use case that you are trying to achieve.

Further information about installing the license key can be found here: Installation Check. Additional info about any related errors can be found here: Error Messages. .. end

Apache: 'timeout specified has expired'

This error looks like this:

(70007)The timeout specified has expired: Error reading chunk

It means that Unified Origin did not receive any data for a specified amount of time. This amount of time is equal to the 'timeout' value that can be set using Apache's mod_reqtimeout module. Note that its default setting might be considered quite aggressive (10 seconds).

Note also that setting the RequestReadTimeout in a virtual host to override the global settings, does not work. You have to set the global RequestReadTimeout to a larger timeout value, or disable it completely with:

RequestReadTimeout header=0 body=0

Make sure to include the following in your (virtual host) configuration:

LimitRequestBody 0

'409 stream closed'

The 'Stream Closed' error occurs when the timestamps sent by the encoder are not increasing.

The webserver log will show entries like the following:

[Mon Mar 09 19:19:32 2015] [error] [client 192.168.12.100] pubpoint_push_input_stream(Streams(Video1))
  returned: 409 Stream is closed, cannot restart (track_name=video stream.time=41863154666 fragment.time=41598557000)

As a basic requirement, the timestamps of a livestream must increase.

As an additional requirement, which will allow the encoder to start, stop and restart (when the restart-on-encoder-reconnect option is enabled for the publishing point, see Options for LIVE ingest), the encoder needs to be configured to use UTC timing. Please refer to the encoder manual on how to configure this.

Discontinuities 

The StreamIndex element in the MSS client manifest contains the timestamps, next to the media type (for instance video) or the QualityLevel element indicating the bitrate.

<StreamIndex>
  <QualityLevel />
  <c t="0" d="20000000" />
  <c d="20000000" />
  <c d="20000000" />
</StreamIndex>

Sometimes you might notice a @t attribute in the client manifest, which indicates a gap in the stream. In most cases, this is caused by an interruption of the ingest of data on the server, but it can be caused by improper encoder implementations as well.

<c t="0" d="20000000" />
<c d="20000000" />
<c t="60000000" d="20000000" />

In the above example, the length of a full segment is missing after the first two. More information can be found on the IIS blog.

Discontinuities can also be found by looking at the archive of the publishing point, which you can access using the /archive REST call, as described in the Publishing Point API. Ideally, every track should have its start and end time displayed on one line. For every discontinuity, there will be a new line.

Background

On the origin we see a discontinuity when the timestamp on a next fragment doesn't equal the timestamp of the previous fragment + its duration. Even if it's only off by 1/10th of a microsecond.

Although the difference may be very small, this triggers a discontinuity and is noticed in playlist generation and eventually by a player implementation that triggers a decoder reset (introducing ticks/clicks/pops) whenever it sees a timestamp discontinuity. This is especially true for HLS streams that use MPEG-TS.

Therefore, it's important that the timestamp and duration that are signaled in the 'tfxd' box (the one that the encoder is inserting for every fragment) is accurate.

In a more mathematical way:

tfxd[fragment N].timestamp = tfxd[fragment N - 1].timestamp + tfxd[fragment N - 1].duration.

Also worth noting is that the 'tfxd[fragment N].duration' should be equal to the sum of the sample durations specified in the 'trun' box.