Google Compute Engine (GCE)

Choosing the Instance

Unlike for Amazon or Azure, Unified Streaming offers no Marketplace image for GCE.

There many GCE Instance Types, choosing depends on the following:

  • is the content SD or HD?
  • what is the expected dominant output (HLS, Smooth, etc)?
  • is DRM needed?
  • is the output Live or VOD?
  • if Live, what are the bitrates ingested?
  • if Live what is the DVR window size (and would you plan a RAM disk for that)?

In short, ingest is IO bound, and egress first IO bound, then network and lastly CPU.

The Cloud Storage Proxy section outlines how the Unified Origin is optimised for cloud storage access, which is enabled by default since Version 1.11.14 (2022-04-04) GA.

The Cloud Storage Reducing Latency setup uses (local) caching of files. There should be enough RAM for the webserver to start processes/threads to handle cache requests.

The Cloud Storage High Availability outlines how to setup high availability / load balancing which can be complemented by following the Origin shield cache use cases.

Lastly, the performance testing blog outlines how to create a vod setup including an installation script and a script that creates a virtual host with S3 access.

Creating a GCE image

  1. Sign up for Google Cloud Platform

Go to https://cloud.google.com and use your Google account to sign up for Google Cloud Platform and complete the guided instructions.

  1. Create a Project

Next, go to the console at https://cloud.google.com/console and create a new Project. Make sure to select your new Project if you are not automatically directed to the Project.

Projects are a way of grouping together related users, services, and billing. You may opt to create multiple Projects and the remaining instructions will need to be completed for each Project.

  1. Enable the Google Compute Engine service

In your Project, either just click Compute Engine to the left, or go to the APIs & auth section and APIs link and enable the Google Compute Engine service.

  1. Create a Service Account

To set up authorization, navigate to APIs & auth section and then the Credentials link and click the CREATE NEW CLIENT ID button. Select Service Account and click the Create Client ID button. This will automatically download a .json file containing the auth settings.

You will need the project id and the json account file in the following section.

Packer is easy to use and automates the creation of any type of machine images.

Packer uses a json template file consisting of builders, provisioners and other (optional) elements.

usp-packer.json contains the template to build an image and install USP in GCE.

The following snippet lists the builder for an instance install in the European zone.

{
  "variables": {
    "account_file": "YOUR_ACCOUNT_FILE",
    "project_id": "YOUR_PROJECT_ID"
  },
  "builders":[{
    "type": "googlecompute",
    "account_file": "{{user `account_file`}}",
    "project_id": "{{user `project_id`}}",
    "source_image": "YOUR_IMAGE",
    "zone": "europe-west1-c",
    "image_name": "YOUR_IMAGE_NAME",
    "image_description": "",
    "machine_type": "n1-standard-1",
    "ssh_username": "root"
  }]
}

Make sure to replace YOUR_ACCOUNT_FILE and YOUR_PROJECT_ID with your actual account file and project id, same for YOUR_IMAGE and YOURIMAGE_NAME.

Adding the license key

Add the following line as the first line in the provisioner described below:

#!/bin/bash

"sudo sh -c 'echo "YOUR_KEY" > /etc/usp-license.key'"

Make sure to replace YOUR_KEY with your actual License Key.

Install USP and demo content

USP and demo content can be installed by using a provisioner as follows:

{
  "provisioners":[{
    "type": "shell",
    "execute_command":  "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
    "inline": [
      "sudo sh -c 'echo \"YOUR_KEY\" > /etc/usp-license.key'",
      "sudo wget http://repository.unified-streaming.com/install-apache-usp.sh",
      "sudo chmod +x install-apache-usp.sh",
      "sudo ./install-apache-usp.sh -y",
      "sudo rm install-apache-usp.sh",
      "wget http://repository.unified-streaming.com/install-tears-of-steel.sh",
      "chmod +x install-tears-of-steel.sh",
      "sudo ./install-tears-of-steel.sh",
      "sudo rm install-tears-of-steel.sh"
    ],
    "inline_shebang": "/bin/sh -x"
  }]
}

Configuration

To use the hostname of the instance (so it's ready after booting) you would need to replace the default hostname - which can be done as follows.

#!/bin/bash

hn=$(curl http://metadata.google.internal/computeMetadata/v1/instance/hostname -H "Metadata-Flavor: Google")
/bin/sed -i -e "s/features.unified-streaming.com/$hn/g" /var/www/tears-of-steel/features.json

This then needs to be run when the instance is starting up, so for instance as conf file in /etc/init.

Create image

To start a build of the image as do the following:

#!/bin/bash

packer build usp-build.json

When the build is finished you can go to your project in the console and start the image.

Using Google Storage

First setup the GCE cli interface.

The cli allows for creating/deleting buckets or copy/move/delete content to a bucket or between buckets.

After uploading content you should set permissions so content can be viewed (the default is 'private').

Following the storage proxy Installation documentation, the just created bucket and uploaded content can be streamed by adding the UspEnableSubreq directive and defining <Proxy> sections for each remote storage server used.

<Location "/">
  UspHandleIsm on
  UspEnableSubreq on
  IsmProxyPass http://storage.googleapis.com/your-bucket/
</Location>

<Proxy "http://storage.googleapis.com/your-bucket/">
  ProxySet connectiontimeout=5 enablereuse=on keepalive=on retry=0 timeout=30 ttl=300
</Proxy>

The URL to the content then becomes the following, for instance for MPEG-DASH:

http://www.example.com/tears-of-steel/tears-of-steel.ism/.mpd

where www.example.com is the webserver running USP and has the previous vhost snippet (and the tears-of-steel content in 'your-bucket' used with both IsmProxPass and Proxy directives.

Attention

For guidelines on how to use Unified Packager with Google Storage see How to write directly to Google Cloud Storage.

If authentication is required, please use Google's Interoperability API allowing for storage access using S3 like methods. Then reconfigure your vhost with the required webserver directives defined in Using S3 with Authentication.