Using S3 with Authentication

When using Amazon S3 for remote storage, it might be required to secure access to the S3 buckets instead of using 'public' access. This means that the requests to the content in the bucket must be authenticated. See Using S3 storage regarding configuring Amazon S3 storage.

There are two ways to do this:

use the Authorization header

send a signature as a URL-encoded query-string parameter.

The mod_unified_s3_auth module supports both.

More information can be found in Amazon's Authenticating Requests documentation.

Using Webserver Directives for S3 authentication 

It is possible to use webserver directives and let USP sign the request automatically with the following webserver directives.

This covers one specific use case: both content (mp4/ismv) and server manifest (ism) are placed in the S3 bucket and the manifest references the content locally (no paths or URLs, just the filename).

The directives for Apache are the following:

Option	Description
S3SecretKey	The AWS secret key.
S3AccessKey	The AWS access key.
S3Region	Region of bucket where data is stored, added in 1.7.29. Required for AWS Signature v4.
S3SecurityToken	The AWS Security Token, if required; see Temporary security credentials in IAM. (Added in 1.11.13)
S3RoleARN	The Amazon Resource Name (ARN) of the role to assume. (Added in 1.15.8)
S3RoleSessionName	An identifier for the assumed role session. (Added in 1.15.8)
S3RoleDuration	The duration, in seconds, of the role session (optional, >= 900 seconds). (Added in 1.15.8)
S3UseHeaders	Set to 'on' to use HTTP headers for AWS authentication, instead of query parameters.

The keys can be created in the AWS IAM portal and managed there as well (active/inactive, delete).

AWS Signature v4 

As of January 30, 2014, newer AWS regions such as eu-central-1 require the use of the Signature v4 for authentication. To enable the use of v4 signatures the name of the S3 region must be set when configuring access to the bucket. If the region is set, v4 signatures will be used whether or not the region requires it. Not setting the S3 region will result in USP using the older v2 signature method that may not be supported by all regions.

AWS Security Tokens 

AWS's Identity and Access Management (IAM) allows creating temporary, limited-privilege Security Tokens, which can be used to access AWS services such as S3. (See Temporary security credentials in IAM for more information.)

In addition to the existing S3-related directives, we have added a new S3SecurityToken directive in 1.11.13, where you can specify such a Security Token, if it is required to access the S3 resource.

In 1.15.8, we have added directives S3RoleARN, S3RoleSessionName and S3RoleDuration, which enable Origin to automatically retrieve temporary Security Tokens. This can be more convenient than the scripting shown below. See the next section, AWS Role Directives.

Note

Please be aware S3 Security Tokens have a limited life or "Expiration" value, therefore a new token will need to be obtained before the active token expires.

Tokens can be obtained in several different ways. A session token can be generated by the aws-cli sts (Security Token Service) API by assuming a role which has been configured with least privilege policy.

The following example demonstrate how this can be achieved.

$ aws sts assume-role \
--role-arn arn:aws:iam::12345678910:role/my-test-role \
--role-session-name session1 \
--duration-seconds 3600

{
    "Credentials": {
        "AccessKeyId": "12345678901",
        "SecretAccessKey": "v/12345678901",
        "SessionToken": "TEST92test48TEST+y6RpoTEST92test48TEST/8oWVAiBqTEsT5Ky7ty2tEStxC1T==",
        "Expiration": "2022-02-25T13:00:00+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "12345678901:my-test-role",
        "Arn": "arn:aws:sts::12345678901:assumed-role/my-role/my-test-role"
    }
}

An alternative method of obtaining credentials is by using the AWS Instance Metadata Service Version 2 (IMDSv2).This is a session-oriented method of obtaining credentials via HTTP request from an AWS EC2 instance.

The following example demonstrate how this can be achieved.

$ TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") && \
  curl -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/my-test-role

{
        "Code": "Success",
        "LastUpdated": "2022-02-25T12:00:00Z",
        "Type": "AWS-HMAC",
        "AccessKeyId": "12345678901",
        "SecretAccessKey": "v/12345678901",
        "Token": "TEST92test48TEST+y6RpoTEST92test48TEST/8oWVAiBqTEsT5Ky7ty2tEStxC1T==",
        "Expiration": "2022-02-25T13:00:00Z"
}

Note

Due to the way in which Unified Origin is configured via apache2/httpd virtual host configurations files any new credentials need to be added to these files and apache2/httpd restarted for these to take affect.

One way to ensure Unified Origin is always using valid credentials could be through the form of a script triggered using through scheduled automation such as a cronjob.

Below is an example shell script demonstrating how the the imdsv2 endpoint used to obtain credentials, update and restart apache.

#!/bin/bash

# Define current config location
APACHE_CUR_CONFIG=/etc/apache2/sites-enabled/unified-streaming.conf

# Define current credential variables
APACHE_S3_ACCESS_KEY=$(grep S3AccessKey "${APACHE_CUR_CONFIG}" | awk '{print $2}' )
APACHE_S3_SECRET_KEY=$(grep S3SecretKey "${APACHE_CUR_CONFIG}" | awk '{print $2}' )
APACHE_S3_SECURITY_TOKEN=$(grep S3SecurityToken "${APACHE_CUR_CONFIG}" | awk '{print $2}' )

# Define imdsv2 endpoint
REQUEST=$(curl -s -X GET -o /dev/null -w "%{http_code}" http://169.254.169.254/latest/meta-data/iam/security-credentials)

# Define if statement to check imdsv2 endpoint is available
# endpoint will only be available if a instance-profile has been assignd to # the ec2 via either the console or launch-template.
if [[ $REQUEST == "200" ]]; then

    # Define imdsv2 variable to obtain credentials
    TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
    ROLE=$(curl -s -X GET -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials)
    CREDS=$(curl -s -X GET -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/"$ROLE")

    # Define temp variables to populate with newly obtains credentials
    S3_ACCESS_KEY=$( jq -r  '.AccessKeyId' <<< "${CREDS}" )
    S3_SECRET_KEY=$( jq -r  '.SecretAccessKey' <<< "${CREDS}" )
    S3_SECURITY_TOKEN=$( jq -r '.Token' <<< "${CREDS}" )

    # Define if statement to compare active with new credentials
    # if these credentials differ (due to aws automatically rotating these)
    # then backup config, update & gracefully restart apache to avoid impacting active requests/connections.
    if [[ $APACHE_S3_ACCESS_KEY != $S3_ACCESS_KEY ]] || [[ $APACHE_S3_SECRET_KEY != $S3_SECRET_KEY ]] || [[ $APACHE_S3_SECURITY_TOKEN != $S3_SECURITY_TOKEN ]]; then

        ## Define message with date/time stamp which can be used for logging
        #echo "$(date) - New S3 Credentials available.. updating apache #config and restarting"

        cp ${APACHE_CUR_CONFIG} /tmp/unified-streaming.conf.old
        sed -i "s@${APACHE_S3_ACCESS_KEY}@${S3_ACCESS_KEY}@g; s@${APACHE_S3_SECRET_KEY}@${S3_SECRET_KEY}@g; s@${APACHE_S3_SECURITY_TOKEN}@${S3_SECURITY_TOKEN}@g" ${APACHE_CUR_CONFIG}
        /usr/sbin/apachectl -k graceful

    else
        echo "$(date) - Current credentials OK... exiting"
        exit 0
    fi

else
    exit 0
fi

Note

For more information on how to use S3 Security Tokens with unified streaming cli-tools, please see Authenticate requests to AWS S3.

Warning

Please be reminded the majority of S3 buckets require requests to be authenticated using AWS Signature v4.

AWS Role Directives 

Instead of running scripts and external tools to obtain AWS temporary Security Tokens, and then restarting Apache, from version 1.15.8 you can use the directives S3RoleARN, S3RoleSessionName and S3RoleDuration, in combination with S3SecretKey, S3AccessKey and S3Region instead, typically in a <Proxy> section.

Using these directives, Origin will automatically obtain temporary Security Credentials from the AWS Security Token Service (STS) API, over secure HTTP. Depending on the S3Region setting, the appropriate regional STS API endpoint will be used. For example, if the region is set to us-east-1, the used STS API endpoint will be https://sts.us-east-1.amazonaws.com/.

The S3RoleARN directive specifies the Amazon Resource Name (ARN) of the role to assume, similar to the --role-arn option in the AWS CLI assume-role command.

The S3RoleSessionName directive specifies the name for the assumed role session, similar to the --role-session-name option in the AWS CLI assume-role command.

The optional S3RoleDuration directive specifies the duration of the role session, in seconds, similar to the --duration-seconds option in the AWS CLI assume-role command. The minimum (and default) value is 900 seconds, or 15 minutes. The maximum value depends on how the S3 administrator configured the role.

Together with these directives, specify credentials for the user that is assuming the role, using S3SecretKey, S3AccessKey, and S3Region.

Example of a <Proxy> section with role directives:

<Proxy "https://yourbucket.s3-us-east-1.amazonaws.com/">
  S3SecretKey YOUR_SECRET_KEY
  S3AccessKey YOUR_ACCESS_KEY
  S3Region us-east-1
  S3RoleARN arn:aws:iam::123456789012:role/my-iamrole-user
  S3RoleSessionName YOUR_SESSION_NAME
  S3RoleDuration 1800
  ProxySet connectiontimeout=5 enablereuse=on keepalive=on retry=0 timeout=30 ttl=300
</Proxy>

<Location /some/location/>
  UspHandleIsm on
  UspEnableSubreq on
  IsmProxyPass https://yourbucket.s3-us-east-1.amazonaws.com/
</Location>

Note

Using role directives, Origin keeps track of the expiration of any temporary Security Tokens, and will automatically retrieve fresh Security Tokens when necessary. There is no need to restart Apache, except when changing the directives themselves.

Apache Configuration 

To secure access to Amazon S3 buckets, <Proxy> sections are the appropriate locations to specify the S3 authentication directives.

The directives can appear in two places:

In a <Proxy> section. This is the recommended location, and the only supported location for AWS Role Directives.
In a <Location> section, which is used in combination with the IsmProxyPass directive in a <Directory> or <Location> section. This is the traditional way of specifying the S3 parameters, and it is still supported for backwards compatibility, but not for AWS Role Directives.

Each Amazon S3 bucket must have its own <Proxy> section. These are then used automatically for each request sent to the Amazon S3 bucket.

An additional benefit for using <Proxy> sections is that performance is improved by keeping the connections to Amazon S3 bucket alive.

For example, if the bucket is called yourbucket, and it is hosted in Amazon's eu-central-1 region, the section becomes:

<Proxy "http://yourbucket.s3-eu-central-1.amazonaws.com/">
  S3SecretKey  YOUR_SECRET_KEY
  S3AccessKey  YOUR_ACCESS_KEY
  S3Region     eu-central-1
  S3UseHeaders on
  ProxySet connectiontimeout=5 enablereuse=on keepalive=on retry=0 timeout=30 ttl=300
</Proxy>

This can be used both when your manifests and/or media files are directly referring to Amazon S3 buckets, and when IsmProxyPass directives redirect requests to Amazon S3 buckets, see Cloud Storage for an outline of the different use cases.

In this case, there is no need to add any S3 parameters to the sections containing IsmProxyPass.

Schematically, it works like the following:

player --> cdn --> shield-cache --> origin (mod-smooth-streaming) --> signing (mod_unified_s3_auth) --> s3

The S3UseHeaders directive determines how the signature information is provided, either by adding a number of additional request headers, or otherwise by adding a number of additional query arguments.

Following is a complete example <VirtualHost>:

<VirtualHost *:80>
  ServerName www.example.com
  DocumentRoot /var/www/yoursite

  # Enable just in time packaging for VOD and using subrequests for I/O backend requests.
  <Location "/">
    UspHandleIsm on
    UspEnableSubreq on
    IsmProxyPass https://yourbucket.s3-eu-central-1.amazonaws.com/
  </Location>

  # If proxying to SSL hosts is desired, you must turn on SSLProxyEngine.
  SSLProxyEngine on

  <Proxy "https://yourbucket.s3-eu-central-1.amazonaws.com/">
    S3SecretKey YOUR_SECRET_KEY
    S3AccessKey YOUR_ACCESS_KEY
    S3Region YOUR_REGION
    S3UseHeaders on
    ProxySet connectiontimeout=5 enablereuse=on keepalive=on retry=0 timeout=30 ttl=300
  </Proxy>
</VirtualHost>

Legacy IsmProxyPass Example 

In previous versions of Unified Origin, when subrequests were not available, S3 authentication parameters were typically placed in a <Location> section, with a corresponding <Directory> section containing an IsmProxyPass directive. E.g.:

<Location /s3/>
  S3SecretKey YOUR_SECRET_KEY
  S3AccessKey YOUR_ACCESS_KEY
  S3Region YOUR_REGION
</Location>

<Directory /var/www/yoursite/s3/>
  IsmProxyPass https://yourbucket.s3-eu-central-1.amazonaws.com/
</Directory>

Assuming the DocumentRoot of the site is /var/www/mysite, the URI path /s3/example.ism would be mapped by IsmProxyPass to https://yourbucket.s3-eu-central-1.amazonaws.com/example.ism, and the S3SecretKey, S3AccessKey and S3Region parameters from the <Location> section would be used for authentication.

Using S3 with Authentication

Using Webserver Directives for S3 authentication

AWS Signature v4

AWS Security Tokens

AWS Role Directives

Apache Configuration

Legacy IsmProxyPass Example

Using Webserver Directives for S3 authentication 

AWS Signature v4 

AWS Security Tokens 

AWS Role Directives 

Apache Configuration 

Legacy IsmProxyPass Example 