Cloud Storage High Availability

Introduction 

Building upon Unified Origin's Cloud Storage Proxy it is now possible to implement load-balancing across multiple HTTP(S) based storage endpoints with the additional of apache's mod_proxy_balancer.

Apache's mod_proxy_hcheck can also be added to provide periodic health checks (using GET/HEAD requests) allowing for automatic fail-over should a remote location become unavailable.

Architecture 

The following diagram described how content maybe hosted on multiple remote locations through the use of a replication service.

../../_images/object-storage-lb-flow.svg

To configure storage proxy for load-balancer please ensure that all modules are enabled as part of the storage proxy Requirements. Please also ensure the additional mod_proxy_balancer and mod_proxy_hcheck modules are enabled.

<Location "/s3-europe/">
  ProxyPass http://0.0.0.0:8081/
  ProxyPassReverse http://0.0.0.0:8081/
</Location>

The main unified-origin.conf contains an appropriately named /s3-europe/ location which will be responsible for routing the requests to the dedicated remote storage configuration.

remote_storage.conf

<Location "/">
  UspHandleIsm on
  UspEnableSubreq on
  IsmProxyPass http://localhost:8081/load-balancer/
</Location>

<Location "/load-balancer/">
  ProxyPass "balancer://load-balancer/"
  ProxyPassReverse "balancer://load-balancer/"
</Location>

To route the incoming requests to the load-balancer its necessary to configure an additional /load-balancer/ location which will forward the request to the <Proxy "balancer://load-balancer/"> configuration.

This is required as IsmProxyPass only supports relative (path based) or absolute (http/s) urls.

<Proxy "balancer://load-balancer/">
  ProxySet lbmethod=bybusyness failonstatus=403 failontimeout=On forcerecovery=Off nofailover=Off

  # Primary 2 Node Cluster
  BalancerMember "http://localhost:8081/usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/" connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=60  hcmethod=GET hcuri=/tears-of-steel/tears-of-steel.ism hcinterval=30 hcpasses=1 hcfails=1
  BalancerMember "http://localhost:8081/usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/" connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=60 hcmethod=GET  hcuri=/tears-of-steel/tears-of-steel.ism hcinterval=30 hcpasses=1 hcfails=1
</Proxy>

Apache's mod_proxy documentation explains in details multiple BalanceMember hosts can be configured within a ProxyPass Directive. This includes parameters to define primary, spare and hot-spare hosts.

The above example demonstrates a simple configuration describing two primary hosts working active/active. Each host is an internal location and proxy directive set within this apacge configuration file.

Additional parameters have been configured that are responsible for:

lbmethod=bybusyness - evenly distributing requests using the mod_lbmethod_bybusyness algorithm.
failonstatus=403 - to fail the BalanceMember host if a 403 response code is received from any request made.
failontimeout=On - to automatically fail the BalanceMember host if the timeout threshold is reached.
hcmethod=GET hcuri=/tears-of-steel/tears-of-steel.ism - to execute a http GET request against the path defined.

Note

Further information on the parameters available for mod_proxy and mod_proxy_hcheck can be found on apache's website.

<Location "/usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/">
  ProxyPass "http://usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/"
  ProxyPassReverse "http://usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/"
</Location>

<Proxy "http://usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/">
  ProxySet connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=0 timeout=5 ttl=300
</Proxy>

<Location "/usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/">
  ProxyPass "http://usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/"
  ProxyPassReverse "http://usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/"
</Location>

<Proxy "http://usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/">
  ProxySet connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=0 timeout=5 ttl=300
</Proxy>

Requests made to each BalancerMember are routed to a Location and Proxy directive. Each has been configured to forward onto the necessary remote storage location.

s3_auth.conf

<Proxy "http://usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/">
  S3AccessKey *REDACTED*
  S3SecretKey *REDACTED*
  S3Region eu-west-1
  S3UseHeaders on
  ProxySet connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=0 timeout=5 ttl=300
</Proxy>

<Proxy "http://usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/">
  S3AccessKey *REDACTED*
  S3SecretKey *REDACTED*
  S3Region eu-central-1
  S3UseHeaders on
  ProxySet connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=0 timeout=5 ttl=300
</Proxy>

The above solution also benefits from the support of Unified-Streaming's Using S3 with Authentication. Therefore allowing each location to use a unique set of access credentials (if desired).

Note

For ease of use the above s3 authorization credentials are defined in a separate configuration file (s3_auth.conf).

Instead the authorization parameters could be set in the <Proxy> location for each remote location within the remote_storage.conf.

Complete Apache Configuration 

unified-origin.conf

<VirtualHost *:80>
  ServerName unified-origin

  AddHandler smooth-streaming.extensions .ism .isml .mp4
  SSLProxyEngine on

  # Location for streaming local server manifests
  <Location "/">
    UspHandleIsm on
    UspEnableSubreq on
  </Location>

  # Location for streaming remote manifests via port 8081 vhost
  <Location "/s3-europe/">
    ProxyPass http://0.0.0.0:8081/
    ProxyPassReverse http://0.0.0.0:8081/
  </Location>

  LogFormat '%h %l %u %t "%r" %>s %b %D "%{Referer}i" "%{User-agent}i" "%{BALANCER_WORKER_NAME}e" ' log_format

  CustomLog /dev/stdout log_format
  ErrorLog /dev/stderr
  LogLevel info

  AddHandler smooth-streaming.extensions .ism .isml .mp4

  DocumentRoot /var/www/unified-origin

  Header set Access-Control-Allow-Headers "origin, range"
  Header set Access-Control-Allow-Methods "GET, HEAD, OPTIONS"
  Header set Access-Control-Allow-Origin "*"
  Header set Access-Control-Expose-Headers "Server,range"

</VirtualHost>

remote_storage.conf

<VirtualHost *:8081>
  ServerName unified-origin-backend

  LogFormat '%h %l %u %t "%r" %>s %b %D "%{Referer}i" "%{User-agent}i" "%{BALANCER_WORKER_NAME}e" ' log_format

  CustomLog /dev/stdout log_format
  ErrorLog /dev/stderr

  LogLevel info

  <Location "/">
    UspHandleIsm on
    UspEnableSubreq on
    IsmProxyPass http://localhost:8081/load-balancer/
  </Location>

  <Location "/load-balancer/">
    ProxyPass "balancer://load-balancer/"
    ProxyPassReverse "balancer://load-balancer/"
  </Location>

  <Proxy "balancer://load-balancer/">
    ProxySet lbmethod=bybusyness failonstatus=403 failontimeout=On forcerecovery=Off nofailover=Off

    # Primary 2 Node Cluster
    BalancerMember "http://localhost:8081/usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/" connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=60  hcmethod=GET hcuri=/tears-of-steel/tears-of-steel.ism hcinterval=30 hcpasses=1 hcfails=1
    BalancerMember "http://localhost:8081/usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/" connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=60 hcmethod=GET  hcuri=/tears-of-steel/tears-of-steel.ism hcinterval=30 hcpasses=1 hcfails=1

  </Proxy>

  <Location "/usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/">
   ProxyPass "http://usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/"
   ProxyPassReverse "http://usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/"
  </Location>

  <Proxy "http://usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/">
    ProxySet connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=0 timeout=5 ttl=300
  </Proxy>

  <Location "/usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/">
   ProxyPass "http://usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/"
   ProxyPassReverse "http://usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/"
  </Location>

  <Proxy "http://usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/">
    ProxySet connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=0 timeout=5 ttl=300
  </Proxy>

</Virtualhost>

s3_auth.conf

<Proxy "http://usp-s3-eu-west-1.s3.eu-west-1.amazonaws.com/">
 S3AccessKey *REDACTED*
  S3SecretKey *REDACTED*
  S3Region eu-west-1
  S3UseHeaders on
  ProxySet connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=0 timeout=5 ttl=300
</Proxy>

<Proxy "http://usp-s3-eu-central-1.s3.eu-central-1.amazonaws.com/">
  S3AccessKey *REDACTED*
  S3SecretKey *REDACTED*
  S3Region eu-central-1
  S3UseHeaders on
  ProxySet connectiontimeout=5 timeout=5 ttl=300 keepalive=on retry=0 timeout=5 ttl=300
</Proxy>

Example Demo 

Unified Streaming's software is offered as cloud-native container images which can be customized and built to the meet desired requirements.

The following origin_storage_loadbalancer project has been pre-configured with the load balancing solution so it can be easily tested (without the need to setup the apache configuration files).

Download the Github repository and follow the instructions to build the project. The origin container can then be ran with a set of environment variables to load-balance requests across two remote storage locations.

Cloud Storage High Availability

Introduction

Architecture

Configuration

Complete Apache Configuration

Example Demo

Introduction 

Architecture 

Configuration 

Complete Apache Configuration 

Example Demo 