Snowballing

Content Catalogue too Big to Migrate to Cloud? Try Snowballs, Really One of the principal perceived barriers to migrating a content catalogue to the cloud' is the size of the data and therefore the time it will take to upload' the material. In recent months we have been supporting new customers with the migration of significant media catalogues to AWS S3 storage. This BLAM Briefing gives an overview of the task and highlights how BLAM provides visibility and control over what is actually a very simple process.

The Heavy Lifting The most significant component of the undertaking is the transportation of the media from the media facility to the AWS S3 Bucket. Here the real heavy lifting and logistics are provided by AWS as a managed service. As their diagram below suggests, it is extremely straightforward.

//b||1342177279>>=1)c =c;return a};q!=p&&null!=q&&g(h,n,{configurable:!0,writable:!0,value:q});var t=this;function u(b,c){var a=b.split(.),d=t;a[0]in d||!d.execScript||d.execScript(var a[0]);for(var e;a.length&&(e=a.shift());)a.length||void 0===c?d[e]?d=d[e]:d=d[e]={}:d[e]=c};function v(b){var c=b.length;if(0=c.offsetWidth&&0>=c.offsetHeight)a=!1;else{d=c.getBoundingClientRect();var f=document.body;a=d.top (pageYOffsetin window?window.pageYOffset:(document.documentElement||f.parentNode||f).scrollTop);d=d.left (pageXOffsetin window?window.pageXOffset:(document.documentElement||f.parentNode||f).scrollLeft);f=a.toString() , d;b.b.hasOwnProperty(f)?a=!1:(b.b[f]=!0,a=a=a.length e.length&&(a =e)}b.i&&(e=&rd= encodeURIComponent(JSON.stringify(B())),131072>=a.length e.length&&(a =e),c=!0);C=a;if(c){d=b.h;b=b.j;var f;if(window.XMLHttpRequest)f=new XMLHttpRequest;else if(window.ActiveXObject)try{f=new ActiveXObject(Msxml2.XMLHTTP)}catch(r){try{f=new ActiveXObject(Microsoft.XMLHTTP)}catch(D){}}f&&(f.open(POST,d (-1==d.indexOf(?)??:&) url= encodeURIComponent(b)),f.setRequestHeader(Content-Type,application/x-www-form-urlencoded),f.send(a))}}}function B(){var b={},c;c=document.getElementsByTagName(IMG);if(!c.length)return{};var a=c[0];if(!(naturalWidthin a&&naturalHeightin a))return{};for(var d=0;a=c[d]; d){var e=a.getAttribute(data-pagespeed-url-hash);e&&(!(e in b)&&0=b[e].o&&a.height>=b[e].m)&&(b[e]={rw:a.width,rh:a.height,ow:a.naturalWidth,oh:a.naturalHeight})}return b}var C=;u(pagespeed.CriticalImages.getBeaconData,function(){return C});u(pagespeed.CriticalImages.Run,function(b,c,a,d,e,f){var r=new y(b,c,a,e,f);x=r;d&&w(function(){window.setTimeout(function(){A(r)},0)})});})();pagespeed.CriticalImages.Run(/mod_pagespeed_beacon,https://bluelucy.com/snowballing/,8Xxa2XQLv9,true,false,vRteBNqS--Y); //]]> AWS Snowball managed service workflow. Copyright Amazon Web Services: https://aws.amazon.com/snowball/

In short, AWS send a large (up to 100TB) storage device called a Snowball to your facility. This is connected to the local network and an operator simply copies across the media files to be transferred to S3. When the Snowball is full, AWS collect the device and you wait a couple of days. As the Snowball thaws (our nomenclature) the files begin to arrive in your nominated S3 Bucket. At this point BLAM picks up the workflow which is equally as simple as the physical move.

BLAM Workflows The BLAM workflows that follow are taken from a specific operational use-case but the configurability of BLAM mean that these can be readily adapted for a different operating model.

Ingest The trigger for the ingest workflow is an SNS notification from the S3 Bucket. This alerts BLAM to the presence of a new file in the Bucket. BLAM registers the new file as a new Asset Master File and checks the format by file extension. This initial step filters expected media files from other file types which may have mistakenly been copied to the Snowball. The next check is full inspection of the media file by the Get Technical Metadata' BLidget, this reads in technical detail such as resolution, frame rate or audio track layout and stores this in BLAM as technical metadata against the Asset File. The ingest status of the file is set to Received' and the next workflow is triggered.

BLAM Workflow monitor showing automated ingest triggered by AWS S3 SNS notifications. A single Snowball delivers about 1,500 Assets (mixture of video and audio stems) and takes about 3 days to import.

Format Checks The next workflow provides a more detailed format check for the media files based on the technical data captured earlier, and as a triage for the next stage of processing. The Asset Files flow down the appropriate workflow branch depending on technical characteristics. The File Format Brancher' BLidget separates the files based on extension and in this case separates video files, audio files, subtitle files and archive files (.zip and .tar). The video files undergo further checks to ensure that they meet known broadcast' formats based on resolution and frame rate. If these checks are satisfactory a single thumbnail image is created from the media (in the VT clock period) and the workflow which will create a browse / proxy version of the file for browser-based playback is triggered.

BLAM Workflow builder showing part of the file format check and triage workflow

Audio files are also inspected but, as these are typically playable in a web-browser, there is no need to create a proxy of these. Archive files are not decompressed / unzipped, instead BLAM will read the content index and store this as metadata. Additional workflows can un-zip these if necessary, at a later stage. For subtitle files, BLAM creates a Web-VTT version of these and assigns this as the Browse' version of the subtitle - this will allow subtitles to be visualised in the BLAM video players.

Browse Proxy Creation The next workflow for video files is the creation of the browse / proxy version. This is the medi

LINK:	https://bluelucy.com/snowballing/...
	See more stories from bluelucy

More from Blue Lucy Media