Data storage for media and entertainment
The media and entertainment industries have not always been heavy users of storage technology. They were used to shooting movies on celluloid or videotape. Old films and tapes were simply stored on shelves or in safes. But then came digitization. Suddenly, data storage was a big deal. The solution in the industry was to use disk, but that approach is now proving too expensive.
Storage industry analyst Tom Coughlin of Coughlin Associates explained the challenge. Frame rates for video content increase from the historic 24 frames per second (fps) to 48 and 60 fps, and can eventually reach 300 fps. And as resolutions increase, file sizes skyrocket. Per movie hour, Standard Definition (SD) is 112GB, HD is 537GB, Ultra HD is 6,880GB, 6k is coming to market, 8K is on the near horizon, and some are considering the 16k. When 8k appears, each hour of movie will be equivalent to 86,000 GB of capacity.
In other words, an 8K movie would use over 100 times the capacity of HD, Coughlin said. And then there’s free point-of-view imaging which uses multiple cameras simultaneously to change the point of view of the film.
“As video resolution and frame rates increase, camera image complexity increases, and stereoscopic projects proliferate, the demands on storage capacity and bandwidth become staggering,” said Coughlin. . “How long will it be before we see video at the scale of an exabyte?”
The 16k movie would eventually require data rates of 115Gb/s and generate around 414TB per hour of content. But if four cameras are used to create data for a free POV presentation, the raw data could add up to 1.66PB per hour of content.
This frightening amount of data is pushing film companies to the cloud. According to a survey by Coughlin Associates, 30.2% of media and entertainment companies said they used cloud storage for post-production work in 2015, up from 25.6% in 2014 and 15.15% in 2012. Additionally, 32.9% said they had 1TB or more of cloud storage in 2015 compared to 23% in 2013. And more than half capture six hours of content for every hour of movie. For some it goes up to 50 hours.
The amount of content makes it essential that theaters clearly define what content should be kept close at hand for quick editing. How it works is that brand new content for movies and shows is widely viewed until the job is done. Once completed and distributed, this content suddenly becomes inactive and is rarely viewed afterward.
Coughlin said 95% of all content therefore resides in massive archives. However, in dollar terms, the archives receive only 45% of available storage funds. Although this material is archived, it may be viewed on occasions such as anniversaries, documentaries, breaking news and for marketing purposes.
“This makes the concept of active archives appealing to the media and entertainment industry,” Coughlin said.
To achieve this, various media are used. Optical disc, tape and hard disk drives (HDD) are used together, depending on the response time and contact frequency required. The mechanics are simple: as response time requirements approach instantaneous and access volume increases, costs skyrocket as the fastest storage is required; as the response time decreases and the contact rate decreases, slower and cheaper storage comes into play, depending on the contact rate and the predefined policy.
Hard disk will clearly play a role in such an arrangement, supported by cheaper optical disk for faster access/lower capacity and tape for higher capacity/slower access. Among media and entertainment today, Coughlin said tape is the most widely used medium for archiving at 40% (expected to grow by 59% this year), followed by hard drives at 28%, local storage at 16% and optical at 6%. .
In terms of capacity market share, HDD holds 48.9% while tape has 40.4%, optical 10.15% and flash 0.6% for the entire industry. Much of this storage migrates from the internal data center to the cloud. “Cloud storage for media and entertainment will surpass $2.1 billion by 2020,” Coughlin said. “It will be primarily for archival purposes and will consist primarily of tapes and discs.”
Film production workflow
Aaron Brenner, executive director of sports television production company Showrunner, was a big fan of the storage area network (SAN) and was happy to gobble up drives for content storage. As the SAN filled up and content needed to be archived, hard drives were pulled out of their slots, labeled, recorded, and stored on shelves.
“We need to save sports content forever because it can be monetized by using clips over and over again,” Brenner said.
In 2008, only 100 GB of standard definition (720 x 480) footage included the entire library of LA Kings footage Brenner was working on. In 2015, each game produced 65 GB and the library was expanded to half a PB. For example, a 13-episode Stanley Cup video series used 100TB of raw media.
“Storing hard drives on shelves gave us no redundancy, was too time consuming and became too expensive,” Brenner said. “But at the same time, the industry had gone tapeless for years getting rid of video tape, so we were opposed to tape storage.”
The workflow at Showrunner now includes storing camera work directly on flash. These flash drives are downloaded to disk and then pushed into the SAN until production and post-production work is complete. From there, the hardware is placed on a Dternity hybrid storage system consisting of disk and tape. This means the business can access archived clips much faster without having to sift through stacks of hard drives. Tapes are also stored offsite for DR purposes.
“When we’re ready to archive, the content is dragged and dropped onto the Dternity, we verify it’s there, and then delete it from the SAN,” Brenner said. “Tape allowed us to focus on our creativity and not worry about archiving or having enough discs to store our data.”
A similar workflow is used by color grading company EFilm. The data captured by the cameras is transferred over the SAN and, once the color job is complete, it is then transferred to LTO tape for archiving. The company makes three copies of each LTO cartridge: one stays on site, one is sent to the office, and one is sent to a secure location for DR purposes. This is done due to the extremely high cost of a shooting day.
“We have about 3TB of disk onsite for film editing and color correction,” said Josh Haynie, senior vice president of operations at EFilm. “But we could use a few thousand LTO cartridges for each major film project.”