Data storage

Next big thing or bust?

The storage of holographic data has been discussed for decades. Once envisioned as the next generation of optical storage, it promised higher densities and access speeds than current Blu-ray discs.

Over the years, many research teams have attempted to build holographic systems capable of meeting the increasing demands for data storage. However, these teams achieved little concrete results beyond the occasional prototype. Their efforts, however, were not in vain. Microsoft has breathed new life into holographic storage with Project HSD, a collaboration between Microsoft Research Cambridge and Microsoft Azure to bring holographic technology to cloud-scale storage.

Why holograms for storage?

Holographic storage – sometimes called 3D storage – is a volumetric storage system that uses lasers to read and write data, similar to other optical storage. However, media such as CDs, DVDs, and hard drives can only store data on the surface of the media, limiting its capacity to two-dimensional storage. Holographic storage uses the entire volume, allowing more data to be stored in less space and increasing data write and read speeds.

Polaroid researcher Pieter J. van Heerden first proposed holographic data storage in the early 1960s, shortly after the invention of the laser. By the early 2000s, research teams from industry and academia had made significant progress in demonstrating the technology’s potential. Two major efforts came from Polaroid spin-off Aprilis and Bell Labs spin-off InPhase Technologies. Both companies attempted to commercialize holographic storage. Ultimately, however, neither achieved commercial success. Dow Corning acquired Aprilis and InPhase eventually filed for bankruptcy.

Many other efforts have also been made, but none have been able to reverse the course of holographic data storage. Most of these attempts focused on using circular media similar to CDs or DVDs to support write-once and read-many (WORM) operations, but holographic storage competed with more established, which themselves had also progressed.

Hard drives, for example, have become faster and denser, and SSDs have become cheaper and more durable. At the same time, there was an increased reliance on cloud computing, which brought scalable storage and expanded streaming capabilities.

Despite these trends, the need for innovative storage platforms continues to grow. According to Microsoft, the world will generate 125 zettabytes of data per year by 2024. Enterprises and cloud service providers must find cost-effective ways to store this data and meet their performance, availability, and durability requirements.

Current storage technologies are insufficient to maintain this volume of data. Hard drives, for example, are limited by their mechanical nature. SSDs are still relatively expensive to implement on a large scale and do not always provide the necessary endurance.

The HSD project tackles holographic storage

To help meet future storage needs, Microsoft launched Project Holographic Storage Device (HSD), a collaborative research effort that revisits holographic technology, but this time with the idea of ​​providing cloud-scale storage for support hot data.

The HSD project is part of the Microsoft Optics for the Cloud group at the Microsoft Research Lab in Cambridge, England. Another of the group’s efforts is Project Silica, which is experimenting with using crystals to provide long-term archival storage. However, Project Silica only focuses on WORM operations, like traditional approaches to storing holographic data. The HSD project allows data to be erased and rewritten and will provide faster read and write speeds.

According to Microsoft, the mission of the project is “to design high-endurance, high-performance, non-mechanical motion cloud storage that is both high-performance and cost-effective.” Microsoft also says the project has already achieved 1.8 times the density of previous volumetric holographic data storage. The team is working to further increase densities and achieve faster access rates.

To help deliver on these promises, Project HSD uses core components like the high-resolution cameras and display screens of today’s smartphones. The project also uses machine learning and deep learning to further improve accuracy and performance. As a result, the team reduced optical distortions and manufacturing tolerance requirements. It uses software to compensate and calibrate the system while running.

The material used for the storage medium also sets Project HSD apart from other attempts to store holographic data. Many other projects used polymers to store permanent changes in the material, so they were limited to WORM operations.

In contrast, the HSD project stores the holograms in electro-optic crystalline materials. The project stores each hologram as a spatial variation of the electron density distribution, which it can alter by exposing the medium to light of a specific wavelength. Holograms can also be erased by exposing the crystalline material to ultraviolet light.

Despite Microsoft’s departure from traditional holographic storage, the basic approach to reading and writing data is much the same. The storage process begins by splitting a laser beam into two signals. One of the beams carries the data to the storage medium. The data-carrying beam – also called a data, object or signal beam – passes through a device called a special light modulator, which then passes or blocks the light at the points corresponding to the binary 1s and 0s. The modulated data beam then continues to the crystalline material.

The second light beam is called the reference beam. The beam does not pass through a light modulator but bounces off a mirror and is redirected to the storage medium, where it intersects with the data beam to create a 3D interference pattern in the optical material.

The pattern forms a tiny hologram that represents a single page of data, which can contain hundreds of kilobytes of data. A page of data occupies a small volume, or area, in the optical material. A zone can contain several pages and the storage medium can contain several zones.

A holographic storage device reads data by diffracting the hologram’s reference beam into the storage medium. The data harness is not required for this operation. A camera captures the diffracted image, which makes it possible to reconstruct the original data page. A holographic storage system can read different holograms by changing the angle of the reference beam, or it can erase holograms with UV light, allowing the data to be rewritten.

Holographic data storage promises a cost-effective way to answer specific business questions efficiently and quickly.

Uses for Holographic Data Storage

In its use of crystalline materials, Project HSD takes advantage of the inherent parallelism of optics. It allows data to be written to and read from the storage medium in parallel, resulting in higher overall throughputs. The project’s approach also requires fewer mechanical parts, like those found in hard drives. Instead, it limits movement to readjusting the angle of the laser beam; all other components remain fixed. Additionally, holographic storage can utilize the entire volume of the medium, rather than just its surface, which provides greater densities than current optical storage types.

Traditional approaches to holographic storage have focused on archiving data and supporting WORM operations. The HSD project targets warm data that supports read and write operations, which could benefit cloud providers and enterprise data centers. Warm data is generally accessed and updated less frequently than data that supports important business applications and is rarely maintained in real time, although it generally requires greater scalability. Performance requirements may vary depending on supported workloads.

Holographic data storage promises a cost-effective way to answer specific business questions efficiently and quickly. Its fast read performance and ability to update data make it well suited for data warehousing, big data analysis, and applications that incorporate advanced technologies such as predictive analytics or artificial intelligence .

Organizations that need to generate regular reports, such as weekly call center statistics or monthly sales figures, could benefit from holographic data storage. Holographic storage could also support less critical operations, such as providing support staff with the basic information they need to help their customers.

How close are we to holographic storage?

Despite its promise, holographic storage still has a long way to go from the research phase to the point where companies can purchase commercial products. Manufacturers are expected to introduce entirely new environments for building storage devices, which will require a high degree of precision to ensure proper alignment between components. Additionally, most research on the HSD project has focused on writing to and reading from a single area. The team still faces the challenge of delivering the same level of performance in multiple areas. Another concern is to ensure that accidental exposure to UV light does not erase the data.

Because holographic data storage is a technology with so many false starts, it’s no surprise that Microsoft has avoided making predictions about the technology’s commercial application. In the meantime, there are many other efforts being made to entertain the industry, ranging from storage class memory to DNA storage.