The Importance of Data Storage Hygiene for Big Tech, CIOSEA News, ETCIO SEA
By the mid-2000s, the importance of data had grown so much that data was said to be the new oil or gold. A 2015 Deloitte article mentioned that organizations driven by data, insights and artificial intelligence must have significant competitive advantages over others. As digital transformation, data and analytics have evolved from the era of on-premises applications and RDBMS to one driven by cloud, mobility, social media, artificial intelligence and the Internet of Things (IoT), this article from the World Economic Forum in 2015 predicted the challenges in storing such gigantic data and that the world could run out of adequate data storage space.
The pandemic has ushered in accelerated digital optimization and transformation, with the additional drivers of 5G, edge computing, blockchain, industry 4.0, hybrid working, metaverse and web 3.0 contributing to a new data increase. This 2020 IDC report estimated that over 64 zettabytes of data was created/replicated across the world. Additionally, it is projected that over the period 2020-2025, data creation and replication across the globe will grow at a CAGR of 23% and this growth will be more than twice the amount of data created since the beginning. invention of digital storage! Unlike data creation and replication, overall storage capacity was 6.7 zettabytes in 2020, and it is estimated that storage growth will be less than data creation/replication. The World Economic Forum in this report at the end of 2019, estimated an even higher quantum of data creation compared to IDC’s figures and corroborated that conventional optical/magnetic storage systems will be unable to handle this phenomenon for more than a century.
On the other hand, it has become essential for CIOs and business leaders to increase the storage of their own data. This will help them leverage data analytics and business insights for organizational growth and KPIs such as launching new products and services, and tracking engagement, satisfaction and reputation indices of customers, employees and of the supply chain.
So what are the considerations for managing and monitoring optimal hygiene for data storage in today’s digital age?
The post-pandemic recovery period has only heightened the importance of data and analytics for organizations that now have on-premises, cloud, and edge data. According to this study by Gartner, some of the top data trends for 2022 and beyond are the importance of data structure, technical business intelligence, regulatory compliance in relevant geographies, increased use of outskirts, among others. There are therefore several important considerations for data storage hygiene, some of which are:
The quality, accuracy and hygiene of the data itself: Corroborate the relevance and quantity of data, implement standardization and normalization policies, especially in cross-platform scenarios, eliminate errors due to duplicity, incompleteness, incorrect analysis, perform cleaning or Data cleansing and updating policies are some of the strategies deployed by CDOs and CIOs.
The basics of data storage: Having policies around data and fields to be stored, historical data management, compliance compliance, robust retention and disposition policies, storage architectures, deployment of the storage mechanism itself spanning the cloud , edge, network attached storage (NAS), storage area network (SAN), security, recovery and access control and forecasting. Consider offsite backups for disaster recovery, BCP, and data backup.
Risk, regulatory and compliance perspective with respect to the storage, retention, retrieval, sharing, access and disposition of data. Multinational corporations and businesses spread across multiple countries should carefully study and comply with relevant regulations
Security and Access covering both physical and cybersecurity. CDOs work with CIOs and CISOs on aspects such as data encryption, multi-factor authentication, password/passphrase management, backups and updates and other aspects
Cultural aspects: Cultural, communication and employee best practices also have a critical role to play in data storage hygiene. Communication and training on data storage and retrieval aspects of information security policies, training, development, best practices, escalation matrices, and rewards and recognition programs play a important role. The ideal environment would foster self-learning, high levels of engagement and curiosity from employees, gig workers and the extended enterprise across all functions and levels. CHROs can also play their part by leveraging communication, collaboration, gamification, training, rewards and recognition to drive awareness and self-learning in this very important area.
Using automation: Automation has immense potential and use cases in storage management. It handles manual data storage tasks spanning structured, unstructured, and archival data, performs data cleansing, streamlines provisioning and allocation of required storage, and provides faster, optimized deployment to storage devices. storage, in addition to forecasts. Thus, providing holistic end-to-end visibility, control, insights and monitoring of data storage in a cost-effective and future-proof manner
Cost considerations and internal evaluation versus storage as a service: Storage as a Service (STaaS) is growing in popularity. Managed service providers are adding STaaS to their portfolio. According to Gartner research here, STaaS is expected to replace over 40% of all on-premises IT storage administration and support costs by 2025.
Leveraging the cloud and offloading historical data to low-cost storage can save huge storage costs without compromising growth, as this McKinsey article shows.
How does automation benefit data storage?
Data storage automation and orchestration helps prioritize processes, tasks, and resources to balance the speed, efficiency, utilization, and cost of the data storage function and free up manual resources for tasks with higher added value. Automation is deployed especially for processes such as provisioning, cleanup, capacity management, workflows and data migration, resource optimization, data protection and recovery policies disaster, capacity and associated bandwidth forecasting, configuration management and software updates/upgrades.
Intelligent automation and hyper automation are leveraged in NAS, SAN, cloud, and edge. Automated data lifecycle management, cloud deployment, workload consolidation, hybrid storage, and integration with artificial intelligence systems are also great use cases. automating.
AI analytics is increasingly being deployed in the areas of capacity needs forecasting, performance bottlenecks, and other KPIs, especially for businesses with data storage in hybrid, multi-cloud and edge environments.
What are some of the future trends in data storage?
Tech leaders are already working on blockchain-powered decentralized data storage as well as edge storage.
There are also developments around the feasibility of using DNA-based data storage (and the resulting data-reading capabilities) to address the storage space exhaustion challenges of conventional optical or magnetic data, such as this World Economic Forum mentions of articles. This DNA-based storage will also potentially address concerns of excessive power consumption by data centers.
Although commercial quantum computing is still a few years away, it will also be interesting to see its impact on data storage.
In this continuous world of uncertainty and data implosion, data storage hygiene is indeed a foundational element to building a successful, competitive, innovative and compliant data-driven business in the times to come!