Dark Data Storage, Security and E.S.G. Compliance Biggest Challenges in Data Infrastructure Management
Building an environment-friendly data infrastructure management, powered by AIOps and cloud computing, tops the list priorities of most business leaders. With the rapid advancements in computing and networking capabilities, modern organizations are embracing newer ways to adopt high-performance data infrastructure. However, only 10% of executives believe that their organizations are mature enough to be called as insights-driven.
What’s their limitation?
According to Deloitte, 63% of executives were aware of analytics, but lacked the data infrastructure to leverage their ad-hoc analytics operations for a full-fledged enterprise transformation.
To achieve great results with analytics and digital transformation, striking a balance between your current data infrastructure management and security policies remains a key.
We all know that modern organizations are drowning in big data, creating an overwhelming amount of pressure on the existing data infrastructures and the security frameworks safeguarding them.
CIOs and CISOs recognize that any chink in their existing data infrastructure management could cost their businesses millions of dollars in wasted resources, lost opportunities, and security incidents. Worse, if left unaddressed, these gaps can result in data breaches and compliance-related penalties. Most leaders understand that data infrastructure is critical to digital transformation, a majority of these front-line professionals also accept that it takes more than just a good data center or a cloud computing platform to address all the issues related to data management. According to a recent report published by Hitachi Vantara, data infrastructure emerges as a successful foundation for digital transformation when it is designed for energy efficiency, resilience and data delivery for a wide range of use cases across all industries.
While business leaders address their pain points related to the resiliency of the data infrastructures, nearly 75% of US-based respondents mentioned these systems are prone to failure amidst the surging demand for better data security and sustainability solutions. In the survey of 1,288 global C-level executives and IT decision makers, the report highlighted the different types of challenges that organizations face while building a strong, secured, resilient and sustainable data infrastructure in 2023.
We reviewed the Hitachi Vantara Modern Data Infrastructure Dynamics Report to come up with the top 10 things you should know about managing your data infrastructure management in 2023.
Finding 1: Data Handling is Getting Over-complicated
The challenge of identifying and managing unused data, also called as ‘dark data’, is toppling digital transformation efforts
17 PB of data — that’s the volume of information organizations currently collect, process, and store in some form within their data centers but never utilize for their purposes.
Most executives, despite belonging to the organizations with a strong data-culture, find themselves struggling with all the tools and resources meant to drive their digital journeys. A majority of them are analytics-aware, yet they express discomfort accessing or utilizing data stored in their systems. This data mishandling causes piling of dark data.
IT leaders believe that this is nearly half of all the data they have stored in their systems but never managed to tap into these for insights and analytics. Hitachi Vantara has cited two main reasons pushing data teams into a spiraling loop of confusion.
Reason 1: Disconnected systems for data handling
Reason 2: Data mismanagement due to patchwork
In complex organizations where data siloes create multiple layers of technology and data handing systems, they end up with under-performing resources, resulting in financial loss to the company.
Over-complicated, siloed data infrastructure end up costing organizations millions of dollars, and a completely broken digital transformation journey.
Finding 2: Current Infrastructures Can’t be Scaled
By 2025, organizations could be spending millions more on data management and storage. On an average, data infrastructures would swell up to handle 65 PB of data annually, twice of what’s handled in 2023. Unfortunately, the current infrastructure would massively lag, with little or no room to grow in terms of size and flexibility.
In fact, Hitachi Vantara’s survey found two-thirds of IT leaders are concerned about their current state of critical data infrastructures crumbling down in the future under the weight of massive data. 71% of leaders also cited the lack of flexibility in their existing systems to adapt to the changing needs.
Meaning, modern data infrastructures would be rendered useless if leaders do not embrace innovative ideas built on hybrid cloud platforms.
Finding 3: Data Infrastructures Eat into 50% of an IT Budget
The prevailing economic conditions have little or no effect on the current IT budgets. In fact, the investments into data infrastructures are expected to grow further to tackle the ever-growing needs related to data handling, security, compliance, and ESG.
By 2025, 30% of IT decision-makers expect to grow their investments into unstructured IoT and surveillance by up to 50 percent.
IT leaders said 50% of their budget goes into data infrastructure management, protecting the most valuable asset of their business — DATA. Therefore, having a good investment strategy to secure the data foundations should be the number one priority of any business.
Data management and subsequent storage cost money in terms of resources, people, and technology investments. In fact, even discarding data is also a costly operation. Data destruction operations have to comply with all the regulatory frameworks to avoid breach and penalties in the future.
Finding 4: Sustainability Goals are Lagging Related to the Unused Data Storage
To thrive in today’s marketplace, organizations are developing their infrastructures around the ESG goals. It’s true that most data-driven companies are committing their efforts toward building a low-carbon economy. However, they still fall short of the expectations.
According to HPE, 30% capacity of all large data centers in the world remain unused, costing the economy 30 billion dollars annually. That’s the financial impact of unused storage. Its impact on the environment is worse. Apparently with a larger carbon footprint than the current aviation industry, data centers snack 2% of world’s electricity. By 2030, data centers will use up 8% of electricity to manage the advanced AI and cloud computing technologies.
Recently, Gijma had selected the Hitachi Virtual Storage Platform (VSP) arrays to reduce energy costs while also improving the performance and scalability of its data infrastructure.
Despite a tremendous push to all sides of the business, organizations still lack the vision and determination to address the impact of storing unused or dark data on the environment. Although 76% of IT leaders approve of their data infrastructure solutions as eco-friendly, the strategies to manage data storage and the prevailing ESG goals are seemingly disconnected from each other.
Hitachi Vantara’s report cited 66% of IT leaders currently measure their data center’s energy consumption. Only 32% agreed that their data infrastructure is over-spending on energy resources and are therefore inefficient. Nonetheless, 57% of organizations with sustainability goals don’t measure the impact of their unused stored data on the environment.
Finding 5: Leadership is Responsible for Developing Sustainability of Technology and Data
The sole responsibility of ensuring sustainability of technology and data lies with the senior management. This includes the top business leaders and the executive management. Interestingly, a significant degree of responsibility is also bestowed upon the third-party vendors, and experts or consultants, followed by the employees, regulatory authorities, and government bodies.
Businesses Must Lead on Sustainability
Finding 6: Confidence on the Existing Data Security Policies is Low
Evolving data threats are taking down infrastructures. The lack of data transparency influence trust in the existing DataOps. There is no middle ground for IT leaders when it comes to trusting their IT infrastructure for data management and security. 22% of business leaders cite their important data has not been backed up; 18% had already lost critical information stored in their systems due to storage corruption in the last 2 years.
Organizations should have a 100% resilient and secured architecture to recover 100% of their data after recovering from an incident such as data breach and ransomware attacks. Hitachi Vantara’s survey reported only 22% admitted their critical data is not backed up for recovery.
69% of decision-makers also stated their inability to detect a data breach in time and protect it from criminals. As threats continue to evolve and target humans with social engineering tactics, it is important to build a resilient data infrastructure that covers for the human errors and undo the mistakes without risking data loss.
Finding 7: Data Distribution Across all Platforms to Stay Consistent
We have already pointed out earlier in this article about the doubling of data stored in the organization in the next two years. This means, IT leaders have to find out newer ways to double their capacity as well to manage all the data across all platforms — public and private clouds, on-prem, and co-located or managed services. As the data landscape gets more complicated, IT leaders have to enable their digital transformations in a hybrid setup.
Finding 8: Users want these from their data infrastructure solutions providers
Business leaders want data infrastructure solutions providers to address security, flexibility and scalability from Day One. Agility, efficiency and cost are other important factors in decision-making while looking out for data infrastructure solutions.
Most importantly, 73% of IT leaders are likely to pick a vendor who actively uses its expertise to aid the environment by reducing carbon footprint and energy consumption without escalating challenges in security, flexibility, and scalability.
Finding 9: Where is third-party assistance required?
36% seek third-party support for creating a resilient data infrastructure and for ensuring data continuity.
Improving cybersecurity is the top-ranking area where business leaders require third-party assistance. According to the report, 23% of decision-makers have already out-sourced data security requirements to the external vendors.
Unused data also plays an important role in deciding a third-party service provider for infrastructure management. 27% of leaders want to understand about the amount of unused data from the third-party help. 56% of leaders want help from third-party for compliance (28%), and for adopting a consumption-based model for their existing DataOps needs.
Finding 10: Regularity Mishaps Happen Because of Missed Audits and Data Inaccessibility
Organizations face an uphill task in data stewardship and governance.
30% business leaders accepted they missed regulatory audits because of inaccessibility. 27% said they lack the necessary skills and training assistance to overcome the barriers in data access. It largely hampers the customers who use cloud storage platforms. 44% of decision-makers also cited a situation where no one in their organization had an idea where all the data gets collected and stored. As risks from ransomware attacks and data breaches escalate, organizations are forced to think and act on their data infrastructure modernization and transformation.
By combining the powers of DataOps and AIOps automation, it is possible to address all the challenges mentioned in the Hitachi Vantara Modern Data Infrastructure Dynamics Report. 76% of respondents already use advanced tools such as AI and automation to minimize their data storage requirements.
And, that’s a wrap.
What did we learn from the Hitachi Vantara Modern Data Infrastructure Dynamics Report?
There are so many activities that an organization can do to build a resilient and transparent data infrastructure for their DataOps teams. For example, utilizing storage tools to prioritize data audits and accessibility to critical data even after the servers have been affected by a data breach or cyber attack. Similarly, data leaders should build a strategy to manage duplicate data, dirty data and unused data, so that the data infrastructure can auto-scale with time. Ensuring IT costs don’t spiral out of control, business leaders should adopt a future-proof data infrastructure that works across all the environments – Cloud, Virtualized, Containerized, and Hybrid.
