AI and the Storage Landscape: A Complex Relationship
An AI transformation is sweeping across the IT storage landscape, creating a highly advantageous environment that necessitates more data to be stored and delivered to AI models and agents, greater data protection, improved data access governance, and significantly enhanced storage operational environments. However, the flip side is that AI can lead to chaos, resulting in data errors and intentional agent-enhanced assaults. We harness AI to improve conditions and require AI to mitigate the risks it can introduce.
Forty-four months ago, following the launch of ChatGPT, the storage sector entered an irrevocable shift into the AI age. The technological advancements essential for providing swift data access to the preferred AI processor, the GPU, transformed the NAND and SSD suppliers, the flash array hardware and software providers, and the HPC/supercomputing domain.
The Established Players vs. Emerging Contenders
The traditional and relatively stable, pre-ChatGPT period dominated by Dell, HPE, and NetApp in the enterprise storage array market confronted a wave of new vendors expanding rapidly from the all-flash array and HPC sectors. Companies such as DDN, Pure Storage, VAST Data, and WEKA experienced swift growth as parallel data access became a crucial software innovation, complementing disaggregated storage array structures, the rising utilization of unstructured data, the rapid rise of analytics, and the development of AI-centric data lakes like Databricks and Snowflake.
The Cloud Revolution: The GPU-as-a-Service Crew
A brand new public cloud segment emerged: the GPU-as-a-Service neoclouds, including CoreWeave and Lambda. In the initial phase of AI storage, AI training took precedence, but it is now being surpassed by AI inference; production AI, with organizations setting up AI factories to create, optimize, implement, and operate their own and external AI agents.
Digital Workforce and Data Chaos
Model Context Protocol (MCP) and graph technologies empower digital workers to act and reason, access, and modify both structured and unstructured data. They are prone to errors, leading to the necessity of documenting their actions to allow rectification if they diverge from the correct path. These digital workers require oversight through Agent Identity Access Management.
Five Key Areas Where Storage and AI Intersect
- Storage supplying data to AI
- Safeguarding AI data and activities
- Storage cyber-resilience expanded to regulate AI data access
- Storage leveraging AI for its own operations
- Storage shielded from AI-driven threats
Data for AI: Keeping GPUs Satisfied
Nvidia has vigorously endorsed storage delivery technologies to ensure its GPUs operate efficiently without being IO-constrained. Simply providing them data from parallel file systems through disk drive arrays was inadequate. These disks have been replaced by SSDs, with NVMe and PCIe interconnects superseding the SAS and SATA protocols of the disk drive era.
Emergence of New Data Management Players
Innovative initiatives like the disaggregated storage array (DASE) technology, led by VAST Data, have been embraced by Dell, Everpure, HPE, and NetApp. Traditional, high-end storage arrays from Hitachi Vantara, IBM, and Lenovo’s Infinidat have yet to adopt DASE, GPUDirect, or KV caching. Their architectures exclude them from participation, and they are becoming the storage counterparts of mainframes, a relegated yet still essential segment.
Vector Databases and AI Workflows
AI models handle tokens, which are transformed into vector embedding data that must be stored and searched. Specialized vector database providers have emerged, such as Pinecone, Qdrant, Weaviate, and Zilliz, while multi-model OLAP and OLTP databases have integrated vector support, with SingleStore serving as an example.
Protection and Resilience: AI’s Alternative Perspective
Secure Your AI with Backup and Cyber-Resilience
Backup and cyber-resilience solutions, including Cohesity, Commvault, Druva, Rubrik, and Veeam, have acknowledged that their backups serve as a valuable data source for AI models and agents. They developed in-house AI agents, like Cohesity’s Gaia, its Gen AI search assistant, to build on this concept.
AI Cyber Resilience: Identity Access Management
Leading backup providers have transitioned to become cyber-resilience providers and offer various forms of identity access management (IAM) and associated data access monitoring. IAM has historically focused on human users but is quickly incorporating AI agent data and resource management as well. Viewing an AI agent as a digital employee clarifies the urgency for agent-centric IAM.
AI Empowering Storage to Enhance Itself
Storage Vendors and AI Innovations
We have observed storage array providers exploiting machine learning to oversee array telemetry and detect malfunctions for quite some time. HPE-acquired Nimble was among the pioneers in this regard, and it has now become standard practice. Contemporary AI can do even more.
AI Chatbots: The Modern Diagnostic Technicians
A clear application for AI chatbots is functioning as the interface between an administrator, utilizing natural language, and the data protection software. The chatbot, or agent, is trained on the features and telemetry of the data protection software and can operate as an adept diagnostic technician assisting administrative personnel.
The Major Concern: Trusting AI Agents
AI agents are abstract, unseen entities that make decisions in microseconds. In contrast, we human operators are concrete, visible, and deliberate in our decision-making. It is straightforward for computers to observe us, but nearly impossible for us to supervise agents. We will need to deploy agents to oversee agents, necessitating trust in our monitoring agents—genuine trust. They must be immutable, resistant to identity theft, and capable of identifying and neutralizing rogue agents swiftly. Consider the potential scenario of a North Korean hacking outfit creating a swarm of attack agents. We must be prepared for this. The threats are looming.
Conclusion: AI and Storage – The Ever-Puzzling Duo
Isn’t AI rejuvenating the storage field? It resembles that awkward couple at the bar – one is uncertain if they are about to share a kiss or engage in a heated argument. Here’s to hoping we can guide it to be beneficial and constructive before it spirals into full-fledged Skynet territory!
