AI Ready Intelligent Data Infrastructure for Data Quality and Governance

AI-Ready Intelligent Data Infrastructure for Data Quality and Governance

Data may no longer be known as the “new oil”, but data represents the foundation of Artificial Intelligence (AI) and Machine Learning (ML) applications. In the newer world of generative AI, data quality becomes even more important. The quantity and quality of data directly affect the accuracy and use of machine learning models. AI is iterative, but the knowledge it gains with each iteration must be relevant. As noted in this year’s AMS 2025 theme, time scales and the ability to improve forecasts, insights and decision-making in a timely fashion is equally important for AI/ML processing.

What technologies can help refine data for AI/ML and gen AI processing and retrain commercial Large Language Models (LLMs) to enable federal agencies like NOAA to calibrate and ensure that they are using the best data for an application? What technologies can be used for data traceability to assess accuracy? These are questions raised in prior AMS forums. AI is today’s biggest disruptor because it has the potential to transform industries, how we work, and deliver quantifiable societal benefits. Technology can help with some of the “guard rails” as well as efficiencies needed to scale AI applications.

The technologies that enable AI must deliver the following key capabilities:

1. Simplify data management with platforms that are available everywhere throughout an AI lifecycle – data processing, training, fine-tuning, inferencing, and monitoring – whether this is at the edge in a WFO or remote location, in a NOAA or partner datacenter or in a cloud service.
2. Maximize performance for data scientists, engineers, and developers. This is where the speed of innovation makes a difference. Technology must accelerate model training to maximize productivity.
3. Deploy and scale AI safely, responsibly, and confidently. Technology must provide easy access by AI models and apps across public and hybrid clouds as well as on premises. AI platforms must enable versioning and traceability of multiple AI production models and keep data protected and secured via built-in real-time cybersecurity and anomaly detection. Finally, technology platforms can deliver built in data classification to assist with data privacy and governance policies.

This presentation will focus on the key features that technology platforms like NetApp provide to enable efficient and responsible AI use. It will walk through a typical AI/ML Workflow (Figure 1 below) and provide several examples of how technology platforms are delivering these capabilities today.

Figure 1 - Typical AI/ML Workflow (see graphic uploaded separately below)

Capabilities such as snapshots and cloning take on new importance in AI data preparation, training and deployment. Generative AI requires additional fine-tuning of commercial Large Language models and Retrieval Augmented Generation (RAG) to maintain data security and governance as commercial models and internal or private data is combined to refine these models for specific applications. Every step of AI/ML workflows benefits from these features that save time, infrastructure costs and improve data governance by enabling comprehensive iterations at scale. This presentation will share NetApp technology and use cases that include using AI to locate and fight wildfires and smart city technologies to provide timely preparedness data such as flood data for urban areas.

Attendees will benefit from learning how these technology features are table stakes for effective data governance at scale as AI, ML and now generative AI models are used in the weather community. They will leave the presentation with knowledge on how they can leverage these features to ensure data quality and governance in AI and GenAI use cases today.