Azure Updates: AI infrastructure; MLPerf; Service retirements

General Manager Omar Khan shared a new Microsoft study called The State of AI Infrastructure. Microsoft is introducing a string of infrastructure improvements to support its AI objectives. The list of changes includes new VMs with NVIDIA Hopper GPUs optimized for high performance computing workloads, custom built silicon, and confidential VMs with NVIDIA H100 GPUs. “2024 is shaping up to be an even more promising year for AI than its predecessor. With the rapid pace of technological advancements, AI infrastructure is becoming more diverse and widespread than ever before. From cloud to edge, CPUs to GPUs, and application-specific integrated circuits (ASICs), the AI hardware and software landscape is expanding at an impressive rate,” Khan wrote. 

Chief Product Officer Sarah Bird discussed AI tools to build “trustworthy” apps. These new tools are coming to AI Studio and include Prompt Shields, groundedness detection, safety evals and system messages, as well as risk and safety monitoring. Modern data governance emerged a few years ago as a theme for Microsoft. That theme continues to receive attention. The team discussed a new data governance scheme in Microsoft Purview, in preview April 8, 2024, which is intended to be scalable, unified, integrated, and extensible.

MLCommons recently published MLPerf Inference results using NC H100 v5 series VMs, powered by NVIDIA H100 NVL Tensor Core GPUs. Generative AI models are rapidly growing in size and scope, like the 70 billion parameters in the Llama2 model, a startling increase in size from Large Language Model GPT-J which has 10 times fewer parameters. To handle much more substantial models, each GPU in the NC H100 v5-series has 94GB of HBM3 memory.