Azure Updates: HDInsight; Generative AI; VM images

Microsoft announced the public preview of HDInsight on Azure Kubernetes Service, an open-source big data service with two new workloads. General Manager Balaji Sankaran discussed the capabilities of HDInsight.  The offering is able to handle Apache Spark, Apache Flink, and Trino workloads, and is integrated with Monitor, Data Factory, and Power BI.

According to Sankaran, HDInsight’s emerging role is to give customers a way to leverage open-source software for analytics, use managed infrastructure, and reduce cost with end-to-end analytics. Microsoft takes care of the managed service component of the offering. As a result, customers can use containerization for scalability and management, running with minimal disruptions.

Principal program manager Ben Ufuk Tezcan explored the growing role of generative AI and large language models. Since the start of 2023, Microsoft has repeatedly emphasized LLMs in services like Microsoft 365 Copilot, Teams, and Power Platform. As part of that initiative, Microsoft has introduced RAG, a natural language processing technique that brings together external retrieval and large pre-trained language models. With RAG, the system initially receives an input sequence, then retrieves relevant documents, which are then merged with the original input to add context. According to Tezcan, RAG has a wide variety of use cases such as financial analysis, personalization, summarization, and automation. Tezcan wrote: