The demand from our clients to understand and adopt Microsoft Fabric has skyrocketed in the last year. In the past our client conversations often started with understanding their unique problem, and working towards a technical solution, however with Microsoft Fabric positioned as a Software-as-a-Solution (SaaS) and the strong marketing messages from Microsoft, many conversations regarding data modernisation now start with Fabric.
Whilst this is helpful, and we continue to understand the problem being addressed, having a solution like Fabric that can act as an accelerator to becoming data driven, and using data to support better decision-making is potentially game-changing for many enterprises.
Fabric at Microsoft Ignite 2024
We’re lucky enough to be attending Microsoft Ignite this week. The sheer number and magnitude of announcements being made around Fabric so far is sizeable. In this post I’d like to summarise some of the key announcements, and what they mean for anyone thinking about adopting Fabric.
One thing that is very clear is that the rate of innovation, resulting in new features in Fabric, is relentless.
As an Microsoft Partner who is actively delivering Fabric in to multiple enterprises, many of the announcements support our view that whilst Fabric is ready for production at scale, the features and innovation that continue to be delivered demonstrate the that Fabric evolution continues at pace, in some cases adding features which many may consider must-haves, to delivering new functionality that provide business benefit underseen in many equivalent products.
Framing Fabric’s goals
One of the messages that Ignite has helped with, is the reframing of Fabric’s goals and where it fits into a modern data-driven organisation.
- It’s an AI-powered data platform. The continued focus on infusing AI into experiences continues, and given Fabric is at the cross-roads between your data and business value, it’s natural that Fabric wants to bridge that gap with AI.
- An open and AI-ready data lake. A single data lake over your multi-cloud data estate, performing analytics on the same data, is a goal, and aspiration, for many organisations today. The one lake concept is central to Fabric.
- AI-enabled business users. Arguably one of the most important goals is ensuring your business users can use the data effectively, and with tools and apps that they are familiar with. The way Microsoft extends and integrates Fabric into existing line-of-business applications will be critical to its success.
- A mission-critical foundation. As Fabric is a fully hosted solution, scalability, resiliency, security, governance and compliance are underpinned by the service, out of the box. Our job as consumers of Fabric is making sure our solutions leverage this underlying technology. How we transition our operational thinking from infrastructure and platform services to Software-as-a-Service has many implications for managing and running at scale.
Given the size and scale of functionality in Fabric I’ve broken down the key announcements to align with the Fabric four goals described above.
Fabric goal 1: It’s an AI-powered data platform
At Ignite there were significant updates across the Fabric Real-Time Intelligence, Fabric Data Factory, Fabric Data Engineering, Power BI and Industry Data Solutions workloads in Fabric.
Fabric databases
At Ignite Microsoft announced a new class of database, that allow transactional and analytic workloads from within the Fabric experience. The first database, SQL database in Fabric, is now in preview.
The integration of Azure AI, vector search and RAG support, as well as the management of the service from within Fabric maybe a good option to build some AI offerings, around your data, without the need to stand up additional databases that are hosted and managed elsewhere.
Ultimately this is a drive to make consuming AI on your data easier and more democratised. It will be interesting to see how Fabric Databases are picked up and used.
Enhancements to Real-Time Intelligence (RTI)
Now generally available, there were plenty of announcements around RTI, some features in preview, showing how Fabric is improving support for different real-time streams, and how different types of data sources can be accommodated within the Fabric and extended Microsoft ecosystem.
As our clients adopt Fabric the question we are repeatedly asked is “what features can I leverage to provide better business outcomes through insight?” The opportunity to handle real-time events can be a powerful enabler to this.
RTI is now generally available in Fabric. You can use both low-code and no-code tools to ingest high-volume streaming data with high granularity, dynamically transform streaming data, query data in real-time for instant insights, and trigger automated actions based on the data. RTI is supported by the Real-time Hub which acts as the central place to discover and manage streaming data.
Some of the other improvements, all currently in preview for RTI, included new Fabric events, enhancements to Eventstreams and Eventhouses, and easier real-time dashboard sharing.
For Eventhouses, Microsoft announced a migration experience from Synapse Data Explorer (SDX), now in preview, to move your SDX clusters to Eventhouse with minimal disruption. Any additional tooling to support migration and adoption is very welcome.
Announcements regarding integrating Fabric CI/CD tools including REST APIs, Git integration and deployment pipelines we also made. This is important to us and our clients, given our continued desire to automate, codify and accelerate technology through repeatable, reusable solutions. Without the underlying platform support this isn’t always possible, so improvements in Git and deployment pipelines are very welcome. We hope to see more of this across the platform over time. Many of the following announcements include improvements in CI/CD and GIT support which is very encouraging.
Fabric Data Factory
Enhancements to data pipelines in Fabric Data Factory include export/import capabilities to support the Apache Iceberg format, and a new Copilot in Fabric Data Pipeline experience, now in preview, to aid in developing data integration solutions via natural language.
As with RTI Fabric is introducing CI/CD support, into preview, for Dataflow Gen2 and Copy Job, including GIT integration, deployment pipelines, and REST APIs. This could be very significant for organization that want to focus on repeatability and automation, like we do at BJSS.
The better Data Factory becomes, and the more it simplifies the ingestion of data, the better for everyone. The support for formats like Iceberg, continue to show that Microsoft understand that you are more likely to bring your data to Fabric if you can integrate and consume via native pipeline support for 3rd party sources.
Fabric Data Science
Announced at Ignite, AI functions are coming soon in preview to Fabric. AI functions provide a simplified API in Fabric notebooks for common AI enrichments for text like summarisation, translation, classification, sentiment analysis, grammar correction. With a few lines of code you can leverage these functions without the needed to wire-up your own solution.
Fabric Data Engineering
Fabric notebooks now support a pure Python experience, helping developers use Fabric notebooks for smaller datasets with a Python runtime. The Fabric notebook Python experience offers multiple built-in Python kernels, easy data integration with Fabric lakehouses and a quick setup with a 5-second spin-up time. Live versioning for Fabric notebooks, enabling you to track change history, compare and restore previous versions, is now included. This, like many other features announced, are going to help manage adoption and effect more controlled usage.
New features to support improved geographic information system (GIS) mapping systems has been added. This enables users to run spatial analytics and visualize geographic data directly within Fabric Data Engineering and Data Science workflows.
The API for GraphQL is now generally available. This is an API that enables developers to access data from multiple sources in Fabric with a single query API. Introduced were several new capabilities including support for new data sources like Azure SQL and Fabric SQL databases. A dashboard has been added to simplify monitoring performance and API activity. Like some of the other announcements CI/CD support with GIT and deployment pipelines were highlighted.
Power BI
A new Tabular Model Definition Language (TMDL) view in Power BI Desktop is coming soon. This provides a semantic-modelling-as-code scripting environment utilising TMDL to modify the semantic model within a code editor. Also announced was the ability to include Copilot-generated reports and page summaries in the emails of a Power BI subscription. This could be very useful when you are providing automated incremental aggregated summary reports to users.
There was more focus on geospatial data, with a new layer type for the Azure Maps visual to help visualise geospatial data was announced. The new layer lets you connect different groups of locations together.
Sustainability Industry Solution is now GA
Microsoft previously announced industry-specific data solutions in Microsoft Fabric for retail, manufacturing, healthcare, and sustainability. Announced at Ignite was the general availability of sustainability data solutions in Microsoft Fabric. Environmental, Social and Governance (ESG) data is supported to aid removing of data silos, standardization, enrichment, benchmarking and governing the data for integrations with business and management systems.
It’s been interesting to see the sustainability solution evolve over the last few years and could help organisations understand their ESG position in a unified way.
Workload development kit is now GA
The Fabric Workload Development Kit announced last spring is now generally available. It was created to help developers design, build, and interoperate applications within Fabric. Applications built using this kit appear as a native workload within Fabric, providing a consistent experience for users directly in their Fabric environment. These applications become available in the workload hub, providing a more marketplace, centralized resource for intra-organizational reuse, and consumption of 3rd party solutions.
Fabric Goal 2: Open and AI-ready data lake
OneLake, Fabric’s unified, multi-cloud data lake, is designed to span an entire organisation, connect to data, and reduce data duplication. Two key announcements have caught my eye, the OneLake catalog, and Open Mirroring.
OneLake catalog
The OneLake catalog was announced at Ignite. It’s a solution to explore, manage and govern your Fabric data estate. It’s an evolution of the OneLake data hub experience with enhancements to help users discover and manage trusted data. It also provides governance for data owners with insights, and recommended actions. The OneLake catalog comes covers two areas: Explore and Govern.
Explore allows users to explore and manage the Fabric items they have access to in a central location with filters for domains, item type, owner, endorsement, tags. Users can select a Fabric item to take actions or explore description, data lineage, permissions, and activity.
Govern allows data owners to see an overview of their data with insights on endorsements, data labelling and data type, getting recommended actions to enhance their data for improved quality and compliance.
Explore tab is now available and the Govern tab will be coming soon in preview.
You can extend the capabilities of the OneLake catalog by connecting it with the Microsoft Purview Data Governance solution. Purview’s catalog offering has capabilities for data discovery, curation and quality.
Announced were deeper data quality support for sources like OneLake, Azure Databricks Unity Catalog and Snowflake Polaris.
Given its advanced features and its ability to ingest data from multiple catalogs, Microsoft Purview Data Catalog is being renamed to Microsoft Purview Unified Catalog to reflect its broader capabilities.
Mirroring
Mirroring support has been extended with new sources available for mirroring, a zero-ETL way of ingesting data in near-real time from existing databases or data warehouses.
Azure SQL DB Mirroring is now general availability and SQL Managed Instance mirroring in now in preview.
Microsoft also introduced a new capability called Open Mirroring. Now in preview this feature extends Fabric by allowing applications or data providers to bring their data estate directly into OneLake. By enabling apps to write change data directly into a mirrored database within Fabric, Open Mirroring simplifies the handling of complex data changes and the resulting mirror.
Open Mirroring could allow a new approach to syncing your data and be analytics-ready from within OneLake.
Goal 3: AI-enabled business users
The right insights can help organisations be more efficient and get more done, but unless those insights are available in the tools your business teams use everyday then there remains a gap in productivity.
AI Skills
At the Microsoft Build event last May, Microsoft announced a new capability called AI skills, which is an AI-powered conversational Q&A experience against your data. Announced today were enhancements, coming soon, for the support of semantic models and Eventhouse KQL databases, and multiple data sources at the same time in AI skills.
AI skills are being integrated with the Azure AI Agent Service. This will allow developers to build AI apps using Fabric data as a core knowledge source. This could be via a low-code chatbot or custom generative AI apps in Azure AI Studio.
We see a lot a benefits of AI skills, the integration with Azure AI Agent Service opens up multiple possibilities for leveraging your Fabric AI insights from within your custom solutions.
Goal 4: Mission-critical foundation
All our clients want to be assured that Fabric means their requirements with regards to running their key line of business and mission critical workloads. It means business continuity and disaster recovery, compliance with standards across industry like GDPR, CCPA, and HIPPA, security and governance are non-negotiable.
Workspace Monitoring
To support observability workspace monitoring in Fabric was announced. It is designed for admins and developers to view detailed diagnostic logs and workload metrics about their workspaces to troubleshoot performance issues, capacity performance, and data downtime.
Fabric will provide a read-only Eventhouse KQL database of workspace logs that users can query ad-hoc, analyse for patterns and anomalies, or save drafted queries as query sets. You can use this database to conduct root-cause analysis for errors, long running queries, refresh failures, and other issues.
This feature is important, aiding operational management. We’ve experienced what happens when you can’t fully uncover what is happening in the workspace, so we are excited by this feature.
Microsoft Purview Integration
There is extended integration with Microsoft Purview, Microsoft’s unified data security, data governance, and compliance solution.
Microsoft Purview Protection policies have been extended to automatically enforce access permissions to Fabric Databases alongside already supported items like semantic models, notebooks and lakehouses. This is also coming soon to Power BI reports, datamarts and warehouses.
Integration with Purview Data Loss Prevention (DLP) policies will enable security admins to detect the upload of sensitive data. You can now automatically restrict access to semantic models with sensitive data.
Billing for DLP, including for Power BI artifacts, will be part of a new Purview pay-as-you-go consumptive model that will begin charging in January.
Cross tenant capabilities
Cross tenant collaboration capabilities in Fabric are being extended, with external data sharing now being general availability. External data sharing allows you to directly share OneLake tables and folders with other Fabric tenants.
As this is built on OneLake’s shortcut capabilities, a welcome benefit is that you can share data in place without copying the data. Coming soon is the ability to share multiple OneLake folders and tables in a single share as well as sharing directly from data warehouses and Eventhouses in Fabric.
Also announced was a new tenant switcher in Fabric. Users with access to more than one Fabric tenant can switch between tenants directly from the account manager in the top right corner of the Fabric portal.
Fabric billing and consumption updates
There were some changes to billing and consumption announced. This area requires its own dedicated post, as there was a lot to unpack, but three announcements stood out for me.
Organisations with multiple capacities can now direct Copilot in Fabric consumption and billing to a specific capacity, no matter where the Copilot in Fabric usage actually takes place.
Admins can assign specific members of their organisation to the specified F64 or higher capacity for all their Copilot requests. These requests will be consumed and billed on that assigned F64+ capacity, ensuring Copilot in Fabric usage doesn’t impact priority jobs while expanding Copilot access to any workspace regardless of its capacity.
Capacity admins now have more control over the Fabric jobs running in their capacities. Surge Protection, now in preview, helps protect capacities from unexpected surges in background workload consumption. Admins can use surge protection to set a limit on background activity consumption which will prevent background jobs from starting when reached. Admins can configure different limits for each capacity in your organisation.
These announcements align with feedback we’ve heard. The continued improvement around the provisioning and management of Fabric SKUs, and how to manage surging are all good steps in the right direction.
So, what is next?
If you’ve managed to get this far, thanks for reading this summary of the announcements made at Ignite 2024 regarding Fabric. I’m sure there is so much more to come at Ignite and beyond.
The question you probably have now, is “So what?” or “What do I do with this information?”
The reality with Fabric is with the rate of change, continuing maturity and functionality will remain a challenge to keep on top of. The size and scale of Fabric, the breadth of personas and use cases it supports means that it will continue to require specialist support to maximise your investment in the product.
At BJSS we believe that Fabric will open up more data opportunities for organisations and help drive data-modernisation and data-driven insights. If you want to know more about what Fabric could do for your organisation, or are interested in the broader Cloud, Data & AI landscape then please reach out.
If you are attending Ignite in person and want to meet up feel free contact me or our team.
By Danny Amini, Chief Technologist at BJSS