Real-Time Intelligence in Microsoft Fabric

Microsoft Fabric’s Real-Time Intelligence, a comprehensive streaming solution optimized for analyzing time-series data at lightning speed. It’s designed to handle data of any scale from various sources and supports automatic partitioning and indexing of any data format, making it a versatile tool for real-time data processing.

The Need for Real-Time Insights

Imagine you’re a data analyst working for a large corporation, grappling with massive volumes of data on a daily basis. You require a solution that can seamlessly ingest data from diverse sources, execute analytical queries directly on raw data, and effortlessly scale to handle an unlimited amount of data. Additionally, you need to visualize data insights through intuitive dashboards and reports, while also setting alerts for specific thresholds.

This is where Microsoft Fabric’s Real-Time Intelligence shines, offering a streamlined workflow from data loading to visualization, empowering you to make informed decisions based on real-time insights.

An End-to-End Streaming Solution

Fabric’s Real-Time Intelligence solution offers a complete streaming solution for high-speed data analysis throughout the Fabric service. It’s specifically designed to handle time-series data and provides automatic partitioning and indexing for any data format, ensuring efficient processing and retrieval.

High Performance and Versatility

Real-Time Intelligence delivers consistent performance across a wide range of data sizes, from gigabytes to petabytes. It seamlessly handles data from diverse sources and in various formats. The Fabric’s Real-Time Intelligence workload finds applications in scenarios like IoT and log analytics across industries such as manufacturing, oil and gas, and automotive.

Understanding Real-Time Intelligence in Microsoft Fabric

Real-Time Intelligence is a fully managed service optimized for streaming time-series data. It enables consistent performance when searching and analyzing all types of data at scale, whether structured, unstructured, or semi-structured. Moreover, its deep integration with the entire Fabric suite streamlines your workflow from data loading to visualization.

Key Capabilities of Real-Time Intelligence in Fabric

Ingest data from any source, in any data format.
Run analytical queries directly on raw data without complex data models or transformation scripts.
Import data with by-default streaming for high-performance, low-latency, and fresh data analysis.
Benefit from automatic partitioning and indexing (both time-based and hash-based).
Work with versatile data structures and query structured, semi-structured, or free text.
Query raw data without transformation with high performance, low response times, and a rich set of operators.
Scale to an unlimited amount of data (gigabytes to petabytes) with unlimited scale on concurrent queries and users.
Seamlessly integrate with other workloads and items in Microsoft Fabric.

Explore the World of Real-Time Data with Real-Time Hub

The Real-Time Hub serves as your central access point to discover and manage the flow of your streaming data. It’s a dynamic catalog encompassing:

Data Connections: Links to various data sources (Event Hubs, IoT Hub, and more) for easy data ingestion.
Tables: Storage for ingested data, queryable using KQL.
Functions: Reusable code snippets for data transformation and analysis.
Queries: Saved KQL queries for repeated use.
Dashboards: Visual representations of data insights powered by KQL.

Visualize Data Insights with Real-Time Dashboards

Transform your data into actionable insights through KQL querysets, Real-Time Dashboards, and Power BI reports. Real-Time Intelligence facilitates a swift transition from data ingestion to visualization. These visualizations cater to both novice and experienced users, allowing them to present data in charts and tables with minimal coding effort.

You can leverage visual cues for filtering and aggregating query results, and a wide range of built-in visualizations is at your disposal. Insights can be readily accessed in Power BI Reports and Real-Time Dashboards, both of which can incorporate alerts triggered by specific data conditions.

Alerts can also be configured directly within non-table visualizations on Real-Time Dashboards during editing mode. These alerts notify you through Microsoft Teams or email when a predefined threshold is met.

Kusto Query Language (KQL)

Kusto Query Language (KQL) is a declarative query language designed to analyze and extract insights from structured, semi-structured, and unstructured data. Its primary focus on efficient large-scale log data searching makes it well-suited for cloud-based data analytics.

We’ll delve into basic KQL syntax later in this module. For now, consider these key benefits of KQL capabilities in Microsoft Fabric:

Efficient data exploration and analysis, enabling work with diverse data sources and versatile visualization options.
Reproducible analyses through Kusto kernel notebooks that capture code, results, and context.
Improved DevOps troubleshooting with notebooks and runbooks detailing issue mitigation using telemetry data.
Enriched DevOps flow by adding KQL files and notebooks to Git repositories and CI/CD pipelines.
Guidance and assistance in building search queries, thanks to the KQL editor’s error identification and hints.
Effortless handling of complex queries pasted from other sources.
Efficient data filtering, presentation, and aggregation using readable and author-friendly operators and functions.

Understand KQL Database and Tables

Let’s delve into the core elements that power Real-Time Intelligence in Microsoft Fabric.

Real-Time Intelligence Core Components

Event House: Think of event houses as highly efficient data libraries. They empower organizations to manage and analyze massive volumes of data rapidly, especially when timely insights are critical. It’s akin to a supercharged database capable of handling continuous data streams from diverse sources and in various formats. Designed to scale seamlessly alongside your project’s data needs, event houses ensure smooth operations without unnecessary resource consumption.
KQL Database: A Kusto database is a high-level entity that houses a collection of tables, stored functions, materialized views, shortcuts, and datastreams.Note: You can continue creating and using standalone KQL databases while event houses are in preview. Post-preview, KQL databases will be managed within the event house as the overarching management layer.
KQL Queryset: This tool allows you to execute queries, view, and manipulate query results on data residing in your KQL database. KQL Queryset lets you save queries for future use or export and share them with others. It employs the Kusto Query Language (KQL) for query creation and also supports T-SQL and some T-SQL functions. For more information on KQL, refer to the “Kusto Query Language overview.”
Real-Time Dashboards: Imagine a dashboard as a personalized control panel on your computer or gaming console. Each section, or “tile,” presents distinct information like scores, health levels, or maps in a game. These tiles are configured to display specific data and can be arranged across multiple pages for organization. You have the flexibility to customize these tiles to show different data or alter their visual appearance.Furthermore, you can seamlessly export complex data queries from KQL directly into these tiles as visualizations. These dashboards go beyond aesthetics; they simplify and accelerate data exploration. Think of it as a high-performance gaming rig, allowing you to switch between screens and data without lag, providing a smooth and integrated experience.
Eventstream: Consider event streams in Microsoft Fabric as a powerful tool for handling live data without the need for coding. It’s like a sophisticated funnel that automatically collects, transforms, and routes data to different destinations. When you configure an eventstream in the system, you’re essentially creating a mini-factory for real-time data processing. You define the data source, destination, and any necessary transformations along the way. It’s an integral part of Real-Time Intelligence, which focuses on deriving insights from data as it unfolds.

KQL Database Objects

Like many databases, the KQL Database encompasses various objects to cater to your data storage, streaming, and querying needs, ultimately supporting your downstream decision-making systems. From the event house landing page, you can select any database and navigate to it for data exploration, adding elements, and more.

Table: A table is a schema entity comprising a structured set of columns and rows of data. It has a well-defined schema (an ordered list of column name and data type pairs). You can create a new table using the .create table command, view the table schema with .show table, and ingest data into it using the .ingest command.
Function: A function is a schema entity that encapsulates a subquery expression, allowing invocation from other KQL queries. Stored functions have a name, an optional parameter list, and a body containing the subquery expression. You can create a new stored function with .create function and view existing ones using .show functions.
Materialized View: A materialized view is a schema entity that stores precomputed query results for faster retrieval. It has a name, optional parameters, and a body with the query expression. You can create a new materialized view with .create materialized-view and list existing ones using .show materialized-views.
Datastream: A datastream represents all attached KQL event streams connected to the KQL database.

By understanding these core components and objects within the KQL Database, you’re laying a solid foundation for working with Real-Time Intelligence in Microsoft Fabric.

This blog post is based on information and concepts derived from the Microsoft Learn module titled “Get started with data science in Microsoft Fabric.” The original content can be found here:
https://learn.microsoft.com/en-us/training/modules/get-started-data-science-fabric/