Explore the unparalleled capabilities of Microsoft Fabric’s Real-Time Intelligence suite, designed to help you build robust analytical solutions for live data streams. This comprehensive guide will walk you through the process of ingesting, analyzing, and visualizing real-time stock market data, culminating in the configuration of dynamic, actionable alerts.
Setting Up Your Microsoft Fabric Environment
1. Establishing Your Workspace:
To begin your journey with real-time analytics in Fabric, you’ll first need a dedicated workspace.
* Navigate to the Microsoft Fabric home page: `https://app.fabric.microsoft.com/home?experience=fabric`.
* Sign in using your Microsoft Fabric credentials.
* On the left-hand menu, select “Workspaces.”
* Create a new workspace, ensuring that you choose a licensing mode that includes Fabric capacity. This workspace will serve as the central hub for all your data engineering and analytical activities.
Ingesting Real-Time Stock Data
2. Creating an Eventstream:
The next step involves setting up an eventstream to capture live data from a streaming source.
* From the left-hand menu, select the “Real-Time hub.” This hub is your primary interface for discovering and managing streaming data sources.
* Within the “Connect to” section of the real-time hub, choose “Data sources.”
* Locate and select the “Stock market sample data source,” then click “Connect.”
* In the subsequent connection wizard, name your source “stock” and modify the default eventstream name to “stock-data.” The system will automatically name the associated stream “stock-data-stream.”
* Proceed through the wizard to finalize the creation of both the data source and the eventstream.
* Once created, open the eventstream. You should see the “stock” source and “stock-data-stream” displayed on the design canvas.
Storing Your Streaming Data with an Eventhouse
3. Setting Up Your Eventhouse:
While an eventstream ingests data, an Eventhouse provides the necessary infrastructure to store and manage this captured data in a structured table format.
* On the left menu, select “Create.” Under the “Real-Time Intelligence” section, select “Eventhouse” and provide it with a unique, descriptive name.
* Close any initial tips or prompts to reveal your new, empty Eventhouse.
* Within your Eventhouse, you’ll observe a KQL database that shares the same name. This database is where you’ll create tables for your real-time data.
* Select the database. You’ll notice an associated queryset containing sample KQL queries, though it’s currently empty as there are no tables yet.
* To resolve this, select “Get data” from the main page of your KQL database.
* For the data source, choose “Eventstream” followed by “Existing eventstream.”
* In the “Select or create a destination table” pane, create a new table and name it “stock.”
* Next, in the “Configure the data source” pane, select your workspace and the “stock-data” eventstream. Name this connection “stock-table.”
* Utilize the “Next” button to navigate through the steps, inspect the data, and ultimately “Finish” the configuration. Close the configuration window to view your Eventhouse now containing the “stock” table.
* To confirm the successful connection, return to the “Real-Time hub” via the left menu. For the “stock-data-stream,” select the ellipsis (…) and choose “Open eventstream.” The eventstream will now clearly indicate a destination for the stream.
* Select the destination on the design canvas. If a data preview isn’t immediately visible, click “Refresh” to load it.
Querying and Analyzing Live Stock Data
4. Exploring Your Data:
With your eventstream actively feeding real-time stock data into a table within your KQL database, you can now begin to query and analyze this information.
* From the left menu, select your Eventhouse database.
* Open the queryset associated with your database.
* In the query pane, modify the first example query to retrieve the first 100 rows of data:
kql
stock
| take 100
* Highlight this query and run it to view the results.
* Review the output, then modify the query to calculate the average price for each stock symbol over the last 5 minutes:
kql
stock
| where ["time"] > ago(5m)
| summarize avgPrice = avg(todecimal(bidPrice)) by symbol
| project symbol, avgPrice
* Highlight and execute this refined query to see the average prices. Run it again after a few seconds to observe how the averages dynamically change as new data flows into the table.
Visualizing Data with a Real-Time Dashboard
5. Creating Your Stock Dashboard:
A live stream of data is most impactful when visualized. You can create a real-time dashboard to monitor your stock data.
* In the query editor, select the KQL query you used to fetch the average stock prices for the past five minutes.
* On the toolbar, select “Save to dashboard.” Pin the query to a new dashboard with the following settings:
* Dashboard name: Stock Dashboard
* Tile name: Average Prices
* Create and then open your new dashboard.
* At the top-right of the dashboard interface, switch from “Viewing” mode to “Editing” mode.
* Select the “Edit” (pencil) icon for the “Average Prices” tile.
* In the “Visual formatting” pane, change the “Visual” type from “Table” to “Column chart.”
* At the top of the dashboard, select “Apply changes” to update and view your new live visualization, now presented as a dynamic column chart.
Setting Up Real-Time Alerts
6. Configuring an Activator Alert:
Microsoft Fabric’s Real-Time Intelligence includes Activator technology, enabling you to trigger actions based on real-time events. Let’s set up an alert for significant changes in average stock prices.
* Within the dashboard window displaying your stock price visualization, select “Set alert” from the toolbar.
* In the “Set alert” pane, configure your alert with these settings:
* Run query every: 5 minutes
* Check: On each event grouped by
* Grouping field: symbol
* When: avgPrice
* Condition: Increases by
* Value: 100
* Action: Send me an email
* Save location: Your workspace, and choose “Create a new item” with a unique name of your choice.
* Create and save the alert. You can then close the confirmation pane.
* Navigate to your workspace via the left menu (saving any unsaved dashboard changes if prompted).
* On your workspace page, you’ll find the various items created during this project, including the activator for your alert.
* Open the activator. Under the avgPrice node, select the unique identifier for your specific alert.
* View its “History” tab. Initially, there might be no data if the alert hasn’t been triggered yet. If the average stock price for any symbol increases by more than 100, the activator will send you an email notification, and the event will be logged in this history.
Conclusion
This project has provided a comprehensive walkthrough of Microsoft Fabric’s Real-Time Intelligence capabilities, demonstrating everything from initial data ingestion and structured storage to dynamic data visualization and automated, event-driven alerting. This foundational knowledge can be extended with additional data transformations and more complex analytical pipelines to build highly sophisticated real-time solutions tailored to specific business needs.