The Real-Time Intelligence Architecture
Sub-second dashboard updates require moving away from traditional scheduled batch refreshes toward event-driven streaming pipelines. In Microsoft Fabric, Real-Time Intelligence provides the infrastructure to ingest, process, and visualize millions of events per second with minimal latency.
[IoT Sensors / Kafka / Webhooks]
│
▼ (Sub-second Event Ingestion)
[Fabric Eventstreams] (Transform, Filter, Route)
│
▼ (Continuous Appends)
[KQL Database] (Columnar Ingestion with Native Anomaly Detection)
│
├── DirectQuery / KQL Visuals ──► [Real-Time Dashboards] (Sub-second telemetry)
└── Eventhouse Mirroring ───────► [OneLake Delta Tables] (Long-term historic analytics)Querying High-Throughput Streams with Kusto (KQL)
Kusto Query Language (KQL) is optimized for ad-hoc exploration of structured and semi-structured time-series data. Unlike relational SQL, which processes data row-by-row or requires complex windowing functions, KQL offers specialized time-series operators that evaluate rolling windows and anomalies natively:
// Aggregate sensor telemetry into 5-minute rolling windows with anomaly detection
DeviceTelemetry
| where Timestamp >= ago(2h)
| summarize
AvgTemperature = avg(Temperature),
MaxPressure = max(Pressure),
EventCount = count()
by DeviceId, bin(Timestamp, 5m)
| make-series
Trend = avg(AvgTemperature) default=0
on Timestamp step 5m by DeviceId
| extend (Anomalies, Score, Baseline) = series_decompose_anomalies(Trend, 2.5)
| render timechart with (title="Telemetry Anomalies across 5-Minute Windows")Handling Out-of-Order Events and Watermarks
Network latency and intermittent edge connectivity inevitably cause telemetry events to arrive out of chronological order. Streaming engines handle this by establishing watermark thresholds. A watermark defines how long the system will wait for delayed events before closing a time window and computing final aggregations.
For financial or billing data where exact accuracy is mandatory, real-time streams must be complemented by an end-of-day batch reconciliation job. The streaming pipeline provides immediate operational visibility, while the batch job overwrites the preliminary aggregations with immutable, verified accounting ledger records in OneLake.
