Azure Data Explorer - Security Data Warehouse: A Reference Implementation

The challenge of efficiently storing and analyzing massive volumes of security telemetry has long been a pain point for security operations teams. Today, I'm pleased to share a reference implementation that addresses this challenge: PUBLIC-adx-basic, a complete solution for deploying and configuring Azure Data Explorer (ADX) as a security data lake.

Two Security Functions, Two Tools

As I've previously written, security operations teams need to address two distinct functions when dealing with logging and alerting:

1. Alert Management: Microsoft Sentinel excels here, providing robust correlation and alert handling capabilities.
2. Data Hunting and Retention: This is where Azure Data Explorer shines, offering unparalleled performance for massive datasets.

While Sentinel was never designed to handle petabyte-scale data retention and hunting, ADX provides the columnar database technology needed for these workloads. This example project bridges the gap between these two critical security functions.

Azure Data Explorer provides a cost effective Data Warehouse for all Enterprise Security signals, including complete backup of all Sentinel data.  It allows for indefinate preservation of Security data and in doing so, allows for a greater depth of data analysis and alerting that would be possible with Sentinel alone.

Key Technical Components

The example architecture follows a modular approach with several key components:

1. Azure Data Explorer Cluster: The core infrastructure providing compute and storage
2. Security Database: A dedicated database with optimised table structures
3. ASIM Parsers: KQL functions that normalise data from typical "_CL" tables
4. Table Definitions: Extensible schemas designed for security data optimisation
5. Deployment Automation: Scripts and Bicept that handle the complete deployment process

This architecture ensures alignment with modern security data practices while leveraging ADX's performance capabilities.

Why This Matters

As security teams move toward working with terabytes or petabytes of data per day, traditional SIEM solutions struggle with long-term retention and high-performance hunting. ADX addresses these challenges by providing:

• Cost-effective storage for years or decades of full-fidelity logs
• Query performance across years of online data measured in seconds, not minutes or hours
• Seamless integration with the Microsoft security ecosystem
• Standardised tables schemas integrated with Microsoft's ASIM Parsers
• Identical KQL query environment to Microsoft Sentinel supporting out-of-the-box parsers and solutions.
• All data directly accessible from Sentinel via Sentinel Functions.
   

Looking Forward

This project is intended as a starting point for organisations to begin experimenting with ADX as a security data warehouse. While not actively developed at this stage, the reference implementation provides a solid foundation that can be extended to meet and SIEM data storage needs.  It should be enough for any SOC or Azure Engineers to rapidly get a functional Data Warehouse deployed while providing enough links to pages in this blog to show how to add new tables and connect data streams through Event Hubs. 
 

If you're interested in exploring this approach further, I encourage you to: review the complete documentation including the architecture overview.