At under $3,000 AUD monthly, Azure Data Explorer can ingest and maintain 1TB of security data daily for 18 months — a cost-efficiency most security professionals overlook..
If you ask your SOC team what data they need to do their job, the answer is always "all of it"! It turns out that the Australian Signals Directorate now agrees with it's recommendation that Australian organisations adopt the M-21-31 log retention directives as best practice SIEM management.
Azure Data Explorer (ADX) is the perfect companion for Microsoft Sentinel as a realisation of a Big Data SIEM. It allows the SOC to utilise KQL and their library of hunting queries seemlessly against an enormous online archive of security signals. It provides an opportunity for Blue teams to have complete environment awareness that used to be reserved for major banks and National Defence.
This blog is intended to provide some guidance on how to estimate SIEM costs when planning to implement Azure Data Explorer as an enterprise SIEM archive.
ADX Data Pricing
As always, the first place to start with Azure pricing estimates is the Azure Pricing Calculator. This will always be slightly subjective as the level of compression achieved is determined by the log types in each environment. I've found that a compression ration of 14-to-1 seems to be a reasonable average estimate with Security logs but the actual range of compression ratio could be as great as 30 with some commonly used Security logs.
- Security Teams don't require much in the way of "Hot Cache" so set the retention as low as possible in the calculator.
- M-21-31 requires agencies plan for "12 Months Active Storage" and "18 Months Cold Data Storage". Azure Data Explorer is so incredibly efficient as a columar database, even using Parquet as a blob file format becomes too expensive to export hundreds of GB a day to "cold" blob storage. The most economical planning for M-21-31 compliance is to hold 30 months of data hot in ADX. The chart at the top of this page shows a current estimate of a TB a day online for 30 months is still under $4,000 AUD a month.
- We should plan for at least one other ADX cluster to act as an Active / Active replica of the Security Data Archive.
Event Hub Pricing
The primary method for ingesting data into Azure Data Explorer is through the use of Event Hub Namespaces.
An enterprise environment is going to want to use a Premium SKU Event Hub and the number of processing units will scale based on the number of distinct logs types being collected and the volume. As a rule of thumb, one processing unit should meet the needs of an organisation ingesting to about 1TB a day with two ADX clusters as consumers of the Event Hub data.
More can be read about the Processing Unit limits for Event Hbs here.
Data Externalisation
One of the enormous benefits of using Event Hubs is the ability to have a secondary ADX cluster in an external Tenant as a consumer of data as it is ingested in the Primary Tenant.
A concern Security Architects have always had over the shift of SIEM to cloud has been the threat to integrity of log preservation should privileged Entra ID accounts become compromised. Using a secondary ADX for externalised data collection in an external tenant mitigates this nightmare scenario, especially if the external Tenant is dedicated to Security and is granted Lighthouse permissions to Sentinel in the monitored tenant.
Data Pipelines
Big Data teams are well versed in the value of data pipelines... Security teams moving into the Big Data space should be too! Organisations that wish to send large volumes of data to ADX and Event Hubs from outside of Azure will discover limitations with the AMA Client that make a Data Pipeline consideration critical. In recent times products like Cribl and Logstash have been heavily used in Australia.
A newcomer to this space that is specifically designed for Sentinel and Azure Data Explorer is VirtualMetric's DataStream. DataStream deserves mention specifically for it's focus on Sentinel integration. It iis exceptionally well priced and will allow smaller organisations to start using Azure Data Explorer event without using a full-blown EventHub Premium architecture.
Rough Order of Magnitude Estimate (AUD)
Scoping for a Premium SKU Event Hub and two Azure Data Explorer clusters to support M-21-31 SIEM requirements will result in a monthly SIEM cost of about $10,000 Australian Dollars a month (roughly $6,000 USD at today's exchange rate).
Some points to note:
- Azure Data Explorer compliments Sentinel as the Big Data archive while Sentinel continues to fulfil its role as the Incident management system and recipient of High Value alert data.
- Azure Data Explorer will act as the archive of all data in Sentinel as well as the primary repository of High Volume lower value data streams from the enterprise.
- The implementation of a Security Data Warehouse shifts the evolution of SOC toward full data engineering. Quite rapidly, SOC Managers will wish to extend SOC capabilities to include Machine Learning Workspace for modelling.
Engineers keen to experiment with Azure Data Explorer further can find an example project with documentation referenced at: Azure Data Explorer - Security Data Warehouse: A Reference Implementation
- Log in to post comments