In my previous blog posts over Azure Data Explorer, I suggested the need to alter the Azure Monitor schema with ADX to include a Timestamp field.
Being able to accurately correlate logs between different systems based on event times is critical for Security Operations. Anyone in a SOC team will have learned that the standard TimeGenerated field used with Log Analytics doesn't represent the time of an event - it represents the time a message was received by Log Analytics.
TimeGenerated is important for being able to run iterative logic against incoming data. It's how the scheduling of Analytics rules works in Sentinel and it's useful for running function apps with processing rules against ADX.
Different tables preserve the event time in dedicated fields. Syslog uses EventTime and the Defender suite use Timestamp.
I had a great example last year with how latency and infrastructure issues completely skewed TimeGenerated in Sentinel from the actual activity time on a source event.
Syslog and Defender tables do preserve Event time while with Common Event Format, the actual syslog timestamp of the event is completely dropped from the record. There is no reliable consistency or for determining an event time for the other 300 odd standard Azure Monitor tables.
Microsoft's hidden time fields
“_TimeReceived” is a hidden field that isn’t shown within the Log Analytics schemas online nor through the Log Analytics blade but you can search and query against it with LogAnalytics.
It's a useful field to be aware of when trying to understand latency and it's idea for using recursive functions against based on time as "TimeGenerated" event times can arrive 30 minuted behind when they occurred due to batching and lag.
Interesting as this is, the lack of a consistent field for event times causes problems for Security Operations.
Timestamp in ADX tables
As we are bringing in enormous amounts of data from different Enterprise systems into ADX we do have the flexibility to amend the Azure Monitor Schemas in a way that we can't with Log Analytics. We can ensure that we always standardise the original event source time for SOC analysts using ADX for hunting.
Although I have been an advovate for strictly adopting the official Azure Monitor table schemas with the ADX table I create, this need for consistency is my exception. In keeping with Microsoft Defender, I decided to duplicate Timestamp with all ADX tables used for hunting.
The benefit for the SOC to have a consistently named field with actual event time is significant and something I would urge anyone using ADX as a security tool to consider.
- Log in to post comments