This example of using REST to populate Syslog is based off a Microsoft guide for using Logstash:
https://learn.microsoft.com/en-us/azure/sentinel/connect-logstash-data-connection-rules#create-a-sample-file
There are only a few tables we can write data back to Sentinel using a DCR – but it’s all the big tables that I want in ADX. This will give me the ability to run queries across enormous data sets and send events of interest to Sentinel for alerting. These tables are:
The PowerShell script below demonstrates the use of REST and a POST. You can use it as a guide for whatever method you are using for getting an Azure authorised header.
$tenantId = "531fefc9-2ea0-4b49-ac25-eaba5d16d2c8" #Tenant ID the data collection endpoint resides in
$appId = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" #Application ID created and granted permissions
$appSecret = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" #Secret created for the application
#information needed to send data to the DCR endpoint
$dcrImmutableId = "dcr-e3c03e1c6b38453da38fd57649720908"; #the immutableId property of the DCR object
$dceEndpoint = https://dce-ase-security-sqxq.australiasoutheast-1.ingest.monitor.azure.com;
$streamDeclaration = "Custom-SyslogStream"
# Get an authorised Azure Header
$authHeader = Get-Header -scope monitor -Tenant "laurierhodes.info" -AppId $AppId -secret $AppSecret
$eventData = @"
[
{
"ls_timestamp": "2024-01-24T02:26:35Z",
"message": "syslog test 2.26pm",
"facility_label": "auth",
"severity_label": "alert",
"host": "DESKTOP-VR7UAGD",
"logsource": ""
}
]
"@
$uri = "$($dceEndpoint)/dataCollectionRules/$($dcrImmutableId)/streams/$($streamDeclaration)?api-version=2021-11-01-preview"
$result = Invoke-RestMethod -Uri $uri -Method 'POST' -Body $eventData -Headers $authHeader
The Data Collection Rule is here. The Data Collection Endpoint Id and Sentinel workspace identifiers will need to be customised for your environment.
For anyone not comfortable in deploying Azure services in REST, Bicep is the best alternative.
{
"properties": {
"dataCollectionEndpointId": "/subscriptions/2be53ae5-6e46-47df-beb9-6f3a795387b8/resourceGroups/rg-ase-datacollection/providers/Microsoft.Insights/dataCollectionEndpoints/dce-ase-security",
"streamDeclarations": {
"Custom-SyslogStream": {
"columns": [
{
"name": "ls_timestamp",
"type": "datetime"
}, {
"name": "timestamp",
"type": "datetime"
},
{
"name": "message",
"type": "string"
},
{
"name": "facility_label",
"type": "string"
},
{
"name": "severity_label",
"type": "string"
},
{
"name": "host",
"type": "string"
},
{
"name": "logsource",
"type": "string"
}
]
}
},
"dataSources": {},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "/subscriptions/2be53ae5-6e46-47df-beb9-XXXXXXXXXXXX/resourceGroups/Sentinel/providers/microsoft.operationalinsights/workspaces/asesentinel6",
"workspaceId": "2b567116-9c70-416e-ba3e-XXXXXXXXXXXX",
"name": "myTargetWorkspace"
}
]
},
"dataFlows": [
{
"streams": [
"Custom-SyslogStream"
],
"destinations": [
"myTargetWorkspace"
],
"transformKql": "source | project TimeGenerated = ls_timestamp, EventTime = todatetime(timestamp), Computer = logsource, HostName = logsource, HostIP = host, SyslogMessage = message, Facility = facility_label, SeverityLevel = severity_label",
"outputStream": "Microsoft-Syslog"
}
]
},
"location": "australiasoutheast",
"id": "/subscriptions/2be53ae5-6e46-47df-beb9-XXXXXXXXXX/resourceGroups/Sentinel/providers/Microsoft.Insights/dataCollectionRules/dcr-write_to_syslog",
"name": "dcr-write_to_syslog",
"type": "Microsoft.Insights/dataCollectionRules"
}Once the Data Collection Rule is deployed, it needs to have assigned the Monitoring Metrics Publisher role assigned to the Application that will be used to submit messages.
The Data collection Rule starts with a custom log stream decloration which defines the fild names expected in a POST of JSON data. Later, the transformKql statement projects the initial fields to match the names and field types of the destination table. You can have issues using the destination field names in the initial custom log stream decloration as thay can be reserved function names. In that scenario, we just Project whatever custom field names we have used initially to the right field names within the Sentinel table schema as a final action.
An example of writing to the SecurityEvent table is below:
{
"properties": {
"immutableId": "dcr-6a7d406f2c8e4f7abcd98d305ae5f7a9",
"dataCollectionEndpointId": "/subscriptions/2be53ae5-6e46-47df-beb9-6f3a795387b8/resourceGroups/rg-ase-datacollection/providers/Microsoft.Insights/dataCollectionEndpoints/dce-ase-security",
"streamDeclarations": {
"Custom-SecurityEvent_CL": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "SourceSystem",
"type": "string"
},
{
"name": "Type",
"type": "string"
},
{
"name": "Account",
"type": "string"
},
{
"name": "AccountType",
"type": "string"
},
{
"name": "Computer",
"type": "string"
},
{
"name": "EventSourceName",
"type": "string"
},
{
"name": "Channel",
"type": "string"
},
{
"name": "Task",
"type": "int"
},
{
"name": "Level",
"type": "string"
},
{
"name": "EventData",
"type": "string"
},
{
"name": "EventID",
"type": "int"
},
{
"name": "Activity",
"type": "string"
},
{
"name": "AuthenticationPackageName",
"type": "string"
},
{
"name": "ElevatedToken",
"type": "string"
},
{
"name": "ImpersonationLevel",
"type": "string"
},
{
"name": "IpAddress",
"type": "string"
},
{
"name": "IpPort",
"type": "string"
},
{
"name": "LmPackageName",
"type": "string"
},
{
"name": "LogonGuid",
"type": "string"
},
{
"name": "LogonProcessName",
"type": "string"
},
{
"name": "LogonType",
"type": "int"
},
{
"name": "LogonTypeName",
"type": "string"
},
{
"name": "TargetAccount",
"type": "string"
},
{
"name": "TargetDomainName",
"type": "string"
}
]
}
},
"dataSources": {},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "/subscriptions/2be53ae5-6e46-47df-XXXXXXXXXXXX/resourceGroups/Sentinel/providers/microsoft.operationalinsights/workspaces/asesentinel",
"workspaceId": "2b567116-9c70-416e-ba3e-XXXXXXXXXXXX",
"name": "Sentinel-SecurityEvent"
}
]
},
"dataFlows": [
{
"streams": [
"Custom-SecurityEvent_CL"
],
"destinations": [
"Sentinel-SecurityEvent"
],
"transformKql": "source | project TimeGenerated, LogonGuid=toguid(LogonGuid), SourceSystem, Type, Account, AccountType, Computer, EventSourceName, Channel, Task, Level, EventData, EventID, Activity, AuthenticationPackageName, ElevatedToken, ImpersonationLevel, IpAddress, IpPort, LmPackageName, LogonProcessName, LogonType, LogonTypeName, TargetAccount, TargetDomainName",
"outputStream": "Microsoft-SecurityEvent"
}
]
},
"location": "australiasoutheast",
"id": "/subscriptions/2be53ae5-6e46-47df-beb9-XXXXXXXXXXX/resourceGroups/Sentinel/providers/Microsoft.Insights/dataCollectionRules/write-to-SecurityEvent",
"name": "write-to-SecurityEvent",
"type": "Microsoft.Insights/dataCollectionRules"
}The script format I'm using to populate the SecurityEvent table is different but ultimately using the same REST methods as the previous Syslog script.
<#
Variables
#>
# Credentials for Azure Active Directory (AAD) to fetch an authentication token
$tenantId = "531fefc9-2ea0-4b49-ac25-XXXXXXXXXXXX" # Unique Tenant ID for endpoint authentication
$appId = "aa73b052-6cea-4f17-b54b-XXXXXXXXXXXX" # Registered Application ID with necessary permissions
$appSecret = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" # Application secret for secure access
# Data Collection Rule (DCR) and Endpoint (DCE) details for data submission
$dcrImmutableId = "dcr-6a7d406f2c8e4f7abcd98d305ae5f7a9"; # Immutable ID of the DCR
# URL for the DCE to receive the data
$dceEndpoint = "https://dce-ase-security-sqxq.australiasoutheast-1.ingest.monitor.azure.com";
<#
First, aquire a token for writing to the Data Collection Endpoint
#>
Add-Type -AssemblyName System.Web
$scope = [System.Web.HttpUtility]::UrlEncode("https://monitor.azure.com/.default")
$tokenRequestBody = "client_id=$appId&scope=$scope&client_secret=$appSecret&grant_type=client_credentials";
$tokenRequestHeaders = @{"Content-Type"="application/x-www-form-urlencoded"};
$tokenUri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
$bearerToken = (Invoke-RestMethod -Uri $tokenUri -Method "POST" -Body $tokenRequestBody -Headers $tokenRequestHeaders).access_token
<#
Example data file
#>
$currentTime = Get-Date ([datetime]::UtcNow) -Format O
$sampleEventData = @"
[ {
"TimeGenerated": "$($currentTime)",
"Account": "NT AUTHORITY\\SYSTEM",
"AccountType": "Machine",
"Computer": "LAURIE-PC",
"EventSourceName": "Microsoft-Windows-Security-Auditing",
"Channel": "Security",
"Task": 12544,
"Level": 0,
"EventData": "",
"EventID": 4624,
"Activity": "4624 - An account was successfully logged on.",
"AuthenticationPackageName": "Negotiate",
"ElevatedToken": "%%1842",
"ImpersonationLevel": "%%1833",
"IpAddress": "-",
"IpPort": "-",
"LmPackageName": "-",
"LogonGuid": "00000000-0000-0000-0000-000000000000",
"LogonProcessName": "MyExampleProcess",
"LogonType": 5
}]
"@
<#
Transmit Data to the Data Collecvtion Endpoint
#>
$Body = $sampleEventData
$Header = @{"Authorization"="Bearer $bearerToken";"Content-Type"="application/json"}
$Uri = "$dceEndpoint/dataCollectionRules/$dcrImmutableId/streams/Custom-SecurityEvent_CL?api-version=2021-11-01-preview"
$Response = Invoke-RestMethod -Uri $Uri -Method "POST" -Body $Body -Headers $Header
Once the Data Collection Rule has been sucessfully deployed, I can take the Immutable Id and Data Collection endpoint Id from the JSON view of the DCR.
As you can see, my payload is written to Sentinel as expected.
- Log in to post comments