There can be a number of reasons for wanting to backup Azure (or Office 365) to GitHub. As an increasing number of SaaS services (like Microsoft Sentinel) are designed for being configured and deploying Azure services through the console, traditional CI/CD code promotion doesn't work.
For some years I've been backing up my Azure subscription to Github using automated workflows. It ensures that I can compare changes in my subscription over time and by using MarkDown I can look through backups to reference previous versions of KQL queries.
Directly through Github I can select complete backups to reference:
An example of MarkDown being used to show KQL is below.
There are a few parts for getting this working. Firstly is the schedule and powershell backup script in Github. These reside in a "/.github/workflows/" directory structure off the root of the subscription.
The YAML GitHub file is listed below. It's purely running my Powershell script on a schedule.
# This is a basic workflow to help you get started with Actions
name: generate-backup
# Controls when the workflow will run
on:
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
schedule:
# * is a special character in YAML so you have to quote this string
- cron: '30 17 * * *'
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "backup"
backup:
# The type of runner that the job will run on
runs-on: ubuntu-latest
env:
resourceGroupName: 'sentinel'
tenant: 'laurierhodes.info'
subscription: 'xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx'
scope: 'azure'
workspaceName: 'asesentinel6'
directory: '${{ github.workspace }}'
cloudEnv: 'AzureCloud'
appid: ${{ secrets.SENTINEL_APP_ID }}
appsecret: ${{ secrets.SENTINEL_APP_SECRET }}
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v2
- name: Backup Azure
shell: pwsh
run: |
pwsh -File '${{ github.workspace }}/.github/workflows/generate-backup.ps1'
- name: Add & Commit
# You may pin to the exact commit or the version.
# uses: EndBug/add-and-commit@d77762158d703e60c60cf5baa4de52697d1414a3
uses: EndBug/add-and-commit@v8
with:
author_name: Laurie Rhodes
author_email: username@domainname.com
message: "Standard Backup"
add: "*"
cwd: '.'The script below has been running since 2021, you would references GitHub secrets differently now but it shows how the daily backup works. There is a lot more going on than just backing up Azure. I'm importing a custom Powershell Module with a number of functions - these functions are documented here as well:
PowerShell Function - Microsoft Cloud Tokens | Laurie Rhodes' Info
Programmatically retrieving ‘latest’ Azure REST API versions | Laurie Rhodes' Info
Understanding how to programatically retrieve the latest API verstion for an Azure object is critical. I'd urge everyone to read that blog before trying to understand how this backup works.
This is more complex than it needs to be for just a backup because I've used this as a solution for different problems since it's first use. I'm also using the JSON for natively deploying to Azure directly off API's (no bicep or ARM templates) but that forces me to remove a number of ReadOnly properties from backed up Azure objects. You'll see a large part of the script is a "Clean-AzureObject" function that served that purpose. Most people who just want a backup for reference could delete that entire function to make the script more readable.
I also had to account for particularly long file names with some Azure objects and test each object for Diagnostings being enabled on them for monitoring. This is all probably legacy now with major changes with monitoring over the years. I also have different queries for backing up Azure Policies and permission assignemnts.
#! /bin/pwsh
<#
Purpose: Exports all objects from a subscription to a local directory
#>
$BackupDir = $Env:directory
$resourcegroup = $Env:resourceGroupName
$subscription = $Env:subscription
$tenant = $Env:tenant
$scope = $Env:scope
$appid = $Env:appid
$secret = $Env:appsecret
#$DebugPreference = 'Continue'
# Set Resourcegroup to null if backing up all of Azure
$resourceGroup = $null
#Used for testing outside of github
$rootDir = "C:\GitHub\Sentinel-as-Code"
if ($Env:directory){$rootDir = $Env:directory}
#$ScriptDir = Split-Path $script:MyInvocation.MyCommand.Path
# Determine if script is being run on linux or windows by the direction
# of slashed in the PATH statement
if ((Get-ChildItem Env:PATH).Value -Match '/'){ $OS='linux'}else{$OS='win'}
# Create a slash variable based on the OS type of the host
if($OS -eq 'win' ) { $slash = "\" }
if($OS -eq 'linux'){ $slash = "/"}
$ScriptDir = Split-Path $script:MyInvocation.MyCommand.Path
$BackupDir = "$($rootDir)$($slash)json"
$ReportDir = "$($rootDir)$($slash)reports"
$ScriptDir = Split-Path $script:MyInvocation.MyCommand.Path
$ModuleDir = "$($ScriptDir)$($slash)modules"
Import-Module "$($moduledir)$($slash)AZRest$($slash)AZRest.psm1" | write-debug
function Clean-AzureObject(){
<#
Purpose: Parsing function to remove read only properties from an object so the exported defintion may be used for redeployment
Not neceassry for a basic audit.
Also provides the ability to perform additional GET against specific object types
#>
[CmdletBinding()]
param(
[Parameter(mandatory=$true)]
[string]$azobjectjson,
[Parameter(mandatory=$true)]
[string]$BackupDir,
[Parameter(mandatory=$true)]
[Hashtable]$AzAPIVersions,
[Parameter(mandatory=$true)]
[Hashtable]$authHeader
)
$azobject = ConvertFrom-Json $azobjectjson
# Remove common properties
if ($azobject.PSObject.properties -match 'etag'){$azobject.PSObject.properties.remove('etag')}
switch($azobject.type){
"Microsoft.ApiManagement/service" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('CreatedAtUTC')
}
"Microsoft.Automation/AutomationAccounts" {
($object.properties).PSObject.properties.remove('state')
($object.properties).PSObject.properties.remove('creationTime')
($object.properties).PSObject.properties.remove('lastModifiedBy')
($object.properties).PSObject.properties.remove('lastModifiedTime')
}
"Microsoft.Automation/AutomationAccounts/Runbooks" {
($object.properties).PSObject.properties.remove('creationTime')
($object.properties).PSObject.properties.remove('lastModifiedBy')
($object.properties).PSObject.properties.remove('lastModifiedTime')
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Cache/Redis" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Compute/virtualMachines/extensions" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Compute/virtualMachines" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('vmId')
($object.identity).PSObject.properties.remove('principalId')
($object.identity).PSObject.properties.remove('tenantId')
#
($object.properties.osProfile).PSObject.properties.remove('requireGuestProvisionSignal')
#Disks will be managed separately & before the VM. Disk option attach will need to be used.
#more work will be needed to accomodate data disks
($object.properties.storageProfile.osDisk.managedDisk).PSObject.properties.remove('id')
# # Handle each vm extension
# For ($i=0; $i -le ($object.resources.Count -1); $i++) {
# $null = Invoke-azobjectbackup -Id $object.resources[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
# }
# $object.resources=@()
}
"Microsoft.Compute/disks" {
($object.properties).PSObject.properties.remove('timeCreated')
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('uniqueId')
($object.properties).PSObject.properties.remove('diskSizeBytes')
}
"Microsoft.ContainerInstance/containerGroups" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.DataFactory/factories" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('createTime')
}
"Microsoft.DesktopVirtualization/applicationgroups" {
($object.properties).PSObject.properties.remove('objectId')
}
"Microsoft.DocumentDB/databaseAccounts" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.systemData).PSObject.properties.remove('createdAt')
# Handle each Private Endpoint Connection as separate objects
For ($i=0; $i -le ($object.properties.PrivateEndpointConnections.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.PrivateEndpointConnections[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
$object.properties.PrivateEndpointConnections=@()
}
"Microsoft.EventHub" {
($object.properties).PSObject.properties.remove('createdAt')
($object.properties).PSObject.properties.remove('updatedAt')
}
"Microsoft.EventHub/clusters" {
($object.properties).PSObject.properties.remove('createdAt')
($object.properties).PSObject.properties.remove('updatedAt')
}
"Microsoft.EventHub/Namespaces" {
($object.properties).PSObject.properties.remove('createdAt')
($object.properties).PSObject.properties.remove('updatedAt')
($object.properties).PSObject.properties.remove('provisioningState')
# Handle each Private Endpoint Connection as separate objects
For ($i=0; $i -le ($object.properties.PrivateEndpointConnections.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.PrivateEndpointConnections[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
$object.properties.PrivateEndpointConnections=@()
}
"Microsoft.EventHub/Namespaces/PrivateEndpointConnections" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.HybridCompute/machines" {
($object.properties).PSObject.properties.remove('lastStatusChange')
($object.identity).PSObject.properties.remove('principalId')
}
"microsoft.insights/components" {
($object.properties).PSObject.properties.remove('CreationDate')
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Insights/scheduledqueryrules" {
($object).PSObject.properties.remove('systemData')
}
"Microsoft.Insights/workbooks" {
}
"Microsoft.KeyVault/vaults" {
$object.PSObject.properties.remove('systemData')
($object.properties).PSObject.properties.remove('provisioningState')
# Handle each Private Endpoint Connection as separate objects
For ($i=0; $i -le ($object.properties.PrivateEndpointConnections.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.PrivateEndpointConnections[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
if ($object.properties.PrivateEndpointConnections){
$object.properties.PrivateEndpointConnections=@()
}
}
"Microsoft.Logic/workflows" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('createdTime')
($object.properties).PSObject.properties.remove('changedTime')
($object.properties).PSObject.properties.remove('endpointsConfiguration')
($object.properties).PSObject.properties.remove('version')
}
"Microsoft.MachineLearningServices/workspaces" {
$object.PSObject.properties.remove('etag')
# ($object.identity).PSObject.properties.remove('principalId')
# ($object.identity).PSObject.properties.remove('tenantId')
($object.systemData).PSObject.properties.remove('createdAt')
($object.systemData).PSObject.properties.remove('createdBy')
($object.systemData).PSObject.properties.remove('createdByType')
($object.systemData).PSObject.properties.remove('lastModifiedAt')
($object.systemData).PSObject.properties.remove('lastModifiedBy')
($object.systemData).PSObject.properties.remove('lastModifiedByType')
($object.systemData).PSObject.properties.remove('provisioningState')
}
"Microsoft.Network/loadBalancers" {
$object.PSObject.properties.remove('etag')
($object.properties).PSObject.properties.remove('provisioningState')
# Handle each nat rule as separate objects
For ($i=0; $i -le ($object.properties.inboundNatRules.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.inboundNatRules[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
# ($object.properties.inboundNatRules[$i]).PSObject.properties.remove('etag')
# ($object.properties.inboundNatRules[$i].properties).PSObject.properties.remove('provisioningState')
}
# Handle each frontendIPConfigurations as separate objects
For ($i=0; $i -le ($object.properties.frontendIPConfigurations.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.frontendIPConfigurations[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
# Handle each backendAddressPools as separate objects
For ($i=0; $i -le ($object.properties.backendAddressPools.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.backendAddressPools[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
# Handle each loadBalancingRules as separate objects
For ($i=0; $i -le ($object.properties.loadBalancingRules.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.loadBalancingRules[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
# Handle each probes as separate objects
For ($i=0; $i -le ($object.properties.probes.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.probes[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
}
"Microsoft.Network/networkInterfaces" {
$object.PSObject.properties.remove('etag')
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('resourceGuid')
($object.properties).PSObject.properties.remove('macAddress')
# IP Configurations must exist in a Network interface for deployment
# Clean inplace
# duplicate with export to aid auditing
For ($i=0; $i -le ($object.properties.ipConfigurations.Count -1); $i++) {
($object.properties.ipConfigurations[$i].properties).PSObject.properties.remove('provisioningState')
($object.properties.ipConfigurations[$i]).PSObject.properties.remove('etag')
#export as well
$null = Invoke-azobjectbackup -Id $object.properties.ipConfigurations[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
}
"Microsoft.Network/loadBalancers/inboundNatRules" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Network/networkInterfaces/ipConfigurations" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Network/networkProfiles" {
$object.PSObject.properties.remove('etag')
($object.properties).PSObject.properties.remove('provisioningState')
# Clean inplace
For ($i=0; $i -le ($object.properties.containerNetworkInterfaceConfigurations.Count -1); $i++) {
($object.properties.containerNetworkInterfaceConfigurations[$i].properties).PSObject.properties.remove('provisioningState')
($object.properties.containerNetworkInterfaceConfigurations[$i]).PSObject.properties.remove('etag')
}
For ($i=0; $i -le ($object.properties.containerNetworkInterfaces.Count -1); $i++) {
($object.properties.containerNetworkInterfaces[$i].properties).PSObject.properties.remove('provisioningState')
($object.properties.containerNetworkInterfaces[$i]).PSObject.properties.remove('etag')
}
}
"Microsoft.Network/networkSecurityGroups" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('resourceGuid')
# Handle each security rule as separate objects
For ($i=0; $i -le ($object.properties.securityRules.Count -1); $i++) {
# $null = Invoke-azobjectbackup -Id $object.properties.securityRules[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
($object.properties.SecurityRules[$i]).PSObject.properties.remove('etag')
($object.properties.SecurityRules[$i].properties).PSObject.properties.remove('provisioningState')
}
#$object.properties.securityRules=@()
# Handle each default security rules - must be part of the nsg
# questions if some objects (like nsg rules shouldnt be separated for deployment
# Just clean the rules in place
For ($i=0; $i -le ($object.properties.defaultSecurityRules.Count -1); $i++) {
($object.properties.defaultSecurityRules[$i]).PSObject.properties.remove('etag')
($object.properties.defaultSecurityRules[$i].properties).PSObject.properties.remove('provisioningState')
}
}
"Microsoft.Network/networkSecurityGroups/defaultSecurityRules" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Network/networkSecurityGroups/securityRules" {
($object.properties).PSObject.properties.remove('provisioningState')
}
# Sentinel
"Microsoft.OperationsManagement/solutions" {
if ($object.plan.product -eq "OMSGallery/SecurityInsights"){
# Retrieve Sentinel Specific settings
# This is a Sentinel Workspace and should have a number of elements backedup
$children=@(
"/providers/Microsoft.SecurityInsights/sourcecontrols",
"/providers/Microsoft.SecurityInsights/settings",
"/providers/Microsoft.SecurityInsights/alertRules",
# "/providers/Microsoft.SecurityInsights/alertRuleTemplates",
"/providers/Microsoft.SecurityInsights/automationRules",
"/providers/Microsoft.SecurityInsights/bookmarks",
"/providers/Microsoft.SecurityInsights/dataConnectors",
"/providers/Microsoft.SecurityInsights/entityQueryTemplates",
"/providers/Microsoft.SecurityInsights/entityQueries"
)
foreach ($child in $children){
$response = Get-AzureObject -id "$($object.properties.workspaceResourceId)$($child)" -authHeader $authHeader -apiversions $AzAPIVersions
foreach ($element in $response.value){
$null = Invoke-azobjectbackup -Id $element.id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
}
}
}
"Microsoft.Network/privateEndpoints" {
$object.PSObject.properties.remove('etag')
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('resourceGuid')
# Handle each default service connections - must be part of the private endpoint
# Just clean the rules in place
For ($i=0; $i -le ($object.properties.privateLinkServiceConnections.Count -1); $i++) {
($object.properties.privateLinkServiceConnections[$i]).PSObject.properties.remove('etag')
($object.properties.privateLinkServiceConnections[$i].properties).PSObject.properties.remove('provisioningState')
}
}
"Microsoft.Network/privateDnsZones" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Network/publicIPAddresses" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('resourceGuid')
($object.properties).PSObject.properties.remove('ipAddress')
}
"Microsoft.Network/routeTables" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('resourceGuid')
# Handle routes as separate objects
For ($i=0; $i -le ($object.properties.routes.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.routes[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
$object.properties.routes=@()
}
"Microsoft.Network/virtualNetworks" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('resourceGuid')
# Handle each subnet as separate objects
For ($i=0; $i -le ($object.properties.subnets.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.subnets[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
$object.properties.subnets=@()
# Handle vnet Peering separate objects
For ($i=0; $i -le ($object.properties.virtualNetworkPeerings.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.virtualNetworkPeerings[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
$object.properties.virtualNetworkPeerings=@()
}
"Microsoft.Network/virtualNetworks/subnets" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Network/virtualNetworks/virtualNetworkPeerings" {
$object.PSObject.properties.remove('etag')
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('resourceGuid')
}
"Microsoft.OperationsManagement/solutions" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('creationTime')
($object.properties).PSObject.properties.remove('lastModifiedTime')
}
"Microsoft.OperationalInsights/workspaces" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('createdDate')
($object.properties).PSObject.properties.remove('modifiedDate')
($object.properties.sku).PSObject.properties.remove('lastSkuUpdate')
($object.properties.workspaceCapping).PSObject.properties.remove('quotaNextResetTime')
# Try to capture Security EventCollection Configuration if the workspace is a security workspace
$children=@(
"/scopedPrivateLinkProxies",
"/query",
"/metadata",
"/dataSources/SecurityEventCollectionConfiguration",
"/linkedStorageAccounts",
"/tables",
"/storageInsightConfigs",
"/linkedServices",
"/dataExports"
)
foreach ($child in $children){
$response = Get-AzureObject -id "$($object.Id)$($child)" -authHeader $authHeader -apiversions $AzAPIVersions
foreach ($element in $response.value){
# $null = Invoke-azobjectbackup -Id $element.id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
}
}
"Microsoft.Portal/dashboards" {
}
"Microsoft.RecoveryServices/vaults" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.identity).PSObject.properties.remove('tenantId')
($object.identity).PSObject.properties.remove('principalId')
}
"Microsoft.Resources/resourceGroups" {
}
"Microsoft.SecurityInsights/alertRules" {
# All alert rules have actions
$children=@(
"/actions"
)
foreach ($child in $children){
$response = Get-AzureObject -id "$($object.Id)$($child)" -authHeader $authHeader -apiversions $AzAPIVersions
foreach ($element in $response.value){
$null = Invoke-azobjectbackup -Id $element.id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
}
}
"Microsoft.Storage/storageAccounts" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('creationTime')
($object.properties).PSObject.properties.remove('secondaryLocation')
($object.properties).PSObject.properties.remove('statusOfSecondary')
# Handle each Private Endpoint Connection as separate objects
For ($i=0; $i -le ($object.properties.PrivateEndpointConnections.Count -1); $i++) {
$null = Invoke-azobjectbackup -Id $object.properties.PrivateEndpointConnections[$i].id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
$object.properties.PrivateEndpointConnections=@()
#type containers need recursion if they exist
$null = Invoke-azobjectbackup -Id "$($object.id)/blobServices/default" -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
$null = Invoke-azobjectbackup -Id "$($object.id)/fileServices/default" -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
$null = Invoke-azobjectbackup -Id "$($object.id)/queueServices/default" -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
$null = Invoke-azobjectbackup -Id "$($object.id)/tableServices/default" -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
"Microsoft.SqlVirtualMachine/sqlVirtualMachines" {
($object.properties).PSObject.properties.remove('provisioningState')
}
"Microsoft.Web/connections" {
($object.properties).PSObject.properties.remove('provisioningState')
($object.properties).PSObject.properties.remove('createdTime')
($object.properties).PSObject.properties.remove('changedTime')
}
}
if ($object ){ convertto-json -InputObject $object -Depth 50}else{return $null}
}
function Invoke-azobjectbackup(){
<#
Purpose: Backs Up an Azure object onto the file system as a json object
#>
[CmdletBinding()]
param(
[Parameter(mandatory=$true)]
[string]$Id,
[Parameter(mandatory=$true)]
[string]$BackupDir,
[Parameter(mandatory=$true)]
[Hashtable]$AzAPIVersions,
[Parameter(mandatory=$true)]
[Hashtable]$authHeader,
[Parameter(mandatory=$false)]
[switch]$norecurse
)
$object = $null
$object = Get-Azureobject -AuthHeader $authHeader -apiversions $AzAPIVersions -id $id
if($object.name){
$objectname = ($object.name).replace(' ','')
}
$azobjectjson = $null
$azobjectjson = Clean-AzureObject -azobjectjson (convertto-json -InputObject $object -Depth 50) -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
# Split the Id string into an array so that different pieces can be retrieved
$filename = $Id.split('/')[-1]
$idarray = $Id.Split("/")
# Delay requests to Microsoft Authorisation to prevent being throttled - Microsoft only all 120 a minute
if ($idarray[2] -eq "Microsoft.Authorization"){Start-Sleep -Milliseconds 500 }
# Objects will be written into folders based on their Resource Group unless they are special objects
$dirpath = "$($BackupDir)$($slash)$($idarray[4].ToLower())$($slash)"
# Keep all exported Policy Definitions together
if ($idarray[3] -eq "policyDefinitions"){$dirpath = "$($BackupDir)$($slash)policydefinitions$($slash)" }
if ($idarray[3] -eq "policySetDefinitions"){$dirpath = "$($BackupDir)$($slash)policyDefinition$($slash)" }
# Keep Security objects together
if ($idarray[4] -eq "Microsoft.Security"){$dirpath = "$($BackupDir)$($slash)Microsoft.Security$($slash)" }
#Role Assignments I want to keep with the objects
if ($object.type -eq "Microsoft.Authorization/roleAssignments" ){
# The actual path will be the scope - not the object ID... either at the subscription level
$dirpath = "$($BackupDir)$($slash)$($($object.properties.scope.split('/')[2]).ToLower())$($slash)roleAssignments$($slash)"
# ... or at the Resource Group Level
if ($object.properties.scope.split('/')[4]){$dirpath = "$($BackupDir)$($slash)$($($object.properties.scope.split('/')[4]).ToLower())$($slash)roleAssignments$($slash)" }
}
# Take special characters and spaces out of the backup file name
$backupfile = "$($idarray[8])"
$backupfile = $backupfile.Replace(' ','')
$backupfile = $backupfile.Replace('[','')
$backupfile = $backupfile.Replace(']','')
# Find the last 'provider' element
for ($i=0; $i -lt $IDArray.length; $i++) {
if ($IDArray[$i] -eq 'providers'){$provIndex = $i}
}
<#
Backup file name gets messy
Odd cases must be accounted for like virtual machine attached packages where many with the same names will exist in the
same Resource Group.
Others will have ids larger than the Windows character limit
Normally just the object type plust its given name will be enough to be unique but exceptions must be accomodated
#>
if ($objecttype){ $objecttype = (($object.type).split('/'))[-1]}
#There are some objects that dont have a type property - default to deriving type from the ID
if ($objecttype -eq $null){ $objecttype = $IDArray[$provIndex + 2]}
if ($IDArray.length -lt 8 ){
$outputfilename = "$($IDArray[-1])__$($IDArray[-2])"
}
else{
#$backupfile = $backupfile + "__$($IDArray[$provIndex + 2])"
if(!($provIndex -eq 10 )){$outputfilename = $backupfile + "$($IDArray[9])" }
if(!($IDArray[$provIndex + 1] -eq $idarray[8] )){
#$outputfilename = $IDArray[$provIndex + 2] + "__$($objectname )"
$outputfilename = $objecttype + "__$($objectname )"
if(!($objectname -eq $backupfile )){ $outputfilename = "$( $outputfilename)__`($($backupfile)`)" }
}
}
#$outputfilename = "$($objecttype)__$($outputfilename)"
# account for 260 character limitation with names of query rules
# truncate the file name
if ("$($dirpath)\$($outputfilename).json".Length -gt 259){
$outputfilename = $outputfilename.Substring(0, 100)
write-debug "truncated backup file = $outputfilename"
}
write-debug "outfile = $($dirpath)$($outputfilename).json"
# White the output to file and probe for object types not listed by Microsoft
if ( $azobjectjson ){
# If the directory doesnt exist, create it.
if (!(Test-Path -Path $dirpath)){ $null = New-Item -Path $dirpath -ItemType 'Directory' -Force }
# somewhere quoted null is being produced with workbooks?
# hack to stop those files being written - only write json
if ($azobjectjson.Contains('{')){
if ($OS -eq 'linux'){
Out-File -FilePath "$($dirpath)/$( $outputfilename ).json" -InputObject $azobjectjson -Force
}
else{
Out-File -FilePath "$($dirpath)\$( $outputfilename ).json" -InputObject $azobjectjson -Force
}
}
if (!($norecurse)){
Invoke-DiagnosticsConfigSearch -Id $id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
}
}
function Invoke-DiagnosticsConfigSearch(){
<#
Purpose: Backs Up hidden Azure object onto the file system as a json objects
Diagnostics Settings can only be determined by querying for a diagnostics settings file
#>
[CmdletBinding()]
param(
[Parameter(mandatory=$true)]
[string]$Id,
[Parameter(mandatory=$true)]
[string]$BackupDir,
[Parameter(mandatory=$true)]
[Hashtable]$AzAPIVersions,
[Parameter(mandatory=$true)]
[Hashtable]$authHeader
)
#The objects to not look at for analytics
$ignoretypes=@(
"/",
"microsoft.alertsmanagement/smartdetectoralertrules",
"microsoft.authorization/policyassignments",
"Microsoft.Authorization/policyDefinitions",
"microsoft.authorization/policyexemptions",
"Microsoft.Authorization/policySetDefinitions",
"microsoft.authorization/roledefinitions",
"Microsoft.Authorization/roleAssignments",
"microsoft.automation/automationaccounts",
"microsoft.automation/automationaccounts/runbooks",
"microsoft.azureactivedirectory/b2cdirectories",
"microsoft.compute/availabilitysets",
"microsoft.compute/images",
"microsoft.compute/proximityplacementgroups",
"microsoft.compute/restorepointcollections",
"microsoft.compute/snapshots",
"microsoft.compute/sshpublickeys",
"microsoft.compute/virtualmachines/extensions",
"microsoft.hybridcompute/machines",
"Microsoft.Insights/ActivityLogAlerts",
"microsoft.insights/actiongroups",
"microsoft.insights/metricalerts",
"microsoft.insights/scheduledqueryrules",
"microsoft.insights/workbooks",
"microsoft.network/applicationsecuritygroups",
"microsoft.network/firewallpolicies",
"microsoft.network/networkinterfaces/ipconfigurations",
"microsoft.network/networkprofiles",
"microsoft.network/networkwatchers/flowlogs",
"microsoft.network/privatednszones/virtualnetworklinks",
"microsoft.network/routetables",
"microsoft.network/serviceendpointpolicies",
"microsoft.network/virtualhubs",
"microsoft.network/virtualnetworks/subnets",
"microsoft.network/virtualnetworks/virtualnetworkpeerings"
"microsoft.network/networkinterfaces/ipconfigurations",
"microsoft.network/privatednszones/virtualnetworklinks",
"microsoft.network/privatednszones/virtualnetworklinks",
"microsoft.operationsmanagement/solutions",
"microsoft.portal/dashboards",
"Microsoft.Resources/resourceGroups",
"Microsoft.SecurityInsights/sourcecontrols",
"Microsoft.SecurityInsights/settings",
"Microsoft.SecurityInsights/alertRules",
"Microsoft.SecurityInsights/automationRules",
"Microsoft.SecurityInsights/bookmarks",
"Microsoft.SecurityInsights/entityQueries",
"Microsoft.SecurityInsights/entityQueryTemplates",
"microsoft.solutions/applications",
"microsoft.sqlvirtualmachine/sqlvirtualmachines",
"microsoft.visualstudio/account",
"microsoft.web/connections"
)
$backup=$true
$IDArray = ($id).split("/")
# Find the last 'provider' element
for ($i=0; $i -lt $IDArray.length; $i++) {
if ($IDArray[$i] -eq 'providers'){$provIndex = $i}
}
$arraykey = "$($IDArray[$provIndex + 1])/$($IDArray[$provIndex + 2])"
write-debug $arraykey
write-debug "ID = $($Id)"
# Ignore specifc types of objects that can't do diagnostic logging
if ($ignoretypes -contains $arraykey){
#write-debug "found type in ignore array"
$backup=$false
}
#Dont try and and get Diagnostics from a Resource Group
if ($IDArray.Count -eq 4 ){
$backup=$false
}
if ($backup -eq $true){
$object = $null
write-debug "diagnostics search $($arraykey)"
$object = Get-Azureobject -AuthHeader $authHeader -apiversions $AzAPIVersions -id "$($id)/providers/microsoft.insights/diagnosticSettings/"
if ( $object.value){
$azobjectjson = $null
Foreach ($object in $object.value) {
if (!($ignoretypes -contains $object.tyoe )){
$null = Invoke-azobjectbackup -Id $object.id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader -norecurse
}
}
$filename = $Id.split('/')[-1]
$idarray=$Id.Split("/")
#Resource Group
# Force directory case notation to lower for consistency
$dirpath = "$($BackupDir)\$($idarray[4].ToLower())\"
if (!(Test-Path -Path $dirpath)){ New-Item -Path $dirpath -ItemType 'Directory' -Force }
$backupfile = $dirpath
For ($i=($idarray.Count -3); $i -le ($idarray.Count -1); $i++) {
if ($i -eq ($idarray.Count -3)){
$backupfile = $backupfile + "$($idarray[$i])"
}
else
{
$backupfile = $backupfile + "__$($idarray[$i])"
}
}
#Backup File will be an object filename representation within a RG directory structure
#$backupfile
$backupfile = $backupfile.Replace(' ','')
if ( $azobjectjson ){
write-debug "hidden object outfile = $backupfile"
#account for 260 character limitation
#truncate the file name
if ("$($dirpath)\$($backupfile).json".Length -gt 259){
$backupfile = $backupfile.Substring(0, 120)
}
Out-File -FilePath "$($backupfile).json" -InputObject $azobjectjson -Force
}
} # object
}
}
<#
Main
#>
# On some systems 'System.Web' needs to be explicitly added
Add-Type -AssemblyName System.Web;
# Get an authorised Azure Header
$authHeader = Get-Header -scope $scope -Tenant $tenant -AppId $appid `
-secret $secret
<#
Get all resources in a subscription (or just a Resource Group if specified)
There are a number of hidden resources that will need to be grabbed later
#>
# If I'm not backing up just a Resource Group there are probably other elements I really need to get
if ($ResourceGroup){
$queries=@(
# Just Query Resource Group objects
"https://management.azure.com/subscriptions/$($subscription)/resourceGroups/$($resourcegroup)/resources?api-version=2021-04-01"
)
}
else
{
$queries=@(
# Get all Subscription objects
"https://management.azure.com/subscriptions/$($subscription)/resources?api-version=2021-04-01",
# Role Definitions
"https://management.azure.com/subscriptions/$($subscription)/providers/Microsoft.Authorization/roleDefinitions?api-version=2017-05-01",
# Empty Resource Group Details
"https://management.azure.com/subscriptions/$($subscription)/resourcegroups?api-version=2021-04-01",
# Role Assignments
"https://management.azure.com/subscriptions/$($subscription)/providers/Microsoft.Authorization/roleAssignments?api-version=2017-05-01",
# Policy Set definitions
"https://management.azure.com/subscriptions/$($subscription)/providers/Microsoft.Authorization/policySetDefinitions?api-version=2019-09-01",
# Policy Definition
"https://management.azure.com/subscriptions/$($subscription)/providers/Microsoft.Authorization/policyDefinitions?api-version=2019-09-01",
# Policy Assignments
"https://management.azure.com/subscriptions/$($subscription)/providers/Microsoft.Authorization/policyAssignments?api-version=2019-09-01",
# Policy Exemptions
"https://management.azure.com/subscriptions/$($subscription)/providers/Microsoft.Authorization/policyExemptions?api-version=2020-07-01-preview",
# Security Center Subscriptions
"https://management.azure.com/subscriptions/$($subscription)/providers/Microsoft.Security/pricings?api-version=2022-03-01"
)
}
# Make sure the subscription objects collection is empty
$subscriptionobjects =@()
foreach ($queryuri in $queries){
write-debug "+ QueryURI = $($queryuri)"
# Get the first set of returned objects into an array
# There are likely to be many pages worth - if so, the response will have a nextlink url
$response = Invoke-RestMethod -Uri $queryuri -Method GET -Headers $authHeader
$subscriptionobjects += $response.value
while($response.nextLink)
{
# Grab any additional pages of objects recursively until no more nextlink responses exist
$nextLink = $response.nextLink
$response = Invoke-RestMethod -Uri $nextLink -Method GET -Headers $authHeader
$subscriptionobjects += $response.value
}
}
# At this point $subscriptionobjects contains object references for each object in the subscription
# Remove any old objects from the Backup folder
if ($BackupDir){
if($OS -eq 'win' ){ Remove-Item $BackupDir\* -Recurse -Force }
if($OS -eq 'linux' ){
$foldersToDel = Get-ChildItem $BackupDir -Directory
$ExcludeFolders=@(".git",".github")
foreach ($folder in $foldersToDel){
if ($ExcludeFolders.Contains($folder.Name ) ){write-output "Folder Name in Exclude list = $($folder.Name)"}else{
$folder | remove-item -force -Recurse
}
}
}
}
# Recurse through all subscription objects and retrieve content
# This needs a dictionary of all API versions to namespace types
# Get the API Versions dictionary for retrieving objects
# This lets us know what version needs to be used with a GET request.
$authHeader = Get-Header -scope $scope -Tenant $tenant -AppId $appid `
-secret $secret
if (!$AzAPIVersions){$AzAPIVersions = Get-AzureAPIVersions -header $authHeader -SubscriptionID $subscription}
foreach ($azureobject in $subscriptionobjects ){
$authHeader = Get-Header -scope $scope -Tenant $tenant -AppId $appid -secret $secret
Invoke-azobjectbackup -Id $azureobject.id -BackupDir $BackupDir -AzAPIVersions $AzAPIVersions -authHeader $authHeader
}
To be continued:
- Log in to post comments