Data Lake Integration (Azure) - Manual Deploy

Last updated
Save as PDF

Web Central REST API

Overview

The instructions in this article are for System Integrators who want to deploy the Archibus - Azure Data Lake integration without Azure DevOps. When you have completed these instructions, you will have a running Azure Function that automatically syncs Archibus space data into an ADLS Gen2 account using a medallion (bronze/silver) architecture — deployed entirely through the Azure Portal and your local machine. After deployed, operational management is the same regardless of how you deployed. These instructions include the full Operator Configuration Reference and Scenario Verification sections.

Contents

Prerequisites
Step 1. Deploy Infrastructure (Bicep via Azure Portal)
Step 2. Create the Function App
Step 3. Assign RBAC Roles
Step 4. Configure Application Settings
Step 5. Deploy the Function Code
Step 6. Populate SyncTableRegistry
Step 7. Verify the Deployment
Update the Function (Re-deployment)
Operator Configuration Reference
Scenario Verification

Prerequisites

Ensure the following are in place before starting.

Deployment Bundle

Before starting, download the versioned deployment bundle (archibus-sync-vX.Y.Z.zip) from Allbound (the partner/client portal). This bundle contains everything referenced throughout this article:

functionapp.zip - the built function code
infra/main.bicep and infra/main.json - the infrastructure template (Step 1 uses main.json if your Portal version rejects .bicep uploads directly)
client-deploy-pipeline.yml - only needed if you're following the separate Data Lake Integration (Azure) - DevOps Deploy

Extract the zip locally and the file paths referenced later in this article (e.g. infra/main.bicep, functionapp.zip) are relative to this extracted folder.

Azure

An Azure subscription and an existing resource group where all resources will be created.
User Assigned Managed Identity (UAMI) with Contributor and User Access Administrator roles on the resource group.

Why two roles?

Contributor lets you create resources. User Access Administrator lets you assign Storage Blob Data Contributor and Storage Table Data Contributor roles to the UAMI on the new storage accounts; without it Step 3 will fail with a 403.

Tools

Azure CLI - Install from https://learn.microsoft.com/en-us/cli/azure/install-azure-cli and it requires version 2.60.0 or later for correct Flex Consumption support. This is the only tool needed for Step 5 (code deployment) as documented in this article.

Archibus OIDC Credentials

The function authenticates to the Archibus REST API using the OAuth2 client credentials flow via your OIDC identity provider.

Contact your Archibus administrator to obtain the following:

ARCHIBUS_CLIENT_ID - the OAuth2 client ID registered in your OIDC identity provider
ARCHIBUS_CLIENT_SECRET - the corresponding client secret
ARCHIBUS_TOKEN_URL - your OIDC token endpoint e.g. https://<your-idp-domain>/oauth2/token
ARCHIBUS_BASE_URL - the base URL of your Archibus instance e.g. https://<your-instance>/archibus

These credentials will be entered as application settings in Step 4 and they are never stored in source control.

Network Connectivity

The Azure Function requires outbound internet access to both:

ARCHIBUS_BASE_URL - the Archibus REST API endpoint
ARCHIBUS_TOKEN_URL - the OIDC token endpoint

If your Archibus instance is behind a VPN, corporate proxy, or private network, you must configure VNet integration or outbound firewall rules in the Function App before the sync will work. This is an infrastructure issue outside the scope of these instructions; make sure you contact your network/cloud team.

Step 1. Deploy Infrastructure (Bicep via Azure Portal)

This step creates the storage accounts, tables, containers, Application Insights, and UAMI using the existing Bicep template.

Navigate to Azure Portal > your Resource Group > Create.
In the Marketplace search bar, type Template deployment.
From the results, click Template deployment (deploy using custom templates).
On the Template deployment (deploy using custom templates) page, if Subscription and Plan fields are visible, leave at their defaults, and click the Create button
On the Custom Deployment page, click Build your own template in the editor.
On Edit Template page:
1. Click Load file.
2. Upload infra/main.bicep from the extracted bundle
3. Click Save. Note: If the load file rejects the bicep file, some Azure Portal versions only accept ARM JSON in this step, not raw Bicep. The bundle already includes a pre-compiled infra/main.json right alongside main.bicep for this situation. There is no need to compile anything yourself, instead, load the main.json file.
Complete the Bicep parameters using the table below.


Label	Template Parameter	Sample Value	What it represents	Naming rules
`Storage Account Name`	`StorageAccountName`	archibussyncfunc	Function runtime + Table Storage account	3–24 chars, lowercase letters and digits only (no hyphens). Must be globally unique across all of Azure.
`Adls Account Name`	`adlsAccountName`	archibussyncadls	ADLS Gen2 data lake account	3–24 chars, lowercase letters and digits only (no hyphens). Must be globally unique across all of Azure.
`Bronze Container Name`	`bronzeContainerName`	bronze	Bronze blob container name	3–63 chars, lowercase letters, numbers, and hyphens only. Must start and end with a letter or number. and no leading/trailing or consecutive hyphens.
`Silver Container Name`	`silverContainerName`	silver	Silver blob container name	3–63 chars, lowercase letters, numbers, and hyphens only. Must start and end with a letter or number. and no leading/trailing or consecutive hyphens.
`Sync Table Registry Name`	`syncTableRegistryName`	SyncTableRegistry	SyncTableRegistry table name	3–63 chars, must start with a letter. Must differ from watermark table (defined in step 4 - Application Settings). Defaults to this value `SyncTableRegistry` if omitted
`Managed Identity Name`	`managedIdentityName`	archibus-sync-devqa-mi	UAMI resource name	Name of the existing UAMI created as a prerequisite, must already exist before running this deployment 3–128 chars, letters/digits/hyphens/underscores
`Function App Name`	`functionAppName`	func-archibus-sync-dev	Must match the Function App name you'll create in Step 2 — used only to name the Application Insights resource (`<functionAppName>-insights`)	—
`Region`	`location`	(US)East US	Azure region	Defaults to resource group location if omitted

Parameters Subscription and Resource group are auto populated.

The SyncLock table is NOT created by Bicep. It is auto-created by the function on its first sync run, so no manual action is needed.

The SyncSchemaHistory and SyncWatermarks tables are NOT pre-created by Bicep. They are auto-created by the function, but only after the first valid entity is confirmed, as a row in SyncTableRegistry whose table name actually exists in the Archibus API. No manual action is needed.

If you need to delete and recreate the bronze/silver containers (e.g. during testing, or resetting an environment) and later redeployment fails, you'll see something like:

{ 
      "code": "DeploymentFailed", 
      "message": "At least one resource deployment operation failed. List deployment operations for details and see https://aka.ms/arm-deployment-operations for usage details.", 
      "details": [ 
        { 
           "code": "ContainerOperationFailure", 
           "message": "The specified container is being deleted. Try operation later." 
         }
     ]
 }

The top-level DeploymentFailed/"At least one resource deployment operation failed" message is generic and the actual cause is in the nested details array: ContainerOperationFailure / "The specified container is being deleted. Try operation later." This is Azure's Container soft delete on the ADLS account keeping the old container's name reserved for a retention window (commonly 7 days) and note that this is not a bug in this deployment.

- To resolve immediately: restore the soft-deleted container (az storage container restore --deleted-version <version>) instead of waiting out the retention window.
- Soft-delete is generally worth keeping enabled on a production deployment and the bronze container holds your append-only audit history, and soft-delete is what lets you recover it if a container is ever deleted accidentally. Only disable it (az storage account blob-service-properties update --enable-container-delete-retention false) if this is a disposable test/dev account where you expect to repeatedly delete and recreate containers.

The Bicep template only provisions supporting infrastructure (storage accounts, tables, containers, Application Insights) and it does not create the Function App itself. That's Step 2, as a deliberately separate step.

Click Review + Create.
Azure validates the template and configuration:
- Checks for syntax errors in the Bicep file.
- Verifies parameter values and resource naming rules.
Review the deployment summary:
- Confirm subscription and resource group.
- Verify all resource names and configurations.
After validation passes, click Create to start the deployment.
Azure provisions the resources (Storage Accounts, Tables, Containers, Application Insights, UAMI).
Monitor deployment progress from the Deployments section until status shows Success.

Step 2. Create the Function App

This step is divided into two parts:

Create Function App
Attach the UAMI to the Function App

Step 2.1 Create Function App

There are two ways to create function:

Azure Portal
Azure CLI

Option A - Azure Portal

Navigate to Azure Portal.
Navigate to Resource Group.
Create a resource.
From the Create Resource page, search for "Function App".
Click the Create and choose Flex Consumption for Hosting Plan.
Complete following settings.


Field	Value
Subscription and Resource Group	Select the same resource group used in Step 1 (where the storage accounts and Application Insights were created), plus subscription, name, and region
Function App name	It must be the exact same value you used for the functionAppName parameter in Step 1. A mismatch here will create an issue at Step 4.
Region	It must be the same as Resource Group’s Region.
Runtime stack	Python
Version	3.14
Instance Size	Choose the appropriate instance size.
Zone redundancy	Keep it Disabled

For the Storage tab, click the Next button.
In the Storage Account field, select the Storage Account (storageAccountName and not the ADLS Gen2 account (adlsAccountName)) created in Step 1.
Click the Next button until Monitoring tab displays.
In the Application Insight field, select the Application Insights instance created in Step 1
Leave the remaining tabs with the default selections.
In Review + create tab, verify all details.
Click the Create button.

Option B - Azure CLI

<FUNCTION_APP_NAME> below must be the exact same value you used for the functionAppName parameter in Step 1 and that parameter is what named the Application Insights resource (<functionAppName>-insights), so a mismatch here means Step 4's Application Insights connection string won't correspond to this Function App.

This ties directly into what we already confirmed earlier (Step 1) that functionAppName in the bicep template is used only to name the App Insights resource, so the reasoning is grounded in something already established on the page, not a new claim.

The--storage-account must be the function runtime storage account (storageAccountName from Step 1), not the ADLS Gen2 account (adlsAccountName). There are two storage accounts in play, and it is easy to mix these up.

The ADLS account (HNS enabled) holds bronze/silver; the function storage account (HNS disabled) is the one the Function App runtime actually needed here.

Execute below command to create a function.

az functionapp create --name <FUNCTION_APP_NAME> --resource-group <RESOURCE_GROUP_NAME> --storage-account <FUNC_STORAGE_ACCOUNT_NAME> --flexconsumption-location <REGION> --runtime python --runtime-version 3.14 --os-type linux

The CLI may fail to auto-create/link Application Insights (a benign, informational error "Please use the Azure Portal to create and configure the Application Insights, if needed"). This is fine and you already have your own Application Insights instance from Step 1, and its connection string is set manually in Step 4 regardless of which option you use here.

Step 2.2 Attach UAMI to the Function App

The UAMI is a standalone Azure resource and this is created as a prerequisite and creating it doesn't automatically connect it to any compute resource.

The function code authenticates to Azure Storage (ADLS blob access, Table Storage) using DefaultAzureCredential/ManagedIdentityCredential; no connection strings, no secrets, just "ask Azure for a token as whatever managed identity I'm running as." For that to work, the Function App has to actually be that identity at runtime, which only happens once the UAMI is explicitly attached to it. The RBAC roles from Step 3 are also granted to the UAMI's principal, not to the Function App directly, if the Function App never attaches that UAMI, it has no way to present itself as that identity, so those role grants are irrelevant to it and every storage call from the code will fail with an authentication error, even though the roles technically exist.

Make sure you give the Function App a moment before attaching it!

At this point it's just a blank container; an empty function host running the Flex Consumption runtime with no code deployed yet (that happens in Step 5). Take a short pause after creation before attaching the identity or confirm it's actually up first by executing following CLI command.

az functionapp show --name <FUNCTION_APP_NAME> --resource-group <RESOURCE_GROUP_NAME> --query "properties.state" -o tsv

After this returns Running, proceed with attaching the UAMI. Complete the steps for either Azure Portal or Azure CLI.

Portal

Go to the new Function App > Settings (left menu) > Identity.
From the User assigned tab, click + Add.
Select the UAMI used in Step 1.
Click Add.

CLI

Identities parameter needs the UAMI's full Azure Resource ID (UAMI_RESOURCE_ID), a path uniquely identifying it across all of Azure, in this form:

/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<UAMI_NAME>

Instead of constructing UAMI_RESOURCE_ID manually, retrieve it with another CLI command:

az identity show --name <UAMI_NAME> --resource-group <RESOURCE_GROUP_NAME> --query id -o tsv

Then attach UAMI to azure function with below CLI command:

az functionapp identity assign --name <FUNCTION_APP_NAME> --resource-group <RESOURCE_GROUP_NAME> --identities <UAMI_RESOURCE_ID>

A successful response looks like this — confirming the UAMI is now attached, with its clientId being the value you'll need for the AZURE_CLIENT_ID app setting in Step 4:

{
  "type": "UserAssigned",
  "userAssignedIdentities": {
    "/subscriptions/.../userAssignedIdentities/<UAMI_NAME>": {
      "clientId": "...",
      "principalId": "..."
    }
  }
}

Step 3. Assign RBAC Roles

The Function App's UAMI needs two role assignments, one on each storage account.

There are two ways to create function:

Azure Portal
Azure CLI

Option A - Azure Portal

For each row in the table below:

Navigate to the storage account in Azure Portal.
Click Access Control (IAM).
Click + Add and then Add role assignment.
Select the role.
Click Next.
Assign access to Managed Identity.
Select the UAMI.
Review + assign.


Storage account	Role to assign	Assignee
`adlsStorageAccountName` (ADLS Gen2, HNS enabled)	Storage Blob Data Contributor	UAMI
`funcStorageAccountName` (function runtime storage)	Storage Table Data Contributor	UAMI

Option B - Azure CLI

Another alternative is to use set of CLI commands to assign these roles to storage account created in Step 1. Follow below steps

First get the UAMI's principal ID with following command:

az identity show --name <UAMI_NAME> --resource-group <RESOURCE_GROUP_NAME> --query principalId -o tsv

Then run below commands to assign above roles to UAMI on given storage account (mentioned in CLI command):

az role assignment create --assignee-object-id <UAMI_PRINCIPAL_ID> --assignee-principal-type ServicePrincipal --role "Storage Blob Data Contributor" --scope /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.Storage/storageAccounts/<ADLS_STORAGE_ACCOUNT>

az role assignment create --assignee-object-id <UAMI_PRINCIPAL_ID> --assignee-principal-type ServicePrincipal --role "Storage Table Data Contributor" --scope /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.Storage/storageAccounts/<FUNC_STORAGE_ACCOUNT>

Running the above az role assignment create commands requires the logged-in CLI identity (via az login) to have Microsoft.Authorization/roleAssignments/write permission - typically User Access Administrator or Owner roles on the resource group or storage account.

The Bicep template does not assign these roles itself and they must always be assigned separately, exactly as described here, regardless of which deployment path you're using.

Step 4. Configure Application Settings

Configure application settings either via the Azure Portal or Azure CLI.

Option A - Azure Portal

Navigate to the Function App > Settings > Environment Variables > Application settings tab.
Click + New application setting for each entry in the table below.
Click Apply after all settings are added and the Function App restarts automatically.


Setting Name	Value
`AZURE_CLIENT_ID`	Client ID of the UAMI and you will find it in the Portal: UAMI > Overview > Client ID
`ARCHIBUS_BASE_URL`	e.g. `https://<your-instance>/archibus`
`ARCHIBUS_CLIENT_ID`	OAuth2 client ID from your OIDC identity provider.
`ARCHIBUS_CLIENT_SECRET`	OAuth2 client secret — treat as a password.
`ARCHIBUS_TOKEN_URL`	OIDC token endpoint URL
`ADLS_ACCOUNT_URL`	`https://<adlsStorageAccountName>.dfs.core.windows.net`
`FUNC_STORAGE_ACCOUNT`	`https://<funcStorageAccountName>.table.core.windows.net`
`ADLS_BRONZE_CONTAINER`	Bronze container name e.g. `bronze`
`ADLS_SILVER_CONTAINER`	Silver container name e.g. `silver`
`WATERMARK_TABLE`	Watermark table name e.g. `SyncWatermarks`
`SYNC_TABLE_REGISTRY`	Registry table name e.g. `SyncTableRegistry`
`SYNC_SCHEDULE`	6-field Azure Functions cron expression e.g. `0 0 /4 * *` Minimum interval: 10 minutes.
`PAGE_SIZE`	`500` (max 10,000 — larger values reduce API round-trips but increase memory pressure)
`APPLICATIONINSIGHTS_CONNECTION_STRING`	Connection string of the Application Insights instance from Step 1 (Portal → Application Insights resource → Overview → Connection String)
`MAX_WORKERS` (optional)	This controls how many Archibus tables are synced in parallel. It defaults to 4 if not set. To use a non-default value, add it here as an additional application setting. Performance Warning Increasing `MAX_WORKERS` raises the number of simultaneous connections to the Archibus REST API. Setting it too high can overwhelm the Archibus API server or corporate proxy, causing HTTP 429 (rate limit) or timeout errors. Start with the default of 4 and increase cautiously, monitoring Archibus API response times before and after each change.

Important

FUNC_STORAGE_ACCOUNT and ADLS_STORAGE_ACCOUNT (plain account names) are not valid settings for this function. The code only reads ADLS_ACCOUNT_URL and FUNC_STORAGE_TABLE_URL (full URLs) and setting the plain account names instead of the URLs will cause the function to fail immediately on startup.

How to construct ADLS_ACCOUNT_URL and FUNC_STORAGE_TABLE_URL

Both options will create the same value, and the Endpoints page is only confirming the same predictable pattern, so for a quick deployment, you can construct it yourself.

Option I - Manual

The simplest way is to construct it directly, since the pattern is fixed and predictable:

ADLS_ACCOUNT_URL = https://<adlsStorageAccountName>.dfs.core.windows.net
FUNC_STORAGE_TABLE_URL = https://<funcStorageAccountName>.table.core.windows.net

Substitute the actual storage account name and there is no need to look anything up.

Option II - Azure Portal

From Azure Portal make sure you double-check that you have the exact right one.

Navigate to the Storage Account Name.
Navigate to Settings > Endpoints.
You'll see a list of service endpoints and copy the Data Lake Storage endpoint for ADLS_ACCOUNT_URL (from the ADLS Gen2 account), and the Table service endpoint for FUNC_STORAGE_TABLE_URL (from the function storage account)

Option B - Azure CLI

Execute below command to apply the settings with CLI

az functionapp config appsettings set --name <FUNCTION_APP_NAME> --resource-group <RESOURCE_GROUP_NAME> --settings AZURE_CLIENT_ID="<UAMI_CLIENT_ID>" ARCHIBUS_BASE_URL="<your Archibus base URL>" ARCHIBUS_CLIENT_ID="<your OAuth2 client ID>" ARCHIBUS_CLIENT_SECRET="<your OAuth2 client secret>" ARCHIBUS_TOKEN_URL="<your OIDC token URL>" ADLS_ACCOUNT_URL="<https://<adlsStorageAccountName>>.dfs.core.windows.net" FUNC_STORAGE_TABLE_URL="<https://<funcStorageAccountName>>.table.core.windows.net" ADLS_BRONZE_CONTAINER="bronze" ADLS_SILVER_CONTAINER="silver" WATERMARK_TABLE="SyncWatermarks" SYNC_TABLE_REGISTRY="SyncTableRegistry" SYNC_SCHEDULE="0 0 */4 * * *" PAGE_SIZE="500" APPLICATIONINSIGHTS_CONNECTION_STRING="<connection string from the Application Insights resource created in Step 1>"

If you're running this from Bash or Git Bash, you can split it across multiple lines with \ line continuations for readability, this only works in bash-family shells, not PowerShell/cmd, which is why the single-line form above is given as the default.

How to get Application Insights connection string

- Azure Portal
  1. Navigate to the Application Insights resource created in Step 1 (its name is <functionAppName>-insights)
  2. On the Overview page, look for the Connection String field (has a copy icon next to it)
- Azure CLI

az resource show --resource-group <RESOURCE_GROUP_NAME> --resource-type "Microsoft.Insights/components" --name "<functionAppName>-insights" --query "properties.ConnectionString" -o tsv

ADLS_BRONZE_CONTAINER, ADLS_SILVER_CONTAINER, WATERMARK_TABLE, and SYNC_TABLE_REGISTRY above use the defaults from Step 1. If you customized any of those Bicep parameters, use your actual values here instead.

Step 5. Deploy the Function Code

This is the sequence validated end-to-end for the Flex Consumption plan this app runs on.

Note: We choose Flex Consumption as our hosting plan and it only supports deployment of the code through CLI.

Step 5.1 Stop the Function App first

This avoids a race condition where stopping the app after deployment could interrupt a deployment still in progress (a 202 Accepted response only means Azure accepted the request, not that it has finished processing):

az functionapp stop --name <FUNCTION_APP_NAME> --resource-group <RESOURCE_GROUP_NAME>

Step 5.2 Deploy the code zip

<FILE_PATH> must be the path to the zip file itself (functionapp.zip the inner file, found after extracting the bundle) not the outer archibus-sync-vX.Y.Z.zip and not a folder of unzipped source files. Use --build-remote true as this tells Azure to install dependencies from requirements.txt remotely during deployment, so you don't need to manually pip install/re-zip beforehand:

az functionapp deployment source config-zip --src <FILE_PATH> --name <FUNCTION_APP_NAME> --resource-group <RESOURCE_GROUP_NAME> --build-remote true

Requires Azure CLI 2.60.0 or later. Occasional transient 503 errors during the remote build are a known rough edge and retry the command once or twice if this happens.

Step 5.3 Start the Function App

This guarantees a genuine cold start on the new code, which reliably fires run_on_startup:

az functionapp start --name <FUNCTION_APP_NAME> --resource-group <RESOURCE_GROUP_NAME>

If the CLI reports a certificate/SSL error at the end of deployment (e.g. CERTIFICATE_VERIFY_FAILED while checking function host health): this is usually a local network/proxy issue on the machine running the CLI (a corporate proxy intercepting TLS traffic), not a deployment failure then the deployment itself has almost certainly already succeeded (look for a 202 Accepted status earlier in the output).

CLI deployment might show "Deployment was successful but the app appears to be unhealthy. Please check the app logs." This is just because the app is still Stopped at that point in the documented sequence.

Confirm independently instead with the following CLI command:

az functionapp function list --name <FUNCTION_APP_NAME> --resource-group <RESOURCE_GROUP_NAME> --query "[].{Name:name, TriggerType:config.bindings[0].type}" -o table

If sync_archibus appears with TriggerType: timerTrigger, the deployment succeeded regardless of the SSL warning.

After deployment

The function will attempt to trigger immediately on cold start (via run_on_startup). Because SyncTableRegistry is empty at this point, the invocation will process 0 entities and exits early. Neither SyncWatermarks rows nor the SyncSchemaHistory table will be created yet — this is expected. Both are created automatically once a valid entity is confirmed (Step 6 below).

The SyncWatermarks and SyncSchemaHistory tables will also not exist yet at this point — they are only created once a valid entry is added to SyncTableRegistry and confirmed to exist in the Archibus API.

Deployment verification

The SyncTableRegistry table storage is created by bicep template. Another table called SyncLock is created by the function at startup. If SyncLock doesn’t appear after deployment, trigger one manual invocation before checking the rest of this list

First get the SUBSCRIPTION_ID with CLI using az account show --query id -o tsv
Use it in following CLI command:

az rest --method post --url "https://management.azure.com/subscriptions/<SUBSCRIPTION_ID>>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.Web/sites/<FUNCTION_APP_NAME>/hostruntime/admin/functions/sync_archibus?api-version=2022-03-01" --body "{\"input\":\"\"}"

After this, SyncLock should appear (and SyncWatermarks/SyncSchemaHistory too, after SyncTableRegistry has at least one valid entry, see Step 6).

Step 6. Populate SyncTableRegistry

The SyncTableRegistry table was created empty in Step 1 and the function will not sync any data until rows are added.

Navigate to Azure Portal > Storage account {FUNC_STORAGE_ACCOUNT} > Storage browser > Tables > SyncTableRegistry.
Click Add entity.
Set the following fields for each Archibus table you want to sync:
- PartitionKey = archibus (all lowercase, exact and see warning below)
- RowKey = the Archibus table name e.g. rm, bl, site
No additional columns are required.
Click Insert.
Repeat for each table. The function will pick up new entries on the next scheduled run (or immediately if triggered manually).

PartitionKey is case-sensitive

It must be exactly archibus (all lowercase). Rows with any other capitalization e.g. Archibus, ARCHIBUS are silently ignored and the function will log a WARNING and skip them.

Notes

Stop syncing a table - Delete its row from SyncTableRegistry. Existing data in the bronze and silver containers is preserved and nothing is deleted from the data lake.
Deferred table creation - The SyncWatermarks and the SyncSchemaHistory tables are only created after the first valid entity is confirmed; this means at least one row in SyncTableRegistry maps to a table that actually exists in the Archibus API. If SyncTableRegistry is empty, or every registered table is absent from the Archibus API, the function exits cleanly without creating either table.
Watch for trailing/leading whitespace in PartitionKey. The registry query is an exact-match filter (PartitionKey eq 'archibus') and a value like archibus (with a trailing space) looks identical to archibus in the Portal's table view, but won't match the query at all. The result is silent: no error, just TableRegistry: 0 table(s) registered for sync in the logs, even though the row is clearly visible in Storage Browser. If you're seeing 0 tables registered despite adding a row, re-type the PartitionKey value from scratch rather than trusting what's currently there.

Step 7. Verify the Deployment

Run through this checklist after Step 6 is complete and SyncTableRegistry has at least one row registered:


Check	Details
Azure Portal > Function App > Functions > sync_archibus > Monitor	Confirm at least one successful invocation after you added rows to `SyncTableRegistry`
Application Insights > Logs	Run the query below to check for exceptions
SyncWatermarks table	One row per registered entity; `status` = `success` or `no_changes`
SyncLock table	A row with `RowKey = sync_lock exists` and `locked = false` after the run completes
ADLS Gen2 bronze container	Partition folder(s) exist under: `bronze/<table-name>/year=YYYY/month=MM/day=DD/`

SyncWatermarks table itself will not exist until after the first successful sync cycle with a valid registry entry. If the table is absent, verify SyncTableRegistry has entries and the Archibus API is reachable.

Application Insights — exception check (KQL):

exceptions
| where timestamp > ago(1h)
| project timestamp, type, outerMessage, severityLevel
| order by timestamp desc

Find sync/delete failures

traces
| where severityLevel == 3
| where message startswith "Sync failed for" or message startswith "Deletion check failed for"
| order by timestamp desc

Update the Function (Re-deployment)

To deploy a new version of the function code, re-run whichever deployment method you used in Step 5.

If only an application setting changed, e.g. SYNC_SCHEDULE, PAGE_SIZE: update the value directly in Azure Portal → Function App → Configuration → Application settings → Save. The change takes effect immediately — no redeploy needed.
If only Bicep/infrastructure changed with no code changes: re-run Step 1 and the Bicep deployment is idempotent and will apply the infrastructure change safely.

Operator Configuration Reference

Post-deployment operational changes can be made without redeploying:


What to change	Where	How
Add a table to sync	`SyncTableRegistry` table	Add row: `PartitionKey=archibus, RowKey=<table-name>`
Remove a table from sync	`SyncTableRegistry` table	Delete the row. Existing lake data is preserved.
Change sync schedule	Function App > Configuration > `SYNC_SCHEDULE`	Update value (6-field cron) > Save
Change page size	Function App > Configuration > `PAGE_SIZE`	Update value > Save
Change the parallel workers	Function App > Configuration > `MAX_WORKERS`	Update value > Save. See MAX_WORKERS Performance Warning above.
Force full re-sync for an entity	`SyncWatermarks` table	Clear or delete the `updatedAfter` field on the entity's row
Adjust stale-lock timeout	`SyncLock` table	Edit `timeoutMinutes` on the sync_lock row (default: 180)
Acknowledge a schema error	`SyncWatermarks` table	Set `schemaStatus` to any value other than `incompatible_schema_change` e.g. `acknowledged`

incompatible_schema_change - What it means and how to clear it

This status is set when the function detects that Archibus has removed a column or changed a column's type.

When triggered, the function automatically performs a full sync and rebuilds the silver file to reflect the new schema. The status is a deliberate signal that a breaking change has occurred and it never auto-clears.

How to investigate

Open {FUNC_STORAGE_ACCOUNT} > Tables > SyncWatermarks> find the affected entity row.
Read schemaRemovedColumns to see which column was removed (this field is intentionally preserved across subsequent syncs so you can see the original cause at a glance).
For full drift history, open SyncSchemaHistory table, then filter by PartitionKey = <entity-name> and then look for rows with severity = error

How to acknowledge

Verify that downstream consumers (reports, Synapse, Databricks, etc.) have been updated to handle the schema change.
Navigate to {FUNC_STORAGE_ACCOUNT} > Tables > SyncWatermarks
Open the affected entity row and then set schemaStatus to any value other than incompatible_schema_change e.g. acknowledged
The next clean sync will automatically reset schemaStatus to ok.

Scenario Verification

Use this section to verify the system is behaving correctly for each data-change scenario.

Storage accounts referenced below:

{FUNC_STORAGE_ACCOUNT} - function runtime storage (Azure Table Storage: SyncWatermarks, SyncTableRegistry, SyncLock, SyncSchemaHistory)
{ADLS_STORAGE_ACCOUNT} - ADLS Gen2 data lake (bronze and silver Parquet files)

Scenario 1. First Full Sync

Applies to: first deployment, or after clearing updatedAfter on a watermark row to force a re-sync.


What to check	Where	Expected
Invocation completed	Function App > Monitor	Status: Success; duration reflects the number of registered entities.
Run-lock released	`{FUNC_STORAGE_ACCOUNT}` > Tables > SyncLock	Row `RowKey = sync_lock exists`; `locked = false` after sync completes.
Watermark created per entity	`{FUNC_STORAGE_ACCOUNT}` > Tables > SyncWatermarks	One row per registered entity; `status = success`; `updatedAfter` is a recent timestamp; `lastDeleteStatus = success` or `no_deletions`.
Bronze data written	`{ADLS_STORAGE_ACCOUNT}` > bronze container > `<table>/year=YYYY/month=MM/day=DD/`	One or more `.parquet` files present containing all records fetched from Archibus.
Silver data written	`{ADLS_STORAGE_ACCOUNT}` > silver container > `<table>/<table>_current.parquet`	File exists; last-modified timestamp matches the sync run time.

SyncWatermarks table itself is created on this first sync (not just the row). SyncSchemaHistory is also created and seeded with the initial schema baseline on the first sync for each entity.

Scenario 2. Record Updated or Added in Archibus

Applies to: next scheduled sync after a record is created or modified in Archibus.


What to check	Where	Expected
Watermark advanced	`{FUNC_STORAGE_ACCOUNT}` > Tables > `SyncWatermarks` > entity row	`updatedAfter` is newer than the previous sync; `lastRecordCount` > 0; `status = success`
New bronze file written	`{ADLS_STORAGE_ACCOUNT}` > bronze > `<table>/year=.../month=.../day=.../`	A new `.parquet` file appears from the previous sync; it contains only the changed or added record(s)
Silver updated	`{ADLS_STORAGE_ACCOUNT}` > silver > `<table>/<table>_current.parquet`	File last-modified timestamp is newer than the previous sync

If no records have changed since the last sync: status = no_changes and no new bronze file is written, then this is correct and expected behavior.

Scenario 3. Record Deleted in Archibus

Applies to: deletion detection runs inline after each entity's sync on every cycle.


What to check	Where	Expected
Deletion detected	`{FUNC_STORAGE_ACCOUNT}` > Tables > `SyncWatermarks` > entity row	`lastDeleteStatus = success; lastDeleteCount` > 0; `lastDeleteCheckTimestamp updated`
Deleted record archived to bronze	`{ADLS_STORAGE_ACCOUNT}` > bronze > `<table>_deleted/<table>_deleted_current.parquet`	File exists and contains the deleted record (identified by its UUID); row count increased by the number of deletions
Record removed from silver	`{ADLS_STORAGE_ACCOUNT}` > silver > `<table>/<table>_current.parquet`	Deleted record's UUID is no longer present; total row count decreased accordingly

If no deletions occurred then the following will be true. The deletion archive file is unchanged.

lastDeleteStatus = no_deletions
lastDeleteCount = 0

Scenario 4. Column Removed or Type Changed in Archibus

When the Archibus schema API returns a different column set or type compared to the stored watermark. The function detects this before fetching records, performs a full sync, and rebuilds silver.


What to check	Where	Expected
Schema error flagged	`{FUNC_STORAGE_ACCOUNT}` > Tables > `SyncWatermarks` > entity row	`schemaStatus = incompatible_schema_change; schemaRemovedColumns` shows the removed column name (or check `typeChanges` in `SyncSchemaHistory` for type changes).
Drift event recorded	`{FUNC_STORAGE_ACCOUNT}` > Tables >`SyncSchemaHistory` > filter `PartitionKey = <entity>`	A row with `severity = error; columnsRemoved` or `typeChanges` populated; `detectedAt` is the sync timestamp.
Silver rebuilt	`{ADLS_STORAGE_ACCOUNT}` > silver > `<table>/<table>_current.parquet`	File last-modified updated; the removed column is no longer present in the Parquet schema.
Warning in App Insights	App Insights > Logs > KQL: `traces \| where message contains "incompatible_schema_change" \| order by timestamp desc`	Warning-level trace for the affected entity naming the removed column or type change.

Action required: incompatible_schema_change never auto-clears. See the acknowledgment steps in the Operator Configuration Reference above.

Scenario 5. Deleted Infrastructure Recovery

Self-healing behavior after manual deletion:

Deleted bronze/silver ADLS containers > auto-recreated by the function on next sync
Deleted SyncWatermarks/SyncSchemaHistory > auto-recreated on next valid sync cycle
Deleted SyncTableRegistry > only recreated by re-running Step 1 (Bicep deployment); entries must be re-added manually
Deleted SyncLock>> auto-recreated on next sync invocation