Data Lake Integration (Azure) - DevOps Deploy

Last updated
Save as PDF

Web Central REST API

Overview

The instructions in this article are for System Integrators who want to deploy the Archibus - Azure Data Lake integration with DevOps. When you have completed these instructions, you will have a running Azure Function that automatically syncs Archibus space data into an ADLS Gen2 account using a medallion (bronze/silver) architecture.

Contents

Prerequisites
Step 1. Configure Azure DevOps Pipeline Variables
Step 2. Run the Pipeline
Step 3. Populate SyncTableRegistry
Step 4. Verify the Deployment
Update the Function (Re-deployment)
Operator Configuration Reference
Scenario Verification

Prerequisites

Ensure the following are in place before running the pipeline.

Deployment Bundle

Before starting, download the version's deployment bundle (archibus-sync-vX.Y.Z.zip). This bundle contains everything referenced throughout this article:

functionapp.zip - the built function source (dependencies are installed by the pipeline itself, not pre-bundled)
infra/main.bicep and infra/main.json - the infrastructure template
client-deploy-pipeline.yml - the Azure DevOps pipeline definition and this article walks you through setting up

Extract the zip locally before continuing.

Azure

Azure subscription and an existing resource group where all resources will be created.
User Assigned Managed Identity (UAMI) with Contributor and User Access Administrator roles on the resource group. The pipeline does not create this identity; it only assigns storage-level RBAC roles to it automatically. User Access Administrator is required on the resource group before the pipeline runs, or the RBAC assignment step will fail with a 403.

Why two roles?

Contributor lets the pipeline create resources. User Access Administrator lets the pipeline assign Storage Blob Data Contributor and Storage Table Data Contributor roles to the UAMI on the new storage accounts.

Azure DevOps

An Azure DevOps project with a service connection configured for the target subscription. The service principal behind the connection must have Owner or User Access Administrator on the resource group.
Extract client-deploy-pipeline.yml from the downloaded bundle and commit it to a Git repository accessible from your Azure DevOps project, for example, Azure Repos or GitHub both work. This can be a minimal, otherwise-empty repo; the pipeline uses checkout: none and never reads anything else from it. Then it gets the actual function code and infrastructure template from the bundle at run time instead, not from this repo.
Upload the bundle zip (archibus-sync-vX.Y.Z.zip) as a Secure File: Project Settings > Pipelines > Library > Secure files. Note the name you give it as you'll reference it as the BUNDLE_SECURE_FILE_NAME variable in Step 1.
Create a pipeline in Azure DevOps pointing at client-deploy-pipeline.yml in your repo. It runs on a standard Microsoft-hosted agent and no self-hosted agent is required.

Archibus OIDC Credentials

The function authenticates to the Archibus REST API using the OAuth2 client credentials flow via your OIDC identity provider.

Contact your Archibus administrator to obtain the following:

ARCHIBUS_CLIENT_ID - the OAuth2 client ID registered in your OIDC identity provider.
ARCHIBUS_CLIENT_SECRET - the corresponding client secret.
ARCHIBUS_TOKEN_URL - your OIDC token endpoint, e.g. https://<your-idp-domain>/oauth2/token
ARCHIBUS_BASE_URL - the base URL of your Archibus instance e.g. https://<your-instance>/archibus

These credentials must be stored as secret pipeline variables (see Step 1 below). They are never stored in source control.

Network Connectivity

The Azure Function requires outbound internet access to both:

ARCHIBUS_BASE_URL - the Archibus REST API endpoint.
ARCHIBUS_TOKEN_URL - the OIDC token endpoint.

If your Archibus instance is behind a VPN, corporate proxy, or private network, you must configure VNet integration or outbound firewall rules in the Function App before the sync will work. This is an infrastructure concern outside the scope of the pipeline make sure you engage your Network/Cloud team.

Step 1. Configure Azure DevOps Pipeline Variables

Navigate to your Azure DevOps pipeline (created in the prerequisite) > Edit > Variables and add the following. All variables are plain text unless marked as secret.

Bundle Variable


Variable	Example value	Notes
`BUNDLE_SECURE_FILE_NAME`	`archibus-sync-v1.0.0.zip`	Must match the name you gave the bundle zip (with extension) when uploading it as a Secure File (Prerequisites)

Infrastructure Variables


Variable	Example value	Notes
`SERVICE_CONNECTION_NAME`	`my-azure-connection`	Must have Owner or User Access Administrator on the resource group.
`RESOURCE_GROUP_NAME`	`rg-archibus-sync-dev`	Must already exist before the pipeline runs.
`FUNCTION_APP_NAME`	`func-archibus-sync-dev`	2–60 chars, letters/digits/hyphens only.
`MANAGED_IDENTITY_NAME`	`id-archibus-sync-dev`	Name of the existing UAMI created as a prerequisite; 3–128 chars, letters/digits/hyphens/underscores.
`FUNC_STORAGE_ACCOUNT`	`archibusfuncstor`	Function runtime + Table Storage account. 3–24 chars, lowercase letters and digits only (no hyphens). Must be globally unique across all of Azure.
`ADLS_STORAGE_ACCOUNT`	`archibussyncadls`	ADLS Gen2 account for bronze/silver data. Same naming rules. Must be globally unique across all of Azure.
`ADLS_BRONZE_CONTAINER`	`bronze`	Blob container for raw (append-only) Parquet data. 3–63 chars, lowercase letters, numbers, and hyphens only. Must start and end with a letter or number — no leading/trailing or consecutive hyphens.
`ADLS_SILVER_CONTAINER`	`silver`	Blob container for current-state Parquet data. 3–63 chars, lowercase letters, numbers, and hyphens only. Must start and end with a letter or number, no leading/trailing or consecutive hyphens.
`WATERMARK_TABLE`	`SyncWatermarks`	Azure Table for sync progress tracking. Must differ from `SYNC_TABLE_REGISTRY`. 3–63 chars, start with letter.
`SYNC_TABLE_REGISTRY`	`SyncTableRegistry`	Azure Table listing which Archibus tables to sync. Must differ from `WATERMARK_TABLE`. Same naming rules.

These are pipeline input variables, not raw Function App settings. The pipeline itself constructs the actual app settings (ADLS_ACCOUNT_URL, FUNC_STORAGE_TABLE_URL) from ADLS_STORAGE_ACCOUNT/FUNC_STORAGE_ACCOUNT during the deploy steps, so you never need to build those URLs yourself when using the pipeline (unlike the manual/no-ADO deployment path, where they must be entered directly).

Function Behavior Variables


Variable	Default	Notes
`PAGE_SIZE`	`500`	Number of records fetched per API call. Maximum 10,000. Larger values reduce API round-trips but increase memory pressure.
`SYNC_SCHEDULE`	`0 0 /4 * *`	6-field Azure Functions cron expression (seconds field included). Minimum interval: 10 minutes.
`ARCHIBUS_BASE_URL`	—	Base URL for your Archibus instance. e.g. `https://<instance>/archibus`
`ARCHIBUS_TOKEN_URL`	—	OIDC token endpoint URL

MAX_WORKERS - Parallel API Workers

MAX_WORKERS controls how many Archibus tables are synced in parallel. For each worker, the function opens a concurrent connection to the Archibus REST API. The default is 4.

Do not set MAX_WORKERS as a pipeline variable, as it has no effect.

The pipeline's Stage 4 (Provision Function App) does not inject MAX_WORKERS into the Function App's application settings. If you set it in Azure DevOps Variables, it will be silently ignored and the function will always use the default of 4.

To use a non-default value, set it manually after deployment, complete the following:

Go to the Azure Portal.
Select Function App > Settings > Environment variables.
Add/update MAX_WORKERS
Save the settings.

Performance Warning

Increasing MAX_WORKERS raises the number of simultaneous connections to the Archibus REST API. Setting it too high can overwhelm the Archibus API server or corporate proxy, causing HTTP 429 (rate limit) or timeout errors. Start with the default of 4 and increase cautiously, monitoring Archibus API response times before and after each change.

Secret Variables (mark as secret in Azure DevOps)


Variable	Description
ARCHIBUS_CLIENT_ID	OAuth2 client ID for the Archibus OIDC integration
ARCHIBUS_CLIENT_SECRET	OAuth2 client secret — treat as a password

AZURE_CLIENT_ID is not a pipeline variable. It is automatically injected from the Bicep deployment output (the UAMI client ID) during the deploy steps. You do not need to set it.

APPLICATIONINSIGHTS_CONNECTION_STRING is not a pipeline variable, and the pipeline reads it directly from the Application Insights resource created by the Bicep deployment and sets it automatically.

Naming Constraints

The pipeline validates these constraints before deploying:

Storage account names: 3–24 characters, lowercase letters and digits only — no hyphens, underscores, or uppercase. Must be globally unique across all Azure storage accounts worldwide. If the name is taken, choose a different one.
Table names: 3–63 characters, must start with a letter, letters, and digits only (no hyphens).
WATERMARK_TABLE and SYNC_TABLE_REGISTRY must be different from each other and from the internal SyncLock table name.
SYNC_SCHEDULE minimum interval: 10 minutes.

Step 2. Run the Pipeline

Trigger the pipeline manually in Azure DevOps. Unlike a typical multi-stage ADO pipeline, client-deploy-pipeline.yml runs as a single job with sequential steps. It's a self-contained file meant to be handed to a client with no access to any other pipeline infrastructure, so everything happens in one place.

Stage 1 - Fetch and extract the bundle

Downloads the bundle from the Secure File you uploaded (BUNDLE_SECURE_FILE_NAME)
Extracts it and there is no git checkout of any kind occurs (checkout: none)

Stage 2 - Build from source

Installs Python dependencies from requirements.txt and rebuilds the deployable functionapp.zip. The bundle ships source only, so this step does what the internal azure-pipelines.yml's Build stage does, just at deploy time instead of ahead of time.

Stage 3 - Validate

Validates all pipeline variable formats (naming constraints, schedule, table uniqueness).
Checks whether the resource group and existing Azure resources are accessible.
Run the az deployment group what-if and az deployment group validate against the Bicep template. At this point, no changes have been made to the Azure resources.

Stage 4 - Deploy Infrastructure

Deploys the Bicep template (infra/main.bicep), which creates or updates:

Function runtime storage account (Standard_LRS, HNS disabled) is required for Azure Table Storage compatibility.
SyncTableRegistry table in the function runtime storage account.
ADLS Gen2 storage account (HNS enabled) with bronze and silver blob containers.
Application Insights instance.

This step does not create the UAMI - It must already exist as a Prerequisite. This is also where RBAC role assignment happens and immediately after the Bicep deployment, in the same step: Storage Blob Data Contributor is assigned on the ADLS Gen2 account, and Storage Table Data Contributor on the function runtime storage account, both to the existing UAMI.

The SyncLock table is not created by Bicep or the pipeline. It is auto-created by the function on its first sync run using the Azure Data Tables SDK. No manual action is needed, SyncWatermarks and SyncSchemaHistory follows the same lazy-creation pattern.

If you see the message "The specified container is being deleted. Try operation later" when this step recreates bronze/silver containers. This happens when the target storage account has Container soft delete enabled and a same-named container was deleted recently (e.g. during earlier testing). The name stays reserved for the retention window (commonly 7 days). It is not the Bicep template deleting anything; nothing in this pipeline issues a delete. Restore the soft-deleted container (az storage container restore --deleted-version <version>) or disable container soft-delete on the account if it's a disposable test account.

Stage 5 - Provision Function App

Creates the Function App (Flex Consumption plan, Python 3.14, Linux) if it doesn't already exist.
Attaches the UAMI to the Function App.
Sets all application settings including (AZURE_CLIENT_ID is read from the Bicep output automatically) andAZURE_CLIENT_ID (read from the Bicep deployment output) and APPLICATIONINSIGHTS_CONNECTION_STRING (read from the Bicep-created Application Insights resource) both resolved automatically and neither is a pipeline variable.

The Function App uses the Flex Consumption hosting plan, which scales to zero between invocations. The pipeline uses the AzureFunctionApp@2 task (not @1) and note that the Flex Consumption does not have a Kudu/SCM endpoint, and the older @1 task will fail with a 404.

Stage 6 - Deploy Function

Deploys the function code zip via AzureFunctionApp@2
Triggers an initial sync run via the Azure Functions admin API immediately after deploy.

Why an explicit trigger call, rather than relying on the deploy to restart the app. Flex Consumption is scale-to-zero, and a runFromPackage-style deploy doesn't guarantee an immediate host restart if the app was already warm. The pipeline force-triggers sync_archibus directly via the ARM admin API after deploying, rather than relying on the deploy itself to restart the host.

Initial sync fires before SyncTableRegistry is populated. The pipeline triggers the function at the end of Stage 5. On a first deployment, SyncTableRegistry is still empty, so the function runs but processes 0 entities. You will see an invocation in the Function Monitor with 0 watermarks written, and this is expected behavior, not an error. Proceed to Step 3 to register the tables you want to sync. SyncWatermarks and SyncSchemaHistory tables will not exist yet at this point as they are only created after a valid entry is added to SyncTableRegistry and confirmed to exist in the Archibus API; this is expected behavior.

Step 3. Populate SyncTableRegistry

The SyncTableRegistry table is created empty by the pipeline. The function will not sync any data until rows are added.

Navigate to Azure Portal > Storage account {FUNC_STORAGE_ACCOUNT} > Storage browser > Tables > SyncTableRegistry
Click Add entity.
Set the following fields for each Archibus table you want to sync:
- PartitionKey = archibus (all lowercase and see the warning below)
- RowKey = the Archibus table name e.g. rm, bl, site
No additional columns are required.
Click Insert.
Repeat for each table.
The function will pick up new entries on the next scheduled run (or immediately if triggered manually).

After a valid entry is added and the next sync cycle runs, the function automatically creates SyncWatermarks (with all required columns) and SyncSchemaHistory. No manual table creation is needed. If the entry's table name does not exist in the Archibus API, these tables are still not created; they appear only on the first confirmed valid sync.

Caution

PartitionKey is case-sensitive - It must be exactly archibus (all lowercase). Rows with any other capitalization (e.g. Archibus, ARCHIBUS) are silently ignored and the function logs a WARNING and skips them.
Watch for trailing/leading whitespace in PartitionKey - The registry query is an exact-match filter; for example, a value like archibus (with a trailing space) looks identical to archibus in the Portal's table view but won't match at all. The result is silent: no error, just 0 tables registered in the logs, even though the row is clearly visible in Storage Browser. If you're seeing 0 tables registered despite adding a row, re-type the PartitionKey value from scratch rather than trusting what's currently there.
Stop syncing a table - Delete its row from SyncTableRegistry. Existing data in the bronze and silver containers is preserved and nothing is deleted from the data lake.

Step 4. Verify the Deployment

Run through this checklist after the pipeline completes and SyncTableRegistry has at least one row registered:

Azure Portal > Function App > Functions > sync_archibus > Monitor: confirm at least one successful invocation after you added rows to SyncTableRegistry
Application Insights > Logs: run the query below to check for exceptions
SyncWatermarks table: one row per registered entity, status = success or no_changes
SyncLock table: a row with RowKey = sync_lock exists and locked = false after the run completes
ADLS Gen2 bronze container: partition folder(s) exist under bronze/<table-name>/year=YYYY/month=MM/day=DD/

SyncWatermarks table itself will not exist until after the first successful sync cycle with a valid registry entry. If the table is absent, verify SyncTableRegistry it has entries and the Archibus API is reachable.

Application Insights — exception check (KQL):

exceptions
| where timestamp > ago(1h)
| project timestamp, type, outerMessage, severityLevel
| order by timestamp desc

Find logs for sync/delete failures

traces
| where severityLevel == 3
| where message startswith "Sync failed for" or message startswith "Deletion check failed for"
| order by timestamp desc

Update the Function (Re-deployment)

After any code change, a new bundle must be published and re-uploaded as the Secure File (this pipeline builds from the bundle, not from a live source repo), then re-run the pipeline using a manual trigger.

If only a pipeline variable changed e.g. SYNC_SCHEDULE, PAGE_SIZE: you can either re-run the pipeline, or update the value directly in Azure Portal > Function App > Settings > Environment variables > Save. The change takes effect immediately without a redeploy.
If only Bicep/infrastructure changed with no code changes: re-run the pipeline and the infrastructure deployment step is idempotent and will apply the infrastructure change safely.

Operator Configuration Reference

Post-deployment operational changes can be made without re-running the pipeline:


What to change	Where	How
Add a table to sync	`SyncTableRegistry` table	Add row: `PartitionKey=archibus`, `RowKey=<table-name>`
Remove a table from sync	`SyncTableRegistry` table	Delete the row. Existing lake data is preserved.
Change sync schedule	Function App > Settings > Environment variables > `SYNC_SCHEDULE`	Update value (6-field cron) > Save
Change page size	Function App > Settings > Environment variables > `PAGE_SIZE`	Update value > Save
Change parallel workers	Function App > Settings > Environment variables > `MAX_WORKERS`	Update value > Save. See MAX_WORKERS warning above.
Force full re-sync for an entity	`SyncWatermarks` table	Clear or delete the `updatedAfter` field on the entity's row
Adjust stale-lock timeout	`SyncLock` table	Edit `timeoutMinutes` on the `sync_lock` row (default: 180 minutes)
Acknowledge a schema error	`SyncWatermarks` table	Set schemaStatus to any value other than `incompatible_schema_change` e.g. `acknowledged`

incompatible_schema_change — what it means and how to clear it

This status is set when the function detects that Archibus has removed a column or changed a column's type. When triggered, the function automatically performs a full sync and rebuilds the silver file to reflect the new schema. The status is a deliberate signal that a breaking change has occurred, and it never auto-clears.

How to investigate

Open {FUNC_STORAGE_ACCOUNT} > Tables > SyncWatermarks > find the affected entity row.
Read schemaRemovedColumns to see which column was removed (this field is intentionally preserved across subsequent syncs so you can see the original cause at a glance).
For full drift history, open SyncSchemaHistory table > filter by PartitionKey = <entity-name> > look for rows with severity = error

How to acknowledge

Verify that downstream consumers (reports, Synapse, Databricks, etc.) have been updated to handle the schema change.
Navigate to {FUNC_STORAGE_ACCOUNT} > Tables > SyncWatermarks
Open the affected entity row > set schemaStatus to any value other than incompatible_schema_change (e.g. acknowledged)
The next clean sync will automatically reset schemaStatus to ok

Scenario Verification

Use this section to verify the system is behaving correctly for each data-change scenario. For each scenario, the tables below show what to check, where to look, and what a healthy result looks like.

Storage accounts referenced below:

{FUNC_STORAGE_ACCOUNT} - function runtime storage (Azure Table Storage: SyncWatermarks, SyncTableRegistry, SyncLock, SyncSchemaHistory)
{ADLS_STORAGE_ACCOUNT} - ADLS Gen2 data lake (bronze and silver Parquet files)

Scenario 1 - First Full Sync

Applies to: first deployment, or after clearing updatedAfter on a watermark row to force a re-sync.


What to check	Where	Expected
Invocation completed	Function App > Monitor	Status: Success; duration reflects the number of registered entities
Run-lock released	`{FUNC_STORAGE_ACCOUNT}` > Tables > SyncLock	Row `RowKey = sync_lock exists`; `locked = false` after sync completes
Watermark created per entity	`{FUNC_STORAGE_ACCOUNT}` > Tables > SyncWatermarks	One row per registered entity; `status = success`; `updatedAfter` is a recent timestamp; `lastDeleteStatus = success` or `no_deletions`
Bronze data written	`{ADLS_STORAGE_ACCOUNT}` > bronze container > `<table>/year=YYYY/month=MM/day=DD/`	One or more `.parquet` files present containing all records fetched from Archibus
Silver data written	`{ADLS_STORAGE_ACCOUNT}` > silver container > `<table>/<table>_current.parquet`	File exists; last-modified timestamp matches the sync run time

SyncWatermarks table itself is created on this first sync (not just the row). Similarly, add a row for SyncSchemaHistory and it is created and seeded with the initial schema baseline on the first sync for each entity.

Scenario 2 - Record Updated or Added in Archibus

Applies to: next scheduled sync after a record is created or modified in Archibus.


What to check	Where	Expected
Watermark advanced	`{FUNC_STORAGE_ACCOUNT}` > Tables > `SyncWatermarks` > entity row	`updatedAfter` is newer than the previous sync; `lastRecordCount` > 0; `status = success`
New bronze file written	`{ADLS_STORAGE_ACCOUNT}` > bronze > `<table>/year=.../month=.../day=.../`	A new `.parquet` file appeared since the previous sync; it contains only the changed or added record(s)
Silver updated	`{ADLS_STORAGE_ACCOUNT}` > silver > `<table>/<table>_current.parquet`	File last-modified timestamp is newer than the previous sync

If no records have changed since the last sync: status = no_changes and no new bronze file is written, then this is correct and expected behavior.

Scenario 3 - Record Deleted in Archibus

Applies to: deletion detection runs inline after each entity's sync on every cycle.


What to check	Where	Expected
Deletion detected	{FUNC_STORAGE_ACCOUNT} > Tables > SyncWatermarks > entity row	lastDeleteStatus = success; lastDeleteCount > 0; lastDeleteCheckTimestamp updated
Deleted record archived to bronze	{ADLS_STORAGE_ACCOUNT} > bronze > <table>_deleted/<table>_deleted_current.parquet	File exists and contains the deleted record (identified by its UUID); row count increased by the number of deletions
Record removed from silver	{ADLS_STORAGE_ACCOUNT} > silver > <table>/<table>_current.parquet	Deleted record's UUID is no longer present; total row count decreased accordingly

If no deletions occurred, then the following should be true. The deletion archive file is unchanged.

lastDeleteStatus = no_deletions
lastDeleteCount = 0

Scenario 4 - Column Removed or Type Changed in Archibus

Applies to: when the Archibus schema API returns a different column set or type compared to the stored watermark. The function detects this before fetching records, performs a full sync, and rebuilds silver.


What to check	Where	Expected
Schema error flagged	{FUNC_STORAGE_ACCOUNT} > Tables > SyncWatermarks > entity row	schemaStatus = incompatible_schema_change; schemaRemovedColumns shows the removed column name (or check typeChanges in SyncSchemaHistory for type changes).
Drift event recorded	{FUNC_STORAGE_ACCOUNT} > Tables > SyncSchemaHistory > filter PartitionKey = <entity>	A row with severity = error; columnsRemoved or typeChanges populated; detectedAt is the sync timestamp
Silver rebuilt	{ADLS_STORAGE_ACCOUNT} > silver > <table>/<table>_current.parquet	File last-modified updated; the removed column is no longer present in the Parquet schema.
Warning in App Insights	App Insights > Logs > KQL: traces \| where message contains "incompatible_schema_change" \| order by timestamp desc	Warning-level trace for the affected entity naming the removed column or type change.

Action required:

incompatible_schema_change never auto-clears. See the acknowledgment steps in the Operator Configuration Reference above.

Scenario 5 - Deleted Infrastructure Recovery covering self-healing behavior after manual deletion

Deleted bronze/silver ADLS containers > auto-recreated by the function on next sync
Deleted SyncWatermarks/SyncSchemaHistory > auto-recreated on next valid sync cycle
Deleted SyncTableRegistry > only recreated by re-running the pipeline (entries must be re-added manually)
Deleted SyncLock > auto-recreated on next sync invocation