Skip to main content
Eptura Knowledge Center

Data Lake Integration (Azure) - Manual Deploy

  1. Last updated
  2. Save as PDF
Web Central REST API

Overview

The instructions in this article are for System Integrators who want to deploy the Archibus - Azure Data Lake integration without Azure DevOps. When you have completed these instructions, you will have a running Azure Function that automatically syncs Archibus space data into an ADLS Gen2 account using a medallion (bronze/silver) architecture — deployed entirely through the Azure Portal and your local machine. After deployed, operational management is the same regardless of how you deployed. These instructions include the full Operator Configuration Reference and Scenario Verification sections. 

Contents

Prerequisites

Ensure the following are in place before starting.

Azure

  • An Azure subscription and an existing resource group where all resources will be created.
  • User Assigned Managed Identity (UAMI) with Contributor and User Access Administrator roles on the resource group.

Why two roles?

Contributor lets you create resources. User Access Administrator lets you assign Storage Blob Data Contributor and Storage Table Data Contributor roles to the UAMI on the new storage accounts; without it Step 3 will fail with a 403.

Tools

Pick one for Step 5 (code deployment)

Archibus OIDC Credentials

The function authenticates to the Archibus REST API using the OAuth2 client credentials flow via your OIDC identity provider.

Contact your Archibus administrator to obtain the following:

  • ARCHIBUS_CLIENT_ID - the OAuth2 client ID registered in your OIDC identity provider
  • ARCHIBUS_CLIENT_SECRET - the corresponding client secret
  • ARCHIBUS_TOKEN_URL - your OIDC token endpoint e.g. https://<your-idp-domain>/oauth2/token
  • ARCHIBUS_BASE_URL - the base URL of your Archibus instance e.g. https://<your-instance>/archibus

These credentials will be entered as application settings in Step 4 and they are never stored in source control.

Network Connectivity

The Azure Function requires outbound internet access to both:

  • ARCHIBUS_BASE_URL - the Archibus REST API endpoint
  • ARCHIBUS_TOKEN_URL - the OIDC token endpoint

If your Archibus instance is behind a VPN, corporate proxy, or private network, you must configure VNet integration or outbound firewall rules in the Function App before the sync will work. This is an infrastructure issue outside the scope of these instructions; make sure you contact your network/cloud team.

Step 1. Deploy Infrastructure (Bicep via Azure Portal)

This step creates the storage accounts, tables, containers, Application Insights, and UAMI using the existing Bicep template.

  1. Navigate to Azure Portal > your Resource Group > Create.
  2. Search for "Custom deployment".
  3. Select Deploy a custom template.
  4. In the editor, click Build your own template.
  5. Click Load file and upload infra/main.bicep from the repository and then Save.
  6. Complete the Bicep parameters using the table below.
Parameter What it represents Naming rules
funcStorageAccountName Function runtime + Table Storage account 3–24 chars, lowercase letters and digits only (no hyphens). Must be globally unique across all of Azure.

adlsStorageAccountName

 ADLS Gen2 data lake account Same rules. Must be globally unique.

bronzeContainerName

 Bronze blob container name e.g. bronze

silverContainerName

 Silver blob container name e.g. silver

watermarkTableName

 SyncWatermarks table name

3–63 chars, must start with a letter.

e.g. SyncWatermarks

syncTableRegistryName

 SyncTableRegistry table name

Must differ from watermark table. Same naming rules.

e.g. SyncTableRegistry

managedIdentityName

 UAMI resource name 3–128 chars, letters/digits/hyphens/underscores

appInsightsName

 Application Insights resource name —

location

 Azure region Defaults to resource group location if omitted

The SyncLock table is NOT created by Bicep. It is auto-created by the function on its first sync run, so no manual action is needed.

The SyncSchemaHistory and SyncWatermarks tables are NOT pre-created by Bicep. They are auto-created by the function, but only after the first valid entity is confirmed, as a row in SyncTableRegistry whose table name actually exists in the Archibus API. No manual action is needed.

  1. Click Review + Create.
  2. Azure validates the template and configuration:
    • Checks for syntax errors in the Bicep file.
    • Verifies parameter values and resource naming rules.
  3. Review the deployment summary:
    • Confirm subscription and resource group.
    • Verify all resource names and configurations.
  4. After validation passes, click Create to start the deployment.
  5. Azure provisions the resources (Storage Accounts, Tables, Containers, Application Insights, UAMI).
  6. Monitor deployment progress from the Deployments section until status shows Success.

Step 2. Create the Function App

  1. Navigate to Azure Portal.
  2. Create a resource.
  3. Search "Function App" and click the Create.
  4. Complete following settings.
Field Value

Hosting plan

 Flex Consumption

Runtime stack

 Python

Version

 3.14

Operating System

 Linux

Storage account

 Select the funcStorageAccountName storage account created in Step 1.

Application Insights

 Select the Application Insights instance created in Step 1.
  1. Complete the remaining wizard fields (subscription, resource group, name, region).
  2. Click Review + create and then click the Create button.

Attach the UAMI after creation:

  1. Go to the new Function App > Identity (left menu) > User assigned tab.
  2. Click + Add.
  3. Select the UAMI created/used in Step 1 and then Add.

The SyncLock table is NOT created by Bicep. It is auto-created by the function on its first sync run and no manual action is needed.

The SyncSchemaHistory and SyncWatermarks tables are NOT pre-created by Bicep. They are auto-created by the function, but only after the first valid entity is confirmed and a row in SyncTableRegistry whose table name actually exists in the Archibus API. No manual action is needed.

Step 3. Assign RBAC Roles

The Function App's UAMI needs two role assignments, one on each storage account.

For each row in the table below:

  1. Navigate to the storage account in Azure Portal.
  2. Click Access Control (IAM).
  3. Click + Add and then Add role assignment.
  4. Select the role.
  5. Click Next.
  6. Assign access to Managed Identity.
  7. Select the UAMI.
  8. Review + assign.
Storage account Role to assign Assignee

adlsStorageAccountName

(ADLS Gen2, HNS enabled)

 Storage Blob Data Contributor UAMI

funcStorageAccountName

(function runtime storage)

 Storage Table Data Contributor UAMI

Step 4. Configure Application Settings

  1. Navigate to the Function App > Configuration (left menu) > Application settings tab.
  2. Click + New application setting for each entry in the table below.
  3. Click Save after all settings are added and the Function App restarts automatically.
Setting Name Value

AZURE_CLIENT_ID

Client ID of the UAMI and you will find it in the Portal:  

UAMI → Overview → Client ID

ARCHIBUS_BASE_URL

 e.g. https://<your-instance>/archibus

ARCHIBUS_CLIENT_ID

 OAuth2 client ID from your OIDC identity provider.

ARCHIBUS_CLIENT_SECRET

 OAuth2 client secret — treat as a password.

ARCHIBUS_TOKEN_URL

 OIDC token endpoint URL

FUNC_STORAGE_ACCOUNT

 Name of the function runtime storage account from Step 1.

ADLS_STORAGE_ACCOUNT

 Name of the ADLS Gen2 storage account from Step 1.

ADLS_BRONZE_CONTAINER

Bronze container name

e.g. bronze

ADLS_SILVER_CONTAINER

Silver container name

e.g. silver

WATERMARK_TABLE

Watermark table name

e.g. SyncWatermarks

SYNC_TABLE_REGISTRY

Registry table name

e.g. SyncTableRegistry

SYNC_SCHEDULE

6-field Azure Functions cron expression

e.g. 0 0 */4 * * *

Minimum interval: 10 minutes.

PAGE_SIZE

 500 (max 10,000 — larger values reduce API round-trips but increase memory pressure)
MAX_WORKERS (optional)

 This controls how many Archibus tables are synced in parallel. It defaults to 4 if not set. To use a non-default value, add it here as an additional application setting.

Performance Warning

Increasing MAX_WORKERS raises the number of simultaneous connections to the Archibus REST API. Setting it too high can overwhelm the Archibus API server or corporate proxy, causing HTTP 429 (rate limit) or timeout errors. Start with the default of 4 and increase cautiously, monitoring Archibus API response times before and after each change.

Step 5.  Deploy the Function Code

Choose one of the three options below.

Option A - VS Code (recommended)

  1. In VS Code, open the repository folder.
  2. Install the Azure Functions extension (search in the Extensions panel)
  3. Sign in to Azure via the Azure extension (left sidebar)
  4. In the Azure panel, right-click the function project folder and select Deploy to Function App.
  5. Select the Function App created in Step 2.
  6. Confirm the deployment.

Option B - Azure CLI

# Install dependencies into the package folder
pip install -r requirements.txt --target .python_packages/lib/site-packages
# Create the deployment zip
zip -r function.zip . -x "*.git*" -x ".venv/*" -x "__pycache__/*"
# Deploy
az functionapp deployment source config-zip \
  --resource-group <RESOURCE_GROUP_NAME> \
  --name <FUNCTION_APP_NAME> \
  --src function.zip

Option C - Azure Portal (Deployment Center)

  1. Navigate to Function App > Deployment Center (left menu)
  2. Select your source e.g. External Git or ZIP Deploy.
  3. For ZIP Deploy: upload the packaged zip file directly in the portal.

After the deployment,  the function will attempt to trigger immediately. Because SyncTableRegistry is empty at this point, the invocation will process 0 entities and exits early. Neither SyncWatermarks rows nor the SyncSchemaHistory table will be created yet, and this is expected behavior. Both are created automatically after a valid entity is confirmed (Step 6 below).

SyncWatermarks and SyncSchemaHistory tables will also not exist yet at this point, and they are only created when a valid entry is added to SyncTableRegistry and confirmed to exist in the Archibus API.

Step 6. Populate SyncTableRegistry

The SyncTableRegistry table was created empty in Step 1 and the function will not sync any data until rows are added.

  1. Navigate to Azure Portal > Storage account {FUNC_STORAGE_ACCOUNT} > Storage browser > Tables > SyncTableRegistry.
  2. Click Add entity.
  3. Set the following fields for each Archibus table you want to sync:
    • PartitionKey = archibus (all lowercase, exact and see warning below)
    • RowKey = the Archibus table name e.g. rm, bl, site
  4. No additional columns are required.
  5. Click Insert.
  6. Repeat for each table. The function will pick up new entries on the next scheduled run (or immediately if triggered manually).

PartitionKey is case-sensitive

It must be exactly archibus (all lowercase). Rows with any other capitalization e.g. Archibus, ARCHIBUS are silently ignored and the function will log a WARNING and skip them.

Notes

  • To stop syncing a table, delete its row from SyncTableRegistry. Existing data in the bronze and silver containers is preserved and nothing is deleted from the data lake.
  • Deferred table creation - The SyncWatermarks and the SyncSchemaHistory tables are only created after the first valid entity is confirmed; this  means at least one row in SyncTableRegistry maps to a table that actually exists in the Archibus API. If SyncTableRegistry is empty, or every registered table is absent from the Archibus API, the function exits cleanly without creating either table.

Step 7. Verify the Deployment

Run through this checklist after Step 6 is complete and SyncTableRegistry has at least one row registered:

Check Details

Azure Portal > Function App > Functions > sync_archibus > Monitor

 Confirm at least one successful invocation after you added rows to SyncTableRegistry

Application Insights > Logs

 Run the query below to check for exceptions

SyncWatermarks table

 One row per registered entity; status = success or no_changes

SyncLock table

 A row with RowKey = sync_lock exists and locked = false after the run completes

ADLS Gen2 bronze container

Partition folder(s) exist under:

bronze/<table-name>/year=YYYY/month=MM/day=DD/

    SyncWatermarks table itself will not exist until after the first successful sync cycle with a valid registry entry. If the table is absent, verify SyncTableRegistry has entries and the Archibus API is reachable.

    Application Insights — exception check (KQL):

    exceptions
| where timestamp > ago(1h)
| project timestamp, type, outerMessage, severityLevel
| order by timestamp desc

    Find sync/delete failures

    traces
| where severityLevel == 3
| where message startswith "Sync failed for" or message startswith "Deletion check failed for"
| order by timestamp desc

    Update the Function (Re-deployment)

    To deploy a new version of the function code, re-run whichever deployment method you used in Step 5.

    • If only an application setting changed, e.g. SYNC_SCHEDULE, PAGE_SIZE: update the value directly in Azure Portal → Function App → Configuration → Application settings → Save. The change takes effect immediately — no redeploy needed.
    • If only Bicep/infrastructure changed with no code changes: re-run Step 1 and the Bicep deployment is idempotent and will apply the infrastructure change safely.

    Operator Configuration Reference

    Post-deployment operational changes can be made without redeploying:

    What to change Where How

    Add a table to sync

    		 SyncTableRegistry table

    Add row:

    PartitionKey=archibus, RowKey=<table-name>

    Remove a table from sync

    		 SyncTableRegistry table Delete the row. Existing lake data is preserved.

    Change sync schedule

    		 Function App > Configuration > SYNC_SCHEDULE Update value (6-field cron) > Save

    Change page size

    		 Function App > Configuration > PAGE_SIZE Update value > Save

    Change the parallel workers

    		 Function App > Configuration > MAX_WORKERS

    Update value > Save.

    See MAX_WORKERS Performance Warning above.

    Force full re-sync for an entity

    		 SyncWatermarks table Clear or delete the updatedAfter field on the entity's row

    Adjust stale-lock timeout

    		 SyncLock table Edit timeoutMinutes on the sync_lock row (default: 180)

    Acknowledge a schema error

    		 SyncWatermarks table

    Set schemaStatus to any value other than incompatible_schema_change

    e.g. acknowledged

    incompatible_schema_change - What it means and how to clear it

    This status is set when the function detects that Archibus has removed a column or changed a column's type.

    When triggered, the function automatically performs a full sync and rebuilds the silver file to reflect the new schema. The status is a deliberate signal that a breaking change has occurred and it never auto-clears.

    How to investigate

    1. Open {FUNC_STORAGE_ACCOUNT} → Tables → SyncWatermarks → find the affected entity row.
    2. Read schemaRemovedColumns to see which column was removed (this field is intentionally preserved across subsequent syncs so you can see the original cause at a glance).
    3. For full drift history, open SyncSchemaHistory table, then filter by PartitionKey = <entity-name> and then look for rows with severity = error

    How to acknowledge 

    1. Verify that downstream consumers (reports, Synapse, Databricks, etc.) have been updated to handle the schema change.
    2. Navigate to {FUNC_STORAGE_ACCOUNT} → Tables → SyncWatermarks
    3. Open the affected entity row and then set schemaStatus to any value other than incompatible_schema_change e.g. acknowledged
    4. The next clean sync will automatically reset schemaStatus to ok.

    Scenario Verification

    Use this section to verify the system is behaving correctly for each data-change scenario.

    Storage accounts referenced below:

    • {FUNC_STORAGE_ACCOUNT} - function runtime storage (Azure Table Storage: SyncWatermarks, SyncTableRegistry, SyncLock, SyncSchemaHistory)
    • {ADLS_STORAGE_ACCOUNT} - ADLS Gen2 data lake (bronze and silver Parquet files)

    Scenario 1. First Full Sync

    Applies to: first deployment, or after clearing updatedAfter on a watermark row to force a re-sync.

    What to check Where Expected

    Invocation completed

    		 Function App → Monitor Status: Success; duration reflects the number of registered entities.

    Run-lock released

    		 {FUNC_STORAGE_ACCOUNT} → Tables → SyncLock Row RowKey = sync_lock exists; locked = false  after sync completes.

    Watermark created per entity

    		 {FUNC_STORAGE_ACCOUNT} → Tables → SyncWatermarks One row per registered entity; status = success; updatedAfter is a recent timestamp; lastDeleteStatus = success or no_deletions.

    Bronze data written

    {ADLS_STORAGE_ACCOUNT} → bronze container →

    <table>/year=YYYY/month=MM/day=DD/

    		 One or more .parquet files present containing all records fetched from Archibus.

    Silver data written

    {ADLS_STORAGE_ACCOUNT} → silver container →

     <table>/<table>_current.parquet

    		 File exists; last-modified timestamp matches the sync run time.

    SyncWatermarks table itself is created on this first sync (not just the row). SyncSchemaHistory is also created and seeded with the initial schema baseline on the first sync for each entity.

    Scenario 2. Record Updated or Added in Archibus

    Applies to: next scheduled sync after a record is created or modified in Archibus.

    What to check Where Expected

    Watermark advanced

    		 {FUNC_STORAGE_ACCOUNT} → Tables → SyncWatermarks → entity row updatedAfter is newer than the previous sync; lastRecordCount > 0; status = success

    New bronze file written

    {ADLS_STORAGE_ACCOUNT} → bronze →

    <table>/year=.../month=.../day=.../

    		 A new .parquet file appears from the previous sync; it contains only the changed or added record(s)

    Silver updated

    {ADLS_STORAGE_ACCOUNT} → silver →

    <table>/<table>_current.parquet

    		 File last-modified timestamp is newer than the previous sync

    If no records have changed since the last sync: status = no_changes and no new bronze file is written, then this is correct and expected behavior.

    Scenario 3. Record Deleted in Archibus

    Applies to: deletion detection runs inline after each entity's sync on every cycle.

    What to check Where Expected

    Deletion detected

    {FUNC_STORAGE_ACCOUNT} → Tables → SyncWatermarks → entity row

    		 lastDeleteStatus = success; lastDeleteCount > 0; lastDeleteCheckTimestamp updated

    Deleted record archived to bronze

    {ADLS_STORAGE_ACCOUNT} → bronze →

    <table>_deleted/<table>_deleted_current.parquet

    		 File exists and contains the deleted record (identified by its UUID); row count increased by the number of deletions

    Record removed from silver

    {ADLS_STORAGE_ACCOUNT} → silver →

    <table>/<table>_current.parquet

    		 Deleted record's UUID is no longer present; total row count decreased accordingly

    If no deletions occurred then the following will be true. The deletion archive file is unchanged.

    • lastDeleteStatus = no_deletions
    • lastDeleteCount = 0 

    Scenario 4. Column Removed or Type Changed in Archibus

    When the Archibus schema API returns a different column set or type compared to the stored watermark. The function detects this before fetching records, performs a full sync, and rebuilds silver.

    What to check Where Expected
    Schema error flagged {FUNC_STORAGE_ACCOUNT} → Tables → SyncWatermarks → entity row

    schemaStatus = incompatible_schema_change; schemaRemovedColumns shows the removed column name (or check typeChanges in SyncSchemaHistory for type changes).
    Drift event recorded {FUNC_STORAGE_ACCOUNT} → Tables → SyncSchemaHistory → filter PartitionKey = <entity>

    A row with severity = error; columnsRemoved or typeChanges populated; detectedAt is the sync timestamp.
    Silver rebuilt {ADLS_STORAGE_ACCOUNT} → silver → <table>/<table>_current.parquet

    File last-modified updated; the removed column is no longer present in the Parquet schema.
    Warning in App Insights

    App Insights → Logs → KQL:

    traces | where message contains "incompatible_schema_change" | order by timestamp desc

    Warning-level trace for the affected entity naming the removed column or type change.

    Action required: incompatible_schema_change never auto-clears. See the acknowledgment steps in the Operator Configuration Reference above. 

    Scenario 5. Deleted Infrastructure Recovery

    Self-healing behavior after manual deletion:

    • Deleted bronze/silver ADLS containers → auto-recreated by the function on next sync
    • Deleted SyncWatermarks/SyncSchemaHistory → auto-recreated on next valid sync cycle
    • Deleted SyncTableRegistry → only recreated by re-running Step 1 (Bicep deployment); entries must be re-added manually
    • Deleted SyncLock → auto-recreated on next sync invocation