Call a Fabric REST API from Azure Data Factory

Suppose you want to call a certain Microsoft Fabric REST API endpoint from Azure Data Factory (or Synapse Pipelines). This can be done using a Web Activity, and most Fabric APIs now support service principals or managed identities. Let’s illustrate with an example. I’m going to call the REST API endpoint to create a new lakehouse. First, create a system-managed identity for Azure Data Factory in the Azure Portal:

Next, add this managed identity to a security group in Azure Entra ID.

This is important, as we cannot directly give permissions to a service principal (or managed identity) for the Fabric REST APIs in the Fabric Admin portal; this needs to be a security group.

Once the security group can access the APIs, we can add the managed identity to a Fabric workspace:

To create a lakehouse, the managed identity needs to be at least a contributor in the workspace, as specified in the documentation. Depending on the specific REST API endpoint, different permissions might be needed. For example, listing existing objects typically only requires viewer permissions.

Now we can add a Web Activity to an ADF pipeline. For the URL, you need to use https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/lakehouses, with workspaceId as the ID of the workspace you want to create the lakehouse in. You can find this ID in the URL when you’re inside the workspace in the Fabric service:

I use the following expression, with a global parameter containing the ID of the workspace:

@concat('https://api.fabric.microsoft.com/v1/workspaces/',pipeline().globalParameters.fabric_workspace,'/lakehouses')

Unfortunately, at the time of writing we cannot specify a specific folder so the lakehouse will be created in the root folder of the workspace. For the body, the following expression can be specified:

@json(concat('{
  "displayName": "my_lakehouse",
  "description": "This is my first lakehouse",
  "creationPayload": {
    "enableSchemas": true
  }',''))

I added the concat function in case some parameterization is needed. As authentication, the managed identity is chosen and the method is set to POST. For the resource URL, you can specify https://api.fabric.microsoft.com (I found this through trial&error, this isn’t mentioned in the documentation). The following header is added:

If everything is set correctly, the pipeline can be executed and the REST API call should be successful.

------------------------------------------------
Do you like this blog post? You can thank me by buying me a beer 🙂