You can find the slides of my session on the €100 DWH in Azure on Github.
Tag: adf
Secure Logic Apps with OAuth Authorization
I’ve used Logic Apps a couple of times over the past years for simple workflows, like sending an email (still not supported in ADF, booooo), reading a SharePoint List or copying some files from A to B (cheaper than the Copy Data Activity in ADF). When you create a Logic App with an HTTP trigger […]
Free Online Sessions – Building the €100 DWH and Indexing for Dummies
I’m giving two online sessions soon on virtual events that are free to attend. The first one is at Retro Data (20th of September), a new virtual conference focusing on established technology, instead of preview features. My session Building the €100 data warehouse with the Azure Data Platform at 10AM UTC+2. The other session is […]
Parameterize Linked Service in ADF with no Dynamic Content available
When you create linked services in Azure Data Factory (ADF) or in a Synapse Workspace, you typically want to parameterize certain properties of the connection. When you deploy your artefacts to another environment, you can change the parameter values to switch the connection to the new environment. For example, the server name of the connection […]
Execute Fabric Data Pipeline from Azure Data Factory
In the blog post Call a Fabric REST API from Azure Data Factory I explained how you can call a Fabric REST API endpoint from Azure Data Factory (or Synapse if you will). Let’s go a step further and execute a Fabric Data Pipeline from an ADF pipeline, which is a common request. A Fabric […]
Azure Data Factory Pipeline Debugging Fails with BadRequest
I recently had a new pipeline fail. It was actually a copy of an old pipeline where I had made some adjustments into as part of a database migration. When triggered during an execution run, it failed saying some expression could not be parsed. When I went into the pipeline and triggered a debug, it […]
Call a Fabric REST API from Azure Data Factory
Suppose you want to call a certain Microsoft Fabric REST API endpoint from Azure Data Factory (or Synapse Pipelines). This can be done using a Web Activity, and most Fabric APIs now support service principals or managed identities. Let’s illustrate with an example. I’m going to call the REST API endpoint to create a new […]
How to use a Script Activity in ADF as a Lookup
In Azure Data Factory (ADF, but also Synapse Pipelines and Fabric Pipelines), you have a typical scenario where you retrieve some metadata from a database and then loop over the metadata with a ForEach activity. For example, to copy all the table from a source database in a metadata-driven pipeline. The obvious choice would be […]
Dynamic Warehouse & Lakehouse Connections in Microsoft Fabric Data Pipelines
When you develop data pipelines in Microsoft Fabric (the Azure Data Factory equivalent in Fabric, not to be confused with deployment pipelines), you will most likely have some activities with a connection to a warehouse, a lakehouse or a KQL database (for the remainder of the blog post I’ll talk about a warehouse, but it […]
PostgreSQL error in Fabric Pipelines – No pg_hba.conf Entry for Host
I came across a bit of a weird error when I was trying to connect to a PostGreSQL database (hosted as an Azure DB for PostGreSQL), where “xx.xx.xx.xxx” is some IP address: No pg_hba.conf entry for host “xx.xx.xx.xxx” Some context to get us started. I’m using Pipelines in Microsoft Fabric to copy data from an […]