In Azure Data Factory (ADF, but also Synapse Pipelines and Fabric Pipelines), you have a typical scenario where you retrieve some metadata from a database and then loop over the metadata with a ForEach activity. For example, to copy all the table from a source database in a metadata-driven pipeline. The obvious choice would be […]
Tag: adf
Dynamic Warehouse & Lakehouse Connections in Microsoft Fabric Data Pipelines
When you develop data pipelines in Microsoft Fabric (the Azure Data Factory equivalent in Fabric, not to be confused with deployment pipelines), you will most likely have some activities with a connection to a warehouse, a lakehouse or a KQL database (for the remainder of the blog post I’ll talk about a warehouse, but it […]
PostgreSQL error in Fabric Pipelines – No pg_hba.conf Entry for Host
I came across a bit of a weird error when I was trying to connect to a PostGreSQL database (hosted as an Azure DB for PostGreSQL), where “xx.xx.xx.xxx” is some IP address: No pg_hba.conf entry for host “xx.xx.xx.xxx” Some context to get us started. I’m using Pipelines in Microsoft Fabric to copy data from an […]
Planview Portfolios REST API with ADF – Retrieving Actual Data
In the introduction of this blog post series, I explained the use case: extracting data from the Planview Portfolios REST API using Azure Data Factory. Any tool that can send HTTPS requests can work with the REST API, but our focus here is ADF (which can also be Synapse Pipelines or Fabric Pipelines). In the […]
Planview Portfolios REST API with ADF – The OData Feed
In the introduction, I explained the setup of the use case and why we need to use the OData feed: to get the list of Project IDs, because the REST API doesn’t have an endpoint for this. Weird, right? In this part of the series, we’ll dive into the Lookup activity, which uses the OData […]
Extracting Data from the Planview Portfolios REST API using ADF – Introduction
I have a client using Planview Portfolios to manage their PMO. As part of the data warehouse, we want to extract the project data from the Planview REST API. There are other methods to get data, such as using OData (which is used in the Power BI connector), SOAP Web Services (2005 called) and a […]
#TSQL2sday 171 – Describe the Most Recent Issue You Closed
It’s #TSQL2sday time! This month’s invitation has been sent out by Brent Ozar and he asks us to describe the most recent issue – or the last ticket – we closed. In my case, I actually blogged about it in this blog post: Azure Function App doesn’t retrieve most recent secret value from Azure Key […]
Fabulous Stuff in Fabric – Part 4: The Office365 Outlook Activity
A while ago I had a little blog post series about cool stuff in Snowflake. I’m doing a similar series now, but this time for Microsoft Fabric. I’m not going to cover the basics of Fabric, hundreds of bloggers have already done that. I’m going to cover little bits & pieces that I find interesting, that […]
Session Materials for Cloudbrew & Data Meetup Groningen
I presented “Microsoft Fabric for Dummies” at Cloudbrew and “How I saved 80% on my ADF costs” as a lightning talk at Data Meetup Groningen. Both slidedecks can be found at github.
Fabulous Stuff in Fabric – Part 2: Deactivating Activities in Pipelines
A while ago I had a little blog post series about cool stuff in Snowflake. I’m doing a similar series now, but this time for Microsoft Fabric. I’m not going to cover the basic of Fabric, hundreds of bloggers have already done that. I’m going to cover little bits & pieces that I find interesting, that […]