I recently had a new pipeline fail. It was actually a copy of an old pipeline where I had made some adjustments into as part of a database migration. When triggered during an execution run, it failed saying some expression could not be parsed. When I went into the pipeline and triggered a debug, it […]
Category: Azure Data Factory
Call a Fabric REST API from Azure Data Factory
Suppose you want to call a certain Microsoft Fabric REST API endpoint from Azure Data Factory (or Synapse Pipelines). This can be done using a Web Activity, and most Fabric APIs now support service principals or managed identities. Let’s illustrate with an example. I’m going to call the REST API endpoint to create a new […]
How to use a Script Activity in ADF as a Lookup
In Azure Data Factory (ADF, but also Synapse Pipelines and Fabric Pipelines), you have a typical scenario where you retrieve some metadata from a database and then loop over the metadata with a ForEach activity. For example, to copy all the table from a source database in a metadata-driven pipeline. The obvious choice would be […]
Planview Portfolios REST API with ADF – Retrieving Actual Data
In the introduction of this blog post series, I explained the use case: extracting data from the Planview Portfolios REST API using Azure Data Factory. Any tool that can send HTTPS requests can work with the REST API, but our focus here is ADF (which can also be Synapse Pipelines or Fabric Pipelines). In the […]
Planview Portfolios REST API with ADF – Fetching the Access Token
In the introduction of this blog post series, I explained the use case: extracting data from the Planview Portfolios RESET API using Azure Data Factory. Any tool that can send HTTPS requests can work with the REST API, but our focus here is ADF (which can also be Synapse Pipelines or Fabric Pipelines). In the […]
Planview Portfolios REST API with ADF – The OData Feed
In the introduction, I explained the setup of the use case and why we need to use the OData feed: to get the list of Project IDs, because the REST API doesn’t have an endpoint for this. Weird, right? In this part of the series, we’ll dive into the Lookup activity, which uses the OData […]
Extracting Data from the Planview Portfolios REST API using ADF – Introduction
I have a client using Planview Portfolios to manage their PMO. As part of the data warehouse, we want to extract the project data from the Planview REST API. There are other methods to get data, such as using OData (which is used in the Power BI connector), SOAP Web Services (2005 called) and a […]
#TSQL2sday 171 – Describe the Most Recent Issue You Closed
It’s #TSQL2sday time! This month’s invitation has been sent out by Brent Ozar and he asks us to describe the most recent issue – or the last ticket – we closed. In my case, I actually blogged about it in this blog post: Azure Function App doesn’t retrieve most recent secret value from Azure Key […]
Allow a user to only trigger pipelines in Azure Data Factory
Typically you have a bunch of pipelines that are started by one or more triggers. Sometimes, a pipeline needs to be manually triggered. For example, when the finance department is closing the fiscal year, they probably want to run the ETL pipeline a couple of times on-demand, to make sure their latest changes are reflected […]
Read data from Azure Synapse Serverless SQL Pools with Azure Data Factory
We have some data we can query using the serverless SQL pools in Azure Synapse Analytics. For this blog post, I’m querying data that is stored in Azure Cosmos DB. Read the blog post How to Store Normalized SQL Server Data into Azure Cosmos DB to learn more about how that data got there. Suppose […]