Typically you have a bunch of pipelines that are started by one or more triggers. Sometimes, a pipeline needs to be manually triggered. For example, when the finance department is closing the fiscal year, they probably want to run the ETL pipeline a couple of times on-demand, to make sure their latest changes are reflected […]
Category: Azure Data Factory
Read data from Azure Synapse Serverless SQL Pools with Azure Data Factory
We have some data we can query using the serverless SQL pools in Azure Synapse Analytics. For this blog post, I’m querying data that is stored in Azure Cosmos DB. Read the blog post How to Store Normalized SQL Server Data into Azure Cosmos DB to learn more about how that data got there. Suppose […]
How to Store Normalized SQL Server Data into Azure Cosmos DB
Recently I got a hold of my Untappd data. For those who don’t know the app, it’s a rating app for beers. You can check-in a beer each time you drink one (by scanning the barcode for example), and then rate it, assign flavors et cetera. I did a GDPR request to get all my […]
Granular Cost Management for Azure Data Factory
A new feature has been launched that allows you to have more granular control over the execution costs of your pipelines. By enabling the feature, you have a cost overview per pipeline in your cost analysis in the Azure portal and in your billing report. This is great news if you want to figure out […]
How you can save up to 80% on Azure Data Factory pricing
… with this one weird little trick 🙂 Sorry for the clickbait, but you can actually save a lot on ADF pricing if you design your pipelines carefully. And as we will see, don’t do everything in ADF. Before we proceed further, I advise to watch this short YouTube video by Cathrine Wilhelmsen on ADF […]
Book Review – Azure Data Factory by Example
I just finished the book Azure Data Factory by Example: Practical Implementation for Data Engineers by Richard Swinbank. The book was not exactly what I was expecting, but that’s actually my fault and not the book’s. The book itself is great and is a great introduction to Azure Data Factory (ADF) and it covers almost […]
Execute SQL Scripts on Snowflake using Azure Data Factory
Azure Data Factory has a new activity introduced this week (around the 10th of March 2022 for you future readers): the Script activity! This is not to be confused with the script task/component of SSIS, which allows you to execute .NET script (C# for most people, or VB if you’re Ben Weissman). No, this task […]
Extracting Data from the AFAS REST API with Azure Data Factory
AFAS is a Dutch business software company, proving companies with ERP and HR software-as-a-service. You can extract data from the service using a REST API (there are also SOAP web services available for those who appreciate the trip down memory lane). In this blog post I’ll show you how you can get data out of […]
Dynamically Map JSON to SQL in Azure Data Factory
We’re reading in some JSON files in Azure Data Factory (ADF), for example for a REST API. We’re storing the data in a relational table (SQL Server, Azure SQL DB…). The data volume is low, so we’re going to use a Copy Data activity in a pipeline, rather than a mapping data flow (or whatever […]
Dynamic Datasets in Azure Data Factory
With “dynamic datasets” I mean the following: a dataset that doesn’t have any schema or properties defined, but rather only parameters. Why would you do this? With a dynamic – or generic – dataset, you can use it inside a ForEach loop and then loop over metadata which will populate the values of the parameter. […]