Recently I got a hold of my Untappd data. For those who don’t know the app, it’s a rating app for beers. You can check-in a beer each time you drink one (by scanning the barcode for example), and then rate it, assign flavors et cetera. I did a GDPR request to get all my […]
Category: Azure Data Factory
Granular Cost Management for Azure Data Factory
A new feature has been launched that allows you to have more granular control over the execution costs of your pipelines. By enabling the feature, you have a cost overview per pipeline in your cost analysis in the Azure portal and in your billing report. This is great news if you want to figure out […]
How you can save up to 80% on Azure Data Factory pricing
… with this one weird little trick 🙂 Sorry for the clickbait, but you can actually save a lot on ADF pricing if you design your pipelines carefully. And as we will see, don’t do everything in ADF. Before we proceed further, I advise to watch this short YouTube video by Cathrine Wilhelmsen on ADF […]
Book Review – Azure Data Factory by Example
I just finished the book Azure Data Factory by Example: Practical Implementation for Data Engineers by Richard Swinbank. The book was not exactly what I was expecting, but that’s actually my fault and not the book’s. The book itself is great and is a great introduction to Azure Data Factory (ADF) and it covers almost […]
Execute SQL Scripts on Snowflake using Azure Data Factory
Azure Data Factory has a new activity introduced this week (around the 10th of March 2022 for you future readers): the Script activity! This is not to be confused with the script task/component of SSIS, which allows you to execute .NET script (C# for most people, or VB if you’re Ben Weissman). No, this task […]
Extracting Data from the AFAS REST API with Azure Data Factory
AFAS is a Dutch business software company, proving companies with ERP and HR software-as-a-service. You can extract data from the service using a REST API (there are also SOAP web services available for those who appreciate the trip down memory lane). In this blog post I’ll show you how you can get data out of […]
Dynamically Map JSON to SQL in Azure Data Factory
We’re reading in some JSON files in Azure Data Factory (ADF), for example for a REST API. We’re storing the data in a relational table (SQL Server, Azure SQL DB…). The data volume is low, so we’re going to use a Copy Data activity in a pipeline, rather than a mapping data flow (or whatever […]
Dynamic Datasets in Azure Data Factory
With “dynamic datasets” I mean the following: a dataset that doesn’t have any schema or properties defined, but rather only parameters. Why would you do this? With a dynamic – or generic – dataset, you can use it inside a ForEach loop and then loop over metadata which will populate the values of the parameter. […]
Azure Data Factory and the Exact Online REST API – Getting around the 10 Minute Limit
This is the final blog post of this series on how to extract data from the Exact Online REST API with Azure Data Factory. I’ve you followed along with the previous posts, you’ve seen how you can request new refresh/access tokens from the API, how you can extract data from an endpoint for multiple divisions […]
Azure Data Factory and the Exact Online REST API – Dealing with Pagination
Some endpoints don’t have that much data, both others certainly will, for example financial transactions. The REST API limits the number of records it returns, and for most endpoints this is 60 records. After the first 60 records, you get a link at the bottom of the response that points to the next 60 records. […]