Recently I had to read some REST API data using an Azure Logic App. The data itself comes from AFAS, a Dutch SaaS ERP company. In general, the process is mostly the same as reading the data with Azure Data Factory (check out this blog post to learn more): There’s a minor difference in how […]
Speaking at Data Saturday Rheinland – Migrating SSIS to ADF
I’m pleased to announce that I’ll be talking about how you can migrate your SSIS skills to Azure Data Factory at Data Saturday Rheinland. At the time of writing this post, the schedule isn’t online yet, but I assume it will be up shortly. The abstract for my session: Integration Services is the new Cobol. […]
Speaking at Techorama 2023
I’m excited to announce I’ve been selected again as a speaker for Techorama 2023. Techorama is one of the biggest Microsoft-focused events in the European mainland, and it’s always a pleasure to be there. It’s organization is amazing and there are tons of interesting sessions. The first batch of sessions has just been announced, so […]
Build Once – Add Metadata (the importance of metadata-driven frameworks)
When working in business intelligence, data engineering or data in general, there are some “mantras” that are being adopted by the larger community as “best practices”. For example, I shout “STAR SCHEMA ALL THE THINGS” anywhere I can, because a star schema is the most optimal way to design your model in Power BI. We […]
Why Managed Identities are Awesome
For the first workdays of this new year, I had the pleasurable task of resetting a bunch of passwords. The passwords belong to Azure AD users, service principals, SQL Server users etc. and all of these are tied into the ETL process of a data warehouse. Some are used to connect to source data, others […]
CONCAT_WS Function Truncates Text
I was writing some dynamic SQL that generates some SQL statements to load my facts and dimensions into a data warehouse. Some of those SQL statements can become very long, for example if a dimension has a lot of columns. When debugging, I noticed a couple of statements failing with various errors. Turns out, they […]
Allow a user to only trigger pipelines in Azure Data Factory
Typically you have a bunch of pipelines that are started by one or more triggers. Sometimes, a pipeline needs to be manually triggered. For example, when the finance department is closing the fiscal year, they probably want to run the ETL pipeline a couple of times on-demand, to make sure their latest changes are reflected […]
Logic App errors out when using variables in a SharePoint Action
I have a Logic App that reads out a SharePoint library and stores all the documents found into Azure Blob Storage (ADF only supports Lists). I was trying to make this Logic App “generic”, meaning I could change the source folder and the destination container by using variables. That way, I have one single Logic […]
Speaking at Data Community Austria Day 2023
Let me start by wishing everyone the best for the year 2023! We’re going to kick off the new year with a virtual conference: the Data Community Austria Day 2023. I’m going to present the session Migrate your SSIS skills to ADF. You can find the schedule with all the abstracts on Sessionize. Hope to […]
Book Review – Designing Data-Intensive Applications
The book Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann was recommended to me by a colleague. The author has worked at companies such as LinkedIn, where he has built large distributed systems to handle data, so I guess he knows what he’s talking about 🙂 (he’s also a […]