I was writing some dynamic SQL that generates some SQL statements to load my facts and dimensions into a data warehouse. Some of those SQL statements can become very long, for example if a dimension has a lot of columns. When debugging, I noticed a couple of statements failing with various errors. Turns out, they […]
Tag: syndicated
Allow a user to only trigger pipelines in Azure Data Factory
Typically you have a bunch of pipelines that are started by one or more triggers. Sometimes, a pipeline needs to be manually triggered. For example, when the finance department is closing the fiscal year, they probably want to run the ETL pipeline a couple of times on-demand, to make sure their latest changes are reflected […]
Logic App errors out when using variables in a SharePoint Action
I have a Logic App that reads out a SharePoint library and stores all the documents found into Azure Blob Storage (ADF only supports Lists). I was trying to make this Logic App “generic”, meaning I could change the source folder and the destination container by using variables. That way, I have one single Logic […]
Speaking at Data Community Austria Day 2023
Let me start by wishing everyone the best for the year 2023! We’re going to kick off the new year with a virtual conference: the Data Community Austria Day 2023. I’m going to present the session Migrate your SSIS skills to ADF. You can find the schedule with all the abstracts on Sessionize. Hope to […]
Book Review – Designing Data-Intensive Applications
The book Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann was recommended to me by a colleague. The author has worked at companies such as LinkedIn, where he has built large distributed systems to handle data, so I guess he knows what he’s talking about 🙂 (he’s also a […]
How to Specify the Format of the Request Body of an Azure Function
This is a bit of a niche blog post, because you don’t actually need this to make your Azure Function work :). When you create a new HTTP-triggered Azure Function (using C# code) in Visual Studio, you’ll get the following boilerplate code: This code reads the body of the HTTP request (called req) and tries […]
Book Review – SQL Server Query Tuning and Optimization (2nd Edition)
The nice people from Packt Publishing sent me a digital copy of Benjamin Nevarez new book to review: SQL Server Query Tuning and Optimization – Optimize Microsoft SQL Server 2022 queries and applications. It says first edition on the cover, but that’s because it’s the first time this book has been published by Packt. The […]
User Group Meeting – Migrate your SSIS skills to ADF & More
This Monday (the 5th of December 2022), my employer AE hosts a user group meeting for dataminds. There will be two sessions: It’ll be a very interesting evening, and catering is provided. For registration and practical details, check out the dataminds website. I’ve uploaded my slidedeck to Github.
How to Install Integration Services 2022
The title of this blog post is a bit misleading, because what we actually want to do is to install the SSIS development functionality in Visual Studio 2022. SSAS and SSRS have been available for some time now in VS 2022, but the extension for SSIS has only just been released in preview. For those […]
Read data from Azure Synapse Serverless SQL Pools with Azure Data Factory
We have some data we can query using the serverless SQL pools in Azure Synapse Analytics. For this blog post, I’m querying data that is stored in Azure Cosmos DB. Read the blog post How to Store Normalized SQL Server Data into Azure Cosmos DB to learn more about how that data got there. Suppose […]