In dbt, you can link your project to a github account, which is obviously the right thing to do. By default, the project uses the main branch. But what if you have two dbt projects; one for dev and one for prd? And in the dev project, you want to change the default branch to […]
Tag: syndicated
Get your Query History in Redshift
Because my Windows machine apparently decides to install updates over night (and thus reboot my machine), it has happened that I lost the query that I was writing for Redshift in the tool DBeaver. When you work with SQL Server Management Studio (SSMS), you typically don’t have this issue as a temporary copy is always […]
Star Schema Merchandising Available
Those of you who follow me on social media (twitter | LinkedIn | mastadon) or who read my blog here, might be familiar with my catchphrase “STAR SCHEMA ALL THE THINGS”. This slogan originated a couple of years ago when I had to convince some people that the star schema (aka the dimensional model) should […]
Book Review – SQL Query Design Patterns and Best Practices
Disclaimer: The publisher – Packt – sent me a free copy of the book for the intended purpose of a review. At the time of writing, the book is yet to be released (the date is set at the 22nd of March 2023). The book SQL Query Design Patterns and Best Practices: A practical guide […]
Reading AFAS REST API Data with Azure Logic Apps
Recently I had to read some REST API data using an Azure Logic App. The data itself comes from AFAS, a Dutch SaaS ERP company. In general, the process is mostly the same as reading the data with Azure Data Factory (check out this blog post to learn more): There’s a minor difference in how […]
Speaking at Data Saturday Rheinland – Migrating SSIS to ADF
I’m pleased to announce that I’ll be talking about how you can migrate your SSIS skills to Azure Data Factory at Data Saturday Rheinland. At the time of writing this post, the schedule isn’t online yet, but I assume it will be up shortly. The abstract for my session: Integration Services is the new Cobol. […]
Build Once – Add Metadata (the importance of metadata-driven frameworks)
When working in business intelligence, data engineering or data in general, there are some “mantras” that are being adopted by the larger community as “best practices”. For example, I shout “STAR SCHEMA ALL THE THINGS” anywhere I can, because a star schema is the most optimal way to design your model in Power BI. We […]
Why Managed Identities are Awesome
For the first workdays of this new year, I had the pleasurable task of resetting a bunch of passwords. The passwords belong to Azure AD users, service principals, SQL Server users etc. and all of these are tied into the ETL process of a data warehouse. Some are used to connect to source data, others […]
CONCAT_WS Function Truncates Text
I was writing some dynamic SQL that generates some SQL statements to load my facts and dimensions into a data warehouse. Some of those SQL statements can become very long, for example if a dimension has a lot of columns. When debugging, I noticed a couple of statements failing with various errors. Turns out, they […]
Allow a user to only trigger pipelines in Azure Data Factory
Typically you have a bunch of pipelines that are started by one or more triggers. Sometimes, a pipeline needs to be manually triggered. For example, when the finance department is closing the fiscal year, they probably want to run the ETL pipeline a couple of times on-demand, to make sure their latest changes are reflected […]