I’m hosting a free webinar at MSSQLTips.com at the 19th of December 2024, 6PM UTC. The topic is about window functions, something that really interests me and that should be in the toolbox of every season SQL developer. The abstract: The gaps and islands problem is a common problem in data analysis. Some examples are: […]
dataMinds Connect 2024 – Session Materials
The slides and scripts for my session “Tackling the Gaps & Islands Problem with T-SQL Window Functions” at dataMinds Connect 2024 can be downloaded from GitHub. If you want to run the sample scripts, some of them require an installation of the AdventureWorks DW sample database.
Connect to Power BI as a Guest User in another Tenant
Sometimes your Microsoft Entra ID account (formerly known as Azure Active Directory) is added as a guest user in another tenant. This happens quite a lot when you’re a consultant and your client can’t create a new user in their own tenant, so they add the account of your own company as a guest instead. […]
How to use a Script Activity in ADF as a Lookup
In Azure Data Factory (ADF, but also Synapse Pipelines and Fabric Pipelines), you have a typical scenario where you retrieve some metadata from a database and then loop over the metadata with a ForEach activity. For example, to copy all the table from a source database in a metadata-driven pipeline. The obvious choice would be […]
Database Build Error – Incorrect syntax near DISTINCT
I wrote a piece of SQL that had some new T-SQL syntax in it: IS DISTINCT FROM. This predicate checks if two expressions are equal taking NULL into account. I successfully ran the code on an Azure SQL Database. However, when I tried to deploy the code using an Azure Devops pipeline, it returned the […]
Speaking at dataMinds Connect 2024
I’m very excited to announce I’ve been selected as a speaker for dataMinds Connect 2024, Belgium’s (and maybe/probably/most likely mainland Europe’s) best and greatest Microsoft data platform conference. My session is about a topic I really love: writing window functions in T-SQL. Tackling the Gaps & Islands Problem with T-SQL Window Functions The gaps & […]
Dynamic Warehouse & Lakehouse Connections in Microsoft Fabric Data Pipelines
When you develop data pipelines in Microsoft Fabric (the Azure Data Factory equivalent in Fabric, not to be confused with deployment pipelines), you will most likely have some activities with a connection to a warehouse, a lakehouse or a KQL database (for the remainder of the blog post I’ll talk about a warehouse, but it […]
System Views in Microsoft Fabric – Query references an object that is not supported in distributed processing mode
I have a metadata-driven ELT framework that heavily relies on dynamic SQL to generate SQL statements that load data from views into a respective fact or dimension. Such a task is well suited for generation, since the pattern to load a type 1 SCD, type 2 SCD or a fact table is always the same. […]
PostgreSQL error in Fabric Pipelines – No pg_hba.conf Entry for Host
I came across a bit of a weird error when I was trying to connect to a PostGreSQL database (hosted as an Azure DB for PostGreSQL), where “xx.xx.xx.xxx” is some IP address: No pg_hba.conf entry for host “xx.xx.xx.xxx” Some context to get us started. I’m using Pipelines in Microsoft Fabric to copy data from an […]
Book Review – Fundamentals of Data Engineering
This book was recommended by some of my colleagues, so I decided to give it a try. After finishing the book, I was pleased that I did. As the title might give away, it’s a book about data engineering, but not from a pure technical perspective. It won’t teach you how to write Python or […]