This Monday (the 5th of December 2022), my employer AE hosts a user group meeting for dataminds. There will be two sessions: It’ll be a very interesting evening, and catering is provided. For registration and practical details, check out the dataminds website. I’ve uploaded my slidedeck to Github.
Tag: adf
Read data from Azure Synapse Serverless SQL Pools with Azure Data Factory
We have some data we can query using the serverless SQL pools in Azure Synapse Analytics. For this blog post, I’m querying data that is stored in Azure Cosmos DB. Read the blog post How to Store Normalized SQL Server Data into Azure Cosmos DB to learn more about how that data got there. Suppose […]
How to Store Normalized SQL Server Data into Azure Cosmos DB
Recently I got a hold of my Untappd data. For those who don’t know the app, it’s a rating app for beers. You can check-in a beer each time you drink one (by scanning the barcode for example), and then rate it, assign flavors et cetera. I did a GDPR request to get all my […]
Granular Cost Management for Azure Data Factory
A new feature has been launched that allows you to have more granular control over the execution costs of your pipelines. By enabling the feature, you have a cost overview per pipeline in your cost analysis in the Azure portal and in your billing report. This is great news if you want to figure out […]
Speaking Engagements for Fall 2022
The summer holiday is almost over, and this means conference season will be back in full swing again. I’ll be speaking at a couple of events in the second half of this year: Data Bash. A new free virtual conference focusing on the German-speaking nations in Europe. And on data of course. There are sessions […]
Book Review – Azure Data Factory by Example
I just finished the book Azure Data Factory by Example: Practical Implementation for Data Engineers by Richard Swinbank. The book was not exactly what I was expecting, but that’s actually my fault and not the book’s. The book itself is great and is a great introduction to Azure Data Factory (ADF) and it covers almost […]
Execute SQL Scripts on Snowflake using Azure Data Factory
Azure Data Factory has a new activity introduced this week (around the 10th of March 2022 for you future readers): the Script activity! This is not to be confused with the script task/component of SSIS, which allows you to execute .NET script (C# for most people, or VB if you’re Ben Weissman). No, this task […]
Extracting Data from the AFAS REST API with Azure Data Factory
AFAS is a Dutch business software company, proving companies with ERP and HR software-as-a-service. You can extract data from the service using a REST API (there are also SOAP web services available for those who appreciate the trip down memory lane). In this blog post I’ll show you how you can get data out of […]
Dynamically Map JSON to SQL in Azure Data Factory
We’re reading in some JSON files in Azure Data Factory (ADF), for example for a REST API. We’re storing the data in a relational table (SQL Server, Azure SQL DB…). The data volume is low, so we’re going to use a Copy Data activity in a pipeline, rather than a mapping data flow (or whatever […]
Dynamic Datasets in Azure Data Factory
With “dynamic datasets” I mean the following: a dataset that doesn’t have any schema or properties defined, but rather only parameters. Why would you do this? With a dynamic – or generic – dataset, you can use it inside a ForEach loop and then loop over metadata which will populate the values of the parameter. […]