Skip to main content
Under the kover of business intelligence
  • Home
  • Copyright
  • About

Tag: azure data factory

How you can save up to 80% on Azure Data Factory pricing

May 11, 2022May 11, 2022 Koen Verbeeck 1 Comment

… with this one weird little trick 🙂 Sorry for the clickbait, but you can actually save a lot on ADF pricing if you design your pipelines carefully. And as we will see, don’t do everything in ADF. Before we proceed further, I advise to watch this short YouTube video by Cathrine Wilhelmsen on ADF […]

Read More

Book Review – Pro Serverless Data Handling with Microsoft Azure

April 5, 2022April 5, 2022 Koen Verbeeck 2 Comments

Yet another book review! I just finished the book Pro Serverless Data Handling with Microsoft Azure: Architecting ETL and Data-Driven Applications in the Cloud (what a title :), written by German MVPs (and metalheads) Benjamin Kettner and Frank Geisler. The book handles the topic of “serverless” computing in the Microsoft Data Platform, where serverless is […]

Read More

Book Review – Azure Data Factory by Example

March 29, 2022March 29, 2022 Koen Verbeeck 2 Comments

I just finished the book Azure Data Factory by Example: Practical Implementation for Data Engineers by Richard Swinbank. The book was not exactly what I was expecting, but that’s actually my fault and not the book’s. The book itself is great and is a great introduction to Azure Data Factory (ADF) and it covers almost […]

Read More

Execute SQL Scripts on Snowflake using Azure Data Factory

March 13, 2022March 13, 2022 Koen Verbeeck 1 Comment

Azure Data Factory has a new activity introduced this week (around the 10th of March 2022 for you future readers): the Script activity! This is not to be confused with the script task/component of SSIS, which allows you to execute .NET script (C# for most people, or VB if you’re Ben Weissman). No, this task […]

Read More

Speaking at Techorama 2022

February 22, 2022February 22, 2022 Koen Verbeeck Leave a comment

I’ve received some good news: Techorama is back with an in-person conference! I’m selected as a speaker and I’ll be talking about Azure Data Factory. Azure Data Factory (ADF) is the tool in the Azure cloud for creating your data pipelines. It allows you to move and transform data from a multitude of sources to […]

Read More

Dynamic Datasets in Azure Data Factory

May 27, 2021May 27, 2021 Koen Verbeeck 4 Comments

With “dynamic datasets” I mean the following: a dataset that doesn’t have any schema or properties defined, but rather only parameters. Why would you do this? With a dynamic – or generic – dataset, you can use it inside a ForEach loop and then loop over metadata which will populate the values of the parameter. […]

Read More

Azure Data Factory and the Exact Online REST API – Getting around the 10 Minute Limit

May 25, 2021May 25, 2021 Koen Verbeeck 7 Comments

This is the final blog post of this series on how to extract data from the Exact Online REST API with Azure Data Factory. I’ve you followed along with the previous posts, you’ve seen how you can request new refresh/access tokens from the API, how you can extract data from an endpoint for multiple divisions […]

Read More

Azure Data Factory and the Exact Online REST API – Dealing with Pagination

May 25, 2021May 25, 2021 Koen Verbeeck 16 Comments

Some endpoints don’t have that much data, both others certainly will, for example financial transactions. The REST API limits the number of records it returns, and for most endpoints this is 60 records. After the first 60 records, you get a link at the bottom of the response that points to the next 60 records. […]

Read More

Azure Data Factory and the Exact Online REST API – Retrieving Data from an Endpoint

May 24, 2021May 25, 2021 Koen Verbeeck 2 Comments

Finally we’ve come to the core of this blog post series: extracting data from a REST API endpoint. Just to recap, you need the following: an access token that is currently valid. See this blog post. a list of divisions. See the previous blog post. As an example, we’re going to read from the Projects […]

Read More

Azure Data Factory and the Exact Online REST API – Retrieving the Divisions

May 24, 2021May 25, 2021 Koen Verbeeck 2 Comments

In many ERP software you have the concept of “companies” or “divisions”, a way of logically dividing resources and assets. For example, if you have a company and a sister-company, you can create two divisions to keep everything separate. When you read data from an Exact Online API endpoint, you need to specify for which […]

Read More

Posts pagination

1 2 Older posts

Feed

RSS

Social

  • Twitter
  • LinkedIn

MVP

Microsoft Data Platform MVP

star schema all the things
Get your star schema merchandise

Author of the Year 2014/2020/2023

Support this blog and buy me a beer

Recent Posts

  • Execute Fabric Data Pipeline from Azure Data Factory
  • Azure Data Factory Pipeline Debugging Fails with BadRequest
  • Call a Fabric REST API from Azure Data Factory
  • Cool Stuff in Snowflake – Part 14: Asynchronous Execution of SQL Statements
  • How I passed the DP-700 Exam

Categories

  • Azure (25)
  • Azure Data Factory (25)
  • Azure Devops (4)
  • Book Review (26)
  • Business Intelligence (29)
  • Events (56)
  • Fabric (19)
  • Power BI (41)
  • Professional Development (9)
  • Snowflake (14)
  • SQL Server (112)
    • MDS (8)
    • SSAS (12)
    • SSIS (46)
    • SSRS (13)
    • TSQL (9)
  • Varia (37)
  • Twitter
  • LinkedIn
  • Home
  • Copyright
  • About
Under the kover of business intelligence All rights reserved. Theme by Colorlib Powered by WordPress