Categories: Azure

SharePoint List only returns 100 items in Azure Logic App

I was reading a SharePoint List using the “Get Items” activity in an Azure Logic App. I explain how you can create such a Logic App in the blog post Reading a SharePoint List with Azure Logic App.

It all worked fine for a while, but recently the list grew larger than 100 items. Suddenly, I started getting complaints that some items didn’t make it into the data warehouse. What’s going on? I ran the Logic App and I could see only 100 items were inserted into the SQL Server table:

(I realize the Logic App is doing a row-by-row insert, which is really slow. But for a small data set loaded only once a day, I don’t really care)

You can check the exact JSON output of the Get Items activity by clicking on “Show raw outputs”. And the output also only had 100 items. Luckily, the solution wasn’t that complicated. The OData feed responsible for exporting the SharePoint List applies pagination. This means only the first 100 items of the list are returned, alongside with an URL on how to retrieve the next 100 items (you can find it in the JSON if you’re really interested in the details). Fortunately, we can just enable pagination in the settings!

Switch the slider for pagination to on, and specify a threshold value (I read somewhere this is maximum 5000, but don’t take my word for it). Don’t forget to hit Done at the bottom AND to save your Logic App!


------------------------------------------------
Do you like this blog post? You can thank me by buying me a beer 🙂
Koen Verbeeck

Koen Verbeeck is a Microsoft Business Intelligence consultant at AE, helping clients to get insight in their data. Koen has a comprehensive knowledge of the SQL Server BI stack, with a particular love for Integration Services. He's also a speaker at various conferences.

Recent Posts

Free webinar – Tackling the Gaps and Islands Problem with T-SQL Window Functions

I'm hosting a free webinar at MSSQLTips.com at the 19th of December 2024, 6PM UTC.…

5 days ago

dataMinds Connect 2024 – Session Materials

The slides and scripts for my session "Tackling the Gaps & Islands Problem with T-SQL…

4 weeks ago

Connect to Power BI as a Guest User in another Tenant

Sometimes your Microsoft Entra ID account (formerly known as Azure Active Directory) is added as…

2 months ago

How to use a Script Activity in ADF as a Lookup

In Azure Data Factory (ADF, but also Synapse Pipelines and Fabric Pipelines), you have a…

4 months ago

Database Build Error – Incorrect syntax near DISTINCT

I wrote a piece of SQL that had some new T-SQL syntax in it: IS…

4 months ago

Speaking at dataMinds Connect 2024

I'm very excited to announce I've been selected as a speaker for dataMinds Connect 2024,…

5 months ago