Read data from Azure Synapse Serverless SQL Pools with Azure Data Factory

We have some data we can query using the serverless SQL pools in Azure Synapse Analytics. For this blog post, I’m querying data that is stored in Azure Cosmos DB. Read the blog post How to Store Normalized SQL Server Data into Azure Cosmos DB to learn more about how that data got there.

Suppose I now want to read the data using Azure Data Factory. You can read data from Cosmos DB directly, but let’s pretend I want to do some transformations first using my favorite language: SQL. How can we do this?

First we need to create a database and a view in the serverless SQL pool. In the Synapse workspace, we can create a new database in the Data tab.

Choose the serverless option and give the database a name.

Next, you create a view on top of the SELECT statement you want to run, using CREATE OR ALTER VIEW.

In Azure Data Factory (ADF), we create a linked service to Azure Synapse.

There’s one problem however: ADF expects dedicated pools and doesn’t display our newly created database in the list.

As a work around, you can type it in yourself:

To make sure ADF can actually connect to the database, we need to add it as a login and user to the serverless pool.

CREATE LOGIN [mssqltips-df]
	FROM EXTERNAL PROVIDER;

CREATE USER [mssqltips-df]
   FROM LOGIN [mssqltips-df];

ALTER ROLE db_datareader ADD MEMBER [mssqltips-df];

To access data, we need to create a dataset. Again, we choose Azure Synapse Analytics as the type.

In the dataset properties, specify a name, choose the linked service we just created and either enter the table name or select it from the dropdown:

Since we’re using a credential in the FROM OPENROWSET clause of the view (to connect to Cosmos DB), we need to give the managed identity permissions to use this credentials, otherwise we get the error “credential has not been found or you do not have permissions to access it” when trying to import the schema.

This can be done with the following SQL statement:

GRANT REFERENCES ON CREDENTIAL::[mssqltips-cosmoslink] TO [mssqltips-df];

Note: in this case I used a server credential, but it’s also possible to use database credentials (which changes the SQL statement slightly). Make sure you are connected to the correct database when executing the statement.

If everything is configured successfully, you can preview the data:

The dataset can now be used in a Copy data activity or in a data flow.


------------------------------------------------
Do you like this blog post? You can thank me by buying me a beer 🙂
Koen Verbeeck

Koen Verbeeck is a Microsoft Business Intelligence consultant at AE, helping clients to get insight in their data. Koen has a comprehensive knowledge of the SQL Server BI stack, with a particular love for Integration Services. He's also a speaker at various conferences.

View Comments

Recent Posts

Free webinar – Tackling the Gaps and Islands Problem with T-SQL Window Functions

I'm hosting a free webinar at MSSQLTips.com at the 19th of December 2024, 6PM UTC.…

5 days ago

dataMinds Connect 2024 – Session Materials

The slides and scripts for my session "Tackling the Gaps & Islands Problem with T-SQL…

4 weeks ago

Connect to Power BI as a Guest User in another Tenant

Sometimes your Microsoft Entra ID account (formerly known as Azure Active Directory) is added as…

2 months ago

How to use a Script Activity in ADF as a Lookup

In Azure Data Factory (ADF, but also Synapse Pipelines and Fabric Pipelines), you have a…

4 months ago

Database Build Error – Incorrect syntax near DISTINCT

I wrote a piece of SQL that had some new T-SQL syntax in it: IS…

4 months ago

Speaking at dataMinds Connect 2024

I'm very excited to announce I've been selected as a speaker for dataMinds Connect 2024,…

5 months ago