Categories: Fabric

How to Retrieve all the Spark Session Configuration Variables in Microsoft Fabric

I was trying some stuff out in a notebook on top of a Microsoft Fabric Lakehouse. I was wondering what some of the default values are of the configuration variables, and if there’s an easy way to retrieve them all. Luckily there is. In the code, I’m using Scala because it has a nice GetAll() function.

%%spark
spark.conf.getAll

This returns a map type with all of the configurations and their current values:

Not exactly super useful to look through, so let’s try to dump this data into a table. Let’s store the map into a variable, convert it to a sequence and then to a data frame. Finally, we can write that data frame to a delta table:

%%spark
val m = spark.conf.getAll
val df = m.toSeq.toDF("config","value")
df.write.mode("overwrite").saveAsTable("session_config")

The result is a nicely query-able table:

UPDATE

I got an interesting comment on LinkedIn from Gerhard Brueckl:

So I tried the statement and it returns a big list of config values (which you can export to CSV):

However, this is a different list than the one I extracted in this blog post. My post is about session configs, while SET -v returns the Spark configuration. Nonetheless, still very interesting, especially because of the additional explanation for each config.

UPDATE 2

If you run SET without the -v parameter, you do get the current session config instead of the Spark config.

If you just want a quick export to CSV, this is a good option. If you want it to dump into a table in Fabric, the Scala option is better.


------------------------------------------------
Do you like this blog post? You can thank me by buying me a beer 🙂
Koen Verbeeck

Koen Verbeeck is a Microsoft Business Intelligence consultant at AE, helping clients to get insight in their data. Koen has a comprehensive knowledge of the SQL Server BI stack, with a particular love for Integration Services. He's also a speaker at various conferences.

Recent Posts

Free webinar – Tackling the Gaps and Islands Problem with T-SQL Window Functions

I'm hosting a free webinar at MSSQLTips.com at the 19th of December 2024, 6PM UTC.…

5 days ago

dataMinds Connect 2024 – Session Materials

The slides and scripts for my session "Tackling the Gaps & Islands Problem with T-SQL…

4 weeks ago

Connect to Power BI as a Guest User in another Tenant

Sometimes your Microsoft Entra ID account (formerly known as Azure Active Directory) is added as…

2 months ago

How to use a Script Activity in ADF as a Lookup

In Azure Data Factory (ADF, but also Synapse Pipelines and Fabric Pipelines), you have a…

4 months ago

Database Build Error – Incorrect syntax near DISTINCT

I wrote a piece of SQL that had some new T-SQL syntax in it: IS…

4 months ago

Speaking at dataMinds Connect 2024

I'm very excited to announce I've been selected as a speaker for dataMinds Connect 2024,…

5 months ago