You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Felipe Pessoto <fe...@hotmail.com> on 2023/01/04 00:14:41 UTC

How to set a config for a single query?

Hi,

In Scala is it possible to set a config value to a single query?

I could set/unset the value, but it won't work for multithreading scenarios.

Example:

spark.sql.adaptive.coalescePartitions.enabled = false
                queryA_df.collect
spark.sql.adaptive.coalescePartitions.enabled=original value
                queryB_df.collect
                queryC_df.collect
                queryD_df.collect


If I execute that block of code multiple times using multiple thread, I can end up executing Query A with coalescePartitions.enabled=true, and Queries B, C and D with the config set to false, because another thread could set it between the executions.

Is there any good alternative to this?

Thanks.

Re: How to set a config for a single query?

Posted by Shay Elbaz <sh...@gm.com>.

Hi Felipe,

I had the same problem - needed to execute multiple jobs/actions multithreaded, with slightly different sql configs per job (mainly spark.sql.shuffle.partitions). I'm not sure if this is the best solution, but I ended up using newSession() per thread. It works well except for the new SparkSession does not contain custom configurations from the original session. I had to re-apply the important configurations (catalogs, etc.) on the new Sessions as well. Hope that helps.

Shay
________________________________
From: Saurabh Gulati <sa...@fedex.com.INVALID>
Sent: Wednesday, January 4, 2023 11:54 AM
To: Felipe Pessoto <fe...@hotmail.com>; user@spark.apache.org <us...@spark.apache.org>
Subject: [EXTERNAL] Re: How to set a config for a single query?

ATTENTION: This email originated from outside of GM.

Hey Felipe,
Since you are collecting the dataframes, you might as well run them separately with desired configs and store them in your storage.

Regards
Saurabh
________________________________
From: Felipe Pessoto <fe...@hotmail.com>
Sent: 04 January 2023 01:14
To: user@spark.apache.org <us...@spark.apache.org>
Subject: [EXTERNAL] How to set a config for a single query?

Caution! This email originated outside of FedEx. Please do not open attachments or click links from an unknown or suspicious origin.

Hi,

In Scala is it possible to set a config value to a single query?

I could set/unset the value, but it won’t work for multithreading scenarios.

Example:

spark.sql.adaptive.coalescePartitions.enabled = false

                queryA_df.collect

spark.sql.adaptive.coalescePartitions.enabled=original value

                queryB_df.collect

                queryC_df.collect

                queryD_df.collect

If I execute that block of code multiple times using multiple thread, I can end up executing Query A with coalescePartitions.enabled=true, and Queries B, C and D with the config set to false, because another thread could set it between the executions.

Is there any good alternative to this?

Thanks.

Re: How to set a config for a single query?

Posted by Saurabh Gulati <sa...@fedex.com.INVALID>.

Hey Felipe,
Since you are collecting the dataframes, you might as well run them separately with desired configs and store them in your storage.

Regards
Saurabh
________________________________
From: Felipe Pessoto <fe...@hotmail.com>
Sent: 04 January 2023 01:14
To: user@spark.apache.org <us...@spark.apache.org>
Subject: [EXTERNAL] How to set a config for a single query?

Caution! This email originated outside of FedEx. Please do not open attachments or click links from an unknown or suspicious origin.

Hi,

In Scala is it possible to set a config value to a single query?

I could set/unset the value, but it won’t work for multithreading scenarios.

Example:

spark.sql.adaptive.coalescePartitions.enabled = false

                queryA_df.collect

spark.sql.adaptive.coalescePartitions.enabled=original value

                queryB_df.collect

                queryC_df.collect

                queryD_df.collect

If I execute that block of code multiple times using multiple thread, I can end up executing Query A with coalescePartitions.enabled=true, and Queries B, C and D with the config set to false, because another thread could set it between the executions.

Is there any good alternative to this?

Thanks.

Re: How to set a config for a single query?

Posted by Khalid Mammadov <kh...@gmail.com>.

Hi

I believe there is a feature in Spark specifically for this purpose. You
can create a new spark session and set those configs.
Note that it's not the same as creating a separate driver processes with
separate sessions, here you will still have the same SparkContext that
works as a backend for both or more spark sessions and does all the heavy
work.

*spark.newSession()*

https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.SparkSession.newSession.html#pyspark.sql.SparkSession.newSession

Hope this helps
Khalid


On Wed, 4 Jan 2023, 00:25 Felipe Pessoto, <fe...@hotmail.com> wrote:

> Hi,
>
>
>
> In Scala is it possible to set a config value to a single query?
>
>
>
> I could set/unset the value, but it won’t work for multithreading
> scenarios.
>
>
>
> Example:
>
>
>
> spark.sql.adaptive.coalescePartitions.enabled = false
>
>                 queryA_df.collect
>
> spark.sql.adaptive.coalescePartitions.enabled=original value
>
>                 queryB_df.collect
>
>                 queryC_df.collect
>
>                 queryD_df.collect
>
>
>
>
>
> If I execute that block of code multiple times using multiple thread, I
> can end up executing Query A with coalescePartitions.enabled=true, and
> Queries B, C and D with the config set to false, because another thread
> could set it between the executions.
>
>
>
> Is there any good alternative to this?
>
>
>
> Thanks.
>

[UNSUBSCRIBE]

Posted by Sebastian Schere <ss...@gmail.com>.

UNSUBSCRIBE