Using HiveContext.set in multipul threads

[Amir Gershman <am...@fb.com>]

Hi,

I have a DataFrame I compute from a long chain of transformations.
I cache it, and then perform two additional transformations on it.
I use two Futures - each Future will insert the content of one of the above Dataframe to a different hive table.
One Future must SET hive.exec.dynamic.partition=true and the other must set it to false.


How can I run both INSERT commands in parallel, but guarantee each runs with its own settings?


If I don't use the same HiveContext then the initial long chain of transformations which I cache is not reusable between HiveContexts. If I use the same HiveContext, race conditions between threads my cause one INSERT to execute with the wrong config.

Re: Using HiveContext.set in multipul threads

[Silvio Fiorito <si...@granturing.com>]

If you’re using DataFrame API you can achieve that by simply using (or not) the “partitionBy” method on the DataFrameWriter:

val originalDf = ….

val df1 = originalDf….
val df2 = originalDf…

df1.write.partitionBy(”col1”).save(…)

df2.write.save(…)

From: Amir Gershman <am...@fb.com>
Date: Tuesday, May 24, 2016 at 7:01 AM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Using HiveContext.set in multipul threads

Hi,

I have a DataFrame I compute from a long chain of transformations.
I cache it, and then perform two additional transformations on it.
I use two Futures - each Future will insert the content of one of the above Dataframe to a different hive table.
One Future must SET hive.exec.dynamic.partition=true and the other must set it to false.



How can I run both INSERT commands in parallel, but guarantee each runs with its own settings?



If I don't use the same HiveContext then the initial long chain of transformations which I cache is not reusable between HiveContexts. If I use the same HiveContext, race conditions between threads my cause one INSERT to execute with the wrong config.