You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by prosp4300 <pr...@163.com> on 2015/06/30 15:41:57 UTC

DataFrame registerTempTable Concurrent Access

Hi, Spark Users

I'm trying to update registered DataFrame temp table by invoke DataFrame.registerTempTable again and again.

Assume I have DataFrame temp table "table1", below is the concurrent logic

sqlContext.table("table1").filter(***).unionAll(dummy1DF).registerTempTable("table1")
sqlContext.table("table1").filter(***).unionAll(dummy2DF).registerTempTable("table1")

And I expect after above logic completed, the final temp table "table1" contain all the changes.
Is this a correct expectation even without proper locking system to make the executing ordered?
Such updating logic seems not functional programming, correct?

Thanks a lot
prosp4300