You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by prosp4300 <pr...@163.com> on 2015/06/30 15:41:57 UTC
DataFrame registerTempTable Concurrent Access
Hi, Spark Users
I'm trying to update registered DataFrame temp table by invoke DataFrame.registerTempTable again and again.
Assume I have DataFrame temp table "table1", below is the concurrent logic
sqlContext.table("table1").filter(***).unionAll(dummy1DF).registerTempTable("table1")
sqlContext.table("table1").filter(***).unionAll(dummy2DF).registerTempTable("table1")
And I expect after above logic completed, the final temp table "table1" contain all the changes.
Is this a correct expectation even without proper locking system to make the executing ordered?
Such updating logic seems not functional programming, correct?
Thanks a lot
prosp4300