[jira] [Updated] (SPARK-19798) Query returns stale results when tables are modified on other sessions

["Giambattista (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-19798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giambattista updated SPARK-19798:
---------------------------------
    Affects Version/s: 3.0.0

> Query returns stale results when tables are modified on other sessions
> ----------------------------------------------------------------------
>
>                 Key: SPARK-19798
>                 URL: https://issues.apache.org/jira/browse/SPARK-19798
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0, 3.0.0
>            Reporter: Giambattista
>            Priority: Major
>
> I observed the problem on master branch with thrift server in multisession mode (default), but I was able to replicate also with spark-shell as well (see below the sequence for replicating).
> I observed cases where changes made in a session (table insert, table renaming) are not visible to other derived sessions (created with session.newSession).
> The problem seems due to the fact that each session has its own tableRelationCache and it does not get refreshed.
> IMO tableRelationCache should be shared in sharedState, maybe in the cacheManager so that refresh of caches for data that is not session-specific such as temporary tables gets centralized.  
> --- Spark shell script
> val spark2 = spark.newSession
> spark.sql("CREATE TABLE test (a int) using parquet")
> spark2.sql("select * from test").show // OK returns empty
> spark.sql("select * from test").show // OK returns empty
> spark.sql("insert into TABLE test values 1,2,3")
> spark2.sql("select * from test").show // ERROR returns empty
> spark.sql("select * from test").show // OK returns 3,2,1
> spark.sql("create table test2 (a int) using parquet")
> spark.sql("insert into TABLE test2 values 4,5,6")
> spark2.sql("select * from test2").show // OK returns 6,4,5
> spark.sql("select * from test2").show // OK returns 6,4,5
> spark.sql("alter table test rename to test3")
> spark.sql("alter table test2 rename to test")
> spark.sql("alter table test3 rename to test2")
> spark2.sql("select * from test").show // ERROR returns empty
> spark.sql("select * from test").show // OK returns 6,4,5
> spark2.sql("select * from test2").show // ERROR throws java.io.FileNotFoundException
> spark.sql("select * from test2").show // OK returns 3,1,2



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org