[jira] [Created] (SPARK-27827) File does not exist notice is misleading in FileScanRDD

["zhoukang (JIRA)" <ji...@apache.org>]

zhoukang created SPARK-27827:
--------------------------------

             Summary: File does not exist notice is misleading in FileScanRDD
                 Key: SPARK-27827
                 URL: https://issues.apache.org/jira/browse/SPARK-27827
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.4.3, 2.3.2
            Reporter: zhoukang


When we encounter error below, we will try "refresh table" and will think the error will not thrown again.
{code:java}
Error: java.lang.IllegalStateException: Can't overwrite cause with java.io.FileNotFoundException: File does not exist: /user/s_xdata/kuduhive_warehouse/info_dev/dws_quality_time_dictionary/part-00003-92c84bf9-99c0-49d9-8cdf-78b1844d75c3.snappy.parquet
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved. (state=,code=0)
{code}
The cause is 'InMemoryFileIndex' will be cached in 'HiveMetaStoreCatalog'.And refresh command will only invalidate table of current session.The notice is misleading when we have a long-running thriftserver.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org