You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Gengliang Wang (JIRA)" <ji...@apache.org> on 2019/04/29 05:46:00 UTC

[jira] [Updated] (SPARK-27136) Remove data source option check_files_exist

     [ https://issues.apache.org/jira/browse/SPARK-27136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gengliang Wang updated SPARK-27136:
-----------------------------------
    Issue Type: Sub-task  (was: Task)
        Parent: SPARK-27589

> Remove data source option check_files_exist
> -------------------------------------------
>
>                 Key: SPARK-27136
>                 URL: https://issues.apache.org/jira/browse/SPARK-27136
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Gengliang Wang
>            Assignee: Gengliang Wang
>            Priority: Major
>             Fix For: 3.0.0
>
>
> The data source option check_files_exist is introduced in In https://github.com/apache/spark/pull/23383 when the file source V2 framework is implemented. In the PR, FileIndex was created as a member of FileTable, so that we could implement partition pruning like 0f9fcab in the future. At that time FileIndexes will always be created for file writes, so we needed the option to decide whether to check file existence.
> After https://github.com/apache/spark/pull/23774, the option is not needed anymore.  This PR is to clean the option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org