You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gengliang Wang (JIRA)" <ji...@apache.org> on 2019/04/29 05:47:00 UTC
[jira] [Updated] (SPARK-26871) File Source V2: avoid creating
unnecessary FileIndex in the write path
[ https://issues.apache.org/jira/browse/SPARK-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gengliang Wang updated SPARK-26871:
-----------------------------------
Issue Type: Sub-task (was: Task)
Parent: SPARK-27589
> File Source V2: avoid creating unnecessary FileIndex in the write path
> ----------------------------------------------------------------------
>
> Key: SPARK-26871
> URL: https://issues.apache.org/jira/browse/SPARK-26871
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Gengliang Wang
> Assignee: Gengliang Wang
> Priority: Major
> Fix For: 3.0.0
>
>
> In https://github.com/apache/spark/pull/23383, the file source V2 framework is implemented. In the PR, FileIndex is created as a member of FileTable, so that we can implement partition pruning like https://github.com/apache/spark/commit/0f9fcabb4ac2e8afec14d010e86467372a85d334 in the future(As data source V2 catalog is under development, partition pruning is removed from the PR)
> However, after write path of file source V2 is implemented, I find that a simple write will create FileIndex, which is required by FileTable. This is a sort of regression.
> This PR is to make FileIndex as a lazy value in FileTable, so that we can avoid creating unnecessary FileIndex in the write path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org