You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andreas Chatzistergiou (Jira)" <ji...@apache.org> on 2022/01/18 16:53:00 UTC
[jira] [Created] (SPARK-37955) PartitioningAwareFileIndex->basePath incorrectly contains the partition filters
Andreas Chatzistergiou created SPARK-37955:
----------------------------------------------
Summary: PartitioningAwareFileIndex->basePath incorrectly contains the partition filters
Key: SPARK-37955
URL: https://issues.apache.org/jira/browse/SPARK-37955
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.2.0
Reporter: Andreas Chatzistergiou
PartitioningAwareFileIndex.getBasePath method returns paths that contain the partitioning directories. This violates the definition of the basePath per FileIndex, i.e. the parent directory of a file path with all the partitioning directories being stripped off.
This PR fixes the issue by separating the notion of the partitioningPaths and the basePaths in the PartitioningAwareFileIndex. The basePaths are derived by removing from the partitioningPaths any partitioning columns with the aid of the PartitioningSchema.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org