[jira] [Created] (SPARK-32716) Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD

["Patrick Grandjean (Jira)" <ji...@apache.org>]

Patrick Grandjean created SPARK-32716:
-----------------------------------------

             Summary: Add function mapPartitionsWithInputSplitAndIndex to NewHadoopRDD
                 Key: SPARK-32716
                 URL: https://issues.apache.org/jira/browse/SPARK-32716
             Project: Spark
          Issue Type: New Feature
          Components: Spark Core
    Affects Versions: 3.0.0
            Reporter: Patrick Grandjean


This is a proposal to add the following function to NewHadoopRDD:
{code:java}
def mapPartitionsWithInputSplitAndIndex[U: ClassTag](
    f: (InputSplit, Int, Iterator[(K, V)]) => Iterator[U],
    preservesPartitioning: Boolean = false
): RDD[U]
{code}
This new function would provide the input split with its partition index.

Having the partition index may be useful when trying to determine the mapping between part-XXXXX files and input splits.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org