You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2026/02/06 04:24:03 UTC

[PR] [WIP][CORE][SQL] Mitigate the recomputation in zipWithIndex [spark]

zhengruifeng opened a new pull request, #54169:
URL: https://github.com/apache/spark/pull/54169

   ### What changes were proposed in this pull request?
   Mitigate the recomputation in `zipWithIndex`
   
   ### Why are the changes needed?
   `zipWithIndex` triggers an extra job to compute the `startIndices`.
   If the parent RDD holds the same data distribution: number of partitions and number of rows per partition, then we can use parent RDD to compute the `startIndices`, so that the internal computation might be skipped.
   
   It should benefit such patterns:
   
   ```
   rdd.map(expensive computation).zipWithIndex 
   
   df.select(expensive computation).zipWithIndex 
   ```
   
   It should be able to add other operators, but this PR focus on `RDD.map` and `ProjectExec`.
   
   
   manually check with
   ```scala
   val rdd = sc.range(0, 10, 1, 4)
   val start = System.currentTimeMillis()
   rdd.map(x => {Thread.sleep(10000); x + 1}).zipWithIndex().collect()
   val duration = System.currentTimeMillis() - start
   ```
   
   master:
   ```scala
   val rdd: org.apache.spark.rdd.RDD[Long] = MapPartitionsRDD[1] at range at <console>:1
   val start: Long = 1770351594037
   val duration: Long = 60651
   ```
   
   this PR:
   ```scala
   val rdd: org.apache.spark.rdd.RDD[Long] = MapPartitionsRDD[12] at range at <console>:1
   val start: Long = 1770351397114
   val duration: Long = 30040
   ```
   
   The original RDD `sc.range(0, 10, 1, 4)` was used to compute `startIndices`, thus the expensive computation `x => {Thread.sleep(10000); x + 1}` was skipped. 
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   CI, will add tests
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [WIP][CORE][SQL] Mitigate the recomputation in zipWithIndex [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54169:
URL: https://github.com/apache/spark/pull/54169#issuecomment-3857879180

   ## ⚠️ Pull Request Title Validation
   
   This pull request title does not contain a JIRA issue ID.
   
   Please update the title to either:
   - Include a JIRA ID: `[SPARK-12345] Your description`
   - Mark as minor change: `[MINOR] Your description`
   
   For minor changes that don't require a JIRA ticket (e.g., typo fixes), please prefix the title with `[MINOR]`.
   
   ---
   *This comment was automatically generated by GitHub Actions*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-55385][CORE][SQL] Mitigate the recomputation in `zipWithIndex` [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #54169: [SPARK-55385][CORE][SQL] Mitigate the recomputation in `zipWithIndex`
URL: https://github.com/apache/spark/pull/54169


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-55385][CORE][SQL] Mitigate the recomputation in `zipWithIndex` [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54169:
URL: https://github.com/apache/spark/pull/54169#issuecomment-3868406474

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org