You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/07/28 21:49:43 UTC
[PR] [SPARK-58413][SQL][TESTS] Rename `sequential fetch` label to `pipelined fetch (1 client)` in `NettyTransportBenchmark` [spark]
dongjoon-hyun opened a new pull request, #57616:
URL: https://github.com/apache/spark/pull/57616
### What changes were proposed in this pull request?
This PR renames the benchmark case label `sequential fetch` to `pipelined fetch (1 client)` in the `File-Backed Shuffle Block Fetch` suite of `NettyTransportBenchmark`.
### Why are the changes needed?
The label is misleading. The case fires all 100 `fetchChunk` requests at once on a single connection and then waits for all of them to complete via a semaphore (`fetchChunksSync`), so the client never waits for one chunk before requesting the next. This is a pipelined fetch over one connection, not a sequential request-response loop.
The mislabel skews the interpretation of the results. For example:
```
NIO, sequential fetch 384 386 3 0.0 3844405.1 1.0X
NIO, parallel fetch (4 clients) 209 217 9 0.0 2091358.0 1.8X
```
Read as "sequential vs parallel", the 1.8X looks like poor parallelization efficiency. The actual comparison is "pipelined over 1 socket vs pipelined over 4 sockets", where the single pipelined connection already runs at ~4.2 GB/s. The new label makes this clear and contrasts naturally with `parallel fetch (4 clients)`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Label-only change; manually reviewed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58413][SQL][TESTS] Rename `sequential fetch` label to `pipelined fetch (1 client)` in `NettyTransportBenchmark` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57616:
URL: https://github.com/apache/spark/pull/57616#issuecomment-5114723585
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/1fba3cc49139a5130febc2f408b378cc8b64d598
- merged into branch-4.x https://github.com/apache/spark/commit/f8bbbdfccc9645032ffc35bddfd84abe3e8d7b2d
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58413][SQL][TESTS] Rename `sequential fetch` label to `pipelined fetch (1 client)` in `NettyTransportBenchmark` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57616:
URL: https://github.com/apache/spark/pull/57616#issuecomment-5114717145
Thank you, @uros-b .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58413][SQL][TESTS] Rename `sequential fetch` label to `pipelined fetch (1 client)` in `NettyTransportBenchmark` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57616:
URL: https://github.com/apache/spark/pull/57616#issuecomment-5110054933
cc @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58413][SQL][TESTS] Rename `sequential fetch` label to `pipelined fetch (1 client)` in `NettyTransportBenchmark` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #57616: [SPARK-58413][SQL][TESTS] Rename `sequential fetch` label to `pipelined fetch (1 client)` in `NettyTransportBenchmark`
URL: https://github.com/apache/spark/pull/57616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58413][SQL][TESTS] Rename `sequential fetch` label to `pipelined fetch (1 client)` in `NettyTransportBenchmark` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57616:
URL: https://github.com/apache/spark/pull/57616#issuecomment-5113387146
Thank you, @LuciferYang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org