You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zeruibao (via GitHub)" <gi...@apache.org> on 2025/09/12 22:11:53 UTC
[PR] [SPARK-53361][SS][1/2] Optimizing JVM–Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch [spark]
zeruibao opened a new pull request, #52331:
URL: https://github.com/apache/spark/pull/52331
### What changes were proposed in this pull request?
This PR introduces an optimization to JVM–Python communication in TWS by allowing multiple keys to be grouped into a single Arrow batch.
Currently, each Arrow batch is restricted to contain records for a single key. In high-cardinality scenarios, this results in many small Arrow batches (e.g., [(key1, value1), (key1, value2)], [(key2, value1), (key2, value2)]), which increases the overhead of Arrow batch transmission between the JVM and Python.
With this change, records with different keys can be bin-packed into the same Arrow batch, reducing the number of batches transmitted. On the Python side, we leverage groupBy to regroup records by key, mirroring the behavior of the Scala GroupedIterator implementation.
This PR only handle `TransformWithStateInPySparkPythonRunner`. `TransformWithStateInPySparkPythonInitialStateRunner` would only affect the batch 0 so that we will leave to another PR.
This approach significantly reduces transmission overhead while preserving correct grouping semantics.
### Why are the changes needed?
Benchmark results show that in high-cardinality scenarios, this optimization improves throughput by ~20% by reducing the overhead of Arrow batch transmission. For low-cardinality scenarios, the change introduces no observable regression,.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing UT and Benchmark.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53361][SS][1/2] Optimizing JVM–Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch [spark]
Posted by "zeruibao (via GitHub)" <gi...@apache.org>.
zeruibao commented on PR #52331:
URL: https://github.com/apache/spark/pull/52331#issuecomment-3290689385
> The change looks good to me. Could you please provide the performance improvement for both Pandas type and Row type? Is it both ~20%? Let's update the PR description.
Hey @HeartSaVioR, just update the PR description with performance improvement. Both Pandas and Pyspark can provide > 20% improvement. But the specific improvement number would depends the number of shuffle partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53361][SS][1/2] Optimizing JVM–Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #52331: [SPARK-53361][SS][1/2] Optimizing JVM–Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch
URL: https://github.com/apache/spark/pull/52331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53361][SS][1/2] Optimizing JVM–Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch [spark]
Posted by "zeruibao (via GitHub)" <gi...@apache.org>.
zeruibao commented on PR #52331:
URL: https://github.com/apache/spark/pull/52331#issuecomment-3293031078
> Thanks @zeruibao for your contribution, I merged this to master. Let's file another ticket for the second one - we need to mark the ticket as resolved whenever the fix has landed, so that we track the change properly.
Thanks! Just create one https://issues.apache.org/jira/browse/SPARK-53583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53361][SS][1/2] Optimizing JVM–Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #52331:
URL: https://github.com/apache/spark/pull/52331#issuecomment-3290939550
Thanks @zeruibao for your contribution, I merged this to master.
Let's file another ticket for the second one - we need to mark the ticket as resolved whenever the fix has landed, so that we track the change properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53361][SS][1/2] Optimizing JVM–Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #52331:
URL: https://github.com/apache/spark/pull/52331#issuecomment-3290932980
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org