You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "grundprinzip (via GitHub)" <gi...@apache.org> on 2025/11/03 12:59:54 UTC
[PR] [CONNECT]Add BatchExecutePlan RPC and reattach support [spark]
grundprinzip opened a new pull request, #52852:
URL: https://github.com/apache/spark/pull/52852
### What changes were proposed in this pull request?
This PR introduces a new BatchExecutePlan RPC for Spark Connect that allows clients to submit multiple execution plans in a single batch operation, minimizing RPC overhead. It also adds reattach support for both Scala and Python clients to consume results from batch-submitted operations.
Key changes:
1. New BatchExecutePlan RPC with unary request/response model
2. Support for client-provided operation IDs
3. Rollback mechanism for submission failures
4. Reattach methods in Scala and Python clients
5. Operations marked as reattachable for later result consumption
### Why are the changes needed?
- **Performance**: Reduces RPC overhead when submitting multiple operations
- **Flexibility**: Allows fire-and-forget execution pattern with later result retrieval
- **Control**: Provides rollback capability for submission failures
### Does this PR introduce any user-facing change?
Yes. New APIs added:
- Scala: `SparkConnectClient.batchExecute()` and `reattach()`
- Python: `SparkConnectClient.batch_execute()` and `reattach_execute()`
### How was this patch tested?
- 11 server-side tests in SparkConnectBatchExecuteSuite
- 11 client-side tests in SparkConnectClientBatchExecuteSuite
- All tests passing with proper formatting and linting
### Key Implementation Details:
- rollbackOnFailure only applies to submission failures (invalid UUID, duplicate operation ID), not execution failures
- Operations are submitted sequentially and execute independently
- Operations are marked as reattachable, allowing clients to consume results via reattach
- Comprehensive documentation clarifying the behavior
Closes #XXXXX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54165][CONNECT]Add BatchExecutePlan RPC and reattach support [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52852:
URL: https://github.com/apache/spark/pull/52852#issuecomment-3482949776
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54165][CONNECT]Add BatchExecutePlan RPC and reattach support [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #52852:
URL: https://github.com/apache/spark/pull/52852#issuecomment-3482947526
I created a JIRA for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54165][CONNECT]Add BatchExecutePlan RPC and reattach support [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #52852: [SPARK-54165][CONNECT]Add BatchExecutePlan RPC and reattach support
URL: https://github.com/apache/spark/pull/52852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [CONNECT]Add BatchExecutePlan RPC and reattach support [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #52852:
URL: https://github.com/apache/spark/pull/52852#issuecomment-3482233291
Absolutely, I'll do that as soon as it's more ready to leave draft mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54165][CONNECT]Add BatchExecutePlan RPC and reattach support [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #52852:
URL: https://github.com/apache/spark/pull/52852#issuecomment-3939793475
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org