You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "xi-db (via GitHub)" <gi...@apache.org> on 2025/06/04 11:15:37 UTC
[PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: second ExecutePlan with same operationId and plan reattaches [spark]
xi-db opened a new pull request, #51084:
URL: https://github.com/apache/spark/pull/51084
### What changes were proposed in this pull request?
In Spark Connect, queries can fail with the error INVALID_HANDLE.OPERATION_ALREADY_EXISTS, when a client retries an ExecutePlan RPC—often due to transient network issues—causing the server to receive the same request multiple times. Since each ExecutePlan request includes an operation_id, the server interprets the duplicate as an attempt to create an already existing operation, which results in the OPERATION_ALREADY_EXISTS exception. This behavior interrupts query execution and breaks the user experience under otherwise recoverable conditions.
To resolve this, the PR introduces idempotent handling of ExecutePlan on the server side. When a request with a previously seen operation_id is received with the same plan, instead of returning an error, the server now reattaches the response stream to the already running execution associated with that operation. This ensures that retries due to network flakiness no longer result in failed queries, thereby improving the resilience and robustness of query executions.
### Why are the changes needed?
It will improve the stability of Spark Connect in case of transient network issues.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Yes, new tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2126901403
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -381,4 +381,15 @@ object Connect {
.internal()
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10g")
+
+ val CONNECT_SESSION_IDEMPOTENT_EXECUTE_PLAN_ENABLED =
Review Comment:
I don't think we need to feature flag this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128584758
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
Once the first ExecutePlan response stream is established, it always retries with ReattachExecute. https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/client/reattach.py#L256-L262
This PR handles the case that retries happen before the first ExecutePlan response stream is established, where multiple ExecutePlan requests are received by server.
Thus, there won't be any duplicate response received by client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry will use ReattachExecute, but I was actually concerned about [the gRPC-level retry](https://github.com/apache/spark/blob/970ba3dbff28361424b15ccd54ef6a43983624c4/python/pyspark/sql/connect/client/retries.py), which may be transparent to the code pointer you gave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry uses ReattachExecute, but I was actually concerned about the [gRPC-level retry](https://grpc.io/docs/guides/retry/), which may be transparent to the code pointer you gave. And I was wondering if it's possible for a grpc retry (due to UNAVAILABLE not OPERATION_ALREADY_EXISTS) to occur before the iterator has been reset to the ReattachExecute iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128386126
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
So it's not possible for the client to receive the second response via ExecutePlan?
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
So it's not possible for the client to receive the second response via ExecutePlan if I understand correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128333483
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
No, the retries of ExecutePlan happen before client receiving any response. But ExecutePlan will always return results from the beginning, which is expected behaviour, because in the second ExecutePlan call, the first call is already dropped due to network issues, so the second call expects the first response.
The behaviour is tested in these line in tests
```
// the first response should be the same as the one we got before, because
// lastConsumedResponseId is unset and the server will start from the start of the stream.
assert(iter2.next().getResponseId == firstResponseId)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128386126
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
So it's not possible for the client to receive the second response via ExecutePlan? My concern is that there is an automatic retry and reattach between the first and second response in the gRPC layer, and the Spark Connect client may still be using the ExecutePlan—it's just not aware of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2127034975
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) && executeHolder.request.getPlan.equals(v.getPlan) =>
Review Comment:
It should be
```scala
case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
```
not
```scala
case Some(executeHolder) && executeHolder.request.getPlan.equals(v.getPlan) =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128386126
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
So, is it not possible for the client to receive the second response via ExecutePlan, which returns a stream of responses? My concern is that there is an automatic retry and reattach between the first and second response in the gRPC streaming layer, and the Spark Connect client may still be using the ExecutePlan—it's just not aware of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128741308
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
Exactly, the OPERATION_ALREADY_EXISTS we're trying to solve in this PR is caused by the gRPC-level retry on the [first ExecutePlan request](https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/client/reattach.py#L125-L127), because ExecutePlan is not idempotent. Now, with this PR, both ExecutePlan and ReattachExecute are idempotent, and the gRPC-level retry can work safely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2127034975
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) && executeHolder.request.getPlan.equals(v.getPlan) =>
Review Comment:
```scala
case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128325886
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
Will the client receive the same data multiple times due to the lack of a lastResponseId?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128388280
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
Yes, exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2127034975
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) && executeHolder.request.getPlan.equals(v.getPlan) =>
Review Comment:
It should be
```scala
case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
```
instead of
```scala
case Some(executeHolder) && executeHolder.request.getPlan.equals(v.getPlan) =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128335545
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
If the client already received at least one response, the reattachable execution logic on client side will call ReattachExecute, which handles the lastResponseId correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128386126
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
So, is it not possible for the client to receive the second response via ExecutePlan, which returns a stream of responses? My concern is that there is an automatic retry and reattach between the first and second response in the gRPC layer, and the Spark Connect client may still be using the ExecutePlan—it's just not aware of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry uses ReattachExecute, but I was actually concerned about [the gRPC-level retry](https://github.com/apache/spark/blob/970ba3dbff28361424b15ccd54ef6a43983624c4/python/pyspark/sql/connect/client/retries.py), which may be transparent to the code pointer you gave. And I was wondering if it's possible for a grpc retry (due to UNAVAILABLE not OPERATION_ALREADY_EXISTS) to occur before the iterator has been reset to the ReattachExecute iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on PR #51084:
URL: https://github.com/apache/spark/pull/51084#issuecomment-2940225393
cc @vicennial Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2126923820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -381,4 +381,15 @@ object Connect {
.internal()
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10g")
+
+ val CONNECT_SESSION_IDEMPOTENT_EXECUTE_PLAN_ENABLED =
Review Comment:
Makes sense, just removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128386126
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
So, is it not possible for the client to receive the second response via ExecutePlan, which returns a stream of responses? My concern is that there is an automatic retry and reattach between the first and second response in the gRPC layer, and the Spark Connect client may still be using the ExecutePlan—it's just not aware of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry will use ReattachExecute, but I was actually concerned about [the gRPC-level retry](https://github.com/apache/spark/blob/970ba3dbff28361424b15ccd54ef6a43983624c4/python/pyspark/sql/connect/client/retries.py), which may be transparent to the code pointer you gave. And I was wondering if it's possible for a retry to occur before the iterator has been reset to the ReattachExecute iterator.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry will use ReattachExecute, but I was actually concerned about [the gRPC-level retry](https://github.com/apache/spark/blob/970ba3dbff28361424b15ccd54ef6a43983624c4/python/pyspark/sql/connect/client/retries.py), which may be transparent to the code pointer you gave. And I was wondering if it's possible for a grpc retry to occur before the iterator has been reset to the ReattachExecute iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db I understand that the Spark Connect client-side retry will use ReattachExecute, but I am actually concerned about [the gRPC-level retry](https://github.com/apache/spark/blob/970ba3dbff28361424b15ccd54ef6a43983624c4/python/pyspark/sql/connect/client/retries.py), which may be transparent to the code pointer you gave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry uses ReattachExecute, but I was actually concerned about [the gRPC-level retry](https://github.com/apache/spark/blob/970ba3dbff28361424b15ccd54ef6a43983624c4/python/pyspark/sql/connect/client/retries.py), which may be transparent to the code pointer you gave. And I was wondering if it's possible for a grpc retry (due to UNAVAILABLE) to occur before the iterator has been reset to the ReattachExecute iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128333483
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
Yes, ExecutePlan will always return results from the beginning, which is expected behaviour, because in the second ExecutePlan call, the first call is already dropped due to network issues, so the second call expects the first response.
The behaviour is tested in these line in tests
```
// the first response should be the same as the one we got before, because
// lastConsumedResponseId is unset and the server will start from the start of the stream.
assert(iter2.next().getResponseId == firstResponseId)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry will use ReattachExecute, but I am actually concerned about [the gRPC-level retry](https://github.com/apache/spark/blob/970ba3dbff28361424b15ccd54ef6a43983624c4/python/pyspark/sql/connect/client/retries.py), which may be transparent to the code pointer you gave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128294405
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) && executeHolder.request.getPlan.equals(v.getPlan) =>
Review Comment:
Right, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry uses ReattachExecute, but I was actually concerned about the gRPC-level retry, which may be transparent to the code pointer you gave. And I was wondering if it's possible for a grpc retry (due to UNAVAILABLE not OPERATION_ALREADY_EXISTS) to occur before the iterator has been reset to the ReattachExecute iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #51084:
URL: https://github.com/apache/spark/pull/51084#issuecomment-2944570732
Merging to master/4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2127034975
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) && executeHolder.request.getPlan.equals(v.getPlan) =>
Review Comment:
```scala
case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "xi-db (via GitHub)" <gi...@apache.org>.
xi-db commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128586579
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@heyihong Thanks for fixing the compile error. The CI is green now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128325886
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
Will the client receive the same data multiple times due to the absence of a lastResponseId and the automatic retry and reattach mechanisms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128386126
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
So it's not possible for the client to receive the second response via ExecutePlan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #51084:
URL: https://github.com/apache/spark/pull/51084#discussion_r2128625820
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutePlanHandler.scala:
##########
@@ -19,32 +19,36 @@ package org.apache.spark.sql.connect.service
import io.grpc.stub.StreamObserver
+import org.apache.spark.SparkSQLException
import org.apache.spark.connect.proto
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender
+import org.apache.spark.sql.connect.config.Connect
class SparkConnectExecutePlanHandler(responseObserver: StreamObserver[proto.ExecutePlanResponse])
extends Logging {
def handle(v: proto.ExecutePlanRequest): Unit = {
- val executeHolder = SparkConnectService.executionManager.createExecuteHolder(v)
- try {
- executeHolder.eventsManager.postStarted()
- executeHolder.start()
- } catch {
- // Errors raised before the execution holder has finished spawning a thread are considered
- // plan execution failure, and the client should not try reattaching it afterwards.
- case t: Throwable =>
- SparkConnectService.executionManager.removeExecuteHolder(executeHolder.key)
- throw t
+ val previousSessionId = v.hasClientObservedServerSideSessionId match {
+ case true => Some(v.getClientObservedServerSideSessionId)
+ case false => None
}
+ val sessionHolder = SparkConnectService
+ .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId, previousSessionId)
+ val executeKey = ExecuteKey(v, sessionHolder)
- try {
- val responseSender =
- new ExecuteGrpcResponseSender[proto.ExecutePlanResponse](executeHolder, responseObserver)
- executeHolder.runGrpcResponseSender(responseSender)
- } finally {
- executeHolder.afterInitialRPC()
+ SparkConnectService.executionManager.getExecuteHolder(executeKey) match {
+ case None =>
+ // Create a new execute holder and attach to it.
+ SparkConnectService.executionManager
+ .createExecuteHolderAndAttach(executeKey, v, sessionHolder, responseObserver)
+ case Some(executeHolder) if executeHolder.request.getPlan.equals(v.getPlan) =>
+ // If the execute holder already exists with the same plan, reattach to it.
+ SparkConnectService.executionManager
+ .reattachExecuteHolder(executeHolder, responseObserver, None)
Review Comment:
@xi-db FYI, I understand that the Spark Connect client-side retry uses ReattachExecute, but I was actually concerned about [the gRPC-level retry](https://github.com/apache/spark/blob/970ba3dbff28361424b15ccd54ef6a43983624c4/python/pyspark/sql/connect/client/retries.py), which may be transparent to the code pointer you gave. And I was wondering if it's possible for a grpc retry to occur before the iterator has been reset to the ReattachExecute iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches [spark]
Posted by "asf-gitbox-commits (via GitHub)" <gi...@apache.org>.
asf-gitbox-commits closed pull request #51084: [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches
URL: https://github.com/apache/spark/pull/51084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org