You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "curtisr7 (via GitHub)" <gi...@apache.org> on 2025/02/27 23:08:12 UTC
[I] Context cancel doesn't work [spark-connect-go]
curtisr7 opened a new issue, #126:
URL: https://github.com/apache/spark-connect-go/issues/126
I have a long running `dataframe.Collect(ctx)` operation and if cancel the context the job continues to run.
Is there some way that I can programatically kill the job? I see that I can get a `PlanID` from the dataframe, but I don't see a way to do anything with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] Context cancel doesn't work [spark-connect-go]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on issue #126:
URL: https://github.com/apache/spark-connect-go/issues/126#issuecomment-3197391044
The server-side query will time-out by default after 5 minutes without the client connecting back. In addition with Spark 4 there are now a bunch more ways to cancel queries:
1. you can use tags to tag a session and then interrupt by tag see https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.SparkSession.interruptTag.html however, this isn't yet fully implemented as the public APIs in the Go DataFrame and session.
2. Spark 4 supports as well cancelling by operation ID - https://github.com/apache/spark/blob/master/sql/connect/common/src/main/protobuf/spark/connect/base.proto#L775-L786 however, as well this isn't implemented in the client yet.
3. The client should properly deal with client side cancelling of the request and cancel the query appropriately.
Generally, this seems like some work that would be great for someone to take and I'm happy to advise and review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org