You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "juliuszsompolski (via GitHub)" <gi...@apache.org> on 2023/09/12 10:08:53 UTC
[GitHub] [spark] juliuszsompolski opened a new pull request, #42889: [SPARK-45133] Make Spark Connect queries be FINISHED when last result task is finished
juliuszsompolski opened a new pull request, #42889:
URL: https://github.com/apache/spark/pull/42889
### What changes were proposed in this pull request?
In the situation before, query will only be FINISHED when all results have been pushed into the output buffers (not necessarily received by client, but pushed out of the server).
For LocalTableScanExec, post FINISHED before sending result batches, because nothing is executed, only cached local results are returned. For regular execution, post FINISHED after all task results have been returned from Spark, not after they have been processed and sent out.
### Why are the changes needed?
Currently, even if a query finished running in Spark, it keeps being RUNNING until all results are sent. Then there is a very small difference between FINISHED and CLOSED. This change makes it behave more similar to e.g. Thriftserver.
### Does this PR introduce _any_ user-facing change?
Yes. Queries will be posted as FINISHED when they finish executing, not when they finish sending results.
### How was this patch tested?
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #42889: [SPARK-45133][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #42889:
URL: https://github.com/apache/spark/pull/42889#issuecomment-1716953774
Hi, @juliuszsompolski and @HyukjinKwon .
I hoped this recovers the master branch, but it seems that CI is still failing since the commit which `ReattachableExecuteSuite` is added. Could you check this comment?
- https://github.com/apache/spark/pull/42560#pullrequestreview-1623698495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #42889: [SPARK-45133][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #42889: [SPARK-45133][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished
URL: https://github.com/apache/spark/pull/42889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on pull request #42889: [SPARK-45133] Make Spark Connect queries be FINISHED when last result task is finished
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #42889:
URL: https://github.com/apache/spark/pull/42889#issuecomment-1715607985
cc @jdesjean @gjxdxh @gengliangwang @hvanhovell @grundprinzip
I think it kind of regressed in https://github.com/apache/spark/pull/42454
I think it's more logical this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jdesjean commented on pull request #42889: [SPARK-45133] Make Spark Connect queries be FINISHED when last result task is finished
Posted by "jdesjean (via GitHub)" <gi...@apache.org>.
jdesjean commented on PR #42889:
URL: https://github.com/apache/spark/pull/42889#issuecomment-1716551621
The change seems logical to me as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on pull request #42889: [SPARK-45133][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #42889:
URL: https://github.com/apache/spark/pull/42889#issuecomment-1717158555
In description I wrote "Will add test in https://github.com/apache/spark/pull/42560", but since that one was merged yesterday, I planned to rebase and add test here. I will add test in a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42889: [SPARK-45133][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42889:
URL: https://github.com/apache/spark/pull/42889#issuecomment-1716870850
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org