You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/11 09:28:41 UTC
[GitHub] [spark] uncleGen opened a new pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
uncleGen opened a new pull request #28497:
URL: https://github.com/apache/spark/pull/28497
### What changes were proposed in this pull request?
Use KVStore instead to cache streaming query progress information.
### Why are the changes needed?
Streaming query progress information are cached twice in `StreamExecution` and `StreamingQueryStatusListener`. It is memory-wasting. We can make this two usage unified.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
update ut.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-638233281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626600275
**[Test build #122501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122501/testReport)** for PR 28497 at commit [`f415a48`](https://github.com/apache/spark/commit/f415a484ce76a63eb6535b0d34f074ce94f12f4f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-638051393
**[Test build #123474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123474/testReport)** for PR 28497 at commit [`4483ace`](https://github.com/apache/spark/commit/4483aceb174ca365019960387a8f2c67c6fdf5a3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-638051935
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626589051
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122497/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626588844
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-638231840
**[Test build #123474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123474/testReport)** for PR 28497 at commit [`4483ace`](https://github.com/apache/spark/commit/4483aceb174ca365019960387a8f2c67c6fdf5a3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626588259
**[Test build #122497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122497/testReport)** for PR 28497 at commit [`c4878dd`](https://github.com/apache/spark/commit/c4878dde090310545c1b43c640422e364587a75d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626600984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626588844
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-695139730
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626600984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626589025
**[Test build #122497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122497/testReport)** for PR 28497 at commit [`c4878dd`](https://github.com/apache/spark/commit/c4878dde090310545c1b43c640422e364587a75d).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] uncleGen commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
uncleGen commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-641884118
Pending on #28781
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626635758
Just to speak about numbers how much is the gain for a simple application? I would like to get some feelings what that could mean.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #28497:
URL: https://github.com/apache/spark/pull/28497
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626589041
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-638051935
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-638051393
**[Test build #123474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123474/testReport)** for PR 28497 at commit [`4483ace`](https://github.com/apache/spark/commit/4483aceb174ca365019960387a8f2c67c6fdf5a3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-638233281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626588259
**[Test build #122497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122497/testReport)** for PR 28497 at commit [`c4878dd`](https://github.com/apache/spark/commit/c4878dde090310545c1b43c640422e364587a75d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] uncleGen commented on pull request #28497: [SPARK-31677][SS] Use kvstore to cache stream query progress
Posted by GitBox <gi...@apache.org>.
uncleGen commented on pull request #28497:
URL: https://github.com/apache/spark/pull/28497#issuecomment-626586465
cc @xuanyuanking
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org