You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/29 15:20:19 UTC
[GitHub] [spark] Olwn opened a new pull request #29578: Fix batch submission delay caused by actions in dstream transform
Olwn opened a new pull request #29578:
URL: https://github.com/apache/spark/pull/29578
### What changes were proposed in this pull request?
Currently dstream.getOrCompute runs at JobGenerator, which has a single thread event loop.
This pull request moves that to JobScheduler.
### Why are the changes needed?
Some of our spark applications have batch creation delay after running for some time. For instance, Batch 10:03 is submitted at 10:06. In spark UI, the latest batch doesn't match current time.
We observe such applications share a commonality that rdd actions exist in dstream.transfrom. Those actions will be executed in dstream.compute, which is called by JobGenerator. JobGenerator runs with a single thread event loop so any synchronized operations will block event processing.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added two tests
1. ForEachDStreamSuite to make sure batch execution doesn't block batch submission
2. JobSchedulerSuite to make sure all jobs in a batch can be associated with the BatchTime and display at Spark UI
### JIRAs
https://issues.apache.org/jira/browse/SPARK-32734
https://issues.apache.org/jira/browse/SPARK-32735
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Olwn commented on pull request #29578: [SPARK-32734][SPARK-32735][DSTREAM] Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
Olwn commented on pull request #29578:
URL: https://github.com/apache/spark/pull/29578#issuecomment-688851023
hi @tdas , could you help review this pull request? Seems you are the main contributor of spark streaming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Olwn commented on pull request #29578: [SPARK-32734][SPARK-32735][DSTREAM] Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
Olwn commented on pull request #29578:
URL: https://github.com/apache/spark/pull/29578#issuecomment-686174954
Thanks for correcting the title @maropu. I have read the contributing guide but missed this point.
Regarding `[SPARK-32734](https://issues.apache.org/jira/browse/SPARK-32734)` and `[SPARK-32735](https://issues.apache.org/jira/browse/SPARK-32735)` , yes this PR is a small change but fixes both issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29578: Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29578:
URL: https://github.com/apache/spark/pull/29578#issuecomment-683305195
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Olwn edited a comment on pull request #29578: [SPARK-32734][SPARK-32735][DSTREAM] Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
Olwn edited a comment on pull request #29578:
URL: https://github.com/apache/spark/pull/29578#issuecomment-686174954
Thanks for correcting the title @maropu. I have read the contributing guide but missed this point.
Regarding [SPARK-32734](https://issues.apache.org/jira/browse/SPARK-32734) and [SPARK-32735](https://issues.apache.org/jira/browse/SPARK-32735), yes this PR is a small change but fixes both issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #29578: [SPARK-32734][SPARK-32735][DSTREAM] Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #29578:
URL: https://github.com/apache/spark/pull/29578
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29578: [SPARK-32734][SPARK-32735][DSTREAM] Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29578:
URL: https://github.com/apache/spark/pull/29578#issuecomment-686130011
Thanks for the work, @Olwn. Btw, have you checked the guide? https://spark.apache.org/contributing.html You need to set correct tags in the title like `[SPARK-XXXXX][some_component_tag]`, so I've fixed it. And, do you target to fix both issue `SPARK-32734` and `SPARK-32735` in a single PR? Basically, we need to make a PR for each issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29578: Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29578:
URL: https://github.com/apache/spark/pull/29578#issuecomment-683305035
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #29578: [SPARK-32734][SPARK-32735][DSTREAM] Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #29578:
URL: https://github.com/apache/spark/pull/29578#issuecomment-747802131
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29578: Fix batch submission delay caused by actions in dstream transform
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29578:
URL: https://github.com/apache/spark/pull/29578#issuecomment-683305035
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org