You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dirkraft <gi...@git.apache.org> on 2017/08/04 23:35:30 UTC
[GitHub] spark pull request #18850: Remove dropping bus logic.
GitHub user dirkraft opened a pull request:
https://github.com/apache/spark/pull/18850
Remove dropping bus logic.
Log warning when event uptake takes too long, but block anyways. Since dropping events is no longer possible, removed that code.
## What changes were proposed in this pull request?
LiveListenerBus events are now never dropped. I believe that critical components communicate through this bus. I will also add the argument that a buggy spark UI (because of dropped events) is just as useless, and so I am unaware of any kind of event which can be lossy.
## How was this patch tested?
Basically we ran it in our production environment. Our past couple spark runs now reliably complete with this change since crucial events never get dropped, but it is unclear how much lag this change might be contributing to the overall run time.
We have partition counts in the thousands and executors in the hundreds.
----
Perhaps the events can be tagged as critical (`queue.put`) or not (`queue.offer`), but this small change is meant to get spark stable again. We have turned up the eventqueue size to 1,000,000 (and fiddled with all available settings), but it still isn't enough. With some probability, enough crucial events are dropped and leaves the spark job hung indefinitely unable to recover (sometimes it does). The large eventqueue also maxed out our driver process with 64GB of memory, so that's pretty much untenable.
With this change and over tens (hundreds?) of millions of events flowing through this bus, only 1100 triggered the slow warning, usually around 20ms with a max of 100ms. The current working hypothesis is that the events tend to arrive in bursts and so quickly overwhelm the queue and then quickly empty out.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dirkraft/spark dont-drop-events-ever
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18850.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18850
----
commit 56c4c1d87716b6f629be83fff8197b36a607a9ae
Author: Jason Dunkelberger <ja...@allenai.org>
Date: 2017-08-04T23:21:02Z
Remove droppy logic. Log warning when event uptake takes too long, but block anyways.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18850: Remove dropping bus logic.
Posted by dirkraft <gi...@git.apache.org>.
Github user dirkraft closed the pull request at:
https://github.com/apache/spark/pull/18850
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18850: Remove dropping bus logic.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18850
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18850: Remove dropping bus logic.
Posted by dirkraft <gi...@git.apache.org>.
Github user dirkraft commented on the issue:
https://github.com/apache/spark/pull/18850
Closing out this noise for now. Discussions continue in https://issues.apache.org/jira/browse/SPARK-18838
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org