You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by deshanxiao <gi...@git.apache.org> on 2018/08/15 03:06:45 UTC
[GitHub] spark pull request #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem...
GitHub user deshanxiao opened a pull request:
https://github.com/apache/spark/pull/22109
[SPARK-25120][CORE][HistoryServer]Fix the problem of EventLogListener may miss driver SparkListenerBloc…
## What changes were proposed in this pull request?
Sometimes in spark history tab "Executors" , it couldn't find driver information because the information of this page created by EventLog replaying the event of SparkListenerBlockManagerAdded. In SparkContext, driver registers blockmanager before adding the EventLogEventListener to LiveListenerBus. In this case,the EventLogEventListener may miss the driver's SparkListenerBlockManagerAdded event and the history ui won't show the info of driver in "Executors" .
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/deshanxiao/spark fix-jira25120
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22109.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22109
----
commit 26ca9c2c08c62961183e6461183c2963b6a00474
Author: xiaodeshan <xi...@...>
Date: 2018-08-15T02:49:25Z
fix the problem of EventLogListener may miss driver SparkListenerBlockManagerAdded event
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22109
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/22109
can you please close this pr @deshanxiao ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by deshanxiao <gi...@git.apache.org>.
Github user deshanxiao commented on the issue:
https://github.com/apache/spark/pull/22109
@squito @vanzin Thanks, the first time I find it in our cluster is Spark2.1. Spark2.1 has the method `setupAndStartListenerBus` too, but it still looks like wrong. The phenomenon of executor lose I find it in yesterday. Maybe we should fix them together.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22109
To be more explicit: events should be queued until `SparkContext.setupAndStartListenerBus()` is called, at which point the queued events should be delivered to all registered listeners even if they were added after the event was posted.
Since that method is called after the logging listener is installed, it should be getting all events. It it's not, there's a race in the listener bus, and this would be the wrong fix.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22109
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22109
**[Test build #4263 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4263/testReport)** for PR 22109 at commit [`26ca9c2`](https://github.com/apache/spark/commit/26ca9c2c08c62961183e6461183c2963b6a00474).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22109
**[Test build #94807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94807/testReport)** for PR 22109 at commit [`26ca9c2`](https://github.com/apache/spark/commit/26ca9c2c08c62961183e6461183c2963b6a00474).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/22109
@vanzin @squito
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22109
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by deshanxiao <gi...@git.apache.org>.
Github user deshanxiao commented on the issue:
https://github.com/apache/spark/pull/22109
please help me cc @HyukjinKwon
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22109
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94807/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22109
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22109
Now that you mention that, the symptoms here look very similar to SPARK-22850. But this one was filed against 2.3.1...
I wonder if there's still some race in the queuing of initial events that we didn't catch.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem...
Posted by deshanxiao <gi...@git.apache.org>.
Github user deshanxiao closed the pull request at:
https://github.com/apache/spark/pull/22109
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/22109
this looks reasonable, but now I'm wondering whether this will only affect the driver. Couldn't it also effect the executors? Executors might get created as soon as there is a [`schedulerBackend`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L493) and they can register as soon as there is a blockManagerMaster, which is done as part of the [SparkEnv creation](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L424) -- both happen before the EventLoggingListener is created. so we'd need to do something different for that case, probably have the listenerBus buffer up events until we tell it all "primimary" listeners are registered.
I suppose that's an extremely unlikely race, so I might be overthinking this. I do recall seeing cases where the executors didn't have info either, but I doubt this was the cause.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22109
**[Test build #4263 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4263/testReport)** for PR 22109 at commit [`26ca9c2`](https://github.com/apache/spark/commit/26ca9c2c08c62961183e6461183c2963b6a00474).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22109
**[Test build #94807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94807/testReport)** for PR 22109 at commit [`26ca9c2`](https://github.com/apache/spark/commit/26ca9c2c08c62961183e6461183c2963b6a00474).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/22109
ah, right, I thought the listenerbus was doing that, but couldn't find it, I was looking in the wrong place.
so @deshanxiao , given the discussion above, any chance you can share more info about what you're observing so we can determine the right fix? ideally a snippet of the event logs and driver logs where you see this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22109
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22109: [SPARK-25120][CORE][HistoryServer]Fix the problem of Eve...
Posted by deshanxiao <gi...@git.apache.org>.
Github user deshanxiao commented on the issue:
https://github.com/apache/spark/pull/22109
@vanzin Sorry..SPARK-22850 has fix the problem. Maybe I will track the executor lose problem next. Thank you! @vanzin @squito
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org