You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by vanzin <gi...@git.apache.org> on 2017/11/14 22:00:50 UTC
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/19750
[SPARK-20650][core] Remove JobProgressListener.
The only remaining use of this class was the SparkStatusTracker, which
was modified to use the new status store. The test code to wait for
executors was moved to TestUtils and now uses the SparkStatusTracker API.
Indirectly, ConsoleProgressBar also uses this data. Because it has
some lower latency requirements, a shortcut to efficiently get the
active stages from the active listener was added to the AppStateStore.
Now that all UI code goes through the status store to get its data,
the FsHistoryProvider can be cleaned up to only replay event logs
when needed - that is, when there is no pre-existing disk store for
the application.
As part of this change I also modified the streaming UI to read the needed
data from the store, which was missed in the previous patch that made
JobProgressListener redundant.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-20650
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19750.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19750
----
commit 406779bf05cbab7afd8c632ebb7035fb0f2cbd28
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2017-03-29T18:18:23Z
[SPARK-20650][core] Remove JobProgressListener.
The only remaining use of this class was the SparkStatusTracker, which
was modified to use the new status store. The test code to wait for
executors was moved to TestUtils and now uses the SparkStatusTracker API.
Indirectly, ConsoleProgressBar also uses this data. Because it has
some lower latency requirements, a shortcut to efficiently get the
active stages from the active listener was added to the AppStateStore.
Now that all UI code goes through the status store to get its data,
the FsHistoryProvider can be cleaned up to only replay event logs
when needed - that is, when there is no pre-existing disk store for
the application.
As part of this change I also modified the streaming UI to read the needed
data from the store, which was missed in the previous patch that made
JobProgressListener redundant.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #83919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83919/testReport)** for PR 19750 at commit [`81b3f3d`](https://github.com/apache/spark/commit/81b3f3dd223785e02ad2df2a6e40046a76539681).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151282422
--- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -315,48 +315,43 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
(new InMemoryStore(), true)
}
- val listener = if (needReplay) {
- val _listener = new AppStatusListener(kvstore, conf, false,
+ if (needReplay) {
+ val replayBus = new ReplayListenerBus()
--- End diff --
Doesn't this val already exist?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #83919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83919/testReport)** for PR 19750 at commit [`81b3f3d`](https://github.com/apache/spark/commit/81b3f3dd223785e02ad2df2a6e40046a76539681).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #84281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84281/testReport)** for PR 19750 at commit [`81b3f3d`](https://github.com/apache/spark/commit/81b3f3dd223785e02ad2df2a6e40046a76539681).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/19750
lgtm
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151281951
--- Diff: core/src/main/scala/org/apache/spark/TestUtils.scala ---
@@ -232,6 +232,30 @@ private[spark] object TestUtils {
}
}
+ /**
+ * Wait until at least `numExecutors` executors are up, or throw `TimeoutException` if the waiting
+ * time elapsed before `numExecutors` executors up. Exposed for testing.
+ *
+ * @param numExecutors the number of executors to wait at least
+ * @param timeout time to wait in milliseconds
+ */
+ private[spark] def waitUntilExecutorsUp(
+ sc: SparkContext,
+ numExecutors: Int,
+ timeout: Long): Unit = {
+ val finishTime = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout)
+ while (System.nanoTime() < finishTime) {
+ if (sc.statusTracker.getExecutorInfos.length > numExecutors) {
--- End diff --
Based on your comment `at least` above, shouldn't this be `>=`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151287944
--- Diff: core/src/main/scala/org/apache/spark/TestUtils.scala ---
@@ -232,6 +232,30 @@ private[spark] object TestUtils {
}
}
+ /**
+ * Wait until at least `numExecutors` executors are up, or throw `TimeoutException` if the waiting
+ * time elapsed before `numExecutors` executors up. Exposed for testing.
+ *
+ * @param numExecutors the number of executors to wait at least
+ * @param timeout time to wait in milliseconds
+ */
+ private[spark] def waitUntilExecutorsUp(
+ sc: SparkContext,
+ numExecutors: Int,
+ timeout: Long): Unit = {
+ val finishTime = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout)
+ while (System.nanoTime() < finishTime) {
+ if (sc.statusTracker.getExecutorInfos.length > numExecutors) {
--- End diff --
`getExecutorInfos` includes the driver...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin closed the pull request at:
https://github.com/apache/spark/pull/19750
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151281491
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -89,39 +83,33 @@ class SparkStatusTracker private[spark] (sc: SparkContext) {
* garbage collected.
*/
def getStageInfo(stageId: Int): Option[SparkStageInfo] = {
- jobProgressListener.synchronized {
- for (
- info <- jobProgressListener.stageIdToInfo.get(stageId);
- data <- jobProgressListener.stageIdToData.get((stageId, info.attemptId))
- ) yield {
- new SparkStageInfoImpl(
- stageId,
- info.attemptId,
- info.submissionTime.getOrElse(0),
- info.name,
- info.numTasks,
- data.numActiveTasks,
- data.numCompleteTasks,
- data.numFailedTasks)
- }
+ store.asOption(store.lastStageAttempt(stageId)).map { stage =>
+ new SparkStageInfoImpl(
+ stageId,
+ stage.attemptId,
+ stage.submissionTime.map(_.getTime()).getOrElse(0L),
+ stage.name,
+ stage.numTasks,
+ stage.numActiveTasks,
+ stage.numCompleteTasks,
+ stage.numFailedTasks)
}
}
/**
* Returns information of all known executors, including host, port, cacheSize, numRunningTasks.
*/
def getExecutorInfos: Array[SparkExecutorInfo] = {
- val executorIdToRunningTasks: Map[String, Int] =
- sc.taskScheduler.asInstanceOf[TaskSchedulerImpl].runningTasksByExecutors
-
- sc.getExecutorStorageStatus.map { status =>
- val bmId = status.blockManagerId
+ store.executorList(true).map { exec =>
+ val (host, port) = exec.hostPort.split(":", 2) match {
+ case Array(h, p) => (h, p.toInt)
+ case Array(h) => (h, -1)
+ }
new SparkExecutorInfoImpl(
- bmId.host,
- bmId.port,
- status.cacheSize,
- executorIdToRunningTasks.getOrElse(bmId.executorId, 0)
- )
- }
+ host,
+ port,
+ exec.maxMemory,
--- End diff --
Why this change from cacheSize to maxMemory, are they synonymous?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83915/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151283135
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -563,7 +575,7 @@ private[spark] class AppStatusListener(
/** Flush all live entities' data to the underlying store. */
def flush(): Unit = {
val now = System.nanoTime()
- liveStages.values.foreach { stage =>
+ liveStages.values.asScala.foreach { stage =>
--- End diff --
Why add `asScala`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151288247
--- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -315,48 +315,43 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
(new InMemoryStore(), true)
}
- val listener = if (needReplay) {
- val _listener = new AppStatusListener(kvstore, conf, false,
+ if (needReplay) {
+ val replayBus = new ReplayListenerBus()
--- End diff --
It's still there in this PR though (line 297)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #83904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83904/testReport)** for PR 19750 at commit [`4f91e9d`](https://github.com/apache/spark/commit/4f91e9d7a5fce7daea04de634293570a4f531722).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19750
For context:
* Project link: https://issues.apache.org/jira/browse/SPARK-18085
* Upcoming PRs that build on this code: https://github.com/vanzin/spark/pulls
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83865/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19750
Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151287705
--- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -315,48 +315,43 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
(new InMemoryStore(), true)
}
- val listener = if (needReplay) {
- val _listener = new AppStatusListener(kvstore, conf, false,
+ if (needReplay) {
+ val replayBus = new ReplayListenerBus()
--- End diff --
Removed stale one.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151288363
--- Diff: core/src/main/scala/org/apache/spark/SparkStatusTracker.scala ---
@@ -89,39 +83,33 @@ class SparkStatusTracker private[spark] (sc: SparkContext) {
* garbage collected.
*/
def getStageInfo(stageId: Int): Option[SparkStageInfo] = {
- jobProgressListener.synchronized {
- for (
- info <- jobProgressListener.stageIdToInfo.get(stageId);
- data <- jobProgressListener.stageIdToData.get((stageId, info.attemptId))
- ) yield {
- new SparkStageInfoImpl(
- stageId,
- info.attemptId,
- info.submissionTime.getOrElse(0),
- info.name,
- info.numTasks,
- data.numActiveTasks,
- data.numCompleteTasks,
- data.numFailedTasks)
- }
+ store.asOption(store.lastStageAttempt(stageId)).map { stage =>
+ new SparkStageInfoImpl(
+ stageId,
+ stage.attemptId,
+ stage.submissionTime.map(_.getTime()).getOrElse(0L),
+ stage.name,
+ stage.numTasks,
+ stage.numActiveTasks,
+ stage.numCompleteTasks,
+ stage.numFailedTasks)
}
}
/**
* Returns information of all known executors, including host, port, cacheSize, numRunningTasks.
*/
def getExecutorInfos: Array[SparkExecutorInfo] = {
- val executorIdToRunningTasks: Map[String, Int] =
- sc.taskScheduler.asInstanceOf[TaskSchedulerImpl].runningTasksByExecutors
-
- sc.getExecutorStorageStatus.map { status =>
- val bmId = status.blockManagerId
+ store.executorList(true).map { exec =>
+ val (host, port) = exec.hostPort.split(":", 2) match {
+ case Array(h, p) => (h, p.toInt)
+ case Array(h) => (h, -1)
+ }
new SparkExecutorInfoImpl(
- bmId.host,
- bmId.port,
- status.cacheSize,
- executorIdToRunningTasks.getOrElse(bmId.executorId, 0)
- )
- }
+ host,
+ port,
+ exec.maxMemory,
--- End diff --
```
/** Return the memory used by caching RDDs */
def cacheSize: Long = onHeapCacheSize.getOrElse(0L) + offHeapCacheSize.getOrElse(0L)
```
So you're right, this isn't correct.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #84281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84281/testReport)** for PR 19750 at commit [`81b3f3d`](https://github.com/apache/spark/commit/81b3f3dd223785e02ad2df2a6e40046a76539681).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19750
Merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by ajbozarth <gi...@git.apache.org>.
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151288302
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -563,7 +575,7 @@ private[spark] class AppStatusListener(
/** Flush all live entities' data to the underlying store. */
def flush(): Unit = {
val now = System.nanoTime()
- liveStages.values.foreach { stage =>
+ liveStages.values.asScala.foreach { stage =>
--- End diff --
Ah I missed that that class change was scala -> java
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #83865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83865/testReport)** for PR 19750 at commit [`406779b`](https://github.com/apache/spark/commit/406779bf05cbab7afd8c632ebb7035fb0f2cbd28).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #83904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83904/testReport)** for PR 19750 at commit [`4f91e9d`](https://github.com/apache/spark/commit/4f91e9d7a5fce7daea04de634293570a4f531722).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83904/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #83915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83915/testReport)** for PR 19750 at commit [`a004ce9`](https://github.com/apache/spark/commit/a004ce915a21e0dc8093bfdde4fe4290e9fa5fc2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #83865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83865/testReport)** for PR 19750 at commit [`406779b`](https://github.com/apache/spark/commit/406779bf05cbab7afd8c632ebb7035fb0f2cbd28).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83919/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19750
**[Test build #83915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83915/testReport)** for PR 19750 at commit [`a004ce9`](https://github.com/apache/spark/commit/a004ce915a21e0dc8093bfdde4fe4290e9fa5fc2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19750#discussion_r151287617
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -563,7 +575,7 @@ private[spark] class AppStatusListener(
/** Flush all live entities' data to the underlying store. */
def flush(): Unit = {
val now = System.nanoTime()
- liveStages.values.foreach { stage =>
+ liveStages.values.asScala.foreach { stage =>
--- End diff --
`liveStages` is not a Scala map anymore.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19750: [SPARK-20650][core] Remove JobProgressListener.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19750
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84281/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org