You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by andrewor14 <gi...@git.apache.org> on 2014/02/28 02:36:17 UTC
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/42
[WIP] [SPARK-1132] Persisting Web UI through refactoring the SparkListener interface
The fleeting nature of the Spark Web UI has long been a problem reported by many users: The existing Web UI disappears as soon as the associated application terminates. This is because SparkUI is tightly coupled with SparkContext, and cannot be instantiated independently from it. To solve this, some state must be saved to persistent storage while the application is still running.
The approach taken by this PR involves persisting the UI state through SparkListenerEvents. This requires a major refactor of the SparkListener interface because existing events (1) maintain deep references, making de/serialization is difficult, and (2) do not encode all the information displayed on the UI.
The new architecture is as follows - The SparkUI registers a central gateway listener with the SparkContext for events. For each event this gateway listener receives, it logs the event to a file and relays it to all of its children listeners (e.g. ExecutorsListener) used by the UI. Each of these children listeners then construct the appropriate information from these events and supplies it to its parent UI (e.g. ExecutorsUI), which renders the associated page(s) on demand. Then, after the SparkContext has stopped, the SparkUI can be revived by replaying all the logged events to the children listeners through the gateway listener.
This patch is WIP and still expects additional features. The main TODO's at this point include adding support for logging into HDFS, handling long running jobs (perhaps through checkpointing), and performance testing. To try this patch out, run your Spark application to completion as usual, then run ```bin/spark-class org.apache.spark.ui.UIReloader /tmp/spark-<user name>```. Your revived Spark Web UI awaits on port 14040.
More details can be found in the commit messages, comments within the code, and the [design doc](https://spark-project.atlassian.net/secure/attachment/12900/PersistingSparkWebUI.pdf). Comments and feedback are most welcome.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/42.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #42
----
commit 164489d6f176bdecfa9dabec2dfce5504d1ee8af
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-04T02:18:04Z
Relax assumptions on compressors and serializers when batching
This commit introduces an intermediate layer of an input stream on the batch level.
This guards against interference from higher level streams (i.e. compression and
deserialization streams), especially pre-fetching, without specifically targeting
particular libraries (Kryo) and forcing shuffle spill compression to use LZF.
commit a531d2e347acdcecf2d0ab72cd4f965ab5e145d8
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-04T02:18:04Z
Relax assumptions on compressors and serializers when batching
This commit introduces an intermediate layer of an input stream on the batch level.
This guards against interference from higher level streams (i.e. compression and
deserialization streams), especially pre-fetching, without specifically targeting
particular libraries (Kryo) and forcing shuffle spill compression to use LZF.
commit 3df700509955f7074821e9aab1e74cb53c58b5a5
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-04T02:27:49Z
Merge branch 'master' of github.com:andrewor14/incubator-spark
commit 287ef44e593ad72f7434b759be3170d9ee2723d2
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-04T21:38:32Z
Avoid reading the entire batch into memory; also simplify streaming logic
Additionally, address formatting comments.
commit bd5a1d7350467ed3dc19c2de9b2c9f531f0e6aa3
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-04T21:44:24Z
Typo: phyiscal -> physical
commit 13920c918efe22e66a1760b14beceb17a61fd8cc
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-05T00:34:15Z
Update docs
commit 090544a87a0767effd0c835a53952f72fc8d24f0
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-05T18:58:23Z
Privatize methods
commit 3ddeb7ef89a0af2b685fb5d071aa0f71c975cc82
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-05T20:09:32Z
Also privatize fields
commit e3ae35f4fb1ce8e2d7398afdbabab3dbf4bb2ffe
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-11T00:15:15Z
Merge github.com:apache/incubator-spark
Conflicts:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
commit 8e09306f6dd4ab421447d769572de58035d3d66a
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-12T01:48:16Z
Use JSON for ExecutorsUI
commit 10ed49dffe4a515bff42762cb025a3f64d9cd407
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-12T18:53:32Z
Merge github.com:apache/incubator-spark into persist-ui
commit dcbd312b1e4585445868dfb562f9c64ac2fc8cda
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-12T23:58:39Z
Add JSON Serializability for all SparkListenerEvent's
This also involves a clean-up in the way these events are structured. The existing way
in which these events are defined maintains a lot of extraneous information. To avoid
serializing the whole tree of RDD dependencies, for instance, this commit cherry-picks
only the relevant fields. However, this means sacrificing JobLogger's functionality of
tracing the entire RDD tree.
Additionally, this commit also involves minor formatting and naming clean-ups within
the scope of the above changes.
commit bb222b9f7422cdf9e3a4c682bb271da1f75f4f75
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-13T04:35:09Z
ExecutorUI: render completely from JSON
Additionally, this commit fixes the bug in the local mode, where executor IDs of tasks
do not match those of storage statuses (more detail in ExecutorsUI.scala).
This commit currently does not serialize the SparkListenerEvents yet, but instead
serializes changes to each executor JSON. This is a big TODO in the upcoming commit.
commit bf0b2e9e92d760d49ba7b26aaa41b9e3aef2420f
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-14T03:12:53Z
ExecutorUI: Serialize events rather than arbitary executor information
This involves adding a new SparkListenerStorageFetchEvent, and adding JSON serializability
to all of the objects it depends on.
commit de8a1cdb833d80423aba629ba932b6f403ecd4ab
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-15T03:22:50Z
Serialize events both to and from JSON (rather than just to)
This requires every field of every event to be completely reconstructible from its
JSON representation. This commit may contain incomplete state.
commit 8a2ebe6ba37b2d5efe344aa3bea343cda1411212
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-15T06:01:21Z
Fix bugs for EnvironmentUI and ExecutorsUI
In particular, EnvironmentUI was not rendering until a job begins, and ExecutorsUI
reports an incorrect number (format) of total tasks.
commit c4cd48022b3a8dbf60f458196e21ba8c9cb3b88f
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-15T06:53:43Z
Also deserialize new events
This includes SparkListenerLoadEnvironment and SparkListenerStorageStatusFetch
commit d859efc34c9a5f07bae7eca7b4ab72fa19fb7e29
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-15T22:01:14Z
BlockManagerUI: Add JSON functionality
commit 8add36bb08126fbcd02d23c446dd3ec970f1f549
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-15T22:40:49Z
JobProgressUI: Add JSON functionality
In addition, refactor FileLogger to log in one directory per logger
commit b3976b0a2eb21b4a887d01fd16869a0f37c36f8b
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-16T06:52:43Z
Add functionality of reconstructing a persisted UI from SparkContext
With this commit, any reconstruct SparkUI resides on default port of 14040 onwards.
Logged events are posted separately from live events, such that the live SparkListeners
are not affected.
This commit also fixes a few JSON de/serialization bugs.
commit 4dfcd224504f392302a49ac82280b294c381f381
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-17T19:14:20Z
Merge git://git.apache.org/incubator-spark into persist-ui
commit f3fc13b53725cdfeddcecb2068ab5a533566772f
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-17T21:22:01Z
General refactor
This includes reverting previous formatting and naming changes that are irrelevant to
this patch.
commit 3fd584e30aaf6552179bf9e9b350b130fa92d0ad
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-18T02:01:12Z
Fix two major bugs
First, JobProgessListener uses HashSets of TaskInfo and StageInfo, and relies on the equality
of these objects to remove from the corresponding HashSets correctly. This is not a luxury that
deserialized StageInfo's and TaskInfo's have. Instead, when removing from these collections, we
must match by the ID rather than the object itself.
Second, although SparkUI differentiates between persisted and live UI's, its children UI's and
their corresponding listeners do not. Thus, each revived UI essentially duplicated all the logs
that reconstructed it in the first place. Further, these zombie UI's continued to respond to
live SparkListenerEvents. This has been fixed by requiring that revived UI's do not register
their listeners with the current SparkContext.
With the former fix, there were major incompatibility issues with the existing way UI classes
access and mutate the collections. Formatting improvements associated with smoothing out these
inconsistencies are included as part of this commit.
commit 5ac906d4dfd546c5d6b6e80540c8774f3985fecc
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-18T05:38:16Z
Mostly naming, formatting, and code style changes
commit 904c7294ac221a0cd9806af843219aaa8a847085
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-18T06:06:46Z
Fix another major bug
Previously, rendering the old, persisted UI continues to trigger load environment and
storage status fetch events. These are now only triggered for the live UI.
A related TODO: Under JobProgressUI, the total duration is inaccurate; right now it uses
the time when the old UI is revived, rather than when it was live. This should be fixed.
commit 427301371117e9e7889f5df0f6bba51e5916e425
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-18T23:27:39Z
Add a gateway SparkListener to simplify event logging
Instead of having each SparkListener log an independent set of events, centralize event
logging to avoid differentiating events across UI's and thus duplicating logged events.
Also rename the "fromDisk" parameter to "live".
TODO: Storage page currently still relies on the previous SparkContext and is not
rendering correctly.
commit 64d2ce1efee3aa5a8166c5fe108932b2279217fc
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-19T02:29:21Z
Fix BlockManagerUI bug by introducing new event
Previously, the storage information of persisted RDD's continued to rely on the old SparkContext,
which is no longer accessible if the UI is rendered from disk. This fix solves it by introducing
an event, SparkListenerGetRDDInfo, which captures this information.
Per discussion with Patrick, an alternative is to encapsulate this information within
SparkListenerTaskEnd. This would bypass the need to create a new event, but would also require
a non-trivial refactor of BlockManager / BlockStore.
commit 6814da0cf9af2a29810b6773463acee3b259c95f
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-19T18:36:01Z
Explicitly register each UI listener rather than through some magic
This (1) allows UISparkListener to be a simple trait and (2) is more intuitive, since it
mirrors sc.addSparkListener(listener), for all other non-UI listeners.
commit d646df6786737d67d5ca1dbf593740a02a600991
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-20T02:47:35Z
Completely decouple SparkUI from SparkContext
This involves storing additional fields, such as the scheduling mode and the app name, into the
new event, SparkListenerApplicationStart, since these attributes are no longer accessible without
a SparkContext. Further, environment information is refactored to be loaded on application start
(rather than on job start).
Persisted Spark UI's can no longer be created from SparkContext. The new way of constructing them
is through a standalone scala program. org.apache.spark.ui.UIReloader is introduced as an example
of how to do this.
commit e9e1c6dede36788d3cefe3c65366f5a79be97a1d
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-21T07:51:08Z
Move all JSON de/serialization logic to JsonProtocol
This makes all classes involved appear less cluttered.
commit 70e7e7acf09d8efd2c7e459ee450c1db140b8f5a
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-22T02:56:26Z
Formatting changes
commit 6631c02a8791d0321f003bb339344445f4dd0cab
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-24T18:52:21Z
More formatting changes, this time mainly for Json DSL
commit bbe3501c63029ffa9c1fd9053e7ab868d0f28b10
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-26T23:27:43Z
Embed storage status and RDD info in Task events
This commit achieves three main things. First and foremost, it embeds the information
from the SparkListenerFetchStorageStatus and SparkListenerGetRDDInfo events into events
that are more descriptive of the SparkListenerInterface. In particular, every Task now
maintains a list of blocks whose storage status have been updated as a result of the task.
Previously, this information is retrieved from fetching storage status from the driver,
an action arbitrarily associated with a stage. This change involves keeping track of
what blocks are dropped during each call to an RDD persist. A big TODO is to also capture
the behavior of an RDD unpersist in a SparkListenerEvent.
Second, the SparkListenerEvent interface now handles the dynamic nature of Executors.
In particular, a new event, SparkListenerExecutorStateChange, is introduced, which triggers
a storage status fetch from the driver. The purpose of this is mainly to decouple fetching
storage status from the driver from the Stage. Note that storage status is not ready until
the remote BlockManagers have been registered, so this involves attaching a registration
listener to the BlockManagerMasterActor.
Third, changes in environment properties is now supported. This accounts for the fact that
the user can invoke sc.addFile and sc.addJar in his/her own application, which should be
reflected appropriately on the EnvironmentUI. In the previous implementation, coupling this
information with application start prevents this from happening.
Other relatively minor changes include: 1) Refactoring BlockStatus and BlockManagerInfo to
not be a part of the BlockManagerMasterActor object, 2) Formatting changes, especially those
involving multi-line arguments, and 3) Making all UI widgets and listeners private[ui] instead
of private[spark].
commit 28019caa5712b8d7f1db039dc41876d91e530998
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-27T00:47:00Z
Merge github.com:apache/spark
Conflicts:
core/src/main/scala/org/apache/spark/CacheManager.scala
core/src/main/scala/org/apache/spark/SparkEnv.scala
core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala
core/src/main/scala/org/apache/spark/storage/BlockManager.scala
core/src/main/scala/org/apache/spark/storage/MemoryStore.scala
core/src/main/scala/org/apache/spark/storage/StorageUtils.scala
core/src/main/scala/org/apache/spark/ui/SparkUI.scala
core/src/main/scala/org/apache/spark/ui/env/EnvironmentUI.scala
core/src/main/scala/org/apache/spark/ui/exec/ExecutorsUI.scala
core/src/main/scala/org/apache/spark/ui/jobs/IndexPage.scala
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala
core/src/main/scala/org/apache/spark/ui/jobs/PoolPage.scala
core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
core/src/main/scala/org/apache/spark/ui/storage/IndexPage.scala
core/src/main/scala/org/apache/spark/ui/storage/RDDPage.scala
core/src/main/scala/org/apache/spark/util/TimeStampedHashMap.scala
core/src/main/scala/org/apache/spark/util/Utils.scala
core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
commit d1f428591d6c33c2bb86f85468c7842b5ca00311
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-27T01:19:20Z
Migrate from lift-json to json4s-jackson
commit 7b2f8112795a53c35b10bc3d72e5be7b699ceb65
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-27T19:24:32Z
Guard against TaskMetrics NPE + Fix tests
commit 996d7a2f42d4e02c1e40ec22b0c4d7db86aa03e3
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-27T20:26:23Z
Reflect RDD unpersist on UI
This introduces a new event, SparkListenerUnpersistRDD.
commit 472fd8a4845e39a38f8d993a3527a7e77571ffad
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-27T23:03:59Z
Fix a couple of tests
commit d47585f22f243fc7e840af90132edb7e84b003ed
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-28T00:15:21Z
Clean up FileLogger
commit faa113e674a276ddf5cd7dc643c16b7bed2b5e44
Author: Andrew Or <an...@gmail.com>
Date: 2014-02-28T01:12:19Z
General clean up
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10551815
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -895,4 +894,42 @@ private[spark] object Utils extends Logging {
}
count
}
+
+ /** Return the class name of the given object, removing all dollar signs */
+ def getFormattedClassName(obj: AnyRef) = {
+ obj.getClass.getSimpleName.replace("$", "")
+ }
+
+ def jsonOption(json: JValue): Option[JValue] = {
+ json match {
+ case JNothing => None
+ case value: JValue => Some(value)
+ }
+ }
+
+ def emptyJson = JObject(List[JField]())
+
+ /**
+ * Return a Hadoop FileSystem with the scheme encoded in the given path.
+ * File systems currently supported include HDFS, S3, and the local file system.
+ */
+ def getHadoopFileSystem(path: URI): FileSystem = {
--- End diff --
Restricting us to specific filesystem's is not a good idea. There are many implementations of the Hadoop FileSystem API that we expect people to use. Why not just do something like:
```
val fs = FileSystem.get(uri, SparkHadoopUtil.get.newConfiguration())
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10730503
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -614,12 +622,27 @@ private[spark] class Master(host: String, port: Int, webUiPort: Int,
if (completedApps.size >= RETAINED_APPLICATIONS) {
val toRemove = math.max(RETAINED_APPLICATIONS / 10, 1)
completedApps.take(toRemove).foreach( a => {
+ appIdToUI.get(a.id).foreach { ui =>
--- End diff --
Should this be `appIdToUI.remove(a.id...`. Otherwise that HashMap will grow forever right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36708975
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642128
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala ---
@@ -57,21 +74,20 @@ class MasterWebUI(val master: Master, requestedPort: Int) extends Logging {
}
}
- val metricsHandlers = master.masterMetricsSystem.getServletHandlers ++
- master.applicationMetricsSystem.getServletHandlers
-
- val handlers = metricsHandlers ++ Seq[ServletContextHandler](
- createStaticHandler(MasterWebUI.STATIC_RESOURCE_DIR, "/static/*"),
- createServletHandler("/app/json",
- createServlet((request: HttpServletRequest) => applicationPage.renderJson(request),
- master.securityMgr)),
- createServletHandler("/app", createServlet((request: HttpServletRequest) => applicationPage
- .render(request), master.securityMgr)),
- createServletHandler("/json", createServlet((request: HttpServletRequest) => indexPage
- .renderJson(request), master.securityMgr)),
- createServletHandler("*", createServlet((request: HttpServletRequest) => indexPage.render
- (request), master.securityMgr))
- )
+ /** Attach a reconstructed UI to this Master UI. Only valid after bind(). */
+ def attachUI(ui: SparkUI) {
+ rootHandler.foreach { root =>
--- End diff --
Unless there is a normal case where this can be called with `rootHandler=None` I'd make this an assertion at the beginning and use `rootHandler.get`:
```
assert(serverInfo.isDefined, "Tried to attach SparkUI to master that wasn't initialized")
```
or something...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37903798
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13227/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10186874
--- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import java.io._
+import java.text.SimpleDateFormat
+import java.util.Date
+
+import org.apache.spark.Logging
+
+/**
+ * A generic class for logging information to file
+ * @param logDir Path to the directory in which files are logged
+ * @param name An identifier of each FileLogger instance
+ */
+class FileLogger(
+ logDir: String = Option(System.getenv("SPARK_LOG_DIR"))
+ .getOrElse("/tmp/spark-%s".format(System.getProperty("user.name", "user"))),
+ name: String = String.valueOf(System.currentTimeMillis()))
+ extends Logging {
+
+ private val logPath = logDir.stripSuffix("/") + "/" + name
+ private val DATE_FORMAT = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss")
+ private var fileIndex = 0
+
+ private var writer: Option[PrintWriter] = {
+ createLogDir()
+ Some(createWriter())
+ }
+
+ /** Create a logging directory with the given path */
+ private def createLogDir() = {
+ val dir = new File(logPath)
+ if (dir.exists) {
+ logWarning("Logging directory already exists: " + logDir)
+ }
+ if (!dir.exists && !dir.mkdirs()) {
+ // Logger should throw a exception rather than continue to construct this object
+ throw new IOException("Error in creating log directory:" + logDir)
+ }
+ }
+
+ /** Create a new writer to the file identified with the given path */
+ private def createWriter() = {
+ // Overwrite any existing file
+ val fileWriter = new FileWriter(logPath + "/" + fileIndex)
--- End diff --
For this we should use the Hadoop FileSystem API rather than a java file writer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36959518
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13032/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36416706
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36486545
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37244892
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37763540
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10363140
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -53,27 +62,28 @@ private[spark] class JobProgressListener(val sc: SparkContext) extends SparkList
val stageIdToShuffleWrite = HashMap[Int, Long]()
val stageIdToMemoryBytesSpilled = HashMap[Int, Long]()
val stageIdToDiskBytesSpilled = HashMap[Int, Long]()
- val stageIdToTasksActive = HashMap[Int, HashSet[TaskInfo]]()
+ val stageIdToTasksActive = HashMap[Int, HashMap[Long, TaskInfo]]()
val stageIdToTasksComplete = HashMap[Int, Int]()
val stageIdToTasksFailed = HashMap[Int, Int]()
- val stageIdToTaskInfos =
- HashMap[Int, HashSet[(TaskInfo, Option[TaskMetrics], Option[ExceptionFailure])]]()
+ val stageIdToTaskInfos = HashMap[Int, HashMap[Long, TaskUIData]]()
--- End diff --
update variable name
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597551
--- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.io.InputStream
+import java.net.URI
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
+import org.apache.hadoop.fs.{Path, FileSystem}
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+/**
+ * An EventBus that replays logged events from persisted storage
--- End diff --
Does this get unit tested anywhere? It would be good to have tests that log events to a file and read them back and make sure everything gets serialized correctly (actually both logging to a file and just a simpler one that serializes/deserializes). If you wrote test code to generate randomized events that could be useful to contribute here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10458444
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -69,19 +100,55 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
// DAGScheduler() requires that the port of this server is known
--- End diff --
Ok sounds like maybe the right thing to do is to ignore this for now and deal with it in a separate PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10357702
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerRegistrationListener.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import org.apache.spark.scheduler._
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+/** A listener for block manager registration */
--- End diff --
Also maybe a better name for this would be BlockManagerRegistrationBuffer (since it's not really listening for the registration so much as it's buffering the messages for other listeners)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36418482
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504474
--- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala ---
@@ -17,13 +17,17 @@
package org.apache.spark.storage
+import scala.collection.Map
+import scala.collection.mutable
+
import org.apache.spark.SparkContext
-import org.apache.spark.storage.BlockManagerMasterActor.BlockStatus
import org.apache.spark.util.Utils
private[spark]
-case class StorageStatus(blockManagerId: BlockManagerId, maxMem: Long,
- blocks: Map[BlockId, BlockStatus]) {
+class StorageStatus(
+ val blockManagerId: BlockManagerId,
+ val maxMem: Long,
+ val blocks: mutable.Map[BlockId, BlockStatus] = mutable.Map.empty) {
--- End diff --
No longer a case class, because blocks is now mutable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36398321
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37717262
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36588479
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10367126
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -978,6 +971,11 @@ class DAGScheduler(
logDebug("Additional executor lost message for " + execId +
"(epoch " + currentEpoch + ")")
}
+ // Block manager master actor should not be null except during tests
+ if (blockManagerMaster.driverActor != null) {
+ val storageStatusList = blockManagerMaster.getStorageStatus
+ listenerBus.post(SparkListenerExecutorsStateChange(storageStatusList))
--- End diff --
This also addresses the comment that @kayousterhout brought up above. JobProgressListener no longer needs to keep track of a storage status list (which it never really uses), but instead only listens for ExecutorStateChange events.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10284630
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+/**
+ * Reload a persisted UI independently from a SparkContext
+ */
+object UIReloader {
--- End diff --
Right, the purpose of UIReloader is just to present a quick example to demonstrate how this can be done. I should make that more explicit in the comments.
@pwendell and I discussed this more offline. The proposal is to extract the server logic from SparkUI, and treat it as a handler for an existing server. This could be the server that holds the Master UI, for example. Further, we need to figure out a way to pass the log directory of the SparkUI to the external server, which should not be too complicated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10356042
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -164,9 +167,18 @@ object SparkEnv extends Logging {
}
}
- val blockManagerMaster = new BlockManagerMaster(registerOrLookup(
- "BlockManagerMaster",
- new BlockManagerMasterActor(isLocal, conf)), conf)
+ // Listen for block manager registration
+ val blockManagerListener = new BlockManagerRegistrationListener
+ lazy val blockManagerMasterActor = {
+ val actor = new BlockManagerMasterActor(isLocal, conf)
+ actor.registerListener(blockManagerListener)
+ actor
+ }
+
+ val blockManagerMaster =
+ new BlockManagerMaster(registerOrLookup("BlockManagerMaster", blockManagerMasterActor), conf)
+ blockManagerMaster.registrationListener = Some(blockManagerListener)
--- End diff --
Is there a reason you didn't make the listener a constructor parameter for BlockManagerMaster?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10361035
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -50,6 +51,9 @@ private[spark] class Master(host: String, port: Int, webUiPort: Int) extends Act
val REAPER_ITERATIONS = conf.getInt("spark.dead.worker.persistence", 15)
val RECOVERY_DIR = conf.get("spark.deploy.recoveryDirectory", "")
val RECOVERY_MODE = conf.get("spark.deploy.recoveryMode", "NONE")
+ val MAX_NUM_PERSISTED_UI = conf.getInt("spark.persisted.ui.maxConcurrent", 25)
+ val PERSISTED_SPARK_UI_PORT =
--- End diff --
Actually is there a reason for defining this separate variable? It looks like it's only used for nextPersistedUIPort
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10183713
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -194,12 +194,19 @@ class DAGScheduler(
}
}
}))
+
+ // Start listening for block manager registration
+ blockManagerMaster.registrationListener.foreach(_.setListenerBus(listenerBus))
}
def addSparkListener(listener: SparkListener) {
listenerBus.addListener(listener)
}
+ def post(event: SparkListenerEvent) {
--- End diff --
it would be better not to have other parts of Spark posting events to the DAGScheduler. Maybe we could do the following:
1. Move the listener bus to the SparkContext (approved by @kayousterhout)
2. Pass a SparkContext to the DAGScheduler and access it through that. Then we could remove the Spark env and other arguments currently passed in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10461463
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class, since the UI thread and the DAGScheduler event loop may otherwise
* be reading/updating the internal data structures concurrently.
--- End diff --
Ok. To be consistent I will synchronize the other listeners as well
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504421
--- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala ---
@@ -40,35 +37,16 @@ private[spark] class SparkListenerBus extends Logging {
override def run() {
while (true) {
val event = eventQueue.take
- event match {
- case stageSubmitted: SparkListenerStageSubmitted =>
- sparkListeners.foreach(_.onStageSubmitted(stageSubmitted))
- case stageCompleted: SparkListenerStageCompleted =>
- sparkListeners.foreach(_.onStageCompleted(stageCompleted))
- case jobStart: SparkListenerJobStart =>
- sparkListeners.foreach(_.onJobStart(jobStart))
- case jobEnd: SparkListenerJobEnd =>
- sparkListeners.foreach(_.onJobEnd(jobEnd))
- case taskStart: SparkListenerTaskStart =>
- sparkListeners.foreach(_.onTaskStart(taskStart))
- case taskGettingResult: SparkListenerTaskGettingResult =>
- sparkListeners.foreach(_.onTaskGettingResult(taskGettingResult))
- case taskEnd: SparkListenerTaskEnd =>
- sparkListeners.foreach(_.onTaskEnd(taskEnd))
- case SparkListenerShutdown =>
- // Get out of the while loop and shutdown the daemon thread
- return
- case _ =>
--- End diff --
Moved to EventBus.scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36815479
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13008/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642111
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala ---
@@ -32,23 +33,39 @@ import org.apache.spark.util.{AkkaUtils, Utils}
*/
private[spark]
class MasterWebUI(val master: Master, requestedPort: Int) extends Logging {
- val timeout = AkkaUtils.askTimeout(master.conf)
- val host = Utils.localHostName()
- val port = requestedPort
-
val masterActorRef = master.self
-
+ val timeout = AkkaUtils.askTimeout(master.conf)
var server: Option[Server] = None
--- End diff --
It might be nice to group these options together to make it more clear why they are options and under which case they are defined.
```
// Information set if bind() is called on this UI
case class ServerInfo(server: Server, port: Int, rootHandler: ContextHandlerCollection)
var serverInfo: Option[ServerInfo] = None
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37120319
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13081/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37120318
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10238309
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io.{FileInputStream, File}
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
import org.eclipse.jetty.server.{Handler, Server}
+import org.json4s.jackson.JsonMethods._
import org.apache.spark.{Logging, SparkContext, SparkEnv}
+import org.apache.spark.scheduler._
import org.apache.spark.ui.JettyUtils._
import org.apache.spark.ui.env.EnvironmentUI
import org.apache.spark.ui.exec.ExecutorsUI
import org.apache.spark.ui.jobs.JobProgressUI
import org.apache.spark.ui.storage.BlockManagerUI
import org.apache.spark.util.Utils
+import org.apache.spark.util.JsonProtocol
-/** Top level user interface for Spark */
-private[spark] class SparkUI(sc: SparkContext) extends Logging {
+/** Top level user interface for Spark. */
+private[spark] class SparkUI(val sc: SparkContext) extends Logging {
--- End diff --
It might make more sense for this to be an `Option[SparkContext]` rather than potentially null
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36313763
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37595662
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13170/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36815477
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36314537
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10584092
--- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.io.InputStream
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
+import org.apache.hadoop.fs.{Path, FileSystem}
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+/**
+ * An EventBus that replays logged events from persisted storage
+ */
+private[spark] class SparkReplayerBus(conf: SparkConf) extends EventBus with Logging {
--- End diff --
I think "something you can attach listeners to" pretty much means pub-sub (the difference is how often you publish; in replay mode we publish once in the beginning). I propose the following:
trait SparkListenerBus
ReplayListenerBus extends SparkListenerBus
LiveListenerBus / PollingListenerBus / AsynchronousListenerBus extends SparkListenerBus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36983520
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10458380
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -69,19 +100,55 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
// DAGScheduler() requires that the port of this server is known
--- End diff --
Neither are the two lines that follow. I don't see a necessary ordering between start() and bind(), now that DAGScheduler no longer requires the UI port. The only ordering requirement is that SparkListenerBus must be instantiated before start(), and this is quite clear in SparkContext.
(That said, I prefer to keep bind() and start() separate as they already are, for code clarity reasons)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10360902
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import org.apache.spark.SparkConf
+
+/**
+ * A simple example that reloads a persisted UI independently from a SparkContext
--- End diff --
Maybe remote "A simple example that" here? Implies that this isn't actually supposed to be used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36314544
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10357634
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -50,6 +50,8 @@ class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf) extends Act
private val akkaTimeout = AkkaUtils.askTimeout(conf)
+ private val listeners = new ArrayBuffer[BlockManagerRegistrationListener]
--- End diff --
Can this just be an option (is there any reason you'd ever need more than one?)? It looks like right now there's always only just one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38029328
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37265019
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642210
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -69,32 +70,47 @@ private[spark] class CacheManager(blockManager: BlockManager) extends Logging {
// If we got here, we have to load the split
logInfo("Partition %s not found, computing it".format(key))
val computedValues = rdd.computeOrReadCheckpoint(split, context)
+
// Persist the result, so long as the task is not running locally
if (context.runningLocally) { return computedValues }
- if (storageLevel.useDisk && !storageLevel.useMemory) {
- // In the case that this RDD is to be persisted using DISK_ONLY
- // the iterator will be passed directly to the blockManager (rather then
- // caching it to an ArrayBuffer first), then the resulting block data iterator
- // will be passed back to the user. If the iterator generates a lot of data,
- // this means that it doesn't all have to be held in memory at one time.
- // This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
- // blocks aren't dropped by the block store before enabling that.
- blockManager.put(key, computedValues, storageLevel, tellMaster = true)
- return blockManager.get(key) match {
- case Some(values) =>
- return new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
- case None =>
- logInfo("Failure to store %s".format(key))
- throw new Exception("Block manager failed to return persisted valued")
+
+ // Keep track of blocks with updated statuses
+ var updatedBlocks = Seq[(BlockId, BlockStatus)]()
+ val returnValue: Iterator[T] = {
+ if (storageLevel.useDisk && !storageLevel.useMemory) {
+ /* In the case that this RDD is to be persisted using DISK_ONLY
+ * the iterator will be passed directly to the blockManager (rather then
+ * caching it to an ArrayBuffer first), then the resulting block data iterator
+ * will be passed back to the user. If the iterator generates a lot of data,
+ * this means that it doesn't all have to be held in memory at one time.
+ * This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
+ * blocks aren't dropped by the block store before enabling that. */
+ updatedBlocks = blockManager.put(key, computedValues, storageLevel, tellMaster = true)
+ blockManager.get(key) match {
+ case Some(values) =>
+ new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
+ case None =>
+ logInfo("Failure to store %s".format(key))
+ throw new Exception("Block manager failed to return persisted valued")
+ }
+ } else {
+ // In this case the RDD is cached to an array buffer. This will save the results
+ // if we're dealing with a 'one-time' iterator
+ val elements = new ArrayBuffer[Any]
+ elements ++= computedValues
+ updatedBlocks = blockManager.put(key, elements, storageLevel, tellMaster = true)
+ elements.iterator.asInstanceOf[Iterator[T]]
}
- } else {
- // In this case the RDD is cached to an array buffer. This will save the results
- // if we're dealing with a 'one-time' iterator
- val elements = new ArrayBuffer[Any]
- elements ++= computedValues
- blockManager.put(key, elements, storageLevel, tellMaster = true)
- return elements.iterator.asInstanceOf[Iterator[T]]
}
+
+ // Update task metrics to include any blocks whose storage status is updated
+ val metrics = context.taskMetrics
--- End diff --
Yeah, I didn't have the null check at first but a few of the tests threw NPE
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10580179
--- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.io.InputStream
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
+import org.apache.hadoop.fs.{Path, FileSystem}
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+/**
+ * An EventBus that replays logged events from persisted storage
+ */
+private[spark] class SparkReplayerBus(conf: SparkConf) extends EventBus with Logging {
+ private val compressed = conf.getBoolean("spark.eventLog.compress", false)
+
+ // Only used if compression is enabled
+ private lazy val compressionCodec = CompressionCodec.createCodec(conf)
+
+ /**
+ * Return a list of paths representing log files in the given directory.
+ */
+ private def getLogFilePaths(logDir: String, fileSystem: FileSystem): Array[Path] = {
+ val path = new Path(logDir)
+ if (!fileSystem.exists(path) || !fileSystem.getFileStatus(path).isDir) {
+ logWarning("Log path provided is not a valid directory: %s".format(logDir))
+ return Array[Path]()
+ }
+ val logStatus = fileSystem.listStatus(path)
+ if (logStatus == null || !logStatus.exists(!_.isDir)) {
+ logWarning("Log path provided contains no log files: %s".format(logDir))
+ return Array[Path]()
+ }
+ logStatus.filter(!_.isDir).map(_.getPath).sortBy(_.getName)
+ }
+
+ /**
+ * Replay each event in the order maintained in the given logs.
+ */
+ def replay(logDir: String): Boolean = {
+ val fileSystem = Utils.getHadoopFileSystem(logDir)
+ val logPaths = getLogFilePaths(logDir, fileSystem)
+ if (logPaths.length == 0) {
+ return false
+ }
+
+ logPaths.foreach { path =>
+ // In case there is an exception, keep track of the highest level stream to close it later
+ var streamToClose: Option[InputStream] = None
+ var currentLine = ""
+ try {
+ val fstream = fileSystem.open(path)
+ val bstream = new FastBufferedInputStream(fstream)
+ val cstream = if (compressed) compressionCodec.compressedInputStream(bstream) else bstream
+ streamToClose = Some(cstream)
+
+ // Parse each line as an event and post it to all attached listeners
+ val lines = Source.fromInputStream(cstream).getLines()
+ lines.foreach { line =>
+ currentLine = line
+ val event = JsonProtocol.sparkEventFromJson(parse(line))
+ postToAll(event)
+ }
+ } catch {
+ case e: Exception =>
+ logWarning("Exception in parsing UI logs for %s".format(path))
--- End diff --
Oops. That was left over from before I moved it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10284454
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io.{FileInputStream, File}
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
import org.eclipse.jetty.server.{Handler, Server}
+import org.json4s.jackson.JsonMethods._
import org.apache.spark.{Logging, SparkContext, SparkEnv}
+import org.apache.spark.scheduler._
import org.apache.spark.ui.JettyUtils._
import org.apache.spark.ui.env.EnvironmentUI
import org.apache.spark.ui.exec.ExecutorsUI
import org.apache.spark.ui.jobs.JobProgressUI
import org.apache.spark.ui.storage.BlockManagerUI
import org.apache.spark.util.Utils
+import org.apache.spark.util.JsonProtocol
-/** Top level user interface for Spark */
-private[spark] class SparkUI(sc: SparkContext) extends Logging {
+/** Top level user interface for Spark. */
+private[spark] class SparkUI(val sc: SparkContext) extends Logging {
+
+ // If SparkContext is not provided, assume this UI is rendered from persisted storage
+ val live = sc != null
val host = Option(System.getenv("SPARK_PUBLIC_DNS")).getOrElse(Utils.localHostName())
- val port = sc.conf.get("spark.ui.port", SparkUI.DEFAULT_PORT).toInt
+ var port = if (live) {
+ sc.conf.get("spark.ui.port", SparkUI.DEFAULT_PORT).toInt
+ } else {
+ SparkUI.DEFAULT_PERSISTED_PORT.toInt
+ }
var boundPort: Option[Int] = None
var server: Option[Server] = None
+ var started = false
+ var appName = ""
- val handlers = Seq[(String, Handler)](
+ private val handlers = Seq[(String, Handler)](
("/static", createStaticHandler(SparkUI.STATIC_RESOURCE_DIR)),
("/", createRedirectHandler("/stages"))
)
- val storage = new BlockManagerUI(sc)
- val jobs = new JobProgressUI(sc)
- val env = new EnvironmentUI(sc)
- val exec = new ExecutorsUI(sc)
+ private val storage = new BlockManagerUI(this)
+ private val jobs = new JobProgressUI(this)
+ private val env = new EnvironmentUI(this)
+ private val exec = new ExecutorsUI(this)
// Add MetricsServlet handlers by default
- val metricsServletHandlers = SparkEnv.get.metricsSystem.getServletHandlers
+ private val metricsServletHandlers = if (live) {
+ SparkEnv.get.metricsSystem.getServletHandlers
+ } else {
+ Array[(String, Handler)]()
+ }
- val allHandlers = storage.getHandlers ++ jobs.getHandlers ++ env.getHandlers ++
+ private val allHandlers = storage.getHandlers ++ jobs.getHandlers ++ env.getHandlers ++
exec.getHandlers ++ metricsServletHandlers ++ handlers
+ // Maintain a gateway listener for all events to simplify event logging
+ private var _gatewayListener: Option[GatewayUISparkListener] = None
+
+ def gatewayListener = _gatewayListener.getOrElse {
+ val gateway = new GatewayUISparkListener(this, sc)
+ _gatewayListener = Some(gateway)
+ gateway
+ }
+
+ // Only meaningful if port is set before binding
+ def setPort(p: Int) = {
--- End diff --
We can even do ```def this(conf: SparkConf) = this(null, conf)``` and ```def this(sc: SparkContext) = this(sc, sc.conf)```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10369800
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -978,6 +971,11 @@ class DAGScheduler(
logDebug("Additional executor lost message for " + execId +
"(epoch " + currentEpoch + ")")
}
+ // Block manager master actor should not be null except during tests
+ if (blockManagerMaster.driverActor != null) {
+ val storageStatusList = blockManagerMaster.getStorageStatus
+ listenerBus.post(SparkListenerExecutorsStateChange(storageStatusList))
--- End diff --
This sounds good to me!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37372997
@pwendell and @kayousterhout - Thanks for taking the first pass through this patch. I have made the relevant changes we discussed and updated to master (non-trivial as there were many conflicts). It is now ready for another round of review. I am also in the process of performance testing, but soon after that I think this PR will be more or less in a merge-able state.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597778
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -196,13 +152,46 @@ class DAGScheduler(
}))
}
- def addSparkListener(listener: SparkListener) {
- listenerBus.addListener(listener)
+ // Called by TaskScheduler to report task's starting.
+ def taskStarted(task: Task[_], taskInfo: TaskInfo) {
+ eventProcessActor ! BeginEvent(task, taskInfo)
+ }
+
+ // Called to report that a task has completed and results are being fetched remotely.
+ def taskGettingResult(task: Task[_], taskInfo: TaskInfo) {
+ eventProcessActor ! GettingResultEvent(task, taskInfo)
+ }
+
+ // Called by TaskScheduler to report task completions or failures.
+ def taskEnded(
+ task: Task[_],
+ reason: TaskEndReason,
+ result: Any,
+ accumUpdates: Map[Long, Any],
+ taskInfo: TaskInfo,
+ taskMetrics: TaskMetrics) {
+ eventProcessActor ! CompletionEvent(task, reason, result, accumUpdates, taskInfo, taskMetrics)
+ }
+
+ // Called by TaskScheduler when an executor fails.
+ def executorLost(execId: String) {
+ eventProcessActor ! ExecutorLost(execId)
+ }
+
+ // Called by TaskScheduler when a host is added
+ def executorGained(execId: String, host: String) {
--- End diff --
I'd go with `executorAdded` here and everywhere else we have this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10368558
--- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -19,33 +19,43 @@ package org.apache.spark.scheduler
import java.util.Properties
+import scala.collection.Map
+
+import org.apache.spark.util.{Utils, Distribution}
import org.apache.spark.{Logging, TaskEndReason}
import org.apache.spark.executor.TaskMetrics
-import org.apache.spark.util.{Distribution, Utils}
+import org.apache.spark.storage.StorageStatus
+
+sealed trait SparkListenerEvent
+
+case class SparkListenerStageSubmitted(stageInfo: StageInfo, properties: Properties = null)
+ extends SparkListenerEvent
+
+case class SparkListenerStageCompleted(stageInfo: StageInfo) extends SparkListenerEvent
-sealed trait SparkListenerEvents
+case class SparkListenerTaskStart(stageId: Int, taskInfo: TaskInfo) extends SparkListenerEvent
-case class SparkListenerStageSubmitted(stage: StageInfo, properties: Properties)
- extends SparkListenerEvents
+case class SparkListenerTaskGettingResult(taskInfo: TaskInfo) extends SparkListenerEvent
-case class SparkListenerStageCompleted(stage: StageInfo) extends SparkListenerEvents
+case class SparkListenerTaskEnd(stageId: Int, taskType: String, reason: TaskEndReason,
+ taskInfo: TaskInfo, taskMetrics: TaskMetrics) extends SparkListenerEvent
-case class SparkListenerTaskStart(task: Task[_], taskInfo: TaskInfo) extends SparkListenerEvents
+case class SparkListenerJobStart(jobId: Int, stageIds: Seq[Int], properties: Properties = null)
+ extends SparkListenerEvent
-case class SparkListenerTaskGettingResult(
- task: Task[_], taskInfo: TaskInfo) extends SparkListenerEvents
+case class SparkListenerJobEnd(jobId: Int, jobResult: JobResult) extends SparkListenerEvent
-case class SparkListenerTaskEnd(task: Task[_], reason: TaskEndReason, taskInfo: TaskInfo,
- taskMetrics: TaskMetrics) extends SparkListenerEvents
+case class SparkListenerEnvironmentUpdate(environmentDetails: Map[String, Seq[(String, String)]])
+ extends SparkListenerEvent
-case class SparkListenerJobStart(job: ActiveJob, stageIds: Array[Int],
- properties: Properties = null) extends SparkListenerEvents
+case class SparkListenerExecutorsStateChange(storageStatusList: Seq[StorageStatus])
--- End diff --
Sounds reasonable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36929100
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36380733
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36959517
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37718880
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13192/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10362915
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -978,6 +971,11 @@ class DAGScheduler(
logDebug("Additional executor lost message for " + execId +
"(epoch " + currentEpoch + ")")
}
+ // Block manager master actor should not be null except during tests
+ if (blockManagerMaster.driverActor != null) {
+ val storageStatusList = blockManagerMaster.getStorageStatus
+ listenerBus.post(SparkListenerExecutorsStateChange(storageStatusList))
--- End diff --
@pwendell is the plan still to avoid having shared data structures between the UI and the main spark code? This seems to be contradictory to that plan but not sure if that's something you're planning to fix later/separately?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37903796
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36486544
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597261
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -634,6 +652,30 @@ private[spark] class Master(host: String, port: Int, webUiPort: Int,
}
}
+ /**
+ * Start a new SparkUI rendered from persisted storage. If this is unsuccessful for any reason,
+ * return None. Otherwise return the reconstructed UI.
+ */
+ def startPersistedSparkUI(app: ApplicationInfo): Option[SparkUI] = {
+ val appName = app.desc.name
+ val eventLogInfo = app.desc.eventLogInfo.getOrElse { return None }
+ val eventLogDir = eventLogInfo.logDir
+ val eventCompressionCodec = eventLogInfo.compressionCodec
+ val appConf = new SparkConf
+ eventCompressionCodec.foreach { codec =>
+ appConf.set("spark.eventLog.compress", "true")
+ appConf.set("spark.io.compression.codec", codec)
+ }
+ val ui = new SparkUI(appConf, "%s (finished)".format(appName), "/history/%s".format(app.id))
--- End diff --
If you did the other changes suggested then the logic here would be a bit different. You'd first create a ReplayBus then you'd pass it to the SparkUI then you'd call the utility function and pass in the logdir to the bus. That seems a bit more modular to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616570
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -196,13 +152,46 @@ class DAGScheduler(
}))
}
- def addSparkListener(listener: SparkListener) {
- listenerBus.addListener(listener)
+ // Called by TaskScheduler to report task's starting.
+ def taskStarted(task: Task[_], taskInfo: TaskInfo) {
+ eventProcessActor ! BeginEvent(task, taskInfo)
+ }
+
+ // Called to report that a task has completed and results are being fetched remotely.
+ def taskGettingResult(task: Task[_], taskInfo: TaskInfo) {
+ eventProcessActor ! GettingResultEvent(task, taskInfo)
+ }
+
+ // Called by TaskScheduler to report task completions or failures.
+ def taskEnded(
+ task: Task[_],
+ reason: TaskEndReason,
+ result: Any,
+ accumUpdates: Map[Long, Any],
+ taskInfo: TaskInfo,
+ taskMetrics: TaskMetrics) {
+ eventProcessActor ! CompletionEvent(task, reason, result, accumUpdates, taskInfo, taskMetrics)
+ }
+
+ // Called by TaskScheduler when an executor fails.
+ def executorLost(execId: String) {
+ eventProcessActor ! ExecutorLost(execId)
+ }
+
+ // Called by TaskScheduler when a host is added
+ def executorGained(execId: String, host: String) {
--- End diff --
This is actually existing code but yeah I'll change it too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10684119
--- Diff: core/src/test/java/org/apache/spark/JavaAPISuite.java ---
@@ -402,16 +403,16 @@ public Boolean call(Double x) {
@Test
public void javaDoubleRDDHistoGram() {
- JavaDoubleRDD rdd = sc.parallelizeDoubles(Arrays.asList(1.0, 2.0, 3.0, 4.0));
- // Test using generated buckets
- Tuple2<double[], long[]> results = rdd.histogram(2);
- double[] expected_buckets = {1.0, 2.5, 4.0};
- long[] expected_counts = {2, 2};
- Assert.assertArrayEquals(expected_buckets, results._1, 0.1);
- Assert.assertArrayEquals(expected_counts, results._2);
- // Test with provided buckets
- long[] histogram = rdd.histogram(expected_buckets);
- Assert.assertArrayEquals(expected_counts, histogram);
+ JavaDoubleRDD rdd = sc.parallelizeDoubles(Arrays.asList(1.0, 2.0, 3.0, 4.0));
+ // Test using generated buckets
+ Tuple2<double[], long[]> results = rdd.histogram(2);
+ double[] expected_buckets = {1.0, 2.5, 4.0};
+ long[] expected_counts = {2, 2};
+ Assert.assertArrayEquals(expected_buckets, results._1, 0.1);
+ Assert.assertArrayEquals(expected_counts, results._2);
+ // Test with provided buckets
+ long[] histogram = rdd.histogram(expected_buckets);
+ Assert.assertArrayEquals(expected_counts, histogram);
--- End diff --
Indent only
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38086119
Thanks I'll merge it when tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10641868
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1020,6 +1042,24 @@ class SparkContext(
/** Register a new RDD, returning its RDD ID */
private[spark] def newRddId(): Int = nextRddId.getAndIncrement()
+ /** Post the environment update event if the listener bus is ready */
+ private def postEnvironmentUpdate() {
+ Option(listenerBus).foreach { bus =>
--- End diff --
If there is no normal case where it can be undefined, I'd rather just do `listenerBus.get`. In other words, I think it's not a great pattern to use `Option` to hide things that are actually code bugs. We should just use `Option` when there is a legitimate case where it hasn't been initialized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10272860
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.SparkContext
+import org.apache.spark.scheduler._
+import org.apache.spark.storage._
+import org.apache.spark.util.FileLogger
+import org.apache.spark.util.JsonProtocol
+
+private[ui] trait UISparkListener extends SparkListener
+
+/**
+ * A SparkListener that serves as an entry point for all events posted to the UI.
+ *
+ * GatewayUISparkListener achieves two functions:
+ *
+ * (1) If the UI is live, GatewayUISparkListener posts each event to all attached listeners
+ * then logs it as JSON. This centralizes event logging and avoids having all attached
+ * listeners log the events on their own.
+ *
+ * (2) If the UI is rendered from disk, GatewayUISparkListener replays each event deserialized
+ * from the event logs to all attached listeners.
+ *
+ * Event logging is specified by three configurable parameters:
+ *
+ * spark.eventLog.enabled - Whether event logging is enabled.
+ * spark.eventLog.dir - Path to the directory in which events are logged.
+ * spark.eventLog.overwrite - Whether to overwrite any existing files.
+ */
+private[ui] class GatewayUISparkListener(parent: SparkUI, sc: SparkContext) extends SparkListener {
+
+ // Log events only if the UI is live and event logging is enabled
+ private val logger: Option[FileLogger] = {
+ if (sc != null && sc.conf.getBoolean("spark.eventLog.enabled", false)) {
+ val logDir = sc.conf.get("spark.eventLog.dir", "/tmp/spark-events")
+ val appName = sc.appName.replaceAll("[ /]", "_").toLowerCase
--- End diff --
I think here we'd want a nonce also (e.g. timestamp) because otherwise if someone runs the same app twice the files will overwrite eachother.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36939032
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13025/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10361480
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -27,28 +28,58 @@ import org.apache.spark.ui.jobs.JobProgressUI
import org.apache.spark.ui.storage.BlockManagerUI
import org.apache.spark.util.Utils
-/** Top level user interface for Spark */
-private[spark] class SparkUI(sc: SparkContext) extends Logging {
- val host = Option(System.getenv("SPARK_PUBLIC_DNS")).getOrElse(Utils.localHostName())
- val port = sc.conf.get("spark.ui.port", SparkUI.DEFAULT_PORT).toInt
- var boundPort: Option[Int] = None
- var server: Option[Server] = None
+/** Top level user interface for Spark. */
+private[spark] class SparkUI(val sc: SparkContext, conf: SparkConf, port: Int) extends Logging {
- val handlers = Seq[(String, Handler)](
+ def this(sc: SparkContext) =
+ this(sc, sc.conf, sc.conf.get("spark.ui.port", SparkUI.DEFAULT_PORT).toInt)
+
+ // Persisted UI constructors
+ def this(conf: SparkConf, port: Int) = this(null, conf, port)
+ def this(conf: SparkConf) =
--- End diff --
Are these two constructors ever used?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36708976
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13000/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37247913
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616549
--- Diff: core/src/main/scala/org/apache/spark/storage/PutResult.scala ---
@@ -20,7 +20,13 @@ package org.apache.spark.storage
import java.nio.ByteBuffer
/**
- * Result of adding a block into a BlockStore. Contains its estimated size, and possibly the
- * values put if the caller asked for them to be returned (e.g. for chaining replication)
+ * Result of adding a block into a BlockStore. This case class contains a few things:
+ * (1) The estimated size of the put,
+ * (2) The values put if the caller asked for them to be returned (e.g. for chaining
+ * replication), and
+ * (3) A list of blocks dropped as a result of this put. This is always empty for DiskStore.
*/
-private[spark] case class PutResult(size: Long, data: Either[Iterator[_], ByteBuffer])
+private[spark] case class PutResult(
+ size: Long,
+ data: Either[Iterator[_], ByteBuffer],
+ droppedBlocks: Seq[(BlockId, BlockStatus)] = Seq())
--- End diff --
Ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38099755
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37891872
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36932993
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36588480
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10284428
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io.{FileInputStream, File}
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
import org.eclipse.jetty.server.{Handler, Server}
+import org.json4s.jackson.JsonMethods._
import org.apache.spark.{Logging, SparkContext, SparkEnv}
+import org.apache.spark.scheduler._
import org.apache.spark.ui.JettyUtils._
import org.apache.spark.ui.env.EnvironmentUI
import org.apache.spark.ui.exec.ExecutorsUI
import org.apache.spark.ui.jobs.JobProgressUI
import org.apache.spark.ui.storage.BlockManagerUI
import org.apache.spark.util.Utils
+import org.apache.spark.util.JsonProtocol
-/** Top level user interface for Spark */
-private[spark] class SparkUI(sc: SparkContext) extends Logging {
+/** Top level user interface for Spark. */
+private[spark] class SparkUI(val sc: SparkContext) extends Logging {
--- End diff --
Having an ```Option[SparkContext]``` argument means we have to do ```new SparkUI(Some(this)) ``` from SparkContext, which seems strange to me.
To hide the ugliness of null SparkContexts, I think a better way is to make a new constructor that looks something like ```def this() = this(null)```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10238954
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.scheduler._
+import org.apache.spark.storage._
+import org.apache.spark.util.FileLogger
+import org.apache.spark.util.JsonProtocol
+import org.apache.spark.SparkContext
+
+private[ui] trait UISparkListener extends SparkListener
+
+/**
+ * A SparkListener that serves as an entry point for all events posted to the UI.
+ *
+ * GatewayUISparkListener achieves two functions:
+ *
+ * (1) If the UI is live, GatewayUISparkListener posts each event to all attached listeners
+ * then logs it as JSON. This centralizes event logging and avoids having all attached
+ * listeners log the events on their own.
+ *
+ * (2) If the UI is rendered from disk, GatewayUISparkListener replays each event deserialized
+ * from the event logs to all attached listeners.
+ */
+private[ui] class GatewayUISparkListener(parent: SparkUI, sc: SparkContext) extends SparkListener {
--- End diff --
This class doesn't seem particularly specific to the UI. It might be a good idea to expose this to the rest of Spark as a general utility that could be used potentially by other higher level applications that hook into the Listener interface (e.g. not just Spark's own UI). Like basically build a listener class that can either get data from a live context or read it from a file, if the file is present. Then you could have a class that extends that and adds the more UI specific stuff, like dispatching to all these internal listeners.
The doc here also implies that the GatewayUISparkListener is responsible for replaying events, but as far as I can tell that happens in another location.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38085639
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10186518
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.scheduler._
+import org.apache.spark.storage._
+import org.apache.spark.util.FileLogger
+import org.apache.spark.util.JsonProtocol
+
+private[ui] trait UISparkListener extends SparkListener
+
+/**
+ * A SparkListener that serves as an entry point for all events posted to the UI.
+ *
+ * GatewayUISparkListener achieves two functions:
+ *
+ * (1) If the UI is live, GatewayUISparkListener posts each event to all attached listeners
+ * then logs it as JSON. This centralizes event logging and avoids having all attached
+ * listeners log the events on their own. By default, GatewayUISparkListener logs one
+ * file per job, though this needs not be the case.
--- End diff --
need not
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10596852
--- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala ---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import java.io._
+import java.net.URI
+import java.text.SimpleDateFormat
+import java.util.Date
+
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream
+import org.apache.hadoop.fs.{FSDataOutputStream, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+
+/**
+ * A generic class for logging information to file.
+ *
+ * @param logDir Path to the directory in which files are logged
+ * @param outputBufferSize The buffer size to use when writing to an output stream in bytes
+ * @param compress Whether to compress output
+ * @param overwrite Whether to overwrite existing files
+ */
+class FileLogger(
+ logDir: String,
+ conf: SparkConf = new SparkConf,
+ outputBufferSize: Int = 8 * 1024,
+ compress: Boolean = false,
+ overwrite: Boolean = true)
+ extends Logging {
+
+ private val DATE_FORMAT = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss")
+ private val fileSystem = Utils.getHadoopFileSystem(new URI(logDir))
+ private var fileIndex = 0
+
+ // Only used if compression is enabled
+ private lazy val compressionCodec = CompressionCodec.createCodec(conf)
+
+ // Only defined if the file system scheme is not local
+ private var hadoopDataStream: Option[FSDataOutputStream] = None
+
+ private var writer: Option[PrintWriter] = {
+ createLogDir()
+ Some(createWriter())
+ }
+
+ /** Create a logging directory with the given path. */
+ private def createLogDir() {
+ val path = new Path(logDir)
+ if (fileSystem.exists(path)) {
+ if (overwrite) {
+ logWarning("Log directory %s already exists. Overwriting...".format(logDir))
+ // Second parameter is whether to delete recursively
+ fileSystem.delete(path, true)
+ } else {
+ throw new IOException("Log directory %s already exists!".format(logDir))
+ }
+ }
+ if (!fileSystem.mkdirs(path)) {
+ throw new IOException("Error in creating log directory: %s".format(logDir))
+ }
+ }
+
+ /**
+ * Create a new writer for the file identified by the given path. File systems currently
+ * supported include HDFS, S3, and the local file system.
--- End diff --
I'd also just support any scheme here and not limit it to HDFS and S3... is there any reason we can't do that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37764131
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10183527
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala ---
@@ -415,6 +415,7 @@ class JavaSparkContext(val sc: SparkContext) extends JavaSparkContextVarargsWork
* Clear the job's list of JARs added by `addJar` so that they do not get downloaded to
* any new nodes.
*/
+ @deprecated("added jars are now temporary files and need not be deleted manually", "1.0.0")
--- End diff --
actually this is more like: "add jars no longer creates local copies so they do not need to be deleted"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36383621
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38029330
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13265/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37897192
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37718879
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13191/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37012312
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13044/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616395
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala ---
@@ -22,24 +22,25 @@ import java.text.SimpleDateFormat
import java.util.{Date, Properties}
import java.util.concurrent.LinkedBlockingQueue
-import scala.collection.mutable.{HashMap, HashSet, ListBuffer}
+import scala.collection.mutable.HashMap
import org.apache.spark._
import org.apache.spark.executor.TaskMetrics
-import org.apache.spark.rdd.RDD
-import org.apache.spark.storage.StorageLevel
/**
* A logger class to record runtime information for jobs in Spark. This class outputs one log file
- * for each Spark job, containing RDD graph, tasks start/stop, shuffle information.
- * JobLogger is a subclass of SparkListener, use addSparkListener to add JobLogger to a SparkContext
- * after the SparkContext is created.
- * Note that each JobLogger only works for one SparkContext
- * @param logDirName The base directory for the log files.
+ * for each Spark job, containing tasks start/stop and shuffle information. JobLogger is a subclass
+ * of SparkListener, use addSparkListener to add JobLogger to a SparkContext after the SparkContext
+ * is created. Note that each JobLogger only works for one SparkContext
+ *
+ * NOTE: The functionality of this class is heavily stripped down to accommodate for a general
+ * refactor of the SparkListener interface. In its place, the EventLoggingListener is introduced
+ * to log application information as SparkListenerEvents through the SparkUI. To enable this
--- End diff --
Previously it does in the sense that if you don't create a SparkUI then it definitely won't log events. With your suggested changes, it no longer does it "through the SparkUI"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597094
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -68,19 +105,53 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
/** Initialize all components of the server */
def start() {
- // NOTE: This is decoupled from bind() because of the following dependency cycle:
- // DAGScheduler() requires that the port of this server is known
- // This server must register all handlers, including JobProgressUI, before binding
- // JobProgressUI registers a listener with SparkContext, which requires sc to initialize
+ storage.start()
jobs.start()
+ env.start()
exec.start()
+
+ // Listen for events from the SparkContext if it exists, otherwise from persisted storage
+ val eventBus = if (live) {
+ val loggingEnabled = conf.getBoolean("spark.eventLog.enabled", false)
--- End diff --
I'm referring to lines 115-120.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37590182
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38009978
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36400692
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642658
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -68,19 +101,33 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
/** Initialize all components of the server */
def start() {
- // NOTE: This is decoupled from bind() because of the following dependency cycle:
- // DAGScheduler() requires that the port of this server is known
- // This server must register all handlers, including JobProgressUI, before binding
- // JobProgressUI registers a listener with SparkContext, which requires sc to initialize
+ storage.start()
jobs.start()
+ env.start()
exec.start()
+
+ // Storage status listener must receive events first, as other listeners depend on its state
+ listenerBus.addListener(storageStatusListener)
+ listenerBus.addListener(storage.listener)
+ listenerBus.addListener(jobs.listener)
+ listenerBus.addListener(env.listener)
+ listenerBus.addListener(exec.listener)
+
+ // Log events only if this UI is live and the feature is enabled
+ if (live && conf.getBoolean("spark.eventLog.enabled", false)) {
--- End diff --
Would this be better to just move into `SparkContext`? Then you don't need to have this live check. Also, the idea of enabling logging is really something that is not specific to the SparkUI but might be used for other things as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37901372
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38080923
@andrewor14 sorry - it's out of date again, I think some other stuff got merged
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10356197
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -32,7 +32,7 @@ import org.apache.spark.executor.TaskMetrics
import org.apache.spark.partial.{ApproximateActionListener, ApproximateEvaluator, PartialResult}
import org.apache.spark.rdd.RDD
import org.apache.spark.storage.{BlockId, BlockManager, BlockManagerMaster, RDDBlockId}
-import org.apache.spark.util.{MetadataCleaner, MetadataCleanerType, TimeStampedHashMap}
+import org.apache.spark.util.{Utils, MetadataCleaner, MetadataCleanerType, TimeStampedHashMap}
--- End diff --
alphabetization
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37251639
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10598046
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerStatusListener.scala ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.scheduler._
+
+/**
+ * A listener for BlockManager status updates.
+ *
+ * This listener provides a way to post executor storage status information as soon as it
+ * is available (i.e. immediately after the associated BlockManager has registered with the
+ * driver). This is necessary because the SparkContext is only notified when an executor is
+ * launched, but by then the storage information is not ready yet.
+ *
+ * Further, it is possible for a BlockManager be registered before the listener bus on the
+ * driver is initialized (e.g. the driver's own BlockManager), in which case the corresponding
+ * event should be buffered.
+ */
+private[spark] class BlockManagerStatusListener extends SparkListener {
--- End diff --
Is it possible to avoid this by just creating the EventBus earlier on in the SparkContext constructor (i.e. changing around the order so it comes before the the block manager)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36417620
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504458
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -307,97 +318,93 @@ class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf) extends Act
}
-private[spark]
-object BlockManagerMasterActor {
-
- case class BlockStatus(storageLevel: StorageLevel, memSize: Long, diskSize: Long)
+private[spark] case class BlockStatus(storageLevel: StorageLevel, memSize: Long, diskSize: Long)
- class BlockManagerInfo(
- val blockManagerId: BlockManagerId,
- timeMs: Long,
- val maxMem: Long,
- val slaveActor: ActorRef)
- extends Logging {
--- End diff --
Slight refactor to move BlockStatus and BlockManagerInfo out of BlockManagerMasterActor. The only change here is moving the two classes out of the BlockManagerMasterActor object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37265020
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38022094
Hey @andrewor14 I'm ready to merge this but it's gotten out of date with master. Mind up-merging it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37054543
I am unable to reproduce the test failure locally. Since the test failure is hidden deep within all the log messages, I have attached them below for convenience.
======================================================================
FAIL: test_termination_stdin (__main__.TestDaemon)
Ensure that daemon and workers terminate when stdin is closed.
----------------------------------------------------------------------
Traceback (most recent call last):
File "pyspark/tests.py", line 230, in test_termination_stdin
self.do_termination_test(lambda daemon: daemon.stdin.close())
File "pyspark/tests.py", line 226, in do_termination_test
self.fail("Expected EnvironmentError to be raised")
AssertionError: Expected EnvironmentError to be raised
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10641986
--- Diff: core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.io.InputStream
+import java.net.URI
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
+import org.apache.hadoop.fs.{Path, FileSystem}
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+/**
+ * An EventBus that replays logged events from persisted storage
+ */
+private[spark] class ReplayListenerBus(conf: SparkConf) extends SparkListenerBus with Logging {
+ private val compressed = conf.getBoolean("spark.eventLog.compress", false)
+
+ // Only used if compression is enabled
+ private lazy val compressionCodec = CompressionCodec.createCodec(conf)
+
+ /**
+ * Return a list of paths representing log files in the given directory.
+ */
+ private def getLogFilePaths(logDir: String, fileSystem: FileSystem): Array[Path] = {
+ val path = new Path(logDir)
+ if (!fileSystem.exists(path) || !fileSystem.getFileStatus(path).isDir) {
+ logWarning("Log path provided is not a valid directory: %s".format(logDir))
+ return Array[Path]()
+ }
+ val logStatus = fileSystem.listStatus(path)
+ if (logStatus == null || !logStatus.exists(!_.isDir)) {
+ logWarning("Log path provided contains no log files: %s".format(logDir))
+ return Array[Path]()
+ }
+ logStatus.filter(!_.isDir).map(_.getPath).sortBy(_.getName)
+ }
+
+ /**
+ * Replay each event in the order maintained in the given logs.
+ */
+ def replay(logDir: String): Boolean = {
+ val fileSystem = Utils.getHadoopFileSystem(new URI(logDir))
+ val logPaths = getLogFilePaths(logDir, fileSystem)
+ if (logPaths.length == 0) {
+ return false
+ }
+
+ logPaths.foreach { path =>
+ // In case there is an exception, keep track of the highest level stream to close it later
+ var streamToClose: Option[InputStream] = None
+ var currentLine = ""
+ try {
+ val fstream = fileSystem.open(path)
+ val bstream = new FastBufferedInputStream(fstream)
+ val cstream = if (compressed) compressionCodec.compressedInputStream(bstream) else bstream
+ streamToClose = Some(cstream)
+
+ // Parse each line as an event and post it to all attached listeners
+ val lines = Source.fromInputStream(cstream).getLines()
+ lines.foreach { line =>
+ currentLine = line
+ val event = JsonProtocol.sparkEventFromJson(parse(line))
+ postToAll(event)
+ }
+ } catch {
+ case e: Exception =>
+ logWarning("Exception in parsing Spark event log %s".format(path))
--- End diff --
I'd use `logError` for all of these messages (this is definitely an error case here if the UI can't be rendered). Also you can just pass the exception here in the first one - the log utilities can take a `Throwable`:
logError("Exception in parsing Spark event log %s".format(path), e)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36398305
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10361732
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class, since the UI thread and the DAGScheduler event loop may otherwise
* be reading/updating the internal data structures concurrently.
*/
-private[spark] class JobProgressListener(val sc: SparkContext) extends SparkListener {
+private[ui] class JobProgressListener(sc: SparkContext, live: Boolean)
+ extends StorageStatusSparkListener {
+
+ import JobProgressListener._
+
// How many stages to remember
- val RETAINED_STAGES = sc.conf.getInt("spark.ui.retainedStages", 1000)
- val DEFAULT_POOL_NAME = "default"
+ val retainedStages = if (live) {
+ sc.conf.getInt("spark.ui.retainedStages", DEFAULT_RETAINED_STAGES)
--- End diff --
Why not let this be configurable for re-created UIs too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10551998
--- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, FileLogger}
+
+/**
+ * A SparkListener that logs events to persistent storage.
+ *
+ * Event logging is specified by the following configurable parameters:
+ * spark.eventLog.enabled - Whether event logging is enabled.
+ * spark.eventLog.compress - Whether to compress logged events
+ * spark.eventLog.overwrite - Whether to overwrite any existing files.
+ * spark.eventLog.dir - Path to the directory in which events are logged.
+ * spark.eventLog.buffer.kb - Buffer size to use when writing to output streams
+ */
+private[spark] class EventLoggingListener(appName: String, conf: SparkConf)
+ extends SparkListener with Logging {
+
+ private val shouldCompress = conf.getBoolean("spark.eventLog.compress", false)
+ private val shouldOverwrite = conf.getBoolean("spark.eventLog.overwrite", true)
--- End diff --
I'd set this one to false and actually throw an exception that fails the job if the directory already exists (and overwrite is not selected).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10633506
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import org.apache.spark.SparkConf
+
+/**
+ * Reload a persisted UI independently from a SparkContext.
+ */
+object UIReloader {
+ def main(args: Array[String]) {
--- End diff --
I don't like it either, but it was convenient for me while I was still developing the feature. I'll remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10362811
--- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -19,33 +19,43 @@ package org.apache.spark.scheduler
import java.util.Properties
+import scala.collection.Map
+
+import org.apache.spark.util.{Utils, Distribution}
import org.apache.spark.{Logging, TaskEndReason}
import org.apache.spark.executor.TaskMetrics
-import org.apache.spark.util.{Distribution, Utils}
+import org.apache.spark.storage.StorageStatus
+
+sealed trait SparkListenerEvent
+
+case class SparkListenerStageSubmitted(stageInfo: StageInfo, properties: Properties = null)
+ extends SparkListenerEvent
+
+case class SparkListenerStageCompleted(stageInfo: StageInfo) extends SparkListenerEvent
-sealed trait SparkListenerEvents
+case class SparkListenerTaskStart(stageId: Int, taskInfo: TaskInfo) extends SparkListenerEvent
-case class SparkListenerStageSubmitted(stage: StageInfo, properties: Properties)
- extends SparkListenerEvents
+case class SparkListenerTaskGettingResult(taskInfo: TaskInfo) extends SparkListenerEvent
-case class SparkListenerStageCompleted(stage: StageInfo) extends SparkListenerEvents
+case class SparkListenerTaskEnd(stageId: Int, taskType: String, reason: TaskEndReason,
+ taskInfo: TaskInfo, taskMetrics: TaskMetrics) extends SparkListenerEvent
-case class SparkListenerTaskStart(task: Task[_], taskInfo: TaskInfo) extends SparkListenerEvents
+case class SparkListenerJobStart(jobId: Int, stageIds: Seq[Int], properties: Properties = null)
+ extends SparkListenerEvent
-case class SparkListenerTaskGettingResult(
- task: Task[_], taskInfo: TaskInfo) extends SparkListenerEvents
+case class SparkListenerJobEnd(jobId: Int, jobResult: JobResult) extends SparkListenerEvent
-case class SparkListenerTaskEnd(task: Task[_], reason: TaskEndReason, taskInfo: TaskInfo,
- taskMetrics: TaskMetrics) extends SparkListenerEvents
+case class SparkListenerEnvironmentUpdate(environmentDetails: Map[String, Seq[(String, String)]])
+ extends SparkListenerEvent
-case class SparkListenerJobStart(job: ActiveJob, stageIds: Array[Int],
- properties: Properties = null) extends SparkListenerEvents
+case class SparkListenerExecutorsStateChange(storageStatusList: Seq[StorageStatus])
--- End diff --
Why not have an explicit executor added and executor removed event? It seems unnecessary to send all of the storateStatuses
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642180
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -54,87 +54,53 @@ import org.apache.spark.util.{MetadataCleaner, MetadataCleanerType, TimeStampedH
*/
private[spark]
class DAGScheduler(
- taskSched: TaskScheduler,
--- End diff --
Is this diff purely code clean-up, or do you change any functionality here (other than passing in the listener)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504353
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -229,10 +243,60 @@ object SparkEnv extends Logging {
broadcastManager,
blockManager,
connectionManager,
+ securityManager,
httpFileServer,
sparkFilesDir,
metricsSystem,
- conf,
- securityManager)
+ conf)
+ }
+
+ /**
+ * Return a map representation of jvm information, Spark properties, system properties, and
+ * class paths. Map keys define the category, and map values represent the corresponding
+ * attributes as a sequence of KV pairs. This is used mainly for SparkListenerEnvironmentUpdate.
+ */
+ private[spark]
+ def environmentDetails(
+ conf: SparkConf,
+ schedulingMode: String,
+ addedJars: Seq[String],
+ addedFiles: Seq[String]): Map[String, Seq[(String, String)]] = {
+
+ val jvmInformation = Seq(
+ ("Java Version", "%s (%s)".format(Properties.javaVersion, Properties.javaVendor)),
+ ("Java Home", Properties.javaHome),
+ ("Scala Version", Properties.versionString),
+ ("Scala Home", Properties.scalaHome)
+ ).sorted
+
+ // Spark properties, including scheduling mode whether or not it is configured
+ var additionalFields = Seq[(String, String)]()
+ conf.getOption("spark.scheduler.mode").getOrElse {
+ additionalFields ++= Seq(("spark.scheduler.mode", schedulingMode))
+ }
+ val sparkProperties = conf.getAll.sorted ++ additionalFields
+
+ // System properties that are not java classpaths
+ val systemProperties = System.getProperties.iterator.toSeq
+ val classPathProperty = systemProperties.find { case (k, v) =>
+ k == "java.class.path"
+ }.getOrElse(("", ""))
+ val otherProperties = systemProperties.filter { case (k, v) =>
+ k != "java.class.path" && !k.startsWith("spark.")
+ }.sorted
+
+ // Class paths including all added jars and files
+ val classPathEntries = classPathProperty._2
+ .split(conf.get("path.separator", ":"))
+ .filterNot(e => e.isEmpty)
+ .map(e => (e, "System Classpath"))
+ val addedJarsAndFiles = (addedJars ++ addedFiles).map((_, "Added By User"))
+ val classPaths = (addedJarsAndFiles ++ classPathEntries).sorted
+
+ Map[String, Seq[(String, String)]](
+ "JVM Information" -> jvmInformation,
+ "Spark Properties" -> sparkProperties,
+ "System Properties" -> otherProperties,
+ "Classpath Entries" -> classPaths)
--- End diff --
This is taken largely from EnvironmentUI.scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10730434
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala ---
@@ -57,27 +67,38 @@ class MasterWebUI(val master: Master, requestedPort: Int) extends Logging {
}
}
- val metricsHandlers = master.masterMetricsSystem.getServletHandlers ++
- master.applicationMetricsSystem.getServletHandlers
+ def boundPort: Int = serverInfo.map(_.boundPort).getOrElse(-1)
- val handlers = metricsHandlers ++ Seq[ServletContextHandler](
- createStaticHandler(MasterWebUI.STATIC_RESOURCE_DIR, "/static/*"),
- createServletHandler("/app/json",
- createServlet((request: HttpServletRequest) => applicationPage.renderJson(request),
- master.securityMgr)),
- createServletHandler("/app", createServlet((request: HttpServletRequest) => applicationPage
- .render(request), master.securityMgr)),
- createServletHandler("/json", createServlet((request: HttpServletRequest) => indexPage
- .renderJson(request), master.securityMgr)),
- createServletHandler("*", createServlet((request: HttpServletRequest) => indexPage.render
- (request), master.securityMgr))
- )
+ /** Attach a reconstructed UI to this Master UI. Only valid after bind(). */
+ def attachUI(ui: SparkUI) {
+ assert(serverInfo.isDefined, "Master UI must be initialized before attaching SparkUIs")
--- End diff --
Tiny nit - but it's not totally clear what initialize means here. It might make sense to say `initialized with bind()` or to add a javadoc in `bind()` that uses the word initialize.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37247719
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37604458
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10360310
--- Diff: core/src/main/scala/org/apache/spark/scheduler/EventBus.scala ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * A SparkListenerEvent bus that relays events to its listeners.
+ */
+private[spark] trait EventBus {
+
+ // SparkListeners attached to this event bus
+ protected val sparkListeners = new ArrayBuffer[SparkListener]
+ with mutable.SynchronizedBuffer[SparkListener]
+
+ def addListener(listener: SparkListener) {
+ sparkListeners += listener
+ }
+
+ /**
+ * Post an event to all attached listeners. Return true if the shutdown event is posted.
+ */
+ protected def postToAll(event: SparkListenerEvent): Boolean = {
+ postToListeners(event, sparkListeners)
+ }
+
+ /**
+ * Post an event to a given list of listeners. Return true if the shutdown event is posted.
+ */
+ protected def postToListeners(
--- End diff --
Is there a reason to have this as a separate method instead of just part of postToAll? It looks like it's only called from postToAll.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36314536
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10240683
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io.{FileInputStream, File}
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
import org.eclipse.jetty.server.{Handler, Server}
+import org.json4s.jackson.JsonMethods._
import org.apache.spark.{Logging, SparkContext, SparkEnv}
+import org.apache.spark.scheduler._
import org.apache.spark.ui.JettyUtils._
import org.apache.spark.ui.env.EnvironmentUI
import org.apache.spark.ui.exec.ExecutorsUI
import org.apache.spark.ui.jobs.JobProgressUI
import org.apache.spark.ui.storage.BlockManagerUI
import org.apache.spark.util.Utils
+import org.apache.spark.util.JsonProtocol
-/** Top level user interface for Spark */
-private[spark] class SparkUI(sc: SparkContext) extends Logging {
+/** Top level user interface for Spark. */
+private[spark] class SparkUI(val sc: SparkContext) extends Logging {
--- End diff --
I went down that road once and ended up having a bunch of ``` sc.map { ... }.getOrElse ``` and soon gave up. I think it's clearer with ``` if (live) { ... } else { ... } ```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36489261
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10239640
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.scheduler._
+import org.apache.spark.storage._
+import org.apache.spark.util.FileLogger
+import org.apache.spark.util.JsonProtocol
+import org.apache.spark.SparkContext
+
+private[ui] trait UISparkListener extends SparkListener
+
+/**
+ * A SparkListener that serves as an entry point for all events posted to the UI.
+ *
+ * GatewayUISparkListener achieves two functions:
+ *
+ * (1) If the UI is live, GatewayUISparkListener posts each event to all attached listeners
+ * then logs it as JSON. This centralizes event logging and avoids having all attached
+ * listeners log the events on their own.
+ *
+ * (2) If the UI is rendered from disk, GatewayUISparkListener replays each event deserialized
+ * from the event logs to all attached listeners.
+ */
+private[ui] class GatewayUISparkListener(parent: SparkUI, sc: SparkContext) extends SparkListener {
--- End diff --
I discussed with @andrewor14 offline, but the proposal was basically to refactor things in a way that makes the components more re-usable. Basically we'll separate the three responsibilities here into distinct classes:
a) logging a stream of events to a file.
b) creating an event bus as the result of logged events.
c) mutliplexing several spark listeners from one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36590957
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12980/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642235
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -472,41 +500,58 @@ private[spark] class BlockManager(
* This is currently used for writing shuffle files out. Callers should handle error
* cases.
*/
- def getDiskWriter(blockId: BlockId, file: File, serializer: Serializer, bufferSize: Int)
- : BlockObjectWriter = {
+ def getDiskWriter(
+ blockId: BlockId,
+ file: File,
+ serializer: Serializer,
+ bufferSize: Int) : BlockObjectWriter = {
--- End diff --
not your code, but maybe clean up the spacing here? `Int): BlockObjectWriter = {`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616671
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -68,19 +105,53 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
/** Initialize all components of the server */
def start() {
- // NOTE: This is decoupled from bind() because of the following dependency cycle:
- // DAGScheduler() requires that the port of this server is known
- // This server must register all handlers, including JobProgressUI, before binding
- // JobProgressUI registers a listener with SparkContext, which requires sc to initialize
+ storage.start()
jobs.start()
+ env.start()
exec.start()
+
+ // Listen for events from the SparkContext if it exists, otherwise from persisted storage
+ val eventBus = if (live) {
+ val loggingEnabled = conf.getBoolean("spark.eventLog.enabled", false)
--- End diff --
Yup, that could get rid of two options that are mutually exclusive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10357546
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerRegistrationListener.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import org.apache.spark.scheduler._
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+/** A listener for block manager registration */
--- End diff --
It looks like the reason you need this is because there's an initialization ordering issue -- where the BlockManager gets initialized before the dag scheduler / listener buses, so you need this intermediate thing? It would be great if you could document this in the doc here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37595661
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37602837
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10596774
--- Diff: docs/configuration.md ---
@@ -490,7 +490,30 @@ Apart from these, the following properties are also available, and may be useful
<td>spark.logConf</td>
<td>false</td>
<td>
- Log the supplied SparkConf as INFO at start of spark context.
+ Whether to log the supplied SparkConf as INFO at start of spark context.
+ </td>
+</tr>
+<tr>
+ <td>spark.eventLog.enabled</td>
+ <td>false</td>
+ <td>
+ Whether to log spark events, useful for reconstructing the Web UI after the application has finished.
+ </td>
+</tr>
+<tr>
+ <td>spark.eventLog.compress</td>
+ <td>false</td>
+ <td>
+ Whether to compress logged events, if <code>spark.eventLog.enabled</code> is true.
+ </td>
+</tr>
+<tr>
+ <td>spark.eventLog.dir</td>
+ <td>/tmp/spark-events</td>
--- End diff --
could we put this as a URL in the example? Like maybe `file://tmp/spark-events`? I just want to to be clear it can be a full filesystem path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10363343
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class, since the UI thread and the DAGScheduler event loop may otherwise
* be reading/updating the internal data structures concurrently.
*/
-private[spark] class JobProgressListener(val sc: SparkContext) extends SparkListener {
+private[ui] class JobProgressListener(sc: SparkContext, live: Boolean)
+ extends StorageStatusSparkListener {
--- End diff --
This seems a little confusing to me...it looks like this extends StorageStatusSparkListener because it needs access to the list of executors and their addresses through the storageStatusList member variable. But for JobProgressListener, storageStatusList isn't really kept up to date correctly -- because onTaskEnd gets overridden by JobProgressListener, so the RDD info won't get correctly updated when the task ends.
I'd propose simplifying this by having JobProgressListener just extend the usual SparkListener. Then, you need a way of getting the executors for the ExecutorTable on the stage status page. But you can get the list of executor Ids / hostnames from the list of tasks (which works because you only care about the executors that are being used by one of the tasks anyway).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37057167
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37895068
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36416705
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10458118
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+import org.apache.spark.storage._
+
+private[ui] trait UISparkListener extends SparkListener
--- End diff --
Not at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597228
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -68,19 +105,53 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
/** Initialize all components of the server */
def start() {
- // NOTE: This is decoupled from bind() because of the following dependency cycle:
- // DAGScheduler() requires that the port of this server is known
- // This server must register all handlers, including JobProgressUI, before binding
- // JobProgressUI registers a listener with SparkContext, which requires sc to initialize
+ storage.start()
jobs.start()
+ env.start()
exec.start()
+
+ // Listen for events from the SparkContext if it exists, otherwise from persisted storage
+ val eventBus = if (live) {
+ val loggingEnabled = conf.getBoolean("spark.eventLog.enabled", false)
+ if (loggingEnabled) {
+ val logger = new EventLoggingListener(appName, conf)
+ eventLogger = Some(logger)
+ sc.listenerBus.addListener(logger)
+ }
+ sc.listenerBus
+ } else {
+ replayerBus = Some(new SparkReplayerBus(conf))
+ replayerBus.get
+ }
+
+ // Storage status listener must receive events first, as other listeners depend on its state
+ eventBus.addListener(storageStatusListener)
+ eventBus.addListener(storage.listener)
+ eventBus.addListener(jobs.listener)
+ eventBus.addListener(env.listener)
+ eventBus.addListener(exec.listener)
+ started = true
+ }
+
+ /**
+ * Reconstruct a previously persisted SparkUI from logs residing in the given directory.
+ *
+ * This method must be invoked after the SparkUI has started. Return true if log files
+ * are found and processed.
+ */
+ def renderFromPersistedStorage(logDir: String): Boolean = {
--- End diff --
This also might be nice to take out of the SparkUI and make a utility function associated with a ReplayBus. For instance the ReplayBus could have an associated object and that has a utility function like:
```
object ReplayBus {
def replayFromPersistentStorage(logDir: String, bus: ReplayBus)
}
```
The concept of replay is sort of distinct from the UI and might be useful for other listener implementations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37374818
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36315929
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642071
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -634,6 +653,36 @@ private[spark] class Master(host: String, port: Int, webUiPort: Int,
}
}
+ /**
+ * Start a new SparkUI rendered from persisted storage. If this is unsuccessful for any reason,
+ * return None. Otherwise return the reconstructed UI.
+ */
+ def startPersistedSparkUI(app: ApplicationInfo): Option[SparkUI] = {
--- End diff --
When a completed application gets removed from `completedApps` we should also remove the UI handlers so that they get garbage collected. Checkout the lines relating to `RETAINED_APPLICATIONS` inside of `removeApplication`. Otherwise, over time, this will cause the master to run out of memory because of all the UI state.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10641969
--- Diff: core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.io.InputStream
+import java.net.URI
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
+import org.apache.hadoop.fs.{Path, FileSystem}
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+/**
+ * An EventBus that replays logged events from persisted storage
+ */
+private[spark] class ReplayListenerBus(conf: SparkConf) extends SparkListenerBus with Logging {
+ private val compressed = conf.getBoolean("spark.eventLog.compress", false)
+
+ // Only used if compression is enabled
+ private lazy val compressionCodec = CompressionCodec.createCodec(conf)
+
+ /**
+ * Return a list of paths representing log files in the given directory.
+ */
+ private def getLogFilePaths(logDir: String, fileSystem: FileSystem): Array[Path] = {
+ val path = new Path(logDir)
+ if (!fileSystem.exists(path) || !fileSystem.getFileStatus(path).isDir) {
+ logWarning("Log path provided is not a valid directory: %s".format(logDir))
+ return Array[Path]()
+ }
+ val logStatus = fileSystem.listStatus(path)
+ if (logStatus == null || !logStatus.exists(!_.isDir)) {
+ logWarning("Log path provided contains no log files: %s".format(logDir))
+ return Array[Path]()
+ }
+ logStatus.filter(!_.isDir).map(_.getPath).sortBy(_.getName)
+ }
+
+ /**
+ * Replay each event in the order maintained in the given logs.
+ */
+ def replay(logDir: String): Boolean = {
+ val fileSystem = Utils.getHadoopFileSystem(new URI(logDir))
+ val logPaths = getLogFilePaths(logDir, fileSystem)
+ if (logPaths.length == 0) {
+ return false
+ }
+
+ logPaths.foreach { path =>
+ // In case there is an exception, keep track of the highest level stream to close it later
+ var streamToClose: Option[InputStream] = None
+ var currentLine = ""
--- End diff --
It might make sense to initialize this as something like "<not started>" in case there is an exception opening the streams.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10363792
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -978,6 +971,11 @@ class DAGScheduler(
logDebug("Additional executor lost message for " + execId +
"(epoch " + currentEpoch + ")")
}
+ // Block manager master actor should not be null except during tests
+ if (blockManagerMaster.driverActor != null) {
+ val storageStatusList = blockManagerMaster.getStorageStatus
+ listenerBus.post(SparkListenerExecutorsStateChange(storageStatusList))
--- End diff --
Hm ya @andrewor14 is there a reason the storage status needs to be updated here? Logging the global storage status is pretty expensive. I figured maybe it was because if an executor fails the blocks could get re-replicated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36706595
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10272735
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io.{FileInputStream, File}
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
import org.eclipse.jetty.server.{Handler, Server}
+import org.json4s.jackson.JsonMethods._
import org.apache.spark.{Logging, SparkContext, SparkEnv}
+import org.apache.spark.scheduler._
import org.apache.spark.ui.JettyUtils._
import org.apache.spark.ui.env.EnvironmentUI
import org.apache.spark.ui.exec.ExecutorsUI
import org.apache.spark.ui.jobs.JobProgressUI
import org.apache.spark.ui.storage.BlockManagerUI
import org.apache.spark.util.Utils
+import org.apache.spark.util.JsonProtocol
-/** Top level user interface for Spark */
-private[spark] class SparkUI(sc: SparkContext) extends Logging {
+/** Top level user interface for Spark. */
+private[spark] class SparkUI(val sc: SparkContext) extends Logging {
+
+ // If SparkContext is not provided, assume this UI is rendered from persisted storage
+ val live = sc != null
val host = Option(System.getenv("SPARK_PUBLIC_DNS")).getOrElse(Utils.localHostName())
- val port = sc.conf.get("spark.ui.port", SparkUI.DEFAULT_PORT).toInt
+ var port = if (live) {
+ sc.conf.get("spark.ui.port", SparkUI.DEFAULT_PORT).toInt
+ } else {
+ SparkUI.DEFAULT_PERSISTED_PORT.toInt
+ }
var boundPort: Option[Int] = None
var server: Option[Server] = None
+ var started = false
+ var appName = ""
- val handlers = Seq[(String, Handler)](
+ private val handlers = Seq[(String, Handler)](
("/static", createStaticHandler(SparkUI.STATIC_RESOURCE_DIR)),
("/", createRedirectHandler("/stages"))
)
- val storage = new BlockManagerUI(sc)
- val jobs = new JobProgressUI(sc)
- val env = new EnvironmentUI(sc)
- val exec = new ExecutorsUI(sc)
+ private val storage = new BlockManagerUI(this)
+ private val jobs = new JobProgressUI(this)
+ private val env = new EnvironmentUI(this)
+ private val exec = new ExecutorsUI(this)
// Add MetricsServlet handlers by default
- val metricsServletHandlers = SparkEnv.get.metricsSystem.getServletHandlers
+ private val metricsServletHandlers = if (live) {
+ SparkEnv.get.metricsSystem.getServletHandlers
+ } else {
+ Array[(String, Handler)]()
+ }
- val allHandlers = storage.getHandlers ++ jobs.getHandlers ++ env.getHandlers ++
+ private val allHandlers = storage.getHandlers ++ jobs.getHandlers ++ env.getHandlers ++
exec.getHandlers ++ metricsServletHandlers ++ handlers
+ // Maintain a gateway listener for all events to simplify event logging
+ private var _gatewayListener: Option[GatewayUISparkListener] = None
+
+ def gatewayListener = _gatewayListener.getOrElse {
+ val gateway = new GatewayUISparkListener(this, sc)
+ _gatewayListener = Some(gateway)
+ gateway
+ }
+
+ // Only meaningful if port is set before binding
+ def setPort(p: Int) = {
--- End diff --
Hm so actually what if you have this class take `Either[SparkContext, SparkConf]`? I think in general we want to use SparkConf for all configurations even when it's outside of a live context.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37372828
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37247912
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37372829
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10363030
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -625,6 +653,30 @@ private[spark] class Master(host: String, port: Int, webUiPort: Int) extends Act
}
}
+ /**
+ * Start a new SparkUI rendered from persisted storage. Assumes event logging information
+ * is available for given application.
+ */
+ def startPersistedSparkUI(app: ApplicationInfo): SparkUI = {
+ val appName = app.desc.name
+ val eventLogInfo = app.desc.eventLogInfo.get
+ val eventLogDir = eventLogInfo.logDir
--- End diff --
Then it would fail...
This will actually be refactored in the following way shortly - Each SparkUI will take in as its argument a Jetty server and simply act as a handler to that server. Then, we'll have the Master pass its own server (the one that hosts the Master UI on port 8080) to the SparkUI. With this change, the rebuilt SparkUI will always be hosted on the Master node.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10361689
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class, since the UI thread and the DAGScheduler event loop may otherwise
* be reading/updating the internal data structures concurrently.
--- End diff --
Is this still true?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504478
--- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala ---
@@ -43,14 +47,18 @@ case class StorageStatus(blockManagerId: BlockManagerId, maxMem: Long,
}
}
-case class RDDInfo(id: Int, name: String, storageLevel: StorageLevel,
- numCachedPartitions: Int, numPartitions: Int, memSize: Long, diskSize: Long)
+private[spark]
+class RDDInfo(val id: Int, val name: String, val numPartitions: Int, val storageLevel: StorageLevel)
extends Ordered[RDDInfo] {
+
+ var numCachedPartitions = 0
+ var memSize = 0L
+ var diskSize = 0L
+
--- End diff --
Same here. RDDInfo now has mutable fields.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36380483
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37374815
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13122/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504393
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -54,87 +54,53 @@ import org.apache.spark.util.{MetadataCleaner, MetadataCleanerType, TimeStampedH
*/
private[spark]
class DAGScheduler(
- taskSched: TaskScheduler,
+ taskScheduler: TaskScheduler,
+ listenerBus: SparkListenerBus,
mapOutputTracker: MapOutputTrackerMaster,
blockManagerMaster: BlockManagerMaster,
env: SparkEnv)
extends Logging {
- def this(taskSched: TaskScheduler) {
- this(taskSched, SparkEnv.get.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster],
- SparkEnv.get.blockManager.master, SparkEnv.get)
- }
- taskSched.setDAGScheduler(this)
-
- // Called by TaskScheduler to report task's starting.
- def taskStarted(task: Task[_], taskInfo: TaskInfo) {
- eventProcessActor ! BeginEvent(task, taskInfo)
- }
-
- // Called to report that a task has completed and results are being fetched remotely.
- def taskGettingResult(task: Task[_], taskInfo: TaskInfo) {
- eventProcessActor ! GettingResultEvent(task, taskInfo)
- }
+ import DAGScheduler._
- // Called by TaskScheduler to report task completions or failures.
- def taskEnded(
- task: Task[_],
- reason: TaskEndReason,
- result: Any,
- accumUpdates: Map[Long, Any],
- taskInfo: TaskInfo,
- taskMetrics: TaskMetrics) {
- eventProcessActor ! CompletionEvent(task, reason, result, accumUpdates, taskInfo, taskMetrics)
+ def this(sc: SparkContext, taskScheduler: TaskScheduler) = {
+ this(
+ taskScheduler,
+ sc.listenerBus,
+ sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster],
+ sc.env.blockManager.master,
+ sc.env)
}
- // Called by TaskScheduler when an executor fails.
- def executorLost(execId: String) {
- eventProcessActor ! ExecutorLost(execId)
- }
-
- // Called by TaskScheduler when a host is added
- def executorGained(execId: String, host: String) {
- eventProcessActor ! ExecutorGained(execId, host)
- }
-
- // Called by TaskScheduler to cancel an entire TaskSet due to either repeated failures or
- // cancellation of the job itself.
- def taskSetFailed(taskSet: TaskSet, reason: String) {
- eventProcessActor ! TaskSetFailed(taskSet, reason)
- }
-
- // The time, in millis, to wait for fetch failure events to stop coming in after one is detected;
- // this is a simplistic way to avoid resubmitting tasks in the non-fetchable map stage one by one
- // as more failure events come in
- val RESUBMIT_TIMEOUT = 200.milliseconds
-
- // The time, in millis, to wake up between polls of the completion queue in order to potentially
- // resubmit failed stages
- val POLL_TIMEOUT = 10L
-
- // Warns the user if a stage contains a task with size greater than this value (in KB)
- val TASK_SIZE_TO_WARN = 100
--- End diff --
These lines are moved to elsewhere in the file, not deleted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10551852
--- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.io.InputStream
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
+import org.apache.hadoop.fs.{Path, FileSystem}
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+/**
+ * An EventBus that replays logged events from persisted storage
+ */
+private[spark] class SparkReplayerBus(conf: SparkConf) extends EventBus with Logging {
+ private val compressed = conf.getBoolean("spark.eventLog.compress", false)
+
+ // Only used if compression is enabled
+ private lazy val compressionCodec = CompressionCodec.createCodec(conf)
+
+ /**
+ * Return a list of paths representing log files in the given directory.
+ */
+ private def getLogFilePaths(logDir: String, fileSystem: FileSystem): Array[Path] = {
+ val path = new Path(logDir)
+ if (!fileSystem.exists(path) || !fileSystem.getFileStatus(path).isDir) {
+ logWarning("Log path provided is not a valid directory: %s".format(logDir))
+ return Array[Path]()
+ }
+ val logStatus = fileSystem.listStatus(path)
+ if (logStatus == null || !logStatus.exists(!_.isDir)) {
+ logWarning("Log path provided contains no log files: %s".format(logDir))
+ return Array[Path]()
+ }
+ logStatus.filter(!_.isDir).map(_.getPath).sortBy(_.getName)
+ }
+
+ /**
+ * Replay each event in the order maintained in the given logs.
+ */
+ def replay(logDir: String): Boolean = {
+ val fileSystem = Utils.getHadoopFileSystem(logDir)
+ val logPaths = getLogFilePaths(logDir, fileSystem)
+ if (logPaths.length == 0) {
+ return false
+ }
+
+ logPaths.foreach { path =>
+ // In case there is an exception, keep track of the highest level stream to close it later
+ var streamToClose: Option[InputStream] = None
+ var currentLine = ""
+ try {
+ val fstream = fileSystem.open(path)
+ val bstream = new FastBufferedInputStream(fstream)
+ val cstream = if (compressed) compressionCodec.compressedInputStream(bstream) else bstream
+ streamToClose = Some(cstream)
+
+ // Parse each line as an event and post it to all attached listeners
+ val lines = Source.fromInputStream(cstream).getLines()
+ lines.foreach { line =>
+ currentLine = line
+ val event = JsonProtocol.sparkEventFromJson(parse(line))
+ postToAll(event)
+ }
+ } catch {
+ case e: Exception =>
+ logWarning("Exception in parsing UI logs for %s".format(path))
--- End diff --
I would just say "Exception parsing spark event log from %s" (i.e. don't make this UI specific).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10596479
--- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.io.InputStream
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
+import org.apache.hadoop.fs.{Path, FileSystem}
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, Utils}
+
+/**
+ * An EventBus that replays logged events from persisted storage
+ */
+private[spark] class SparkReplayerBus(conf: SparkConf) extends EventBus with Logging {
--- End diff --
I think Replay/Live is reasonable here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37764130
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642674
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -30,32 +30,65 @@ import org.apache.spark.ui.jobs.JobProgressUI
import org.apache.spark.ui.storage.BlockManagerUI
import org.apache.spark.util.Utils
-/** Top level user interface for Spark */
-private[spark] class SparkUI(sc: SparkContext) extends Logging {
- val host = Option(System.getenv("SPARK_PUBLIC_DNS")).getOrElse(Utils.localHostName())
- val port = sc.conf.get("spark.ui.port", SparkUI.DEFAULT_PORT).toInt
- var boundPort: Option[Int] = None
- var server: Option[Server] = None
+/** Top level user interface for Spark. */
+private[spark] class SparkUI(
+ val sc: SparkContext,
+ conf: SparkConf,
+ val listenerBus: SparkListenerBus,
+ val appName: String,
+ val basePath: String = "")
+ extends Logging {
+
+ def this(sc: SparkContext) = this(sc, sc.conf, sc.listenerBus, sc.appName)
+ def this(conf: SparkConf, listenerBus: SparkListenerBus, appName: String, basePath: String) =
+ this(null, conf, listenerBus, appName, basePath)
+
+ // If SparkContext is not provided, assume the associated application is not live
+ val live = sc != null
+
+ val securityManager = if (live) sc.env.securityManager else new SecurityManager(conf)
+
+ private val host = Option(System.getenv("SPARK_PUBLIC_DNS")).getOrElse(Utils.localHostName())
+ private val port = conf.get("spark.ui.port", SparkUI.DEFAULT_PORT).toInt
+ private var boundPort: Option[Int] = None
+ private var server: Option[Server] = None
+
+ private val storage = new BlockManagerUI(this)
+ private val jobs = new JobProgressUI(this)
+ private val env = new EnvironmentUI(this)
+ private val exec = new ExecutorsUI(this)
- val handlers = Seq[ServletContextHandler] (
- createStaticHandler(SparkUI.STATIC_RESOURCE_DIR, "/static/*"),
- createRedirectHandler("/stages", "/")
- )
- val storage = new BlockManagerUI(sc)
- val jobs = new JobProgressUI(sc)
- val env = new EnvironmentUI(sc)
- val exec = new ExecutorsUI(sc)
+ val handlers: Seq[ServletContextHandler] = {
+ val metricsServletHandlers = if (live) {
+ SparkEnv.get.metricsSystem.getServletHandlers
+ } else {
+ Array[ServletContextHandler]()
+ }
+ storage.getHandlers ++
+ jobs.getHandlers ++
+ env.getHandlers ++
+ exec.getHandlers ++
+ metricsServletHandlers ++
+ Seq[ServletContextHandler] (
+ createStaticHandler(SparkUI.STATIC_RESOURCE_DIR, "/static/*"),
+ createRedirectHandler("/", "/stages", basePath)
+ )
+ }
+
+ // Maintain executor storage status through Spark events
+ val storageStatusListener = new StorageStatusListener
- // Add MetricsServlet handlers by default
- val metricsServletHandlers = SparkEnv.get.metricsSystem.getServletHandlers
+ // Only log events if this SparkUI is live
+ private var eventLogger: Option[EventLoggingListener] = None
--- End diff --
It seems a bit more intuitive for me to have this event logger tracked inside of `SparkContext` rather than the UI
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642204
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -54,87 +54,53 @@ import org.apache.spark.util.{MetadataCleaner, MetadataCleanerType, TimeStampedH
*/
private[spark]
class DAGScheduler(
- taskSched: TaskScheduler,
--- End diff --
It's just clean-up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38083750
It's up to date again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10360999
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -50,6 +51,9 @@ private[spark] class Master(host: String, port: Int, webUiPort: Int) extends Act
val REAPER_ITERATIONS = conf.getInt("spark.dead.worker.persistence", 15)
val RECOVERY_DIR = conf.get("spark.deploy.recoveryDirectory", "")
val RECOVERY_MODE = conf.get("spark.deploy.recoveryMode", "NONE")
+ val MAX_NUM_PERSISTED_UI = conf.getInt("spark.persisted.ui.maxConcurrent", 25)
+ val PERSISTED_SPARK_UI_PORT =
--- End diff --
don't think you need SPARK in the name here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10272169
--- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import java.io._
+import java.text.SimpleDateFormat
+import java.net.URI
+import java.util.Date
+
+import org.apache.spark.Logging
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.hadoop.fs.{FSDataOutputStream, Path, FileSystem}
+
+/**
+ * A generic class for logging information to file.
+ *
+ * @param logBaseDir Path to the directory in which files are logged
+ * @param name An identifier of each FileLogger instance
+ * @param overwriteExistingFiles Whether to overwrite existing files
+ */
+class FileLogger(
+ logBaseDir: String,
+ name: String = String.valueOf(System.currentTimeMillis()),
+ overwriteExistingFiles: Boolean = true)
+ extends Logging {
+
+ private val logDir = logBaseDir.stripSuffix("/") + "/" + name
+ private val DATE_FORMAT = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss")
+ private var fileIndex = 0
+
+ // Only defined if the file system scheme uses the Hadoop API
+ private var hadoopDataStream: Option[FSDataOutputStream] = None
+ private var hadoopFileSystem: Option[FileSystem] = None
+
+ private var writer: Option[PrintWriter] = {
+ createLogDir()
+ createWriter()
+ }
+
+ /** Create a logging directory with the given path */
+ private def createLogDir() {
+ val dir = new File(logDir)
+ if (dir.exists) {
+ logWarning("Logging directory already exists: " + logDir)
+ }
+ if (!dir.exists && !dir.mkdirs()) {
+ // Logger should throw a exception rather than continue to construct this object
+ throw new IOException("Error in creating log directory:" + logDir)
+ }
+ }
+
+ /**
+ * Create a new writer for the file identified by the given path.
+ *
+ * File systems currently supported include HDFS, S3, and the local file system.
+ * The Hadoop LocalFileSystem (r1.0.4) has known issues with syncing (HADOOP-7844).
+ * Therefore, for local files, use FileOutputStream instead.
+ */
+ private def createWriter(): Option[PrintWriter] = {
+ val logPath = logDir + "/" + fileIndex
+ val uri = new URI(logPath)
+
+ val dataStream = uri.getScheme match {
+ case "hdfs" | "s3" =>
+ val fs = hadoopFileSystem.getOrElse {
+ val conf = SparkHadoopUtil.get.newConfiguration()
+ hadoopFileSystem = Some(FileSystem.get(uri, conf))
+ hadoopFileSystem.get
+ }
+ val path = new Path(logPath)
+ hadoopDataStream = Some(fs.create(path, overwriteExistingFiles))
+ hadoopDataStream.get
+
+ case "file" | null =>
+ // Second parameter is whether to append
+ new FileOutputStream(logPath, !overwriteExistingFiles)
+
+ case unsupportedScheme =>
+ throw new UnsupportedOperationException("File system scheme %s is not supported!"
+ .format(unsupportedScheme))
+ }
+
+ val bufferedStream = new BufferedOutputStream(dataStream)
--- End diff --
Here it would probably be good to make the size of the buffer configurable. And maybe default to 100 KB or something. I think the default is around 8KB for this stream.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37765872
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37765873
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13203/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10674139
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1020,6 +1042,24 @@ class SparkContext(
/** Register a new RDD, returning its RDD ID */
private[spark] def newRddId(): Int = nextRddId.getAndIncrement()
+ /** Post the environment update event if the listener bus is ready */
+ private def postEnvironmentUpdate() {
+ Option(listenerBus).foreach { bus =>
--- End diff --
addJar() calls this, which happens before the spark listener bus is initialized.
Another way is to initialize the spark listener bus super early (before add jars). I don't prefer this, however, because it increases the gap between the initializations of the bus and the UI.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10186902
--- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import java.io._
+import java.text.SimpleDateFormat
+import java.util.Date
+
+import org.apache.spark.Logging
+
+/**
+ * A generic class for logging information to file
+ * @param logDir Path to the directory in which files are logged
+ * @param name An identifier of each FileLogger instance
+ */
+class FileLogger(
+ logDir: String = Option(System.getenv("SPARK_LOG_DIR"))
--- End diff --
I think we want this to be a sparkConf value called `spark.eventLog.dir` or something similar.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37012309
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10360822
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+import org.apache.spark.storage._
+
+private[ui] trait UISparkListener extends SparkListener
--- End diff --
Do you need this intermediate trait?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10580856
--- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+import org.apache.spark.util.{JsonProtocol, FileLogger}
+
+/**
+ * A SparkListener that logs events to persistent storage.
+ *
+ * Event logging is specified by the following configurable parameters:
+ * spark.eventLog.enabled - Whether event logging is enabled.
+ * spark.eventLog.compress - Whether to compress logged events
+ * spark.eventLog.overwrite - Whether to overwrite any existing files.
+ * spark.eventLog.dir - Path to the directory in which events are logged.
+ * spark.eventLog.buffer.kb - Buffer size to use when writing to output streams
+ */
+private[spark] class EventLoggingListener(appName: String, conf: SparkConf)
+ extends SparkListener with Logging {
+
+ private val shouldCompress = conf.getBoolean("spark.eventLog.compress", false)
+ private val shouldOverwrite = conf.getBoolean("spark.eventLog.overwrite", true)
--- End diff --
Sounds good. (Though the chance of a file name collision is actually quite low, since we also attach a timestamp to it, which is why I didn't expose this config to the user)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37776148
Hey @andrewor14 I took another pass on this. Things are looking good but a few notes in the code. In a few places it seemed like `Option` was used in a way that causes what are really runtime failures to be ignored. It's better in these cases to actually cause an exception in less there is a legitimate reason the option could be `None`. Otherwise if there are failures they are really hard to understand/debug.
There are a couple of things in here that I was a bit confused by and we can discuss offline. They were the special case for the BlockManager listener (I still wonder if we can come up with a better solution than is currently there) and the use of redirection for static code.
In general though this is looking good and I think it's close to being ready to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36590955
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10272892
--- Diff: docs/configuration.md ---
@@ -444,7 +444,21 @@ Apart from these, the following properties are also available, and may be useful
<td>spark.logConf</td>
<td>false</td>
<td>
- Log the supplied SparkConf as INFO at start of spark context.
+ Whether to log the supplied SparkConf as INFO at start of spark context.
+ </td>
+</tr>
+<tr>
+ <td>spark.eventLog.enabled</td>
+ <td>false</td>
+ <td>
+ Whether to log spark events, useful for reconstructing the Web UI after the application has finished.
+ </td>
+</tr>
+<tr>
+ <td>spark.eventLog.dir</td>
+ <td>/tmp/spark-events</td>
+ <td>
+ Directory in which spark events are logged, if <code>spark.eventLog.enabled</code> is true.
--- End diff --
It might be good to clarify that each application will automatically create a sub-directory within this that contains the event log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36315930
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12926/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10189124
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala ---
@@ -17,21 +17,23 @@
package org.apache.spark.executor
+import org.apache.spark.storage.{BlockId, BlockStatus}
+
class TaskMetrics extends Serializable {
/**
- * Host's name the task runs on
+ * Host's name the task runs on
*/
var hostname: String = _
/**
* Time taken on the executor to deserialize this task
*/
- var executorDeserializeTime: Int = _
+ var executorDeserializeTime: Long = _
--- End diff --
It's more consistent with the rest of the code - variables denoting time and number of bytes in other places use Long. Also, the way these variables are set is through System.currentTimeMillis() anyway, and then arbitrarily cast to Int.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10175006
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala ---
@@ -339,8 +217,8 @@ class JobLogger(val user: String, val logDirName: String)
* @param jobEnd Job end event
*/
override def onJobEnd(jobEnd: SparkListenerJobEnd) {
- val job = jobEnd.job
- var info = "JOB_ID=" + job.jobId
+ val jobID = jobEnd.jobId
--- End diff --
nit: camel case here? (jobId)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37057138
Maybe try rebasing on master? It looks like the same error has appeared on other PRs: https://github.com/apache/spark/pull/85 in the past. It's also possible it's just a flakey test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37267126
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37374813
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38101979
Woohoo!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10237907
--- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala ---
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import java.io._
+import java.text.SimpleDateFormat
+import java.net.URI
+import java.util.Date
+
+import org.apache.spark.Logging
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+/**
+ * A generic class for logging information to file
+ * @param logBaseDir Path to the directory in which files are logged
+ * @param name An identifier of each FileLogger instance
+ * @param overwriteExistingFiles Whether to overwrite existing files
+ */
+class FileLogger(
+ logBaseDir: String,
+ name: String = String.valueOf(System.currentTimeMillis()),
--- End diff --
It might be good to prepend the folder name with something like `sc.appName.replace(" ", "_").toLowerCase`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10644885
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -472,41 +500,58 @@ private[spark] class BlockManager(
* This is currently used for writing shuffle files out. Callers should handle error
* cases.
*/
- def getDiskWriter(blockId: BlockId, file: File, serializer: Serializer, bufferSize: Int)
- : BlockObjectWriter = {
+ def getDiskWriter(
+ blockId: BlockId,
+ file: File,
+ serializer: Serializer,
+ bufferSize: Int) : BlockObjectWriter = {
--- End diff --
oops I missed this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37884927
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38009938
@pwendell I took a comprehensive pass over this PR again and I think it's ready to merge. If others have any additional feedback, now is your time...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36380718
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36400693
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12934/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36932992
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36811873
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37718878
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36815549
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10273534
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+/**
+ * Reload a persisted UI independently from a SparkContext
+ */
+object UIReloader {
--- End diff --
We should think a bit about the right way to expose the "rebuild a UI" logic inside of Spark. The first goal here is that the existing standalone cluster manager should, once an application completes, be able to re-render the existing UI for that application. The ideal semantics would be that you can just point to a directory in HDFS known to contain the logs and have it add new handlers to an existing jetty server. Right now the SparkUI starts its own server always so that might need to be altered.
A second goal is to have simple standalone version that can run in it's own JVM that basically "watches" a base directory and reports new applications as they arrive. This one would also need to create multiple SparkUI's in the same JVM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10361549
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import org.apache.spark.SparkConf
+
+/**
+ * A simple example that reloads a persisted UI independently from a SparkContext
+ */
+object UIReloader {
+ def main(args: Array[String]) {
+ if (args.length < 1) {
+ println("Usage: ./bin/spark-class org.apache.spark.ui.UIReloader [log path] [port]")
+ System.exit(1)
+ }
+
+ val port = if (args.length == 2) args(1).toInt else 14040
--- End diff --
Use the SparkUI.DEFAULT_PERSISTED_PORT variable here to make sure they stay consistent?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10457836
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class, since the UI thread and the DAGScheduler event loop may otherwise
* be reading/updating the internal data structures concurrently.
--- End diff --
Not sure... though it appears that you wrote this comment a while back. You could imagine a case in which the user refreshes the browser, which reads from these data structures, while the SparkListenerBus thread is updating them through an event, leading to inconsistent state reported on the UI.
However, other UI listeners do not synchronize on their data structures, and I haven't observed a concurrency issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504506
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentUI.scala ---
@@ -19,76 +19,74 @@ package org.apache.spark.ui.env
import javax.servlet.http.HttpServletRequest
-import scala.collection.JavaConversions._
-import scala.util.Properties
import scala.xml.Node
import org.eclipse.jetty.servlet.ServletContextHandler
-import org.apache.spark.SparkContext
+import org.apache.spark.scheduler._
+import org.apache.spark.ui._
import org.apache.spark.ui.JettyUtils._
import org.apache.spark.ui.Page.Environment
-import org.apache.spark.ui.UIUtils
-private[spark] class EnvironmentUI(sc: SparkContext) {
+private[ui] class EnvironmentUI(parent: SparkUI) {
+ private val appName = parent.appName
+ private val basePath = parent.basePath
+ private var _listener: Option[EnvironmentListener] = None
+
+ lazy val listener = _listener.get
+
+ def start() {
+ _listener = Some(new EnvironmentListener)
+ }
def getHandlers = Seq[ServletContextHandler](
createServletHandler("/environment",
- createServlet((request: HttpServletRequest) => envDetails(request), sc.env.securityManager))
+ (request: HttpServletRequest) => render(request), parent.securityManager, basePath)
)
- def envDetails(request: HttpServletRequest): Seq[Node] = {
- val jvmInformation = Seq(
- ("Java Version", "%s (%s)".format(Properties.javaVersion, Properties.javaVendor)),
- ("Java Home", Properties.javaHome),
- ("Scala Version", Properties.versionString),
- ("Scala Home", Properties.scalaHome)
- ).sorted
- def jvmRow(kv: (String, String)) = <tr><td>{kv._1}</td><td>{kv._2}</td></tr>
- def jvmTable =
- UIUtils.listingTable(Seq("Name", "Value"), jvmRow, jvmInformation, fixedWidth = true)
-
- val sparkProperties = sc.conf.getAll.sorted
-
- val systemProperties = System.getProperties.iterator.toSeq
- val classPathProperty = systemProperties.find { case (k, v) =>
- k == "java.class.path"
- }.getOrElse(("", ""))
- val otherProperties = systemProperties.filter { case (k, v) =>
- k != "java.class.path" && !k.startsWith("spark.")
- }.sorted
-
- val propertyHeaders = Seq("Name", "Value")
- def propertyRow(kv: (String, String)) = <tr><td>{kv._1}</td><td>{kv._2}</td></tr>
- val sparkPropertyTable =
- UIUtils.listingTable(propertyHeaders, propertyRow, sparkProperties, fixedWidth = true)
- val otherPropertyTable =
- UIUtils.listingTable(propertyHeaders, propertyRow, otherProperties, fixedWidth = true)
-
- val classPathEntries = classPathProperty._2
- .split(sc.conf.get("path.separator", ":"))
- .filterNot(e => e.isEmpty)
- .map(e => (e, "System Classpath"))
- val addedJars = sc.addedJars.iterator.toSeq.map{case (path, time) => (path, "Added By User")}
- val addedFiles = sc.addedFiles.iterator.toSeq.map{case (path, time) => (path, "Added By User")}
- val classPath = (addedJars ++ addedFiles ++ classPathEntries).sorted
-
- val classPathHeaders = Seq("Resource", "Source")
- def classPathRow(data: (String, String)) = <tr><td>{data._1}</td><td>{data._2}</td></tr>
- val classPathTable =
- UIUtils.listingTable(classPathHeaders, classPathRow, classPath, fixedWidth = true)
-
--- End diff --
Moved to SparkEnv.scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10685165
--- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.LinkedBlockingQueue
+
+import org.apache.spark.Logging
+
+/** Asynchronously passes SparkListenerEvents to registered SparkListeners. */
+private[spark] class LiveListenerBus extends SparkListenerBus with Logging {
+
+ /* Cap the capacity of the SparkListenerEvent queue so we get an explicit error (rather than
+ * an OOM exception) if it's perpetually being added to more quickly than it's being drained. */
+ private val EVENT_QUEUE_CAPACITY = 10000
+ private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent](EVENT_QUEUE_CAPACITY)
+ private var queueFullErrorMessageLogged = false
+
+ /**
+ * Create a new daemon thread to listen for events. Until this thread has started, all posted
--- End diff --
The scaladoc here might be better focusing on the external interface for users rather than the fact that it launches a thread. For instance we could refactor this some day to not launch a separate thread and we'd still preserve the semantics of `start`. So maybe remove the first sentence.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10272195
--- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import java.io._
+import java.text.SimpleDateFormat
+import java.net.URI
+import java.util.Date
+
+import org.apache.spark.Logging
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.hadoop.fs.{FSDataOutputStream, Path, FileSystem}
+
+/**
+ * A generic class for logging information to file.
+ *
+ * @param logBaseDir Path to the directory in which files are logged
+ * @param name An identifier of each FileLogger instance
+ * @param overwriteExistingFiles Whether to overwrite existing files
+ */
+class FileLogger(
+ logBaseDir: String,
+ name: String = String.valueOf(System.currentTimeMillis()),
+ overwriteExistingFiles: Boolean = true)
+ extends Logging {
+
+ private val logDir = logBaseDir.stripSuffix("/") + "/" + name
+ private val DATE_FORMAT = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss")
+ private var fileIndex = 0
+
+ // Only defined if the file system scheme uses the Hadoop API
+ private var hadoopDataStream: Option[FSDataOutputStream] = None
+ private var hadoopFileSystem: Option[FileSystem] = None
+
+ private var writer: Option[PrintWriter] = {
+ createLogDir()
+ createWriter()
+ }
+
+ /** Create a logging directory with the given path */
+ private def createLogDir() {
+ val dir = new File(logDir)
+ if (dir.exists) {
+ logWarning("Logging directory already exists: " + logDir)
+ }
+ if (!dir.exists && !dir.mkdirs()) {
+ // Logger should throw a exception rather than continue to construct this object
+ throw new IOException("Error in creating log directory:" + logDir)
+ }
+ }
+
+ /**
+ * Create a new writer for the file identified by the given path.
+ *
+ * File systems currently supported include HDFS, S3, and the local file system.
+ * The Hadoop LocalFileSystem (r1.0.4) has known issues with syncing (HADOOP-7844).
+ * Therefore, for local files, use FileOutputStream instead.
+ */
+ private def createWriter(): Option[PrintWriter] = {
+ val logPath = logDir + "/" + fileIndex
+ val uri = new URI(logPath)
+
+ val dataStream = uri.getScheme match {
+ case "hdfs" | "s3" =>
+ val fs = hadoopFileSystem.getOrElse {
+ val conf = SparkHadoopUtil.get.newConfiguration()
+ hadoopFileSystem = Some(FileSystem.get(uri, conf))
+ hadoopFileSystem.get
+ }
+ val path = new Path(logPath)
+ hadoopDataStream = Some(fs.create(path, overwriteExistingFiles))
+ hadoopDataStream.get
+
+ case "file" | null =>
+ // Second parameter is whether to append
+ new FileOutputStream(logPath, !overwriteExistingFiles)
+
+ case unsupportedScheme =>
+ throw new UnsupportedOperationException("File system scheme %s is not supported!"
+ .format(unsupportedScheme))
+ }
+
+ val bufferedStream = new BufferedOutputStream(dataStream)
--- End diff --
I'd also add the option of wrapping this stream for compression.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37602838
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36313839
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597888
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala ---
@@ -80,187 +81,78 @@ class JobLogger(val user: String, val logDirName: String)
/**
* Create a log file for one job
- * @param jobID ID of the job
+ * @param jobId ID of the job
* @throws FileNotFoundException Fail to create log file
*/
- protected def createLogWriter(jobID: Int) {
+ protected def createLogWriter(jobId: Int) {
try {
- val fileWriter = new PrintWriter(logDir + "/" + logDirName + "/" + jobID)
- jobIDToPrintWriter += (jobID -> fileWriter)
+ val fileWriter = new PrintWriter(logDir + "/" + logDirName + "/" + jobId)
+ jobIdToPrintWriter += (jobId -> fileWriter)
} catch {
case e: FileNotFoundException => e.printStackTrace()
}
}
/**
- * Close log file, and clean the stage relationship in stageIDToJobID
- * @param jobID ID of the job
+ * Close log file, and clean the stage relationship in stageIdToJobId
+ * @param jobId ID of the job
*/
- protected def closeLogWriter(jobID: Int) {
- jobIDToPrintWriter.get(jobID).foreach { fileWriter =>
+ protected def closeLogWriter(jobId: Int) {
+ jobIdToPrintWriter.get(jobId).foreach { fileWriter =>
fileWriter.close()
- jobIDToStages.get(jobID).foreach(_.foreach{ stage =>
- stageIDToJobID -= stage.id
+ jobIdToStageIds.get(jobId).foreach(_.foreach { stageId =>
+ stageIdToJobId -= stageId
})
- jobIDToPrintWriter -= jobID
- jobIDToStages -= jobID
+ jobIdToPrintWriter -= jobId
+ jobIdToStageIds -= jobId
}
}
/**
+ * Build up the maps that represent stage-job relationships
+ * @param jobId ID of the job
+ * @param stageIds IDs of the associated stages
+ */
+ protected def buildJobStageDependencies(jobId: Int, stageIds: Seq[Int]) = {
+ jobIdToStageIds(jobId) = stageIds
+ stageIds.foreach { stageId => stageIdToJobId(stageId) = jobId }
+ }
+
+ /**
* Write info into log file
- * @param jobID ID of the job
+ * @param jobId ID of the job
* @param info Info to be recorded
* @param withTime Controls whether to record time stamp before the info, default is true
*/
- protected def jobLogInfo(jobID: Int, info: String, withTime: Boolean = true) {
+ protected def jobLogInfo(jobId: Int, info: String, withTime: Boolean = true) {
var writeInfo = info
if (withTime) {
val date = new Date(System.currentTimeMillis())
writeInfo = DATE_FORMAT.format(date) + ": " + info
}
- jobIDToPrintWriter.get(jobID).foreach(_.println(writeInfo))
+ jobIdToPrintWriter.get(jobId).foreach(_.println(writeInfo))
}
/**
* Write info into log file
- * @param stageID ID of the stage
+ * @param stageId ID of the stage
* @param info Info to be recorded
* @param withTime Controls whether to record time stamp before the info, default is true
*/
- protected def stageLogInfo(stageID: Int, info: String, withTime: Boolean = true) {
--- End diff --
Just to make sure - the main stuff we are removing here is the print out of the entire dependency graph... right? Or is there other stuff? I just want to make sure there's nothing more we should be exposing to the listeners that we aren't.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36489262
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12963/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38012549
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13254/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10641797
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -69,32 +70,47 @@ private[spark] class CacheManager(blockManager: BlockManager) extends Logging {
// If we got here, we have to load the split
logInfo("Partition %s not found, computing it".format(key))
val computedValues = rdd.computeOrReadCheckpoint(split, context)
+
// Persist the result, so long as the task is not running locally
if (context.runningLocally) { return computedValues }
- if (storageLevel.useDisk && !storageLevel.useMemory) {
- // In the case that this RDD is to be persisted using DISK_ONLY
- // the iterator will be passed directly to the blockManager (rather then
- // caching it to an ArrayBuffer first), then the resulting block data iterator
- // will be passed back to the user. If the iterator generates a lot of data,
- // this means that it doesn't all have to be held in memory at one time.
- // This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
- // blocks aren't dropped by the block store before enabling that.
- blockManager.put(key, computedValues, storageLevel, tellMaster = true)
- return blockManager.get(key) match {
- case Some(values) =>
- return new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
- case None =>
- logInfo("Failure to store %s".format(key))
- throw new Exception("Block manager failed to return persisted valued")
+
+ // Keep track of blocks with updated statuses
+ var updatedBlocks = Seq[(BlockId, BlockStatus)]()
+ val returnValue: Iterator[T] = {
+ if (storageLevel.useDisk && !storageLevel.useMemory) {
+ /* In the case that this RDD is to be persisted using DISK_ONLY
+ * the iterator will be passed directly to the blockManager (rather then
+ * caching it to an ArrayBuffer first), then the resulting block data iterator
+ * will be passed back to the user. If the iterator generates a lot of data,
+ * this means that it doesn't all have to be held in memory at one time.
+ * This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
+ * blocks aren't dropped by the block store before enabling that. */
+ updatedBlocks = blockManager.put(key, computedValues, storageLevel, tellMaster = true)
+ blockManager.get(key) match {
+ case Some(values) =>
+ new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
+ case None =>
+ logInfo("Failure to store %s".format(key))
+ throw new Exception("Block manager failed to return persisted valued")
+ }
+ } else {
+ // In this case the RDD is cached to an array buffer. This will save the results
+ // if we're dealing with a 'one-time' iterator
+ val elements = new ArrayBuffer[Any]
+ elements ++= computedValues
+ updatedBlocks = blockManager.put(key, elements, storageLevel, tellMaster = true)
+ elements.iterator.asInstanceOf[Iterator[T]]
}
- } else {
- // In this case the RDD is cached to an array buffer. This will save the results
- // if we're dealing with a 'one-time' iterator
- val elements = new ArrayBuffer[Any]
- elements ++= computedValues
- blockManager.put(key, elements, storageLevel, tellMaster = true)
- return elements.iterator.asInstanceOf[Iterator[T]]
}
+
+ // Update task metrics to include any blocks whose storage status is updated
+ val metrics = context.taskMetrics
--- End diff --
Could also write this as:
```
Option(context.taskMetrics).foreach{ metrics =>
metrics.updatedBlocks = Some(updatedBlocks)
}
```
Not sure it's any nicer though :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10685117
--- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.LinkedBlockingQueue
+
+import org.apache.spark.Logging
+
+/** Asynchronously passes SparkListenerEvents to registered SparkListeners. */
+private[spark] class LiveListenerBus extends SparkListenerBus with Logging {
--- End diff --
It might be good to say in the docs here that until you call `start()` it won't send events to attached listeners.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10184513
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala ---
@@ -17,21 +17,23 @@
package org.apache.spark.executor
+import org.apache.spark.storage.{BlockId, BlockStatus}
+
class TaskMetrics extends Serializable {
/**
- * Host's name the task runs on
+ * Host's name the task runs on
*/
var hostname: String = _
/**
* Time taken on the executor to deserialize this task
*/
- var executorDeserializeTime: Int = _
+ var executorDeserializeTime: Long = _
--- End diff --
Any particular reason to change these to Long's?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37718877
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10616539
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerStatusListener.scala ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.scheduler._
+
+/**
+ * A listener for BlockManager status updates.
+ *
+ * This listener provides a way to post executor storage status information as soon as it
+ * is available (i.e. immediately after the associated BlockManager has registered with the
+ * driver). This is necessary because the SparkContext is only notified when an executor is
+ * launched, but by then the storage information is not ready yet.
+ *
+ * Further, it is possible for a BlockManager be registered before the listener bus on the
+ * driver is initialized (e.g. the driver's own BlockManager), in which case the corresponding
+ * event should be buffered.
+ */
+private[spark] class BlockManagerStatusListener extends SparkListener {
--- End diff --
This would require creating the event bus and the scheduler before SparkEnv, since SparkEnv creates all the block manager classes. This is not possible, however, since the DAGScheduler depends on the block manager and the map output tracker.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37372656
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10273889
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+/**
+ * Reload a persisted UI independently from a SparkContext
+ */
+object UIReloader {
--- End diff --
One idea is the SparkUI could just take an existing jetty server along with a content path "/X/Y/" and add handlers to it instead of building it's own server. This way you could have a parent server with multiple UI's bound to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/42
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36383623
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12932/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36417618
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36398304
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36313766
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642848
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -69,32 +70,47 @@ private[spark] class CacheManager(blockManager: BlockManager) extends Logging {
// If we got here, we have to load the split
logInfo("Partition %s not found, computing it".format(key))
val computedValues = rdd.computeOrReadCheckpoint(split, context)
+
// Persist the result, so long as the task is not running locally
if (context.runningLocally) { return computedValues }
- if (storageLevel.useDisk && !storageLevel.useMemory) {
- // In the case that this RDD is to be persisted using DISK_ONLY
- // the iterator will be passed directly to the blockManager (rather then
- // caching it to an ArrayBuffer first), then the resulting block data iterator
- // will be passed back to the user. If the iterator generates a lot of data,
- // this means that it doesn't all have to be held in memory at one time.
- // This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
- // blocks aren't dropped by the block store before enabling that.
- blockManager.put(key, computedValues, storageLevel, tellMaster = true)
- return blockManager.get(key) match {
- case Some(values) =>
- return new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
- case None =>
- logInfo("Failure to store %s".format(key))
- throw new Exception("Block manager failed to return persisted valued")
+
+ // Keep track of blocks with updated statuses
+ var updatedBlocks = Seq[(BlockId, BlockStatus)]()
+ val returnValue: Iterator[T] = {
+ if (storageLevel.useDisk && !storageLevel.useMemory) {
+ /* In the case that this RDD is to be persisted using DISK_ONLY
+ * the iterator will be passed directly to the blockManager (rather then
+ * caching it to an ArrayBuffer first), then the resulting block data iterator
+ * will be passed back to the user. If the iterator generates a lot of data,
+ * this means that it doesn't all have to be held in memory at one time.
+ * This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
+ * blocks aren't dropped by the block store before enabling that. */
+ updatedBlocks = blockManager.put(key, computedValues, storageLevel, tellMaster = true)
+ blockManager.get(key) match {
+ case Some(values) =>
+ new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
+ case None =>
+ logInfo("Failure to store %s".format(key))
+ throw new Exception("Block manager failed to return persisted valued")
+ }
+ } else {
+ // In this case the RDD is cached to an array buffer. This will save the results
+ // if we're dealing with a 'one-time' iterator
+ val elements = new ArrayBuffer[Any]
+ elements ++= computedValues
+ updatedBlocks = blockManager.put(key, elements, storageLevel, tellMaster = true)
+ elements.iterator.asInstanceOf[Iterator[T]]
}
- } else {
- // In this case the RDD is cached to an array buffer. This will save the results
- // if we're dealing with a 'one-time' iterator
- val elements = new ArrayBuffer[Any]
- elements ++= computedValues
- blockManager.put(key, elements, storageLevel, tellMaster = true)
- return elements.iterator.asInstanceOf[Iterator[T]]
}
+
+ // Update task metrics to include any blocks whose storage status is updated
+ val metrics = context.taskMetrics
--- End diff --
Ya ideally that would be better - otherwise we risk masking what is an actual failure during a legitimate execution. If you can just clean up the tests that would be great. If they are creating `TaskContext` objects we should include an empty metrics inside of them
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10368525
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class, since the UI thread and the DAGScheduler event loop may otherwise
* be reading/updating the internal data structures concurrently.
*/
-private[spark] class JobProgressListener(val sc: SparkContext) extends SparkListener {
+private[ui] class JobProgressListener(sc: SparkContext, live: Boolean)
+ extends StorageStatusSparkListener {
--- End diff --
Makes sense. The fact that storage status list isn't updated is a bug; I forgot to add super.onTaskEnd to JobProgressListener. However, this will no longer be relevant in the new proposal, which involves JobProgressListener extending the usual SparkListener as you suggest, and listening for the trimmed ExecutorStateChange events. (More details in discussion above)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10581141
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -895,4 +894,42 @@ private[spark] object Utils extends Logging {
}
count
}
+
+ /** Return the class name of the given object, removing all dollar signs */
+ def getFormattedClassName(obj: AnyRef) = {
+ obj.getClass.getSimpleName.replace("$", "")
+ }
+
+ def jsonOption(json: JValue): Option[JValue] = {
+ json match {
+ case JNothing => None
+ case value: JValue => Some(value)
+ }
+ }
+
+ def emptyJson = JObject(List[JField]())
+
+ /**
+ * Return a Hadoop FileSystem with the scheme encoded in the given path.
+ * File systems currently supported include HDFS, S3, and the local file system.
+ */
+ def getHadoopFileSystem(path: URI): FileSystem = {
--- End diff --
In that case I don't think we even need a helper method. I guess it is sufficiently clear if the user gets a HDFS exception (so we don't actually need to come up with our own).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10598149
--- Diff: core/src/main/scala/org/apache/spark/storage/PutResult.scala ---
@@ -20,7 +20,13 @@ package org.apache.spark.storage
import java.nio.ByteBuffer
/**
- * Result of adding a block into a BlockStore. Contains its estimated size, and possibly the
- * values put if the caller asked for them to be returned (e.g. for chaining replication)
+ * Result of adding a block into a BlockStore. This case class contains a few things:
+ * (1) The estimated size of the put,
+ * (2) The values put if the caller asked for them to be returned (e.g. for chaining
+ * replication), and
+ * (3) A list of blocks dropped as a result of this put. This is always empty for DiskStore.
*/
-private[spark] case class PutResult(size: Long, data: Either[Iterator[_], ByteBuffer])
+private[spark] case class PutResult(
+ size: Long,
+ data: Either[Iterator[_], ByteBuffer],
+ droppedBlocks: Seq[(BlockId, BlockStatus)] = Seq())
--- End diff --
It would be good to add unit tests that test the new logic you've added around these PutResults. If you look in `BlockManagerSuite` there are fairly extensive tests for the Block Manager. You should change the existing tests that do puts and cause things to flush out of memory to verify that the `droppedBlocks` gets updated. And also see if you can add new tests if those don't sufficiently cover the new code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10504565
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -106,121 +114,154 @@ private[spark] class JobProgressListener(val sc: SparkContext) extends SparkList
}
description.map(d => stageIdToDescription(stage.stageId) = d)
- val stages = poolToActiveStages.getOrElseUpdate(poolName, new HashSet[StageInfo]())
- stages += stage
+ val stages = poolToActiveStages.getOrElseUpdate(poolName, new HashMap[Int, StageInfo]())
+ stages(stage.stageId) = stage
}
override def onTaskStart(taskStart: SparkListenerTaskStart) = synchronized {
- val sid = taskStart.task.stageId
- val tasksActive = stageIdToTasksActive.getOrElseUpdate(sid, new HashSet[TaskInfo]())
- tasksActive += taskStart.taskInfo
- val taskList = stageIdToTaskInfos.getOrElse(
- sid, HashSet[(TaskInfo, Option[TaskMetrics], Option[ExceptionFailure])]())
- taskList += ((taskStart.taskInfo, None, None))
- stageIdToTaskInfos(sid) = taskList
+ val sid = taskStart.stageId
+ val taskInfo = taskStart.taskInfo
+ if (taskInfo != null) {
+ val tasksActive = stageIdToTasksActive.getOrElseUpdate(sid, new HashMap[Long, TaskInfo]())
+ tasksActive(taskInfo.taskId) = taskInfo
+ val taskMap = stageIdToTaskData.getOrElse(sid, HashMap[Long, TaskUIData]())
+ taskMap(taskInfo.taskId) = new TaskUIData(taskInfo)
+ stageIdToTaskData(sid) = taskMap
+ }
}
- override def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult)
- = synchronized {
+ override def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult) {
// Do nothing: because we don't do a deep copy of the TaskInfo, the TaskInfo in
// stageToTaskInfos already has the updated status.
}
override def onTaskEnd(taskEnd: SparkListenerTaskEnd) = synchronized {
- val sid = taskEnd.task.stageId
-
- // create executor summary map if necessary
- val executorSummaryMap = stageIdToExecutorSummaries.getOrElseUpdate(key = sid,
- op = new HashMap[String, ExecutorSummary]())
- executorSummaryMap.getOrElseUpdate(key = taskEnd.taskInfo.executorId,
- op = new ExecutorSummary())
-
- val executorSummary = executorSummaryMap.get(taskEnd.taskInfo.executorId)
- executorSummary match {
- case Some(y) => {
- // first update failed-task, succeed-task
+ val sid = taskEnd.stageId
+ val info = taskEnd.taskInfo
+
+ if (info != null) {
+ // create executor summary map if necessary
+ val executorSummaryMap = stageIdToExecutorSummaries.getOrElseUpdate(key = sid,
+ op = new HashMap[String, ExecutorSummary]())
+ executorSummaryMap.getOrElseUpdate(key = info.executorId, op = new ExecutorSummary())
+
+ val executorSummary = executorSummaryMap.get(info.executorId)
+ executorSummary match {
+ case Some(y) => {
+ // first update failed-task, succeed-task
+ taskEnd.reason match {
+ case Success =>
+ y.succeededTasks += 1
+ case _ =>
+ y.failedTasks += 1
+ }
+
+ // update duration
+ y.taskTime += info.duration
+
+ val metrics = taskEnd.taskMetrics
+ if (metrics != null) {
+ metrics.shuffleReadMetrics.foreach { y.shuffleRead += _.remoteBytesRead }
+ metrics.shuffleWriteMetrics.foreach { y.shuffleWrite += _.shuffleBytesWritten }
+ y.memoryBytesSpilled += metrics.memoryBytesSpilled
+ y.diskBytesSpilled += metrics.diskBytesSpilled
+ }
+ }
+ case _ => {}
+ }
+
+ val tasksActive = stageIdToTasksActive.getOrElseUpdate(sid, new HashMap[Long, TaskInfo]())
+ // Remove by taskId, rather than by TaskInfo, in case the TaskInfo is from storage
+ tasksActive.remove(info.taskId)
+
--- End diff --
Intended because of null check above, otherwise largely the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10685267
--- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.LinkedBlockingQueue
+
+import org.apache.spark.Logging
+
+/** Asynchronously passes SparkListenerEvents to registered SparkListeners. */
+private[spark] class LiveListenerBus extends SparkListenerBus with Logging {
+
+ /* Cap the capacity of the SparkListenerEvent queue so we get an explicit error (rather than
+ * an OOM exception) if it's perpetually being added to more quickly than it's being drained. */
+ private val EVENT_QUEUE_CAPACITY = 10000
+ private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent](EVENT_QUEUE_CAPACITY)
+ private var queueFullErrorMessageLogged = false
+
+ /**
+ * Create a new daemon thread to listen for events. Until this thread has started, all posted
--- End diff --
And I'd revers the order of the second two sentences:
`
Starts sending events to attached listeners. Note that this will first send any previously buffered events that were posted before starting.
`
Also I'd add a `var started` and throw an `IllegalStateException` if someone tries to start it twice. Otherwise you'll have two threads in that case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37884928
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10457960
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -30,16 +32,23 @@ import org.apache.spark.scheduler._
* class, since the UI thread and the DAGScheduler event loop may otherwise
* be reading/updating the internal data structures concurrently.
--- End diff --
I personally think that from the user's perspective, they're mostly concerned with aggregate statistics (min/median/max) rather than the fine-grained correctness that this level of synchronization guarantees. So I'm not sure how important this is.
That said, maybe there was a particular case regarding JobProgressListener specifically that I haven't thought of.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597086
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -68,19 +105,53 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
/** Initialize all components of the server */
def start() {
- // NOTE: This is decoupled from bind() because of the following dependency cycle:
- // DAGScheduler() requires that the port of this server is known
- // This server must register all handlers, including JobProgressUI, before binding
- // JobProgressUI registers a listener with SparkContext, which requires sc to initialize
+ storage.start()
jobs.start()
+ env.start()
exec.start()
+
+ // Listen for events from the SparkContext if it exists, otherwise from persisted storage
+ val eventBus = if (live) {
+ val loggingEnabled = conf.getBoolean("spark.eventLog.enabled", false)
--- End diff --
This feels like something that should happen in the initialization of the SparkContext (maybe right after the listener bus is created). The logging of the event bus is a slightly decoupled concept from the UI. It would also make this statement a bit more functional since right now hooking up the event logger is basically a side-effect.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37717261
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37058830
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36380717
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10642214
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -69,32 +70,47 @@ private[spark] class CacheManager(blockManager: BlockManager) extends Logging {
// If we got here, we have to load the split
logInfo("Partition %s not found, computing it".format(key))
val computedValues = rdd.computeOrReadCheckpoint(split, context)
+
// Persist the result, so long as the task is not running locally
if (context.runningLocally) { return computedValues }
- if (storageLevel.useDisk && !storageLevel.useMemory) {
- // In the case that this RDD is to be persisted using DISK_ONLY
- // the iterator will be passed directly to the blockManager (rather then
- // caching it to an ArrayBuffer first), then the resulting block data iterator
- // will be passed back to the user. If the iterator generates a lot of data,
- // this means that it doesn't all have to be held in memory at one time.
- // This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
- // blocks aren't dropped by the block store before enabling that.
- blockManager.put(key, computedValues, storageLevel, tellMaster = true)
- return blockManager.get(key) match {
- case Some(values) =>
- return new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
- case None =>
- logInfo("Failure to store %s".format(key))
- throw new Exception("Block manager failed to return persisted valued")
+
+ // Keep track of blocks with updated statuses
+ var updatedBlocks = Seq[(BlockId, BlockStatus)]()
+ val returnValue: Iterator[T] = {
+ if (storageLevel.useDisk && !storageLevel.useMemory) {
+ /* In the case that this RDD is to be persisted using DISK_ONLY
+ * the iterator will be passed directly to the blockManager (rather then
+ * caching it to an ArrayBuffer first), then the resulting block data iterator
+ * will be passed back to the user. If the iterator generates a lot of data,
+ * this means that it doesn't all have to be held in memory at one time.
+ * This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
+ * blocks aren't dropped by the block store before enabling that. */
+ updatedBlocks = blockManager.put(key, computedValues, storageLevel, tellMaster = true)
+ blockManager.get(key) match {
+ case Some(values) =>
+ new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
+ case None =>
+ logInfo("Failure to store %s".format(key))
+ throw new Exception("Block manager failed to return persisted valued")
+ }
+ } else {
+ // In this case the RDD is cached to an array buffer. This will save the results
+ // if we're dealing with a 'one-time' iterator
+ val elements = new ArrayBuffer[Any]
+ elements ++= computedValues
+ updatedBlocks = blockManager.put(key, elements, storageLevel, tellMaster = true)
+ elements.iterator.asInstanceOf[Iterator[T]]
}
- } else {
- // In this case the RDD is cached to an array buffer. This will save the results
- // if we're dealing with a 'one-time' iterator
- val elements = new ArrayBuffer[Any]
- elements ++= computedValues
- blockManager.put(key, elements, storageLevel, tellMaster = true)
- return elements.iterator.asInstanceOf[Iterator[T]]
}
+
+ // Update task metrics to include any blocks whose storage status is updated
+ val metrics = context.taskMetrics
--- End diff --
I could also create an empty (non-null) TaskMetrics in each of the relevant tests. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597059
--- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import org.apache.spark.SparkConf
+
+/**
+ * Reload a persisted UI independently from a SparkContext.
+ */
+object UIReloader {
+ def main(args: Array[String]) {
--- End diff --
It's fine if you want to leave this class but it might be worth taking it out since it could be interpreted as a public API. I think in general we'll want a standalone thing that runs in YARN deployments but this is just a small example. Also we could maybe move this to the examples directory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10186927
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+
+import org.json4s.jackson.JsonMethods._
+
+import org.apache.spark.scheduler._
+import org.apache.spark.storage._
+import org.apache.spark.util.FileLogger
+import org.apache.spark.util.JsonProtocol
+
+private[ui] trait UISparkListener extends SparkListener
+
+/**
+ * A SparkListener that serves as an entry point for all events posted to the UI.
+ *
+ * GatewayUISparkListener achieves two functions:
+ *
+ * (1) If the UI is live, GatewayUISparkListener posts each event to all attached listeners
+ * then logs it as JSON. This centralizes event logging and avoids having all attached
+ * listeners log the events on their own. By default, GatewayUISparkListener logs one
+ * file per job, though this needs not be the case.
+ *
+ * (2) If the UI is rendered from disk, GatewayUISparkListener replays each event deserialized
+ * from the event logs to all attached listeners.
+ */
+private[ui] class GatewayUISparkListener(parent: SparkUI, live: Boolean) extends SparkListener {
+
+ // Log events only if the UI is live
+ private val logger: Option[FileLogger] = if (live) Some(new FileLogger()) else None
--- End diff --
We probably want to make the use of logging configurable. For instance by introducing a configuration value called `spark.eventLog.enabled` or something similar and have the default be "false". If it's not enabled then even for live code we won't turn it on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38099756
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13271/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10596857
--- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala ---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import java.io._
+import java.net.URI
+import java.text.SimpleDateFormat
+import java.util.Date
+
+import it.unimi.dsi.fastutil.io.FastBufferedOutputStream
+import org.apache.hadoop.fs.{FSDataOutputStream, Path}
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.io.CompressionCodec
+
+/**
+ * A generic class for logging information to file.
+ *
+ * @param logDir Path to the directory in which files are logged
+ * @param outputBufferSize The buffer size to use when writing to an output stream in bytes
+ * @param compress Whether to compress output
+ * @param overwrite Whether to overwrite existing files
+ */
+class FileLogger(
+ logDir: String,
+ conf: SparkConf = new SparkConf,
+ outputBufferSize: Int = 8 * 1024,
+ compress: Boolean = false,
+ overwrite: Boolean = true)
+ extends Logging {
+
+ private val DATE_FORMAT = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss")
+ private val fileSystem = Utils.getHadoopFileSystem(new URI(logDir))
+ private var fileIndex = 0
+
+ // Only used if compression is enabled
+ private lazy val compressionCodec = CompressionCodec.createCodec(conf)
+
+ // Only defined if the file system scheme is not local
+ private var hadoopDataStream: Option[FSDataOutputStream] = None
+
+ private var writer: Option[PrintWriter] = {
+ createLogDir()
+ Some(createWriter())
+ }
+
+ /** Create a logging directory with the given path. */
+ private def createLogDir() {
+ val path = new Path(logDir)
+ if (fileSystem.exists(path)) {
+ if (overwrite) {
+ logWarning("Log directory %s already exists. Overwriting...".format(logDir))
+ // Second parameter is whether to delete recursively
+ fileSystem.delete(path, true)
+ } else {
+ throw new IOException("Log directory %s already exists!".format(logDir))
+ }
+ }
+ if (!fileSystem.mkdirs(path)) {
+ throw new IOException("Error in creating log directory: %s".format(logDir))
+ }
+ }
+
+ /**
+ * Create a new writer for the file identified by the given path. File systems currently
+ * supported include HDFS, S3, and the local file system.
+ */
+ private def createWriter(): PrintWriter = {
+ val logPath = logDir + "/" + fileIndex
+ val uri = new URI(logPath)
+
+ /**
+ * The Hadoop LocalFileSystem (r1.0.4) has known issues with syncing (HADOOP-7844).
+ * Therefore, for local files, use FileOutputStream instead.
+ */
+ val dstream = uri.getScheme match {
+ case "hdfs" | "s3" =>
+ val path = new Path(logPath)
--- End diff --
I'd support all schemes here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597200
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -68,19 +105,53 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
/** Initialize all components of the server */
def start() {
- // NOTE: This is decoupled from bind() because of the following dependency cycle:
- // DAGScheduler() requires that the port of this server is known
- // This server must register all handlers, including JobProgressUI, before binding
- // JobProgressUI registers a listener with SparkContext, which requires sc to initialize
+ storage.start()
jobs.start()
+ env.start()
exec.start()
+
+ // Listen for events from the SparkContext if it exists, otherwise from persisted storage
+ val eventBus = if (live) {
--- End diff --
I think it would be a bit nicer to have the `SparkUI` take an `EventBus` as a constructor argument rather than reasoning about it inside of the UI. This way you decoupled the UI functionality from whether or not the bus is running live (it seems like there are some places where you still have to do that, but as much as possible it might be nice to separate this)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37590183
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37883411
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10669388
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -68,19 +101,33 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
/** Initialize all components of the server */
def start() {
- // NOTE: This is decoupled from bind() because of the following dependency cycle:
- // DAGScheduler() requires that the port of this server is known
- // This server must register all handlers, including JobProgressUI, before binding
- // JobProgressUI registers a listener with SparkContext, which requires sc to initialize
+ storage.start()
jobs.start()
+ env.start()
exec.start()
+
+ // Storage status listener must receive events first, as other listeners depend on its state
+ listenerBus.addListener(storageStatusListener)
+ listenerBus.addListener(storage.listener)
+ listenerBus.addListener(jobs.listener)
+ listenerBus.addListener(env.listener)
+ listenerBus.addListener(exec.listener)
+
+ // Log events only if this UI is live and the feature is enabled
+ if (live && conf.getBoolean("spark.eventLog.enabled", false)) {
--- End diff --
That would be cleaner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10183222
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala ---
@@ -339,8 +217,8 @@ class JobLogger(val user: String, val logDirName: String)
* @param jobEnd Job end event
*/
override def onJobEnd(jobEnd: SparkListenerJobEnd) {
- val job = jobEnd.job
- var info = "JOB_ID=" + job.jobId
+ val jobID = jobEnd.jobId
--- End diff --
Yeah I realized after that comment that there seems to be inconsistency in
the way ID is named so feel free to leave it however you prefer.
On Fri, Feb 28, 2014 at 1:55 PM, andrewor14 <no...@github.com>wrote:
> In core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala:
>
> > @@ -339,8 +217,8 @@ class JobLogger(val user: String, val logDirName: String)
> > * @param jobEnd Job end event
> > */
> > override def onJobEnd(jobEnd: SparkListenerJobEnd) {
> > - val job = jobEnd.job
> > - var info = "JOB_ID=" + job.jobId
> > + val jobID = jobEnd.jobId
>
> Yeah, actually I had jobId before, but to make it consistent with the rest
> of the file I reverted it back to jobID. I'll change it back.
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/42/files#r10183037>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36821926
Looks like test failed because of git problems. Retesting later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36958962
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36706594
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10455635
--- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+import org.apache.spark.storage._
+
+private[ui] trait UISparkListener extends SparkListener
+
+/**
+ * A SparkListener that maintains executor storage status
+ */
+private[ui] class StorageStatusSparkListener extends UISparkListener {
--- End diff --
I believe the UI is the only usage of a SparkListener that builds storage status from events. If the SparkContext is alive, then you can just ask the driver for this information. So I think it should live in the UI.
As for whether to extend this class... I need to give more thought to it. One reason why it's not a good idea is because we end up maintaining two separate storage status lists with exactly the same information (because both ExecutorsUI and BlockManagerUI use this). The alternative is to keep it in SparkUI.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37372657
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36418483
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12942/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37604459
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13171/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10361374
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -625,6 +653,30 @@ private[spark] class Master(host: String, port: Int, webUiPort: Int) extends Act
}
}
+ /**
+ * Start a new SparkUI rendered from persisted storage. Assumes event logging information
+ * is available for given application.
+ */
+ def startPersistedSparkUI(app: ApplicationInfo): SparkUI = {
+ val appName = app.desc.name
+ val eventLogInfo = app.desc.eventLogInfo.get
+ val eventLogDir = eventLogInfo.logDir
--- End diff --
What happens when the SparkContext was created on a different machine than the Master?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37119592
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36958961
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37064297
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13053/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597813
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala ---
@@ -22,24 +22,25 @@ import java.text.SimpleDateFormat
import java.util.{Date, Properties}
import java.util.concurrent.LinkedBlockingQueue
-import scala.collection.mutable.{HashMap, HashSet, ListBuffer}
+import scala.collection.mutable.HashMap
import org.apache.spark._
import org.apache.spark.executor.TaskMetrics
-import org.apache.spark.rdd.RDD
-import org.apache.spark.storage.StorageLevel
/**
* A logger class to record runtime information for jobs in Spark. This class outputs one log file
- * for each Spark job, containing RDD graph, tasks start/stop, shuffle information.
- * JobLogger is a subclass of SparkListener, use addSparkListener to add JobLogger to a SparkContext
- * after the SparkContext is created.
- * Note that each JobLogger only works for one SparkContext
- * @param logDirName The base directory for the log files.
+ * for each Spark job, containing tasks start/stop and shuffle information. JobLogger is a subclass
+ * of SparkListener, use addSparkListener to add JobLogger to a SparkContext after the SparkContext
+ * is created. Note that each JobLogger only works for one SparkContext
+ *
+ * NOTE: The functionality of this class is heavily stripped down to accommodate for a general
+ * refactor of the SparkListener interface. In its place, the EventLoggingListener is introduced
+ * to log application information as SparkListenerEvents through the SparkUI. To enable this
--- End diff --
does it log them "through the SparkUI"? Could you just say:
```
In its place, the EventLoggingListener is introduced to log application information as serialized SparkListenerEvents
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37718089
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37897189
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37064296
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36980633
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-36313780
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10366355
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -978,6 +971,11 @@ class DAGScheduler(
logDebug("Additional executor lost message for " + execId +
"(epoch " + currentEpoch + ")")
}
+ // Block manager master actor should not be null except during tests
+ if (blockManagerMaster.driverActor != null) {
+ val storageStatusList = blockManagerMaster.getStorageStatus
+ listenerBus.post(SparkListenerExecutorsStateChange(storageStatusList))
--- End diff --
I have given more thought to the SparkListenerExecutorsStateChange event and have come to the following conclusions.
Having the event take in the storage status list is a little arbitrary in two ways - (1) Every ExecutorStateChange event is supposed to be triggered only by one executor, but we fetch (and log) the storage status of all executors. (2) The event doesn't really need to maintain the blocks map in StorageStatus. When new executors have joined, they start off with no blocks, so the map is empty anyway. When existing executors have failed, we don't care about their blocks any more, so the map holds little value to us.
This leads to the proposal to refactor this event in the following way - Keep around just enough arguments to reconstruct a StorageStatus on the UI side, ignoring the blocks map, which could potentially be huge and expensive to log. This means the event will now hold a (blockManagerId, maxMem) that corresponds to the given executor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-37374819
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13123/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10641806
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -69,32 +70,47 @@ private[spark] class CacheManager(blockManager: BlockManager) extends Logging {
// If we got here, we have to load the split
logInfo("Partition %s not found, computing it".format(key))
val computedValues = rdd.computeOrReadCheckpoint(split, context)
+
// Persist the result, so long as the task is not running locally
if (context.runningLocally) { return computedValues }
- if (storageLevel.useDisk && !storageLevel.useMemory) {
- // In the case that this RDD is to be persisted using DISK_ONLY
- // the iterator will be passed directly to the blockManager (rather then
- // caching it to an ArrayBuffer first), then the resulting block data iterator
- // will be passed back to the user. If the iterator generates a lot of data,
- // this means that it doesn't all have to be held in memory at one time.
- // This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
- // blocks aren't dropped by the block store before enabling that.
- blockManager.put(key, computedValues, storageLevel, tellMaster = true)
- return blockManager.get(key) match {
- case Some(values) =>
- return new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
- case None =>
- logInfo("Failure to store %s".format(key))
- throw new Exception("Block manager failed to return persisted valued")
+
+ // Keep track of blocks with updated statuses
+ var updatedBlocks = Seq[(BlockId, BlockStatus)]()
+ val returnValue: Iterator[T] = {
+ if (storageLevel.useDisk && !storageLevel.useMemory) {
+ /* In the case that this RDD is to be persisted using DISK_ONLY
+ * the iterator will be passed directly to the blockManager (rather then
+ * caching it to an ArrayBuffer first), then the resulting block data iterator
+ * will be passed back to the user. If the iterator generates a lot of data,
+ * this means that it doesn't all have to be held in memory at one time.
+ * This could also apply to MEMORY_ONLY_SER storage, but we need to make sure
+ * blocks aren't dropped by the block store before enabling that. */
+ updatedBlocks = blockManager.put(key, computedValues, storageLevel, tellMaster = true)
+ blockManager.get(key) match {
+ case Some(values) =>
+ new InterruptibleIterator(context, values.asInstanceOf[Iterator[T]])
+ case None =>
+ logInfo("Failure to store %s".format(key))
+ throw new Exception("Block manager failed to return persisted valued")
+ }
+ } else {
+ // In this case the RDD is cached to an array buffer. This will save the results
+ // if we're dealing with a 'one-time' iterator
+ val elements = new ArrayBuffer[Any]
+ elements ++= computedValues
+ updatedBlocks = blockManager.put(key, elements, storageLevel, tellMaster = true)
+ elements.iterator.asInstanceOf[Iterator[T]]
}
- } else {
- // In this case the RDD is cached to an array buffer. This will save the results
- // if we're dealing with a 'one-time' iterator
- val elements = new ArrayBuffer[Any]
- elements ++= computedValues
- blockManager.put(key, elements, storageLevel, tellMaster = true)
- return elements.iterator.asInstanceOf[Iterator[T]]
}
+
+ // Update task metrics to include any blocks whose storage status is updated
+ val metrics = context.taskMetrics
--- End diff --
Actually - is there any case where the metrics could be null here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/42#issuecomment-38012548
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10272629
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -17,38 +17,80 @@
package org.apache.spark.ui
+import java.io.{FileInputStream, File}
+
+import scala.io.Source
+
+import it.unimi.dsi.fastutil.io.FastBufferedInputStream
import org.eclipse.jetty.server.{Handler, Server}
+import org.json4s.jackson.JsonMethods._
import org.apache.spark.{Logging, SparkContext, SparkEnv}
+import org.apache.spark.scheduler._
import org.apache.spark.ui.JettyUtils._
import org.apache.spark.ui.env.EnvironmentUI
import org.apache.spark.ui.exec.ExecutorsUI
import org.apache.spark.ui.jobs.JobProgressUI
import org.apache.spark.ui.storage.BlockManagerUI
import org.apache.spark.util.Utils
+import org.apache.spark.util.JsonProtocol
-/** Top level user interface for Spark */
-private[spark] class SparkUI(sc: SparkContext) extends Logging {
+/** Top level user interface for Spark. */
+private[spark] class SparkUI(val sc: SparkContext) extends Logging {
--- End diff --
I'm just saying it would be better to keep this ugliness internal and have the public constructor take an `Option[SparkContext]`. Then you could have something like this:
```
val live = sc.isDefined
// Only valid if live
def _sc = sc.get
```
Would that work?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10360102
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -69,19 +100,55 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging {
// DAGScheduler() requires that the port of this server is known
--- End diff --
This isn't really related to your change but as long as you're changing things here, I don't think this comment is relevant anymore...or at least I don't think DAGScheduler requires that the port of this server is known? So maybe just delete that line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10357808
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -194,10 +148,46 @@ class DAGScheduler(
}
}
}))
+
+ // Start listening for block manager registration
+ blockManagerMaster.registrationListener.foreach(_.setListenerBus(listenerBus))
--- End diff --
Now that the listenerBus lives in Spark Context, can you move this line there? It seems a little unrelated to the DAGScheduler
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---