You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by andrewor14 <gi...@git.apache.org> on 2014/07/09 01:27:35 UTC

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

GitHub user andrewor14 opened a pull request:

    https://github.com/apache/spark/pull/1336

    [SPARK-2411] Add a history-not-found page to standalone Master

    Right now, if you click on an application after it has finished, it simply refreshes the page if there are no event logs for the application. This is not super intuitive especially because event logging is not enabled by default. We should direct the user to enable this if they attempt to view a SparkUI after the fact without event logs.
    
    Here is what the new page looks like:
    
    <img src="https://issues.apache.org/jira/secure/attachment/12654708/Screen%20Shot%202014-07-08%20at%204.23.51%20PM.png">

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/andrewor14/spark master-link

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/1336.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1336
    
----
commit e7df7ed945fcefcbefec990e6d4d643654c1db52
Author: Andrew Or <an...@gmail.com>
Date:   2014-07-08T23:24:31Z

    Add a history not found page to standalone Master

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-49227999
  
    I have updated the screenshot. Anything else?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48516854
  
    Jenkins test please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48666793
  
    QA tests have started for PR 1336. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16524/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48492241
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r15072986
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -667,29 +664,47 @@ private[spark] class Master(
        */
       def rebuildSparkUI(app: ApplicationInfo): Boolean = {
         val appName = app.desc.name
    -    val eventLogDir = app.desc.eventLogDir.getOrElse { return false }
    +    val eventLogDir = app.desc.eventLogDir.getOrElse {
    +      // Event logging is not enabled for this application
    +      app.desc.appUiUrl = "/history/not-found"
    +      return false
    +    }
         val fileSystem = Utils.getHadoopFileSystem(eventLogDir)
         val eventLogInfo = EventLoggingListener.parseLoggingInfo(eventLogDir, fileSystem)
         val eventLogPaths = eventLogInfo.logPaths
         val compressionCodec = eventLogInfo.compressionCodec
    -    if (!eventLogPaths.isEmpty) {
    -      try {
    -        val replayBus = new ReplayListenerBus(eventLogPaths, fileSystem, compressionCodec)
    -        val ui = new SparkUI(
    -          new SparkConf, replayBus, appName + " (completed)", "/history/" + app.id)
    -        replayBus.replay()
    -        app.desc.appUiUrl = ui.basePath
    -        appIdToUI(app.id) = ui
    -        webUi.attachSparkUI(ui)
    -        return true
    -      } catch {
    -        case e: Exception =>
    -          logError("Exception in replaying log for application %s (%s)".format(appName, app.id), e)
    -      }
    -    } else {
    -      logWarning("Application %s (%s) has no valid logs: %s".format(appName, app.id, eventLogDir))
    +
    +    if (eventLogPaths.isEmpty) {
    --- End diff --
    
    What happens here if `eventLogDir` doesn't exist?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r14837627
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/HistoryNotFoundPage.scala ---
    @@ -0,0 +1,40 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.deploy.master.ui
    +
    +import javax.servlet.http.HttpServletRequest
    +
    +import scala.xml.Node
    +
    +import org.apache.spark.ui.{UIUtils, WebUIPage}
    +
    +private[spark] class HistoryNotFoundPage(parent: MasterWebUI)
    +  extends WebUIPage("history/not-found") {
    +
    +  def render(request: HttpServletRequest): Seq[Node] = {
    +    val content =
    +      <div class="row-fluid">
    +        <div class="span12" style="font-size:14px;font-weight:bold">
    +          No event logs were found for this application. To enable event logging, please set
    --- End diff --
    
    Should you also mention the setting for the event log directory? Otherwise they might think they can just set `enabled` and it will auto-magically work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48527671
  
    Jenkins test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r15081330
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -667,29 +664,47 @@ private[spark] class Master(
        */
       def rebuildSparkUI(app: ApplicationInfo): Boolean = {
         val appName = app.desc.name
    -    val eventLogDir = app.desc.eventLogDir.getOrElse { return false }
    +    val eventLogDir = app.desc.eventLogDir.getOrElse {
    +      // Event logging is not enabled for this application
    +      app.desc.appUiUrl = "/history/not-found"
    +      return false
    +    }
         val fileSystem = Utils.getHadoopFileSystem(eventLogDir)
         val eventLogInfo = EventLoggingListener.parseLoggingInfo(eventLogDir, fileSystem)
         val eventLogPaths = eventLogInfo.logPaths
         val compressionCodec = eventLogInfo.compressionCodec
    -    if (!eventLogPaths.isEmpty) {
    -      try {
    -        val replayBus = new ReplayListenerBus(eventLogPaths, fileSystem, compressionCodec)
    -        val ui = new SparkUI(
    -          new SparkConf, replayBus, appName + " (completed)", "/history/" + app.id)
    -        replayBus.replay()
    -        app.desc.appUiUrl = ui.basePath
    -        appIdToUI(app.id) = ui
    -        webUi.attachSparkUI(ui)
    -        return true
    -      } catch {
    -        case e: Exception =>
    -          logError("Exception in replaying log for application %s (%s)".format(appName, app.id), e)
    -      }
    -    } else {
    -      logWarning("Application %s (%s) has no valid logs: %s".format(appName, app.id, eventLogDir))
    +
    +    if (eventLogPaths.isEmpty) {
    --- End diff --
    
    (If you mean the directory does not exist on the filesystem itself, then it does go into this case, and it displays view (2) in the PR message)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-49254656
  
    QA results for PR 1336:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16759/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48769071
  
    LGTM with one small comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48417491
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48492234
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48415413
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48535197
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16476/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48416653
  
    Jenkins test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-49230195
  
    Hey actually - could this differentiate between the case where the event log was not enabled (and give a message like "Application did not enable event logging") and a case where the event log was expected but could not be read (and give a message like "Error reading event log"). Bonus points if you can encode the error message in the second case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48415414
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16429/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48527742
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48574441
  
    Please don't merge this yet. @aarondav made a comment about the wording offline that I want to address.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r15024426
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/HistoryNotFoundPage.scala ---
    @@ -0,0 +1,40 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.deploy.master.ui
    +
    +import javax.servlet.http.HttpServletRequest
    +
    +import scala.xml.Node
    +
    +import org.apache.spark.ui.{UIUtils, WebUIPage}
    +
    +private[spark] class HistoryNotFoundPage(parent: MasterWebUI)
    +  extends WebUIPage("history/not-found") {
    +
    +  def render(request: HttpServletRequest): Seq[Node] = {
    +    val content =
    +      <div class="row-fluid">
    +        <div class="span12" style="font-size:14px;font-weight:bold">
    +          No event logs were found for this application. To enable event logging, please set
    --- End diff --
    
    This is for the standalone Master, not the history server. Here the event log directory for each application is shipped to the master as part of the application description, so the Master knows where to find the logs automatically.
    
    (For history server, however, the user should explicitly set the path in all cases as you describe)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48676585
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16524/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-49250081
  
    QA tests have started for PR 1336. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16759/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-49228680
  
    Looks good, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48491094
  
    Jenkins test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r14919256
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/HistoryNotFoundPage.scala ---
    @@ -0,0 +1,40 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.deploy.master.ui
    +
    +import javax.servlet.http.HttpServletRequest
    +
    +import scala.xml.Node
    +
    +import org.apache.spark.ui.{UIUtils, WebUIPage}
    +
    +private[spark] class HistoryNotFoundPage(parent: MasterWebUI)
    +  extends WebUIPage("history/not-found") {
    +
    +  def render(request: HttpServletRequest): Seq[Node] = {
    +    val content =
    +      <div class="row-fluid">
    +        <div class="span12" style="font-size:14px;font-weight:bold">
    +          No event logs were found for this application. To enable event logging, please set
    --- End diff --
    
    I mean that if the are joining an existing cluster, they'll need to figure out what HDFS or FS path to set this too such that it's consistent with the path set by the person running the history server.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r14851932
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/HistoryNotFoundPage.scala ---
    @@ -0,0 +1,40 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.deploy.master.ui
    +
    +import javax.servlet.http.HttpServletRequest
    +
    +import scala.xml.Node
    +
    +import org.apache.spark.ui.{UIUtils, WebUIPage}
    +
    +private[spark] class HistoryNotFoundPage(parent: MasterWebUI)
    +  extends WebUIPage("history/not-found") {
    +
    +  def render(request: HttpServletRequest): Seq[Node] = {
    +    val content =
    +      <div class="row-fluid">
    +        <div class="span12" style="font-size:14px;font-weight:bold">
    +          No event logs were found for this application. To enable event logging, please set
    --- End diff --
    
    Not sure what you mean, but in most environments users will have to add an HDFS directory there that is shared... right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48666447
  
    Provided the tests pass this is ready from my side.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r15081226
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
    @@ -667,29 +664,47 @@ private[spark] class Master(
        */
       def rebuildSparkUI(app: ApplicationInfo): Boolean = {
         val appName = app.desc.name
    -    val eventLogDir = app.desc.eventLogDir.getOrElse { return false }
    +    val eventLogDir = app.desc.eventLogDir.getOrElse {
    +      // Event logging is not enabled for this application
    +      app.desc.appUiUrl = "/history/not-found"
    +      return false
    +    }
         val fileSystem = Utils.getHadoopFileSystem(eventLogDir)
         val eventLogInfo = EventLoggingListener.parseLoggingInfo(eventLogDir, fileSystem)
         val eventLogPaths = eventLogInfo.logPaths
         val compressionCodec = eventLogInfo.compressionCodec
    -    if (!eventLogPaths.isEmpty) {
    -      try {
    -        val replayBus = new ReplayListenerBus(eventLogPaths, fileSystem, compressionCodec)
    -        val ui = new SparkUI(
    -          new SparkConf, replayBus, appName + " (completed)", "/history/" + app.id)
    -        replayBus.replay()
    -        app.desc.appUiUrl = ui.basePath
    -        appIdToUI(app.id) = ui
    -        webUi.attachSparkUI(ui)
    -        return true
    -      } catch {
    -        case e: Exception =>
    -          logError("Exception in replaying log for application %s (%s)".format(appName, app.id), e)
    -      }
    -    } else {
    -      logWarning("Application %s (%s) has no valid logs: %s".format(appName, app.id, eventLogDir))
    +
    +    if (eventLogPaths.isEmpty) {
    --- End diff --
    
    See a few lines up. If `eventLogDir` doesn't exist, we never even reach here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r14846862
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/HistoryNotFoundPage.scala ---
    @@ -0,0 +1,40 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.deploy.master.ui
    +
    +import javax.servlet.http.HttpServletRequest
    +
    +import scala.xml.Node
    +
    +import org.apache.spark.ui.{UIUtils, WebUIPage}
    +
    +private[spark] class HistoryNotFoundPage(parent: MasterWebUI)
    +  extends WebUIPage("history/not-found") {
    +
    +  def render(request: HttpServletRequest): Seq[Node] = {
    +    val content =
    +      <div class="row-fluid">
    +        <div class="span12" style="font-size:14px;font-weight:bold">
    +          No event logs were found for this application. To enable event logging, please set
    --- End diff --
    
    Well, it kind of does auto-magically work, but sure I'll add it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1336#discussion_r14855100
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/HistoryNotFoundPage.scala ---
    @@ -0,0 +1,40 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.deploy.master.ui
    +
    +import javax.servlet.http.HttpServletRequest
    +
    +import scala.xml.Node
    +
    +import org.apache.spark.ui.{UIUtils, WebUIPage}
    +
    +private[spark] class HistoryNotFoundPage(parent: MasterWebUI)
    +  extends WebUIPage("history/not-found") {
    +
    +  def render(request: HttpServletRequest): Seq[Node] = {
    +    val content =
    +      <div class="row-fluid">
    +        <div class="span12" style="font-size:14px;font-weight:bold">
    +          No event logs were found for this application. To enable event logging, please set
    --- End diff --
    
    Right now the code actually creates the directory for you


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-49336964
  
    I have updated the screenshots again. Anything else?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48535196
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48417492
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16433/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48508591
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16466/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48416911
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48412700
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48676561
  
    QA results for PR 1336:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16524/consoleFull


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/1336


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48412687
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-49390208
  
    Okay cool - I'm merging tihs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48416901
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48527729
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1336#issuecomment-48508587
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---