You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sundeepn <gi...@git.apache.org> on 2014/03/27 02:38:51 UTC
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
GitHub user sundeepn opened a pull request:
https://github.com/apache/spark/pull/246
SPARK-1202 - Add a "cancel" button in the UI for stages
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sundeepn/spark uikilljob
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/246.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #246
----
commit f8b1c4ce654ec1e60a6c684a18c8c69ffd535b47
Author: Sundeep Narravula <su...@dhcpx-204-110.corp.yahoo.com>
Date: 2014-03-27T00:04:55Z
Expose canceljob by job ID.
Author: Sundeep Narravula <ns...@yahoo.com>
commit 988faf1852efe45c1343093fc2163659ce6485c5
Author: Sundeep Narravula <su...@dhcpx-204-110.corp.yahoo.com>
Date: 2014-03-27T00:09:53Z
Ability to kill jobs thru the UI.
This behavior can be turned on be settings the following variable: spark.ui.killEnabled=true (default=false)
Author: Sundeep Narravula <ns...@yahoo.com>
commit a30d15d78b52312e43a3db1a58608a29fee39322
Author: Sundeep Narravula <su...@dhcpx-204-110.corp.yahoo.com>
Date: 2014-03-27T01:34:52Z
Formatting fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40158944
Merged into master and 1.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39012183
> what happens for stages that are used for multiple jobs?
I misunderstood our conversation on job to stage mapping the other day. As you see, the code will currently not handle multiple job mappings. Is there a simple example I can use to generate such a scenario?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11189917
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1028,6 +1028,16 @@ class SparkContext(
dagScheduler.cancelAllJobs()
}
+ /** Cancel a given job if it's scheduled or running */
+ def cancelJob(jobId: Int) {
--- End diff --
I think these should be private[spark]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40134385
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14012/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39364146
I was referring to this part of the code: https://github.com/apache/spark/pull/246/files#diff-1f32bcb61f51133bd0959a4177a066a5R230
I don't think this works properly if a stage is used by multiple jobs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39011464
It looks like github is just moving slowly today...the commit just got pulled in. I took another look at this and have a question: what happens for stages that are used for multiple jobs? Right now, stageIdToJobId in the UI code you added just maps a stage to a single job id. So, if stage0 is used by JobA and jobB, the ui code only stores one of these jobs, and then cancelJob() will only be called for one of the jobs. cancelJob() ultimately calls DAGScheduler.handleJobCancellation(), which only cancels the stages that are independent to the job. So, because stage0 is not independent to either of the jobs, it won't get cancelled. Did I misunderstand this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40148964
Yes. It is a format issue. Updated the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39011323
Build triggered. Build is starting -or- tests failed to complete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39404658
Added #309 to deal with telling the UI about finished stages. This should allow you to simplify your patch by removing the code you added in JobProgressListener.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40152575
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14021/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011282
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1028,6 +1028,11 @@ class SparkContext(
dagScheduler.cancelAllJobs()
}
+ /** Cancel a given job if its scheduled or running*/
--- End diff --
Space after "running"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11037021
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -226,8 +239,17 @@ private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
}
case _ =>
}
+ // Clear out tracking for jobs that have ended
+ jobIdToStageIds.remove(jobEnd.jobId)
}
+ override def onJobStart(jobStart: SparkListenerJobStart) = synchronized {
+ // Track job to stage ID mapping to allow users to kill jobs
+ // Cleanup as jobs/stages are completed
+ jobIdToStageIds + (jobStart.jobId -> jobStart.stageIds)
+ jobStart.stageIds.foreach(id => stageIdToJobId + (id -> jobStart.jobId))
--- End diff --
You are correct. I will fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39159181
Build triggered. Build is starting -or- tests failed to complete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39404791
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13710/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40150594
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11425473
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1028,6 +1028,16 @@ class SparkContext(
dagScheduler.cancelAllJobs()
}
+ /** Cancel a given job if it's scheduled or running */
+ def cancelJob(jobId: Int) {
--- End diff --
I'd like to make these private for now because a stage id and job id is not a concept we expose directly to users in the main API at present. I the main cancellation API we have at the moment is to use the future actions and then call cancel on the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-38843794
Currently, only the IndexPage links to the killPage. The KillPage just has the uniform header on top so it links back to the other pages.
The unit of killing is a job by requirement. However, the initiation of kill action is through any stage contained in the job. The KillPage is actually a Job level message page, even though we arrive there by a stageId. Maybe I should change the title to "Job x containing stage y killed" ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39043810
I'm not sure how to generate an example of this...I think it can happen with Shark but maybe @pwendell or @markhamstra can comment here?
I wonder if maybe a better way to handle this would be to add a cancelStage() method to the DAGScheduler, which already has the mappings of stages to jobs and vice versa. That way the UI wouldn't have to duplicate this information.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11423117
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -224,8 +225,31 @@ private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
failedStages += s
trimIfNecessary(failedStages)
}
+
+ // Clean up other stages associated with this job; typically when job is cancelled
+ jobIdToStageIds.get(jobEnd.jobId).getOrElse(Seq[Int]()).foreach { sId =>
--- End diff --
I don't think this code is necessary anymore, now that we explicitly tell the listeners when stages end
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40134382
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40152503
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11426078
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -31,11 +31,21 @@ private[ui] class StagePage(parent: JobProgressUI) {
private val appName = parent.appName
private val basePath = parent.basePath
private lazy val listener = parent.listener
+ private lazy val sc = parent.sc
+ private val killEnabled = parent.killEnabled
def render(request: HttpServletRequest): Seq[Node] = {
listener.synchronized {
val stageId = request.getParameter("id").toInt
+ if (killEnabled) {
+ val killFlag = Option(request.getParameter("terminate")).getOrElse("false").toBoolean
--- End diff --
Could we just re-direct to the index page here? I think that would be more natural after killing a stage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011334
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -18,12 +18,12 @@
package org.apache.spark.ui.jobs
import scala.collection.mutable.{HashMap, ListBuffer}
-
import org.apache.spark.{ExceptionFailure, SparkConf, SparkContext, Success}
import org.apache.spark.executor.TaskMetrics
import org.apache.spark.scheduler._
import org.apache.spark.scheduler.SchedulingMode.SchedulingMode
import org.apache.spark.storage.BlockManagerId
+import org.apache.spark.Logging
--- End diff --
nit: merge this with `org.apache.spark.{...}`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39159190
Build started. Build is starting -or- tests failed to complete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39011326
Build started. Build is starting -or- tests failed to complete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39126865
Yes, right now the landing page of the UI is the stage page, which is a little arbitrary. I think it makes sense to have an overview page, that displays the number of running executors, master URL, total duration, event log location. (@tdas and I discussed this a bunch in designing for a brother UI for Spark Streaming.)
On this overview page, maybe we can also add a Job summary table, in which each row links to a job-specific stages page (what we already have). Then, in the context of canceling jobs, we can have the `cancel` button at the end of each Job row.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/246
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11424350
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala ---
@@ -56,5 +57,5 @@ private[ui] class JobProgressUI(parent: SparkUI) {
(request: HttpServletRequest) => poolPage.render(request), parent.securityManager, basePath),
createServletHandler("/stages",
(request: HttpServletRequest) => indexPage.render(request), parent.securityManager, basePath)
- )
+ )
--- End diff --
unnecessary added space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011447
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala ---
@@ -71,15 +71,28 @@ private[ui] class StageTable(stages: Seq[StageInfo], parent: JobProgressUI) {
</div>
}
- /** Render an HTML row that represents a stage */
- private def stageRow(s: StageInfo): Seq[Node] = {
- val poolName = listener.stageIdToPool.get(s.stageId)
+ private def makeDescription(s: StageInfo): Seq[Node]=
+ {
--- End diff --
Formatting is weird here... make sure you leave a space before `=`. Also you don't need to break the line here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011317
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -103,6 +105,7 @@ private[spark] class SparkUI(
listenerBus.addListener(jobs.listener)
listenerBus.addListener(env.listener)
listenerBus.addListener(exec.listener)
+
--- End diff --
nit: remove extra new line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39931895
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39933343
Just a few small comments -- other than those this looks good
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39159316
Build is starting -or- tests failed to complete.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13615/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40050758
Jenkins is listing some style errors - mind fixing them?
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13987/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39128899
Thanks. Just what I was going to suggest. I will clean up the current cancel stage/job functionality and complete this pull. It will be good to have a different pull with a clear redesign of the UI.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39905833
PRs #305 and #309 have now been merged so this should be good to go as soon as you rebase on the latest code!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40045810
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11095628
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -116,6 +118,16 @@ private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
val stages = poolToActiveStages.getOrElseUpdate(poolName, new HashMap[Int, StageInfo]())
stages(stage.stageId) = stage
+
+ // Extract Job ID and double check if we have the details
+ val jobId = Option(stageSubmitted.properties).flatMap {
+ p => Option(p.getProperty("spark.job.id"))
+ }.getOrElse("-1").toInt
--- End diff --
Well, this is only to ensure we can handle things if we get any scenarios where the onJobStart does not arrive before stageSubmitted. I am not familiar with the scheduling code sufficiently to rule that out. If you are sure, I can take this out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39931975
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13936/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40158483
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39044622
You can look at Matei's comments from way back: https://github.com/mesos/spark/pull/414
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39404610
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11424258
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1028,6 +1028,16 @@ class SparkContext(
dagScheduler.cancelAllJobs()
}
+ /** Cancel a given job if it's scheduled or running */
+ def cancelJob(jobId: Int) {
--- End diff --
Did you see this comment @sundeepn ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39011376
Build is starting -or- tests failed to complete.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13581/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39358127
@kayousterhout The current code iterates thru the the different jobs sharing the stage and cancels them all. It should handle the scenario of multiple jobs sharing the stage. Unless ofcourse, the jobs sharing the stage are interleaved with the 'cancel stage' signal (the race condition). In which case, the user should either see an exception or will have to cancel the stage again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39129521
@andrewor14 I think cancelStage() has to cancel all jobs that depend on the
stage -- otherwise those jobs would hang. It would be nice if there was a
way to propagate an error message that says something like "This job was
cancelled because one of its stages was cancelled in the UI" back to the
JobListener, so it's clear what happened.
On Mon, Mar 31, 2014 at 12:16 PM, andrewor14 <no...@github.com>wrote:
> @pwendell <https://github.com/pwendell> Yeah Spark Streaming also has
> that problem. A streaming job may contain many Spark jobs, so from a
> streaming user's point of view, our breakdown of Spark jobs doesn't make
> much sense. But if you think about it, jumping directly to stages is even
> lower level for streaming users, which is why we are making a custom
> Streaming UI that links to the SparkUI. I could be wrong but I think this
> issue is also common to Shark.
>
> @kayousterhout <https://github.com/kayousterhout> Separating the two
> changes this way makes sense. I have one question for (1): does
> cancelStage() just cancels the current stage, as the name suggests? Or does
> it cancel all the jobs that depend on that stage?
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/246#issuecomment-39129206>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11190015
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -505,6 +505,21 @@ class DAGScheduler(
}
/**
+ * Cancel all jobs associated with a running or scheduled stage.
+ */
+ def cancelStage(stageId: Int) {
--- End diff --
Also, it would be good to chance the handleJobCancellation() method to accept a string exceptionMessage parameter to use when creating the SparkException. For the normal case it can stay as "Job X cancelled", but for this case we can say "Job X cancelled because stage Y was cancelled" to make it more clear to the user what's going on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39129206
@pwendell Yeah Spark Streaming also has that problem. A streaming job may contain many Spark jobs, so from a streaming user's point of view, our breakdown of Spark jobs doesn't make much sense. But if you think about it, jumping directly to stages is even lower level for streaming users, which is why we are making a custom Streaming UI that links to the SparkUI. I could be wrong but I think this issue is also common to Shark.
@kayousterhout Separating the two changes this way makes sense. I have one question for (1): does cancelStage() just cancels the current stage, as the name suggests? Or does it cancel all the jobs that depend on that stage?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40154263
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39011375
Build finished. Build is starting -or- tests failed to complete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40130248
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39928157
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11424318
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -991,11 +1001,23 @@ class DAGScheduler(
}
}
- private def handleJobCancellation(jobId: Int) {
+ private def handleStageCancellation(stageId: Int) {
+ if (stageIdToJobIds.contains(stageId)) {
+ val jobsThatUseStage: Array[Int] = stageIdToJobIds(stageId).toArray.sorted
--- End diff --
I don't think you need the sorted here anymore right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011288
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -439,7 +439,7 @@ class DAGScheduler(
if (partitions.size == 0) {
return new JobWaiter[U](this, jobId, 0, resultHandler)
}
-
+
--- End diff --
Remove white space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-38881832
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39128529
It seems like there are two things at play here:
(1) Adding a cancel button in the UI. It sounds like this can be done in a not-hugely-invasive way by adding a cancelStage() method in the DAGScheduler that can be called by the UI.
(2) Adding job-level information to the UI. When we do this we might want to rearrange how the cancellation gets implemented -- probably there will need to be a refactoring of how the DAGScheduler keeps track of job / stage information so that it can easily be shared with the UI (and at this point we might want to move how the cancelStage() stuff happens).
I'd guess (1) can easily be done for 1.0 but (2) may not be able to be and involves much larger changes. Seems fine for Sundeep to finish (1) here and then @andrewor14 / @tdas / @pwendell to implement (2) later/separately?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39928152
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39367380
Looking forward to your patch. I will send out a updated patch with the other changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40150432
please trigger tests again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40154535
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39404617
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39159315
Build finished. Build is starting -or- tests failed to complete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11426142
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -46,6 +46,7 @@ private[spark] class SparkUI(
val live = sc != null
val securityManager = if (live) sc.env.securityManager else new SecurityManager(conf)
+ val killEnabled = conf.get("spark.ui.killEnabled", "false").toBoolean
--- End diff --
Also - could you document this in `docs/configuration.md`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39928173
@kayousterhout Here is the rebased UI code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39127366
I think adding the cancel functionality is desirable, but to do it cleanly requires a broader effort (specifically adding job-level information). @sundeepn Feel free to keeping working on this, but if you have other things on your plate then we can take over. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39125321
@kayousterhout I have thought about adding cancelStage to SparkContext/DAGScheduler. My earlier take was that the Job level Info is useful in the UI as well. Currently, its not shown/used in the UI, but lots of our users get confused looking at the stage level UI showing their query multiple times. It will be good to expose the Job ID in the stage table as well.
Having said that, I do not see any other way of doing this. We will have to move this to the DAGScheduler as a cancelStage. If two jobs can share stages, then any handling from the UI side can have a race condition in the cancel work flow and it will be a lot cleaner to handle upstream. I will submit a revision shortly.
@andrewor14, What do you think about adding a jobId column to the Stage table?
Thanks Mark for the pointer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11095648
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -116,6 +118,16 @@ private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
val stages = poolToActiveStages.getOrElseUpdate(poolName, new HashMap[Int, StageInfo]())
stages(stage.stageId) = stage
+
+ // Extract Job ID and double check if we have the details
+ val jobId = Option(stageSubmitted.properties).flatMap {
+ p => Option(p.getProperty("spark.job.id"))
+ }.getOrElse("-1").toInt
--- End diff --
Ah cool -- I looked at the ordering of the JobStart and StageSubmitted events more closely and I think you can safely remove this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40158903
Hey I'm gonna go ahead and merge this - I'll have a follow-on patch with some changes... felt that was easier than doing another round-trip on the review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40130268
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39931898
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11042971
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/KillPage.scala ---
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.jobs
+
+import java.util.Date
+import javax.servlet.http.HttpServletRequest
+
+import scala.xml.Node
+
+import org.apache.spark.ui.Page._
+import org.apache.spark.ui.{WebUI, UIUtils}
+import org.apache.spark.util.{Utils, Distribution}
+
+/** Page showing statistics and task list for a given stage */
+private[ui] class KillPage(parent: JobProgressUI) {
+ private val appName = parent.appName
+ private val basePath = parent.basePath
+ private lazy val listener = parent.listener
+ private lazy val sc = parent.sc
+
+ def render(request: HttpServletRequest): Seq[Node] = {
+ listener.synchronized {
+ val stageId = request.getParameter("id").toInt
+ val jobIdOpt = listener.stageIdToJobId.get(stageId)
+
+ if (!parent.killEnabled) {
+ val content =
+ <div>
+ <h4>Option disbaled</h4> Cannot kill jobs through the UI.
+ Please set spark.ui.killEnabled to change this behavior
+ </div>
+ return UIUtils.headerSparkPage(
+ content, basePath, appName, "Disabled option", Stages)
+
+ }
+
+ jobIdOpt match {
+ case Some(jobId) => {
+ sc.cancelJob(jobId)
+ val content =
+ <div>
+ Stage {stageId} and Job {jobId} are being killed
+ </div>
+ return UIUtils.headerSparkPage(
+ content, basePath, appName, "Stage %s is being terminated".format(stageId), Stages)
+ }
+ case _ => {
--- End diff --
Yep.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39932952
Jenkins is reporting a failure due to >100char lines: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13936/console
Can you fix these?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40149800
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39011156
@andrewor14 / @kayousterhout. Thanks for the comments.
I have posted an additional commit with the changes. I have checked them in my branch but some reason they are not showing up here. Do you want me to issue another new pull request? Let me know.
https://github.com/sundeepn/spark/commit/32c2ea58e5053623e5bf5955e52f7b7cd640ff64
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11426135
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -46,6 +46,7 @@ private[spark] class SparkUI(
val live = sc != null
val securityManager = if (live) sc.env.securityManager else new SecurityManager(conf)
+ val killEnabled = conf.get("spark.ui.killEnabled", "false").toBoolean
--- End diff --
maybe make this enabled by default? This is a cool feature. Also I think you can use `conf.getBoolean` here to have better type safety.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40158484
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14026/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40153259
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14024/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by sundeepn <gi...@git.apache.org>.
Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39931844
@kayousterhout ah! cleaned up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011382
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -226,8 +239,17 @@ private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
}
case _ =>
}
+ // Clear out tracking for jobs that have ended
+ jobIdToStageIds.remove(jobEnd.jobId)
}
+ override def onJobStart(jobStart: SparkListenerJobStart) = synchronized {
+ // Track job to stage ID mapping to allow users to kill jobs
+ // Cleanup as jobs/stages are completed
+ jobIdToStageIds + (jobStart.jobId -> jobStart.stageIds)
+ jobStart.stageIds.foreach(id => stageIdToJobId + (id -> jobStart.jobId))
--- End diff --
Did you mean `+=` here? Is this a bug?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11189924
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -224,8 +225,31 @@ private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
failedStages += s
trimIfNecessary(failedStages)
}
+
+ // Clean up other stages associated with this job; typically when job is cancelled
+ jobIdToStageIds.get(jobEnd.jobId).getOrElse(Seq[Int]()).foreach { sId =>
+ // CLean up the stage
--- End diff --
nit: Clean
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39931974
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-38772149
Hi @sundeepn. Thanks for doing this. I left a few relatively minor comments. I haven't looked too closely into it but could you explain on a high level, what page links to the KillPage? If I understand correctly, each KillPage is associated with a stage ID, which seems a little strange when the "unit of killing" here is the job. Is there a reason why you created a new page for killing jobs, as opposed to embedding in existing pages (e.g. the StagePage)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40149629
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011422
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala ---
@@ -49,12 +51,23 @@ private[ui] class JobProgressUI(parent: SparkUI) {
def formatDuration(ms: Long) = Utils.msDurationToString(ms)
- def getHandlers = Seq[ServletContextHandler](
- createServletHandler("/stages/stage",
- (request: HttpServletRequest) => stagePage.render(request), parent.securityManager, basePath),
- createServletHandler("/stages/pool",
- (request: HttpServletRequest) => poolPage.render(request), parent.securityManager, basePath),
- createServletHandler("/stages",
- (request: HttpServletRequest) => indexPage.render(request), parent.securityManager, basePath)
- )
+ def getHandlers = {
+
+ val handlers = Seq[ServletContextHandler](
+ createServletHandler("/stages/stage",
+ (request: HttpServletRequest) => stagePage.render(request), parent.securityManager, basePath),
+ createServletHandler("/stages/pool",
+ (request: HttpServletRequest) => poolPage.render(request), parent.securityManager, basePath),
+ createServletHandler("/stages",
+ (request: HttpServletRequest) => indexPage.render(request), parent.securityManager, basePath)
+ )
+
+ if (killEnabled) {
+ (createServletHandler("/stages/kill",
+ (request: HttpServletRequest) => killPage.render(request), parent.securityManager, basePath)
+ ) +: handlers
+ } else {
+ handlers
+ }
--- End diff --
I would do something like
```
val killHandler = if (killEnabled) {
...
} else {
Array[ServletContextHandler]
}
```
and then in the end `++` them together. See how this is done in SparkUI's handlers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11095610
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -116,6 +118,16 @@ private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
val stages = poolToActiveStages.getOrElseUpdate(poolName, new HashMap[Int, StageInfo]())
stages(stage.stageId) = stage
+
+ // Extract Job ID and double check if we have the details
+ val jobId = Option(stageSubmitted.properties).flatMap {
+ p => Option(p.getProperty("spark.job.id"))
+ }.getOrElse("-1").toInt
--- End diff --
When we chatted about this I remember you saying that this code is to handle the case where the stage runs locally at the driver...but from glancing at the DAGScheduler code, it looks like onStageSubmitted() never gets called for the locally-run tasks, and they never show up at the UI.
When do you need this code / when will the jobIdToStageIds mapping not already be set up correctly by OnJobStart?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011337
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -32,7 +32,7 @@ import org.apache.spark.storage.BlockManagerId
* class, since the UI thread and the EventBus loop may otherwise be reading and
* updating the internal data structures concurrently.
*/
-private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
+private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener with Logging {
--- End diff --
nit: remove this? I don't see where this is used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011300
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -512,7 +512,10 @@ class DAGScheduler(
*/
private[scheduler] def processEvent(event: DAGSchedulerEvent): Boolean = {
event match {
- case JobSubmitted(jobId, rdd, func, partitions, allowLocal, callSite, listener, properties) =>
+ case JobSubmitted(jobId, rdd, func, partitions, allowLocal, callSite, listener, props) =>
+ var properties = {
+ if (props != null) props else new java.util.Properties
+ }
--- End diff --
How about, `Option(props).getOrElse(new Properties)`? Also we can just `import java.util.Properties` at the top.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011762
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1028,6 +1028,11 @@ class SparkContext(
dagScheduler.cancelAllJobs()
}
+ /** Cancel a given job if its scheduled or running*/
--- End diff --
As long as we're being nitpicky... should be "it's"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11011437
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/KillPage.scala ---
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.jobs
+
+import java.util.Date
+import javax.servlet.http.HttpServletRequest
+
+import scala.xml.Node
+
+import org.apache.spark.ui.Page._
+import org.apache.spark.ui.{WebUI, UIUtils}
+import org.apache.spark.util.{Utils, Distribution}
+
+/** Page showing statistics and task list for a given stage */
+private[ui] class KillPage(parent: JobProgressUI) {
+ private val appName = parent.appName
+ private val basePath = parent.basePath
+ private lazy val listener = parent.listener
+ private lazy val sc = parent.sc
+
+ def render(request: HttpServletRequest): Seq[Node] = {
+ listener.synchronized {
+ val stageId = request.getParameter("id").toInt
+ val jobIdOpt = listener.stageIdToJobId.get(stageId)
+
+ if (!parent.killEnabled) {
+ val content =
+ <div>
+ <h4>Option disbaled</h4> Cannot kill jobs through the UI.
+ Please set spark.ui.killEnabled to change this behavior
+ </div>
+ return UIUtils.headerSparkPage(
+ content, basePath, appName, "Disabled option", Stages)
+
+ }
+
+ jobIdOpt match {
+ case Some(jobId) => {
+ sc.cancelJob(jobId)
+ val content =
+ <div>
+ Stage {stageId} and Job {jobId} are being killed
+ </div>
+ return UIUtils.headerSparkPage(
+ content, basePath, appName, "Stage %s is being terminated".format(stageId), Stages)
+ }
+ case _ => {
--- End diff --
Isn't this always going to be None?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39297535
Hey @sundeepn, so there's one other problem with your change which is that the code you added in the UI to clean up the stages for a cancelled job doesn't quite work because of the issue we discussed above, where a stage can be used by multiple jobs. Fixing this is a big pain that I don't want to subject you to, so I'm working on a separate pull request that properly cleans up stuff in the UI when a job gets cancelled (since this fact that this is broken is not an artifact of your change; it's just made more obvious by your change). I've started this but I'm not quite done; I'll submit this in the next day or so and cc you on the pull request, and then hopefully we can get both of these things into the 1.0 release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-38762651
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13489/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11425917
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala ---
@@ -71,15 +71,28 @@ private[ui] class StageTable(stages: Seq[StageInfo], parent: JobProgressUI) {
</div>
}
- /** Render an HTML row that represents a stage */
- private def stageRow(s: StageInfo): Seq[Node] = {
- val poolName = listener.stageIdToPool.get(s.stageId)
+ private def makeDescription(s: StageInfo): Seq[Node] = {
val nameLink =
<a href={"%s/stages/stage?id=%s".format(UIUtils.prependBaseUri(basePath), s.stageId)}>
{s.name}
</a>
+ val killButton = if (killEnabled) {
--- End diff --
extra space here after `=`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-38762601
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39125783
+1 on job-level UI, including job-level progress indication.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11426048
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala ---
@@ -71,15 +71,28 @@ private[ui] class StageTable(stages: Seq[StageInfo], parent: JobProgressUI) {
</div>
}
- /** Render an HTML row that represents a stage */
- private def stageRow(s: StageInfo): Seq[Node] = {
- val poolName = listener.stageIdToPool.get(s.stageId)
+ private def makeDescription(s: StageInfo): Seq[Node] = {
val nameLink =
<a href={"%s/stages/stage?id=%s".format(UIUtils.prependBaseUri(basePath), s.stageId)}>
{s.name}
</a>
+ val killButton = if (killEnabled) {
+ <form action={"%s/stages/stage/".format(UIUtils.prependBaseUri(basePath))}>
--- End diff --
instead of using a form here with POST parameters and a submit button, could we just have a link? The button looks really out of place in our UI and a bit clunky. For now what about having a small link right next to the stage name that says "[kill]"? Then we can style it more later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40153258
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39404790
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40152573
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39128026
Hm - the only tricky part about a job level UI is that Spark's notion of a Job is not the same thing as the query submitted by a higher level application like Shark. It's possible for a given shark query to spawn multiple Spark jobs. And having this in the UI means that now there are three levels of abstractions for the user to worry about (a) the Shark query or other higher level query (b) Spark jobs (c) Spark stages.
Anyways just wanted to note this distinction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40045886
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40143186
Test failure here:
```
[info] - trivial job cancellation *** FAILED *** (2 milliseconds)
[info] "Job 0 cancelled[ ]" did not equal "Job 0 cancelled[]" (DAGSchedulerSuite.scala:293)
[info] org.scalatest.exceptions.TestFailedException:
```
Seems like maybe a simple string format issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40149814
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40150587
Jenkins, retest this please
On Thu, Apr 10, 2014 at 3:26 PM, sundeepn <no...@github.com> wrote:
> please trigger tests again.
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/246#issuecomment-40150432>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-38762600
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39928286
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-38762650
Build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40154524
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40045887
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13987/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39928287
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13928/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11189913
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -505,6 +505,21 @@ class DAGScheduler(
}
/**
+ * Cancel all jobs associated with a running or scheduled stage.
+ */
+ def cancelStage(stageId: Int) {
--- End diff --
I think adding an event adds just a few lines of code and handles the race condition discussed offline (where a job submitted after the cancelStage() function gets called but before the JobCancellation() event gets processed won't be cancelled). I'd create a StageCancelled event that this function calls, and then just move the body of the function to inside the DAGScheduler event loop in the StageCancelled case (and change the cancelJob() function to handleJobCancellation()).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-40045814
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1202 - Add a "cancel" button in the UI f...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11424341
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -61,6 +61,7 @@ private[ui] class JobProgressListener(conf: SparkConf) extends SparkListener {
val stageIdToPool = HashMap[Int, String]()
val stageIdToDescription = HashMap[Int, String]()
val poolToActiveStages = HashMap[String, HashMap[Int, StageInfo]]()
+ val jobIdToStageIds = HashMap[Int, Seq[Int]]()
--- End diff --
remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---