You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by yunjzhang <gi...@git.apache.org> on 2018/08/21 23:29:21 UTC

[GitHub] spark pull request #22177: stages in wrong order within job page DAG chart

GitHub user yunjzhang opened a pull request:

    https://github.com/apache/spark/pull/22177

    stages in wrong order within job page DAG chart

    if spark job contains multiple tasks , the order in DAG chart might be incorrect.
    sample and screen snapshot can be found in jira ticket https://issues.apache.org/jira/browse/SPARK-25119
    
    to fix this issue, just sort RDDOperationGraph Array on "task id" before UIUtils.showDagVizForJob()
    
    ## How was this patch tested?
    packaging with all existing UT passed. 
    run complex query in Spark-sql console with the new jars
    check the DAG chart in job page. 


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yunjzhang/spark SPARK-25119

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22177.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22177
    
----
commit f75b3fa1a09b63617d940e779a24614b82d22905
Author: Zhang <yu...@...>
Date:   2018-08-21T23:06:08Z

    stages in wrong order within job page DAG chart

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22177: [SPARK-25119][Web UI] stages in wrong order within job p...

Posted by yunjzhang <gi...@git.apache.org>.

Github user yunjzhang commented on the issue:

    https://github.com/apache/spark/pull/22177
  
    my fault, just fixed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22177: [SPARK-25119][Web UI] stages in wrong order withi...

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22177#discussion_r212134229
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala ---
    @@ -18,18 +18,18 @@
     package org.apache.spark.ui.jobs
     
     import java.util.Locale
    +
     import javax.servlet.http.HttpServletRequest
     
     import scala.collection.mutable.{Buffer, ListBuffer}
     import scala.xml.{Node, NodeSeq, Unparsed, Utility}
    -
    --- End diff --
    
    +1


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22177: stages in wrong order within job page DAG chart

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22177
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22177: [SPARK-25119][Web UI] stages in wrong order within job p...

Posted by yunjzhang <gi...@git.apache.org>.

Github user yunjzhang commented on the issue:

    https://github.com/apache/spark/pull/22177
  
    I respect your decision.
    this PR does not fix the order issue from the root.
    I saw differences when jobs/tasks were shown in spark 2.1/2.3.
    same insert...select was treated as 2 jobs in 2.1, while which was 1 job in 2.3.
    If 2.3 can push tasks into an array in right order, extract sort is not necessary in this PR and also PR#21680


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22177: [SPARK-25119][Web UI] stages in wrong order within job p...

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/22177
  
    That looks better indeed. I'll backport that one to 2.3.
    
    @yunjzhang could you close this PR?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22177: [SPARK-25119][Web UI] stages in wrong order within job p...

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/22177
  
    You're sorting on stage ID but your description says task ID. Just a small inconsistency.
    
    This is also against branch-2.3, it should be against master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22177: [SPARK-25119][Web UI] stages in wrong order withi...

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22177#discussion_r212134537
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala ---
    @@ -337,7 +337,9 @@ private[ui] class JobPage(parent: JobsTab, store: AppStatusStore) extends WebUIP
           store.executorList(false), appStartTime)
     
         val operationGraphContent = store.asOption(store.operationGraphForJob(jobId)) match {
    -      case Some(operationGraph) => UIUtils.showDagVizForJob(jobId, operationGraph)
    +      case Some(operationGraph) => UIUtils.showDagVizForJob(jobId, operationGraph.sortWith(
    +        _.rootCluster.id.replaceAll(RDDOperationGraph.STAGE_CLUSTER_PREFIX, "").toInt
    --- End diff --
    
    +1. `replaceAll` also seems like the wrong API to use, `substring` seems more correct.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22177: [SPARK-25119][Web UI] stages in wrong order within job p...

Posted by wangyum <gi...@git.apache.org>.

Github user wangyum commented on the issue:

    https://github.com/apache/spark/pull/22177
  
    How about backport https://github.com/apache/spark/pull/21680?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22177: [SPARK-25199][Web UI] stages in wrong order within job p...

Posted by MaxGekk <gi...@git.apache.org>.

Github user MaxGekk commented on the issue:

    https://github.com/apache/spark/pull/22177
  
    Probably you put wrong JIRA in the title. `SPARK-25199`  -> `SPARK-25119`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org