You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yoshidakuy <gi...@git.apache.org> on 2016/01/08 09:10:35 UTC

[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stag...

GitHub user yoshidakuy opened a pull request:

    https://github.com/apache/spark/pull/10663

    [SPARK-12708][UI] Sorting task error in Stages Page when yarn mode.

    If sort column contains slash(e.g. "Executor ID / Host") when yarn mode,sort fail with following message.
    
    ![spark-12708](https://cloud.githubusercontent.com/assets/6679275/12193320/80814f8c-b62a-11e5-9914-7bf3907029df.png)
    
    
    It's similar to SPARK-4313 .

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yoshidakuy/spark SPARK-12708

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/10663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #10663
    
----
commit fc45425bc80a039994e57d2d98c0a6f281427654
Author: root <root@r520t1.(none)>
Date:   2016-01-08T07:59:02Z

    [SPARK-12708][UI] Sorting task error in Stages Page when yarn mode

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49760805
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/PoolTable.scala ---
    @@ -59,7 +61,8 @@ private[ui] class PoolTable(pools: Seq[Schedulable], parent: StagesTab) {
           case None => 0
         }
         val href = "%s/stages/pool?poolname=%s"
    -      .format(UIUtils.prependBaseUri(parent.basePath), p.name)
    +      .format(UIUtils.prependBaseUri(parent.basePath),
    +        URLEncoder.encode(p.name, "UTF-8"))
    --- End diff --
    
    This two lines can be fit into one line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169937595
  
    **[Test build #49013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49013/consoleFull)** for PR 10663 at commit [`fc45425`](https://github.com/apache/spark/commit/fc45425bc80a039994e57d2d98c0a6f281427654).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49734793
  
    --- Diff: core/src/test/scala/org/apache/spark/ui/UIUtilsSuite.scala ---
    @@ -67,6 +67,20 @@ class UIUtilsSuite extends SparkFunSuite {
           s"\nRunning progress bar should round down\n\nExpected:\n$expected\nGenerated:\n$generated")
       }
     
    +  test("decodeURLParameter (SPARK-12708: Sorting task error in Stages Page when yarn mode.)") {
    +    val encoded1 = "%252F"
    +    val decoded1 = "/"
    +    val encoded2 = "%253Cdriver%253E"
    +    val decoded2 = "<driver>"
    +
    +    assert(decoded1.sameElements(decodeURLParameter(encoded1)))
    +    assert(decoded2.sameElements(decodeURLParameter(encoded2)))
    --- End diff --
    
    Why don't you just compare by `===` instead of `sameElements`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-170018160
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49018/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/10663


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171780948
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171838535
  
    **[Test build #49414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49414/consoleFull)** for PR 10663 at commit [`384b990`](https://github.com/apache/spark/commit/384b990e237d6c0b311042301ed0252dc689569f).
     * This patch **fails from timeout after a configured wait of \`250m\`**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171782133
  
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stag...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49166120
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
    @@ -100,7 +101,19 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIPage("stage") {
           val parameterTaskPrevPageSize = request.getParameter("task.prevPageSize")
     
           val taskPage = Option(parameterTaskPage).map(_.toInt).getOrElse(1)
    -      val taskSortColumn = Option(parameterTaskSortColumn).getOrElse("Index")
    +      val taskSortColumn = Option(parameterTaskSortColumn).map {
    +        sortColumn =>
    --- End diff --
    
    `sortColumn` should be on previous line like `.map { sortColumn =>`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stag...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169934471
  
    ok to test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49760292
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/PoolPage.scala ---
    @@ -31,7 +31,7 @@ private[ui] class PoolPage(parent: StagesTab) extends WebUIPage("pool") {
     
       def render(request: HttpServletRequest): Seq[Node] = {
         listener.synchronized {
    -      val poolName = request.getParameter("poolname")
    +      val poolName = UIUtils.decodeURLParameter(request.getParameter("poolname"))
    --- End diff --
    
    NullPointerException if `poolname` is not set. Could you use `Option....getOrElse` like other place?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by yoshidakuy <gi...@git.apache.org>.
Github user yoshidakuy commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169992628
  
    @sarutak Thanks. I have addressed your comment.
    
    > Does this issue affect both yarn-client and yarn-cluster right?
    
    Yes.
    Fixed screenshot is following.
    ![spark-12708-2](https://cloud.githubusercontent.com/assets/6679275/12198261/a583aa56-b651-11e5-8cd8-1289b7b5a408.PNG)
    
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stag...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169934802
  
    @yoshidakuy Does this issue affect both `yarn-client` and `yarn-cluster` right?
    Also, could you show the screen shot after applying this patch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171780735
  
    **[Test build #49407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49407/consoleFull)** for PR 10663 at commit [`384b990`](https://github.com/apache/spark/commit/384b990e237d6c0b311042301ed0252dc689569f).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171863995
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by yoshidakuy <gi...@git.apache.org>.
Github user yoshidakuy commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-170205527
  
    Thanks for comments and I agree. will fix later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171449382
  
    Ping @yoshidakuy, could you update this pull request to address our comments? It would be great to merge this soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171838672
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171632053
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171780951
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49407/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-170018157
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stag...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169928271
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49734713
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala ---
    @@ -30,19 +29,7 @@ private[ui] class ExecutorThreadDumpPage(parent: ExecutorsTab) extends WebUIPage
       private val sc = parent.sc
     
       def render(request: HttpServletRequest): Seq[Node] = {
    -    val executorId = Option(request.getParameter("executorId")).map {
    -      executorId =>
    -        // Due to YARN-2844, "<driver>" in the url will be encoded to "%25253Cdriver%25253E" when
    -        // running in yarn-cluster mode. `request.getParameter("executorId")` will return
    -        // "%253Cdriver%253E". Therefore we need to decode it until we get the real id.
    -        var id = executorId
    -        var decodedId = URLDecoder.decode(id, "UTF-8")
    -        while (id != decodedId) {
    -          id = decodedId
    -          decodedId = URLDecoder.decode(id, "UTF-8")
    -        }
    -        id
    -    }.getOrElse {
    +    val executorId = Option(request.getParameter("executorId")).getOrElse {
    --- End diff --
    
    `decodeURLParameter` is not invoked.
    
    We should say like as follows right?
    
    ```
    Option(...).map { executorId =>
      UIUtils.decodeURLParameter(executorId)
    }.getOrElse {
      ...
    }
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171838673
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49414/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171839437
  
    **[Test build #49433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49433/consoleFull)** for PR 10663 at commit [`384b990`](https://github.com/apache/spark/commit/384b990e237d6c0b311042301ed0252dc689569f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49217150
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
    @@ -100,7 +101,20 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIPage("stage") {
           val parameterTaskPrevPageSize = request.getParameter("task.prevPageSize")
     
           val taskPage = Option(parameterTaskPage).map(_.toInt).getOrElse(1)
    -      val taskSortColumn = Option(parameterTaskSortColumn).getOrElse("Index")
    +      val taskSortColumn = Option(parameterTaskSortColumn).map { sortColumn =>
    +          // SPARK-12708
    +          // Due to YARN-2844, "/" in the url will be encoded to "%252F" when
    +          // running in yarn mode. `request.getParameter("task.sort")` will return
    +          // "%252F". Therefore we need to decode it until we get the real column name.
    +          // SPARK-4313 is similar to this issue.
    +          var column = sortColumn
    --- End diff --
    
    Since this is very similar to `ExecutorThreadDumpPage`, could you create a new method in UIUtils (maybe call it decodeURLParameter and add comments to explain it) and update here and ExecutorThreadDumpPage to use it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171760196
  
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171863848
  
    **[Test build #49433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49433/consoleFull)** for PR 10663 at commit [`384b990`](https://github.com/apache/spark/commit/384b990e237d6c0b311042301ed0252dc689569f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171449456
  
    (I think we're going to do a 1.6.1 soon, so if we get this in soon it can be in that release)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by yoshidakuy <gi...@git.apache.org>.
Github user yoshidakuy commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171759663
  
    @sarutak @zsxwing Thanks for your review. I have addressed your comments. Could you please check it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49225126
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
    @@ -100,7 +101,20 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIPage("stage") {
           val parameterTaskPrevPageSize = request.getParameter("task.prevPageSize")
     
           val taskPage = Option(parameterTaskPage).map(_.toInt).getOrElse(1)
    -      val taskSortColumn = Option(parameterTaskSortColumn).getOrElse("Index")
    +      val taskSortColumn = Option(parameterTaskSortColumn).map { sortColumn =>
    +          // SPARK-12708
    +          // Due to YARN-2844, "/" in the url will be encoded to "%252F" when
    +          // running in yarn mode. `request.getParameter("task.sort")` will return
    +          // "%252F". Therefore we need to decode it until we get the real column name.
    +          // SPARK-4313 is similar to this issue.
    +          var column = sortColumn
    --- End diff --
    
    Make sense. 
    @yoshidakuy can you add a new method to `UIUtils` to reduce the similar logic.
    Then, call it from `ExecutorThreadDumpPage` and `StagePage`.
    Also, could you add a test case for the new method to `UIUtilsSuite`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171755452
  
    **[Test build #49407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49407/consoleFull)** for PR 10663 at commit [`384b990`](https://github.com/apache/spark/commit/384b990e237d6c0b311042301ed0252dc689569f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stag...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49167186
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
    @@ -100,7 +101,19 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIPage("stage") {
           val parameterTaskPrevPageSize = request.getParameter("task.prevPageSize")
     
           val taskPage = Option(parameterTaskPage).map(_.toInt).getOrElse(1)
    -      val taskSortColumn = Option(parameterTaskSortColumn).getOrElse("Index")
    +      val taskSortColumn = Option(parameterTaskSortColumn).map {
    +        sortColumn =>
    +          // If sortColumn contains "/", `getParameter("task.sort")` will return 
    +          // "%252F" when yarn mode. we need additional decode.
    +          // See also SPARK-4313, YARN-2844.
    --- End diff --
    
    How about say like as follows?
    
    ```
    // SPARK-12708
    // Due to YARN-2844, "/" in the url will be encoded to "%252F" when
    // running in yarn mode. `request.getParameter("task.sort")` will return
    // "%252F". Therefore we need to decode it until we get the real id.
    // SPARK-4313 is similar to this issue.
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171610253
  
    **[Test build #49396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49396/consoleFull)** for PR 10663 at commit [`3d508c6`](https://github.com/apache/spark/commit/3d508c6f6c69c2b2e4b29f8b9edf7048694e7f23).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171869236
  
    LGTM as well. Merging into `master` and `branch-1.6`. Thanks @yoshidakuy !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169937602
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49013/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169937600
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171838681
  
    retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171632055
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49396/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171784154
  
    **[Test build #49414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49414/consoleFull)** for PR 10663 at commit [`384b990`](https://github.com/apache/spark/commit/384b990e237d6c0b311042301ed0252dc689569f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171631873
  
    **[Test build #49396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49396/consoleFull)** for PR 10663 at commit [`3d508c6`](https://github.com/apache/spark/commit/3d508c6f6c69c2b2e4b29f8b9edf7048694e7f23).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169988742
  
    **[Test build #49018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49018/consoleFull)** for PR 10663 at commit [`62c04ca`](https://github.com/apache/spark/commit/62c04ca239d160d24bfb55dee6d2e827b8183f9a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/10663#discussion_r49223834
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
    @@ -100,7 +101,20 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIPage("stage") {
           val parameterTaskPrevPageSize = request.getParameter("task.prevPageSize")
     
           val taskPage = Option(parameterTaskPage).map(_.toInt).getOrElse(1)
    -      val taskSortColumn = Option(parameterTaskSortColumn).getOrElse("Index")
    +      val taskSortColumn = Option(parameterTaskSortColumn).map { sortColumn =>
    +          // SPARK-12708
    +          // Due to YARN-2844, "/" in the url will be encoded to "%252F" when
    +          // running in yarn mode. `request.getParameter("task.sort")` will return
    +          // "%252F". Therefore we need to decode it until we get the real column name.
    +          // SPARK-4313 is similar to this issue.
    +          var column = sortColumn
    --- End diff --
    
    +1; good to reduce duplication here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by yoshidakuy <gi...@git.apache.org>.
Github user yoshidakuy commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171605226
  
    @JoshRosen @sarutak Sorry for late.
    I found same issue in PoolTable, so I fixed it too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-169937305
  
    **[Test build #49013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49013/consoleFull)** for PR 10663 at commit [`fc45425`](https://github.com/apache/spark/commit/fc45425bc80a039994e57d2d98c0a6f281427654).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-170017967
  
    **[Test build #49018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49018/consoleFull)** for PR 10663 at commit [`62c04ca`](https://github.com/apache/spark/commit/62c04ca239d160d24bfb55dee6d2e827b8183f9a).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-12708][UI] Sorting task error in Stages...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10663#issuecomment-171863996
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49433/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org