You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by sandeep-katta <gi...@git.apache.org> on 2018/09/27 17:31:32 UTC

[GitHub] spark pull request #22571: [SPARK-25392][Spark Job History]Inconsistent beha...

GitHub user sandeep-katta opened a pull request:

    https://github.com/apache/spark/pull/22571

    [SPARK-25392][Spark Job History]Inconsistent behaviour for pool details in spark web UI and history server page 

    
    ## What changes were proposed in this pull request?
    
    1. Added the pool information to the "Pool Information" field in the environmentDetails by using the poolname as the key value, the xml string of some other fields of the pool is Value.
    This event is persist in event log dir
    
    2.AppStatusListener::onEnvironmentUpdate will parse the xml String and construct the Map[poolName,Pool] which will be stored in kvstore.
    
    2, the page is divided into two types: one is AllStagesPage, and the other is PoolPage,both will fetch the pool Details from kvstore and display accordingly
    
    In summary, modify the environmentDetails, save the pool information through the SparkListenerEnvironmentUpdate event information; modify it according to the modification scheme. 
    
    ## How was this patch tested?
    
    Added Testcases and also verified manually in the cluster
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sandeep-katta/spark historyserver

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22571.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22571
    
----
commit 65e3ca79c3ce8bcd16d9e92d7e9b5e69b532edef
Author: sandeep-katta <sa...@...>
Date:   2018-09-27T10:18:45Z

    RootCause:History server does not maintain Pool Details as no event was sent to maintain pool Details
    Modification content:Pool Information is sent as a part of EnvironmentInfo Event

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22571: [SPARK-25392][Spark Job History]Inconsistent beha...

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22571#discussion_r222841579
  
    --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
    @@ -2434,8 +2434,15 @@ class SparkContext(config: SparkConf) extends Logging {
           val schedulingMode = getSchedulingMode.toString
           val addedJarPaths = addedJars.keys.toSeq
           val addedFilePaths = addedFiles.keys.toSeq
    +      // SPARK-25392 pool Information should be stored in the event
    +      val poolInformation = getAllPools.map { it =>
    +        val xmlString = ("<pool><item PoolName=\"%s\" MinimumShare=\"%d\"" +
    --- End diff --
    
    Hmm, really don't like this kind of ad-hoc serialization format.
    
    Instead, create a proper pool class for the REST API, and use that class to transfer pool information around.
    
    Because this is a legacy event (not serialized by Jackson) you'll need some custom serialization in `JsonProtocol.scala`, but that's cleaner than what you have in this change.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour f...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22571
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour f...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/22571
  
    I'm not sure I know enough to evaluate this. Is the solution to send all this info around, or just not link to a pool in this case? I don't know.
    
    For some reason the Github diff is highlighting strings in dark red, as if there's a problem. I don't see one. Not sure why it is doing that.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour f...

Posted by sandeep-katta <gi...@git.apache.org>.

Github user sandeep-katta commented on the issue:

    https://github.com/apache/spark/pull/22571
  
    cc @vanzin @srowen ,Please review this improvement in History UI


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour f...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22571
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour f...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22571
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour f...

Posted by sandeep-katta <gi...@git.apache.org>.

Github user sandeep-katta commented on the issue:

    https://github.com/apache/spark/pull/22571
  
    Yes I am sending the pool Info around,so history server also can have details.
    
    And regarding the dark red ,I think git unable to parse it correctly.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour f...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22571
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour f...

Posted by sandeep-katta <gi...@git.apache.org>.

Github user sandeep-katta commented on the issue:

    https://github.com/apache/spark/pull/22571
  
    cc @cloud-fan Please review 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org