You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by vanzin <gi...@git.apache.org> on 2017/11/06 21:03:06 UTC

[GitHub] spark pull request #19679: [SPARK-20647][core] Port StorageTab to the new UI...

GitHub user vanzin opened a pull request:

    https://github.com/apache/spark/pull/19679

    [SPARK-20647][core] Port StorageTab to the new UI backend.

    This required adding information about StreamBlockId to the store,
    which is not available yet via the API. So an internal type was added
    until there's a need to expose that information in the API.
    
    The UI only lists RDDs that have cached partitions, and that information
    wasn't being correctly captured in the listener, so that's also fixed,
    along with some minor (internal) API adjustments so that the UI can
    get the correct data.
    
    Because of the way partitions are cached, some optimizations w.r.t. how
    often the data is flushed to the store could not be applied to this code;
    because of that, some different ways to make the code more performant
    were added to the data structures tracking RDD blocks, with the goal of
    avoiding expensive copies when lots of blocks are being updated.
    
    Tested with existing and updated unit tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vanzin/spark SPARK-20647

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19679.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19679
    
----
commit 7147bd241b8acd6a944d3bba9170f98f8233cc3b
Author: Marcelo Vanzin <va...@cloudera.com>
Date:   2017-01-30T22:48:30Z

    [SPARK-20647][core] Port StorageTab to the new UI backend.
    
    This required adding information about StreamBlockId to the store,
    which is not available yet via the API. So an internal type was added
    until there's a need to expose that information in the API.
    
    The UI only lists RDDs that have cached partitions, and that information
    wasn't being correctly captured in the listener, so that's also fixed,
    along with some minor (internal) API adjustments so that the UI can
    get the correct data.
    
    Because of the way partitions are cached, some optimizations w.r.t. how
    often the data is flushed to the store could not be applied to this code;
    because of that, some different ways to make the code more performant
    were added to the data structures tracking RDD blocks, with the goal of
    avoiding expensive copies when lots of blocks are being updated.
    
    Tested with existing and updated unit tests.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19679: [SPARK-20647][core] Port StorageTab to the new UI backen...

Posted by vanzin <gi...@git.apache.org>.

Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19679
  
    For context:
    
    * Project link: https://issues.apache.org/jira/browse/SPARK-18085
    * Upcoming PRs that build on this code: https://github.com/vanzin/spark/pulls


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19679: [SPARK-20647][core] Port StorageTab to the new UI backen...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19679
  
    **[Test build #83619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83619/testReport)** for PR 19679 at commit [`fd59a24`](https://github.com/apache/spark/commit/fd59a24ee89ced2b74b52d702806547aa0c578e8).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19679: [SPARK-20647][core] Port StorageTab to the new UI backen...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19679
  
    **[Test build #83604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83604/testReport)** for PR 19679 at commit [`e257a1a`](https://github.com/apache/spark/commit/e257a1a3e4ec9180e78176824fef96bc705cdee0).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19679: [SPARK-20647][core] Port StorageTab to the new UI backen...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19679
  
    **[Test build #83509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83509/testReport)** for PR 19679 at commit [`7147bd2`](https://github.com/apache/spark/commit/7147bd241b8acd6a944d3bba9170f98f8233cc3b).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19679: [SPARK-20647][core] Port StorageTab to the new UI backen...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19679
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83509/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19679: [SPARK-20647][core] Port StorageTab to the new UI backen...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19679
  
    **[Test build #83509 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83509/testReport)** for PR 19679 at commit [`7147bd2`](https://github.com/apache/spark/commit/7147bd241b8acd6a944d3bba9170f98f8233cc3b).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19679: [SPARK-20647][core] Port StorageTab to the new UI backen...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19679
  
    **[Test build #83604 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83604/testReport)** for PR 19679 at commit [`e257a1a`](https://github.com/apache/spark/commit/e257a1a3e4ec9180e78176824fef96bc705cdee0).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org