You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/10/28 00:34:04 UTC

[GitHub] spark pull request #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is re...

GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/19593

    [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required after using loginUserFromKeytab

    ## What changes were proposed in this pull request?
    
    In a secure cluster, FileSystem.Cache grows indefinitely when we use
    1. `spark.yarn.principal` and `spark.yarn.keytab` is used.
    2. Spark Thrift Server run with `hive.server2.enable.doAs=false`.
    
    For example, with 6GB (-Xmx6144m) options, `HiveConf` consumes 4GB inside FileSystem.CACHE and OOM occurs. This PR aims to clear up `FileSystem.Cache` by using `closeAllForUGI`.
    
    ![2](https://user-images.githubusercontent.com/9700541/32129492-b09f6cbc-bb3c-11e7-8f75-fe027d626816.png)
    
    ![3](https://user-images.githubusercontent.com/9700541/32129494-bafa2cce-bb3c-11e7-80f8-5b03e3471109.png)
    
    ## How was this patch tested?
    
    N/A

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-22374

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19593.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19593
    
----
commit ea3943e977d378609c9440c049bdb86a308ba428
Author: Dongjoon Hyun <do...@apache.org>
Date:   2017-10-27T03:23:13Z

    [SPARK-22374][SQL][2.2] closeAllForUGI is required after using loginUserFromKeytab

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is re...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun closed the pull request at:

    https://github.com/apache/spark/pull/19593


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19593
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19593
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83142/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19593
  
    **[Test build #83142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83142/testReport)** for PR 19593 at commit [`ea3943e`](https://github.com/apache/spark/commit/ea3943e977d378609c9440c049bdb86a308ba428).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required ...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/19593
  
    Thank you for review, @vanzin .
    Sorry, I'll close this PR.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required ...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19593
  
    Does the problem not exist on master? Why isn't the PR against master?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19593: [WIP][SPARK-22374][SQL][2.2] closeAllForUGI is required ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19593
  
    **[Test build #83142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83142/testReport)** for PR 19593 at commit [`ea3943e`](https://github.com/apache/spark/commit/ea3943e977d378609c9440c049bdb86a308ba428).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org