You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2016/12/29 07:39:55 UTC

[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

GitHub user jerryshao opened a pull request:

    https://github.com/apache/spark/pull/16432

    [SPARK-19021][YARN] Generailize HDFSCredentialProvider to support non HDFS security filesystems

    Change-Id: I85d7963c4980cf9660f495f377ba27227da4b1b1
    
    Currently Spark can only get token renewal interval from security HDFS (hdfs://), if Spark runs with other security file systems like webHDFS (webhdfs://), wasb (wasb://), ADLS, it will ignore these tokens and not get token renewal intervals from these tokens. These will make Spark unable to work with these security clusters. So instead of only checking HDFS token, we should generalize to support different DelegationTokenIdentifier.
    
    ## How was this patch tested?
    
    Manually verified in security cluster.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jerryshao/apache-spark SPARK-19021

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/16432.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #16432
    
----
commit 0c1ec23626b74544e456778262030cee1c623b30
Author: jerryshao <ss...@hortonworks.com>
Date:   2016-12-29T07:30:19Z

    Generailize HDFSCredentialProvider to support non HDFS security FS
    
    Change-Id: I85d7963c4980cf9660f495f377ba27227da4b1b1

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70910/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    so the changes look good.  I just thought of one other thing though and that is really the name and description of spark.yarn.access.namenodes is off.   I'll file a separate jira for that though since unrelated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71056/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/16432


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #70715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70715/testReport)** for PR 16432 at commit [`0c1ec23`](https://github.com/apache/spark/commit/0c1ec23626b74544e456778262030cee1c623b30).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71199/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #70715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70715/testReport)** for PR 16432 at commit [`0c1ec23`](https://github.com/apache/spark/commit/0c1ec23626b74544e456778262030cee1c623b30).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #70712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70712/testReport)** for PR 16432 at commit [`0c1ec23`](https://github.com/apache/spark/commit/0c1ec23626b74544e456778262030cee1c623b30).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #71199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71199/testReport)** for PR 16432 at commit [`cedd482`](https://github.com/apache/spark/commit/cedd482e74bb0f939fb5907cdefc0c3fdc76308f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r95104135
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala ---
    @@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends ServiceCredentialProvider
         }
     
         // Get the time of next renewal.
    -    tokenRenewalInterval.map { interval =>
    -      creds.getAllTokens.asScala
    -        .filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
    +    tokenRenewalInterval.flatMap { interval =>
    +      val nextRenewalDates = creds.getAllTokens.asScala
    +        .filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier])
    --- End diff --
    
    Yeah, that's the issue, let me think about how to handle it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r94971916
  
    --- Diff: docs/running-on-yarn.md ---
    @@ -479,12 +479,12 @@ Hadoop services issue *hadoop tokens* to grant access to the services and data.
     Clients must first acquire tokens for the services they will access and pass them along with their
     application as it is launched in the YARN cluster.
     
    -For a Spark application to interact with HDFS, HBase and Hive, it must acquire the relevant tokens
    +For a Spark application to interact with Hadoop filesystem, HBase and Hive, it must acquire the relevant tokens
    --- End diff --
    
    I think perhaps we should say interact with any of the Hadoop filesystems (for example hdfs, webhdfs, etc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #71054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71054/testReport)** for PR 16432 at commit [`86c76a6`](https://github.com/apache/spark/commit/86c76a6666f0bdb4fda15e420d1103dffcb375e3).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #71065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71065/testReport)** for PR 16432 at commit [`2673af2`](https://github.com/apache/spark/commit/2673af288c0c9472af3aa8e68782d6e572d12346).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Ok can we change the name of this from FileSystem to be HadoopFileSystem or HadoopFs to be a bit more clear and then update the docs accordingly to make sure we specify what the requirements are for that to work on those filesystems.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #71054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71054/testReport)** for PR 16432 at commit [`86c76a6`](https://github.com/apache/spark/commit/86c76a6666f0bdb4fda15e420d1103dffcb375e3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r95104079
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala ---
    @@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends ServiceCredentialProvider
         }
     
         // Get the time of next renewal.
    -    tokenRenewalInterval.map { interval =>
    -      creds.getAllTokens.asScala
    -        .filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
    +    tokenRenewalInterval.flatMap { interval =>
    +      val nextRenewalDates = creds.getAllTokens.asScala
    +        .filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier])
             .map { t =>
    -          val identifier = new DelegationTokenIdentifier()
    -          identifier.readFields(new DataInputStream(new ByteArrayInputStream(t.getIdentifier)))
    +          val identifier = t.decodeIdentifier().asInstanceOf[AbstractDelegationTokenIdentifier]
               identifier.getIssueDate + interval
    -      }.foldLeft(0L)(math.max)
    +        }
    +      if (nextRenewalDates.isEmpty) None else Some(nextRenewalDates.min)
    --- End diff --
    
    I think the logic was incorrect previously, we should get the minimum renewal date, that's why I changed here. But the code was probably OK if only we only have one default HDFS configured. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    @tgravescs , would you please review again, thanks a lot.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #70910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70910/testReport)** for PR 16432 at commit [`131d420`](https://github.com/apache/spark/commit/131d420c779702f8a7129f29a4d1d361eb92a166).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70712/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #71056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71056/testReport)** for PR 16432 at commit [`3b4c4b0`](https://github.com/apache/spark/commit/3b4c4b0bed43a04fbbb1f3de7e45e34f1007a222).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71065/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #71056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71056/testReport)** for PR 16432 at commit [`3b4c4b0`](https://github.com/apache/spark/commit/3b4c4b0bed43a04fbbb1f3de7e45e34f1007a222).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #71199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71199/testReport)** for PR 16432 at commit [`cedd482`](https://github.com/apache/spark/commit/cedd482e74bb0f939fb5907cdefc0c3fdc76308f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r94972595
  
    --- Diff: docs/running-on-yarn.md ---
    @@ -494,7 +494,7 @@ Similarly, a Hive token will be obtained if Hive is on the classpath, its config
     includes a URI of the metadata store in `"hive.metastore.uris`, and
     `spark.yarn.security.credentials.hive.enabled` is not set to `false`.
     
    -If an application needs to interact with other secure HDFS clusters, then
    +If an application needs to interact with other secure Hadoop filesystem clusters, then
    --- End diff --
    
    can you also change the example below to include webhdfs 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70715/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    @tgravescs thanks for your comments, I also tested with webhdfs with configuration like:
    
    ```
    --conf spark.yarn.access.namenodes=webhdfs://c6402.ambari.apache.org:50070/
    ```
    
    And the related logs are:
    
    ```
    17/01/09 03:15:49 INFO HadoopFSCredentialProvider: getting token for: webhdfs://c6402.ambari.apache.org:50070/
    17/01/09 03:15:50 INFO HadoopFSCredentialProvider: getting token for: hdfs://c6402.ambari.apache.org:8020/user/spark
    17/01/09 03:15:50 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 138 for spark on 192.168.64.102:8020
    17/01/09 03:15:50 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 140 for spark on 192.168.64.102:8020
    17/01/09 03:15:50 INFO HadoopFSCredentialProvider: Renewal interval is 86400112 for token WEBHDFS delegation
    17/01/09 03:15:50 INFO HadoopFSCredentialProvider: Renewal interval is 86400146 for token HDFS_DELEGATION_TOKEN
    ```
    
    As you could see also get tokens and token renewal interval from webhdfs token.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #71065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71065/testReport)** for PR 16432 at commit [`2673af2`](https://github.com/apache/spark/commit/2673af288c0c9472af3aa8e68782d6e572d12346).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r94972291
  
    --- Diff: docs/running-on-yarn.md ---
    @@ -479,12 +479,12 @@ Hadoop services issue *hadoop tokens* to grant access to the services and data.
     Clients must first acquire tokens for the services they will access and pass them along with their
     application as it is launched in the YARN cluster.
     
    -For a Spark application to interact with HDFS, HBase and Hive, it must acquire the relevant tokens
    +For a Spark application to interact with Hadoop filesystem, HBase and Hive, it must acquire the relevant tokens
     using the Kerberos credentials of the user launching the application
     \u2014that is, the principal whose identity will become that of the launched Spark application.
     
     This is normally done at launch time: in a secure cluster Spark will automatically obtain a
    -token for the cluster's HDFS filesystem, and potentially for HBase and Hive.
    +token for the cluster's Hadoop filesystem, and potentially for HBase and Hive.
    --- End diff --
    
    Perhaps say the default hadoop filesystem


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r94972451
  
    --- Diff: docs/running-on-yarn.md ---
    @@ -494,7 +494,7 @@ Similarly, a Hive token will be obtained if Hive is on the classpath, its config
     includes a URI of the metadata store in `"hive.metastore.uris`, and
     `spark.yarn.security.credentials.hive.enabled` is not set to `false`.
     
    -If an application needs to interact with other secure HDFS clusters, then
    +If an application needs to interact with other secure Hadoop filesystem clusters, then
    --- End diff --
    
    remove "clusters" here.  should just say "interact with other secure Hadoop filesystems".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71054/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    what kind of testing have you done on this? 
    
    A couple of concerns:  
    
    1) FileSystem is kind of obscure, really I think it means hadoop filesystems.  It won't work for just any filesystem.
    
    2) this is documented: https://spark.apache.org/docs/latest/running-on-yarn.html  I'm not fond of changing config names.  Its basically an api.  However, I doubt many people are specifically setting it at this point though since its in the defaults.  Either way we would need to update that documentation. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r95534171
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala ---
    @@ -17,37 +17,39 @@
     
     package org.apache.spark.deploy.yarn.security
     
    -import java.io.{ByteArrayInputStream, DataInputStream}
    -
     import scala.collection.JavaConverters._
    +import scala.util.Try
     
     import org.apache.hadoop.conf.Configuration
     import org.apache.hadoop.fs.{FileSystem, Path}
    -import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
     import org.apache.hadoop.mapred.Master
     import org.apache.hadoop.security.Credentials
    +import org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier
     
     import org.apache.spark.{SparkConf, SparkException}
     import org.apache.spark.deploy.yarn.config._
     import org.apache.spark.internal.Logging
     import org.apache.spark.internal.config._
     
    -private[security] class HDFSCredentialProvider extends ServiceCredentialProvider with Logging {
    +private[security] class HadoopFSCredentialProvider
    +    extends ServiceCredentialProvider with Logging {
       // Token renewal interval, this value will be set in the first call,
    -  // if None means no token renewer specified, so cannot get token renewal interval.
    +  // if None means no token renewer specified or no token can be renewed,
    +  // so cannot get token renewal interval.
       private var tokenRenewalInterval: Option[Long] = null
     
    -  override val serviceName: String = "hdfs"
    +  override val serviceName: String = "hadoopfs"
     
       override def obtainCredentials(
           hadoopConf: Configuration,
           sparkConf: SparkConf,
           creds: Credentials): Option[Long] = {
         // NameNode to access, used to get tokens from different FileSystems
    +    val tmpCreds = new Credentials()
         nnsToAccess(hadoopConf, sparkConf).foreach { dst =>
           val dstFs = dst.getFileSystem(hadoopConf)
    -      logInfo("getting token for namenode: " + dst)
    -      dstFs.addDelegationTokens(getTokenRenewer(hadoopConf), creds)
    +      logInfo("getting token for: " + dst)
    +      dstFs.addDelegationTokens(getTokenRenewer(hadoopConf), tmpCreds)
    --- End diff --
    
    Can you put `getTokenRenewer` outside the foreach loop and use a variable for it? We don't need to get the token renewer everytime.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r95537856
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala ---
    @@ -17,37 +17,39 @@
     
     package org.apache.spark.deploy.yarn.security
     
    -import java.io.{ByteArrayInputStream, DataInputStream}
    -
     import scala.collection.JavaConverters._
    +import scala.util.Try
     
     import org.apache.hadoop.conf.Configuration
     import org.apache.hadoop.fs.{FileSystem, Path}
    -import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier
     import org.apache.hadoop.mapred.Master
     import org.apache.hadoop.security.Credentials
    +import org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier
     
     import org.apache.spark.{SparkConf, SparkException}
     import org.apache.spark.deploy.yarn.config._
     import org.apache.spark.internal.Logging
     import org.apache.spark.internal.config._
     
    -private[security] class HDFSCredentialProvider extends ServiceCredentialProvider with Logging {
    +private[security] class HadoopFSCredentialProvider
    +    extends ServiceCredentialProvider with Logging {
       // Token renewal interval, this value will be set in the first call,
    -  // if None means no token renewer specified, so cannot get token renewal interval.
    +  // if None means no token renewer specified or no token can be renewed,
    +  // so cannot get token renewal interval.
       private var tokenRenewalInterval: Option[Long] = null
     
    -  override val serviceName: String = "hdfs"
    +  override val serviceName: String = "hadoopfs"
     
       override def obtainCredentials(
           hadoopConf: Configuration,
           sparkConf: SparkConf,
           creds: Credentials): Option[Long] = {
         // NameNode to access, used to get tokens from different FileSystems
    +    val tmpCreds = new Credentials()
         nnsToAccess(hadoopConf, sparkConf).foreach { dst =>
           val dstFs = dst.getFileSystem(hadoopConf)
    -      logInfo("getting token for namenode: " + dst)
    -      dstFs.addDelegationTokens(getTokenRenewer(hadoopConf), creds)
    +      logInfo("getting token for: " + dst)
    +      dstFs.addDelegationTokens(getTokenRenewer(hadoopConf), tmpCreds)
    --- End diff --
    
    Done, thanks for your comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Thanks @tgravescs for your comments. Yes actually it should be Hadoop compatible file systems, or in other words should be Hadoop `FileSystem` API supported FS. I tested with kerberized HDFS and wasb.
    
    In HDI, it ships with kerberized wasb or ADSL, they will also have delegation token but with different token kind. In the current code we assume that token kind should be `HDFS_DELEGATION_KIND`, this is not true if the default FS is not HDFS or there's no HDFS in the cluster. In such case currently we simply ignore the tokens from these FS, which will be an issue for long running Spark application. This also applies to webHDFS (it also has different token kind).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    **[Test build #70910 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70910/testReport)** for PR 16432 at commit [`131d420`](https://github.com/apache/spark/commit/131d420c779702f8a7129f29a4d1d361eb92a166).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r95103785
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala ---
    @@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends ServiceCredentialProvider
         }
     
         // Get the time of next renewal.
    -    tokenRenewalInterval.map { interval =>
    -      creds.getAllTokens.asScala
    -        .filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
    +    tokenRenewalInterval.flatMap { interval =>
    +      val nextRenewalDates = creds.getAllTokens.asScala
    +        .filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier])
    --- End diff --
    
    Can we make sure all instances of `AbstractDelegationTokenIdentifier` are tokens for Hadoop file systems? Isn't Hive's `DelegationTokenIdentifier` also an `AbstractDelegationTokenIdentifier`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    few minor doc comments, otherwise looks good. 
    
    Could you also please test with webhdfs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16432
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16432#discussion_r95103643
  
    --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala ---
    @@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends ServiceCredentialProvider
         }
     
         // Get the time of next renewal.
    -    tokenRenewalInterval.map { interval =>
    -      creds.getAllTokens.asScala
    -        .filter(_.getKind == DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
    +    tokenRenewalInterval.flatMap { interval =>
    +      val nextRenewalDates = creds.getAllTokens.asScala
    +        .filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier])
             .map { t =>
    -          val identifier = new DelegationTokenIdentifier()
    -          identifier.readFields(new DataInputStream(new ByteArrayInputStream(t.getIdentifier)))
    +          val identifier = t.decodeIdentifier().asInstanceOf[AbstractDelegationTokenIdentifier]
               identifier.getIssueDate + interval
    -      }.foldLeft(0L)(math.max)
    +        }
    +      if (nextRenewalDates.isEmpty) None else Some(nextRenewalDates.min)
    --- End diff --
    
    Why previously we use math.max and now changed to minimum value?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org