[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650831#comment-16650831 ] 

Steve Loughran edited comment on HADOOP-14556 at 10/15/18 9:45 PM:
-------------------------------------------------------------------

HADOOP-14556 patch 013
* ITestDelegatedMRJob mixes a mock job submission API with a real miniYarn cluster to verify that MR job submission collects DTs for source and destination paths.
  To do this the MockJob class had to go into hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java and job.connect() made an override point (so it can be skipped)
* default assumed role duration returned to 1h; it had been extended to 6h but that only works if your role has been explicitly extended to > 1h duration.
* and docs on increasing it (plus error messages you get if you don't) improved/extended in assumed_roles.md as well as delegation_tokens.md.
 All AWS error messages related to STS/session and role requests are now in assumed_roles.md to avoid duplication & inconsistencies.
* ITestS3ADelegationTokenSupport tests that the Session DT binding will forward any session creds it gets from its own auth chain, rather than ask for new ones (which it can't do with session creds)
* Also: I'm using a Hadoop cred provider for storing secrets; this broke the AssumeRole and delegation tests which were clearing or overwriting the fs.s3a.{auth, secret, session} options, as those in the creds file were still being picked up. Fix: explicitly reset hadoop.security.credential.provider.path for all the tests which were now failing.
* minor checkstyle fixup

tested, S3A ireland. Apart from the cred problem (fixed), I got a failure of {{ITestS3GuardToolLocal\#testDestroyNoBucket}} *even when I was running with dynamodb*. I think that test suite is running when it shouldn't. More research needed there


was (Author: stevel@apache.org):
HADOOP-14556 patch 013
* ITestDelegatedMRJob mixes a mock job submission API with a real miniYarn cluster to verify that MR job submission collects DTs for source and destination paths.
  To do this the MockJob class had to go into hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java and job.connect() made an override point (so it can be skipped)
* default assumed role duration returned to 1h; it had been extended to 6h but that only works if your role has been explicitly extended to > 1h duration.
* and docs on increasing it (plus error messages you get if you don't) improved/extended in assumed_roles.md as well as delegation_tokens.md.
 All AWS error messages related to STS/session and role requests are now in assumed_roles.md to avoid duplication & inconsistencies.
* ITestS3ADelegationTokenSupport tests that the Session DT binding will forward any session creds it gets from its own auth chain, rather than ask for new ones (which it can't do with session creds)
* Also: I'm using a Hadoop cred provider for storing secrets; this broke the AssumeRole and delegation tests which were clearing or overwriting the fs.s3a.{auth, secret, session} options, as those in the creds file were still being picked up. Fix: explicitly reset hadoop.security.credential.provider.path for all the tests which were now failing.
* minor checkstyle fixup

tested, S3A ireland. Apart from the cred problem (fixed), I got a failure of {{ITestS3GuardToolLocal\#testDestroyNoBucket }} *even when I was running with dynamodb*. I think that test suite is running when it shouldn't. More research needed there

> S3A to support Delegation Tokens
> --------------------------------
>
>                 Key: HADOOP-14556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14556
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id; these will be saved in the token and  marshalled with jobs
> * A new authentication provider will look for a token for the current user and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to the initial duration. Also, as you can't request an STS token from a temporary session, IAM instances won't be able to issue tokens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org