You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2018/10/01 18:01:00 UTC
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634404#comment-16634404 ]
Steve Loughran commented on HADOOP-14556:
-----------------------------------------
HADOOP-14556 patch 008
* coalesce config options so use assumed.role config options instead of adding new ones
* role policies setting things up properly, including the listBucket command (which is per bucket, not bucket path)
* tests for role tokens verify that you can't access other S3 resources
* AWS credential provider subclassing now does on-demand init, so that you can have some cred providers (temporary) which only raise an AWS exception on call for creds, others can fail fast in the constructor
* discussing use in hadoop-aws docs
* clarifying implementation/subclassing details in javadocs
* Deprecate test.fs.s3a.sts.endpoint in favour of fs.s3a.assumed.role.sts.endpoint; register it as a Configuration deprecation.
* Logging @ debug (correct?) time to acquire/refresh DTs; useful given there are potentially long-haul calls taking place.
* s3guard bucket-info prints token info, including a warning if delegation is set but you aren't logged in.
+ fix up the findbugs/checkstyle warnings from the previous run
Testing: S3 london; got one error (HADOOP-15807) without S3Guard: Consistency? Or some race condition in tests?
The `s3guard bucket-info` command now prints status
{code}
$ bin/hadoop s3guard bucket-info s3a://landsat-pds/
Filesystem s3a://landsat-pds
Location: us-west-2
Filesystem s3a://landsat-pds is not using S3Guard
The "magic" committer is supported
S3A Client
Endpoint: fs.s3a.endpoint=s3.amazonaws.com
Encryption: fs.s3a.server-side-encryption-algorithm=none
Input seek policy: fs.s3a.experimental.input.fadvise=normal
Delegation token support is disabled
{code}
And when you enable it to the default value (session tokens); it prints the kind & then warns that security is off
{code}
$ bin/hadoop s3guard -D fs.s3a.delegation.tokens.enabled=true bucket-info s3a://landsat-pds/
Filesystem s3a://landsat-pds
Location: us-west-2
Filesystem s3a://landsat-pds is not using S3Guard
The "magic" committer is supported
S3A Client
Endpoint: fs.s3a.endpoint=s3.amazonaws.com
Encryption: fs.s3a.server-side-encryption-algorithm=none
Input seek policy: fs.s3a.experimental.input.fadvise=normal
Delegation Support enabled; token kind = S3ADelegationToken/Session
Warning: Hadoop security is disabled; delegation tokens will not be generated.
{code}
It might be good to make this an option where you can declare the token value, e.g -dtkind S3ADelegationToken/Role" which would fail if the token mapping wasn't right (type, flags, kinit, etc). That way: no ambiguity. Indeed, it could actually ask for a DT, though I'd rather that dttuil worked there: if it doesn't right now, that's something to fix.
One issue here which I've made "go away" is to declare the FS default port is 0; the canonical URI == the normal s3 URI (no port, no user prefix). Putting the port #into URIs broke many things (FileContext, mock staging committer tests, S3Guard's existing URIs). I hope everyone is happy with that.
> S3A to support Delegation Tokens
> --------------------------------
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id; these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to the initial duration. Also, as you can't request an STS token from a temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org