You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2019/01/02 19:23:00 UTC

[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens

    [ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732350#comment-16732350 ] 

Steve Loughran commented on HADOOP-14556:
-----------------------------------------

Patch 027; tested s3 ireland +dynamodb. All happy.
h3. Larry's comments
{quote}If delegation tokens are enabled it seems that there is no fallback to other providers. This seems appropriate to me but just want to make sure that interpretation is correct.
{quote}
Its up to the DT binding to provide the fallback chain of its choice. This lets it choose "fallback to some other property", "fail dramatically".
{quote}While there are specific implementations provided in the patch it also seems to be extensible by 3rd parties that which to provide their own DT and binding code.
{quote}
that's the goal.
{quote}Can you provide any details on renewal of DT's in the provided implementations?
{quote}
There is no renewal of DTs. It doesn't make sense.
{quote}AWSCredentialProviderList + name
{quote}
it's there to provide slightly more informative errors. I didn't bother enforcing it or being clever.

change: I patch things up in setName().
{quote}ASSUMED_ROLE_STS_ENDPOINT_REGION and ASSUMED_ROLE_STS_ENDPOINT_REGION
{quote}
this is really complex. Have a look at {{STSClientFactory.builder()}}. If you don't set a region then it defaults to the central endpoint, but you can't declare that as the endpoint without declaring a signing region in your EndpointConfiguration class. Which you don't need to do with the empty string. Really, *you do not want to begin trying to understand the logic here as there is no logic to understand*
{quote}S3A.java: it possible for the toString() to have credentials from the URI?
{quote}
not since we took away the ability to put credentials in URIs, no.
{quote}DurationInfo logAtInfo - should this be javadoc'd to explain how and when it is set to debug?
{quote}
done

 
{quote}S3AEncryptionMethods. Can the following javadoc be better explained?
{quote}
done. They just round off the enum with all supported methods. Added tests to verify the matching works.
{quote}S3AEncryptionMethods. should we consider making encryption algorithms such as AES256 configurable rather than hardcoded?
{quote}
nope. These are the exact strings which AWS requires in its headers.

 
h3. Other Changes
 * add Optional<OffsetDateTime> expiry time accessor of MarshalledCredentials; always returns a UTC-zoned value or empty() when no expiry is known.
 . Diving into the AWS SDKs, they seem to expect it to be UTC, though it's not ever spelt out in the docs properly. This is used in tests and toString. The wire format is still seconds in the Unix epoch with no TZ marker. It MUST be UTC.

 * Tests working in this probably work even if the person running the tests isn't in/near UTC. I think up until now, some of the assertions about expiry only worked when the offset between UTC and local time was close enough. that (UTC-expiry-time - local-clock-time) was positive

 * EncryptionSecrets don't have a ref to S3AUtils in, so no indirect ref to any AWS classes. Harder to isolate tokens than you'd think. I'm almost tempted to move the token identifiers into hadoop-common to guarantee that (a) they get everywhere and (b) no accidental extra dependencies. But that'd probably create other problems.

Token providers have the ability to provide a UA suffix for the client, which gets passed all the way down to the S3 client. Why so? Makes it possible to debug what's going wrong from S3A access logs, as the DT UUID can make it into the logs. Once you start playing with assumed role, user IDs get lost, and of course, that's when things start to go wrong. This propagates all the way into the S3 Client factories, I'm afraid.

 

> S3A to support Delegation Tokens
> --------------------------------
>
>                 Key: HADOOP-14556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14556
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id; these will be saved in the token and  marshalled with jobs
> * A new authentication provider will look for a token for the current user and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to the initial duration. Also, as you can't request an STS token from a temporary session, IAM instances won't be able to issue tokens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org