[jira] [Commented] (FLUME-1691) Allow use of EC2 roles with S3 sink

["D. Granit (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597057#comment-13597057 ] 

D. Granit commented on FLUME-1691:
----------------------------------

A patch was committed to HADOOP-9384 that allows using the HDFS sink with the latest Amazon SDK to support the use of the instance meta data service to establish credentials provided via a role applied to an instance.

if you leave your {{hdfs-site.xml}} without any configuration and set your flume config to 
{code}
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = s3n://yourBucket
{code}
without credentials provided (It's important to use {{s3n://}} and not {{s3://}} as the patch only patches the native s3 implementation and not the block fs one) then it'll use the {{AmazonS3Client}} constructor following the order you have described above and as such will pick up the credentials provided by the meta data service. 

                
> Allow use of EC2 roles with S3 sink
> -----------------------------------
>
>                 Key: FLUME-1691
>                 URL: https://issues.apache.org/jira/browse/FLUME-1691
>             Project: Flume
>          Issue Type: Improvement
>          Components: Sinks+Sources
>    Affects Versions: v0.9.4
>            Reporter: Steve Stogner
>            Priority: Minor
>
> If you assign an IAM role to an EC2 instance, then AWS exposes role credentials through the metadata interface.  These credentials are temporary credentials that AWS rolls periodically.  When making calls to AWS with temporary credentials, you have to use a token in addition to the access ID and secret key.  Flume would impress if it would default to the EC2 role credentials when using an S3 sink with no credentials configuration required.  Flume would either refresh the credentials from the metadata with every call to S3 or when it detects that the credentials have expired.  Users could still override the use of role credentials with user credentials via the current configuration method (fs.s3.awsAccessKeyId, fs.s3.awsSecretAccessKey, fs.s3n.awsAccessKeyId, fs.s3n.awsSecretAccessKey).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira