Using Flume with S3 Sink

[Ajay <aj...@gmail.com>]

Hi,

I could like to know how to use Flume with S3 sink with IAM role. I found
the JIRA ticket for the same  -
https://issues.apache.org/jira/browse/HADOOP-15141 but the
steps/configuration required for the same are not documented anywhere.

If anyone could point me to the documentation or provide the configuration
required, it could be of great help.

Thanks
Ajay

Re: Using Flume with S3 Sink

[Ajay <aj...@gmail.com>]

Hi,

Yes i mean the HDFS sink only for s3 as well. But with s3n protocol it need
access and secret keys. As per the mentioned Jira ticket, it is said that
with s3a protocol, it could work with IAM role. But I couldn't find any
documentation for the same but from the code merged in the Jira ticket that
few properties need to configured.

Thanks
Ajay


On Fri, Mar 16, 2018, 4:54 PM Suresh V <ve...@gmail.com> wrote:

> The HDFS Sink will function as a S3 sink as well, we have done that. But
> sometimes we had HTTP Connectivity timeout issues with S3 from the EC2
> where Flume was running, so we moved away from S3 and used HDFS for
> production.
>
> Make sure the IAM role attached to the EC2 that is running Flume has the
> write permissions to S3 bucket you are writing to.
>
> I am not aware of a separate S3 sink.
>
> Suresh.
>
>
> On Fri, Mar 16, 2018 at 5:16 AM, Ajay <aj...@gmail.com> wrote:
>
>> Hi,
>>
>> I could like to know how to use Flume with S3 sink with IAM role. I found
>> the JIRA ticket for the same  -
>> https://issues.apache.org/jira/browse/HADOOP-15141 but the
>> steps/configuration required for the same are not documented anywhere.
>>
>> If anyone could point me to the documentation or provide the
>> configuration required, it could be of great help.
>>
>> Thanks
>> Ajay
>>
>
>

Re: Using Flume with S3 Sink

[Suresh V <ve...@gmail.com>]

The HDFS Sink will function as a S3 sink as well, we have done that. But
sometimes we had HTTP Connectivity timeout issues with S3 from the EC2
where Flume was running, so we moved away from S3 and used HDFS for
production.

Make sure the IAM role attached to the EC2 that is running Flume has the
write permissions to S3 bucket you are writing to.

I am not aware of a separate S3 sink.

Suresh.


On Fri, Mar 16, 2018 at 5:16 AM, Ajay <aj...@gmail.com> wrote:

> Hi,
>
> I could like to know how to use Flume with S3 sink with IAM role. I found
> the JIRA ticket for the same  - https://issues.apache.org/
> jira/browse/HADOOP-15141 but the steps/configuration required for the
> same are not documented anywhere.
>
> If anyone could point me to the documentation or provide the configuration
> required, it could be of great help.
>
> Thanks
> Ajay
>