Getting Data from S3

[Jonathan <jo...@gmail.com>]

Hi,

I was wondering if Flume supports retrieving data from an S3 bucket and
sinking into HDFS. The s3 bucket is getting updated on a regular basis so I
would like for flume to catch any changes on s3 and move them accordingly.
 I have seen a lot of documentation on using S3 as a sink but I haven't been
able to find anything going the other way. Is this even possible?


Thanks
Jonathan

Re: Getting Data from S3

[Jonathan Hsieh <jo...@cloudera.com>]

Jonathan,

This isn't available out-of-the-box currently but one could create a custom
source that can grab data from s3.

This would be an interesting source plugin. (I've thought about building
something similar that uses an hdfs dir as a source).

Jon.

On Thu, Aug 18, 2011 at 11:24 AM, Jonathan <jo...@gmail.com> wrote:

> Hi,
>
> I was wondering if Flume supports retrieving data from an S3 bucket and
> sinking into HDFS. The s3 bucket is getting updated on a regular basis so I
> would like for flume to catch any changes on s3 and move them accordingly.
>  I have seen a lot of documentation on using S3 as a sink but I haven't been
> able to find anything going the other way. Is this even possible?
>
>
> Thanks
> Jonathan
>
>


-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com