You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by igor Finkelshteyn <ie...@gmail.com> on 2012/08/28 00:39:01 UTC
Flume (NG?) with S3 Sink?
Hi Folks,
I'm completely new to flume, but was hoping to use it as a way to collect logs on a couple web servers and funnel them to an s3 sink in Avro. I've found a tutorial for doing this on Flume OG, but am thinking I should probably just work with Flume NG for a new project if that's possible. Is there a prebuilt s3 sink for Flume NG that I can use? Can I just use the HDFS sink and provide some extra parameters to get it to work with s3? Or should I just use Flume OG?I'd also be open to using something else entirely that has better s3 support, like FluentD, if that makes more sense. Any ideas?
Thanks!
Igor
Re: Flume (NG?) with S3 Sink?
Posted by Brock Noland <br...@cloudera.com>.
FLUME-1015 is saying we could use the Hadoop S3 FileSystem
abstraction. If it doesn't work, it's likely a hadoop problem.
FLUME-951 is for implementing a direct to S3 sink.
On Tue, Aug 28, 2012 at 4:29 PM, Julian Henry Alcala <ze...@gmail.com> wrote:
> I tried flume-OG with s3 and it was not reliable. I am interested in what
> people are doing with flume-ng and s3 as well.
>
>
> On Tue, Aug 28, 2012 at 1:47 PM, igor Finkelshteyn <ie...@gmail.com>
> wrote:
>>
>> Has no one needed to send logs through Flume NG to s3? This JIRA makes it
>> sound like it's something that's already been implemented with
>> documentation, but there is 0 documentation anywhere (definitely not in the
>> user guide), and the JIRA was resolved months ago.
>>
>>
>> On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:
>>
>> Hi Folks,
>> I'm completely new to flume, but was hoping to use it as a way to collect
>> logs on a couple web servers and funnel them to an s3 sink in Avro. I've
>> found a tutorial for doing this on Flume OG, but am thinking I should
>> probably just work with Flume NG for a new project if that's possible. Is
>> there a prebuilt s3 sink for Flume NG that I can use? Can I just use the
>> HDFS sink and provide some extra parameters to get it to work with s3? Or
>> should I just use Flume OG?I'd also be open to using something else entirely
>> that has better s3 support, like FluentD, if that makes more sense. Any
>> ideas?
>>
>> Thanks!
>> Igor
>>
>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Flume (NG?) with S3 Sink?
Posted by "Bhaskar V. Karambelkar" <bh...@gmail.com>.
There's a project over at Github
https://github.com/Hobsons/hobsons-flumeng-s3-sink
haven't tried it myself though.
On Tue, Aug 28, 2012 at 5:29 PM, Julian Henry Alcala <ze...@gmail.com>wrote:
> I tried flume-OG with s3 and it was not reliable. I am interested in what
> people are doing with flume-ng and s3 as well.
>
>
> On Tue, Aug 28, 2012 at 1:47 PM, igor Finkelshteyn <ie...@gmail.com>wrote:
>
>> Has no one needed to send logs through Flume NG to s3? This JIRA<https://issues.apache.org/jira/browse/FLUME-1015> makes
>> it sound like it's something that's already been implemented with
>> documentation, but there is 0 documentation anywhere (definitely not in the
>> user guide), and the JIRA was resolved months ago.
>>
>>
>> On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:
>>
>> Hi Folks,
>> I'm completely new to flume, but was hoping to use it as a way to collect
>> logs on a couple web servers and funnel them to an s3 sink in Avro. I've
>> found a tutorial for doing this on Flume OG, but am thinking I should
>> probably just work with Flume NG for a new project if that's possible. Is
>> there a prebuilt s3 sink for Flume NG that I can use? Can I just use the
>> HDFS sink and provide some extra parameters to get it to work with s3? Or
>> should I just use Flume OG?I'd also be open to using something else
>> entirely that has better s3 support, like FluentD, if that makes more
>> sense. Any ideas?
>>
>> Thanks!
>> Igor
>>
>>
>>
>>
>
Re: Flume (NG?) with S3 Sink?
Posted by Julian Henry Alcala <ze...@gmail.com>.
I tried flume-OG with s3 and it was not reliable. I am interested in what
people are doing with flume-ng and s3 as well.
On Tue, Aug 28, 2012 at 1:47 PM, igor Finkelshteyn <ie...@gmail.com>wrote:
> Has no one needed to send logs through Flume NG to s3? This JIRA<https://issues.apache.org/jira/browse/FLUME-1015> makes
> it sound like it's something that's already been implemented with
> documentation, but there is 0 documentation anywhere (definitely not in the
> user guide), and the JIRA was resolved months ago.
>
>
> On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:
>
> Hi Folks,
> I'm completely new to flume, but was hoping to use it as a way to collect
> logs on a couple web servers and funnel them to an s3 sink in Avro. I've
> found a tutorial for doing this on Flume OG, but am thinking I should
> probably just work with Flume NG for a new project if that's possible. Is
> there a prebuilt s3 sink for Flume NG that I can use? Can I just use the
> HDFS sink and provide some extra parameters to get it to work with s3? Or
> should I just use Flume OG?I'd also be open to using something else
> entirely that has better s3 support, like FluentD, if that makes more
> sense. Any ideas?
>
> Thanks!
> Igor
>
>
>
>
Re: Flume (NG?) with S3 Sink?
Posted by igor Finkelshteyn <ie...@gmail.com>.
Has no one needed to send logs through Flume NG to s3? This JIRA makes it sound like it's something that's already been implemented with documentation, but there is 0 documentation anywhere (definitely not in the user guide), and the JIRA was resolved months ago.
On Aug 27, 2012, at 3:39 PM, igor Finkelshteyn wrote:
> Hi Folks,
> I'm completely new to flume, but was hoping to use it as a way to collect logs on a couple web servers and funnel them to an s3 sink in Avro. I've found a tutorial for doing this on Flume OG, but am thinking I should probably just work with Flume NG for a new project if that's possible. Is there a prebuilt s3 sink for Flume NG that I can use? Can I just use the HDFS sink and provide some extra parameters to get it to work with s3? Or should I just use Flume OG?I'd also be open to using something else entirely that has better s3 support, like FluentD, if that makes more sense. Any ideas?
>
> Thanks!
> Igor
>
>