You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Jan W <jc...@gmail.com> on 2011/10/11 00:01:03 UTC

problem with fanout sink

Hello:

So I'm having problems configuring a fanout sink.  Here's what I have for my
collector config:

config [ip-10-88-101-167.ec2.internal, collectorSource(35853),
collectorSink("hdfs://flume-logs/%Y%m%d/","%H00-%{tailSrcFile}-%{host}-"),
collectorSink("file:///flume-logs/%Y%m%d/","%H00-%{tailSrcFile}-%{host}-")]

So the command succeeds, but never actually writes anything to the second
sink listed.  So if hdfs is listed as the first sink, all collected events
sink there, and none get copied to the second.

Am I doing something wrong?  CDH3 for everything...

--thanks for any help

jan

-- 
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
Sent from my brain.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=

Re: problem with fanout sink

Posted by Jan W <jc...@gmail.com>.

Thanks, David.

Brackets all the way.  Fixed.

--cheers

jan

On Mon, Oct 10, 2011 at 4:38 PM, David Zülke
<da...@bitextender.com>wrote:

> > Hello:
> >
> > So I'm having problems configuring a fanout sink.  Here's what I have for
> my collector config:
> >
> > config [ip-10-88-101-167.ec2.internal, collectorSource(35853),
> collectorSink("hdfs://flume-logs/%Y%m%d/","%H00-%{tailSrcFile}-%{host}-"),
> collectorSink("file:///flume-logs/%Y%m%d/","%H00-%{tailSrcFile}-%{host}-")]
> >
> > So the command succeeds, but never actually writes anything to the second
> sink listed.  So if hdfs is listed as the first sink, all collected events
> sink there, and none get copied to the second.
> >
> > Am I doing something wrong?  CDH3 for everything...
>
> Fanout sinks are listed in square brackets, so I think you ned to do:
>
> collectorSource(35853), [ collectorSink(...), collectorSink(...) ]
>
> - David
>
>
>


-- 
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
Sent from my brain.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=

Re: problem with fanout sink

Posted by David Zülke <da...@bitextender.com>.

> Hello:
> 
> So I'm having problems configuring a fanout sink.  Here's what I have for my collector config:
> 
> config [ip-10-88-101-167.ec2.internal, collectorSource(35853), collectorSink("hdfs://flume-logs/%Y%m%d/","%H00-%{tailSrcFile}-%{host}-"), collectorSink("file:///flume-logs/%Y%m%d/","%H00-%{tailSrcFile}-%{host}-")]
> 
> So the command succeeds, but never actually writes anything to the second sink listed.  So if hdfs is listed as the first sink, all collected events sink there, and none get copied to the second.
> 
> Am I doing something wrong?  CDH3 for everything...

Fanout sinks are listed in square brackets, so I think you ned to do:

collectorSource(35853), [ collectorSink(...), collectorSink(...) ]

- David