Fanout [a, b] send to a even if b is down?

[Björn Edström <bj...@spotify.com>]

Hello.

My collector currently has a single collectorSink writing to HDFS. Now
I'd like to add a real time "tap" into flume. The fanout feature seems
suited for this purpose.

However, I would like to do so in such a way that the availability of
the tap node does not affect the writing of data to the HDFS
collectorSink. Basically, I want flume to not care if this node is
down.

Any idea how I can do this? I tried using
[collectorSink("hdfs..."), agentBESink("tap...")]
but agentBESink will block writing to the collectorSink if the tap node is down.

Best regards
Björn

Re: Fanout [a, b] send to a even if b is down?

[Björn Edström <bj...@spotify.com>]

Anyone?

I was thinking of perhaps using the failover sink in combination with
fanout, like

< [a, b] ? a >

But that seems like a hacky solution, since it's not really what I want.

It'd be nice to have some way of "attempt to connect, attempt to send,
always report success"

Best regards
Björn


On 12/9/11, Björn Edström <bj...@spotify.com> wrote:
> Hello.
>
> My collector currently has a single collectorSink writing to HDFS. Now
> I'd like to add a real time "tap" into flume. The fanout feature seems
> suited for this purpose.
>
> However, I would like to do so in such a way that the availability of
> the tap node does not affect the writing of data to the HDFS
> collectorSink. Basically, I want flume to not care if this node is
> down.
>
> Any idea how I can do this? I tried using
> [collectorSink("hdfs..."), agentBESink("tap...")]
> but agentBESink will block writing to the collectorSink if the tap node is
> down.
>
> Best regards
> Björn
>


-- 
Björn Edström
Developer