You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by João Ascenso <jo...@gmail.com> on 2012/09/16 22:18:51 UTC
Several sinks from the same channel
Greetings,
Anyone knows if i put n sinks listening to one channel, if every sink is
going to get copies of the message or if the messages
are distributed between the sinks?
Also, every new sink is gonna run on his own thread, or do i need several
flume instances for that?
-------------------
João Ascenso
Re: Several sinks from the same channel
Posted by João Ascenso <jo...@gmail.com>.
Thank you for the response
João Ascenso
2012/9/17 Hari Shreedharan <hs...@cloudera.com>
> Yes, each sink will be run by a different thread called the SinkRunner.
> Several sinks are multithreaded internally too(like the
> HDFSEventSink/AsyncHBaseSink).
>
> Hari
>
> --
> Hari Shreedharan
>
> On Sunday, September 16, 2012 at 4:24 PM, João Ascenso wrote:
>
> Thanks for the response
>
> Very nice design on flume part for the sink :)
>
> So, just to clarify, if i have 6 sinks running operations on logs before
> writing, each one will be on a diferent thread/ not hogging the same core
> on the cpu?
>
> ----------------
> João Ascenso
>
>
> 2012/9/16 Hari Shreedharan <hs...@cloudera.com>
>
> Reply inline.
>
> --
> Hari Shreedharan
>
> On Sunday, September 16, 2012 at 1:18 PM, João Ascenso wrote:
>
> Greetings,
>
> Anyone knows if i put n sinks listening to one channel, if every sink is
> going to get copies of the message or if the messages
> are distributed between the sinks?
>
> Events will get distributed across multiple sinks. The channel does not
> really care if one sink or multiple sinks take the event. Once taken
> and committed, the channel deletes the event.
>
>
> Also, every new sink is gonna run on his own thread, or do i need several
> flume instances for that?
>
> No, you can have multiple sinks in the same agent.
>
>
> -------------------
> João Ascenso
>
>
>
>
>
Re: Several sinks from the same channel
Posted by Hari Shreedharan <hs...@cloudera.com>.
Yes, each sink will be run by a different thread called the SinkRunner. Several sinks are multithreaded internally too(like the HDFSEventSink/AsyncHBaseSink).
Hari
--
Hari Shreedharan
On Sunday, September 16, 2012 at 4:24 PM, João Ascenso wrote:
> Thanks for the response
>
> Very nice design on flume part for the sink :)
>
> So, just to clarify, if i have 6 sinks running operations on logs before writing, each one will be on a diferent thread/ not hogging the same core on the cpu?
> ----------------
> João Ascenso
>
>
> 2012/9/16 Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)>
> > Reply inline.
> >
> > --
> > Hari Shreedharan
> >
> >
> > On Sunday, September 16, 2012 at 1:18 PM, João Ascenso wrote:
> >
> > > Greetings,
> > >
> > > Anyone knows if i put n sinks listening to one channel, if every sink is going to get copies of the message or if the messages are distributed between the sinks?
> > Events will get distributed across multiple sinks. The channel does not really care if one sink or multiple sinks take the event. Once taken and committed, the channel deletes the event.
> > >
> > > Also, every new sink is gonna run on his own thread, or do i need several flume instances for that?
> > No, you can have multiple sinks in the same agent.
> > >
> > > -------------------
> > > João Ascenso
> >
>
Re: Several sinks from the same channel
Posted by João Ascenso <jo...@gmail.com>.
Thanks for the response
Very nice design on flume part for the sink :)
So, just to clarify, if i have 6 sinks running operations on logs before
writing, each one will be on a diferent thread/ not hogging the same core
on the cpu?
----------------
João Ascenso
2012/9/16 Hari Shreedharan <hs...@cloudera.com>
> Reply inline.
>
> --
> Hari Shreedharan
>
> On Sunday, September 16, 2012 at 1:18 PM, João Ascenso wrote:
>
> Greetings,
>
> Anyone knows if i put n sinks listening to one channel, if every sink is
> going to get copies of the message or if the messages
> are distributed between the sinks?
>
> Events will get distributed across multiple sinks. The channel does not
> really care if one sink or multiple sinks take the event. Once taken
> and committed, the channel deletes the event.
>
>
> Also, every new sink is gonna run on his own thread, or do i need several
> flume instances for that?
>
> No, you can have multiple sinks in the same agent.
>
>
> -------------------
> João Ascenso
>
>
>
Re: Several sinks from the same channel
Posted by Hari Shreedharan <hs...@cloudera.com>.
Reply inline.
--
Hari Shreedharan
On Sunday, September 16, 2012 at 1:18 PM, João Ascenso wrote:
> Greetings,
>
> Anyone knows if i put n sinks listening to one channel, if every sink is going to get copies of the message or if the messages are distributed between the sinks?
Events will get distributed across multiple sinks. The channel does not really care if one sink or multiple sinks take the event. Once taken and committed, the channel deletes the event.
>
> Also, every new sink is gonna run on his own thread, or do i need several flume instances for that?
No, you can have multiple sinks in the same agent.
>
> -------------------
> João Ascenso