You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Rakos, Rudolf" <Ru...@morganstanley.com> on 2012/08/16 14:30:52 UTC
Aggregate channel or sink
Hi All,
We are developing a tool which embeds Flume. I have some questions about the planned / upcoming features of Flume.
I understand that it is possible to create fan out configurations using channel processors or sink processors.
My question would be that is it planned to support some sink or channel configuration which aggregates messages from different channels or sources?
I’m thinking of a configuration like this:
(source1) -> [channel1] -> -> (aggregate sink)-> [channel3] -> (final sink)
(source2) -> [channel2] -> /
The messages would flow from both channel1 and channel2 to the final sink.
I know this is possible by using multiple Avro sinks and an Avro source, but I think that is only a workaround for this problem, not a nice solution.
Regards,
Rudolf
Rudolf Rakos
Morgan Stanley | ISG Technology
Rudolf.Rakos@morganstanley.com<ma...@morganstanley.com>
Be carbon conscious. Please consider our environment before printing this email.
--------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
RE: Aggregate channel or sink
Posted by "Rakos, Rudolf" <Ru...@morganstanley.com>.
Hari,
Thanks for the suggestion, it works as expected.
I tried the same configuration you suggested a few weeks ago with Flume 1.1.0, but it ignored source1 completely. So I started complicating things with the concept of an aggregate channel or sink. I don't know whether I did something wrong or this issue got fixed in Flume 1.2.0, but it works now.
Thank you for your help.
Regards,
Rudolf
-----Original Message-----
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Thursday, August 16, 2012 8:12 PM
To: user@flume.apache.org
Subject: Re: Aggregate channel or sink
Rudolf,
Is there a reason this configuration would not work:
source1 -> channel1 -> final sink
and source2->channel1-> final sink.
Multiple sources can write to the same channel. So you could have source1 and source2 writing to the same channel which the final sink reads from(assuming all these are on the same physical machine).
Hari
--
Hari Shreedharan
On Thursday, August 16, 2012 at 5:30 AM, Rakos, Rudolf wrote:
> Hi All,
>
> We are developing a tool which embeds Flume. I have some questions about the planned / upcoming features of Flume.
>
> I understand that it is possible to create fan out configurations using channel processors or sink processors.
> My question would be that is it planned to support some sink or channel configuration which aggregates messages from different channels or sources?
>
> I’m thinking of a configuration like this:
> (source1) -> [channel1] -> -> (aggregate sink)-> [channel3] -> (final sink)
> (source2) -> [channel2] -> /
> The messages would flow from both channel1 and channel2 to the final sink.
>
> I know this is possible by using multiple Avro sinks and an Avro source, but I think that is only a workaround for this problem, not a nice solution.
>
> Regards,
> Rudolf
>
> Rudolf Rakos
> Morgan Stanley | ISG Technology
> Rudolf.Rakos@morganstanley.com (mailto:Rudolf.Rakos@morganstanley.com)
>
>
> Be carbon conscious. Please consider our environment before printing this email.
>
>
>
>
>
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
--------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
Re: Aggregate channel or sink
Posted by Hari Shreedharan <hs...@cloudera.com>.
Rudolf,
Is there a reason this configuration would not work:
source1 -> channel1 -> final sink
and source2->channel1-> final sink.
Multiple sources can write to the same channel. So you could have source1 and source2 writing to the same channel which the final sink reads from(assuming all these are on the same physical machine).
Hari
--
Hari Shreedharan
On Thursday, August 16, 2012 at 5:30 AM, Rakos, Rudolf wrote:
> Hi All,
>
> We are developing a tool which embeds Flume. I have some questions about the planned / upcoming features of Flume.
>
> I understand that it is possible to create fan out configurations using channel processors or sink processors.
> My question would be that is it planned to support some sink or channel configuration which aggregates messages from different channels or sources?
>
> I’m thinking of a configuration like this:
> (source1) -> [channel1] -> -> (aggregate sink)-> [channel3] -> (final sink)
> (source2) -> [channel2] -> /
> The messages would flow from both channel1 and channel2 to the final sink.
>
> I know this is possible by using multiple Avro sinks and an Avro source, but I think that is only a workaround for this problem, not a nice solution.
>
> Regards,
> Rudolf
>
> Rudolf Rakos
> Morgan Stanley | ISG Technology
> Rudolf.Rakos@morganstanley.com (mailto:Rudolf.Rakos@morganstanley.com)
>
>
> Be carbon conscious. Please consider our environment before printing this email.
>
>
>
>
>
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.