You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Babu, Prashanth" <Pr...@nttdata.com> on 2013/04/02 10:25:39 UTC
Flume-NG -- 2 Sources and a single Sink
Hi,
I have been using Flume-OG till now and currently I am evaluating Flume-NG.
I have followed the guidelines in http://flume.apache.org/FlumeDeveloperGuide.html.
In my use case, I have 2 Sources on a machine, but a single custom sink.
2 Sources are tails on 2 different files which are configured in flume.conf.
I have tried giving the same channel for both the sources, but my sink is getting data only from one of the sources.
Is there any way wherein I can get data from both the sources with a single sink, as is required in my case?
Please let me know.
TIA.
Regards,
Prashanth.
______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
Re: Flume-NG -- 2 Sources and a single Sink
Posted by Israel Ekpo <is...@aicer.org>.
Prashanth,
I am glad things are working out for you.
Good luck with your project.
On Wed, Apr 3, 2013 at 12:06 AM, Babu, Prashanth <Prashanth.Babu@nttdata.com
> wrote:
> Thanks for your inputs Israel.****
>
> As of now, both my sources are using separate channels [Memory] and my
> custom sink is just getting all the events and pushing them to a flat file
> on the disk.****
>
> I am trying to learn and understand Flume-NG with this prototype.****
>
> I will go thru the Flume User Guide as well.****
>
> ** **
>
> Regards,****
>
> Prashanth.****
>
> ** **
>
> *From:* Israel Ekpo [mailto:israel@aicer.org]
> *Sent:* Tuesday, April 02, 2013 4:34 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Flume-NG -- 2 Sources and a single Sink****
>
> ** **
>
> Checkout the user guide for Flume NG instead
>
> http://flume.apache.org/FlumeUserGuide.html
>
> I would suggest that you use separate channels (preferably the
> FileChannel) for each source.
>
> I am not sure what your custom sink looks like, but if it capable of
> handling multiple incoming requests simultaneously like MongoDB, Elastic
> Search, Apache Solr, MySQL or CouchBase you should still be able to set
> them up as separate sinks but technically all the events are going to the
> same centralized store.
>
> The user guide is more up to date and contains more examples that could
> help you decide which way to go.****
>
> ** **
>
> Review the various examples and data flow designs.****
>
> On Tue, Apr 2, 2013 at 4:25 AM, Babu, Prashanth <
> Prashanth.Babu@nttdata.com> wrote:****
>
> Hi,
>
> I have been using Flume-OG till now and currently I am evaluating Flume-NG.
> I have followed the guidelines in
> http://flume.apache.org/FlumeDeveloperGuide.html.
> In my use case, I have 2 Sources on a machine, but a single custom sink.
> 2 Sources are tails on 2 different files which are configured in
> flume.conf.
> I have tried giving the same channel for both the sources, but my sink is
> getting data only from one of the sources.
> Is there any way wherein I can get data from both the sources with a
> single sink, as is required in my case?
> Please let me know.
> TIA.
>
> Regards,
> Prashanth.
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding****
>
> ** **
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding
>
RE: Flume-NG -- 2 Sources and a single Sink
Posted by "Babu, Prashanth" <Pr...@nttdata.com>.
Thanks for your inputs Israel.
As of now, both my sources are using separate channels [Memory] and my custom sink is just getting all the events and pushing them to a flat file on the disk.
I am trying to learn and understand Flume-NG with this prototype.
I will go thru the Flume User Guide as well.
Regards,
Prashanth.
From: Israel Ekpo [mailto:israel@aicer.org]
Sent: Tuesday, April 02, 2013 4:34 PM
To: user@flume.apache.org
Subject: Re: Flume-NG -- 2 Sources and a single Sink
Checkout the user guide for Flume NG instead
http://flume.apache.org/FlumeUserGuide.html
I would suggest that you use separate channels (preferably the FileChannel) for each source.
I am not sure what your custom sink looks like, but if it capable of handling multiple incoming requests simultaneously like MongoDB, Elastic Search, Apache Solr, MySQL or CouchBase you should still be able to set them up as separate sinks but technically all the events are going to the same centralized store.
The user guide is more up to date and contains more examples that could help you decide which way to go.
Review the various examples and data flow designs.
On Tue, Apr 2, 2013 at 4:25 AM, Babu, Prashanth <Pr...@nttdata.com>> wrote:
Hi,
I have been using Flume-OG till now and currently I am evaluating Flume-NG.
I have followed the guidelines in http://flume.apache.org/FlumeDeveloperGuide.html.
In my use case, I have 2 Sources on a machine, but a single custom sink.
2 Sources are tails on 2 different files which are configured in flume.conf.
I have tried giving the same channel for both the sources, but my sink is getting data only from one of the sources.
Is there any way wherein I can get data from both the sources with a single sink, as is required in my case?
Please let me know.
TIA.
Regards,
Prashanth.
______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
Re: Flume-NG -- 2 Sources and a single Sink
Posted by Israel Ekpo <is...@aicer.org>.
Checkout the user guide for Flume NG instead
http://flume.apache.org/FlumeUserGuide.html
I would suggest that you use separate channels (preferably the FileChannel)
for each source.
I am not sure what your custom sink looks like, but if it capable of
handling multiple incoming requests simultaneously like MongoDB, Elastic
Search, Apache Solr, MySQL or CouchBase you should still be able to set
them up as separate sinks but technically all the events are going to the
same centralized store.
The user guide is more up to date and contains more examples that could
help you decide which way to go.
Review the various examples and data flow designs.
On Tue, Apr 2, 2013 at 4:25 AM, Babu, Prashanth
<Pr...@nttdata.com>wrote:
> Hi,
>
> I have been using Flume-OG till now and currently I am evaluating Flume-NG.
> I have followed the guidelines in
> http://flume.apache.org/FlumeDeveloperGuide.html.
> In my use case, I have 2 Sources on a machine, but a single custom sink.
> 2 Sources are tails on 2 different files which are configured in
> flume.conf.
> I have tried giving the same channel for both the sources, but my sink is
> getting data only from one of the sources.
> Is there any way wherein I can get data from both the sources with a
> single sink, as is required in my case?
> Please let me know.
> TIA.
>
> Regards,
> Prashanth.
>
> ______________________________________________________________________
> Disclaimer:This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding
>