You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Abhijeet Pathak <Ab...@kpitcummins.com> on 2012/12/28 07:36:38 UTC
Source and Sink on different machine
Hi,
It was possible to have source and sinks on different machines using Flume 0.9x.
Is it possible to have source and sinks on different machines in Flume-NG (1.3.0)?
I want to use tail source on one machine and hbase sink on another machine.
How should I go about it?
Thanks,
Abhijeet
Re: Source and Sink on different machine
Posted by Nitin Pawar <ni...@gmail.com>.
yes thats doable
what you are trying to do is agnet1 ->agenet2|agent3 -> hbase
second portion of the flume part you can have it on same host where you
have logs being generated.
you can look at flume failover configuration at
http://flume.apache.org/FlumeUserGuide.html#flume-sink-processors
also your question on stackoverflow is answered by Dmitry in Detail
On Fri, Dec 28, 2012 at 1:23 PM, Abhijeet Pathak <
Abhijeet.Pathak@kpitcummins.com> wrote:
> I want to setup a failover configuration.****
>
> ** **
>
> On Machine1 I’ve a source.****
>
> On Machine2 & Machine3, I have (hbase) sinks.****
>
> I want to configure them in failover mode. That is, by default events
> should go to sink on Machine2’s sink. But if Machine2 fails then they
> should automatically go to Machine3’s sink.****
>
> ** **
>
> Is that possible in Flume-NG? How?****
>
> ** **
>
> Thanks,****
>
> Abhijeet****
>
> ** **
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* Friday, December 28, 2012 1:18 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Source and Sink on different machine****
>
> ** **
>
> if I understood it correctly, you want to put your log data into hbase
> which is hosted remotely? ****
>
> ** **
>
> yes thats easily doable. ****
>
> ** **
>
> Flume-NG does everything alone instead of older master-slave architecture.
> It has currently source channel and sink. you can collect logs on a machine
> via an agent and using same agent you can sink them into hbase on a remote
> host ****
>
> ** **
>
> On Fri, Dec 28, 2012 at 12:06 PM, Abhijeet Pathak <
> Abhijeet.Pathak@kpitcummins.com> wrote:****
>
> Hi,****
>
> ****
>
> It was possible to have source and sinks on different machines using Flume
> 0.9x.****
>
> ****
>
> Is it possible to have source and sinks on different machines in Flume-NG
> (1.3.0)?****
>
> ****
>
> I want to use tail source on one machine and hbase sink on another machine.
> ****
>
> How should I go about it?****
>
> ****
>
> Thanks,****
>
> Abhijeet****
>
>
> This message contains information that may be privileged or confidential
> and is the property of the KPIT Cummins Infosystems Ltd. It is intended
> only for the person to whom it is addressed. If you are not the intended
> recipient, you are not authorized to read, print, retain copy, disseminate,
> distribute, or use this message or any part thereof. If you receive this
> message in error, please notify the sender immediately and delete all
> copies of this message. KPIT Cummins Infosystems Ltd. does not accept any
> liability for virus infected mails. ****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
>
> This message contains information that may be privileged or confidential
> and is the property of the KPIT Cummins Infosystems Ltd. It is intended
> only for the person to whom it is addressed. If you are not the intended
> recipient, you are not authorized to read, print, retain copy, disseminate,
> distribute, or use this message or any part thereof. If you receive this
> message in error, please notify the sender immediately and delete all
> copies of this message. KPIT Cummins Infosystems Ltd. does not accept any
> liability for virus infected mails.
>
--
Nitin Pawar
Re: Source and Sink on different machine
Posted by Juhani Connolly <ju...@cyberagent.co.jp>.
You connect up the various machines using avro sink/source.
You set up 2 & 3 with avro source, a channel and hdfs sink.
Then you set up machine one with whatever source, channel and two avro
sinks, and a sink
processor(http://flume.apache.org/FlumeUserGuide.html#flume-sink-processors)
of your personal preference. If you use failover when the main sink
fails, events will go through the backup instead.
On 12/28/2012 04:53 PM, Abhijeet Pathak wrote:
>
> I want to setup a failover configuration.
>
> On Machine1 I've a source.
>
> On Machine2 & Machine3, I have (hbase) sinks.
>
> I want to configure them in failover mode. That is, by default events
> should go to sink on Machine2's sink. But if Machine2 fails then they
> should automatically go to Machine3's sink.
>
> Is that possible in Flume-NG? How?
>
> Thanks,
>
> Abhijeet
>
> *From:*Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* Friday, December 28, 2012 1:18 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Source and Sink on different machine
>
> if I understood it correctly, you want to put your log data into hbase
> which is hosted remotely?
>
> yes thats easily doable.
>
> Flume-NG does everything alone instead of older master-slave
> architecture. It has currently source channel and sink. you can
> collect logs on a machine via an agent and using same agent you can
> sink them into hbase on a remote host
>
> On Fri, Dec 28, 2012 at 12:06 PM, Abhijeet Pathak
> <Abhijeet.Pathak@kpitcummins.com
> <ma...@kpitcummins.com>> wrote:
>
> Hi,
>
> It was possible to have source and sinks on different machines using
> Flume 0.9x.
>
> Is it possible to have source and sinks on different machines in
> Flume-NG (1.3.0)?
>
> I want to use tail source on one machine and hbase sink on another
> machine.
>
> How should I go about it?
>
> Thanks,
>
> Abhijeet
>
>
> This message contains information that may be privileged or
> confidential and is the property of the KPIT Cummins Infosystems Ltd.
> It is intended only for the person to whom it is addressed. If you are
> not the intended recipient, you are not authorized to read, print,
> retain copy, disseminate, distribute, or use this message or any part
> thereof. If you receive this message in error, please notify the
> sender immediately and delete all copies of this message. KPIT Cummins
> Infosystems Ltd. does not accept any liability for virus infected mails.
>
>
>
> --
> Nitin Pawar
>
>
> This message contains information that may be privileged or
> confidential and is the property of the KPIT Cummins Infosystems Ltd.
> It is intended only for the person to whom it is addressed. If you are
> not the intended recipient, you are not authorized to read, print,
> retain copy, disseminate, distribute, or use this message or any part
> thereof. If you receive this message in error, please notify the
> sender immediately and delete all copies of this message. KPIT Cummins
> Infosystems Ltd. does not accept any liability for virus infected mails.
RE: Source and Sink on different machine
Posted by Abhijeet Pathak <Ab...@kpitcummins.com>.
I want to setup a failover configuration.
On Machine1 I've a source.
On Machine2 & Machine3, I have (hbase) sinks.
I want to configure them in failover mode. That is, by default events should go to sink on Machine2's sink. But if Machine2 fails then they should automatically go to Machine3's sink.
Is that possible in Flume-NG? How?
Thanks,
Abhijeet
From: Nitin Pawar [mailto:nitinpawar432@gmail.com]
Sent: Friday, December 28, 2012 1:18 PM
To: user@flume.apache.org
Subject: Re: Source and Sink on different machine
if I understood it correctly, you want to put your log data into hbase which is hosted remotely?
yes thats easily doable.
Flume-NG does everything alone instead of older master-slave architecture. It has currently source channel and sink. you can collect logs on a machine via an agent and using same agent you can sink them into hbase on a remote host
On Fri, Dec 28, 2012 at 12:06 PM, Abhijeet Pathak <Ab...@kpitcummins.com>> wrote:
Hi,
It was possible to have source and sinks on different machines using Flume 0.9x.
Is it possible to have source and sinks on different machines in Flume-NG (1.3.0)?
I want to use tail source on one machine and hbase sink on another machine.
How should I go about it?
Thanks,
Abhijeet
This message contains information that may be privileged or confidential and is the property of the KPIT Cummins Infosystems Ltd. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message. KPIT Cummins Infosystems Ltd. does not accept any liability for virus infected mails.
--
Nitin Pawar
Re: Source and Sink on different machine
Posted by Nitin Pawar <ni...@gmail.com>.
if I understood it correctly, you want to put your log data into hbase
which is hosted remotely?
yes thats easily doable.
Flume-NG does everything alone instead of older master-slave architecture.
It has currently source channel and sink. you can collect logs on a machine
via an agent and using same agent you can sink them into hbase on a remote
host
On Fri, Dec 28, 2012 at 12:06 PM, Abhijeet Pathak <
Abhijeet.Pathak@kpitcummins.com> wrote:
> Hi,****
>
> ** **
>
> It was possible to have source and sinks on different machines using Flume
> 0.9x.****
>
> ** **
>
> Is it possible to have source and sinks on different machines in Flume-NG
> (1.3.0)?****
>
> ** **
>
> I want to use tail source on one machine and hbase sink on another machine.
> ****
>
> How should I go about it?****
>
> ** **
>
> Thanks,****
>
> Abhijeet****
>
> This message contains information that may be privileged or confidential
> and is the property of the KPIT Cummins Infosystems Ltd. It is intended
> only for the person to whom it is addressed. If you are not the intended
> recipient, you are not authorized to read, print, retain copy, disseminate,
> distribute, or use this message or any part thereof. If you receive this
> message in error, please notify the sender immediately and delete all
> copies of this message. KPIT Cummins Infosystems Ltd. does not accept any
> liability for virus infected mails.
>
--
Nitin Pawar