You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by terreyshih <te...@gmail.com> on 2014/09/23 09:09:35 UTC
replicating avrò data to two sinks ?
HI,
I am trying to replicate the data coming in from avro source to two channels connecting to 2 sinks. One goes directory to Hbase, and one goes to some backend process.
The following is my configuration file for the data coming in and from the “main.conf”, I replicate the data and send it to other two avro sinks. “sink.conf” is an example conf file for one of the sinks I am using.
I actually start 3 JVMs on the same machinate to represent 3 agents. This seems straight forward according to the documentation.
However, I don’t seem to be able to get it to work. Any ideas ?
thx
#main.conf
server-agent.sources = avrò-Src
server-agent.sinks = hbase-sink be-sink
server-agent.channels = hbaseC beC
server-agent.sources.avro-Src.type = avro
server-agent.sources.avro-Src.bind = 127.0.0.1
server-agent.sources.avro-Src.port = 5000
server-agent.sources.avro-Src.thread = 10
server-agent.sinks.hbase-sink.type = avro
server-agent.sinks.hbase-sink.channel = hbaseC
server-agent.sinks.hbase-sink.hostname = 127.0.0.1
server-agent.sinks.hbase-sink.port = 5001
server-agent.sinks.hbase-sink.batch-size = 1000
server-agent.sinks.be-sink.type = avro
server-agent.sinks.be-sink.channel = beC
server-agent.sinks.be-sink.hostname = 127.0.0.1
server-agent.sinks.be-sink.port = 5002
server-agent.sinks.be-sink.batch-size = 1000
server-agent.sources.avro-Src.channels = hbaseC beC
server-agent.sources.avro-Src.selector.type = replicating
#sink.conf
be.sources = avro-Src
be.sinks = mySink
be.channels = beC
be.sources.avro-Src.type = avro
be.sources.avro-Src.bind = 127.0.0.1
be.sources.avro-Src.port = 5002
be.sources.avro-Src.channels = beC
be.channels.beC.type = memory
be.channels.beC.capacity = 2000
be.channels.beC.transactionCapacity = 200
be.sinks.mySink.type = com.mycompany.mySink
be.sinks.mySink.channel = beC
Re: replicating avrò data to two sinks ?
Posted by terreyshih <te...@gmail.com>.
never mind. I forgo to put channel information in main.conf so the main.conf was invalid.
On Sep 23, 2014, at 12:09 AM, terreyshih <te...@gmail.com> wrote:
> HI,
>
> I am trying to replicate the data coming in from avro source to two channels connecting to 2 sinks. One goes directory to Hbase, and one goes to some backend process.
>
> The following is my configuration file for the data coming in and from the “main.conf”, I replicate the data and send it to other two avro sinks. “sink.conf” is an example conf file for one of the sinks I am using.
>
> I actually start 3 JVMs on the same machinate to represent 3 agents. This seems straight forward according to the documentation.
>
> However, I don’t seem to be able to get it to work. Any ideas ?
>
> thx
>
> #main.conf
>
> server-agent.sources = avrò-Src
> server-agent.sinks = hbase-sink be-sink
> server-agent.channels = hbaseC beC
>
> server-agent.sources.avro-Src.type = avro
> server-agent.sources.avro-Src.bind = 127.0.0.1
> server-agent.sources.avro-Src.port = 5000
> server-agent.sources.avro-Src.thread = 10
>
> server-agent.sinks.hbase-sink.type = avro
> server-agent.sinks.hbase-sink.channel = hbaseC
> server-agent.sinks.hbase-sink.hostname = 127.0.0.1
> server-agent.sinks.hbase-sink.port = 5001
> server-agent.sinks.hbase-sink.batch-size = 1000
>
> server-agent.sinks.be-sink.type = avro
> server-agent.sinks.be-sink.channel = beC
> server-agent.sinks.be-sink.hostname = 127.0.0.1
> server-agent.sinks.be-sink.port = 5002
> server-agent.sinks.be-sink.batch-size = 1000
>
> server-agent.sources.avro-Src.channels = hbaseC beC
> server-agent.sources.avro-Src.selector.type = replicating
>
>
> #sink.conf
> be.sources = avro-Src
> be.sinks = mySink
> be.channels = beC
>
> be.sources.avro-Src.type = avro
> be.sources.avro-Src.bind = 127.0.0.1
> be.sources.avro-Src.port = 5002
> be.sources.avro-Src.channels = beC
>
> be.channels.beC.type = memory
> be.channels.beC.capacity = 2000
> be.channels.beC.transactionCapacity = 200
> be.sinks.mySink.type = com.mycompany.mySink
> be.sinks.mySink.channel = beC
>
>