You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Alain B." <al...@hotmail.com> on 2013/01/22 11:51:38 UTC
Multiplexing to multiple JdbcChannel (Derby) + event header ?
Hi,
I'd like to use the multiplexing feature of Flume-NG, routing events from one source to 2 jdbc-channels.
My question is: will these 2 channels store their events in separate derby DB by default or do I need to configure my 2 jdbc-channels with specific properties in order to get 2 embedded derby DB started ?
Moreover, the multiplexing feature can base its routing on headers. But no information is given on how to setup custom headers (the example in the documentation uses a "State" header).
I'm using a Log4jAppender to write to an AvroSource.
# list the sources, sinks and channels in the agent
agent_foo.sources = avro-AppSrv-source1
agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
agent_foo.channels = jdbc-channel-1 jdbc-channel-2
# set channels for source
agent_foo.sources.avro-AppSrv-source1.channels = jdbc-channel-1 jdbc-channel-2
# set channel for sinks
agent_foo.sinks.hdfs-Cluster1-sink1.channel = jdbc-channel-1
agent_foo.sinks.avro-forward-sink2.channel = jdbc-channel-2
# channel selector configuration
agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
agent_foo.sources.avro-AppSrv-source1.selector.header = State
agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = jdbc-channel-1
agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = jdbc-channel-2
agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = jdbc-channel-1 jdbc-channel-2
agent_foo.sources.avro-AppSrv-source1.selector.default = jdbc-channel-1
Thanks for your help,
Best regards
RE: Multiplexing to multiple JdbcChannel (Derby) + event header ?
Posted by "Alain B." <al...@hotmail.com>.
Great ! I'll try and implement all this.
Thank you for your quick responses.
Best regards,
Alain
________________________________
> Date: Wed, 23 Jan 2013 00:05:56 -0800
> From: hshreedharan@cloudera.com
> To: user@flume.apache.org
> Subject: Re: Multiplexing to multiple JdbcChannel (Derby) + event header ?
>
> Alan,
>
> This is the path your event would follow:
> log4jappender -> source -> interceptor -> selector (technically this is
> the channel processor, selector simply selects the channels) -> channel
>
>
> Hari
>
> --
> Hari Shreedharan
>
>
> On Wednesday, January 23, 2013 at 12:01 AM, Alain B. wrote:
>
> log4jappender -> source -> interceptor -> selector -> channel
>
Re: Multiplexing to multiple JdbcChannel (Derby) + event header
?
Posted by Hari Shreedharan <hs...@cloudera.com>.
Alan,
This is the path your event would follow:
log4jappender -> source -> interceptor -> selector (technically this is the channel processor, selector simply selects the channels) -> channel
Hari
--
Hari Shreedharan
On Wednesday, January 23, 2013 at 12:01 AM, Alain B. wrote:
> log4jappender -> source -> interceptor -> selector -> channel
RE: Multiplexing to multiple JdbcChannel (Derby) + event header ?
Posted by "Alain B." <al...@hotmail.com>.
Ok, I got it thank you. I'll follow your advice and go for file channels instead.
W.r.t. the interceptor, I'll implement a custom org.apache.flume.interceptor class.
Just one more question: interceptors are configured in the source configuration. But are they executed before the events are inserted in the source or when they get out of the source (before going to the channel). If it's the later case, then I suppose it's done before any selector does its job (multiplexing on the basis of a header in my specific case).
So is it :
log4jappender -> interceptor -> source -> selector -> channel
or
log4jappender -> source -> interceptor -> selector -> channel
Thanks,
Alain
________________________________
> Date: Tue, 22 Jan 2013 10:54:49 -0800
> Subject: Re: Multiplexing to multiple JdbcChannel (Derby) + event header ?
> From: jlord@cloudera.com
> To: user@flume.apache.org
>
>
> On Tue, Jan 22, 2013 at 2:51 AM, Alain B.
> <al...@hotmail.com>> wrote:
> My question is: will these 2 channels store their events in separate
> derby DB by default or do I need to configure my 2 jdbc-channels with
> specific properties in order to get 2 embedded derby DB started ?
>
>
> By default they will use the same derby DB.
> You can specify the location with the following property.
>
> a1.channels.c1.driver.url =
> jdbc:derby:/var/run/flume-ng/channel1/.flume/jdbc-channel/db;create=true
>
> a1.channels.c2.driver.url =
> jdbc:derby:/var/run/flume-ng/channel2/.flume/jdbc-channel/db;create=true
>
> As Hari already mentioned we Highly recommend the file channel for
> durability over the jdbc channel.
Re: Multiplexing to multiple JdbcChannel (Derby) + event header ?
Posted by Jeff Lord <jl...@cloudera.com>.
On Tue, Jan 22, 2013 at 2:51 AM, Alain B. <al...@hotmail.com> wrote:
> My question is: will these 2 channels store their events in separate derby
> DB by default or do I need to configure my 2 jdbc-channels with specific
> properties in order to get 2 embedded derby DB started ?
>
By default they will use the same derby DB.
You can specify the location with the following property.
a1.channels.c1.driver.url = jdbc:derby:/var/run/flume-ng/channel1/
.flume/jdbc-channel/db;create=true
a1.channels.c2.driver.url = jdbc:derby:/var/run/flume-ng/channel2/
.flume/jdbc-channel/db;create=true
As Hari already mentioned we Highly recommend the file channel for
durability over the jdbc channel.
Re: Multiplexing to multiple JdbcChannel (Derby) + event header ?
Posted by Hari Shreedharan <hs...@apache.org>.
I'd recommend using the FileChannel instead of the JDBC Channel. The
FileChannel would give you better performance and the same reliability
guarantees as the JDBC Channel.
To answer your question - you can insert custom headers in an agent using
Interceptors. You can write an interceptor which inserts the headers based
on the content of the event etc. Interceptors are available in version 1.2+.
Hari
On Tue, Jan 22, 2013 at 2:51 AM, Alain B. <al...@hotmail.com> wrote:
>
> Hi,
> I'd like to use the multiplexing feature of Flume-NG, routing events from
> one source to 2 jdbc-channels.
> My question is: will these 2 channels store their events in separate derby
> DB by default or do I need to configure my 2 jdbc-channels with specific
> properties in order to get 2 embedded derby DB started ?
> Moreover, the multiplexing feature can base its routing on headers. But no
> information is given on how to setup custom headers (the example in the
> documentation uses a "State" header).
> I'm using a Log4jAppender to write to an AvroSource.
>
> # list the sources, sinks and channels in the agent
> agent_foo.sources = avro-AppSrv-source1
> agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
> agent_foo.channels = jdbc-channel-1 jdbc-channel-2
>
> # set channels for source
> agent_foo.sources.avro-AppSrv-source1.channels = jdbc-channel-1
> jdbc-channel-2
>
> # set channel for sinks
> agent_foo.sinks.hdfs-Cluster1-sink1.channel = jdbc-channel-1
> agent_foo.sinks.avro-forward-sink2.channel = jdbc-channel-2
>
> # channel selector configuration
> agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
> agent_foo.sources.avro-AppSrv-source1.selector.header = State
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = jdbc-channel-1
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = jdbc-channel-2
> agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = jdbc-channel-1
> jdbc-channel-2
> agent_foo.sources.avro-AppSrv-source1.selector.default = jdbc-channel-1
>
> Thanks for your help,
> Best regards