You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Michael Moores <mm...@realnetworks.com> on 2014/06/27 23:00:42 UTC
ZookeeperRoutePolicy with multiple routes and different znodes- one
always shuts down
I have two routes that I integrated with ZooKeeperRoutePolicy to establish basic distributed route election.
(Using camel 2.13.1)
When used with one route instance, election/failover works fine.
When I set up multiple routes with multiple instances of ZooKeeperRoutePolicy, one route almost always gets immediately shut down but NOT 100% of the time.
The behavior seems nondeterministic- like a race condition.
Each RouteBuilder instance has a separate instance of a ZooKeeperRoutePolicy that uses the same server but a different zookeeper znode:
Like this:
Route 1: zookeeper URI: localhost:2181/scheduler-node-election/test
Route 2: zookeeper URI: localhost:2181/scheduler-node-election/googleplay
In the zookeeper CLI:
[zk: localhost:2181(CONNECTED) 6] ls /scheduler-node-election
[googleplay, test]
(I have one zookeeper node running)
I’m seeing that most of the time one route (either one) just gets shutdown during Spring initialization, while the other one runs. But once in a while, both routes will run.
In this case the “googleplay” route is shut down and the associated session is closed.
Here’s the Camel/Zookeeper logging:
[timer://test-#2] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=5000 watcher=org.apache.camel.component.zookeeper.ConnectionHolder@733ed393
[//googleplay-#1] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=5000 watcher=org.apache.camel.component.zookeeper.ConnectionHolder@21b49301
[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x146de3b813e0032, negotiated timeout = 5000
[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x146de3b813e0031, negotiated timeout = 5000
[timer://test-#2] o.a.c.c.zookeeper.ZookeeperProducer : Node '/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a2623690' did not exist, creating it.
[//googleplay-#1] o.a.c.c.zookeeper.ZookeeperProducer : Node '/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b44e008854db' did not exist, creating it.
[timer://test-#2] o.a.c.c.z.policy.ZooKeeperElection : Candidate node '/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a2623690' has been created
[//googleplay-#1] o.a.c.c.z.policy.ZooKeeperElection : Candidate node '/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b44e008854db' has been created
[//googleplay-#1] o.a.camel.spring.SpringCamelContext : Route: election-route-localhos started and consuming from: Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Starting to graceful shutdown 1 routes (timeout 300 seconds)
[ShutdownTask-#4] o.a.camel.impl.DefaultShutdownStrategy : Route: election-route-localhos shutdown complete, was consuming from: Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Graceful shutdown of 1 routes completed in 0 seconds
[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route: election-route-localhos is stopped, was consuming from: Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
[-#1-EventThread] org.apache.zookeeper.ClientCnxn : EventThread shut down
[timer://test-#2] org.apache.zookeeper.ZooKeeper : Session: 0x146de3b813e0031 closed
[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route: election-route-localhos is shutdown and removed, was consuming from: Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route: election-route-localhos started and consuming from: Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/test]
My route builder code is below. I call it twice, once for vendor = “test”, and once for vendor = “googleplay”.
private RouteBuilder scheduledEventBatchExecutionTimerRoute(
final String vendor,
final int batchSize,
final int timerPeriodSeconds,
final boolean autoStart) {
log.info("scheduledEventBatchExecutionTimerRoute: vendor={}, doStart={}, queryInterval={}s, batchSize={}",
vendor, autoStart, timerPeriodSeconds, batchSize);
return new RouteBuilder() {
@Override
public void configure() throws Exception {
from("timer://" + vendor + "?period=" + timerPeriodSeconds + "s")
.autoStartup(autoStart)
.routePolicy(electionRoutePolicy(vendor))
.setHeader("vendor", constant(vendor)).setHeader("batchSize", constant(batchSize))
.beanRef("scheduleEventRepository", "findExecutableEvents(entitlement.create.${in.header.vendor}, ${in.header.batchSize})")
.choice()
.when().simple("${body.size} == 0")
.log(LoggingLevel.DEBUG, "Route: vendor=${in.header.vendor}: No items to process")
.when().simple("${body.size} > 0")
.to(ROUTE_DIRECT_EVENT_BATCH_EXECUTION + "-" + vendor)
.endChoice()
.routeId(vendor + "-timer-execution-route");
}
};
}
private RoutePolicy electionRoutePolicy(String vendor) {
RoutePolicy routePolicy;
if (doUseDistributedRouteElection) {
log.info("Creating zookeeper distributed route election policy for {}", vendor);
// Ensure that only one node is processing work for the specified vendor.
// See http://camel.apache.org/zookeeper.html - distributed route policy.
routePolicy = new ZooKeeperRoutePolicy(zookeeperUri + "scheduler-node-election/" + vendor, 1);
} else {
// Just return a default route policy that does nothing.
log.info("No-op route policy for {}", vendor);
routePolicy = new RoutePolicySupport() {
@Override
public void onInit(Route route) {
super.onInit(route);
}
};
}
return routePolicy;
}
Re: ZookeeperRoutePolicy with multiple routes and different znodes-
one always shuts down
Posted by Michael Moores <mm...@realnetworks.com>.
Thanks Scott, for the info.
Indeed - it looks like I have run into CAMEL-7501.
Only one election policy- so one route is actually causing another route
to be shut down.
On 6/27/14, 2:53 PM, "Scott Stults" <ss...@opensourceconnections.com>
wrote:
>I was doing something similar, and what I noticed in JConsole was that
>there was only ever one zookeeper election route created even when I had
>multiple route policies. There are two similar Jira issues related to
>this: CAMEL-7501 and CAMEL-5627. They might be the source of your trouble.
>
>Hope that helps. If it really is a bug in the zk routing policy then that
>might save you some hair-pulling in your own code.
>
>-Scott
>
>
>On Fri, Jun 27, 2014 at 5:41 PM, Michael Moores <mm...@realnetworks.com>
>wrote:
>
>> I tried an experiment that seems to point to a race condition in the
>>Camel
>> ZooKeeper component.
>>
>> I initialized the routes with autoStart=false, then started each route
>> manually.
>> When I do this everything works fine - no immediate shutdown of one of
>>the
>> routes.. I think it might have something to do with
>> ZooKeeperElection.awaitElectionResults(), but not sure.
>>
>>
>>
>> On 6/27/14, 2:00 PM, "Michael Moores" <mm...@realnetworks.com> wrote:
>>
>> >I have two routes that I integrated with ZooKeeperRoutePolicy to
>> >establish basic distributed route election.
>> >(Using camel 2.13.1)
>> >
>> >When used with one route instance, election/failover works fine.
>> >When I set up multiple routes with multiple instances of
>> >ZooKeeperRoutePolicy, one route almost always gets immediately shut
>>down
>> >but NOT 100% of the time.
>> >The behavior seems nondeterministic- like a race condition.
>> >
>> >Each RouteBuilder instance has a separate instance of a
>> >ZooKeeperRoutePolicy that uses the same server but a different
>>zookeeper
>> >znode:
>> >Like this:
>> >Route 1: zookeeper URI: localhost:2181/scheduler-node-election/test
>> >Route 2: zookeeper URI:
>>localhost:2181/scheduler-node-election/googleplay
>> >
>> >In the zookeeper CLI:
>> >[zk: localhost:2181(CONNECTED) 6] ls /scheduler-node-election
>> >[googleplay, test]
>> >
>> >(I have one zookeeper node running)
>> >
>> >
>> >I¹m seeing that most of the time one route (either one) just gets
>> >shutdown during Spring initialization, while the other one runs. But
>> >once in a while, both routes will run.
>> >In this case the ³googleplay² route is shut down and the associated
>> >session is closed.
>> >
>> >Here¹s the Camel/Zookeeper logging:
>> >[timer://test-#2] org.apache.zookeeper.ZooKeeper : Initiating
>> >client connection, connectString=127.0.0.1:2181 sessionTimeout=5000
>> >watcher=org.apache.camel.component.zookeeper.ConnectionHolder@733ed393
>> >[//googleplay-#1] org.apache.zookeeper.ZooKeeper : Initiating
>> >client connection, connectString=127.0.0.1:2181 sessionTimeout=5000
>> >watcher=org.apache.camel.component.zookeeper.ConnectionHolder@21b49301
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening
>> >socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt
>>to
>> >authenticate using SASL (unknown error)
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening
>> >socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt
>>to
>> >authenticate using SASL (unknown error)
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket
>> >connection established to 127.0.0.1/127.0.0.1:2181, initiating session
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket
>> >connection established to 127.0.0.1/127.0.0.1:2181, initiating session
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session
>> >establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid =
>> >0x146de3b813e0032, negotiated timeout = 5000
>> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session
>> >establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid =
>> >0x146de3b813e0031, negotiated timeout = 5000
>> >[timer://test-#2] o.a.c.c.zookeeper.ZookeeperProducer : Node
>>
>>>'/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a26
>>>23
>> >690' did not exist, creating it.
>> >[//googleplay-#1] o.a.c.c.zookeeper.ZookeeperProducer : Node
>>
>>>'/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b
>>>44
>> >e008854db' did not exist, creating it.
>> >[timer://test-#2] o.a.c.c.z.policy.ZooKeeperElection : Candidate
>> >node
>>
>>>'/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a26
>>>23
>> >690' has been created
>> >[//googleplay-#1] o.a.c.c.z.policy.ZooKeeperElection : Candidate
>> >node
>>
>>>'/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b
>>>44
>> >e008854db' has been created
>> >[//googleplay-#1] o.a.camel.spring.SpringCamelContext : Route:
>> >election-route-localhos started and consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>> >[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Starting
>>to
>> >graceful shutdown 1 routes (timeout 300 seconds)
>> >[ShutdownTask-#4] o.a.camel.impl.DefaultShutdownStrategy : Route:
>> >election-route-localhos shutdown complete, was consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>> >[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Graceful
>> >shutdown of 1 routes completed in 0 seconds
>> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>> >election-route-localhos is stopped, was consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>> >[-#1-EventThread] org.apache.zookeeper.ClientCnxn :
>>EventThread
>> >shut down
>> >[timer://test-#2] org.apache.zookeeper.ZooKeeper : Session:
>> >0x146de3b813e0031 closed
>> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>> >election-route-localhos is shutdown and removed, was consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>> >election-route-localhos started and consuming from:
>> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/test]
>> >
>> >
>> >
>> >My route builder code is below. I call it twice, once for vendor =
>> >³test², and once for vendor = ³googleplay².
>> >
>> >private RouteBuilder scheduledEventBatchExecutionTimerRoute(
>> > final String vendor,
>> > final int batchSize,
>> > final int timerPeriodSeconds,
>> > final boolean autoStart) {
>> >
>> > log.info("scheduledEventBatchExecutionTimerRoute: vendor={},
>> >doStart={}, queryInterval={}s, batchSize={}",
>> > vendor, autoStart, timerPeriodSeconds, batchSize);
>> >
>> > return new RouteBuilder() {
>> > @Override
>> > public void configure() throws Exception {
>> >
>> > from("timer://" + vendor + "?period=" +
>> >timerPeriodSeconds + "s")
>> > .autoStartup(autoStart)
>> > .routePolicy(electionRoutePolicy(vendor))
>> > .setHeader("vendor",
>> >constant(vendor)).setHeader("batchSize", constant(batchSize))
>> > .beanRef("scheduleEventRepository",
>> >"findExecutableEvents(entitlement.create.${in.header.vendor},
>> >${in.header.batchSize})")
>> >
>> > .choice()
>> > .when().simple("${body.size} == 0")
>> > .log(LoggingLevel.DEBUG, "Route:
>> >vendor=${in.header.vendor}: No items to process")
>> > .when().simple("${body.size} > 0")
>> > .to(ROUTE_DIRECT_EVENT_BATCH_EXECUTION + "-" +
>> >vendor)
>> >
>> > .endChoice()
>> > .routeId(vendor + "-timer-execution-route");
>> >
>> > }
>> > };
>> >
>> > }
>> >
>> > private RoutePolicy electionRoutePolicy(String vendor) {
>> >
>> > RoutePolicy routePolicy;
>> >
>> > if (doUseDistributedRouteElection) {
>> > log.info("Creating zookeeper distributed route election
>> >policy for {}", vendor);
>> > // Ensure that only one node is processing work for the
>> >specified vendor.
>> > // See http://camel.apache.org/zookeeper.html - distributed
>> >route policy.
>> > routePolicy = new ZooKeeperRoutePolicy(zookeeperUri +
>> >"scheduler-node-election/" + vendor, 1);
>> >
>> > } else {
>> > // Just return a default route policy that does nothing.
>> > log.info("No-op route policy for {}", vendor);
>> > routePolicy = new RoutePolicySupport() {
>> > @Override
>> > public void onInit(Route route) {
>> > super.onInit(route);
>> > }
>> > };
>> > }
>> >
>> > return routePolicy;
>> > }
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>
>
>--
>Scott Stults | Founder & Solutions Architect | OpenSource Connections, LLC
>| 434.409.2780
>http://www.opensourceconnections.com
Re: ZookeeperRoutePolicy with multiple routes and different znodes-
one always shuts down
Posted by Scott Stults <ss...@opensourceconnections.com>.
I was doing something similar, and what I noticed in JConsole was that
there was only ever one zookeeper election route created even when I had
multiple route policies. There are two similar Jira issues related to
this: CAMEL-7501 and CAMEL-5627. They might be the source of your trouble.
Hope that helps. If it really is a bug in the zk routing policy then that
might save you some hair-pulling in your own code.
-Scott
On Fri, Jun 27, 2014 at 5:41 PM, Michael Moores <mm...@realnetworks.com>
wrote:
> I tried an experiment that seems to point to a race condition in the Camel
> ZooKeeper component.
>
> I initialized the routes with autoStart=false, then started each route
> manually.
> When I do this everything works fine - no immediate shutdown of one of the
> routes.. I think it might have something to do with
> ZooKeeperElection.awaitElectionResults(), but not sure.
>
>
>
> On 6/27/14, 2:00 PM, "Michael Moores" <mm...@realnetworks.com> wrote:
>
> >I have two routes that I integrated with ZooKeeperRoutePolicy to
> >establish basic distributed route election.
> >(Using camel 2.13.1)
> >
> >When used with one route instance, election/failover works fine.
> >When I set up multiple routes with multiple instances of
> >ZooKeeperRoutePolicy, one route almost always gets immediately shut down
> >but NOT 100% of the time.
> >The behavior seems nondeterministic- like a race condition.
> >
> >Each RouteBuilder instance has a separate instance of a
> >ZooKeeperRoutePolicy that uses the same server but a different zookeeper
> >znode:
> >Like this:
> >Route 1: zookeeper URI: localhost:2181/scheduler-node-election/test
> >Route 2: zookeeper URI: localhost:2181/scheduler-node-election/googleplay
> >
> >In the zookeeper CLI:
> >[zk: localhost:2181(CONNECTED) 6] ls /scheduler-node-election
> >[googleplay, test]
> >
> >(I have one zookeeper node running)
> >
> >
> >I¹m seeing that most of the time one route (either one) just gets
> >shutdown during Spring initialization, while the other one runs. But
> >once in a while, both routes will run.
> >In this case the ³googleplay² route is shut down and the associated
> >session is closed.
> >
> >Here¹s the Camel/Zookeeper logging:
> >[timer://test-#2] org.apache.zookeeper.ZooKeeper : Initiating
> >client connection, connectString=127.0.0.1:2181 sessionTimeout=5000
> >watcher=org.apache.camel.component.zookeeper.ConnectionHolder@733ed393
> >[//googleplay-#1] org.apache.zookeeper.ZooKeeper : Initiating
> >client connection, connectString=127.0.0.1:2181 sessionTimeout=5000
> >watcher=org.apache.camel.component.zookeeper.ConnectionHolder@21b49301
> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening
> >socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
> >authenticate using SASL (unknown error)
> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening
> >socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
> >authenticate using SASL (unknown error)
> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket
> >connection established to 127.0.0.1/127.0.0.1:2181, initiating session
> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket
> >connection established to 127.0.0.1/127.0.0.1:2181, initiating session
> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session
> >establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid =
> >0x146de3b813e0032, negotiated timeout = 5000
> >[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session
> >establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid =
> >0x146de3b813e0031, negotiated timeout = 5000
> >[timer://test-#2] o.a.c.c.zookeeper.ZookeeperProducer : Node
> >'/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a2623
> >690' did not exist, creating it.
> >[//googleplay-#1] o.a.c.c.zookeeper.ZookeeperProducer : Node
> >'/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b44
> >e008854db' did not exist, creating it.
> >[timer://test-#2] o.a.c.c.z.policy.ZooKeeperElection : Candidate
> >node
> >'/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a2623
> >690' has been created
> >[//googleplay-#1] o.a.c.c.z.policy.ZooKeeperElection : Candidate
> >node
> >'/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b44
> >e008854db' has been created
> >[//googleplay-#1] o.a.camel.spring.SpringCamelContext : Route:
> >election-route-localhos started and consuming from:
> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
> >[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Starting to
> >graceful shutdown 1 routes (timeout 300 seconds)
> >[ShutdownTask-#4] o.a.camel.impl.DefaultShutdownStrategy : Route:
> >election-route-localhos shutdown complete, was consuming from:
> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
> >[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Graceful
> >shutdown of 1 routes completed in 0 seconds
> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
> >election-route-localhos is stopped, was consuming from:
> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
> >[-#1-EventThread] org.apache.zookeeper.ClientCnxn : EventThread
> >shut down
> >[timer://test-#2] org.apache.zookeeper.ZooKeeper : Session:
> >0x146de3b813e0031 closed
> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
> >election-route-localhos is shutdown and removed, was consuming from:
> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
> >[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
> >election-route-localhos started and consuming from:
> >Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/test]
> >
> >
> >
> >My route builder code is below. I call it twice, once for vendor =
> >³test², and once for vendor = ³googleplay².
> >
> >private RouteBuilder scheduledEventBatchExecutionTimerRoute(
> > final String vendor,
> > final int batchSize,
> > final int timerPeriodSeconds,
> > final boolean autoStart) {
> >
> > log.info("scheduledEventBatchExecutionTimerRoute: vendor={},
> >doStart={}, queryInterval={}s, batchSize={}",
> > vendor, autoStart, timerPeriodSeconds, batchSize);
> >
> > return new RouteBuilder() {
> > @Override
> > public void configure() throws Exception {
> >
> > from("timer://" + vendor + "?period=" +
> >timerPeriodSeconds + "s")
> > .autoStartup(autoStart)
> > .routePolicy(electionRoutePolicy(vendor))
> > .setHeader("vendor",
> >constant(vendor)).setHeader("batchSize", constant(batchSize))
> > .beanRef("scheduleEventRepository",
> >"findExecutableEvents(entitlement.create.${in.header.vendor},
> >${in.header.batchSize})")
> >
> > .choice()
> > .when().simple("${body.size} == 0")
> > .log(LoggingLevel.DEBUG, "Route:
> >vendor=${in.header.vendor}: No items to process")
> > .when().simple("${body.size} > 0")
> > .to(ROUTE_DIRECT_EVENT_BATCH_EXECUTION + "-" +
> >vendor)
> >
> > .endChoice()
> > .routeId(vendor + "-timer-execution-route");
> >
> > }
> > };
> >
> > }
> >
> > private RoutePolicy electionRoutePolicy(String vendor) {
> >
> > RoutePolicy routePolicy;
> >
> > if (doUseDistributedRouteElection) {
> > log.info("Creating zookeeper distributed route election
> >policy for {}", vendor);
> > // Ensure that only one node is processing work for the
> >specified vendor.
> > // See http://camel.apache.org/zookeeper.html - distributed
> >route policy.
> > routePolicy = new ZooKeeperRoutePolicy(zookeeperUri +
> >"scheduler-node-election/" + vendor, 1);
> >
> > } else {
> > // Just return a default route policy that does nothing.
> > log.info("No-op route policy for {}", vendor);
> > routePolicy = new RoutePolicySupport() {
> > @Override
> > public void onInit(Route route) {
> > super.onInit(route);
> > }
> > };
> > }
> >
> > return routePolicy;
> > }
> >
> >
> >
> >
> >
> >
> >
>
>
--
Scott Stults | Founder & Solutions Architect | OpenSource Connections, LLC
| 434.409.2780
http://www.opensourceconnections.com
Re: ZookeeperRoutePolicy with multiple routes and different znodes-
one always shuts down
Posted by Michael Moores <mm...@realnetworks.com>.
I tried an experiment that seems to point to a race condition in the Camel
ZooKeeper component.
I initialized the routes with autoStart=false, then started each route
manually.
When I do this everything works fine - no immediate shutdown of one of the
routes.. I think it might have something to do with
ZooKeeperElection.awaitElectionResults(), but not sure.
On 6/27/14, 2:00 PM, "Michael Moores" <mm...@realnetworks.com> wrote:
>I have two routes that I integrated with ZooKeeperRoutePolicy to
>establish basic distributed route election.
>(Using camel 2.13.1)
>
>When used with one route instance, election/failover works fine.
>When I set up multiple routes with multiple instances of
>ZooKeeperRoutePolicy, one route almost always gets immediately shut down
>but NOT 100% of the time.
>The behavior seems nondeterministic- like a race condition.
>
>Each RouteBuilder instance has a separate instance of a
>ZooKeeperRoutePolicy that uses the same server but a different zookeeper
>znode:
>Like this:
>Route 1: zookeeper URI: localhost:2181/scheduler-node-election/test
>Route 2: zookeeper URI: localhost:2181/scheduler-node-election/googleplay
>
>In the zookeeper CLI:
>[zk: localhost:2181(CONNECTED) 6] ls /scheduler-node-election
>[googleplay, test]
>
>(I have one zookeeper node running)
>
>
>I¹m seeing that most of the time one route (either one) just gets
>shutdown during Spring initialization, while the other one runs. But
>once in a while, both routes will run.
>In this case the ³googleplay² route is shut down and the associated
>session is closed.
>
>Here¹s the Camel/Zookeeper logging:
>[timer://test-#2] org.apache.zookeeper.ZooKeeper : Initiating
>client connection, connectString=127.0.0.1:2181 sessionTimeout=5000
>watcher=org.apache.camel.component.zookeeper.ConnectionHolder@733ed393
>[//googleplay-#1] org.apache.zookeeper.ZooKeeper : Initiating
>client connection, connectString=127.0.0.1:2181 sessionTimeout=5000
>watcher=org.apache.camel.component.zookeeper.ConnectionHolder@21b49301
>[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening
>socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
>authenticate using SASL (unknown error)
>[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Opening
>socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
>authenticate using SASL (unknown error)
>[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket
>connection established to 127.0.0.1/127.0.0.1:2181, initiating session
>[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Socket
>connection established to 127.0.0.1/127.0.0.1:2181, initiating session
>[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session
>establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid =
>0x146de3b813e0032, negotiated timeout = 5000
>[127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn : Session
>establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid =
>0x146de3b813e0031, negotiated timeout = 5000
>[timer://test-#2] o.a.c.c.zookeeper.ZookeeperProducer : Node
>'/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a2623
>690' did not exist, creating it.
>[//googleplay-#1] o.a.c.c.zookeeper.ZookeeperProducer : Node
>'/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b44
>e008854db' did not exist, creating it.
>[timer://test-#2] o.a.c.c.z.policy.ZooKeeperElection : Candidate
>node
>'/scheduler-node-election/test/localhost-3fae9f79-a91c-4baa-bdcc-bd23a2623
>690' has been created
>[//googleplay-#1] o.a.c.c.z.policy.ZooKeeperElection : Candidate
>node
>'/scheduler-node-election/googleplay/localhost-2b2634e5-a0b3-4c0f-8ddb-b44
>e008854db' has been created
>[//googleplay-#1] o.a.camel.spring.SpringCamelContext : Route:
>election-route-localhos started and consuming from:
>Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Starting to
>graceful shutdown 1 routes (timeout 300 seconds)
>[ShutdownTask-#4] o.a.camel.impl.DefaultShutdownStrategy : Route:
>election-route-localhos shutdown complete, was consuming from:
>Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>[timer://test-#2] o.a.camel.impl.DefaultShutdownStrategy : Graceful
>shutdown of 1 routes completed in 0 seconds
>[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>election-route-localhos is stopped, was consuming from:
>Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>[-#1-EventThread] org.apache.zookeeper.ClientCnxn : EventThread
>shut down
>[timer://test-#2] org.apache.zookeeper.ZooKeeper : Session:
>0x146de3b813e0031 closed
>[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>election-route-localhos is shutdown and removed, was consuming from:
>Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/googleplay]
>[timer://test-#2] o.a.camel.spring.SpringCamelContext : Route:
>election-route-localhos started and consuming from:
>Endpoint[zookeeper://127.0.0.1:2181/scheduler-node-election/test]
>
>
>
>My route builder code is below. I call it twice, once for vendor =
>³test², and once for vendor = ³googleplay².
>
>private RouteBuilder scheduledEventBatchExecutionTimerRoute(
> final String vendor,
> final int batchSize,
> final int timerPeriodSeconds,
> final boolean autoStart) {
>
> log.info("scheduledEventBatchExecutionTimerRoute: vendor={},
>doStart={}, queryInterval={}s, batchSize={}",
> vendor, autoStart, timerPeriodSeconds, batchSize);
>
> return new RouteBuilder() {
> @Override
> public void configure() throws Exception {
>
> from("timer://" + vendor + "?period=" +
>timerPeriodSeconds + "s")
> .autoStartup(autoStart)
> .routePolicy(electionRoutePolicy(vendor))
> .setHeader("vendor",
>constant(vendor)).setHeader("batchSize", constant(batchSize))
> .beanRef("scheduleEventRepository",
>"findExecutableEvents(entitlement.create.${in.header.vendor},
>${in.header.batchSize})")
>
> .choice()
> .when().simple("${body.size} == 0")
> .log(LoggingLevel.DEBUG, "Route:
>vendor=${in.header.vendor}: No items to process")
> .when().simple("${body.size} > 0")
> .to(ROUTE_DIRECT_EVENT_BATCH_EXECUTION + "-" +
>vendor)
>
> .endChoice()
> .routeId(vendor + "-timer-execution-route");
>
> }
> };
>
> }
>
> private RoutePolicy electionRoutePolicy(String vendor) {
>
> RoutePolicy routePolicy;
>
> if (doUseDistributedRouteElection) {
> log.info("Creating zookeeper distributed route election
>policy for {}", vendor);
> // Ensure that only one node is processing work for the
>specified vendor.
> // See http://camel.apache.org/zookeeper.html - distributed
>route policy.
> routePolicy = new ZooKeeperRoutePolicy(zookeeperUri +
>"scheduler-node-election/" + vendor, 1);
>
> } else {
> // Just return a default route policy that does nothing.
> log.info("No-op route policy for {}", vendor);
> routePolicy = new RoutePolicySupport() {
> @Override
> public void onInit(Route route) {
> super.onInit(route);
> }
> };
> }
>
> return routePolicy;
> }
>
>
>
>
>
>
>