elasticsearch component config issues

[batwad <al...@oh.is.it>]

We have been using the camel-elasticsearch component in transport client
configuration for some time, however we were seeing NodeClosedExceptions
when the node with which it was communicating was restarted so switched to
the node client to take advantage of the automatic failover.  We configured
the client with data=false as we didn't want it to store data and things
worked quite nicely.

However we also use Marvel to collect cluster stats and after a cluster
restart we found that the node client in our Camel application had become
the master node and no stats were being collected by Marvel on the other
nodes in the cluster.  When our application restarted one of the regular
nodes became master and Marvel started working again.

This leads me to question whether an application node client should be
eligible to be a master node or not.  I have also seen some plugins that
only run on the master node in a cluster, and if the master node is a node
client that would stop them working, wouldn't it?  Perhaps this could be
configurable.

I'd much rather use the transport client as it feels a better fit for our
application's purpose, however the lack of failover support in the component
is unacceptable. I'd like to be able to specify a list of hosts for the
component to use (maybe in the ActiveMQ style of
failover(host1:9200,host2:9200,host3:9200)) or enable sniffing so that the
transport client could discover the rest of the cluster if it can connect
once.

All things considered, it would be cumbersome to specify all these things in
the URI on a route, and I wonder if it would be better to have a more
flexible way of configuring the component in code or XML (again, like
ActiveMQ :-))

Apologies if that is a bit rambling.  I think Camel and elasticsearch are
fantastic and have been spoiled so far with how easy it is integrating
things that maybe I expect things to be too simple now!

Regards,
Alex.



--
View this message in context: http://camel.465427.n5.nabble.com/elasticsearch-component-config-issues-tp5761369.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: elasticsearch component config issues

[batwad <al...@oh.is.it>]

Hi Willem,

Thanks for your reply.  I should have added that we are using Camel 2.12 and
I see the code you've highlighted is tagged as 2.14, so it's possible you've
fixed a bug already.  I notice the 2.14 code does enable sniffing, which the
2.12 code does not.

However it looks like node.client is only set when using the transport
client (i.e.: when you specify an IP address).  When using the node client
(i.e.: cluster name only, no IP) no settings are given so it doesn't look
like the node will be started in client mode.  Maybe that could be why our
application node became master.

Regards,
Alex.



--
View this message in context: http://camel.465427.n5.nabble.com/elasticsearch-component-config-issues-tp5761369p5761396.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: elasticsearch component config issues

[Willem Jiang <wi...@gmail.com>]

It doesn’t make sense that camel-elasticsearch endpoint became a master node.
In camel we just setup the Client parameter here[1], please feel free to add more option on it if you need.

I think we can also introduce addresses option to add other host address for the transport client to use, please feel free to add a JIRA for it.

[1]https://github.com/apache/camel/blob/master/components/camel-elasticsearch/src/main/java/org/apache/camel/component/elasticsearch/ElasticsearchEndpoint.java#L82-L86

--  
Willem Jiang

Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem



On January 6, 2015 at 8:40:56 PM, batwad (alex@oh.is.it) wrote:
> We have been using the camel-elasticsearch component in transport client
> configuration for some time, however we were seeing NodeClosedExceptions
> when the node with which it was communicating was restarted so switched to
> the node client to take advantage of the automatic failover. We configured
> the client with data=false as we didn't want it to store data and things
> worked quite nicely.
>  
> However we also use Marvel to collect cluster stats and after a cluster
> restart we found that the node client in our Camel application had become
> the master node and no stats were being collected by Marvel on the other
> nodes in the cluster. When our application restarted one of the regular
> nodes became master and Marvel started working again.
>  
> This leads me to question whether an application node client should be
> eligible to be a master node or not. I have also seen some plugins that
> only run on the master node in a cluster, and if the master node is a node
> client that would stop them working, wouldn't it? Perhaps this could be
> configurable.
>  
> I'd much rather use the transport client as it feels a better fit for our
> application's purpose, however the lack of failover support in the component
> is unacceptable. I'd like to be able to specify a list of hosts for the
> component to use (maybe in the ActiveMQ style of
> failover(host1:9200,host2:9200,host3:9200)) or enable sniffing so that the
> transport client could discover the rest of the cluster if it can connect
> once.
>  
> All things considered, it would be cumbersome to specify all these things in
> the URI on a route, and I wonder if it would be better to have a more
> flexible way of configuring the component in code or XML (again, like
> ActiveMQ :-))
>  
> Apologies if that is a bit rambling. I think Camel and elasticsearch are
> fantastic and have been spoiled so far with how easy it is integrating
> things that maybe I expect things to be too simple now!
>  
> Regards,
> Alex.
>  
>  
>  
> --
> View this message in context: http://camel.465427.n5.nabble.com/elasticsearch-component-config-issues-tp5761369.html  
> Sent from the Camel - Users mailing list archive at Nabble.com.
>