Does CloudSolrServer hit zookeeper for every request?

["Jim.Musil" <Ji...@target.com>]

I’m curious how CloudSolrServer works in practice.

I understand that it gets the active solr nodes from zookeeper, but does it do this for every request?

If it does hit zk for every request, that seems to put a lot of pressure on the zk ensemble.

If it does NOT hit zk for every request, then how does it detect changes in the number of nodes and the status of the nodes?

Thanks!
Jim M.

Re: Does CloudSolrServer hit zookeeper for every request?

[Shawn Heisey <ap...@elyograg.org>]

On 10/6/2014 12:15 PM, amitha wrote:
> Yeah I saw that does if I call shutdown after a request and need to make
> another request I get 
> org.apache.solr.client.solrj.SolrServerException:
> java.lang.IllegalStateException: Connection pool shut down

You would only call shutdown if you're done with the server object and
don't plan to use it again.  That method shuts the object down so it
can't be used any more.

I admit that I don't have a lot of experience with CloudSolrServer, and
I haven't looked at the code in enough detail to know exactly how it
works ... but I had always thought that the zookeeper client would
maintain a constant connection with every ZK server that you tellit about.

You haven't put a load balancer in front of your zookeeper servers, have
you?  That's completely unnecessary - just inform CloudSolrServer about
all three of them.

I thought we got some much better javadocs into a recent release, but it
looks like it's only in the new stable branch and trunk.

Here's the new CloudSolrServer(String) javadoc comment that will be in 5.0:

http://apaste.info/BVu

Thanks,
Shawn

Re: Does CloudSolrServer hit zookeeper for every request?

[amitha <am...@yahoo.com>]

Yeah I saw that does if I call shutdown after a request and need to make
another request I get 
org.apache.solr.client.solrj.SolrServerException:
java.lang.IllegalStateException: Connection pool shut down




--
View this message in context: http://lucene.472066.n3.nabble.com/Does-CloudSolrServer-hit-zookeeper-for-every-request-tp4139413p4162969.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Does CloudSolrServer hit zookeeper for every request?

[Joel Bernstein <jo...@gmail.com>]

There is a shutdown() method on CloudSolrServer that releases the resources
for CloudSolrServer.

Joel Bernstein
Search Engineer at Heliosearch

On Mon, Oct 6, 2014 at 11:39 AM, Jonnakuti, Vijayalatha <
Vijayalatha.Jonnakuti@disney.com> wrote:

> We are using solr 4.8.0
>
> From: <Jonnakuti>, Vijay Jonnakuti <vijayalatha.jonnakuti@disney.com
> <ma...@disney.com>>
> Date: Monday, October 6, 2014 at 8:34 AM
> To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <
> solr-user@lucene.apache.org<ma...@lucene.apache.org>>
> Subject: Does CloudSolrServer hit zookeeper for every request?
>
> Hi
> We have too many zookeeper connections and had to increase max client
> connection on zookeeper(3.4.6)
>
> The underlaying problem looks like,
> CloudSolrServer makes a connect call to zookeeper for every request  but
> never disconnects.
>  Am I supposed to call getZkStateReader().close() for every request? Or am
> I missing something?
>
> Solr Configuration is
>
> <autoCommit>
>   <maxDocs>10000</maxDocs>
>   <maxTime>900000</maxTime>
>   <openSearcher>false</openSearcher>
> </autoCommit>
> <commitWithin>
>   <softCommit>true</softCommit>
> </commitWithin>
> <autoSoftCommit>
>        <maxTime>10000</maxTime>
> </autoSoftCommit>
>
> Thanks.,
> Vijaya
>

Re: Does CloudSolrServer hit zookeeper for every request?

["Jonnakuti, Vijayalatha" <Vi...@disney.com>]

We are using solr 4.8.0

From: <Jonnakuti>, Vijay Jonnakuti <vi...@disney.com>>
Date: Monday, October 6, 2014 at 8:34 AM
To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Subject: Does CloudSolrServer hit zookeeper for every request?

Hi
We have too many zookeeper connections and had to increase max client connection on zookeeper(3.4.6)

The underlaying problem looks like,
CloudSolrServer makes a connect call to zookeeper for every request  but never disconnects.
 Am I supposed to call getZkStateReader().close() for every request? Or am I missing something?

Solr Configuration is

<autoCommit>
  <maxDocs>10000</maxDocs>
  <maxTime>900000</maxTime>
  <openSearcher>false</openSearcher>
</autoCommit>
<commitWithin>
  <softCommit>true</softCommit>
</commitWithin>
<autoSoftCommit>
       <maxTime>10000</maxTime>
</autoSoftCommit>

Thanks.,
Vijaya

Re: Does CloudSolrServer hit zookeeper for every request?

[Erick Erickson <er...@gmail.com>]

There's some pinging going on between ZK and registered nodes, and when the
timeout is exceeded there ZK marks the node as down and broadcasts messages
to all the _other_ nodes that the node is down. Then each Solr node knows
not to use the downed node until a message is received indicating it's
healthy again.

Best,
Erick


On Mon, Jun 2, 2014 at 2:17 PM, Jim.Musil <Ji...@target.com> wrote:

> I’m curious how CloudSolrServer works in practice.
>
> I understand that it gets the active solr nodes from zookeeper, but does
> it do this for every request?
>
> If it does hit zk for every request, that seems to put a lot of pressure
> on the zk ensemble.
>
> If it does NOT hit zk for every request, then how does it detect changes
> in the number of nodes and the status of the nodes?
>
> Thanks!
> Jim M.
>

Re: Does CloudSolrServer hit zookeeper for every request?

[Steve McKay <st...@b.abbies.us>]

ZooKeeper allows clients to put watches on paths in the ZK tree. When the cluster state changes, every Solr client is notified by the ZK server and then each client reads the updated state. No polling is needed or even helpful.

In any event, reading from ZK is much more lightweight than writing, because the ZK server keeps all its data in memory and doesn’t have to go through the consensus rigamarole required for a write.

On Jun 2, 2014, at 5:17 PM, Jim.Musil <Ji...@target.com> wrote:

> I’m curious how CloudSolrServer works in practice.
> 
> I understand that it gets the active solr nodes from zookeeper, but does it do this for every request?
> 
> If it does hit zk for every request, that seems to put a lot of pressure on the zk ensemble.
> 
> If it does NOT hit zk for every request, then how does it detect changes in the number of nodes and the status of the nodes?
> 
> Thanks!
> Jim M.