You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by vicky desai <vi...@germinait.com> on 2013/07/03 09:05:13 UTC
Is it possible to find a leader from a list of cores in solr via
java code
Hi ,
I have a set up of 1 leader and 1 replica and I have a requirement where in
I need to find the leader core from the collection. Is there an api in solrj
by means of which this can be achieved.
--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via
java code
Posted by Aloke Ghoshal <al...@gmail.com>.
One option could be to get the clusterstate.json via the following Solr url
& figure out the leader from the response json:
*
http://server:port/solr/zookeeper?detail=true&path=%2Fclusterstate.json*
On Wed, Jul 3, 2013 at 5:57 PM, vicky desai <vi...@germinait.com>wrote:
> Hi,
>
> I have a requirement where in I want to write to the leader and read from
> the replica. Reason being If a write request is sent to the replica it
> relays it to the leader and then the leader relays it to all the replicas.
> This will help me in saving some network traffic as my application performs
> continuous writes
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4075083.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Is it possible to find a leader from a list of cores in solr
via java code
Posted by vicky desai <vi...@germinait.com>.
Hi,
I got the solution to the above problem . Sharing the code so that it could
help people in future
PoolingClientConnectionManager poolingClientConnectionManager = new
PoolingClientConnectionManager();
poolingClientConnectionManager.setMaxTotal(2);
poolingClientConnectionManager.setDefaultMaxPerRoute(1);
HttpClient httpClient = (HttpClient)new
DefaultHttpClient(poolingClientConnectionManager);
LBHttpSolrServer lbServer = new LBHttpSolrServer(httpClient);
server = new CloudSolrServer("zkhost", lbServer);
server.setDefaultCollection(collectionName);
Thanks a lot to every1 in the thread chain. Your suggestions helped a lot
--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4078012.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr
via java code
Posted by vicky desai <vi...@germinait.com>.
Hi,
As per the suggestions above I shifted my focus to using CloudSolrServer. In
terms of sending updates to the leaders and reducing network traffic it
works great. But i faced one problem in using CloudSolrServer is that it
opens too many connections as large as five thousand connections. My Code is
as follows
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 3);
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 2);
HttpClient client = HttpClientUtil.createClient(params);
LBHttpSolrServer lbServer = new LBHttpSolrServer(client);
server = new CloudSolrServer(zkHost,lbServer);
server.setDefaultCollection(defaultColllection);
If there is only one instance of solr up then this works great. But in 1
shard 1 replica system it opens up too many connections in waiting state. Am
I doing something incorrect. Any help would be highly appreciated
--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4077587.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via
java code
Posted by Erick Erickson <er...@gmail.com>.
What Jack and Shalin said....
Jack's comments about leaders changing is absolutely correct,
although practically they won't change all that often unless
you're bouncing nodes all over the place.
But as Shalin says, CloudSolrServer will already to this, at least
as far as sending updates to a leader. If you only have one shard,
it's already been done for you, just use SolrJ and CloudSolrServer.
And https://issues.apache.org/jira/browse/SOLR-4816 will further
refine the process such that the documents are split up into packets,
all the docs for that packet go to the same shard, and the packet
will be sent to the shard leader.
So as Jack says, this is probably a really poor place to put _your_
development efforts since a> in the single-shard case it's already
been done for you and b> in the near future it'll already have been
done for you in the multi-shard case.
If you want to pursue this, I suggest putting your efforts into
helping test SOLR-4816 is both more efficient for you and a help
to the larger community.
Best
Erick
On Sun, Jul 7, 2013 at 2:52 AM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> Update requests are routed only to leader nodes by CloudSolrServer so
> there is a 1/numShards probability of avoiding an extra network hop.
> It is *not* a 1/(number of nodes in cluster) probability as you may
> think.
>
> You may also be interested in
> https://issues.apache.org/jira/browse/SOLR-4816 . Perhaps you can try
> the patch and report your results.
>
> On Sun, Jul 7, 2013 at 11:18 AM, Jack Krupansky <ja...@basetechnology.com> wrote:
>> There are three "concepts" to grasp:
>>
>> 1. You can send Solr update requests to ANY node of the cluster. Period.
>> 2. Any extra network traffic (within the cluster) is likely to be negligible
>> and absolutely not worrying about unless you have definitive evidence to the
>> contrary.
>> 3. Leader nodes in SolrCloud are designed to be dynamic and will change over
>> time. Generally, you won't know which node is the leader for a shard at any
>> given moment - it may have been a particular node one milliscond ago, but
>> now maybe another node has been designated as the leader. Neither the
>> application nor the user designates which node is a leader for a shard - it
>> is a dynamic election process. Sure, you can check to see which node is
>> currently the leader, but one millisecond later some other node may have
>> gotten elected to be the leader.
>>
>> To repeat, you DO NOT need to be concerned with sending Solr update requests
>> to a shard "leader". The SolrCloud concept of "leader" is really only needed
>> WITHIN the cluster. Clients outside of the SolrCloud cluster need not
>> concern themselves with shard "leaders."
>>
>> "Micro-optimization" and "premature optimization" are very poor tools to
>> lead a system design. Focus the energy on the data modeling and the overall
>> application design.
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: vicky desai
>> Sent: Sunday, July 07, 2013 1:17 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Is it possible to find a leader from a list of cores in solr
>> via java code
>>
>>
>> Hi Erik,
>>
>> I just wanted to clarify if u got my concern right. If i send some documents
>> to the replica core wont it first have to send the documents to the leader
>> core which in turn would be sending it back to the replica cores. If yes
>> then this will lead to additional network traffic which can be avoided by
>> sending the documents directly to leader.
>>
>> Please correct me if I have got the concept incorrect.Any help is
>> appreciated
>>
>>
>> Thanks,
>> Vicky
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4076016.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
Re: Is it possible to find a leader from a list of cores in solr via
java code
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Update requests are routed only to leader nodes by CloudSolrServer so
there is a 1/numShards probability of avoiding an extra network hop.
It is *not* a 1/(number of nodes in cluster) probability as you may
think.
You may also be interested in
https://issues.apache.org/jira/browse/SOLR-4816 . Perhaps you can try
the patch and report your results.
On Sun, Jul 7, 2013 at 11:18 AM, Jack Krupansky <ja...@basetechnology.com> wrote:
> There are three "concepts" to grasp:
>
> 1. You can send Solr update requests to ANY node of the cluster. Period.
> 2. Any extra network traffic (within the cluster) is likely to be negligible
> and absolutely not worrying about unless you have definitive evidence to the
> contrary.
> 3. Leader nodes in SolrCloud are designed to be dynamic and will change over
> time. Generally, you won't know which node is the leader for a shard at any
> given moment - it may have been a particular node one milliscond ago, but
> now maybe another node has been designated as the leader. Neither the
> application nor the user designates which node is a leader for a shard - it
> is a dynamic election process. Sure, you can check to see which node is
> currently the leader, but one millisecond later some other node may have
> gotten elected to be the leader.
>
> To repeat, you DO NOT need to be concerned with sending Solr update requests
> to a shard "leader". The SolrCloud concept of "leader" is really only needed
> WITHIN the cluster. Clients outside of the SolrCloud cluster need not
> concern themselves with shard "leaders."
>
> "Micro-optimization" and "premature optimization" are very poor tools to
> lead a system design. Focus the energy on the data modeling and the overall
> application design.
>
> -- Jack Krupansky
>
> -----Original Message----- From: vicky desai
> Sent: Sunday, July 07, 2013 1:17 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Is it possible to find a leader from a list of cores in solr
> via java code
>
>
> Hi Erik,
>
> I just wanted to clarify if u got my concern right. If i send some documents
> to the replica core wont it first have to send the documents to the leader
> core which in turn would be sending it back to the replica cores. If yes
> then this will lead to additional network traffic which can be avoided by
> sending the documents directly to leader.
>
> Please correct me if I have got the concept incorrect.Any help is
> appreciated
>
>
> Thanks,
> Vicky
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4076016.html
> Sent from the Solr - User mailing list archive at Nabble.com.
--
Regards,
Shalin Shekhar Mangar.
Re: Is it possible to find a leader from a list of cores in solr via java code
Posted by Jack Krupansky <ja...@basetechnology.com>.
There are three "concepts" to grasp:
1. You can send Solr update requests to ANY node of the cluster. Period.
2. Any extra network traffic (within the cluster) is likely to be negligible
and absolutely not worrying about unless you have definitive evidence to the
contrary.
3. Leader nodes in SolrCloud are designed to be dynamic and will change over
time. Generally, you won't know which node is the leader for a shard at any
given moment - it may have been a particular node one milliscond ago, but
now maybe another node has been designated as the leader. Neither the
application nor the user designates which node is a leader for a shard - it
is a dynamic election process. Sure, you can check to see which node is
currently the leader, but one millisecond later some other node may have
gotten elected to be the leader.
To repeat, you DO NOT need to be concerned with sending Solr update requests
to a shard "leader". The SolrCloud concept of "leader" is really only needed
WITHIN the cluster. Clients outside of the SolrCloud cluster need not
concern themselves with shard "leaders."
"Micro-optimization" and "premature optimization" are very poor tools to
lead a system design. Focus the energy on the data modeling and the overall
application design.
-- Jack Krupansky
-----Original Message-----
From: vicky desai
Sent: Sunday, July 07, 2013 1:17 AM
To: solr-user@lucene.apache.org
Subject: Re: Is it possible to find a leader from a list of cores in solr
via java code
Hi Erik,
I just wanted to clarify if u got my concern right. If i send some documents
to the replica core wont it first have to send the documents to the leader
core which in turn would be sending it back to the replica cores. If yes
then this will lead to additional network traffic which can be avoided by
sending the documents directly to leader.
Please correct me if I have got the concept incorrect.Any help is
appreciated
Thanks,
Vicky
--
View this message in context:
http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4076016.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr
via java code
Posted by vicky desai <vi...@germinait.com>.
Hi Erik,
I just wanted to clarify if u got my concern right. If i send some documents
to the replica core wont it first have to send the documents to the leader
core which in turn would be sending it back to the replica cores. If yes
then this will lead to additional network traffic which can be avoided by
sending the documents directly to leader.
Please correct me if I have got the concept incorrect.Any help is
appreciated
Thanks,
Vicky
--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4076016.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via
java code
Posted by Erick Erickson <er...@gmail.com>.
bq: Reason being If a write request is sent to the replica it
relays it to the leader and then the leader relays it to all the replicas.
This will help me in saving some network traffic as my application performs
continuous writes
How are you indexing? SolrJ already is "leader aware" and sends
updates to the leaders. Requests should go to any node, there's no
good reason to send requests only to replicas since the replicas
and leaders are both search and index nodes.
And an upcoming enhancement to SolrJ will additionally route the
documents to the correct leader, i.e. it'll divide documents up
into packets that all live on the same shard and send those packets
to the leader for that shard.
Unless you can prove that this is actually causing you trouble,
I'd advise not spending the effort on this and focusing on
other aspects of your project..
This _really_ feels like premature optimization.
Best
Erick
On Wed, Jul 3, 2013 at 8:27 AM, vicky desai <vi...@germinait.com>wrote:
> Hi,
>
> I have a requirement where in I want to write to the leader and read from
> the replica. Reason being If a write request is sent to the replica it
> relays it to the leader and then the leader relays it to all the replicas.
> This will help me in saving some network traffic as my application performs
> continuous writes
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4075083.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Is it possible to find a leader from a list of cores in solr
via java code
Posted by vicky desai <vi...@germinait.com>.
Hi,
I have a requirement where in I want to write to the leader and read from
the replica. Reason being If a write request is sent to the replica it
relays it to the leader and then the leader relays it to all the replicas.
This will help me in saving some network traffic as my application performs
continuous writes
--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4075083.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via
java code
Posted by Giovanni Bricconi <gi...@banzai.it>.
I have the same question. My purpose is to start the dih full process on
the leader and not on a replica.
I tried full import on a replica but watching logs it seemed to me that the
replica was loading data to send it to the leader which in turn has to
update all the replicas.
At least this is what I saw with solr 4.2.1
Giovanni
2013/7/3 Erick Erickson <er...@gmail.com>
> You can always query Zookeeper and find that information out.
> Take a look at CloudSolrServer, maybe ZkCoreNodeProps etc.
> for examples since CloudSolrServer is "leader aware", it
> should have some clues...
>
> Or maybe ZkStateReader? I haven't been in that code much,
> so I can't be more specific...
>
> But why do you have this requirement? What do you hope to
> accomplish? Because this is often the kind of thing that's seems
> more useful than it is...
>
> Best
> Erick
>
>
> On Wed, Jul 3, 2013 at 3:05 AM, vicky desai <vicky.desai@germinait.com
> >wrote:
>
> > Hi ,
> >
> > I have a set up of 1 leader and 1 replica and I have a requirement where
> in
> > I need to find the leader core from the collection. Is there an api in
> > solrj
> > by means of which this can be achieved.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
Re: Is it possible to find a leader from a list of cores in solr via
java code
Posted by Erick Erickson <er...@gmail.com>.
You can always query Zookeeper and find that information out.
Take a look at CloudSolrServer, maybe ZkCoreNodeProps etc.
for examples since CloudSolrServer is "leader aware", it
should have some clues...
Or maybe ZkStateReader? I haven't been in that code much,
so I can't be more specific...
But why do you have this requirement? What do you hope to
accomplish? Because this is often the kind of thing that's seems
more useful than it is...
Best
Erick
On Wed, Jul 3, 2013 at 3:05 AM, vicky desai <vi...@germinait.com>wrote:
> Hi ,
>
> I have a set up of 1 leader and 1 replica and I have a requirement where in
> I need to find the leader core from the collection. Is there an api in
> solrj
> by means of which this can be achieved.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>