You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Satya Nand <sa...@indiamart.com.INVALID> on 2022/10/10 07:57:06 UTC
Shards Parameter causing Infinite loop in solr cloud search
Hi,
Good Morning.
we have 8+1 solr nodes cluster. Where 1 Indexing node contains all(8) NRT
Primary shards. This is where all indexing happens. Then We have another 8
nodes consisting of one pull replica of each primary shard.
To limit the query on replicas we have done the following changes in
solrconfig and shard whitelisting.
<str name="shards.tolerant">true</str>
<str name="shards">
10.128.74.11:6086/solr/im-search,10.128.74.11:6087/solr/im-search</str>
But after the changes, the requests are going in an infinite loop. I found
this in the documentation but I couldn't understand what is standard vs
nonstandard request handler is.
*"Do not add the shards parameter to the standard request handler; doing so
> may cause search queries may enter an infinite loop. Instead, define a new
> request handler that uses the shards parameter, and pass distributed search
> requests to that handler."*
so, How can we use shards parameters in solr config to limit the shards? I
think one alternative will be to pass the shards parameter in URL instead
of solrconfig.
But we would want to use the solrconfig to limit the changes in config only.
Re: Shards Parameter causing Infinite loop in solr cloud search
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Thanks Shawn for sharing all possibilities , we will try to evaluate all
these.
On Mon, 10 Oct, 2022, 6:45 pm Shawn Heisey, <ap...@elyograg.org> wrote:
> On 10/10/22 06:58, Satya Nand wrote:
> > Actually we were using the preference parameter but recently we faced an
> > issue where 1 pull replica got down(due to gcp machine restart) and
> > requests started going to the NRT replica.
> > Machine hosting NRT replica is pretty weak.
> >
> > That's why I was experimenting with with shards parameter with all the
> urls
> > of pull replicas. So request has no option to go to any other shards.
> > Also planning to use shards.tolrent so that in case one or more replica
> is
> > down , we can get the response from remaining replicas.
>
> Some choices:
>
> * Bump up the hardware hosting the NRT replicas so they can also handle
> queries.
> * Add another set of PULL replicas on separate hardware.
> * Adjust your systems so that each one hosts a PULL replica for two
> different shards.
> * Rearrange things so that each system hosts an NRT replica for one
> shard and a PULL replica for a different shard.
>
> Thanks,
> Shawn
>
>
Re: Shards Parameter causing Infinite loop in solr cloud search
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/10/22 06:58, Satya Nand wrote:
> Actually we were using the preference parameter but recently we faced an
> issue where 1 pull replica got down(due to gcp machine restart) and
> requests started going to the NRT replica.
> Machine hosting NRT replica is pretty weak.
>
> That's why I was experimenting with with shards parameter with all the urls
> of pull replicas. So request has no option to go to any other shards.
> Also planning to use shards.tolrent so that in case one or more replica is
> down , we can get the response from remaining replicas.
Some choices:
* Bump up the hardware hosting the NRT replicas so they can also handle
queries.
* Add another set of PULL replicas on separate hardware.
* Adjust your systems so that each one hosts a PULL replica for two
different shards.
* Rearrange things so that each system hosts an NRT replica for one
shard and a PULL replica for a different shard.
Thanks,
Shawn
Re: Shards Parameter causing Infinite loop in solr cloud search
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Shawn,
Actually we were using the preference parameter but recently we faced an
issue where 1 pull replica got down(due to gcp machine restart) and
requests started going to the NRT replica.
Machine hosting NRT replica is pretty weak.
That's why I was experimenting with with shards parameter with all the urls
of pull replicas. So request has no option to go to any other shards.
Also planning to use shards.tolrent so that in case one or more replica is
down , we can get the response from remaining replicas.
Based on the Documentation link I have posted, it says that we can use but
i cam not able to make it work.
On Mon, 10 Oct, 2022, 6:14 pm Shawn Heisey, <ap...@elyograg.org> wrote:
> On 10/10/22 06:00, Satya Nand wrote:
> > Yes, we are using solr cloud. The reason I don't want to specify the
> > shards' names is that a request can be sent to any replica of a shard
> based
> > on preference and availability but I specifically want to limit a request
> > to a PULL-type replica of a shard.
> >
> > I am trying to replicate the behavior on this link,
>
> This is a perfect use case for the shards.preference parameter. Use
> "shards.preference=replica.type:PULL" along with a list of shard names.
> If there is at least one PULL replica available for a shard, it will be
> used. It will only try other replica types as a last resort.
>
>
> https://solr.apache.org/guide/8_4/distributed-requests.html#shards-preference-parameter
>
> Thanks,
> Shawn
>
>
Re: Shards Parameter causing Infinite loop in solr cloud search
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/10/22 06:00, Satya Nand wrote:
> Yes, we are using solr cloud. The reason I don't want to specify the
> shards' names is that a request can be sent to any replica of a shard based
> on preference and availability but I specifically want to limit a request
> to a PULL-type replica of a shard.
>
> I am trying to replicate the behavior on this link,
This is a perfect use case for the shards.preference parameter. Use
"shards.preference=replica.type:PULL" along with a list of shard names.
If there is at least one PULL replica available for a shard, it will be
used. It will only try other replica types as a last resort.
https://solr.apache.org/guide/8_4/distributed-requests.html#shards-preference-parameter
Thanks,
Shawn
Re: Shards Parameter causing Infinite loop in solr cloud search
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Shawn,
>
> The standard request handler is usually the one named "/select". You may
> want to add a new handler for this purpose.
We are already using a custom request handler, Actually, there is no
/select handler in our solr config.
your message subject says you are in cloud mode. If that is true, I
> think you are going to want to specify shards by name, not URL.
Yes, we are using solr cloud. The reason I don't want to specify the
shards' names is that a request can be sent to any replica of a shard based
on preference and availability but I specifically want to limit a request
to a PULL-type replica of a shard.
I am trying to replicate the behavior on this link,
https://solr.apache.org/guide/8_4/distributed-requests.html#limiting-which-shards-are-queried
This section
<--------
> Alternatively, you can specify a list of replicas you wish to use in place
> of a shard IDs by separating the replica IDs with commas:
http://localhost:8983/solr/gettingstarted/select?q=*:*&shards=localhost:7574/solr/gettingstarted,localhost:8983/solr/gettingstarted
---->
But when I do this, my request goes in an infinite loop.
is there anything I can do to make it work? I just want to use some
specific set of replicas with shard.tolrent=true.
On Mon, Oct 10, 2022 at 5:07 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 10/10/22 01:57, Satya Nand wrote:
>
> >> *"Do not add the shards parameter to the standard request handler;
> doing so
> >> may cause search queries may enter an infinite loop. Instead, define a
> new
> >> request handler that uses the shards parameter, and pass distributed
> search
> >> requests to that handler."*
> >
> > so, How can we use shards parameters in solr config to limit the shards?
> I
> > think one alternative will be to pass the shards parameter in URL instead
> > of solrconfig.
> >
> > But we would want to use the solrconfig to limit the changes in config
> only.
>
> The standard request handler is usually the one named "/select". You may
> want to add a new handler for this purpose.
>
> Your message subject says you are in cloud mode. If that is true, I
> think you are going to want to specify shards by name, not URL. If you
> are in standalone mode (no zookeeper) then the way I handled that was to
> build a special core with an empty index that had a predefined list of
> shard URLs in the /select handler. When I did that, I was using the
> "defaults" parameter config. I think if I did it again I would use
> "invariants" so the user would not be able to override the list.
>
> Thanks,
> Shawn
>
>
On Mon, Oct 10, 2022 at 5:07 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 10/10/22 01:57, Satya Nand wrote:
>
> >> *"Do not add the shards parameter to the standard request handler;
> doing so
> >> may cause search queries may enter an infinite loop. Instead, define a
> new
> >> request handler that uses the shards parameter, and pass distributed
> search
> >> requests to that handler."*
> >
> > so, How can we use shards parameters in solr config to limit the shards?
> I
> > think one alternative will be to pass the shards parameter in URL instead
> > of solrconfig.
> >
> > But we would want to use the solrconfig to limit the changes in config
> only.
>
> The standard request handler is usually the one named "/select". You may
> want to add a new handler for this purpose.
>
> Your message subject says you are in cloud mode. If that is true, I
> think you are going to want to specify shards by name, not URL. If you
> are in standalone mode (no zookeeper) then the way I handled that was to
> build a special core with an empty index that had a predefined list of
> shard URLs in the /select handler. When I did that, I was using the
> "defaults" parameter config. I think if I did it again I would use
> "invariants" so the user would not be able to override the list.
>
> Thanks,
> Shawn
>
>
Re: Shards Parameter causing Infinite loop in solr cloud search
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/10/22 01:57, Satya Nand wrote:
>> *"Do not add the shards parameter to the standard request handler; doing so
>> may cause search queries may enter an infinite loop. Instead, define a new
>> request handler that uses the shards parameter, and pass distributed search
>> requests to that handler."*
>
> so, How can we use shards parameters in solr config to limit the shards? I
> think one alternative will be to pass the shards parameter in URL instead
> of solrconfig.
>
> But we would want to use the solrconfig to limit the changes in config only.
The standard request handler is usually the one named "/select". You may
want to add a new handler for this purpose.
Your message subject says you are in cloud mode. If that is true, I
think you are going to want to specify shards by name, not URL. If you
are in standalone mode (no zookeeper) then the way I handled that was to
build a special core with an empty index that had a predefined list of
shard URLs in the /select handler. When I did that, I was using the
"defaults" parameter config. I think if I did it again I would use
"invariants" so the user would not be able to override the list.
Thanks,
Shawn