You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dmitry Kan <dm...@gmail.com> on 2011/06/02 13:00:53 UTC
query routing with shards
Hello all,
We have currently several pretty fat logically isolated shards with the same
schema / solrconfig (indices are separate). We currently have one single
front end SOLR (1.4) for the client code calls. Since a client code query
usually hits only one shard, we are considering making a smart routing of
queries to the shards they map to. Can you please give some pointers as to
what would be an optimal way to achieve such a routing inside the front end
solr? Is there a way to configure mapping inside the solrconfig?
Thanks.
--
Regards,
Dmitry Kan
Re: query routing with shards
Posted by Dmitry Kan <dm...@gmail.com>.
Hi Otis,
I have fixed it by assigning the value to rb same as assigned to sreq:
rb.shards = shards.toString().split(",");
not tested that fully yet, but distributed faceting works at least on my pc
_3 shards 1 router_ setup.
Dmitry
On Thu, Jun 16, 2011 at 4:53 PM, Dmitry Kan <dm...@gmail.com> wrote:
> Hi Otis,
>
> I followed your recommendation and decided to implement the
> SearchComponent::modifyRequest(ResponseBuilder rb, SearchComponent who,
> ShardRequest sreq) method, where the query routing happens. So far it is
> working OK for the non-facet search, this is good news. The bad news is that
> it fails on the facet search.
>
> This is how request modification happens:
>
> [code_snippet, SearchComponent::modifyRequest]
> SolrQueryRequest req_routed = rb.req;
> req_routed = routeRequest(req_routed);
> rb.req = req_routed;
> sreq.shards = shards.toString().split(",");
> [/code_snippet]
>
> where shards is StringBuilder, that accumulates the shards the request
> should go to. req_routed also contains the target shards. Those are set like
> this:
>
>
> [code_snippet, my function routeRequest(SolrQueryRequest req)]
> // could not find clone(), used ref reassignment
> SolrQueryRequest req_local = req;
> ModifiableSolrParams params = new
> ModifiableSolrParams(req_local.getParams());
> ...
> params.remove(ShardParams.SHARDS);
> params.set(ShardParams.SHARDS, getShardsParams(yearToQuarterMap));
> params.remove(ShardParams.IS_SHARD);
> params.set(ShardParams.IS_SHARD, true);
> req_local.setParams(params);
> ...
> return req_local;
> [/code_snippet]
>
> The NPE happens down the road during the facet search, in the
> FacetComponent::countFacets(), the cause of which is that OpenBitSet obs is
> null for shardNum=0.
>
> Do you have any idea why this happens, should some other field
> of ResponseBuilder, SearchComponent or ShardRequest be changed?
>
> BTW, I have tried to call FacetInfo::parse method inside
> FacetComponent::modifyRequest() and countFacets(). Where do
> the fi.facets.values() get initiated, is there some method to call?
>
> Thanks,
> Dmitry
>
> On Fri, Jun 3, 2011 at 8:00 PM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com> wrote:
>
>> Nah, if you can quickly figure out which shard a given query maps to, then
>> all
>> this component needs to do is stick the appropriate shards param value in
>> the
>> request and let the request pass through to the other SearchComponents in
>> the
>> chain, including QueryComponent, which will know what to do with the
>> shards
>> param.
>>
>> Otis
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> Lucene ecosystem search :: http://search-lucene.com/
>>
>>
>>
>> ----- Original Message ----
>> > From: Dmitry Kan <dm...@gmail.com>
>> > To: solr-user@lucene.apache.org
>> > Sent: Fri, June 3, 2011 12:56:15 PM
>> > Subject: Re: query routing with shards
>> >
>> > Hi Otis,
>> >
>> > Thanks! This sounds promising. This custom implementation, will it hurt
>> in
>> > any way the stability of the front end SOLR? After implementing it, can
>> I
>> > run some tests to verify the stability / performance?
>> >
>> > Dmitry
>> > On Fri, Jun 3, 2011 at 4:49 PM, Otis Gospodnetic <
>> otis_gospodnetic@yahoo.com
>> > > wrote:
>> >
>> > > Hi Dmitry,
>> > >
>> > > Yes, you could also implement your own custom SearchComponent. In
>> this
>> > > component you could grab the query param, examine the query value,
>> and
>> > > based on
>> > > that add the shards URL param with appropriate value, so that when
>> the
>> > > regular
>> > > QueryComponent grabs stuff from the request, it has the correct shard
>> in
>> > > there
>> > > already.
>> > >
>> > > Otis
>> > > ----
>> > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> > > Lucene ecosystem search :: http://search-lucene.com/
>> > >
>> > >
>> > >
>> > > ----- Original Message ----
>> > > > From: Dmitry Kan <dm...@gmail.com>
>> > > > To: solr-user@lucene.apache.org
>> > > > Sent: Fri, June 3, 2011 2:47:00 AM
>> > > > Subject: Re: query routing with shards
>> > > >
>> > > > Hi Otis,
>> > > >
>> > > > I merely followed on the gmail's suggestion to include other
>> people
>> into
>> > > the
>> > > > recipients list, Yonik was the first one :) I won't do it next
>> time.
>> > > >
>> > > > Thanks for a rapid reply. The reason for doing this query routing
>> is
>> > > that we
>> > > > abstract the distributed SOLR from the client code for security
>> reasons
>> > > > (that is, we don't want to expose the entire shard farm to the
>> world,
>> > > but
>> > > > only the frontend SOLR) and for better decoupling.
>> > > >
>> > > > Is it possible to implement a plugin to SOLR that would map
>> queries to
>> > > > shards?
>> > > >
>> > > > We have other choices too, they'll take quite some time, that's
>> why I
>> > > > decided to quickly ask, if I was missing something from the SOLR
>> main
>> > > > components design and configuration.
>> > > >
>> > > > Dmitry
>> > > >
>> > > > On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <
>> > > otis_gospodnetic@yahoo.com
>> > > > > wrote:
>> > > >
>> > > > > Hi Dmitry (you may not want to additionally copy Yonik, he's
>> > > subscribed to
>> > > > > this
>> > > > > list, too)
>> > > > >
>> > > > >
>> > > > > It sounds like you have the knowledge of which query maps to
>> which
>> > > shard.
>> > > > > If
>> > > > > so, why not control/change the value of "shards" param in the
>> request
>> > > to
>> > > > > your
>> > > > > front-end Solr (aka distributed request dispatcher) within your
>> app,
>> > > which
>> > > > > is
>> > > > > the one calling Solr?
>> > > > >
>> > > > > Otis
>> > > > > ----
>> > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> > > > > Lucene ecosystem search :: http://search-lucene.com/
>> > > > >
>> > > > >
>> > > > >
>> > > > > ----- Original Message ----
>> > > > > > From: Dmitry Kan <dm...@gmail.com>
>> > > > > > To: solr-user@lucene.apache.org; yonik@lucidimagination.com
>> > > > > > Sent: Thu, June 2, 2011 7:00:53 AM
>> > > > > > Subject: query routing with shards
>> > > > > >
>> > > > > > Hello all,
>> > > > > >
>> > > > > > We have currently several pretty fat logically isolated shards
>> with
>> > > the
>> > > > > same
>> > > > > > schema / solrconfig (indices are separate). We currently have
>> one
>> > > single
>> > > > > > front end SOLR (1.4) for the client code calls. Since a
>> client
>> code
>> > > > > query
>> > > > > > usually hits only one shard, we are considering making a smart
>> > > routing
>> > > > > of
>> > > > > > queries to the shards they map to. Can you please give some
>> > > pointers as
>> > > > > to
>> > > > > > what would be an optimal way to achieve such a routing inside
>> the
>> > > front
>> > > > > end
>> > > > > > solr? Is there a way to configure mapping inside the
>> solrconfig?
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > > --
>> > > > > > Regards,
>> > > > > >
>> > > > > > Dmitry Kan
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Regards,
>> > > >
>> > > > Dmitry Kan
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> >
>> > Dmitry Kan
>> >
>>
>
>
>
--
Regards,
Dmitry Kan
Re: query routing with shards
Posted by Dmitry Kan <dm...@gmail.com>.
Hi Otis,
I followed your recommendation and decided to implement the
SearchComponent::modifyRequest(ResponseBuilder rb, SearchComponent who,
ShardRequest sreq) method, where the query routing happens. So far it is
working OK for the non-facet search, this is good news. The bad news is that
it fails on the facet search.
This is how request modification happens:
[code_snippet, SearchComponent::modifyRequest]
SolrQueryRequest req_routed = rb.req;
req_routed = routeRequest(req_routed);
rb.req = req_routed;
sreq.shards = shards.toString().split(",");
[/code_snippet]
where shards is StringBuilder, that accumulates the shards the request
should go to. req_routed also contains the target shards. Those are set like
this:
[code_snippet, my function routeRequest(SolrQueryRequest req)]
// could not find clone(), used ref reassignment
SolrQueryRequest req_local = req;
ModifiableSolrParams params = new
ModifiableSolrParams(req_local.getParams());
...
params.remove(ShardParams.SHARDS);
params.set(ShardParams.SHARDS, getShardsParams(yearToQuarterMap));
params.remove(ShardParams.IS_SHARD);
params.set(ShardParams.IS_SHARD, true);
req_local.setParams(params);
...
return req_local;
[/code_snippet]
The NPE happens down the road during the facet search, in the
FacetComponent::countFacets(), the cause of which is that OpenBitSet obs is
null for shardNum=0.
Do you have any idea why this happens, should some other field
of ResponseBuilder, SearchComponent or ShardRequest be changed?
BTW, I have tried to call FacetInfo::parse method inside
FacetComponent::modifyRequest() and countFacets(). Where do
the fi.facets.values() get initiated, is there some method to call?
Thanks,
Dmitry
On Fri, Jun 3, 2011 at 8:00 PM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> wrote:
> Nah, if you can quickly figure out which shard a given query maps to, then
> all
> this component needs to do is stick the appropriate shards param value in
> the
> request and let the request pass through to the other SearchComponents in
> the
> chain, including QueryComponent, which will know what to do with the
> shards
> param.
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
> > From: Dmitry Kan <dm...@gmail.com>
> > To: solr-user@lucene.apache.org
> > Sent: Fri, June 3, 2011 12:56:15 PM
> > Subject: Re: query routing with shards
> >
> > Hi Otis,
> >
> > Thanks! This sounds promising. This custom implementation, will it hurt
> in
> > any way the stability of the front end SOLR? After implementing it, can
> I
> > run some tests to verify the stability / performance?
> >
> > Dmitry
> > On Fri, Jun 3, 2011 at 4:49 PM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com
> > > wrote:
> >
> > > Hi Dmitry,
> > >
> > > Yes, you could also implement your own custom SearchComponent. In
> this
> > > component you could grab the query param, examine the query value, and
> > > based on
> > > that add the shards URL param with appropriate value, so that when the
> > > regular
> > > QueryComponent grabs stuff from the request, it has the correct shard
> in
> > > there
> > > already.
> > >
> > > Otis
> > > ----
> > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > > Lucene ecosystem search :: http://search-lucene.com/
> > >
> > >
> > >
> > > ----- Original Message ----
> > > > From: Dmitry Kan <dm...@gmail.com>
> > > > To: solr-user@lucene.apache.org
> > > > Sent: Fri, June 3, 2011 2:47:00 AM
> > > > Subject: Re: query routing with shards
> > > >
> > > > Hi Otis,
> > > >
> > > > I merely followed on the gmail's suggestion to include other people
> into
> > > the
> > > > recipients list, Yonik was the first one :) I won't do it next
> time.
> > > >
> > > > Thanks for a rapid reply. The reason for doing this query routing
> is
> > > that we
> > > > abstract the distributed SOLR from the client code for security
> reasons
> > > > (that is, we don't want to expose the entire shard farm to the
> world,
> > > but
> > > > only the frontend SOLR) and for better decoupling.
> > > >
> > > > Is it possible to implement a plugin to SOLR that would map queries
> to
> > > > shards?
> > > >
> > > > We have other choices too, they'll take quite some time, that's why
> I
> > > > decided to quickly ask, if I was missing something from the SOLR
> main
> > > > components design and configuration.
> > > >
> > > > Dmitry
> > > >
> > > > On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <
> > > otis_gospodnetic@yahoo.com
> > > > > wrote:
> > > >
> > > > > Hi Dmitry (you may not want to additionally copy Yonik, he's
> > > subscribed to
> > > > > this
> > > > > list, too)
> > > > >
> > > > >
> > > > > It sounds like you have the knowledge of which query maps to
> which
> > > shard.
> > > > > If
> > > > > so, why not control/change the value of "shards" param in the
> request
> > > to
> > > > > your
> > > > > front-end Solr (aka distributed request dispatcher) within your
> app,
> > > which
> > > > > is
> > > > > the one calling Solr?
> > > > >
> > > > > Otis
> > > > > ----
> > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > > > > Lucene ecosystem search :: http://search-lucene.com/
> > > > >
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > > From: Dmitry Kan <dm...@gmail.com>
> > > > > > To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> > > > > > Sent: Thu, June 2, 2011 7:00:53 AM
> > > > > > Subject: query routing with shards
> > > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > We have currently several pretty fat logically isolated shards
> with
> > > the
> > > > > same
> > > > > > schema / solrconfig (indices are separate). We currently have
> one
> > > single
> > > > > > front end SOLR (1.4) for the client code calls. Since a client
> code
> > > > > query
> > > > > > usually hits only one shard, we are considering making a smart
> > > routing
> > > > > of
> > > > > > queries to the shards they map to. Can you please give some
> > > pointers as
> > > > > to
> > > > > > what would be an optimal way to achieve such a routing inside
> the
> > > front
> > > > > end
> > > > > > solr? Is there a way to configure mapping inside the
> solrconfig?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Dmitry Kan
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Dmitry Kan
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
Re: query routing with shards
Posted by Dmitry Kan <dm...@gmail.com>.
Got it, I can quickly figure the shard out, thanks a lot Otis!
Dmitry
On Fri, Jun 3, 2011 at 8:00 PM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> wrote:
> Nah, if you can quickly figure out which shard a given query maps to, then
> all
> this component needs to do is stick the appropriate shards param value in
> the
> request and let the request pass through to the other SearchComponents in
> the
> chain, including QueryComponent, which will know what to do with the
> shards
> param.
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
> > From: Dmitry Kan <dm...@gmail.com>
> > To: solr-user@lucene.apache.org
> > Sent: Fri, June 3, 2011 12:56:15 PM
> > Subject: Re: query routing with shards
> >
> > Hi Otis,
> >
> > Thanks! This sounds promising. This custom implementation, will it hurt
> in
> > any way the stability of the front end SOLR? After implementing it, can
> I
> > run some tests to verify the stability / performance?
> >
> > Dmitry
> > On Fri, Jun 3, 2011 at 4:49 PM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com
> > > wrote:
> >
> > > Hi Dmitry,
> > >
> > > Yes, you could also implement your own custom SearchComponent. In
> this
> > > component you could grab the query param, examine the query value, and
> > > based on
> > > that add the shards URL param with appropriate value, so that when the
> > > regular
> > > QueryComponent grabs stuff from the request, it has the correct shard
> in
> > > there
> > > already.
> > >
> > > Otis
> > > ----
> > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > > Lucene ecosystem search :: http://search-lucene.com/
> > >
> > >
> > >
> > > ----- Original Message ----
> > > > From: Dmitry Kan <dm...@gmail.com>
> > > > To: solr-user@lucene.apache.org
> > > > Sent: Fri, June 3, 2011 2:47:00 AM
> > > > Subject: Re: query routing with shards
> > > >
> > > > Hi Otis,
> > > >
> > > > I merely followed on the gmail's suggestion to include other people
> into
> > > the
> > > > recipients list, Yonik was the first one :) I won't do it next
> time.
> > > >
> > > > Thanks for a rapid reply. The reason for doing this query routing
> is
> > > that we
> > > > abstract the distributed SOLR from the client code for security
> reasons
> > > > (that is, we don't want to expose the entire shard farm to the
> world,
> > > but
> > > > only the frontend SOLR) and for better decoupling.
> > > >
> > > > Is it possible to implement a plugin to SOLR that would map queries
> to
> > > > shards?
> > > >
> > > > We have other choices too, they'll take quite some time, that's why
> I
> > > > decided to quickly ask, if I was missing something from the SOLR
> main
> > > > components design and configuration.
> > > >
> > > > Dmitry
> > > >
> > > > On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <
> > > otis_gospodnetic@yahoo.com
> > > > > wrote:
> > > >
> > > > > Hi Dmitry (you may not want to additionally copy Yonik, he's
> > > subscribed to
> > > > > this
> > > > > list, too)
> > > > >
> > > > >
> > > > > It sounds like you have the knowledge of which query maps to
> which
> > > shard.
> > > > > If
> > > > > so, why not control/change the value of "shards" param in the
> request
> > > to
> > > > > your
> > > > > front-end Solr (aka distributed request dispatcher) within your
> app,
> > > which
> > > > > is
> > > > > the one calling Solr?
> > > > >
> > > > > Otis
> > > > > ----
> > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > > > > Lucene ecosystem search :: http://search-lucene.com/
> > > > >
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > > From: Dmitry Kan <dm...@gmail.com>
> > > > > > To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> > > > > > Sent: Thu, June 2, 2011 7:00:53 AM
> > > > > > Subject: query routing with shards
> > > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > We have currently several pretty fat logically isolated shards
> with
> > > the
> > > > > same
> > > > > > schema / solrconfig (indices are separate). We currently have
> one
> > > single
> > > > > > front end SOLR (1.4) for the client code calls. Since a client
> code
> > > > > query
> > > > > > usually hits only one shard, we are considering making a smart
> > > routing
> > > > > of
> > > > > > queries to the shards they map to. Can you please give some
> > > pointers as
> > > > > to
> > > > > > what would be an optimal way to achieve such a routing inside
> the
> > > front
> > > > > end
> > > > > > solr? Is there a way to configure mapping inside the
> solrconfig?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Dmitry Kan
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Dmitry Kan
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
--
Regards,
Dmitry Kan
Re: query routing with shards
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Nah, if you can quickly figure out which shard a given query maps to, then all
this component needs to do is stick the appropriate shards param value in the
request and let the request pass through to the other SearchComponents in the
chain, including QueryComponent, which will know what to do with the shards
param.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Dmitry Kan <dm...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, June 3, 2011 12:56:15 PM
> Subject: Re: query routing with shards
>
> Hi Otis,
>
> Thanks! This sounds promising. This custom implementation, will it hurt in
> any way the stability of the front end SOLR? After implementing it, can I
> run some tests to verify the stability / performance?
>
> Dmitry
> On Fri, Jun 3, 2011 at 4:49 PM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> > wrote:
>
> > Hi Dmitry,
> >
> > Yes, you could also implement your own custom SearchComponent. In this
> > component you could grab the query param, examine the query value, and
> > based on
> > that add the shards URL param with appropriate value, so that when the
> > regular
> > QueryComponent grabs stuff from the request, it has the correct shard in
> > there
> > already.
> >
> > Otis
> > ----
> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > Lucene ecosystem search :: http://search-lucene.com/
> >
> >
> >
> > ----- Original Message ----
> > > From: Dmitry Kan <dm...@gmail.com>
> > > To: solr-user@lucene.apache.org
> > > Sent: Fri, June 3, 2011 2:47:00 AM
> > > Subject: Re: query routing with shards
> > >
> > > Hi Otis,
> > >
> > > I merely followed on the gmail's suggestion to include other people
into
> > the
> > > recipients list, Yonik was the first one :) I won't do it next time.
> > >
> > > Thanks for a rapid reply. The reason for doing this query routing is
> > that we
> > > abstract the distributed SOLR from the client code for security reasons
> > > (that is, we don't want to expose the entire shard farm to the world,
> > but
> > > only the frontend SOLR) and for better decoupling.
> > >
> > > Is it possible to implement a plugin to SOLR that would map queries to
> > > shards?
> > >
> > > We have other choices too, they'll take quite some time, that's why I
> > > decided to quickly ask, if I was missing something from the SOLR main
> > > components design and configuration.
> > >
> > > Dmitry
> > >
> > > On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <
> > otis_gospodnetic@yahoo.com
> > > > wrote:
> > >
> > > > Hi Dmitry (you may not want to additionally copy Yonik, he's
> > subscribed to
> > > > this
> > > > list, too)
> > > >
> > > >
> > > > It sounds like you have the knowledge of which query maps to which
> > shard.
> > > > If
> > > > so, why not control/change the value of "shards" param in the request
> > to
> > > > your
> > > > front-end Solr (aka distributed request dispatcher) within your app,
> > which
> > > > is
> > > > the one calling Solr?
> > > >
> > > > Otis
> > > > ----
> > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > > > Lucene ecosystem search :: http://search-lucene.com/
> > > >
> > > >
> > > >
> > > > ----- Original Message ----
> > > > > From: Dmitry Kan <dm...@gmail.com>
> > > > > To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> > > > > Sent: Thu, June 2, 2011 7:00:53 AM
> > > > > Subject: query routing with shards
> > > > >
> > > > > Hello all,
> > > > >
> > > > > We have currently several pretty fat logically isolated shards with
> > the
> > > > same
> > > > > schema / solrconfig (indices are separate). We currently have one
> > single
> > > > > front end SOLR (1.4) for the client code calls. Since a client
code
> > > > query
> > > > > usually hits only one shard, we are considering making a smart
> > routing
> > > > of
> > > > > queries to the shards they map to. Can you please give some
> > pointers as
> > > > to
> > > > > what would be an optimal way to achieve such a routing inside the
> > front
> > > > end
> > > > > solr? Is there a way to configure mapping inside the solrconfig?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Dmitry Kan
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Dmitry Kan
> > >
> >
>
>
>
> --
> Regards,
>
> Dmitry Kan
>
Re: query routing with shards
Posted by Dmitry Kan <dm...@gmail.com>.
Hi Otis,
Thanks! This sounds promising. This custom implementation, will it hurt in
any way the stability of the front end SOLR? After implementing it, can I
run some tests to verify the stability / performance?
Dmitry
On Fri, Jun 3, 2011 at 4:49 PM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> wrote:
> Hi Dmitry,
>
> Yes, you could also implement your own custom SearchComponent. In this
> component you could grab the query param, examine the query value, and
> based on
> that add the shards URL param with appropriate value, so that when the
> regular
> QueryComponent grabs stuff from the request, it has the correct shard in
> there
> already.
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
> > From: Dmitry Kan <dm...@gmail.com>
> > To: solr-user@lucene.apache.org
> > Sent: Fri, June 3, 2011 2:47:00 AM
> > Subject: Re: query routing with shards
> >
> > Hi Otis,
> >
> > I merely followed on the gmail's suggestion to include other people into
> the
> > recipients list, Yonik was the first one :) I won't do it next time.
> >
> > Thanks for a rapid reply. The reason for doing this query routing is
> that we
> > abstract the distributed SOLR from the client code for security reasons
> > (that is, we don't want to expose the entire shard farm to the world,
> but
> > only the frontend SOLR) and for better decoupling.
> >
> > Is it possible to implement a plugin to SOLR that would map queries to
> > shards?
> >
> > We have other choices too, they'll take quite some time, that's why I
> > decided to quickly ask, if I was missing something from the SOLR main
> > components design and configuration.
> >
> > Dmitry
> >
> > On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com
> > > wrote:
> >
> > > Hi Dmitry (you may not want to additionally copy Yonik, he's
> subscribed to
> > > this
> > > list, too)
> > >
> > >
> > > It sounds like you have the knowledge of which query maps to which
> shard.
> > > If
> > > so, why not control/change the value of "shards" param in the request
> to
> > > your
> > > front-end Solr (aka distributed request dispatcher) within your app,
> which
> > > is
> > > the one calling Solr?
> > >
> > > Otis
> > > ----
> > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > > Lucene ecosystem search :: http://search-lucene.com/
> > >
> > >
> > >
> > > ----- Original Message ----
> > > > From: Dmitry Kan <dm...@gmail.com>
> > > > To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> > > > Sent: Thu, June 2, 2011 7:00:53 AM
> > > > Subject: query routing with shards
> > > >
> > > > Hello all,
> > > >
> > > > We have currently several pretty fat logically isolated shards with
> the
> > > same
> > > > schema / solrconfig (indices are separate). We currently have one
> single
> > > > front end SOLR (1.4) for the client code calls. Since a client code
> > > query
> > > > usually hits only one shard, we are considering making a smart
> routing
> > > of
> > > > queries to the shards they map to. Can you please give some
> pointers as
> > > to
> > > > what would be an optimal way to achieve such a routing inside the
> front
> > > end
> > > > solr? Is there a way to configure mapping inside the solrconfig?
> > > >
> > > > Thanks.
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Dmitry Kan
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
--
Regards,
Dmitry Kan
Re: query routing with shards
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Dmitry,
Yes, you could also implement your own custom SearchComponent. In this
component you could grab the query param, examine the query value, and based on
that add the shards URL param with appropriate value, so that when the regular
QueryComponent grabs stuff from the request, it has the correct shard in there
already.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Dmitry Kan <dm...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, June 3, 2011 2:47:00 AM
> Subject: Re: query routing with shards
>
> Hi Otis,
>
> I merely followed on the gmail's suggestion to include other people into the
> recipients list, Yonik was the first one :) I won't do it next time.
>
> Thanks for a rapid reply. The reason for doing this query routing is that we
> abstract the distributed SOLR from the client code for security reasons
> (that is, we don't want to expose the entire shard farm to the world, but
> only the frontend SOLR) and for better decoupling.
>
> Is it possible to implement a plugin to SOLR that would map queries to
> shards?
>
> We have other choices too, they'll take quite some time, that's why I
> decided to quickly ask, if I was missing something from the SOLR main
> components design and configuration.
>
> Dmitry
>
> On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> > wrote:
>
> > Hi Dmitry (you may not want to additionally copy Yonik, he's subscribed to
> > this
> > list, too)
> >
> >
> > It sounds like you have the knowledge of which query maps to which shard.
> > If
> > so, why not control/change the value of "shards" param in the request to
> > your
> > front-end Solr (aka distributed request dispatcher) within your app, which
> > is
> > the one calling Solr?
> >
> > Otis
> > ----
> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> > Lucene ecosystem search :: http://search-lucene.com/
> >
> >
> >
> > ----- Original Message ----
> > > From: Dmitry Kan <dm...@gmail.com>
> > > To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> > > Sent: Thu, June 2, 2011 7:00:53 AM
> > > Subject: query routing with shards
> > >
> > > Hello all,
> > >
> > > We have currently several pretty fat logically isolated shards with the
> > same
> > > schema / solrconfig (indices are separate). We currently have one single
> > > front end SOLR (1.4) for the client code calls. Since a client code
> > query
> > > usually hits only one shard, we are considering making a smart routing
> > of
> > > queries to the shards they map to. Can you please give some pointers as
> > to
> > > what would be an optimal way to achieve such a routing inside the front
> > end
> > > solr? Is there a way to configure mapping inside the solrconfig?
> > >
> > > Thanks.
> > >
> > > --
> > > Regards,
> > >
> > > Dmitry Kan
> > >
> >
>
>
>
> --
> Regards,
>
> Dmitry Kan
>
Re: query routing with shards
Posted by Dmitry Kan <dm...@gmail.com>.
Hi Otis,
I merely followed on the gmail's suggestion to include other people into the
recipients list, Yonik was the first one :) I won't do it next time.
Thanks for a rapid reply. The reason for doing this query routing is that we
abstract the distributed SOLR from the client code for security reasons
(that is, we don't want to expose the entire shard farm to the world, but
only the frontend SOLR) and for better decoupling.
Is it possible to implement a plugin to SOLR that would map queries to
shards?
We have other choices too, they'll take quite some time, that's why I
decided to quickly ask, if I was missing something from the SOLR main
components design and configuration.
Dmitry
On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> wrote:
> Hi Dmitry (you may not want to additionally copy Yonik, he's subscribed to
> this
> list, too)
>
>
> It sounds like you have the knowledge of which query maps to which shard.
> If
> so, why not control/change the value of "shards" param in the request to
> your
> front-end Solr (aka distributed request dispatcher) within your app, which
> is
> the one calling Solr?
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
> > From: Dmitry Kan <dm...@gmail.com>
> > To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> > Sent: Thu, June 2, 2011 7:00:53 AM
> > Subject: query routing with shards
> >
> > Hello all,
> >
> > We have currently several pretty fat logically isolated shards with the
> same
> > schema / solrconfig (indices are separate). We currently have one single
> > front end SOLR (1.4) for the client code calls. Since a client code
> query
> > usually hits only one shard, we are considering making a smart routing
> of
> > queries to the shards they map to. Can you please give some pointers as
> to
> > what would be an optimal way to achieve such a routing inside the front
> end
> > solr? Is there a way to configure mapping inside the solrconfig?
> >
> > Thanks.
> >
> > --
> > Regards,
> >
> > Dmitry Kan
> >
>
--
Regards,
Dmitry Kan
Re: query routing with shards
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Dmitry (you may not want to additionally copy Yonik, he's subscribed to this
list, too)
It sounds like you have the knowledge of which query maps to which shard. If
so, why not control/change the value of "shards" param in the request to your
front-end Solr (aka distributed request dispatcher) within your app, which is
the one calling Solr?
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Dmitry Kan <dm...@gmail.com>
> To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> Sent: Thu, June 2, 2011 7:00:53 AM
> Subject: query routing with shards
>
> Hello all,
>
> We have currently several pretty fat logically isolated shards with the same
> schema / solrconfig (indices are separate). We currently have one single
> front end SOLR (1.4) for the client code calls. Since a client code query
> usually hits only one shard, we are considering making a smart routing of
> queries to the shards they map to. Can you please give some pointers as to
> what would be an optimal way to achieve such a routing inside the front end
> solr? Is there a way to configure mapping inside the solrconfig?
>
> Thanks.
>
> --
> Regards,
>
> Dmitry Kan
>