Dynamic core selection

[Dzmitry Petrushenka <dp...@gmail.com>]

Hi All!

I need to be able to send requests to 2 different cores based on the value  
of some request parameter.

First core (active) contains most recent docs. This core is used in 99% of  
cases.

The second core (it has 100-1000 times more docs then active core) and  
used in 0.1% of cases.

We wrote our own search handler (mostly based on the standard one but  
handling our own custom params) and I wonder if there is a way to  
customize Solr so we could direct calls to the required core based on  
request params user passes?

Any help would be helpful.

Thanx,

Re: Dynamic core selection

[Amit Nithian <an...@gmail.com>]

I have done something similar in a search component for our search at
Zvents.com. Our use case is where we have a core that invokes searches in
other cores and merges the results together
Basically we have:
1) FederatedComponent implements SolrCoreAware
--> Grab instance of CoreContainer and store (mCoreContainer =
core.getCoreDescriptor().getCoreContainer();)
2) In the process method:
* grab the core requested (SolrCore core = mCoreContainer
.getCore(sCoreName);)
* grab the handler necessary to execute the request (SolrRequestHandler
handler = mCore.getRequestHandler(mQueryParams.get(CommonParams.QT));)
* Create a blank response object to hold the results (SolrQueryResponse
tempResponse = new SolrQueryResponse();)
* create a local query request (newReq = new LocalSolrQueryRequest(mCore,
mQueryParams); )
* execute the request: mCore.execute(handler, newReq, tempResponse);
* Now to avoid your "meta core" from having to deal with schemas from the
child cores, I find the DocList in the tempResponse and convert it to a
SolrDocumentList  (SolrDocumentList newDocList =
SolrPluginUtils.docListToSolrDocumentList(response, searcher,
returnFieldsSet, null);)

Now you can place this in an actual response object that gets returned from
the process() method. The nice part is that the writers that convert to
XML/ JSON etc will understand how to serialize SolrDocumentList which is
nice. This is important for me because otherwise the metacore's
"schema.xml" would have to be a union of any children core's schemas if you
are serializing a DocList out which I didn't want to have.

This is a lot simpler than mucking with the dispatch filters.

Hope this helps!
Amit


On Fri, Nov 2, 2012 at 9:45 AM, Dzmitry Petrushenka <dp...@gmail.com>wrote:

> Hi all!
>
> Just sharing the solution)
>
> I've extended SolrDispatchFilter with my own implementation and did like
> this:
>
> ...
>
> String core = determineCore(req);
> super.doFilter(new CoreRoutingReqWrapper(req, core), response, chain);
>
> ...
>
> code for the CoreRoutingReqWrapper class:
>
> class CoreRoutingReqWrapper extends HttpServletRequestWrapper {
>         private String pathToCore;
>
>         public CoreRoutingReqWrapper(**HttpServletRequest request, String
> core) {
>             super(request);
>             pathToCore = "/" + core + request.getServletPath();
>         }
>
>         @Override
>         public String getServletPath() {
>             return pathToCore;
>         }
>     }
>
> Would be nice to have something like CoreResolver component in Solr
> architecture.
>
> Something like this:
>
> interface CoreResolver {
>         String resolveCore(HttpServlerRequest req);
> }
>
> Would make Solr server more customizable.
>
> What do you think?
>
> Thanx,
>
>
>
>> : as I said we have our own search handler (wrapping handleRequestBody
>> method
>> : and adding logic before it) where we convert those custom_paramX params
>> into
>> : Solr understandable params like q, fq, facet, etc. Then we delegate to
>> Solr to
>> : process them.
>> :
>> : So what I want to do is core0 handle things if custom_param1=aaa and
>> core1 if
>> : custom_param1=ccc.
>>
>> Ah.. i think i'm understanding:
>>  * you know you need a custom search handler
>>  * you have a custom search handler that delegates to some other handler
>> based on some logic
>>  * your customer handler modifies the request params before delegating to
>> the handler it picks.
>>  * the part you are missing is how to delegate to an entirely differnet
>> SolrCore.
>>
>> does that capture your question?
>>
>> The nutshell is you would need to ask your current SolrCore for access to
>> the CoreContainer -- then create a new "LocalSolrQueryRequest" and ask
>> that SolrCore to execute it.  one hitch to watch out for is keeping track
>> of thinkgs like the SolrIndexSearcher used -- because stuff like "DocList"
>> values in the response will come from the *other* SolrIndexSearcher, and
>> you'll need to use that when writting the response out (because the
>> QueryResponseWriter needs to as the SolrInexSearcher for the stored fields
>> from those docids).
>>
>> (Note: i have never tried this ... there may be other gotcha's i'm not
>> aware of)
>>
>>
>>
>>
>>
>>
>> -Hoss
>>
>
>
> --
> Using Opera's revolutionary email client: http://www.opera.com/mail/
>

Re: Dynamic core selection

[Dzmitry Petrushenka <dp...@gmail.com>]

Hi all!

Just sharing the solution)

I've extended SolrDispatchFilter with my own implementation and did like  
this:

...

String core = determineCore(req);
super.doFilter(new CoreRoutingReqWrapper(req, core), response, chain);

...

code for the CoreRoutingReqWrapper class:

class CoreRoutingReqWrapper extends HttpServletRequestWrapper {
         private String pathToCore;

         public CoreRoutingReqWrapper(HttpServletRequest request, String  
core) {
             super(request);
             pathToCore = "/" + core + request.getServletPath();
         }

         @Override
         public String getServletPath() {
             return pathToCore;
         }
     }

Would be nice to have something like CoreResolver component in Solr  
architecture.

Something like this:

interface CoreResolver {
	String resolveCore(HttpServlerRequest req);
}

Would make Solr server more customizable.

What do you think?

Thanx,

>
> : as I said we have our own search handler (wrapping handleRequestBody  
> method
> : and adding logic before it) where we convert those custom_paramX  
> params into
> : Solr understandable params like q, fq, facet, etc. Then we delegate to  
> Solr to
> : process them.
> :
> : So what I want to do is core0 handle things if custom_param1=aaa and  
> core1 if
> : custom_param1=ccc.
>
> Ah.. i think i'm understanding:
>  * you know you need a custom search handler
>  * you have a custom search handler that delegates to some other handler
> based on some logic
>  * your customer handler modifies the request params before delegating to
> the handler it picks.
>  * the part you are missing is how to delegate to an entirely differnet
> SolrCore.
>
> does that capture your question?
>
> The nutshell is you would need to ask your current SolrCore for access to
> the CoreContainer -- then create a new "LocalSolrQueryRequest" and ask
> that SolrCore to execute it.  one hitch to watch out for is keeping track
> of thinkgs like the SolrIndexSearcher used -- because stuff like  
> "DocList"
> values in the response will come from the *other* SolrIndexSearcher, and
> you'll need to use that when writting the response out (because the
> QueryResponseWriter needs to as the SolrInexSearcher for the stored  
> fields
> from those docids).
>
> (Note: i have never tried this ... there may be other gotcha's i'm not
> aware of)
>
>
>
>
>
>
> -Hoss


-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Re: Dynamic core selection

[Chris Hostetter <ho...@fucit.org>]

: as I said we have our own search handler (wrapping handleRequestBody method
: and adding logic before it) where we convert those custom_paramX params into
: Solr understandable params like q, fq, facet, etc. Then we delegate to Solr to
: process them.
: 
: So what I want to do is core0 handle things if custom_param1=aaa and core1 if
: custom_param1=ccc.

Ah.. i think i'm understanding:
 * you know you need a custom search handler
 * you have a custom search handler that delegates to some other handler 
based on some logic
 * your customer handler modifies the request params before delegating to
the handler it picks.
 * the part you are missing is how to delegate to an entirely differnet 
SolrCore.

does that capture your question?

The nutshell is you would need to ask your current SolrCore for access to 
the CoreContainer -- then create a new "LocalSolrQueryRequest" and ask 
that SolrCore to execute it.  one hitch to watch out for is keeping track 
of thinkgs like the SolrIndexSearcher used -- because stuff like "DocList" 
values in the response will come from the *other* SolrIndexSearcher, and 
you'll need to use that when writting the response out (because the 
QueryResponseWriter needs to as the SolrInexSearcher for the stored fields 
from those docids).

(Note: i have never tried this ... there may be other gotcha's i'm not 
aware of)






-Hoss

Re: Dynamic core selection

[Dzmitry Petrushenka <dp...@gmail.com>]

The idea is to send req to http://server.com/solr.

Here are some sample URLs:

http://server.com/solr?custom_param1=aaa&custom_param2=bbb

http://server.com/solr?custom_param1=ccc&custom_param2=dddd

as I said we have our own search handler (wrapping handleRequestBody  
method and adding logic before it) where we convert those custom_paramX  
params into Solr understandable params like q, fq, facet, etc. Then we  
delegate to Solr to process them.

So what I want to do is core0 handle things if custom_param1=aaa and core1  
if custom_param1=ccc.


So I'm looking for another way of core selection then selection by URL  
path part.

Thanx,



> If I understand you correctly, you would use a multicore setup and send  
> the
> request to http://server.com/solr/core0 in one case, and
> http://server.com/solr/core1 in the other.
>
> Is there something else that makes this complicated?
>
> cheers,
>
> Travis
>
> On Thu, Nov 1, 2012 at 12:08 PM, Dzmitry Petrushenka  
> <dp...@gmail.com>wrote:
>
>> Hi All!
>>
>> I need to be able to send requests to 2 different cores based on the  
>> value
>> of some request parameter.
>>
>> First core (active) contains most recent docs. This core is used in 99%  
>> of
>> cases.
>>
>> The second core (it has 100-1000 times more docs then active core) and
>> used in 0.1% of cases.
>>
>> We wrote our own search handler (mostly based on the standard one but
>> handling our own custom params) and I wonder if there is a way to  
>> customize
>> Solr so we could direct calls to the required core based on request  
>> params
>> user passes?
>>
>> Any help would be helpful.
>>
>> Thanx,
>>
>
>
>


-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

Re: Dynamic core selection

[Travis Low <tl...@4centurion.com>]

If I understand you correctly, you would use a multicore setup and send the
request to http://server.com/solr/core0 in one case, and
http://server.com/solr/core1 in the other.

Is there something else that makes this complicated?

cheers,

Travis

On Thu, Nov 1, 2012 at 12:08 PM, Dzmitry Petrushenka <dp...@gmail.com>wrote:

> Hi All!
>
> I need to be able to send requests to 2 different cores based on the value
> of some request parameter.
>
> First core (active) contains most recent docs. This core is used in 99% of
> cases.
>
> The second core (it has 100-1000 times more docs then active core) and
> used in 0.1% of cases.
>
> We wrote our own search handler (mostly based on the standard one but
> handling our own custom params) and I wonder if there is a way to customize
> Solr so we could direct calls to the required core based on request params
> user passes?
>
> Any help would be helpful.
>
> Thanx,
>



-- 

**

*Travis Low, Director of Development*


** <tl...@4centurion.com>* *

*Centurion Research Solutions, LLC*

*14048 ParkEast Circle *•* Suite 100 *•* Chantilly, VA 20151*

*703-956-6276 *•* 703-378-4474 (fax)*

*http://www.centurionresearch.com* <http://www.centurionresearch.com>

**The information contained in this email message is confidential and
protected from disclosure.  If you are not the intended recipient, any use
or dissemination of this communication, including attachments, is strictly
prohibited.  If you received this email message in error, please delete it
and immediately notify the sender.

This email message and any attachments have been scanned and are believed
to be free of malicious software and defects that might affect any computer
system in which they are received and opened. No responsibility is accepted
by Centurion Research Solutions, LLC for any loss or damage arising from
the content of this email.