How to inject custom response data after results have been sorted

[tedsolr <ts...@sciquest.com>]

Hello! With the help of this community I have solved 2 problems on my way to
creating a search that collapses documents based on multiple fields. The
CollapsingQParserPlugin was key.

I have a new problem now. All the custom stats I generate in my custom
QParser makes for way to much data to simply write out in the response. I
need to filter that data so I only return the stats the user will see on one
page. Say my search returns 800K collapsed docs - in the
DelegatingCollector's collect() method I am computing some info for each
collapsed group - that's 800K map entries.

I can't filter the stats in my post filter implementation because the
results have not been sorted. So I need a new downstream component that can
read the sorted results, and grab the custom stats from my post filter. Can
someone recommend a suggestion? Is this a SearchComponent extension? Where
is the proper hook for examining results after sorting?



--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-inject-custom-response-data-after-results-have-been-sorted-tp4181545.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to inject custom response data after results have been sorted

[Joel Bernstein <jo...@gmail.com>]

Another thing to consider... If you only need custom stats for the current
result page then there is no need to keep stats for the full result set. In
this case you could perform your custom collapse and generate the stats
just for the current page. The ExpandComponent could be altered to do that
because it re-issues the query (and filters) and then collects the expanded
groups for the current page. You could also collect the stats for the
current page.

Joel Bernstein
Search Engineer at Heliosearch

On Fri, Jan 23, 2015 at 4:19 PM, tedsolr <ts...@sciquest.com> wrote:

> Thank you so much for your responses Hoss and Shalin. I gather the
> DocTransfomer allows manipulations to the doc list returned in the results.
> That is very cool. So the transformer has access to the Solr Request. I
> haven't seen the hook yet, but I believe you - I'll have to keep looking.
> It
> would certainly be cleaner to return my stats as "fields" within each doc.
> My plan was to attach the stats as a map to the response, and post process
> in my app.
>
> I was able to quickly mock up a custom SearchComponent and verify that it
> receives the doc list in sorted order, and that I could retrieve objects
> form the request context. So this search component would allow me to simply
> "paste" the filtered map of stats to the response.
>
> Is there a performance benefit one way or the other? Is it just easier in
> the DocTransformer since there is a method transform(doc, id) that must get
> called for every return doc?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-inject-custom-response-data-after-results-have-been-sorted-tp4181545p4181602.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: How to inject custom response data after results have been sorted

[tedsolr <ts...@sciquest.com>]

Thank you so much for your responses Hoss and Shalin. I gather the
DocTransfomer allows manipulations to the doc list returned in the results.
That is very cool. So the transformer has access to the Solr Request. I
haven't seen the hook yet, but I believe you - I'll have to keep looking. It
would certainly be cleaner to return my stats as "fields" within each doc.
My plan was to attach the stats as a map to the response, and post process
in my app.

I was able to quickly mock up a custom SearchComponent and verify that it
receives the doc list in sorted order, and that I could retrieve objects
form the request context. So this search component would allow me to simply
"paste" the filtered map of stats to the response.

Is there a performance benefit one way or the other? Is it just easier in
the DocTransformer since there is a method transform(doc, id) that must get
called for every return doc?



--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-inject-custom-response-data-after-results-have-been-sorted-tp4181545p4181602.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to inject custom response data after results have been sorted

[Chris Hostetter <ho...@fucit.org>]

: If you just need to transform an individual result, that can be done by a
: custom DocTransformer. But from your email, I think you need a custom
: SearchComponent.

if your PostFilter has already collected all of the info you need, and you 
now just wnat to return a subset of that information that corrispponds to 
the individual documents being returmed on the current "page" of results 
(ie: the current DocList) then a custom DocTransformer should probably be 
enough as long as your PostFilter puts the computed data in the request 
context.

see for example how the ElevatedMarkerFactory works in conjunction with 
the QueryElevationComponent...

https://cwiki.apache.org/confluence/display/solr/Transforming+Result+Documents
https://cwiki.apache.org/confluence/display/solr/The+Query+Elevation+Component

-Hoss
http://www.lucidworks.com/

Re: How to inject custom response data after results have been sorted

[Shalin Shekhar Mangar <sh...@gmail.com>]

If you just need to transform an individual result, that can be done by a
custom DocTransformer. But from your email, I think you need a custom
SearchComponent.

On Fri, Jan 23, 2015 at 6:23 PM, tedsolr <ts...@sciquest.com> wrote:

> Hello! With the help of this community I have solved 2 problems on my way
> to
> creating a search that collapses documents based on multiple fields. The
> CollapsingQParserPlugin was key.
>
> I have a new problem now. All the custom stats I generate in my custom
> QParser makes for way to much data to simply write out in the response. I
> need to filter that data so I only return the stats the user will see on
> one
> page. Say my search returns 800K collapsed docs - in the
> DelegatingCollector's collect() method I am computing some info for each
> collapsed group - that's 800K map entries.
>
> I can't filter the stats in my post filter implementation because the
> results have not been sorted. So I need a new downstream component that can
> read the sorted results, and grab the custom stats from my post filter. Can
> someone recommend a suggestion? Is this a SearchComponent extension? Where
> is the proper hook for examining results after sorting?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-inject-custom-response-data-after-results-have-been-sorted-tp4181545.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Regards,
Shalin Shekhar Mangar.