You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com> on 2010/01/05 07:56:10 UTC

Why do we need SolrPluginUtils#optimizePreFetchDocs()

This looks like a hack. It currently only uses highlighter for
prefetching docs and fields . There is no standard way of other
components to take part in this.

We should either remove this altogether or have a standard way for all
components to take part in this.

-- 
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com

Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()

Posted by Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>.
ok I have opened a new issue https://issues.apache.org/jira/browse/SOLR-1702

On Tue, Jan 5, 2010 at 5:50 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Jan 5, 2010, at 6:52 AM, Noble Paul നോബിള്‍ नोब्ळ् wrote:
>
>> On Tue, Jan 5, 2010 at 4:52 PM, Grant Ingersoll <gs...@apache.org> wrote:
>>>
>>> On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള്‍ नोब्ळ् wrote:
>>>
>>>> This looks like a hack. It currently only uses highlighter for
>>>> prefetching docs and fields . There is no standard way of other
>>>> components to take part in this.
>>>
>>> Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important.  Of course, it doesn't help if you want to use Term Vectors w/ highlighter
>>>
>>>>
>>>> We should either remove this altogether
>>>
>>> -1.
>>>
>>>
>>>> or have a standard way for all
>>>> components to take part in this.
>>>
>>> Perhaps a component could register what fields it needs?  However, do you have a use case in mind?  What component would you like to have leverage this?
>>
>> I don't know. But the point is can we have a an interface
>> PrefetchAware (or anything nicer) and components can choose to return
>> the list of fields which it is interested in prefetching. I would like
>> to remove the Strong coupling of QueryComponent on highlighting.
>>
>
> Sounds reasonable to me.



-- 
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com

Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()

Posted by Grant Ingersoll <gs...@apache.org>.
On Jan 5, 2010, at 6:52 AM, Noble Paul നോബിള്‍ नोब्ळ् wrote:

> On Tue, Jan 5, 2010 at 4:52 PM, Grant Ingersoll <gs...@apache.org> wrote:
>> 
>> On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള്‍ नोब्ळ् wrote:
>> 
>>> This looks like a hack. It currently only uses highlighter for
>>> prefetching docs and fields . There is no standard way of other
>>> components to take part in this.
>> 
>> Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important.  Of course, it doesn't help if you want to use Term Vectors w/ highlighter
>> 
>>> 
>>> We should either remove this altogether
>> 
>> -1.
>> 
>> 
>>> or have a standard way for all
>>> components to take part in this.
>> 
>> Perhaps a component could register what fields it needs?  However, do you have a use case in mind?  What component would you like to have leverage this?
> 
> I don't know. But the point is can we have a an interface
> PrefetchAware (or anything nicer) and components can choose to return
> the list of fields which it is interested in prefetching. I would like
> to remove the Strong coupling of QueryComponent on highlighting.
> 

Sounds reasonable to me.

Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()

Posted by Chris Hostetter <ho...@fucit.org>.
: >>> This looks like a hack. It currently only uses highlighter for
: >>> prefetching docs and fields . There is no standard way of other
: >>> components to take part in this.

It was an optimization that improved the common when it was added, but it 
predates all of the search component stuff, so that's not much of a 
suprise that it's not easy for components to use.

: Or we can add a method to ResponseBuilder.addPrefetchFields(String[]
: fieldNames) and SearchComponents can use this in prepare()/process()
: to express interest in prefetching.

I would suggest using something like a FieldSelector instead of a 
String[], so that components wanting all files that match a rule don't 
have to manifest a large array.

I can't put my finger on it, but it feels like there is a larger issue 
here ... relating to SOLR-1298, and how/when a DocList might/should get 
manifested as DocumentList ...

It would be fairly easy to modify optimizePreFetchDocs to check properties 
on the ResponseBuilder (via the SolrQueryRequest) to decide which fields 
to ask for, but ultimately all optimizePreFetchDocs does is ask the 
SolrIndexSearche to load the doc and then throws it away (relying on the 
documentCache to have those fields handy for later use).  as long as we're 
changing that, we might as well make it do ... something ... better.


-Hoss

Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()

Posted by Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>.
2010/1/5 Noble Paul നോബിള്‍  नोब्ळ् <no...@corp.aol.com>:
> On Tue, Jan 5, 2010 at 4:52 PM, Grant Ingersoll <gs...@apache.org> wrote:
>>
>> On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള്‍ नोब्ळ् wrote:
>>
>>> This looks like a hack. It currently only uses highlighter for
>>> prefetching docs and fields . There is no standard way of other
>>> components to take part in this.
>>
>> Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important.  Of course, it doesn't help if you want to use Term Vectors w/ highlighter
>>
>>>
>>> We should either remove this altogether
>>
>> -1.
>>
>>
>>> or have a standard way for all
>>> components to take part in this.
>>
>> Perhaps a component could register what fields it needs?  However, do you have a use case in mind?  What component would you like to have leverage this?
>
> I don't know. But the point is can we have a an interface
> PrefetchAware (or anything nicer) and components can choose to return
> the list of fields which it is interested in prefetching. I would like
> to remove the Strong coupling of QueryComponent on highlighting.

Or we can add a method to ResponseBuilder.addPrefetchFields(String[]
fieldNames) and SearchComponents can use this in prepare()/process()
to express interest in prefetching.
>
>
>>
>> -Grant
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Systems Architect| AOL | http://aol.com
>



-- 
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com

Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()

Posted by Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>.
On Tue, Jan 5, 2010 at 4:52 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള്‍ नोब्ळ् wrote:
>
>> This looks like a hack. It currently only uses highlighter for
>> prefetching docs and fields . There is no standard way of other
>> components to take part in this.
>
> Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important.  Of course, it doesn't help if you want to use Term Vectors w/ highlighter
>
>>
>> We should either remove this altogether
>
> -1.
>
>
>> or have a standard way for all
>> components to take part in this.
>
> Perhaps a component could register what fields it needs?  However, do you have a use case in mind?  What component would you like to have leverage this?

I don't know. But the point is can we have a an interface
PrefetchAware (or anything nicer) and components can choose to return
the list of fields which it is interested in prefetching. I would like
to remove the Strong coupling of QueryComponent on highlighting.


>
> -Grant



-- 
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com

Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()

Posted by Grant Ingersoll <gs...@apache.org>.
On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള്‍ नोब्ळ् wrote:

> This looks like a hack. It currently only uses highlighter for
> prefetching docs and fields . There is no standard way of other
> components to take part in this.

Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important.  Of course, it doesn't help if you want to use Term Vectors w/ highlighter

> 
> We should either remove this altogether

-1.  


> or have a standard way for all
> components to take part in this.

Perhaps a component could register what fields it needs?  However, do you have a use case in mind?  What component would you like to have leverage this?

-Grant