potential query performance issue

[Lin Ma <li...@gmail.com>]

Hello guys,

Supposing I have one million documents, and each document has hundreds of
features. For a given query, it also has hundreds of features. I want to
fetch most relevant top 1000 documents by dot product related features of
query and documents (query/document features are in the same feature space).

I am not sure how Lucene implement internally? If we have to go through all
one million document to dot product the query, then I am concerning about
the performance. Appreciate if anyone could confirm (1) how Lucene works
internally for this use case (2) any smart ideas to make improvement for
query efficiency to select top 1000 documents?

thanks in advance,
Lin

Re: potential query performance issue

[Lin Ma <li...@gmail.com>]

Thanks Lukai for the detailed reply,

   - "If you query is too long, it might not very efficient in query
   evaluation process. " -- how does Lucene query evaluation works? Is there
   any document to refer to?
   - "you can read out payload of the match term you have stored" -- what
   do you mean payload of the match term? Could you show me an example?

regards,
Lin

On Sun, Mar 17, 2013 at 7:13 AM, lukai <lu...@gmail.com> wrote:

>
>
> On Fri, Mar 15, 2013 at 10:02 PM, Lin Ma <li...@gmail.com> wrote:
>
>> Hi Lukai, thanks for the detailed reply.
>>
>> Some more comments,
>>
>>    - "You can try score by payload" -- what do you mean score by
>>    payload? Appreciate if you could provide a bit more details;
>>
>>      Write your own query/scorer, you can read out payload of the match
> term you have stored. You can implement your dot product functionality in
> score function of your scorer.
>
>>
>>    - "Lucene focus on search for the default implementation" -- for
>>    default you mean?
>>
>>     I mean the default query parser, query types are designed for search
> application. If you query is too long, it might not very efficient in query
> evaluation process.
>
>>
>>    - "For your requirement, you can do some query re-write process to
>>    reduce your query size" -- I think query re-write you mean rewrite "iPhone
>>    5", "iPhone 4S" to "iPhone" to reduce # of queries? Or you mean something
>>    else?
>>
>>    Query re-write, it really depends on your application. you can
> reduce/expand your query or even change the query type according your
> needs.
>
>>
>>    -
>>
>> regards,
>> Lin
>>
>>
>> On Sat, Mar 16, 2013 at 11:55 AM, lukai <lu...@gmail.com> wrote:
>>
>>> Different application has different requirement and resolve different
>>> problem. Lucene focus on search for the default implementation. For your
>>> requirement, you can do some query re-write process to reduce your query
>>> size if you still want to leverage the search functionality. If you just
>>> want to customize your feature value and do simple dot product calculation.
>>> You can try score by payload, it might not very efficient, cuz you still
>>> need to convert your query into some specified Lucene query type. But you
>>> still can leverage the existing index structure, NRT, distributed search
>>> support by Solr.
>>>
>>> When you refer to performance, it really depends on the document size,
>>> term distribution of your corpus. If you have enough machine, you can just
>>> try reduce document number per instance and distribute your search to
>>> achieve a better performance goal.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 15, 2013 at 7:36 PM, Lin Ma <li...@gmail.com> wrote:
>>>
>>>> Hi lukai, thanks for the reply. Do you mean WAND is a way to resolve
>>>> this issue? For "native support", do you mean there is no built-in
>>>> (existing ready to use externally open source) module in Lucene to
>>>> implement WAND? If so, the performance will really be bad.
>>>>
>>>> regards,
>>>> Lin
>>>>
>>>>
>>>> On Sat, Mar 16, 2013 at 2:49 AM, lukai <lu...@gmail.com> wrote:
>>>>
>>>>> I had implemented wand with solr/lucene. So far there is no performance
>>>>> issue.  There is no native support for this functionality, you need to
>>>>> implement it by yourself..
>>>>>
>>>>> On Fri, Mar 15, 2013 at 10:09 AM, Lin Ma <li...@gmail.com> wrote:
>>>>>
>>>>> > Hello guys,
>>>>> >
>>>>> > Supposing I have one million documents, and each document has
>>>>> hundreds of
>>>>> > features. For a given query, it also has hundreds of features. I
>>>>> want to
>>>>> > fetch most relevant top 1000 documents by dot product related
>>>>> features of
>>>>> > query and documents (query/document features are in the same feature
>>>>> > space).
>>>>> >
>>>>> > I am not sure how Lucene implement internally? If we have to go
>>>>> through all
>>>>> > one million document to dot product the query, then I am concerning
>>>>> about
>>>>> > the performance. Appreciate if anyone could confirm (1) how Lucene
>>>>> works
>>>>> > internally for this use case (2) any smart ideas to make improvement
>>>>> for
>>>>> > query efficiency to select top 1000 documents?
>>>>> >
>>>>> > thanks in advance,
>>>>> > Lin
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Re: potential query performance issue

[Lin Ma <li...@gmail.com>]

Hi lukai, thanks for the reply. Do you mean WAND is a way to resolve this
issue? For "native support", do you mean there is no built-in (existing
ready to use externally open source) module in Lucene to implement WAND? If
so, the performance will really be bad.

regards,
Lin

On Sat, Mar 16, 2013 at 2:49 AM, lukai <lu...@gmail.com> wrote:

> I had implemented wand with solr/lucene. So far there is no performance
> issue.  There is no native support for this functionality, you need to
> implement it by yourself..
>
> On Fri, Mar 15, 2013 at 10:09 AM, Lin Ma <li...@gmail.com> wrote:
>
> > Hello guys,
> >
> > Supposing I have one million documents, and each document has hundreds of
> > features. For a given query, it also has hundreds of features. I want to
> > fetch most relevant top 1000 documents by dot product related features of
> > query and documents (query/document features are in the same feature
> > space).
> >
> > I am not sure how Lucene implement internally? If we have to go through
> all
> > one million document to dot product the query, then I am concerning about
> > the performance. Appreciate if anyone could confirm (1) how Lucene works
> > internally for this use case (2) any smart ideas to make improvement for
> > query efficiency to select top 1000 documents?
> >
> > thanks in advance,
> > Lin
> >
>

Re: potential query performance issue

[lukai <lu...@gmail.com>]

I had implemented wand with solr/lucene. So far there is no performance
issue.  There is no native support for this functionality, you need to
implement it by yourself..

On Fri, Mar 15, 2013 at 10:09 AM, Lin Ma <li...@gmail.com> wrote:

> Hello guys,
>
> Supposing I have one million documents, and each document has hundreds of
> features. For a given query, it also has hundreds of features. I want to
> fetch most relevant top 1000 documents by dot product related features of
> query and documents (query/document features are in the same feature
> space).
>
> I am not sure how Lucene implement internally? If we have to go through all
> one million document to dot product the query, then I am concerning about
> the performance. Appreciate if anyone could confirm (1) how Lucene works
> internally for this use case (2) any smart ideas to make improvement for
> query efficiency to select top 1000 documents?
>
> thanks in advance,
> Lin
>