You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Elaine Li <el...@gmail.com> on 2009/09/09 16:10:32 UTC
slow response
Hi,
I have 20 million docs on solr. If my query would return more than
10,000 results, the response time will be very very long. How to
resolve such problem? Can I slice my docs into pieces and let the
query operate within one piece at a time so the response time and
response data will be more managable? Thanks.
Elaine
Re: slow response
Posted by Alex Baranov <al...@gmail.com>.
Please, take a look at
http://issues.apache.org/jira/browse/SOLR-1379
Alex.
On Wed, Sep 9, 2009 at 5:28 PM, Constantijn Visinescu <ba...@gmail.com>wrote:
> Just wondering, is there an easy way to load the whole index into ram?
>
> On Wed, Sep 9, 2009 at 4:22 PM, Alex Baranov <alex.baranov.v@gmail.com
> >wrote:
>
> > There is a good article on how to scale the Lucene/Solr solution:
> >
> >
> >
> http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr
> >
> > Also, if you have heavy load on the server (large amount of concurrent
> > requests) then I'd suggest to consider loading the index into RAM. It
> > worked
> > well for me on the project with 140+ million documents and 30 concurrent
> > user requests per second. If your index can be placed in RAM you can
> reduce
> > the architecture complexity.
> >
> > Alex Baranov
> >
> > On Wed, Sep 9, 2009 at 5:10 PM, Elaine Li <el...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I have 20 million docs on solr. If my query would return more than
> > > 10,000 results, the response time will be very very long. How to
> > > resolve such problem? Can I slice my docs into pieces and let the
> > > query operate within one piece at a time so the response time and
> > > response data will be more managable? Thanks.
> > >
> > > Elaine
> > >
> >
>
Re: slow response
Posted by Constantijn Visinescu <ba...@gmail.com>.
Just wondering, is there an easy way to load the whole index into ram?
On Wed, Sep 9, 2009 at 4:22 PM, Alex Baranov <al...@gmail.com>wrote:
> There is a good article on how to scale the Lucene/Solr solution:
>
>
> http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr
>
> Also, if you have heavy load on the server (large amount of concurrent
> requests) then I'd suggest to consider loading the index into RAM. It
> worked
> well for me on the project with 140+ million documents and 30 concurrent
> user requests per second. If your index can be placed in RAM you can reduce
> the architecture complexity.
>
> Alex Baranov
>
> On Wed, Sep 9, 2009 at 5:10 PM, Elaine Li <el...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have 20 million docs on solr. If my query would return more than
> > 10,000 results, the response time will be very very long. How to
> > resolve such problem? Can I slice my docs into pieces and let the
> > query operate within one piece at a time so the response time and
> > response data will be more managable? Thanks.
> >
> > Elaine
> >
>
Re: slow response
Posted by Alex Baranov <al...@gmail.com>.
There is a good article on how to scale the Lucene/Solr solution:
http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr
Also, if you have heavy load on the server (large amount of concurrent
requests) then I'd suggest to consider loading the index into RAM. It worked
well for me on the project with 140+ million documents and 30 concurrent
user requests per second. If your index can be placed in RAM you can reduce
the architecture complexity.
Alex Baranov
On Wed, Sep 9, 2009 at 5:10 PM, Elaine Li <el...@gmail.com> wrote:
> Hi,
>
> I have 20 million docs on solr. If my query would return more than
> 10,000 results, the response time will be very very long. How to
> resolve such problem? Can I slice my docs into pieces and let the
> query operate within one piece at a time so the response time and
> response data will be more managable? Thanks.
>
> Elaine
>
Re: slow response
Posted by Elaine Li <el...@gmail.com>.
gwk, thanks a lot.
Elaine
On Wed, Sep 9, 2009 at 11:14 AM, gwk<gi...@eyefi.nl> wrote:
> Hi Elaine,
>
> You can page your resultset with the rows and start parameters
> (http://wiki.apache.org/solr/CommonQueryParameters). So for example to get
> the first 100 results one would use the parameters rows=100&start=0 and the
> second 100 results with rows=100&start=100 etc. etc.
>
> Regards,
>
> gwk
>
> Elaine Li wrote:
>>
>> gwk,
>>
>> Sorry for confusion. I am doing simple phrase search among the
>> sentences which could be in english or other language. Each doc has
>> only several id numbers and the sentence itself.
>>
>> I did not know about paging. Sounds like it is what I need. How to
>> achieve paging from solr?
>>
>> I also need to store all the results into my own tables in javascript
>> to use for connecting with other applications.
>>
>> Elaine
>>
>> On Wed, Sep 9, 2009 at 10:37 AM, gwk<gi...@eyefi.nl> wrote:
>>
>>>
>>> Hi Elaine,
>>>
>>> I think you need to provide us with some more information on what exactly
>>> you are trying to achieve. From your question I also assumed you wanted
>>> paging (getting the first 10 results, than the next 10 etc.) But reading
>>> it
>>> again, "slice my docs into pieces" I now think you might've meant that
>>> you
>>> only want to retrieve certain fields from each document. For that you can
>>> use the fl parameter
>>>
>>> (http://wiki.apache.org/solr/CommonQueryParameters#head-db2785986af2355759faaaca53dc8fd0b012d1ab).
>>> Hope this helps.
>>>
>>> Regards,
>>>
>>> gwk
>>>
>>> Elaine Li wrote:
>>>
>>>>
>>>> I want to get the 10K results, not just the top 10.
>>>> The fields are regular language sentences, they are not large.
>>>>
>>>> Is clustering the technique for what I am doing?
>>>>
>>>> On Wed, Sep 9, 2009 at 10:16 AM, Grant Ingersoll<gs...@apache.org>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> Do you need 10K results at a time or are you just getting the top 10 or
>>>>> so
>>>>> in a set of 10K? Also, are you retrieving really large stored fields?
>>>>> If
>>>>> you add &debugQuery=true to your request, Solr will return timing
>>>>> information for the various components.
>>>>>
>>>>>
>>>>> On Sep 9, 2009, at 10:10 AM, Elaine Li wrote:
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have 20 million docs on solr. If my query would return more than
>>>>>> 10,000 results, the response time will be very very long. How to
>>>>>> resolve such problem? Can I slice my docs into pieces and let the
>>>>>> query operate within one piece at a time so the response time and
>>>>>> response data will be more managable? Thanks.
>>>>>>
>>>>>> Elaine
>>>>>>
>>>>>>
>>>>>
>>>>> --------------------------
>>>>> Grant Ingersoll
>>>>> http://www.lucidimagination.com/
>>>>>
>>>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>>>>> using
>>>>> Solr/Lucene:
>>>>> http://www.lucidimagination.com/search
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>
>
Re: slow response
Posted by gwk <gi...@eyefi.nl>.
Hi Elaine,
You can page your resultset with the rows and start parameters
(http://wiki.apache.org/solr/CommonQueryParameters). So for example to
get the first 100 results one would use the parameters rows=100&start=0
and the second 100 results with rows=100&start=100 etc. etc.
Regards,
gwk
Elaine Li wrote:
> gwk,
>
> Sorry for confusion. I am doing simple phrase search among the
> sentences which could be in english or other language. Each doc has
> only several id numbers and the sentence itself.
>
> I did not know about paging. Sounds like it is what I need. How to
> achieve paging from solr?
>
> I also need to store all the results into my own tables in javascript
> to use for connecting with other applications.
>
> Elaine
>
> On Wed, Sep 9, 2009 at 10:37 AM, gwk<gi...@eyefi.nl> wrote:
>
>> Hi Elaine,
>>
>> I think you need to provide us with some more information on what exactly
>> you are trying to achieve. From your question I also assumed you wanted
>> paging (getting the first 10 results, than the next 10 etc.) But reading it
>> again, "slice my docs into pieces" I now think you might've meant that you
>> only want to retrieve certain fields from each document. For that you can
>> use the fl parameter
>> (http://wiki.apache.org/solr/CommonQueryParameters#head-db2785986af2355759faaaca53dc8fd0b012d1ab).
>> Hope this helps.
>>
>> Regards,
>>
>> gwk
>>
>> Elaine Li wrote:
>>
>>> I want to get the 10K results, not just the top 10.
>>> The fields are regular language sentences, they are not large.
>>>
>>> Is clustering the technique for what I am doing?
>>>
>>> On Wed, Sep 9, 2009 at 10:16 AM, Grant Ingersoll<gs...@apache.org>
>>> wrote:
>>>
>>>
>>>> Do you need 10K results at a time or are you just getting the top 10 or
>>>> so
>>>> in a set of 10K? Also, are you retrieving really large stored fields?
>>>> If
>>>> you add &debugQuery=true to your request, Solr will return timing
>>>> information for the various components.
>>>>
>>>>
>>>> On Sep 9, 2009, at 10:10 AM, Elaine Li wrote:
>>>>
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> I have 20 million docs on solr. If my query would return more than
>>>>> 10,000 results, the response time will be very very long. How to
>>>>> resolve such problem? Can I slice my docs into pieces and let the
>>>>> query operate within one piece at a time so the response time and
>>>>> response data will be more managable? Thanks.
>>>>>
>>>>> Elaine
>>>>>
>>>>>
>>>> --------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com/
>>>>
>>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
>>>> Solr/Lucene:
>>>> http://www.lucidimagination.com/search
>>>>
>>>>
>>>>
>>>>
>>
Re: slow response
Posted by Elaine Li <el...@gmail.com>.
gwk,
Sorry for confusion. I am doing simple phrase search among the
sentences which could be in english or other language. Each doc has
only several id numbers and the sentence itself.
I did not know about paging. Sounds like it is what I need. How to
achieve paging from solr?
I also need to store all the results into my own tables in javascript
to use for connecting with other applications.
Elaine
On Wed, Sep 9, 2009 at 10:37 AM, gwk<gi...@eyefi.nl> wrote:
> Hi Elaine,
>
> I think you need to provide us with some more information on what exactly
> you are trying to achieve. From your question I also assumed you wanted
> paging (getting the first 10 results, than the next 10 etc.) But reading it
> again, "slice my docs into pieces" I now think you might've meant that you
> only want to retrieve certain fields from each document. For that you can
> use the fl parameter
> (http://wiki.apache.org/solr/CommonQueryParameters#head-db2785986af2355759faaaca53dc8fd0b012d1ab).
> Hope this helps.
>
> Regards,
>
> gwk
>
> Elaine Li wrote:
>>
>> I want to get the 10K results, not just the top 10.
>> The fields are regular language sentences, they are not large.
>>
>> Is clustering the technique for what I am doing?
>>
>> On Wed, Sep 9, 2009 at 10:16 AM, Grant Ingersoll<gs...@apache.org>
>> wrote:
>>
>>>
>>> Do you need 10K results at a time or are you just getting the top 10 or
>>> so
>>> in a set of 10K? Also, are you retrieving really large stored fields?
>>> If
>>> you add &debugQuery=true to your request, Solr will return timing
>>> information for the various components.
>>>
>>>
>>> On Sep 9, 2009, at 10:10 AM, Elaine Li wrote:
>>>
>>>
>>>>
>>>> Hi,
>>>>
>>>> I have 20 million docs on solr. If my query would return more than
>>>> 10,000 results, the response time will be very very long. How to
>>>> resolve such problem? Can I slice my docs into pieces and let the
>>>> query operate within one piece at a time so the response time and
>>>> response data will be more managable? Thanks.
>>>>
>>>> Elaine
>>>>
>>>
>>> --------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com/
>>>
>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
>>> Solr/Lucene:
>>> http://www.lucidimagination.com/search
>>>
>>>
>>>
>
>
Re: slow response
Posted by gwk <gi...@eyefi.nl>.
Hi Elaine,
I think you need to provide us with some more information on what
exactly you are trying to achieve. From your question I also assumed you
wanted paging (getting the first 10 results, than the next 10 etc.) But
reading it again, "slice my docs into pieces" I now think you might've
meant that you only want to retrieve certain fields from each document.
For that you can use the fl parameter
(http://wiki.apache.org/solr/CommonQueryParameters#head-db2785986af2355759faaaca53dc8fd0b012d1ab).
Hope this helps.
Regards,
gwk
Elaine Li wrote:
> I want to get the 10K results, not just the top 10.
> The fields are regular language sentences, they are not large.
>
> Is clustering the technique for what I am doing?
>
> On Wed, Sep 9, 2009 at 10:16 AM, Grant Ingersoll<gs...@apache.org> wrote:
>
>> Do you need 10K results at a time or are you just getting the top 10 or so
>> in a set of 10K? Also, are you retrieving really large stored fields? If
>> you add &debugQuery=true to your request, Solr will return timing
>> information for the various components.
>>
>>
>> On Sep 9, 2009, at 10:10 AM, Elaine Li wrote:
>>
>>
>>> Hi,
>>>
>>> I have 20 million docs on solr. If my query would return more than
>>> 10,000 results, the response time will be very very long. How to
>>> resolve such problem? Can I slice my docs into pieces and let the
>>> query operate within one piece at a time so the response time and
>>> response data will be more managable? Thanks.
>>>
>>> Elaine
>>>
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
>> Solr/Lucene:
>> http://www.lucidimagination.com/search
>>
>>
>>
Re: slow response
Posted by Elaine Li <el...@gmail.com>.
I want to get the 10K results, not just the top 10.
The fields are regular language sentences, they are not large.
Is clustering the technique for what I am doing?
On Wed, Sep 9, 2009 at 10:16 AM, Grant Ingersoll<gs...@apache.org> wrote:
> Do you need 10K results at a time or are you just getting the top 10 or so
> in a set of 10K? Also, are you retrieving really large stored fields? If
> you add &debugQuery=true to your request, Solr will return timing
> information for the various components.
>
>
> On Sep 9, 2009, at 10:10 AM, Elaine Li wrote:
>
>> Hi,
>>
>> I have 20 million docs on solr. If my query would return more than
>> 10,000 results, the response time will be very very long. How to
>> resolve such problem? Can I slice my docs into pieces and let the
>> query operate within one piece at a time so the response time and
>> response data will be more managable? Thanks.
>>
>> Elaine
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
> Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
Re: slow response
Posted by Grant Ingersoll <gs...@apache.org>.
Do you need 10K results at a time or are you just getting the top 10
or so in a set of 10K? Also, are you retrieving really large stored
fields? If you add &debugQuery=true to your request, Solr will return
timing information for the various components.
On Sep 9, 2009, at 10:10 AM, Elaine Li wrote:
> Hi,
>
> I have 20 million docs on solr. If my query would return more than
> 10,000 results, the response time will be very very long. How to
> resolve such problem? Can I slice my docs into pieces and let the
> query operate within one piece at a time so the response time and
> response data will be more managable? Thanks.
>
> Elaine
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search