You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Chris Bamford <ch...@talktalk.net> on 2011/07/08 16:39:15 UTC
Large indexes
Hi
I was wondering how to improve search performance over a set of indexes like this:
27G K1-1/index
19G K1-2/index
24G K1-3/index
15G K1-4/index
19G K1-5/index
31G K2-1/index
16G K2-2/index
8.1G K2-3/index
12G K2-4/index
15G K2-5/index
In total it is ~192Gb. All have the same schema, with data partitioned by my application into separate
indexes (rather than just 1 big one).
I have tried searching with both a MultiSearcher and a ParallelMultiSearcher (with lots of memory), but it
is way too slow (sometimes hangs, otherwise it takes tens of minutes). Do you have any tips for improving speed?
I have reduced the stored data as much as possible (without using compression) and use a FieldSelector when querying
but neither seem to help..
Is this the right approach? If I query them one by one I response time is reasonable (a few seconds each), but I was
kind of hoping to treat them as one large index if possible.
Any thoughts welcomed.
Thanks,
- Chris
Re: Large indexes
Posted by Simon Willnauer <si...@googlemail.com>.
On Fri, Jul 8, 2011 at 4:50 PM, Ian Lea <ia...@gmail.com> wrote:
> There are lots of general tips at
> http://wiki.apache.org/lucene-java/ImproveSearchingSpeed.
>
> What version of lucene? Recent releases should be faster. Have you
> tried with one big index? If everything is running on the same server
> that may well be faster.
>
> Even on single indexes, response of a few seconds sounds high. Are the
> searches complex? Are you sorting? Reading lots of hits for each
> search?
>
>
> --
> Ian.
>
> On Fri, Jul 8, 2011 at 3:39 PM, Chris Bamford
> <ch...@talktalk.net> wrote:
>>
>> Hi
>>
>>
>> I was wondering how to improve search performance over a set of indexes like this:
>>
>> 27G K1-1/index
>> 19G K1-2/index
>> 24G K1-3/index
>> 15G K1-4/index
>> 19G K1-5/index
>> 31G K2-1/index
>> 16G K2-2/index
>> 8.1G K2-3/index
>> 12G K2-4/index
>> 15G K2-5/index
>>
>> In total it is ~192Gb. All have the same schema, with data partitioned by my application into separate
>> indexes (rather than just 1 big one).
>> I have tried searching with both a MultiSearcher and a ParallelMultiSearcher (with lots of memory), but it
>> is way too slow (sometimes hangs, otherwise it takes tens of minutes). Do you have any tips for improving speed?
>> I have reduced the stored data as much as possible (without using compression) and use a FieldSelector when querying
>> but neither seem to help..
>>
>> Is this the right approach? If I query them one by one I response time is reasonable (a few seconds each), but I was
>> kind of hoping to treat them as one large index if possible.
a few seconds? what are you doing that a search takes a few seconds.
simon
>>
>> Any thoughts welcomed.
>>
>> Thanks,
>>
>> - Chris
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Large indexes
Posted by Ian Lea <ia...@gmail.com>.
There are lots of general tips at
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed.
What version of lucene? Recent releases should be faster. Have you
tried with one big index? If everything is running on the same server
that may well be faster.
Even on single indexes, response of a few seconds sounds high. Are the
searches complex? Are you sorting? Reading lots of hits for each
search?
--
Ian.
On Fri, Jul 8, 2011 at 3:39 PM, Chris Bamford
<ch...@talktalk.net> wrote:
>
> Hi
>
>
> I was wondering how to improve search performance over a set of indexes like this:
>
> 27G K1-1/index
> 19G K1-2/index
> 24G K1-3/index
> 15G K1-4/index
> 19G K1-5/index
> 31G K2-1/index
> 16G K2-2/index
> 8.1G K2-3/index
> 12G K2-4/index
> 15G K2-5/index
>
> In total it is ~192Gb. All have the same schema, with data partitioned by my application into separate
> indexes (rather than just 1 big one).
> I have tried searching with both a MultiSearcher and a ParallelMultiSearcher (with lots of memory), but it
> is way too slow (sometimes hangs, otherwise it takes tens of minutes). Do you have any tips for improving speed?
> I have reduced the stored data as much as possible (without using compression) and use a FieldSelector when querying
> but neither seem to help..
>
> Is this the right approach? If I query them one by one I response time is reasonable (a few seconds each), but I was
> kind of hoping to treat them as one large index if possible.
>
> Any thoughts welcomed.
>
> Thanks,
>
> - Chris
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Large indexes
Posted by Erick Erickson <er...@gmail.com>.
Simply breaking up your index into separate pieces on the same machine
buys you nothing, in fact it costs you considerably. Have you put
a profiler on the system to see what's happening? I expect you're swapping
all over the place and are memory-constrained.
Have you considered sharding your index across multiple machines? That
is often an approach to use assuming that logically this is a single index...
Best
Erick
On Fri, Jul 8, 2011 at 10:39 AM, Chris Bamford
<ch...@talktalk.net> wrote:
>
> Hi
>
>
> I was wondering how to improve search performance over a set of indexes like this:
>
> 27G K1-1/index
> 19G K1-2/index
> 24G K1-3/index
> 15G K1-4/index
> 19G K1-5/index
> 31G K2-1/index
> 16G K2-2/index
> 8.1G K2-3/index
> 12G K2-4/index
> 15G K2-5/index
>
> In total it is ~192Gb. All have the same schema, with data partitioned by my application into separate
> indexes (rather than just 1 big one).
> I have tried searching with both a MultiSearcher and a ParallelMultiSearcher (with lots of memory), but it
> is way too slow (sometimes hangs, otherwise it takes tens of minutes). Do you have any tips for improving speed?
> I have reduced the stored data as much as possible (without using compression) and use a FieldSelector when querying
> but neither seem to help..
>
> Is this the right approach? If I query them one by one I response time is reasonable (a few seconds each), but I was
> kind of hoping to treat them as one large index if possible.
>
> Any thoughts welcomed.
>
> Thanks,
>
> - Chris
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org