You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Mark Miller <ma...@gmail.com> on 2009/08/03 21:59:36 UTC
Re: ConcurrentMergeScheduler and MergePolicy question
Michael McCandless wrote:
> On the impact of search performance for large vs small mergeFactors, I
> think the jury is still out. People should keep testing that (and
> report back!). Certainly, for the fastest reopen time you never want
> any merging to be done :)
>
Here is the original exchange I referenced:
>>On Fri, Apr 10, 2009 at 3:06 PM, Mark Miller <ma...@gmail.com>
wrote:
>> 24 segments is bound to be quite a bit slower than an optimized
index for most things
>I'd be curious just how true this really is (in general)... my guess
>is the "long tail of tiny segments" gets into the OS's IO cache (as
>long as the system stays hot) and doesn't actually hurt things much.
>
>Has anyone tested this (performance of unoptimized vs optimized
>indexes, in general) recently? To be a fair comparison, there should
>be no deletions in the index.
>
>Mike
After reading that, I played with some sorting code I had and did a
quick cheesy test or two - one segment vs a 10 or 20. In that horrible
test (based on the stress sort code), I don't remember seeing much of a
difference. No sorting. Very, very unscientific, quick and dirty.
This time I loaded up 1.3 million wikipedia articles, gave the test
768MB of RAM, warmed the Searcher with lots of searching before each
measurement, and compared 1 segment vs 5. The optimized index was 15-20%
faster with the queries I was using (approx 100 queries targeted at
wikipedia). Its an odd test system - Ubuntu, Quad core laptop with slow
laptop drives and 4 gig of RAM. Still not very scientific, but better
than before.
Here is the benchmark I was using in various forms:
{ "Rounds"
ResetSystemErase
{ "Populate"
-CreateIndex
{ "MAddDocs" AddDoc > : 15000
-CloseIndex
}
{ "test"
OpenReader
{ "WarmRdrDocs" Warm > : 50
{ "WarmRdr" Search > : 5000
{ "SearchSameRdr" Search > : 50000
CloseReader
OpenIndex
PrintSegmentCount
Optimize
CloseIndex
NewRound
} : 2
}
RepSumByName
RepSumByPrefRound SearchSameRdr
I also did a quick profile for a 15k index, 1seg vs 10 segs. I profiled
each for approx 11 million calls of readVint. The hotspot results are below.
http://myhardshadow.com/images/1seg.png
http://myhardshadow.com/images/10seg.png
Just a quick start at looking into this from over the weekend.
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: ConcurrentMergeScheduler and MergePolicy question
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Mon, Aug 3, 2009 at 3:59 PM, Mark Miller<ma...@gmail.com> wrote:
> This time I loaded up 1.3 million wikipedia articles, gave the test 768MB of
> RAM, warmed the Searcher with lots of searching before each measurement, and
> compared 1 segment vs 5. The optimized index was 15-20% faster with the
> queries I was using (approx 100 queries targeted at wikipedia). Its an odd
> test system - Ubuntu, Quad core laptop with slow laptop drives and 4 gig of
> RAM. Still not very scientific, but better than before.
That's a good result!
It sounds like optimizing is worthwhile.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: ConcurrentMergeScheduler and MergePolicy question
Posted by Earwin Burrfoot <ea...@gmail.com>.
On Sun, Aug 9, 2009 at 08:38, Jason
Rutherglen<ja...@gmail.com> wrote:
>> You don't have to copy. You can have one machine optimize your indexes
>> whilst other serves user requests, then they switch roles, rinse,
>> repeat. This approach also works with sharding, and more than 2-way
>> mirroring.
>
> What does the un-optimized server do after the other server is
> optimized? The search requests go to the newly optimized server,
> however if we're mirroring, the 2nd server now needs the
> optimized index as well?
>
The second server now stops servicing requests and starts optimizing.
You can also keep them running together for some time, depending on
how you're serious about always running on optimized index.
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: ConcurrentMergeScheduler and MergePolicy question
Posted by Jason Rutherglen <ja...@gmail.com>.
> You don't have to copy. You can have one machine optimize your indexes
> whilst other serves user requests, then they switch roles, rinse,
> repeat. This approach also works with sharding, and more than 2-way
> mirroring.
What does the un-optimized server do after the other server is
optimized? The search requests go to the newly optimized server,
however if we're mirroring, the 2nd server now needs the
optimized index as well?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: ConcurrentMergeScheduler and MergePolicy question
Posted by Earwin Burrfoot <ea...@gmail.com>.
> Perhaps the ideal search system architecture that requires
> optimizing is to dedicate a server to it, copy the index to the
> optimize server, do the optimize, copy the index off (to a
> search server) and start again for the next optimize task.
>
> I wonder how/if this would work with Hadoop/HDFS as copying
> 100GB around would presumably tie up the network? Also, I've
> found rsyncing large optimized indexes to be time consuming and
> wreaks havoc on the searcher server's IO subsystem. Usually this
> is unacceptable for the user as the queries will suddenly
> degrade.
You don't have to copy. You can have one machine optimize your indexes
whilst other serves user requests, then they switch roles, rinse,
repeat. This approach also works with sharding, and more than 2-way
mirroring.
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: ConcurrentMergeScheduler and MergePolicy question
Posted by Jason Rutherglen <ja...@gmail.com>.
Mark,
On a system where the size of the index is 10 times the amount
of RAM, lets say 10GB RAM and 100GB index, is it ok for optimize
to take 30-60 minutes? Maybe the performance trade off (10-20%
less search performance) is worth it? Otherwise the optimize
literally takes down the machine.
Perhaps the ideal search system architecture that requires
optimizing is to dedicate a server to it, copy the index to the
optimize server, do the optimize, copy the index off (to a
search server) and start again for the next optimize task.
I wonder how/if this would work with Hadoop/HDFS as copying
100GB around would presumably tie up the network? Also, I've
found rsyncing large optimized indexes to be time consuming and
wreaks havoc on the searcher server's IO subsystem. Usually this
is unacceptable for the user as the queries will suddenly
degrade.
-J
On Mon, Aug 3, 2009 at 12:59 PM, Mark Miller<ma...@gmail.com> wrote:
> Michael McCandless wrote:
>
> After reading that, I played with some sorting code I had and did a quick
> cheesy test or two - one segment vs a 10 or 20. In that horrible test (based
> on the stress sort code), I don't remember seeing much of a difference. No
> sorting. Very, very unscientific, quick and dirty.
>
> This time I loaded up 1.3 million wikipedia articles, gave the test 768MB of
> RAM, warmed the Searcher with lots of searching before each measurement, and
> compared 1 segment vs 5. The optimized index was 15-20% faster with the
> queries I was using (approx 100 queries targeted at wikipedia). Its an odd
> test system - Ubuntu, Quad core laptop with slow laptop drives and 4 gig of
> RAM. Still not very scientific, but better than before.
>
>
> Here is the benchmark I was using in various forms:
>
> { "Rounds"
>
> ResetSystemErase
>
> { "Populate"
> -CreateIndex
> { "MAddDocs" AddDoc > : 15000
> -CloseIndex
> }
> { "test"
> OpenReader { "WarmRdrDocs" Warm > : 50
> { "WarmRdr" Search > : 5000
> { "SearchSameRdr" Search > : 50000
> CloseReader
> OpenIndex
> PrintSegmentCount
> Optimize CloseIndex NewRound
> } : 2
> }
>
> RepSumByName
> RepSumByPrefRound SearchSameRdr
>
>
> I also did a quick profile for a 15k index, 1seg vs 10 segs. I profiled each
> for approx 11 million calls of readVint. The hotspot results are below.
>
> http://myhardshadow.com/images/1seg.png
> http://myhardshadow.com/images/10seg.png
>
>
> Just a quick start at looking into this from over the weekend.
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org