Re: Deferring merging of index segments

[Vitaly Funstein <vf...@gmail.com>]

Any takers on this one or is my inquiry a bit too broad? I can post my
test code if that helps...

On Tue, May 29, 2012 at 6:42 PM, Vitaly Funstein <vf...@gmail.com> wrote:
> Hello,
>
> I am trying to optimize the process of "warming up" an index prior to
> using the search subsystem, i.e. it is guaranteed that no other writes
> or searches can take place in parallel with with the warmup. To that
> end, I have been toying with the idea of turning off segment merging
> altogether until after all the data has been written and committed. I
> am currently using Lucene 3.0.3 and migration to a later version is
> not an option in the short term. So, the way I'm going about turning
> merging off is as follows:
>
> 1. Before warmup, call:
>
> IndexWriter.setMaxMergeDocs(0);
> IndexWriter.getLogMergePolicy().setMaxMergeMB(0);
>
> 2. After the warmup task completes, revert the above parameters to
> their defaults, then call:
>
> IndexWriter.maybeMerge();
> IndexWriter.waitForMerges();
>
>
> Now, I compared my results when deferring segment merges using the
> above method, with a test run letting Lucene do the merging on the
> fly. Curiously, the resulting size of indexes on disk is about 64%
> greater in the former case, although the total time to complete the
> warmup is almost the same.
>
> So I have a few of questions:
> - is the approach for deferring segment merging flawed in some way?
> - what could possibly account for the huge difference in file sizes?
> - what else could I possibly try to further speed up index writing
> during system's "off hours"?
>
> Thanks,
> -V

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org