Impact of max merged segment setting

[Vitaly Funstein <vf...@gmail.com>]

Hello,

I am currently experimenting with tuning of max merged segment MB
parameter on TieredMergePolicy in Lucene 3.5, and seeing significant
gains in index writing speed from values dramatically lower than the
default (5 Gb). For instance, when setting it to 5 or 10 MB, I can see
my writing tests run anywhere from 25% to 40% faster, depending on
physical storage type (rotational vs Fusion IO), data size and type,
and possibly thread access pattern.

In practice, our usage of Lucene can result in merged segment files
whose size sometimes exceeds even the default merge limit, so forcing
partition of these into much smaller chunks raises a concern over how
much of a decrease in search performance I should expect for each
improvement in indexing speed. Is the dependency linear, are there any
specific guidelines to follow here? Are there other tunable settings
related to segment merging to try out that might have a similar
impact?

Thank you,
-V

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org