TieredMergePolicy

[Kireet Reddy <ki...@feedly.com>]

I have a question about the maxMergeAtOnce parameter. We are using elasticsearch and one of our nodes seems to have very high merge activity, However it seems to be high CPU activity and not I/O constrainted. I have enabled the IndexWriter info stream logs, and often times it seems to do merges of quite small segments (100KB) that are much below the floor size (2MB). I suspect this is due to frequent refreshes and/or using lots of threads concurrently to do indexing. 

My supposition is that this is leading to the merge policy doing lots of merges of very small segments into another small segment which will again require a merge to even reach the floor size. My index has 64 segments and 25 are below the floor size. I am wondering if there should be an exception for the maxMergesAtOnce parameter for the first level so that many small segments could be merged at once in this case?

I am considering changing the other parameters (wider tiers, lower floor size, more concurrent merges allowed) but these all seem to have side effects I may not necessarily want. Is there a good solution here?