You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jerven Bolleman <je...@sib.swiss> on 2018/11/02 10:12:52 UTC
Static index, fastest way to do forceMerge
Dear Lucene Devs and Users,
First of all thank you for this wonderful library and API.
forceMerges are normally not recommended but we fall into one of the few
usecases where it makes sense.
In our use case we have a large index (3 actually) and we don't update
them ever after indexing. i.e. we index all the documents and then never
ever add another document to the index, nor are any deleted.
It has proven beneficial for search performance to always foreMerge down
to one segment. However, this takes significant time. Are there any
suggestions on what kind of merge scheduler/policy settings will utilize
the most of the available IO, CPU and RAM capacity? Currently we end up
being single thread bound, leaving lots of potential cpu and bandwidth
not used during the merge.
e.g. we are looking for a MergeEvertyThing use all hardware policy and
scheduler.
We are currently on lucene 7.4 but nothing is stopping us from upgrading.
Regards,
Jerven
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Dawid Weiss <da...@gmail.com>.
That's fine -- distraction is pretty much what defines my work... ;)
D.
On Tue, Dec 18, 2018 at 11:19 AM Jerven Tjalling Bolleman
<je...@sib.swiss> wrote:
>
> Hi Dawid,
>
> Thanks for looking into this! I have been distracted with other work and
> did not get the time I expected to work on it.
>
> Regards,
> Jerven
> On 11/30/18 12:01 PM, Dawid Weiss wrote:
> > Just FYI: I implemented a quick and dirty PoC to see what it'd work
> > like. Not much of a difference on my machine (since postings merging
> > dominates everything else). Interesting problem how to split it up to
> > saturate all of available resources though (CPU and I/O).
> >
> > https://issues.apache.org/jira/browse/LUCENE-8580
> >
> > Dawid
> >
> > On Fri, Nov 2, 2018 at 10:17 PM Dawid Weiss <da...@gmail.com> wrote:
> >>
> >> Thanks for chipping in, Toke. A ~1TB index is impressive.
> >>
> >> Back of the envelope says reading & writing 900GB in 8 hours is
> >> 2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our
> >> SSD machine, but even with SATA II this is only ~1/5th of the possible
> >> fairly sequential IO throughput. So for us at least, NVMe drives are
> >> not needed to have single-threaded CPU as bottleneck.
> >>
> >> The mileage will vary depending on the CPU -- if it can merge the data
> >> from multiple files at ones fast enough then it may theoretically
> >> saturate the bandwidth... but I agree we also seem to be CPU bound on
> >> these N-to-1 merges, a regular SSD is enough.
> >>
> >>> And +1 to the issue BTW.
> >>
> >> I agree. Fine-grained granularity here would be a win even in the
> >> regular "merge is a low-priority citizen" case. At least that's what I
> >> tend to think. And if there are spare CPUs, the gain would be
> >> terrific.
> >>
> >> Dawid
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Jerven Tjalling Bolleman <je...@sib.swiss>.
Hi Dawid,
Thanks for looking into this! I have been distracted with other work and
did not get the time I expected to work on it.
Regards,
Jerven
On 11/30/18 12:01 PM, Dawid Weiss wrote:
> Just FYI: I implemented a quick and dirty PoC to see what it'd work
> like. Not much of a difference on my machine (since postings merging
> dominates everything else). Interesting problem how to split it up to
> saturate all of available resources though (CPU and I/O).
>
> https://issues.apache.org/jira/browse/LUCENE-8580
>
> Dawid
>
> On Fri, Nov 2, 2018 at 10:17 PM Dawid Weiss <da...@gmail.com> wrote:
>>
>> Thanks for chipping in, Toke. A ~1TB index is impressive.
>>
>> Back of the envelope says reading & writing 900GB in 8 hours is
>> 2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our
>> SSD machine, but even with SATA II this is only ~1/5th of the possible
>> fairly sequential IO throughput. So for us at least, NVMe drives are
>> not needed to have single-threaded CPU as bottleneck.
>>
>> The mileage will vary depending on the CPU -- if it can merge the data
>> from multiple files at ones fast enough then it may theoretically
>> saturate the bandwidth... but I agree we also seem to be CPU bound on
>> these N-to-1 merges, a regular SSD is enough.
>>
>>> And +1 to the issue BTW.
>>
>> I agree. Fine-grained granularity here would be a win even in the
>> regular "merge is a low-priority citizen" case. At least that's what I
>> tend to think. And if there are spare CPUs, the gain would be
>> terrific.
>>
>> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Dawid Weiss <da...@gmail.com>.
Just FYI: I implemented a quick and dirty PoC to see what it'd work
like. Not much of a difference on my machine (since postings merging
dominates everything else). Interesting problem how to split it up to
saturate all of available resources though (CPU and I/O).
https://issues.apache.org/jira/browse/LUCENE-8580
Dawid
On Fri, Nov 2, 2018 at 10:17 PM Dawid Weiss <da...@gmail.com> wrote:
>
> Thanks for chipping in, Toke. A ~1TB index is impressive.
>
> Back of the envelope says reading & writing 900GB in 8 hours is
> 2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our
> SSD machine, but even with SATA II this is only ~1/5th of the possible
> fairly sequential IO throughput. So for us at least, NVMe drives are
> not needed to have single-threaded CPU as bottleneck.
>
> The mileage will vary depending on the CPU -- if it can merge the data
> from multiple files at ones fast enough then it may theoretically
> saturate the bandwidth... but I agree we also seem to be CPU bound on
> these N-to-1 merges, a regular SSD is enough.
>
> > And +1 to the issue BTW.
>
> I agree. Fine-grained granularity here would be a win even in the
> regular "merge is a low-priority citizen" case. At least that's what I
> tend to think. And if there are spare CPUs, the gain would be
> terrific.
>
> Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Erick Erickson <er...@gmail.com>.
Do you really need exactly one segment? Or would, say, 5 be good enough?
You see where this is going, set maxsegments to 5 and maybe be able to get
some parallelization...
On Fri, Nov 2, 2018, 14:17 Dawid Weiss <dawid.weiss@gmail.com wrote:
> Thanks for chipping in, Toke. A ~1TB index is impressive.
>
> Back of the envelope says reading & writing 900GB in 8 hours is
> 2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our
> SSD machine, but even with SATA II this is only ~1/5th of the possible
> fairly sequential IO throughput. So for us at least, NVMe drives are
> not needed to have single-threaded CPU as bottleneck.
>
> The mileage will vary depending on the CPU -- if it can merge the data
> from multiple files at ones fast enough then it may theoretically
> saturate the bandwidth... but I agree we also seem to be CPU bound on
> these N-to-1 merges, a regular SSD is enough.
>
> > And +1 to the issue BTW.
>
> I agree. Fine-grained granularity here would be a win even in the
> regular "merge is a low-priority citizen" case. At least that's what I
> tend to think. And if there are spare CPUs, the gain would be
> terrific.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Static index, fastest way to do forceMerge
Posted by Dawid Weiss <da...@gmail.com>.
Thanks for chipping in, Toke. A ~1TB index is impressive.
Back of the envelope says reading & writing 900GB in 8 hours is
2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our
SSD machine, but even with SATA II this is only ~1/5th of the possible
fairly sequential IO throughput. So for us at least, NVMe drives are
not needed to have single-threaded CPU as bottleneck.
The mileage will vary depending on the CPU -- if it can merge the data
from multiple files at ones fast enough then it may theoretically
saturate the bandwidth... but I agree we also seem to be CPU bound on
these N-to-1 merges, a regular SSD is enough.
> And +1 to the issue BTW.
I agree. Fine-grained granularity here would be a win even in the
regular "merge is a low-priority citizen" case. At least that's what I
tend to think. And if there are spare CPUs, the gain would be
terrific.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Toke Eskildsen <to...@kb.dk>.
Dawid Weiss <da...@gmail.com> wrote:
> Merging segments as large as this one requires not just CPU, but also
> serious I/O throughput efficiency. I assume you have fast NVMe drives
> on that machine, otherwise it'll be slow, no matter what. It's just a
> lot of bytes going back and forth.
We have quite a lot of experience with creating fully merged 900GB indexes. On our plain-SSD (Samsung 840) equipped machine this took ~8 hours with a single CPU-core at 100%. On our 7200 RPM spinning drive machines (same CPU class) it took nearly twice as long. Back of the envelope says reading & writing 900GB in 8 hours is 2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our SSD machine, but even with SATA II this is only ~1/5th of the possible fairly sequential IO throughput. So for us at least, NVMe drives are not needed to have single-threaded CPU as bottleneck.
And +1 to the issue BTW. It does not matter too much for us now, as we have shifted to a setup where we build more indexes in parallel, but 3 years ago our process was sequential so the 8 hour delay before building the next part was a bit of an annoyance.
- Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Jerven Tjalling Bolleman <Je...@sib.swiss>.
On 2018-11-02 20:52, Dawid Weiss wrote:
>> int processors = Runtime.getRuntime().availableProcessors();
>> int ConcurrentMergeScheduler cms = new ConcurrentMergeScheduler();
>> cms.setMaxMergesAndThreads(processors,processors);
>
> See the number of threads in the CMS only matters if you have
> concurrent merges of independent segments. What you're doing
> effectively forces an eventual X -> 1 merge, which is done by a single
> thread (regardless of the max processors above).
>
>> 38G _583u.fdt
>> 25M _583u.fdx
>> 13K _583u.fnm
>> 47G _583u_Lucene50_0.doc
>> 54G _583u_Lucene50_0.pos
>> 30G _583u_Lucene50_0.tim
>> 413M _583u_Lucene50_0.tip
>> 2.1G _583u_Lucene70_0.dvd
>> 213 _583u_Lucene70_0.dvm
>
> Merging segments as large as this one requires not just CPU, but also
> serious I/O throughput efficiency. I assume you have fast NVMe drives
> on that machine, otherwise it'll be slow, no matter what. It's just a
> lot of bytes going back and forth.
Yup, it's now cloud so optimizing for quick index and then merge to one
has become financially interesting. Now it's too much cpu and ram being
idle. Nor even maxing out the disk io (about 25% of max rate).
>
>> If we did such a max resource merge code would there be interest to
>> have this merged?
>
> I think so. Try to experiment locally first though and see if what you
> can find out. Hacking that code I pointed at shouldn't be too
> difficult. see what happens.
Yeah, before I left I started with an experiment to have one running
without the
merge scheduler being involved at all.
Will try a few more experiments next week.
>
>> Or should we maybe do something like this assuming 64 cpus
>>
>> writer.forceMerge(64, true);
>> writer.forceMerge(32, true);
>> writer.forceMerge(16, true);
>> writer.forceMerge(8, true);
>> writer.forceMerge(4, true);
>> writer.forceMerge(2, true);
>> writer.forceMerge(1, true);
>
> No, this doesn't make much sense. If your goal is 1 segment then you
> want to read from as many of them as once as possible and merge into a
> single segment. Doing what you did above would only bump I/O traffic a
> lot.
Thanks, I always thought so but wasn't sure anymore.
Have a nice weekend everyone!
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
--
Jerven Tjalling Bolleman
SIB | Swiss Institute of Bioinformatics
CMU - 1, rue Michel Servet - 1211 Geneva 4
t: +41 22 379 58 85 - f: +41 22 379 58 58
Jerven.Bolleman@sib.swiss - http://www.sib.swiss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Dawid Weiss <da...@gmail.com>.
> int processors = Runtime.getRuntime().availableProcessors();
> int ConcurrentMergeScheduler cms = new ConcurrentMergeScheduler();
> cms.setMaxMergesAndThreads(processors,processors);
See the number of threads in the CMS only matters if you have
concurrent merges of independent segments. What you're doing
effectively forces an eventual X -> 1 merge, which is done by a single
thread (regardless of the max processors above).
> 38G _583u.fdt
> 25M _583u.fdx
> 13K _583u.fnm
> 47G _583u_Lucene50_0.doc
> 54G _583u_Lucene50_0.pos
> 30G _583u_Lucene50_0.tim
> 413M _583u_Lucene50_0.tip
> 2.1G _583u_Lucene70_0.dvd
> 213 _583u_Lucene70_0.dvm
Merging segments as large as this one requires not just CPU, but also
serious I/O throughput efficiency. I assume you have fast NVMe drives
on that machine, otherwise it'll be slow, no matter what. It's just a
lot of bytes going back and forth.
> If we did such a max resource merge code would there be interest to have this merged?
I think so. Try to experiment locally first though and see if what you
can find out. Hacking that code I pointed at shouldn't be too
difficult. see what happens.
> Or should we maybe do something like this assuming 64 cpus
>
> writer.forceMerge(64, true);
> writer.forceMerge(32, true);
> writer.forceMerge(16, true);
> writer.forceMerge(8, true);
> writer.forceMerge(4, true);
> writer.forceMerge(2, true);
> writer.forceMerge(1, true);
No, this doesn't make much sense. If your goal is 1 segment then you
want to read from as many of them as once as possible and merge into a
single segment. Doing what you did above would only bump I/O traffic a
lot.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Jerven Tjalling Bolleman <Je...@sib.swiss>.
Hi Dawid, Erick,
Thanks for the reply. We are using pure lucene and currently this is
what I am doing
int processors = Runtime.getRuntime().availableProcessors();
int ConcurrentMergeScheduler cms = new ConcurrentMergeScheduler();
cms.setMaxMergesAndThreads(processors,processors);
cms.disableAutoIOThrottle();
config.setMergeScheduler(cms);
TieredMergePolicy tms = new TieredMergePolicy();
int merges = Math.max(2, processors);
tms.setMaxMergeAtOnce(merges);
tms.setMaxMergeAtOnceExplicit(merges);
tms.setSegmentsPerTier(merges * 2);
config.setMergePolicy(new TieredMergePolicy());
IndexWriter writer = new IndexWriter(this.dir, config);
writer.forceMerge(1, true);
Regarding, seeing a performance decrease without merging. Yes
significant.
That the one time we tried we broke every SLA and haven't tried in long
time.
Of course now our largest index is more than 200 million documents so
perhaps we
should retry this.
38G _583u.fdt
25M _583u.fdx
13K _583u.fnm
47G _583u_Lucene50_0.doc
54G _583u_Lucene50_0.pos
30G _583u_Lucene50_0.tim
413M _583u_Lucene50_0.tip
2.1G _583u_Lucene70_0.dvd
213 _583u_Lucene70_0.dvm
If we did such a max resource merge code would there be interest to have
this merged?
Or should we maybe do something like this assuming 64 cpus
writer.forceMerge(64, true);
writer.forceMerge(32, true);
writer.forceMerge(16, true);
writer.forceMerge(8, true);
writer.forceMerge(4, true);
writer.forceMerge(2, true);
writer.forceMerge(1, true);
Regards,
Jerven
On 2018-11-02 19:19, Dawid Weiss wrote:
> We are faced with a similar situation. Yes, the merge process can take
> a long time and is mostly single-threaded (if you're merging from N
> segments into a single segment, only one thread does the job). As
> Erick pointed out, the merge process takes a backseat compared to
> indexing and searches (in most cases), so it's not a priority, but
> it's definitely something people like you (and me) could utilize, if
> given the opportunity.
>
> I actually don't see any reasons why merging of individual parts of a
> segment can't be done in parallel (this would be a start; later on a
> splittable strategy of merging single things could make use of things
> like the fork-join executor). I'd love to work on this at some point,
> but I honestly don't see any time soon this could be happening. If you
> have a spare cycle, take a look at how index writer merges a single
> segment; there are quite trivial ways this could be split into
> parallel subtasks and executed with, for example, the system fork-join
> executor (even without forkable tasks).
>
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L2999-L3007
>
> As a side note, you may want to make absolutely sure your merge
> scheduler (if it's the CMS) is not using any I/O throttling -- this is
> theoretically self-adjustable, but in practice if you only care about
> the wall-clock end of a single merge, it's better to turn it off.
>
> Dawid
> On Fri, Nov 2, 2018 at 5:57 PM Erick Erickson <er...@gmail.com>
> wrote:
>>
>> The merge process is rather tricky, and there's nothing that I know of
>> that will use all resources available. In fact the merge code is
>> written to _not_ use up all the possible resources on the theory that
>> there should be some left over to handle queries etc.
>>
>> Yeah, the situation you describe is indeed one of the few where
>> merging down to 1 segment makes sense. Out of curiosity, what kind of
>> performance gains to you see?
>>
>> This applies to the default TieredMergePolicy (TMP):
>>
>> 1> there is a limit to the number of segments that can be merged at
>> once, so sometimes it can take more than one pass. If you have more
>> than 30 segments, it'll be multi-pass. You can try (and I haven't done
>> this personally) setting maxMergeAtOnceExplicit in your solrconfig.xml
>> to see if it helps. That only takes effect when you forceMerge.
>> There's a trick bit of reflection that handles this, see the very end
>> of TieredMergePolicy.java for the parameters you can set.
>>
>> 2> As of Solr 7.5 (see LUCENE-7976) the default behavior has changed
>> from automatically merging down to 1 segment to respecting
>> "maxMergedSegmentMB" (default 5G). You will have to explicitly pass
>> maxSegments=1 to get the old behavior.
>>
>> Best,
>> Erick
>> On Fri, Nov 2, 2018 at 3:13 AM Jerven Bolleman
>> <je...@sib.swiss> wrote:
>> >
>> > Dear Lucene Devs and Users,
>> >
>> > First of all thank you for this wonderful library and API.
>> >
>> > forceMerges are normally not recommended but we fall into one of the few
>> > usecases where it makes sense.
>> >
>> > In our use case we have a large index (3 actually) and we don't update
>> > them ever after indexing. i.e. we index all the documents and then never
>> > ever add another document to the index, nor are any deleted.
>> >
>> > It has proven beneficial for search performance to always foreMerge down
>> > to one segment. However, this takes significant time. Are there any
>> > suggestions on what kind of merge scheduler/policy settings will utilize
>> > the most of the available IO, CPU and RAM capacity? Currently we end up
>> > being single thread bound, leaving lots of potential cpu and bandwidth
>> > not used during the merge.
>> >
>> > e.g. we are looking for a MergeEvertyThing use all hardware policy and
>> > scheduler.
>> >
>> > We are currently on lucene 7.4 but nothing is stopping us from upgrading.
>> >
>> > Regards,
>> > Jerven
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
--
Jerven Tjalling Bolleman
SIB | Swiss Institute of Bioinformatics
CMU - 1, rue Michel Servet - 1211 Geneva 4
t: +41 22 379 58 85 - f: +41 22 379 58 58
Jerven.Bolleman@sib.swiss - http://www.sib.swiss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Dawid Weiss <da...@gmail.com>.
We are faced with a similar situation. Yes, the merge process can take
a long time and is mostly single-threaded (if you're merging from N
segments into a single segment, only one thread does the job). As
Erick pointed out, the merge process takes a backseat compared to
indexing and searches (in most cases), so it's not a priority, but
it's definitely something people like you (and me) could utilize, if
given the opportunity.
I actually don't see any reasons why merging of individual parts of a
segment can't be done in parallel (this would be a start; later on a
splittable strategy of merging single things could make use of things
like the fork-join executor). I'd love to work on this at some point,
but I honestly don't see any time soon this could be happening. If you
have a spare cycle, take a look at how index writer merges a single
segment; there are quite trivial ways this could be split into
parallel subtasks and executed with, for example, the system fork-join
executor (even without forkable tasks).
https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L2999-L3007
As a side note, you may want to make absolutely sure your merge
scheduler (if it's the CMS) is not using any I/O throttling -- this is
theoretically self-adjustable, but in practice if you only care about
the wall-clock end of a single merge, it's better to turn it off.
Dawid
On Fri, Nov 2, 2018 at 5:57 PM Erick Erickson <er...@gmail.com> wrote:
>
> The merge process is rather tricky, and there's nothing that I know of
> that will use all resources available. In fact the merge code is
> written to _not_ use up all the possible resources on the theory that
> there should be some left over to handle queries etc.
>
> Yeah, the situation you describe is indeed one of the few where
> merging down to 1 segment makes sense. Out of curiosity, what kind of
> performance gains to you see?
>
> This applies to the default TieredMergePolicy (TMP):
>
> 1> there is a limit to the number of segments that can be merged at
> once, so sometimes it can take more than one pass. If you have more
> than 30 segments, it'll be multi-pass. You can try (and I haven't done
> this personally) setting maxMergeAtOnceExplicit in your solrconfig.xml
> to see if it helps. That only takes effect when you forceMerge.
> There's a trick bit of reflection that handles this, see the very end
> of TieredMergePolicy.java for the parameters you can set.
>
> 2> As of Solr 7.5 (see LUCENE-7976) the default behavior has changed
> from automatically merging down to 1 segment to respecting
> "maxMergedSegmentMB" (default 5G). You will have to explicitly pass
> maxSegments=1 to get the old behavior.
>
> Best,
> Erick
> On Fri, Nov 2, 2018 at 3:13 AM Jerven Bolleman
> <je...@sib.swiss> wrote:
> >
> > Dear Lucene Devs and Users,
> >
> > First of all thank you for this wonderful library and API.
> >
> > forceMerges are normally not recommended but we fall into one of the few
> > usecases where it makes sense.
> >
> > In our use case we have a large index (3 actually) and we don't update
> > them ever after indexing. i.e. we index all the documents and then never
> > ever add another document to the index, nor are any deleted.
> >
> > It has proven beneficial for search performance to always foreMerge down
> > to one segment. However, this takes significant time. Are there any
> > suggestions on what kind of merge scheduler/policy settings will utilize
> > the most of the available IO, CPU and RAM capacity? Currently we end up
> > being single thread bound, leaving lots of potential cpu and bandwidth
> > not used during the merge.
> >
> > e.g. we are looking for a MergeEvertyThing use all hardware policy and
> > scheduler.
> >
> > We are currently on lucene 7.4 but nothing is stopping us from upgrading.
> >
> > Regards,
> > Jerven
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Static index, fastest way to do forceMerge
Posted by Erick Erickson <er...@gmail.com>.
The merge process is rather tricky, and there's nothing that I know of
that will use all resources available. In fact the merge code is
written to _not_ use up all the possible resources on the theory that
there should be some left over to handle queries etc.
Yeah, the situation you describe is indeed one of the few where
merging down to 1 segment makes sense. Out of curiosity, what kind of
performance gains to you see?
This applies to the default TieredMergePolicy (TMP):
1> there is a limit to the number of segments that can be merged at
once, so sometimes it can take more than one pass. If you have more
than 30 segments, it'll be multi-pass. You can try (and I haven't done
this personally) setting maxMergeAtOnceExplicit in your solrconfig.xml
to see if it helps. That only takes effect when you forceMerge.
There's a trick bit of reflection that handles this, see the very end
of TieredMergePolicy.java for the parameters you can set.
2> As of Solr 7.5 (see LUCENE-7976) the default behavior has changed
from automatically merging down to 1 segment to respecting
"maxMergedSegmentMB" (default 5G). You will have to explicitly pass
maxSegments=1 to get the old behavior.
Best,
Erick
On Fri, Nov 2, 2018 at 3:13 AM Jerven Bolleman
<je...@sib.swiss> wrote:
>
> Dear Lucene Devs and Users,
>
> First of all thank you for this wonderful library and API.
>
> forceMerges are normally not recommended but we fall into one of the few
> usecases where it makes sense.
>
> In our use case we have a large index (3 actually) and we don't update
> them ever after indexing. i.e. we index all the documents and then never
> ever add another document to the index, nor are any deleted.
>
> It has proven beneficial for search performance to always foreMerge down
> to one segment. However, this takes significant time. Are there any
> suggestions on what kind of merge scheduler/policy settings will utilize
> the most of the available IO, CPU and RAM capacity? Currently we end up
> being single thread bound, leaving lots of potential cpu and bandwidth
> not used during the merge.
>
> e.g. we are looking for a MergeEvertyThing use all hardware policy and
> scheduler.
>
> We are currently on lucene 7.4 but nothing is stopping us from upgrading.
>
> Regards,
> Jerven
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org