You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Tom Burton-West <tb...@umich.edu> on 2015/02/25 18:11:12 UTC
Fwd: Optimize maxSegments="2" not working right with Solr 4.10.2
No replies on the Solr users list, so I thought I would repost to dev.
We are continuing to see inconsistent behavior with <optimize
maxSegments="2"/>
Out of 12 shards 1-3 of them end up with more than 2 segments at the finish
of the optimize command. (and we see no errors in the logs)
The pattern seems the same in that after almost all of the segments are
merged, one or two new segments are created when a startFullFlush happens
after the big merge.
Any suggestions on how to troubleshoot this would be appreciated.
Tom
---------- Forwarded message ----------
From: Tom Burton-West <tb...@umich.edu>
Date: Mon, Feb 23, 2015 at 12:41 PM
Subject: Optimize maxSegments="2" not working right with Solr 4.10.2
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Cc: Phillip Farber <pf...@umich.edu>, Sebastien Korner <sk...@umich.edu>
Hello,
We normally run an optimize with maxSegments="2" after our daily indexing.
This has worked without problem on Solr 3.6. We recently moved to Solr
4.10.2 and on several shards the optimize completed with no errors in the
logs, but left more than 2 segments.
We send this xml to Solr
<optimize maxSegments="2"/>
I've attached a copy of the indexwriter log for one of the segments where
there were 4 segments rather than the requested number (i.e. there should
have been only 2 segments) at the end of the optimize. It looks like a
merge was done down to two segments and then somehow another process
flushed some postings to disk creating two more segments. Then there are
messages about 2 of the remaining 4 segments being too big. (See below)
What we expected is that the remainng 2 small segments (about 40MB) would
get merged with the smaller of the two large segments, i.e. with the 56GB
segment, since we gave the argument maxSegments=2. This didn't happen.
Any suggestions about how to troubleshoot this issue would be appreciated.
Tom
-------
Excerpt from indexwriter log:
TMP][http-8091-Processor5]: findForcedMerges maxSegmentCount=2 ...
...
[IW][Lucene Merge Thread #0]: merge time 3842310 msec for 65236 docs
...
[TMP][http-8091-Processor5]: findMerges: 4 segments
[TMP][http-8091-Processor5]: seg=_1fzb(4.10.2):C1081559/24089:delGen=9
size=672402.066 MB [skip: too large]
[TMP][http-8091-Processor5]: seg=_1gj2(4.10.2):C65236/2:delGen=1
size=56179.245 MB [skip: too large]
[TMP][http-8091-Processor5]: seg=_1gj0(4.10.2):C16 size=44.280 MB
[TMP][http-8091-Processor5]: seg=_1gj1(4.10.2):C8 size=40.442 MB
[TMP][http-8091-Processor5]: allowedSegmentCount=3 vs count=4 (eligible
count=2) tooBigCount=2
Re: Optimize maxSegments="2" not working right with Solr 4.10.2
Posted by Tom Burton-West <tb...@umich.edu>.
Hello all,
We are continuing to see inconsistent behavior with <optimize
maxSegments="2"/>
Out of 12 shards one or two of them end up with more than 2 segments at
the finish of the optimize command. (and we see no errors in the logs) So
far we have found no consistent pattern in which of the shards end up with
more than two segments.
I've opened a ticket: Solr-7175 and attached some sample indexwriter logs
and our solrconfig.xml file.
Any suggestions on how to troubleshoot this would be appreciated.
Tom
Re: Optimize maxSegments="2" not working right with Solr 4.10.2
Posted by Tom Burton-West <tb...@umich.edu>.
Hi Toke,
We are using TieredMergePolicy and have had no problems with Solr 3.6
merging to 60GB or larger. With Solr 4.10.2 our second shards are on the
order of 60-100GB so a 2GB limit is unlikely to be the cause of the problem
(unless I need to read the code to see what the context of the 2GB limit is
:)
As far as 2 vs 10 segments, it probably doesn't make too much difference,
but we have been optimizing down to two for several years now. (On the
other hand I assume that more segments means more disk seeks, but given our
huge index extra seeks are probably dwarfed by the disk transfer time to
pull data from the 600GB segment. At this point we are just trying to
understand why Solr isn't consistently doing what we tell it. i.e.
optimize down to two shards.
Tom
On Wed, Feb 25, 2015 at 12:40 PM, Toke Eskildsen <te...@statsbiblioteket.dk>
wrote:
> I'm guessing that you are using LogByteSizeMergePolicy? As far as I can
> see from the code for Solr 4.8, normal maxMergeSize defaults to 2GB.
>
> There also seems to be a bug that effectively disables forced merge as the
> default value for maxMergeSizeForForcedMerge (used when forcing a merge) is
> set with
>
> public static final double DEFAULT_MAX_MERGE_MB_FOR_FORCED_MERGE =
> Long.MAX_VALUE;
> maxMergeSizeForForcedMerge = (long)
> (DEFAULT_MAX_MERGE_MB_FOR_FORCED_MERGE*1024*1024);
>
> which overflows to -1048576. You should be able to fix this by setting
> maxMergeSizeForForcedMerge explicitly.
>
> Guessing here: Your forced merge attempt does nothing and the standard
> merge maxMergeSize is too low?
> (I should fine-read the code and make a JIRA if the error is present in
> Solr 5)
>
> BTW: Why 2 segments? Single segment indexes has memory and performance
> benefits, but there is not - to my knowledge - much difference between 2 or
> 4 (or 10) segments.
>
> - Toke Eskildsen
> ________________________________________
> From: Tom Burton-West [tburtonw@umich.edu]
> Sent: 25 February 2015 18:11
> To: dev@lucene.apache.org
> Subject: Fwd: Optimize maxSegments="2" not working right with Solr 4.10.2
>
> No replies on the Solr users list, so I thought I would repost to dev.
>
> We are continuing to see inconsistent behavior with <optimize
> maxSegments="2"/>
> Out of 12 shards 1-3 of them end up with more than 2 segments at the
> finish of the optimize command. (and we see no errors in the logs)
>
> The pattern seems the same in that after almost all of the segments are
> merged, one or two new segments are created when a startFullFlush happens
> after the big merge.
>
> Any suggestions on how to troubleshoot this would be appreciated.
>
> Tom
>
> ---------- Forwarded message ----------
> From: Tom Burton-West <tb...@umich.edu>>
> Date: Mon, Feb 23, 2015 at 12:41 PM
> Subject: Optimize maxSegments="2" not working right with Solr 4.10.2
> To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <
> solr-user@lucene.apache.org<ma...@lucene.apache.org>>
> Cc: Phillip Farber <pf...@umich.edu>>,
> Sebastien Korner <sk...@umich.edu>>
>
>
> Hello,
>
> We normally run an optimize with maxSegments="2" after our daily
> indexing. This has worked without problem on Solr 3.6. We recently moved
> to Solr 4.10.2 and on several shards the optimize completed with no errors
> in the logs, but left more than 2 segments.
>
> We send this xml to Solr
> <optimize maxSegments="2"/>
>
> I've attached a copy of the indexwriter log for one of the segments where
> there were 4 segments rather than the requested number (i.e. there should
> have been only 2 segments) at the end of the optimize. It looks like a
> merge was done down to two segments and then somehow another process
> flushed some postings to disk creating two more segments. Then there are
> messages about 2 of the remaining 4 segments being too big. (See below)
>
> What we expected is that the remainng 2 small segments (about 40MB) would
> get merged with the smaller of the two large segments, i.e. with the 56GB
> segment, since we gave the argument maxSegments=2. This didn't happen.
>
>
> Any suggestions about how to troubleshoot this issue would be appreciated.
>
> Tom
>
> -------
> Excerpt from indexwriter log:
>
> TMP][http-8091-Processor5]: findForcedMerges maxSegmentCount=2 ...
> ...
> [IW][Lucene Merge Thread #0]: merge time 3842310 msec for 65236 docs
> ...
> [TMP][http-8091-Processor5]: findMerges: 4 segments
> [TMP][http-8091-Processor5]: seg=_1fzb(4.10.2):C1081559/24089:delGen=9
> size=672402.066 MB [skip: too large]
> [TMP][http-8091-Processor5]: seg=_1gj2(4.10.2):C65236/2:delGen=1
> size=56179.245 MB [skip: too large]
> [TMP][http-8091-Processor5]: seg=_1gj0(4.10.2):C16 size=44.280 MB
> [TMP][http-8091-Processor5]: seg=_1gj1(4.10.2):C8 size=40.442 MB
> [TMP][http-8091-Processor5]: allowedSegmentCount=3 vs count=4 (eligible
> count=2) tooBigCount=2
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
RE: Optimize maxSegments="2" not working right with Solr 4.10.2
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
I'm guessing that you are using LogByteSizeMergePolicy? As far as I can see from the code for Solr 4.8, normal maxMergeSize defaults to 2GB.
There also seems to be a bug that effectively disables forced merge as the default value for maxMergeSizeForForcedMerge (used when forcing a merge) is set with
public static final double DEFAULT_MAX_MERGE_MB_FOR_FORCED_MERGE = Long.MAX_VALUE;
maxMergeSizeForForcedMerge = (long) (DEFAULT_MAX_MERGE_MB_FOR_FORCED_MERGE*1024*1024);
which overflows to -1048576. You should be able to fix this by setting maxMergeSizeForForcedMerge explicitly.
Guessing here: Your forced merge attempt does nothing and the standard merge maxMergeSize is too low?
(I should fine-read the code and make a JIRA if the error is present in Solr 5)
BTW: Why 2 segments? Single segment indexes has memory and performance benefits, but there is not - to my knowledge - much difference between 2 or 4 (or 10) segments.
- Toke Eskildsen
________________________________________
From: Tom Burton-West [tburtonw@umich.edu]
Sent: 25 February 2015 18:11
To: dev@lucene.apache.org
Subject: Fwd: Optimize maxSegments="2" not working right with Solr 4.10.2
No replies on the Solr users list, so I thought I would repost to dev.
We are continuing to see inconsistent behavior with <optimize maxSegments="2"/>
Out of 12 shards 1-3 of them end up with more than 2 segments at the finish of the optimize command. (and we see no errors in the logs)
The pattern seems the same in that after almost all of the segments are merged, one or two new segments are created when a startFullFlush happens after the big merge.
Any suggestions on how to troubleshoot this would be appreciated.
Tom
---------- Forwarded message ----------
From: Tom Burton-West <tb...@umich.edu>>
Date: Mon, Feb 23, 2015 at 12:41 PM
Subject: Optimize maxSegments="2" not working right with Solr 4.10.2
To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Cc: Phillip Farber <pf...@umich.edu>>, Sebastien Korner <sk...@umich.edu>>
Hello,
We normally run an optimize with maxSegments="2" after our daily indexing. This has worked without problem on Solr 3.6. We recently moved to Solr 4.10.2 and on several shards the optimize completed with no errors in the logs, but left more than 2 segments.
We send this xml to Solr
<optimize maxSegments="2"/>
I've attached a copy of the indexwriter log for one of the segments where there were 4 segments rather than the requested number (i.e. there should have been only 2 segments) at the end of the optimize. It looks like a merge was done down to two segments and then somehow another process flushed some postings to disk creating two more segments. Then there are messages about 2 of the remaining 4 segments being too big. (See below)
What we expected is that the remainng 2 small segments (about 40MB) would get merged with the smaller of the two large segments, i.e. with the 56GB segment, since we gave the argument maxSegments=2. This didn't happen.
Any suggestions about how to troubleshoot this issue would be appreciated.
Tom
-------
Excerpt from indexwriter log:
TMP][http-8091-Processor5]: findForcedMerges maxSegmentCount=2 ...
...
[IW][Lucene Merge Thread #0]: merge time 3842310 msec for 65236 docs
...
[TMP][http-8091-Processor5]: findMerges: 4 segments
[TMP][http-8091-Processor5]: seg=_1fzb(4.10.2):C1081559/24089:delGen=9 size=672402.066 MB [skip: too large]
[TMP][http-8091-Processor5]: seg=_1gj2(4.10.2):C65236/2:delGen=1 size=56179.245 MB [skip: too large]
[TMP][http-8091-Processor5]: seg=_1gj0(4.10.2):C16 size=44.280 MB
[TMP][http-8091-Processor5]: seg=_1gj1(4.10.2):C8 size=40.442 MB
[TMP][http-8091-Processor5]: allowedSegmentCount=3 vs count=4 (eligible count=2) tooBigCount=2
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org