You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by er...@apache.org on 2018/06/29 01:45:58 UTC
lucene-solr:branch_7x: LUCENE-8370: Reproducing TestLucene{54,
70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()
failures
Repository: lucene-solr
Updated Branches:
refs/heads/branch_7x 4212da569 -> 1f5c75cb9
LUCENE-8370: Reproducing TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failures
(cherry picked from commit c303c5f)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/1f5c75cb
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/1f5c75cb
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/1f5c75cb
Branch: refs/heads/branch_7x
Commit: 1f5c75cb9a3704db395cd13140005130dcf726c0
Parents: 4212da5
Author: Erick Erickson <er...@apache.org>
Authored: Thu Jun 28 18:28:37 2018 -0700
Committer: Erick Erickson <er...@apache.org>
Committed: Thu Jun 28 18:35:40 2018 -0700
----------------------------------------------------------------------
lucene/CHANGES.txt | 4 ++++
.../apache/lucene/index/TieredMergePolicy.java | 19 +++++++++++++------
.../apache/lucene/index/RandomIndexWriter.java | 4 +++-
solr/solr-ref-guide/src/solr-upgrade-notes.adoc | 2 +-
.../src/uploading-data-with-index-handlers.adoc | 2 +-
5 files changed, 22 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1f5c75cb/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index bb5f1c1..4fe236b 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -15,6 +15,10 @@ Bug Fixes:
via LUCENE-8165 since we are now checking for offset+length going out of bounds.
(Robert Muir, Nhat Nyugen, Simon Willnauer)
+* LUCENE-8370: Reproducing
+ TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()
+ failures (Erick Erickson)
+
Changes in Runtime Behavior:
* LUCENE-7976: TieredMergePolicy now respects maxSegmentSizeMB by default when executing
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1f5c75cb/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java b/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java
index 3d51c9c..94a284b 100644
--- a/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java
+++ b/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java
@@ -60,15 +60,22 @@ import java.util.Set;
* LogMergePolicy}.
*
* <p><b>NOTE</b>: This policy always merges by byte size
- * of the segments, always pro-rates by percent deletes,
- * and does not apply any maximum segment size during
- * forceMerge (unlike {@link LogByteSizeMergePolicy}).
+ * of the segments, always pro-rates by percent deletes
*
- * <p><b>NOTE</b> As of Lucene 7.5, forceMerge (aka optimize)
- * and expungeDeletes (findForcedMerges and
- * findForcedDeletesMerges) respect the max segment
+ * <p><b>NOTE</b> Starting with Lucene 7.5, there are several changes:
+ *
+ * - findForcedMerges and findForcedDeletesMerges) respect the max segment
* size by default.
*
+ * - When findforcedmerges is called with maxSegmentCount other than 1,
+ * the resulting index is not guaranteed to have <= maxSegmentCount segments.
+ * Rather it is on a "best effort" basis. Specifically the theoretical ideal
+ * segment size is calculated and a "fudge factor" of 25% is added as the
+ * new maxSegmentSize, which is respected.
+ *
+ * - findForcedDeletesMerges will not produce segments greater than
+ * maxSegmentSize.
+ *
* @lucene.experimental
*/
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1f5c75cb/lucene/test-framework/src/java/org/apache/lucene/index/RandomIndexWriter.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/index/RandomIndexWriter.java b/lucene/test-framework/src/java/org/apache/lucene/index/RandomIndexWriter.java
index e2db533..33c4421 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/index/RandomIndexWriter.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/index/RandomIndexWriter.java
@@ -369,7 +369,9 @@ public class RandomIndexWriter implements Closeable {
System.out.println("RIW: doRandomForceMerge(" + limit + ")");
}
w.forceMerge(limit);
- assert !doRandomForceMergeAssert || w.getSegmentCount() <= limit: "limit=" + limit + " actual=" + w.getSegmentCount();
+ if (limit == 1 || (w.getConfig().getMergePolicy() instanceof TieredMergePolicy) == false) {
+ assert !doRandomForceMergeAssert || w.getSegmentCount() <= limit : "limit=" + limit + " actual=" + w.getSegmentCount();
+ }
} else {
if (LuceneTestCase.VERBOSE) {
System.out.println("RIW: do random forceMergeDeletes()");
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1f5c75cb/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solr-upgrade-notes.adoc b/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
index 0230625..52fc9d4 100644
--- a/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
+++ b/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
@@ -30,7 +30,7 @@ Detailed steps for upgrading a Solr cluster are in the section <<upgrading-a-sol
=== Solr 7.5
When upgrading to Solr 7.4, users should be aware of the following major changes from v7.3:
-* When using the default TieredMergePolicy (TMP), optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5,000 (5 gigaBytes). If it is absolutely necessary to control the number of segments present after optimize, specify maxSegments=# where # is a positive integer. TMP will also reclaim resources from segments that exceed maxMergedSegmentMB more aggressively.
+* When using the default TieredMergePolicy (TMP), optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5,000 (5 gigaBytes). If it is absolutely necessary to control the number of segments present after optimize, specify maxSegments=# where # is a positive integer. maxSegments > 1 are honored on a "best effort" basis. TMP will also reclaim resources from segments that exceed maxMergedSegmentMB more aggressively.
=== Solr 7.4
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1f5c75cb/solr/solr-ref-guide/src/uploading-data-with-index-handlers.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/uploading-data-with-index-handlers.adoc b/solr/solr-ref-guide/src/uploading-data-with-index-handlers.adoc
index 26d9bfe..381c0f6 100644
--- a/solr/solr-ref-guide/src/uploading-data-with-index-handlers.adoc
+++ b/solr/solr-ref-guide/src/uploading-data-with-index-handlers.adoc
@@ -100,7 +100,7 @@ Default is `true`. Blocks until a new searcher is opened and registered as the m
WARNING: expungeDeletes is "less expensive" than optimize, but the same warnings apply.
-`maxSegments`:: (optimize only) Default is unlimited, resulting segments respect the maxMergedSegmentMB setting. Merges the segments down to no more than this number of segments. Unless there is tangible evidence that optimizing to a small number of segments is beneficial, this parameter should be omitted and the default behavior accepted.
+`maxSegments`:: (optimize only) Default is unlimited, resulting segments respect the maxMergedSegmentMB setting. Makes a "best effort" attempt to merge the segments down to no more than this number of segments but does not guarantee that the goal will be achieved. Unless there is tangible evidence that optimizing to a small number of segments is beneficial, this parameter should be omitted and the default behavior accepted.
Here are examples of <commit> and <optimize> using optional attributes: