You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by dw...@apache.org on 2021/03/10 10:02:18 UTC
[lucene] branch branch_7_4 created (now 8cfc86b)
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a change to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git.
at 8cfc86b SOLR-12765: Incorrect format of JMX cache stats.
This branch includes the following new commits:
new 46d2187 LUCENE-8365: Fix ArrayIndexOutOfBoundsException in UnifiedHighlighter
new c8f5307 Add 7.4.0 bw indices.
new 891aa2a Add 7.4.0 to the doap files.
new a9ff152 Bump version to 7.4.1.
new ebc1b17 remove duplicate opening <release> tag
new 1cd10c1 SOLR-12450: Add CVE number to changelog
new a35f18e LUCENE-8381: Do not count hard-deletes as soft-deletes in merges
new 911f96b LUCENE-8381: Add change log entry
new c4f3c27 LUCENE-8384: Fix missing advance DVGen in PendingSoftDeletes
new f3a984e SOLR-11277: Documentation for maxSize based auto hard commit
new 05787c5 SOLR-12547: Fix typo in documentation for maxSize based auto-commit
new ae3cdc1 SOLR-12570: OpenNLPExtractNamedEntitiesUpdateProcessor cannot support multi fields because pattern replacement doesn't work correctly
new c205d5a SOLR-12594: MetricsHistoryHandler.getOverseerLeader fails when hostname contains hyphen.
new c1259f8 LUCENE-8472: Always rewrite soft-deletes retention query
new 8cfc86b SOLR-12765: Incorrect format of JMX cache stats.
The 15 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[lucene] 08/15: LUCENE-8381: Add change log entry
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 911f96b0f3ccf9416f04195b87b2d5296de24a8f
Author: Nhat Nguyen <nh...@elastic.co>
AuthorDate: Wed Jul 4 20:16:56 2018 -0400
LUCENE-8381: Add change log entry
---
lucene/CHANGES.txt | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 8cf709f..2583bf9 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -12,6 +12,9 @@ Bug Fixes:
a "off by one" error in the UnifiedHighlighter's code that is only triggered when
two nested SpanNearQueries contain the same term. (Marc-Andre Morissette via Simon Willnauer)
+ * LUCENE-8381: Fix IndexWriter incorrectly interprets hard-deletes as soft-deletes
+ while wrapping reader for merges. (Simon Willnauer, Nhat Nguyen)
+
======================= Lucene 7.4.0 =======================
Upgrading
[lucene] 09/15: LUCENE-8384: Fix missing advance DVGen in
PendingSoftDeletes
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit c4f3c27998b7ae5f3247e81ea1b0a940cc3e834f
Author: Nhat Nguyen <nh...@elastic.co>
AuthorDate: Wed Jul 4 12:46:42 2018 -0400
LUCENE-8384: Fix missing advance DVGen in PendingSoftDeletes
Today we only advance the docValues update generation in
PendingSoftDeletes for the soft-deletes field. If we update a
soft-deletes DV field, then update a non-soft-deletes DV field, then
onNewReader will consider that we never update DV field although we did.
This commit makes sure that we always advance the docValues update
generation when handling docValues update in PendingSoftDeletes.
Co-authored-by: Simon Willnauer <si...@apache.org>
---
lucene/CHANGES.txt | 3 +++
.../apache/lucene/index/PendingSoftDeletes.java | 6 +++---
.../org/apache/lucene/index/SegmentCommitInfo.java | 4 ++--
.../index/TestSoftDeletesRetentionMergePolicy.java | 22 ++++++++++++++++++++++
4 files changed, 30 insertions(+), 5 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 2583bf9..d3435c1 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -15,6 +15,9 @@ Bug Fixes:
* LUCENE-8381: Fix IndexWriter incorrectly interprets hard-deletes as soft-deletes
while wrapping reader for merges. (Simon Willnauer, Nhat Nguyen)
+ * LUCENE-8384: Fix missing advance docValues generation while handling docValues
+ update in PendingSoftDeletes. (Simon Willnauer, Nhat Nguyen)
+
======================= Lucene 7.4.0 =======================
Upgrading
diff --git a/lucene/core/src/java/org/apache/lucene/index/PendingSoftDeletes.java b/lucene/core/src/java/org/apache/lucene/index/PendingSoftDeletes.java
index 4074903..eb6e4ff 100644
--- a/lucene/core/src/java/org/apache/lucene/index/PendingSoftDeletes.java
+++ b/lucene/core/src/java/org/apache/lucene/index/PendingSoftDeletes.java
@@ -146,12 +146,12 @@ final class PendingSoftDeletes extends PendingDeletes {
if (this.field.equals(info.name)) {
pendingDeleteCount += applySoftDeletes(iterator, getMutableBits());
assert assertPendingDeletes();
- assert dvGeneration < info.getDocValuesGen() : "we have seen this generation update already: " + dvGeneration + " vs. " + info.getDocValuesGen();
- assert dvGeneration != -2 : "docValues generation is still uninitialized";
- dvGeneration = info.getDocValuesGen();
this.info.setSoftDelCount(this.info.getSoftDelCount() + pendingDeleteCount);
super.dropChanges();
}
+ assert dvGeneration < info.getDocValuesGen() : "we have seen this generation update already: " + dvGeneration + " vs. " + info.getDocValuesGen();
+ assert dvGeneration != -2 : "docValues generation is still uninitialized";
+ dvGeneration = info.getDocValuesGen();
}
private boolean assertPendingDeletes() {
diff --git a/lucene/core/src/java/org/apache/lucene/index/SegmentCommitInfo.java b/lucene/core/src/java/org/apache/lucene/index/SegmentCommitInfo.java
index 954a138..6911757 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SegmentCommitInfo.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SegmentCommitInfo.java
@@ -328,7 +328,7 @@ public class SegmentCommitInfo {
if (delCount < 0 || delCount > info.maxDoc()) {
throw new IllegalArgumentException("invalid delCount=" + delCount + " (maxDoc=" + info.maxDoc() + ")");
}
- assert softDelCount + delCount <= info.maxDoc();
+ assert softDelCount + delCount <= info.maxDoc() : "maxDoc=" + info.maxDoc() + ",delCount=" + delCount + ",softDelCount=" + softDelCount;
this.delCount = delCount;
}
@@ -336,7 +336,7 @@ public class SegmentCommitInfo {
if (softDelCount < 0 || softDelCount > info.maxDoc()) {
throw new IllegalArgumentException("invalid softDelCount=" + softDelCount + " (maxDoc=" + info.maxDoc() + ")");
}
- assert softDelCount + delCount <= info.maxDoc();
+ assert softDelCount + delCount <= info.maxDoc() : "maxDoc=" + info.maxDoc() + ",delCount=" + delCount + ",softDelCount=" + softDelCount;
this.softDelCount = softDelCount;
}
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java b/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
index bb36ba3..ee2b650 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
@@ -40,6 +40,8 @@ import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.MatchAllDocsQuery;
import org.apache.lucene.search.MatchNoDocsQuery;
import org.apache.lucene.search.Query;
+import org.apache.lucene.search.SearcherFactory;
+import org.apache.lucene.search.SearcherManager;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.Directory;
@@ -607,6 +609,26 @@ public class TestSoftDeletesRetentionMergePolicy extends LuceneTestCase {
IOUtils.close(writer, dir);
}
+ public void testSoftDeleteWithTryUpdateDocValue() throws Exception {
+ Directory dir = newDirectory();
+ IndexWriterConfig config = newIndexWriterConfig().setSoftDeletesField("soft_delete")
+ .setMergePolicy(new SoftDeletesRetentionMergePolicy("soft_delete", MatchAllDocsQuery::new, newLogMergePolicy()));
+ IndexWriter writer = new IndexWriter(dir, config);
+ SearcherManager sm = new SearcherManager(writer, new SearcherFactory());
+ Document d = new Document();
+ d.add(new StringField("id", "0", Field.Store.YES));
+ writer.addDocument(d);
+ sm.maybeRefreshBlocking();
+ doUpdate(new Term("id", "0"), writer,
+ new NumericDocValuesField("soft_delete", 1), new NumericDocValuesField("other-field", 1));
+ sm.maybeRefreshBlocking();
+ assertEquals(1, writer.segmentInfos.asList().size());
+ SegmentCommitInfo si = writer.segmentInfos.asList().get(0);
+ assertEquals(1, si.getSoftDelCount());
+ assertEquals(1, si.info.maxDoc());
+ IOUtils.close(sm, writer, dir);
+ }
+
static void doUpdate(Term doc, IndexWriter writer, Field... fields) throws IOException {
long seqId = -1;
do { // retry if we just committing a merge
[lucene] 03/15: Add 7.4.0 to the doap files.
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 891aa2a891dfc3e08d8aebb2e18a4b0c2956d1f7
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Wed Jun 27 09:33:00 2018 +0200
Add 7.4.0 to the doap files.
---
dev-tools/doap/lucene.rdf | 7 +++++++
dev-tools/doap/solr.rdf | 8 ++++++++
2 files changed, 15 insertions(+)
diff --git a/dev-tools/doap/lucene.rdf b/dev-tools/doap/lucene.rdf
index 1c76fee..f498bcb 100644
--- a/dev-tools/doap/lucene.rdf
+++ b/dev-tools/doap/lucene.rdf
@@ -69,6 +69,13 @@
<!-- NOTE: please insert releases in numeric order, NOT chronologically. -->
<release>
<Version>
+ <name>lucene-7.4.0</name>
+ <created>2018-06-27</created>
+ <revision>7.4.0</revision>
+ </Version>
+ </release>
+ <release>
+ <Version>
<name>lucene-7.3.1</name>
<created>2018-05-15</created>
<revision>7.3.1</revision>
diff --git a/dev-tools/doap/solr.rdf b/dev-tools/doap/solr.rdf
index b020f6d..25fb295 100644
--- a/dev-tools/doap/solr.rdf
+++ b/dev-tools/doap/solr.rdf
@@ -69,6 +69,14 @@
<!-- NOTE: please insert releases in numeric order, NOT chronologically. -->
<release>
<Version>
+ <name>solr-7.4.0</name>
+ <created>2018-06-27</created>
+ <revision>7.4.1</revision>
+ </Version>
+ </release>
+ <release>
+ <release>
+ <Version>
<name>solr-7.3.1</name>
<created>2018-05-15</created>
<revision>7.3.1</revision>
[lucene] 05/15: remove duplicate opening tag
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit ebc1b17c2188dfe4f2f13730f536bc968b6c3689
Author: Steve Rowe <sa...@apache.org>
AuthorDate: Wed Jun 27 05:25:00 2018 -0400
remove duplicate opening <release> tag
---
dev-tools/doap/solr.rdf | 1 -
1 file changed, 1 deletion(-)
diff --git a/dev-tools/doap/solr.rdf b/dev-tools/doap/solr.rdf
index 25fb295..e10285f 100644
--- a/dev-tools/doap/solr.rdf
+++ b/dev-tools/doap/solr.rdf
@@ -75,7 +75,6 @@
</Version>
</release>
<release>
- <release>
<Version>
<name>solr-7.3.1</name>
<created>2018-05-15</created>
[lucene] 04/15: Bump version to 7.4.1.
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit a9ff1529f6a024848ec8faf20c4548393ffd4632
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Wed Jun 27 10:21:50 2018 +0200
Bump version to 7.4.1.
---
.../core/src/java/org/apache/lucene/util/Version.java | 11 +++++++++--
lucene/version.properties | 2 +-
solr/CHANGES.txt | 17 +++++++++++++++++
.../solr/configsets/_default/conf/solrconfig.xml | 2 +-
solr/example/example-DIH/solr/atom/conf/solrconfig.xml | 2 +-
solr/example/example-DIH/solr/db/conf/solrconfig.xml | 2 +-
solr/example/example-DIH/solr/mail/conf/solrconfig.xml | 2 +-
solr/example/example-DIH/solr/solr/conf/solrconfig.xml | 2 +-
solr/example/example-DIH/solr/tika/conf/solrconfig.xml | 2 +-
solr/example/files/conf/solrconfig.xml | 2 +-
.../server/solr/configsets/_default/conf/solrconfig.xml | 2 +-
.../sample_techproducts_configs/conf/solrconfig.xml | 2 +-
12 files changed, 36 insertions(+), 12 deletions(-)
diff --git a/lucene/core/src/java/org/apache/lucene/util/Version.java b/lucene/core/src/java/org/apache/lucene/util/Version.java
index 418c9ff..d00746b 100644
--- a/lucene/core/src/java/org/apache/lucene/util/Version.java
+++ b/lucene/core/src/java/org/apache/lucene/util/Version.java
@@ -194,11 +194,18 @@ public final class Version {
/**
* Match settings and bugs in Lucene's 7.4.0 release.
+ * @deprecated (7.4.1) Use latest
+ */
+ @Deprecated
+ public static final Version LUCENE_7_4_0 = new Version(7, 4, 0);
+
+ /**
+ * Match settings and bugs in Lucene's 7.4.1 release.
* <p>
* Use this to get the latest & greatest settings, bug
* fixes, etc, for Lucene.
*/
- public static final Version LUCENE_7_4_0 = new Version(7, 4, 0);
+ public static final Version LUCENE_7_4_1 = new Version(7, 4, 1);
// To add a new version:
// * Only add above this comment
@@ -219,7 +226,7 @@ public final class Version {
* some defaults may have changed and may break functionality
* in your application.
*/
- public static final Version LATEST = LUCENE_7_4_0;
+ public static final Version LATEST = LUCENE_7_4_1;
/**
* Constant for backwards compatibility.
diff --git a/lucene/version.properties b/lucene/version.properties
index 8780350..0bb9b9e 100644
--- a/lucene/version.properties
+++ b/lucene/version.properties
@@ -2,7 +2,7 @@
# RELEASE MANAGER must change this file after creating a release and
# enter new base version (format "x.y.z", no prefix/appendix):
-version.base=7.4.0
+version.base=7.4.1
# Other version property defaults, don't change:
version.suffix=SNAPSHOT
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 7ac0e07..9f8dc4b 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -16,6 +16,23 @@ In this release, there is an example Solr server including a bundled
servlet container in the directory named "example".
See the Solr tutorial at https://lucene.apache.org/solr/guide/solr-tutorial.html
+================== 7.4.1 ==================
+
+Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
+
+Versions of Major Components
+---------------------
+Apache Tika 1.17
+Carrot2 3.16.0
+Velocity 1.7 and Velocity Tools 2.0
+Apache UIMA 2.3.1
+Apache ZooKeeper 3.4.11
+Jetty 9.4.10.v20180503
+
+
+(No Changes)
+
+
================== 7.4.0 ==================
Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
diff --git a/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml b/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml
index 694e797..672731e 100644
--- a/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml
+++ b/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml
@@ -35,7 +35,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<!-- <lib/> directives can be used to instruct Solr to load any Jars
identified and use them to resolve any "plugins" specified in
diff --git a/solr/example/example-DIH/solr/atom/conf/solrconfig.xml b/solr/example/example-DIH/solr/atom/conf/solrconfig.xml
index 97a5d36..afcb060 100644
--- a/solr/example/example-DIH/solr/atom/conf/solrconfig.xml
+++ b/solr/example/example-DIH/solr/atom/conf/solrconfig.xml
@@ -36,7 +36,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar"/>
diff --git a/solr/example/example-DIH/solr/db/conf/solrconfig.xml b/solr/example/example-DIH/solr/db/conf/solrconfig.xml
index a52d7ab..076d843 100644
--- a/solr/example/example-DIH/solr/db/conf/solrconfig.xml
+++ b/solr/example/example-DIH/solr/db/conf/solrconfig.xml
@@ -35,7 +35,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<!-- <lib/> directives can be used to instruct Solr to load any Jars
identified and use them to resolve any "plugins" specified in
diff --git a/solr/example/example-DIH/solr/mail/conf/solrconfig.xml b/solr/example/example-DIH/solr/mail/conf/solrconfig.xml
index 50445bc..ed1e34d 100644
--- a/solr/example/example-DIH/solr/mail/conf/solrconfig.xml
+++ b/solr/example/example-DIH/solr/mail/conf/solrconfig.xml
@@ -35,7 +35,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<!-- <lib/> directives can be used to instruct Solr to load any Jars
identified and use them to resolve any "plugins" specified in
diff --git a/solr/example/example-DIH/solr/solr/conf/solrconfig.xml b/solr/example/example-DIH/solr/solr/conf/solrconfig.xml
index 850f491..5b996ec 100644
--- a/solr/example/example-DIH/solr/solr/conf/solrconfig.xml
+++ b/solr/example/example-DIH/solr/solr/conf/solrconfig.xml
@@ -35,7 +35,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<!-- <lib/> directives can be used to instruct Solr to load any Jars
identified and use them to resolve any "plugins" specified in
diff --git a/solr/example/example-DIH/solr/tika/conf/solrconfig.xml b/solr/example/example-DIH/solr/tika/conf/solrconfig.xml
index 76b9ed5..79bbd42 100644
--- a/solr/example/example-DIH/solr/tika/conf/solrconfig.xml
+++ b/solr/example/example-DIH/solr/tika/conf/solrconfig.xml
@@ -36,7 +36,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<!-- Load Data Import Handler and Apache Tika (extraction) libraries -->
<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar"/>
diff --git a/solr/example/files/conf/solrconfig.xml b/solr/example/files/conf/solrconfig.xml
index bed93dd..cab8b04 100644
--- a/solr/example/files/conf/solrconfig.xml
+++ b/solr/example/files/conf/solrconfig.xml
@@ -35,7 +35,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<!-- <lib/> directives can be used to instruct Solr to load any Jars
identified and use them to resolve any "plugins" specified in
diff --git a/solr/server/solr/configsets/_default/conf/solrconfig.xml b/solr/server/solr/configsets/_default/conf/solrconfig.xml
index 694e797..672731e 100644
--- a/solr/server/solr/configsets/_default/conf/solrconfig.xml
+++ b/solr/server/solr/configsets/_default/conf/solrconfig.xml
@@ -35,7 +35,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<!-- <lib/> directives can be used to instruct Solr to load any Jars
identified and use them to resolve any "plugins" specified in
diff --git a/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml b/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
index 5c27099..8af4717 100644
--- a/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
+++ b/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
@@ -35,7 +35,7 @@
that you fully re-index after changing this setting as it can
affect both how text is indexed and queried.
-->
- <luceneMatchVersion>7.4.0</luceneMatchVersion>
+ <luceneMatchVersion>7.4.1</luceneMatchVersion>
<!-- <lib/> directives can be used to instruct Solr to load any Jars
identified and use them to resolve any "plugins" specified in
[lucene] 01/15: LUCENE-8365: Fix ArrayIndexOutOfBoundsException in
UnifiedHighlighter
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 46d2187dc2326c50c49478a5b57bcdc29c4b01db
Author: Marc-Andre Morissette <mo...@lexum.com>
AuthorDate: Tue Jun 19 22:42:27 2018 -0400
LUCENE-8365: Fix ArrayIndexOutOfBoundsException in UnifiedHighlighter
This fixes a "off by one" error in the UnifiedHighlighter's code that
is only triggered when two nested SpanNearQueries contain the same term.
Signed-off-by: Simon Willnauer <si...@apache.org>
Closes #408
---
lucene/CHANGES.txt | 9 +++++++
.../lucene/search/uhighlight/PhraseHelper.java | 4 +--
.../TestUnifiedHighlighterStrictPhrases.java | 30 +++++++++++++++++++++-
3 files changed, 40 insertions(+), 3 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 28bb128..8cf709f 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -3,6 +3,15 @@ Lucene Change Log
For more information on past and future Lucene versions, please see:
http://s.apache.org/luceneversions
+
+======================= Lucene 7.4.1 =======================
+
+Bug Fixes:
+
+ * LUCENE-8365: Fix ArrayIndexOutOfBoundsException in UnifiedHighlighter. This fixes
+ a "off by one" error in the UnifiedHighlighter's code that is only triggered when
+ two nested SpanNearQueries contain the same term. (Marc-Andre Morissette via Simon Willnauer)
+
======================= Lucene 7.4.0 =======================
Upgrading
diff --git a/lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/PhraseHelper.java b/lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/PhraseHelper.java
index dbaac76..cc92fbc 100644
--- a/lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/PhraseHelper.java
+++ b/lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/PhraseHelper.java
@@ -383,8 +383,8 @@ public class PhraseHelper {
// shift right any pairs by one to make room
final int shiftLen = numPairs - (pairIdx + 1);
if (shiftLen > 0) {
- System.arraycopy(startOffsets, pairIdx + 2, startOffsets, pairIdx + 3, shiftLen);
- System.arraycopy(endOffsets, pairIdx + 2, endOffsets, pairIdx + 3, shiftLen);
+ System.arraycopy(startOffsets, pairIdx + 1, startOffsets, pairIdx + 2, shiftLen);
+ System.arraycopy(endOffsets, pairIdx + 1, endOffsets, pairIdx + 2, shiftLen);
}
// now we can place the offset pair
startOffsets[pairIdx + 1] = startOffset;
diff --git a/lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/TestUnifiedHighlighterStrictPhrases.java b/lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/TestUnifiedHighlighterStrictPhrases.java
index 9ddce08..1888942 100644
--- a/lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/TestUnifiedHighlighterStrictPhrases.java
+++ b/lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/TestUnifiedHighlighterStrictPhrases.java
@@ -463,6 +463,35 @@ public class TestUnifiedHighlighterStrictPhrases extends LuceneTestCase {
assertArrayEquals(new String[]{"There is no <b>accord</b> <b>and</b> <b>satisfaction</b> with this - <b>Consideration</b> of the <b>accord</b> is arbitrary."}, snippets);
}
+ // Tests that terms collected out of order due to being present in multiple Spans are handled correctly
+ // See LUCENE-8365
+ public void testReverseOrderSpanCollection() throws IOException {
+ // Processing order may depend on various optimizations or other weird factor.
+ indexWriter.addDocument(newDoc("alpha bravo - alpha charlie"));
+ indexWriter.addDocument(newDoc("alpha charlie - alpha bravo"));
+ initReaderSearcherHighlighter();
+
+ SpanNearQuery query = new SpanNearQuery(new SpanQuery[]{
+ new SpanNearQuery(new SpanQuery[]{
+ new SpanTermQuery(new Term("body", "alpha")),
+ new SpanTermQuery(new Term("body", "bravo"))
+ }, 0, true),
+ new SpanNearQuery(new SpanQuery[]{
+ new SpanTermQuery(new Term("body", "alpha")),
+ new SpanTermQuery(new Term("body", "charlie"))
+ }, 0, true)
+ }, 10, false);
+
+ TopDocs topDocs = searcher.search(query, 10, Sort.INDEXORDER);
+ String[] snippets = highlighter.highlight("body", query, topDocs);
+
+ assertArrayEquals(new String[]{
+ "<b>alpha</b> <b>bravo</b> - <b>alpha</b> <b>charlie</b>",
+ "<b>alpha</b> <b>charlie</b> - <b>alpha</b> <b>bravo</b>",
+ },
+ snippets);
+ }
+
private static class MyQuery extends Query {
private final Query wrapped;
@@ -563,5 +592,4 @@ public class TestUnifiedHighlighterStrictPhrases extends LuceneTestCase {
if (VERBOSE) System.out.println("Expected: \"" + expected + "\n" + "Observed: \"" + observed);
assertEquals("Nested SpanNear query within SpanOr not properly highlighted.", expected, observed);
}
-
}
[lucene] 14/15: LUCENE-8472: Always rewrite soft-deletes retention
query
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit c1259f8274ddfc4e138fb6b7de4f0f15872cff51
Author: Nhat Nguyen <nh...@elastic.co>
AuthorDate: Mon Sep 3 11:13:42 2018 -0400
LUCENE-8472: Always rewrite soft-deletes retention query
This change ensures that we always rewrite the soft-deletes merge
retention policy before passing it into `createWeight` as some Query
does not support `createWeight` directly.
---
lucene/CHANGES.txt | 2 ++
.../index/SoftDeletesRetentionMergePolicy.java | 2 +-
.../index/TestSoftDeletesRetentionMergePolicy.java | 27 ++++++++++++++++++++++
3 files changed, 30 insertions(+), 1 deletion(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index d3435c1..e58750d 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -18,6 +18,8 @@ Bug Fixes:
* LUCENE-8384: Fix missing advance docValues generation while handling docValues
update in PendingSoftDeletes. (Simon Willnauer, Nhat Nguyen)
+ * LUCENE-8472: Always rewrite the soft-deletes merge retention query. (Adrien Grand, Nhat Nguyen)
+
======================= Lucene 7.4.0 =======================
Upgrading
diff --git a/lucene/core/src/java/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.java b/lucene/core/src/java/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.java
index 182af50..007e222 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.java
@@ -140,7 +140,7 @@ public final class SoftDeletesRetentionMergePolicy extends OneMergeWrappingMerge
private static Scorer getScorer(Query query, CodecReader reader) throws IOException {
IndexSearcher s = new IndexSearcher(reader);
s.setQueryCache(null);
- Weight weight = s.createWeight(query, false, 1.0f);
+ Weight weight = s.createWeight(s.rewrite(query), false, 1.0f);
return weight.scorer(reader.getContext());
}
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java b/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
index ee2b650..33f5cc6 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
@@ -39,6 +39,7 @@ import org.apache.lucene.search.DocValuesFieldExistsQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.MatchAllDocsQuery;
import org.apache.lucene.search.MatchNoDocsQuery;
+import org.apache.lucene.search.PrefixQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.SearcherFactory;
import org.apache.lucene.search.SearcherManager;
@@ -629,6 +630,32 @@ public class TestSoftDeletesRetentionMergePolicy extends LuceneTestCase {
IOUtils.close(sm, writer, dir);
}
+ public void testRewriteRetentionQuery() throws Exception {
+ Directory dir = newDirectory();
+ IndexWriterConfig config = newIndexWriterConfig().setSoftDeletesField("soft_deletes")
+ .setMergePolicy(new SoftDeletesRetentionMergePolicy("soft_deletes",
+ () -> new PrefixQuery(new Term("id", "foo")), newMergePolicy()));
+ IndexWriter writer = new IndexWriter(dir, config);
+
+ Document d = new Document();
+ d.add(new StringField("id", "foo-1", Field.Store.YES));
+ writer.addDocument(d);
+ d = new Document();
+ d.add(new StringField("id", "foo-2", Field.Store.YES));
+ writer.softUpdateDocument(new Term("id", "foo-1"), d, new NumericDocValuesField("soft_deletes", 1));
+
+ d = new Document();
+ d.add(new StringField("id", "bar-1", Field.Store.YES));
+ writer.addDocument(d);
+ d.add(new StringField("id", "bar-2", Field.Store.YES));
+ writer.softUpdateDocument(new Term("id", "bar-1"), d, new NumericDocValuesField("soft_deletes", 1));
+
+ writer.forceMerge(1);
+ assertEquals(2, writer.numDocs()); // foo-2, bar-2
+ assertEquals(3, writer.maxDoc()); // foo-1, foo-2, bar-2
+ IOUtils.close(writer, dir);
+ }
+
static void doUpdate(Term doc, IndexWriter writer, Field... fields) throws IOException {
long seqId = -1;
do { // retry if we just committing a merge
[lucene] 02/15: Add 7.4.0 bw indices.
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit c8f53075cd2b4f898f8197d36886e60d1640b6b6
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Fri Jun 22 13:31:42 2018 +0200
Add 7.4.0 bw indices.
---
.../lucene/index/TestBackwardsCompatibility.java | 7 +++++--
.../test/org/apache/lucene/index/index.7.4.0-cfs.zip | Bin 0 -> 15484 bytes
.../test/org/apache/lucene/index/index.7.4.0-nocfs.zip | Bin 0 -> 15493 bytes
.../src/test/org/apache/lucene/index/sorted.7.4.0.zip | Bin 0 -> 158939 bytes
4 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index b7adc5a..83a9358 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -332,7 +332,9 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
"7.3.0-cfs",
"7.3.0-nocfs",
"7.3.1-cfs",
- "7.3.1-nocfs"
+ "7.3.1-nocfs",
+ "7.4.0-cfs",
+ "7.4.0-nocfs"
};
public static String[] getOldNames() {
@@ -359,7 +361,8 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
"sorted.7.2.0",
"sorted.7.2.1",
"sorted.7.3.0",
- "sorted.7.3.1"
+ "sorted.7.3.1",
+ "sorted.7.4.0"
};
public static String[] getOldSortedNames() {
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.4.0-cfs.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.4.0-cfs.zip
new file mode 100644
index 0000000..b810fd0
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.4.0-cfs.zip differ
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.4.0-nocfs.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.4.0-nocfs.zip
new file mode 100644
index 0000000..f2c4674
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.4.0-nocfs.zip differ
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.4.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.4.0.zip
new file mode 100644
index 0000000..b427a50
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.4.0.zip differ
[lucene] 07/15: LUCENE-8381: Do not count hard-deletes as
soft-deletes in merges
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit a35f18e56b7c178cdd90bce1ea873d328683fd1c
Author: Nhat Nguyen <nh...@elastic.co>
AuthorDate: Tue Jul 3 22:56:46 2018 -0400
LUCENE-8381: Do not count hard-deletes as soft-deletes in merges
Today if a document is soft-deleted, then hard-deleted, IW, however,
considers that doc as soft-deleted when wrapping readers for merges.
This change makes sure that IW excludes the hard-deleted documents from
a merge reader, and does not count them as soft-deletes.
Co-authored-by: Simon Willnauer <si...@apache.org>
---
.../org/apache/lucene/index/FilterCodecReader.java | 23 +++++++++++
.../java/org/apache/lucene/index/IndexWriter.java | 44 ++++++++++++++++++----
.../index/SoftDeletesRetentionMergePolicy.java | 35 ++---------------
.../index/TestSoftDeletesRetentionMergePolicy.java | 38 +++++++++++++++++++
4 files changed, 101 insertions(+), 39 deletions(-)
diff --git a/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java b/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java
index fd36ecb..4187051 100644
--- a/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java
@@ -127,4 +127,27 @@ public abstract class FilterCodecReader extends CodecReader {
in.checkIntegrity();
}
+ /**
+ * Returns a filtered codec reader with the given live docs and numDocs.
+ */
+ static FilterCodecReader wrapLiveDocs(CodecReader reader, Bits liveDocs, int numDocs) {
+ return new FilterCodecReader(reader) {
+ @Override
+ public CacheHelper getCoreCacheHelper() {
+ return reader.getCoreCacheHelper();
+ }
+ @Override
+ public CacheHelper getReaderCacheHelper() {
+ return null; // we are altering live docs
+ }
+ @Override
+ public Bits getLiveDocs() {
+ return liveDocs;
+ }
+ @Override
+ public int numDocs() {
+ return numDocs;
+ }
+ };
+ }
}
diff --git a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
index c4c5c57..e8ed2bb 100644
--- a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
@@ -48,6 +48,7 @@ import org.apache.lucene.index.DocValuesUpdate.BinaryDocValuesUpdate;
import org.apache.lucene.index.DocValuesUpdate.NumericDocValuesUpdate;
import org.apache.lucene.index.FieldInfos.FieldNumbers;
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
+import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.search.DocValuesFieldExistsQuery;
import org.apache.lucene.search.MatchAllDocsQuery;
import org.apache.lucene.search.Query;
@@ -4384,25 +4385,52 @@ public class IndexWriter implements Closeable, TwoPhaseCommit, Accountable,
// Let the merge wrap readers
List<CodecReader> mergeReaders = new ArrayList<>();
- int numSoftDeleted = 0;
- for (SegmentReader reader : merge.readers) {
+ int softDeleteCount = 0;
+ for (int r = 0; r < merge.readers.size(); r++) {
+ SegmentReader reader = merge.readers.get(r);
CodecReader wrappedReader = merge.wrapForMerge(reader);
validateMergeReader(wrappedReader);
- mergeReaders.add(wrappedReader);
if (softDeletesEnabled) {
if (reader != wrappedReader) { // if we don't have a wrapped reader we won't preserve any soft-deletes
- Bits liveDocs = wrappedReader.getLiveDocs();
- numSoftDeleted += PendingSoftDeletes.countSoftDeletes(
- DocValuesFieldExistsQuery.getDocValuesDocIdSetIterator(config.getSoftDeletesField(), wrappedReader),
- liveDocs);
+ Bits hardLiveDocs = merge.hardLiveDocs.get(r);
+ Bits wrappedLiveDocs = wrappedReader.getLiveDocs();
+ int hardDeleteCount = 0;
+ DocIdSetIterator softDeletedDocs = DocValuesFieldExistsQuery.getDocValuesDocIdSetIterator(config.getSoftDeletesField(), wrappedReader);
+ if (softDeletedDocs != null) {
+ int docId;
+ while ((docId = softDeletedDocs.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
+ if (wrappedLiveDocs == null || wrappedLiveDocs.get(docId)) {
+ if (hardLiveDocs == null || hardLiveDocs.get(docId)) {
+ softDeleteCount++;
+ } else {
+ hardDeleteCount++;
+ }
+ }
+ }
+ }
+ // Wrap the wrapped reader again if we have excluded some hard-deleted docs
+ if (hardLiveDocs != null && hardDeleteCount > 0) {
+ Bits liveDocs = wrappedLiveDocs == null ? hardLiveDocs : new Bits() {
+ @Override
+ public boolean get(int index) {
+ return hardLiveDocs.get(index) && wrappedLiveDocs.get(index);
+ }
+ @Override
+ public int length() {
+ return hardLiveDocs.length();
+ }
+ };
+ wrappedReader = FilterCodecReader.wrapLiveDocs(wrappedReader, liveDocs, wrappedReader.numDocs() - hardDeleteCount);
+ }
}
}
+ mergeReaders.add(wrappedReader);
}
final SegmentMerger merger = new SegmentMerger(mergeReaders,
merge.info.info, infoStream, dirWrapper,
globalFieldNumberMap,
context);
- merge.info.setSoftDelCount(numSoftDeleted);
+ merge.info.setSoftDelCount(softDeleteCount);
merge.checkAborted();
merge.mergeStartNS = System.nanoTime();
diff --git a/lucene/core/src/java/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.java b/lucene/core/src/java/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.java
index 36a2208..182af50 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SoftDeletesRetentionMergePolicy.java
@@ -73,7 +73,7 @@ public final class SoftDeletesRetentionMergePolicy extends OneMergeWrappingMerge
public boolean keepFullyDeletedSegment(IOSupplier<CodecReader> readerIOSupplier) throws IOException {
CodecReader reader = readerIOSupplier.get();
/* we only need a single hit to keep it no need for soft deletes to be checked*/
- Scorer scorer = getScorer(retentionQuerySupplier.get(), wrapLiveDocs(reader, null, reader.maxDoc()));
+ Scorer scorer = getScorer(retentionQuerySupplier.get(), FilterCodecReader.wrapLiveDocs(reader, null, reader.maxDoc()));
if (scorer != null) {
DocIdSetIterator iterator = scorer.iterator();
boolean atLeastOneHit = iterator.nextDoc() != DocIdSetIterator.NO_MORE_DOCS;
@@ -88,7 +88,7 @@ public final class SoftDeletesRetentionMergePolicy extends OneMergeWrappingMerge
if (liveDocs == null) { // no deletes - just keep going
return reader;
}
- CodecReader wrappedReader = wrapLiveDocs(reader, new Bits() { // only search deleted
+ CodecReader wrappedReader = FilterCodecReader.wrapLiveDocs(reader, new Bits() { // only search deleted
@Override
public boolean get(int index) {
return liveDocs.get(index) == false;
@@ -114,7 +114,7 @@ public final class SoftDeletesRetentionMergePolicy extends OneMergeWrappingMerge
}
}
assert reader.numDocs() + numExtraLiveDocs <= reader.maxDoc() : "numDocs: " + reader.numDocs() + " numExtraLiveDocs: " + numExtraLiveDocs + " maxDoc: " + reader.maxDoc();
- return wrapLiveDocs(reader, cloneLiveDocs, reader.numDocs() + numExtraLiveDocs);
+ return FilterCodecReader.wrapLiveDocs(reader, cloneLiveDocs, reader.numDocs() + numExtraLiveDocs);
} else {
return reader;
}
@@ -144,33 +144,6 @@ public final class SoftDeletesRetentionMergePolicy extends OneMergeWrappingMerge
return weight.scorer(reader.getContext());
}
- /**
- * Returns a codec reader with the given live docs
- */
- private static CodecReader wrapLiveDocs(CodecReader reader, Bits liveDocs, int numDocs) {
- return new FilterCodecReader(reader) {
- @Override
- public CacheHelper getCoreCacheHelper() {
- return reader.getCoreCacheHelper();
- }
-
- @Override
- public CacheHelper getReaderCacheHelper() {
- return null; // we are altering live docs
- }
-
- @Override
- public Bits getLiveDocs() {
- return liveDocs;
- }
-
- @Override
- public int numDocs() {
- return numDocs;
- }
- };
- }
-
@Override
public int numDeletesToMerge(SegmentCommitInfo info, int delCount, IOSupplier<CodecReader> readerSupplier) throws IOException {
final int numDeletesToMerge = super.numDeletesToMerge(info, delCount, readerSupplier);
@@ -180,7 +153,7 @@ public final class SoftDeletesRetentionMergePolicy extends OneMergeWrappingMerge
BooleanQuery.Builder builder = new BooleanQuery.Builder();
builder.add(new DocValuesFieldExistsQuery(field), BooleanClause.Occur.FILTER);
builder.add(retentionQuerySupplier.get(), BooleanClause.Occur.FILTER);
- Scorer scorer = getScorer(builder.build(), wrapLiveDocs(reader, null, reader.maxDoc()));
+ Scorer scorer = getScorer(builder.build(), FilterCodecReader.wrapLiveDocs(reader, null, reader.maxDoc()));
if (scorer != null) {
DocIdSetIterator iterator = scorer.iterator();
Bits liveDocs = reader.getLiveDocs();
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java b/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
index be1f7ac..bb36ba3 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java
@@ -569,6 +569,44 @@ public class TestSoftDeletesRetentionMergePolicy extends LuceneTestCase {
IOUtils.close(writer, dir);
}
+ public void testMergeSoftDeleteAndHardDelete() throws Exception {
+ Directory dir = newDirectory();
+ String softDelete = "soft_delete";
+ IndexWriterConfig config = newIndexWriterConfig()
+ .setSoftDeletesField(softDelete)
+ .setMergePolicy(new SoftDeletesRetentionMergePolicy("soft_delete",
+ MatchAllDocsQuery::new, new LogDocMergePolicy()));
+ config.setReaderPooling(true);
+ IndexWriter writer = new IndexWriter(dir, config);
+ Document d = new Document();
+ d.add(new StringField("id", "0", Field.Store.YES));
+ writer.addDocument(d);
+ d = new Document();
+ d.add(new StringField("id", "1", Field.Store.YES));
+ d.add(new NumericDocValuesField("soft_delete", 1));
+ writer.addDocument(d);
+ try (DirectoryReader reader = writer.getReader()) {
+ assertEquals(2, reader.maxDoc());
+ assertEquals(1, reader.numDocs());
+ }
+ while (true) {
+ try (DirectoryReader reader = writer.getReader()) {
+ TopDocs topDocs = new IndexSearcher(new NoDeletesWrapper(reader)).search(new TermQuery(new Term("id", "1")), 1);
+ assertEquals(1, topDocs.totalHits);
+ if (writer.tryDeleteDocument(reader, topDocs.scoreDocs[0].doc) > 0) {
+ break;
+ }
+ }
+ }
+ writer.forceMergeDeletes(true);
+ assertEquals(1, writer.segmentInfos.size());
+ SegmentCommitInfo si = writer.segmentInfos.info(0);
+ assertEquals(0, si.getSoftDelCount()); // hard-delete should supersede the soft-delete
+ assertEquals(0, si.getDelCount());
+ assertEquals(1, si.info.maxDoc());
+ IOUtils.close(writer, dir);
+ }
+
static void doUpdate(Term doc, IndexWriter writer, Field... fields) throws IOException {
long seqId = -1;
do { // retry if we just committing a merge
[lucene] 12/15: SOLR-12570:
OpenNLPExtractNamedEntitiesUpdateProcessor cannot support multi fields
because pattern replacement doesn't work correctly
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit ae3cdc187239c4daad57749b993a9a62d81de4da
Author: koji <ko...@apache.org>
AuthorDate: Mon Jul 23 18:02:40 2018 +0900
SOLR-12570: OpenNLPExtractNamedEntitiesUpdateProcessor cannot support multi fields because pattern replacement doesn't work correctly
---
solr/CHANGES.txt | 5 ++++-
...penNLPExtractNamedEntitiesUpdateProcessorFactory.java | 16 ++++++++--------
2 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 937315e..ae65ca9 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -29,8 +29,11 @@ Apache UIMA 2.3.1
Apache ZooKeeper 3.4.11
Jetty 9.4.10.v20180503
+Bug Fixes
+----------------------
-(No Changes)
+* SOLR-12570: OpenNLPExtractNamedEntitiesUpdateProcessor cannot support multi fields because pattern replacement
+ doesn't work correctly. (Koji Sekiguchi)
================== 7.4.0 ==================
diff --git a/solr/contrib/analysis-extras/src/java/org/apache/solr/update/processor/OpenNLPExtractNamedEntitiesUpdateProcessorFactory.java b/solr/contrib/analysis-extras/src/java/org/apache/solr/update/processor/OpenNLPExtractNamedEntitiesUpdateProcessorFactory.java
index 2a7514d..d69c367 100644
--- a/solr/contrib/analysis-extras/src/java/org/apache/solr/update/processor/OpenNLPExtractNamedEntitiesUpdateProcessorFactory.java
+++ b/solr/contrib/analysis-extras/src/java/org/apache/solr/update/processor/OpenNLPExtractNamedEntitiesUpdateProcessorFactory.java
@@ -17,8 +17,6 @@
package org.apache.solr.update.processor;
-import static org.apache.solr.common.SolrException.ErrorCode.SERVER_ERROR;
-
import java.io.IOException;
import java.lang.invoke.MethodHandles;
import java.util.ArrayList;
@@ -57,6 +55,8 @@ import org.apache.solr.util.plugin.SolrCoreAware;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static org.apache.solr.common.SolrException.ErrorCode.SERVER_ERROR;
+
/**
* Extracts named entities using an OpenNLP NER <code>modelFile</code> from the values found in
* any matching <code>source</code> field into a configured <code>dest</code> field, after
@@ -500,13 +500,13 @@ public class OpenNLPExtractNamedEntitiesUpdateProcessorFactory
SolrInputField destField = null;
String entityName = entity.first();
String entityType = entity.second();
- resolvedDest = resolvedDest.replace(ENTITY_TYPE, entityType);
- if (doc.containsKey(resolvedDest)) {
- destField = doc.getField(resolvedDest);
+ final String resolved = resolvedDest.replace(ENTITY_TYPE, entityType);
+ if (doc.containsKey(resolved)) {
+ destField = doc.getField(resolved);
} else {
- SolrInputField targetField = destMap.get(resolvedDest);
+ SolrInputField targetField = destMap.get(resolved);
if (targetField == null) {
- destField = new SolrInputField(resolvedDest);
+ destField = new SolrInputField(resolved);
} else {
destField = targetField;
}
@@ -514,7 +514,7 @@ public class OpenNLPExtractNamedEntitiesUpdateProcessorFactory
destField.addValue(entityName);
// put it in map to avoid concurrent modification...
- destMap.put(resolvedDest, destField);
+ destMap.put(resolved, destField);
}
}
}
[lucene] 06/15: SOLR-12450: Add CVE number to changelog
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 1cd10c108a00225919c3b1c4ad98d2f418be6b28
Author: Uwe Schindler <us...@apache.org>
AuthorDate: Wed Jul 4 18:35:06 2018 +0200
SOLR-12450: Add CVE number to changelog
---
solr/CHANGES.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 9f8dc4b..937315e 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -314,7 +314,7 @@ Bug Fixes
* SOLR-12416: When creating a time routed alias, the router.autoDeleteAge option wasn't considered.
(Joachim Sauer via David Smiley)
-* SOLR-12450: Don't allow referal to external resources in various config files.
+* SOLR-12450: Don't allow referal to external resources in various config files (CVE-2018-8026).
(Yuyang Xiao, Uwe Schindler)
Optimizations
[lucene] 13/15: SOLR-12594: MetricsHistoryHandler.getOverseerLeader
fails when hostname contains hyphen.
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit c205d5ac133cd516ab7602588e8e16b9bd1dd9b5
Author: Andrzej Bialecki <ab...@apache.org>
AuthorDate: Thu Aug 2 18:47:58 2018 +0200
SOLR-12594: MetricsHistoryHandler.getOverseerLeader fails when hostname contains hyphen.
---
solr/CHANGES.txt | 1 +
.../org/apache/solr/handler/admin/MetricsHistoryHandler.java | 11 +++++++----
.../solr/metrics/reporters/solr/SolrClusterReporter.java | 11 +++++++----
.../org/apache/solr/metrics/rrd/SolrRrdBackendFactory.java | 9 +++++++--
.../apache/solr/handler/admin/MetricsHistoryHandlerTest.java | 2 +-
5 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index ae65ca9..e031271 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -35,6 +35,7 @@ Bug Fixes
* SOLR-12570: OpenNLPExtractNamedEntitiesUpdateProcessor cannot support multi fields because pattern replacement
doesn't work correctly. (Koji Sekiguchi)
+* SOLR-12594: MetricsHistoryHandler.getOverseerLeader fails when hostname contains hyphen. (ab)
================== 7.4.0 ==================
diff --git a/solr/core/src/java/org/apache/solr/handler/admin/MetricsHistoryHandler.java b/solr/core/src/java/org/apache/solr/handler/admin/MetricsHistoryHandler.java
index 419f021..9a46d04 100644
--- a/solr/core/src/java/org/apache/solr/handler/admin/MetricsHistoryHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/admin/MetricsHistoryHandler.java
@@ -62,6 +62,7 @@ import org.apache.solr.client.solrj.cloud.SolrCloudManager;
import org.apache.solr.client.solrj.cloud.autoscaling.ReplicaInfo;
import org.apache.solr.client.solrj.cloud.autoscaling.VersionedData;
import org.apache.solr.client.solrj.impl.HttpClientUtil;
+import org.apache.solr.cloud.LeaderElector;
import org.apache.solr.cloud.Overseer;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.cloud.ClusterState;
@@ -332,12 +333,14 @@ public class MetricsHistoryHandler extends RequestHandlerBase implements Permiss
if (oid == null) {
return null;
}
- String[] ids = oid.split("-");
- if (ids.length != 3) { // unknown format
- log.warn("Unknown format of leader id, skipping: " + oid);
+ String nodeName = null;
+ try {
+ nodeName = LeaderElector.getNodeName(oid);
+ } catch (Exception e) {
+ log.warn("Unknown format of leader id, skipping: " + oid, e);
return null;
}
- return ids[1];
+ return nodeName;
}
private boolean amIOverseerLeader() {
diff --git a/solr/core/src/java/org/apache/solr/metrics/reporters/solr/SolrClusterReporter.java b/solr/core/src/java/org/apache/solr/metrics/reporters/solr/SolrClusterReporter.java
index 9e8861e..17390e1 100644
--- a/solr/core/src/java/org/apache/solr/metrics/reporters/solr/SolrClusterReporter.java
+++ b/solr/core/src/java/org/apache/solr/metrics/reporters/solr/SolrClusterReporter.java
@@ -28,6 +28,7 @@ import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
import org.apache.http.client.HttpClient;
+import org.apache.solr.cloud.LeaderElector;
import org.apache.solr.cloud.Overseer;
import org.apache.solr.cloud.ZkController;
import org.apache.solr.common.cloud.SolrZkClient;
@@ -270,13 +271,15 @@ public class SolrClusterReporter extends SolrCoreContainerReporter {
if (oid == null) {
return lastKnownUrl;
}
- String[] ids = oid.split("-");
- if (ids.length != 3) { // unknown format
- log.warn("Unknown format of leader id, skipping: " + oid);
+ String nodeName = null;
+ try {
+ nodeName = LeaderElector.getNodeName(oid);
+ } catch (Exception e) {
+ log.warn("Unknown format of leader id, skipping: " + oid, e);
return lastKnownUrl;
}
// convert nodeName back to URL
- String url = zk.getZkStateReader().getBaseUrlForNodeName(ids[1]);
+ String url = zk.getZkStateReader().getBaseUrlForNodeName(nodeName);
// check that it's parseable
try {
new java.net.URL(url);
diff --git a/solr/core/src/java/org/apache/solr/metrics/rrd/SolrRrdBackendFactory.java b/solr/core/src/java/org/apache/solr/metrics/rrd/SolrRrdBackendFactory.java
index a3c6f64..5448d8f 100644
--- a/solr/core/src/java/org/apache/solr/metrics/rrd/SolrRrdBackendFactory.java
+++ b/solr/core/src/java/org/apache/solr/metrics/rrd/SolrRrdBackendFactory.java
@@ -206,7 +206,8 @@ public class SolrRrdBackendFactory extends RrdBackendFactory implements SolrClos
return null;
}
if (o instanceof byte[]) {
- Long time = (Long)doc.getFieldValue("timestamp_l");
+ Object timeObj = doc.getFieldValue("timestamp_l");
+ Long time = timeObj instanceof Number ? ((Number)timeObj).longValue() : Long.parseLong(String.valueOf(timeObj));
return new SolrRrdBackend.SyncData((byte[])o, time);
} else {
throw new SolrServerException("Unexpected value of '" + DATA_FIELD + "' field: " + o.getClass().getName() + ": " + o);
@@ -248,7 +249,11 @@ public class SolrRrdBackendFactory extends RrdBackendFactory implements SolrClos
SolrDocumentList docs = rsp.getResults();
if (docs != null) {
docs.forEach(d -> {
- Long time = (Long)d.getFieldValue("timestamp_l");
+ Object o = d.getFieldValue("timestamp_l");
+ if (o == null) {
+ return;
+ }
+ Long time = o instanceof Number ? ((Number)o).longValue() : Long.parseLong(String.valueOf(o));
Pair<String, Long> p = new Pair<>(((String)d.getFieldValue("id")).substring(idPrefixLength), time);
byName.put(p.first(), p);
});
diff --git a/solr/core/src/test/org/apache/solr/handler/admin/MetricsHistoryHandlerTest.java b/solr/core/src/test/org/apache/solr/handler/admin/MetricsHistoryHandlerTest.java
index 7c84c16..735b427 100644
--- a/solr/core/src/test/org/apache/solr/handler/admin/MetricsHistoryHandlerTest.java
+++ b/solr/core/src/test/org/apache/solr/handler/admin/MetricsHistoryHandlerTest.java
@@ -58,7 +58,7 @@ public class MetricsHistoryHandlerTest extends SolrCloudTestCase {
@BeforeClass
public static void beforeClass() throws Exception {
- simulated = random().nextBoolean() || true;
+ simulated = random().nextBoolean();
Map<String, Object> args = new HashMap<>();
args.put(MetricsHistoryHandler.SYNC_PERIOD_PROP, 1);
args.put(MetricsHistoryHandler.COLLECT_PERIOD_PROP, 1);
[lucene] 15/15: SOLR-12765: Incorrect format of JMX cache stats.
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 8cfc86b419d0db5cd2bc692a144e32bbdee7ae2f
Author: Andrzej Bialecki <ab...@apache.org>
AuthorDate: Mon Sep 17 14:40:36 2018 +0200
SOLR-12765: Incorrect format of JMX cache stats.
---
solr/CHANGES.txt | 2 ++
.../apache/solr/metrics/reporters/jmx/JmxMetricsReporter.java | 10 +++++++---
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index e031271..42bb0c7 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -37,6 +37,8 @@ Bug Fixes
* SOLR-12594: MetricsHistoryHandler.getOverseerLeader fails when hostname contains hyphen. (ab)
+* SOLR-12765: Incorrect format of JMX cache stats. (Bojan Smid, ab)
+
================== 7.4.0 ==================
Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
diff --git a/solr/core/src/java/org/apache/solr/metrics/reporters/jmx/JmxMetricsReporter.java b/solr/core/src/java/org/apache/solr/metrics/reporters/jmx/JmxMetricsReporter.java
index 54da5fa..7b0b8ac 100644
--- a/solr/core/src/java/org/apache/solr/metrics/reporters/jmx/JmxMetricsReporter.java
+++ b/solr/core/src/java/org/apache/solr/metrics/reporters/jmx/JmxMetricsReporter.java
@@ -50,6 +50,7 @@ import com.codahale.metrics.ObjectNameFactory;
import com.codahale.metrics.Reporter;
import com.codahale.metrics.Timer;
import org.apache.solr.metrics.MetricsMap;
+import org.apache.solr.metrics.SolrMetricManager;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -558,9 +559,12 @@ public class JmxMetricsReporter implements Reporter, Closeable {
try {
if (filter.matches(name, gauge)) {
final ObjectName objectName = createName("gauges", name);
- if (gauge instanceof MetricsMap) {
- ((MetricsMap)gauge).setAttribute(new Attribute(INSTANCE_TAG, tag));
- registerMBean(gauge, objectName);
+ if (gauge instanceof SolrMetricManager.GaugeWrapper &&
+ ((SolrMetricManager.GaugeWrapper)gauge).getGauge() instanceof MetricsMap) {
+ MetricsMap mm = (MetricsMap)((SolrMetricManager.GaugeWrapper)gauge).getGauge();
+ mm.setAttribute(new Attribute(INSTANCE_TAG, tag));
+ // don't wrap it in a JmxGauge, it already supports all necessary JMX attributes
+ registerMBean(mm, objectName);
} else {
registerMBean(new JmxGauge(gauge, objectName, tag), objectName);
}
[lucene] 10/15: SOLR-11277: Documentation for maxSize based auto
hard commit
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit f3a984e320ccee1006058d4ef924c4dbd359dbaf
Author: Anshum Gupta <an...@apache.org>
AuthorDate: Mon Jun 18 09:56:43 2018 -0700
SOLR-11277: Documentation for maxSize based auto hard commit
---
solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc b/solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc
index cd60315..bad06d9 100644
--- a/solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc
@@ -47,10 +47,13 @@ The number of updates that have occurred since the last commit.
`maxTime`::
The number of milliseconds since the oldest uncommitted update.
+`maxSize`::
+The maxSize of the transaction log, after which a hard commit is triggered. This is useful when the size of documents is unknown and the intention is to restric the size of the tlog to reasonable size.
+
`openSearcher`::
Whether to open a new searcher when performing a commit. If this is `false`, the commit will flush recent index changes to stable storage, but does not cause a new searcher to be opened to make those changes visible. The default is `true`.
-If either of the `maxDocs` or `maxTime` limits are reached, Solr automatically performs a commit operation. If the `autoCommit` tag is missing, then only explicit commits will update the index. The decision whether to use auto-commit or not depends on the needs of your application.
+If any of the `maxDocs`, `maxTime`, or `maxSize` limits are reached, Solr automatically performs a commit operation. If the `autoCommit` tag is missing, then only explicit commits will update the index. The decision whether to use auto-commit or not depends on the needs of your application.
Determining the best auto-commit settings is a tradeoff between performance and accuracy. Settings that cause frequent updates will improve the accuracy of searches because new content will be searchable more quickly, but performance may suffer because of the frequent updates. Less frequent updates may improve performance but it will take longer for updates to show up in queries.
@@ -59,6 +62,7 @@ Determining the best auto-commit settings is a tradeoff between performance and
<autoCommit>
<maxDocs>10000</maxDocs>
<maxTime>30000</maxTime>
+ <maxSize>512m</maxSize>
<openSearcher>false</openSearcher>
</autoCommit>
----
[lucene] 11/15: SOLR-12547: Fix typo in documentation for maxSize
based auto-commit
Posted by dw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_4
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 05787c586781339d63fd13282f3135c75f63743d
Author: Anshum Gupta <an...@apache.org>
AuthorDate: Wed Jul 11 23:00:49 2018 -0700
SOLR-12547: Fix typo in documentation for maxSize based auto-commit
---
solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc b/solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc
index bad06d9..0af467f 100644
--- a/solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/updatehandlers-in-solrconfig.adoc
@@ -48,7 +48,7 @@ The number of updates that have occurred since the last commit.
The number of milliseconds since the oldest uncommitted update.
`maxSize`::
-The maxSize of the transaction log, after which a hard commit is triggered. This is useful when the size of documents is unknown and the intention is to restric the size of the tlog to reasonable size.
+The maxSize of the transaction log, after which a hard commit is triggered. This is useful when the size of documents is unknown and the intention is to restrict the size of the tlog to reasonable size.
`openSearcher`::
Whether to open a new searcher when performing a commit. If this is `false`, the commit will flush recent index changes to stable storage, but does not cause a new searcher to be opened to make those changes visible. The default is `true`.