You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ab...@apache.org on 2017/10/05 11:56:22 UTC
[01/50] lucene-solr:feature/autoscaling_72: LUCENE-7978: Add some
more documentation about setting up build environment;
merge branch 'LUCENE-7978' of https://github.com/antonmry/lucene-solr
Repository: lucene-solr
Updated Branches:
refs/heads/feature/autoscaling_72 f28904bf9 -> c0abcff11
LUCENE-7978: Add some more documentation about setting up build environment; merge branch 'LUCENE-7978' of https://github.com/antonmry/lucene-solr
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/f1bb9147
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/f1bb9147
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/f1bb9147
Branch: refs/heads/feature/autoscaling_72
Commit: f1bb91478449ee7613a28feb596f497dbcd82042
Parents: eaa5586 e148574
Author: Uwe Schindler <us...@apache.org>
Authored: Fri Sep 29 14:53:56 2017 +0200
Committer: Uwe Schindler <us...@apache.org>
Committed: Fri Sep 29 14:53:56 2017 +0200
----------------------------------------------------------------------
README.md | 8 ++++++++
lucene/CHANGES.txt | 3 +++
2 files changed, 11 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f1bb9147/README.md
----------------------------------------------------------------------
diff --cc README.md
index 7181297,50da250..0dac902
--- a/README.md
+++ b/README.md
@@@ -41,6 -41,12 +41,14 @@@ Lucene and Solr, run
`ant compile`
-If you see an error about Ivy missing when invoking ant (e.g., `.ant/lib does not exist`), run `ant ivy-bootstrap` and retry.
++If you see an error about Ivy missing while invoking Ant (e.g., `.ant/lib does
++not exist`), run `ant ivy-bootstrap` and retry.
+
-Sometimes you may face issues with Ivy (e.g., instance a wrongly downloaded artifact). Cleaning up the Ivy cache and retrying is a workaround for most of such issues:
++Sometimes you may face issues with Ivy (e.g., an incompletely downloaded artifact).
++Cleaning up the Ivy cache and retrying is a workaround for most of such issues:
+
+ `rm -rf ~/.ivy2/cache`
+
The Solr server can then be packaged and prepared for startup by running the
following command from the `solr/` directory:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f1bb9147/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --cc lucene/CHANGES.txt
index ffc901c,ffc901c..ae61d86
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@@ -92,6 -92,6 +92,9 @@@ Othe
* LUCENE-7933: LongBitSet now validates the numBits parameter (Won
Jonghoon, Mike McCandless)
++* LUCENE-7978: Add some more documentation about setting up build
++ environment. (Anton R. Yuste via Uwe Schindler)
++
======================= Lucene 7.0.0 =======================
New Features
[28/50] lucene-solr:feature/autoscaling_72: LUCENE-7985: Upgrade
forbiddenapis to 2.4.1
Posted by ab...@apache.org.
LUCENE-7985: Upgrade forbiddenapis to 2.4.1
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/fb19528f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/fb19528f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/fb19528f
Branch: refs/heads/feature/autoscaling_72
Commit: fb19528f6c4d12694b2af3b42516b048317eb423
Parents: fc42939
Author: Uwe Schindler <us...@apache.org>
Authored: Wed Oct 4 01:35:40 2017 +0200
Committer: Uwe Schindler <us...@apache.org>
Committed: Wed Oct 4 01:35:40 2017 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 2 ++
lucene/common-build.xml | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fb19528f/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 24885d6..d82889b 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -93,6 +93,8 @@ Build
* LUCENE-6673: Maven build fails for target javadoc:jar.
(Ramkumar Aiyengar, Daniel Collins via Steve Rowe)
+* LUCENE-7985: Upgrade forbiddenapis to 2.4.1. (Uwe Schindler)
+
Other
* LUCENE-7948, LUCENE-7937: Upgrade randomizedtesting to 2.5.3 (minor fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fb19528f/lucene/common-build.xml
----------------------------------------------------------------------
diff --git a/lucene/common-build.xml b/lucene/common-build.xml
index cc0b356..03c6c26 100644
--- a/lucene/common-build.xml
+++ b/lucene/common-build.xml
@@ -2411,7 +2411,7 @@ ${ant.project.name}.test.dependencies=${test.classpath.list}
<!-- GROOVY scripting engine for ANT tasks -->
<target name="resolve-groovy" unless="groovy.loaded" depends="ivy-availability-check,ivy-configure">
- <ivy:cachepath organisation="org.codehaus.groovy" module="groovy-all" revision="2.4.8"
+ <ivy:cachepath organisation="org.codehaus.groovy" module="groovy-all" revision="2.4.12"
inline="true" conf="default" type="jar" transitive="true" pathid="groovy.classpath"/>
<taskdef name="groovy"
classname="org.codehaus.groovy.ant.Groovy"
@@ -2425,7 +2425,7 @@ ${ant.project.name}.test.dependencies=${test.classpath.list}
<property name="forbidden-sysout-excludes" value=""/>
<target name="-install-forbidden-apis" unless="forbidden-apis.loaded" depends="ivy-availability-check,ivy-configure">
- <ivy:cachepath organisation="de.thetaphi" module="forbiddenapis" revision="2.3"
+ <ivy:cachepath organisation="de.thetaphi" module="forbiddenapis" revision="2.4.1"
inline="true" conf="default" transitive="true" pathid="forbidden-apis.classpath"/>
<taskdef name="forbidden-apis" classname="de.thetaphi.forbiddenapis.ant.AntTask" classpathref="forbidden-apis.classpath"/>
<property name="forbidden-apis.loaded" value="true"/>
[40/50] lucene-solr:feature/autoscaling_72: fix precommit errors
after merging autoscaling branch into master
Posted by ab...@apache.org.
fix precommit errors after merging autoscaling branch into master
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/fe38dcaf
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/fe38dcaf
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/fe38dcaf
Branch: refs/heads/feature/autoscaling_72
Commit: fe38dcafbeae6188d67891dd6518d78070b17786
Parents: 15d17ed
Author: Steve Rowe <sa...@gmail.com>
Authored: Wed Oct 4 12:17:40 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Wed Oct 4 12:17:40 2017 -0400
----------------------------------------------------------------------
solr/core/src/java/org/apache/solr/cloud/AddReplicaCmd.java | 1 -
.../src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java | 2 +-
solr/solrj/src/java/org/apache/solr/common/MapWriter.java | 1 -
.../org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java | 1 -
4 files changed, 1 insertion(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fe38dcaf/solr/core/src/java/org/apache/solr/cloud/AddReplicaCmd.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/AddReplicaCmd.java b/solr/core/src/java/org/apache/solr/cloud/AddReplicaCmd.java
index 6535d34..62f01c5 100644
--- a/solr/core/src/java/org/apache/solr/cloud/AddReplicaCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/AddReplicaCmd.java
@@ -24,7 +24,6 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
-import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.commons.lang.StringUtils;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fe38dcaf/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
index dedaebe..f9ec691 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
@@ -49,7 +49,7 @@ public class AutoScaling {
/**
* Interface for a Solr trigger. Each trigger implements Runnable and Closeable interface. A trigger
* is scheduled using a {@link java.util.concurrent.ScheduledExecutorService} so it is executed as
- * per a configured schedule to check whether the trigger is ready to fire. The {@link Trigger#setProcessor(TriggerEventProcessor)}
+ * per a configured schedule to check whether the trigger is ready to fire. The {@link AutoScaling.Trigger#setProcessor(AutoScaling.TriggerEventProcessor)}
* method should be used to set a processor which is used by implementation of this class whenever
* ready.
* <p>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fe38dcaf/solr/solrj/src/java/org/apache/solr/common/MapWriter.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/common/MapWriter.java b/solr/solrj/src/java/org/apache/solr/common/MapWriter.java
index 080d750..c8ef3fb 100644
--- a/solr/solrj/src/java/org/apache/solr/common/MapWriter.java
+++ b/solr/solrj/src/java/org/apache/solr/common/MapWriter.java
@@ -20,7 +20,6 @@ package org.apache.solr.common;
import java.io.IOException;
import java.util.ArrayList;
-import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fe38dcaf/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java b/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
index a91698d..e13a29a 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
@@ -35,7 +35,6 @@ import org.apache.solr.client.solrj.SolrRequest;
import org.apache.solr.client.solrj.cloud.autoscaling.Clause.Violation;
import org.apache.solr.client.solrj.cloud.autoscaling.Policy.Suggester.Hint;
import org.apache.solr.client.solrj.request.CollectionAdminRequest;
-import org.apache.solr.cloud.autoscaling.TriggerEvent;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.ReplicaPosition;
import org.apache.solr.common.cloud.ZkStateReader;
[37/50] lucene-solr:feature/autoscaling_72: LUCENE-7982: add missing
import
Posted by ab...@apache.org.
LUCENE-7982: add missing import
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a6052ba7
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a6052ba7
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a6052ba7
Branch: refs/heads/feature/autoscaling_72
Commit: a6052ba78d496373bb121339d5806c3e9ab672e9
Parents: 0b11ee5
Author: Mike McCandless <mi...@apache.org>
Authored: Wed Oct 4 10:22:10 2017 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Wed Oct 4 10:22:10 2017 -0400
----------------------------------------------------------------------
.../src/java/org/apache/lucene/search/NormsFieldExistsQuery.java | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a6052ba7/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java b/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java
index be0311e..10a6ffa 100644
--- a/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java
+++ b/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java
@@ -20,6 +20,7 @@ package org.apache.lucene.search;
import java.io.IOException;
import java.util.Objects;
+import org.apache.lucene.document.StringField;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.FieldInfos;
import org.apache.lucene.index.LeafReader;
[15/50] lucene-solr:feature/autoscaling_72: LUCENE-6673: Maven build
fails for target javadoc:jar.
Posted by ab...@apache.org.
LUCENE-6673: Maven build fails for target javadoc:jar.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/0062e999
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/0062e999
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/0062e999
Branch: refs/heads/feature/autoscaling_72
Commit: 0062e999203fb6dbb7bd9d37e1c4a95cc2367504
Parents: d52564c
Author: Steve Rowe <sa...@apache.org>
Authored: Sun Oct 1 19:03:55 2017 -0400
Committer: Steve Rowe <sa...@apache.org>
Committed: Sun Oct 1 19:03:55 2017 -0400
----------------------------------------------------------------------
dev-tools/maven/pom.xml.template | 5 +++++
lucene/CHANGES.txt | 3 +++
2 files changed, 8 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0062e999/dev-tools/maven/pom.xml.template
----------------------------------------------------------------------
diff --git a/dev-tools/maven/pom.xml.template b/dev-tools/maven/pom.xml.template
index 6b7f915..c676f6e 100644
--- a/dev-tools/maven/pom.xml.template
+++ b/dev-tools/maven/pom.xml.template
@@ -230,6 +230,11 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
+ <configuration>
+ <quiet>true</quiet>
+ <additionalparam>-Xdoclint:all</additionalparam>
+ <additionalparam>-Xdoclint:-missing</additionalparam>
+ </configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0062e999/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index e0c8124..9cc0fbd 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -86,6 +86,9 @@ Build
* SOLR-11181: Switch order of maven artifact publishing procedure: deploy first
instead of locally installing first, to workaround a double repository push of
*-sources.jar and *-javadoc.jar files. (Lynn Monson via Steve Rowe)
+
+* LUCENE-6673: Maven build fails for target javadoc:jar.
+ (Ramkumar Aiyengar, Daniel Collins via Steve Rowe)
Other
[48/50] lucene-solr:feature/autoscaling_72: SOLR-11440: Not a fix.
Swallowing the exception for the time being.
Posted by ab...@apache.org.
SOLR-11440: Not a fix. Swallowing the exception for the time being.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6b327fa4
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/6b327fa4
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/6b327fa4
Branch: refs/heads/feature/autoscaling_72
Commit: 6b327fa4046bdae94210fb4faa76e66ebe133fb1
Parents: 080232f
Author: Noble Paul <no...@apache.org>
Authored: Thu Oct 5 16:34:13 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Thu Oct 5 16:34:13 2017 +1030
----------------------------------------------------------------------
.../solr/cloud/autoscaling/ScheduledTriggers.java | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6b327fa4/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
index 7a96552..7bbbcd9 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
@@ -37,6 +37,7 @@ import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
@@ -146,7 +147,18 @@ public class ScheduledTriggers implements Closeable {
if (isClosed) {
throw new AlreadyClosedException("ScheduledTriggers has been closed and cannot be used anymore");
}
- ScheduledTrigger scheduledTrigger = new ScheduledTrigger(newTrigger, zkClient, queueStats);
+ ScheduledTrigger st = null;
+ try {
+ st = new ScheduledTrigger(newTrigger, zkClient, queueStats);
+ } catch (Exception e) {
+ //TODO to decide what to do
+ log.error("Failed to add trigger", e);
+ return;
+ }
+
+ ScheduledTrigger scheduledTrigger = st;
+
+
ScheduledTrigger old = scheduledTriggers.putIfAbsent(newTrigger.getName(), scheduledTrigger);
if (old != null) {
if (old.trigger.equals(newTrigger)) {
[20/50] lucene-solr:feature/autoscaling_72: Ref Guide: fix bad line
breaks lost during conversion from Confluence
Posted by ab...@apache.org.
Ref Guide: fix bad line breaks lost during conversion from Confluence
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/266cc82d
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/266cc82d
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/266cc82d
Branch: refs/heads/feature/autoscaling_72
Commit: 266cc82df0f422c977107d2bd4d403a58eea3f09
Parents: 2a13e78
Author: Cassandra Targett <ct...@apache.org>
Authored: Mon Oct 2 12:42:21 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Mon Oct 2 12:42:21 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/266cc82d/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc b/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc
index 96ccbca..ea790a8 100644
--- a/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc
+++ b/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc
@@ -59,9 +59,7 @@ Please follow the instructions to install Solr as a Service on Linux documented
[NOTE]
====
-
If you have a `/var/solr/solr.in.sh` file for your existing Solr install, running the `install_solr_service.sh` script will move this file to its new location: `/etc/default/solr.in.sh` (see https://issues.apache.org/jira/browse/SOLR-8101[SOLR-8101] for more details)
-
====
@@ -69,7 +67,11 @@ If you have a `/var/solr/solr.in.sh` file for your existing Solr install, runnin
Open `/etc/default/solr.in.sh` with a text editor and verify that the following variables are set correctly, or add them bottom of the include file as needed:
-`ZK_HOST=SOLR_HOST=SOLR_PORT=SOLR_HOME=`
+[source]
+ZK_HOST=
+SOLR_HOST=
+SOLR_PORT=
+SOLR_HOME=
Make sure the user you plan to own the Solr process is the owner of the `SOLR_HOME` directory. For instance, if you plan to run Solr as the "solr" user and `SOLR_HOME` is `/var/solr/data`, then you would do: `sudo chown -R solr: /var/solr/data`
@@ -80,7 +82,7 @@ You are now ready to start the upgraded Solr node by doing: `sudo service solr s
=== Step 5: Run Healthcheck
-You should run the Solr *healthcheck* command for all collections that are hosted on the upgraded node before proceeding to upgrade the next node in your cluster. For instance, if the newly upgraded node hosts a replica for the *MyDocuments* collection, then you can run the following command (replace ZK_HOST with the ZooKeeper connection string):
+You should run the Solr *healthcheck* command for all collections that are hosted on the upgraded node before proceeding to upgrade the next node in your cluster. For instance, if the newly upgraded node hosts a replica for the *MyDocuments* collection, then you can run the following command (replace `ZK_HOST` with the ZooKeeper connection string):
[source,bash]
----
[23/50] lucene-solr:feature/autoscaling_72: Merge branch 'master' of
https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Posted by ab...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/310bcd7c
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/310bcd7c
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/310bcd7c
Branch: refs/heads/feature/autoscaling_72
Commit: 310bcd7cbe7cd1563fbd83c85a4c25df7d3eb02f
Parents: 942b671 fd2b4f3
Author: Noble Paul <no...@apache.org>
Authored: Tue Oct 3 23:57:15 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Tue Oct 3 23:57:15 2017 +1030
----------------------------------------------------------------------
.gitignore | 1 +
README.md | 8 +
dev-tools/doap/lucene.rdf | 7 +
dev-tools/doap/solr.rdf | 7 +
.../lucene/backward-codecs/pom.xml.template | 13 +
.../lucene/spatial-extras/pom.xml.template | 7 +
.../maven/lucene/spatial3d/pom.xml.template | 13 +
dev-tools/maven/pom.xml.template | 5 +
.../maven/solr/core/src/test/pom.xml.template | 7 +
dev-tools/scripts/addBackcompatIndexes.py | 69 +-
dev-tools/scripts/buildAndPushRelease.py | 11 +-
lucene/CHANGES.txt | 22 +
.../index/TestBackwardsCompatibility.java | 49 +-
.../org/apache/lucene/index/dvupdates.7.0.0.zip | Bin 0 -> 13401 bytes
.../org/apache/lucene/index/empty.7.0.0.zip | Bin 0 -> 375 bytes
.../org/apache/lucene/index/index.7.0.0-cfs.zip | Bin 0 -> 15594 bytes
.../apache/lucene/index/index.7.0.0-nocfs.zip | Bin 0 -> 15586 bytes
.../org/apache/lucene/index/moreterms.7.0.0.zip | Bin 0 -> 175822 bytes
.../org/apache/lucene/index/sorted.7.0.0.zip | Bin 0 -> 74558 bytes
.../classification/ClassificationTestBase.java | 1 +
.../java/org/apache/lucene/util/Version.java | 7 +
.../lucene/search/TestIntRangeFieldQueries.java | 2 +-
.../directory/DirectoryTaxonomyWriter.java | 2 +-
lucene/ivy-versions.properties | 8 +-
.../morfologik-ukrainian-search-3.7.6.jar.sha1 | 1 -
.../morfologik-ukrainian-search-3.9.0.jar.sha1 | 1 +
.../apache/lucene/queries/mlt/MoreLikeThis.java | 16 +-
.../document/FloatPointNearestNeighbor.java | 383 ++++++
.../lucene/document/LatLonBoundingBox.java | 231 ++++
.../org/apache/lucene/document/LatLonPoint.java | 3 +-
.../document/TestFloatPointNearestNeighbor.java | 239 ++++
.../search/TestLatLonBoundingBoxQueries.java | 299 +++++
lucene/site/xsl/index.xsl | 2 +-
lucene/spatial-extras/build.xml | 9 +
.../spatial/spatial4j/Geo3dBinaryCodec.java | 130 ++
.../spatial/spatial4j/Geo3dCircleShape.java | 82 ++
.../spatial4j/Geo3dDistanceCalculator.java | 154 +++
.../spatial/spatial4j/Geo3dPointShape.java | 78 ++
.../spatial/spatial4j/Geo3dRectangleShape.java | 163 +++
.../lucene/spatial/spatial4j/Geo3dShape.java | 164 ++-
.../spatial/spatial4j/Geo3dShapeFactory.java | 396 ++++++
.../spatial4j/Geo3dSpatialContextFactory.java | 94 ++
.../apache/lucene/spatial/SpatialArgsTest.java | 12 +-
.../lucene/spatial/spatial4j/Geo3dRptTest.java | 193 +--
.../Geo3dShapeRectRelationTestCase.java | 20 +-
.../Geo3dShapeSphereModelRectRelationTest.java | 29 +-
.../Geo3dShapeWGS84ModelRectRelationTest.java | 21 +
.../apache/lucene/spatial3d/geom/Bounded.java | 35 +
.../lucene/spatial3d/geom/GeoCircleFactory.java | 16 +
.../lucene/spatial3d/geom/GeoExactCircle.java | 373 ++++++
.../apache/lucene/spatial3d/geom/GeoShape.java | 10 +-
.../lucene/spatial3d/geom/PlanetModel.java | 61 +-
.../lucene/spatial3d/geom/StandardObjects.java | 1 +
.../lucene/spatial3d/geom/GeoCircleTest.java | 83 ++
.../geom/RandomGeo3dShapeGenerator.java | 40 +-
.../java/org/apache/lucene/geo/GeoTestUtil.java | 5 +
.../search/BaseRangeFieldQueryTestCase.java | 10 +-
solr/common-build.xml | 9 +-
solr/core/build.xml | 3 +-
.../org/apache/solr/cloud/DeleteShardCmd.java | 1 +
.../org/apache/solr/cloud/DistributedQueue.java | 481 -------
.../org/apache/solr/cloud/ElectionContext.java | 38 +-
.../java/org/apache/solr/cloud/Overseer.java | 28 +-
.../apache/solr/cloud/OverseerTaskQueue.java | 4 +-
.../org/apache/solr/cloud/RecoveryStrategy.java | 1 +
.../apache/solr/cloud/ReplicateFromLeader.java | 7 +-
.../org/apache/solr/cloud/ZkController.java | 6 +-
.../apache/solr/cloud/ZkDistributedQueue.java | 489 +++++++
.../org/apache/solr/core/CoreContainer.java | 42 +-
.../apache/solr/handler/CdcrRequestHandler.java | 9 +-
.../org/apache/solr/handler/IndexFetcher.java | 16 +-
.../apache/solr/handler/ReplicationHandler.java | 31 +-
.../org/apache/solr/handler/StreamHandler.java | 15 +-
.../solr/handler/admin/PrepRecoveryOp.java | 33 -
.../solr/highlight/UnifiedSolrHighlighter.java | 18 +-
.../solr/index/SlowCompositeReaderWrapper.java | 16 +-
.../apache/solr/metrics/SolrMetricManager.java | 2 +-
.../solr/schema/AbstractSpatialFieldType.java | 5 +
.../apache/solr/schema/ExternalFileField.java | 1 -
.../java/org/apache/solr/update/UpdateLog.java | 12 +-
.../ClassificationUpdateProcessorParams.java | 2 +-
.../processor/DistributedUpdateProcessor.java | 2 +-
.../processor/UpdateRequestProcessor.java | 1 +
.../solr/collection1/conf/schema-spatial.xml | 4 +
.../collection1/conf/solrconfig-spatial.xml | 6 +
.../solr/configsets/backcompat/conf/schema.xml | 22 +
.../configsets/backcompat/conf/solrconfig.xml | 43 +
.../backcompat/TestLuceneIndexBackCompat.java | 96 ++
.../org/apache/solr/cloud/DeleteShardTest.java | 4 +-
.../apache/solr/cloud/DistributedQueueTest.java | 15 +-
.../org/apache/solr/cloud/ForceLeaderTest.java | 4 +-
.../apache/solr/cloud/MultiThreadedOCPTest.java | 5 +-
.../org/apache/solr/cloud/OverseerTest.java | 22 +-
.../cloud/TestCloudJSONFacetJoinDomain.java | 51 +-
.../cloud/TestRandomRequestDistribution.java | 2 +-
.../solr/cloud/TestShardsWithSingleReplica.java | 84 ++
.../org/apache/solr/cloud/TestTlogReplica.java | 50 +-
.../highlight/TestUnifiedSolrHighlighter.java | 3 +
.../solr/schema/SpatialRPTFieldTypeTest.java | 12 +-
.../apache/solr/search/TestSolr4Spatial2.java | 21 +-
.../asciidoctor-ant-1.6.0-alpha.3.jar.sha1 | 1 -
.../asciidoctor-ant-1.6.0-alpha.5.jar.sha1 | 1 +
solr/site/quickstart.mdtext | 614 +--------
.../src/basic-authentication-plugin.adoc | 4 +-
solr/solr-ref-guide/src/blob-store-api.adoc | 5 +-
.../src/collapse-and-expand-results.adoc | 20 +-
solr/solr-ref-guide/src/enabling-ssl.adoc | 18 +-
solr/solr-ref-guide/src/errata.adoc | 4 +-
solr/solr-ref-guide/src/faceting.adoc | 4 +-
.../solr-tutorial/tutorial-add-copy-field.png | Bin 123194 -> 93190 bytes
.../images/solr-tutorial/tutorial-add-field.png | Bin 212142 -> 110529 bytes
.../src/implicit-requesthandlers.adoc | 19 +-
solr/solr-ref-guide/src/index-replication.adoc | 4 +-
solr/solr-ref-guide/src/installing-solr.adoc | 2 +-
.../src/major-changes-in-solr-7.adoc | 12 +-
.../src/making-and-restoring-backups.adoc | 5 +-
solr/solr-ref-guide/src/managed-resources.adoc | 2 +-
solr/solr-ref-guide/src/managing-solr.adoc | 2 +-
.../src/mbean-request-handler.adoc | 13 +-
.../src/meta-docs/asciidoc-syntax.adoc | 10 +-
solr/solr-ref-guide/src/metrics-reporting.adoc | 21 +-
.../src/near-real-time-searching.adoc | 8 +-
.../src/pdf/SolrRefGuide-all.adoc | 1 +
.../src/performance-statistics-reference.adoc | 160 ++-
solr/solr-ref-guide/src/query-re-ranking.adoc | 2 +-
solr/solr-ref-guide/src/response-writers.adoc | 28 +-
.../src/rule-based-authorization-plugin.adoc | 23 +-
.../src/running-solr-on-hdfs.adoc | 3 +-
solr/solr-ref-guide/src/solr-tutorial.adoc | 10 +-
.../src/solrcloud-autoscaling-actions.adoc | 28 +
.../src/solrcloud-autoscaling-api.adoc | 4 +-
...solrcloud-autoscaling-auto-add-replicas.adoc | 21 +
.../solrcloud-autoscaling-fault-tolerance.adoc | 60 +
.../src/solrcloud-autoscaling-listeners.adoc | 179 +++
.../src/solrcloud-autoscaling-overview.adoc | 43 +-
.../src/solrcloud-autoscaling-triggers.adoc | 106 ++
.../src/solrcloud-autoscaling.adoc | 7 +-
...rcloud-query-routing-and-read-tolerance.adoc | 6 +-
solr/solr-ref-guide/src/spatial-search.adoc | 71 +-
.../src/statistical-programming.adoc | 190 ++-
.../src/stream-decorator-reference.adoc | 1216 ++++++++++++++++++
solr/solr-ref-guide/src/stream-decorators.adoc | 1216 ------------------
.../src/stream-evaluator-reference.adoc | 966 ++++++++++++++
solr/solr-ref-guide/src/stream-evaluators.adoc | 1030 ---------------
.../src/stream-source-reference.adoc | 539 ++++++++
solr/solr-ref-guide/src/stream-sources.adoc | 540 --------
.../src/streaming-expressions.adoc | 9 +-
.../src/the-extended-dismax-query-parser.adoc | 3 +-
.../src/the-query-elevation-component.adoc | 24 +-
.../solr-ref-guide/src/the-stats-component.adoc | 8 +-
.../src/the-term-vector-component.adoc | 7 +-
.../solr-ref-guide/src/the-terms-component.adoc | 18 +-
.../src/updating-parts-of-documents.adoc | 6 +-
.../src/upgrading-a-solr-cluster.adoc | 18 +-
...g-data-with-solr-cell-using-apache-tika.adoc | 2 -
.../solr-ref-guide/src/using-jmx-with-solr.adoc | 70 +-
solr/solr-ref-guide/src/v2-api.adoc | 5 -
solr/solr-ref-guide/src/working-with-dates.adoc | 14 +-
.../io/eval/BetaDistributionEvaluator.java | 48 +
.../io/eval/ConstantDistributionEvaluator.java | 44 +
.../eval/EnumeratedDistributionEvaluator.java | 19 +-
.../solrj/io/eval/FactorialEvaluator.java | 54 +
.../io/eval/GammaDistributionEvaluator.java | 48 +
.../io/eval/LogNormalDistributionEvaluator.java | 48 +
.../client/solrj/io/eval/MeanEvaluator.java | 57 +
.../client/solrj/io/eval/ModeEvaluator.java | 64 +
.../solrj/io/eval/MonteCarloEvaluator.java | 76 ++
.../solrj/io/eval/MovingMedianEvaluator.java | 68 +
.../client/solrj/io/eval/SampleEvaluator.java | 30 +-
.../io/eval/WeibullDistributionEvaluator.java | 48 +
.../io/eval/ZipFDistributionEvaluator.java | 48 +
.../solr/client/solrj/io/stream/LetStream.java | 27 +-
.../solr/client/solrj/io/stream/SolrStream.java | 4 +-
.../solr/client/solrj/io/stream/SqlStream.java | 7 +-
.../io/stream/expr/StreamExpressionParser.java | 29 +
.../solrj/io/stream/expr/StreamFactory.java | 6 +
.../solr/common/cloud/DistributedQueue.java | 41 +
.../solrj/io/stream/StreamExpressionTest.java | 324 ++++-
178 files changed, 9053 insertions(+), 4748 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/core/src/java/org/apache/solr/cloud/Overseer.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/core/src/java/org/apache/solr/cloud/ZkController.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/core/src/java/org/apache/solr/core/CoreContainer.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/solr-ref-guide/src/solrcloud-autoscaling-api.adoc
----------------------------------------------------------------------
[43/50] lucene-solr:feature/autoscaling_72: SOLR-11436: Add polyfit
and polyfitDerivative Stream Evaluators
Posted by ab...@apache.org.
SOLR-11436: Add polyfit and polyfitDerivative Stream Evaluators
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/1782dd9c
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/1782dd9c
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/1782dd9c
Branch: refs/heads/feature/autoscaling_72
Commit: 1782dd9ca901eced0adbe9932cdbc50ad5792663
Parents: b10eb11
Author: Joel Bernstein <jb...@apache.org>
Authored: Wed Oct 4 13:56:46 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Wed Oct 4 14:02:13 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/handler/StreamHandler.java | 2 +
.../io/eval/PolyFitDerivativeEvaluator.java | 104 +++++++++++++++++++
.../client/solrj/io/eval/PolyFitEvaluator.java | 102 ++++++++++++++++++
.../solrj/io/stream/StreamExpressionTest.java | 38 +++++++
4 files changed, 246 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1782dd9c/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index 16a31fb..69017ac 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -308,6 +308,8 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
.withFunctionName("zipFDistribution", ZipFDistributionEvaluator.class)
.withFunctionName("gammaDistribution", GammaDistributionEvaluator.class)
.withFunctionName("betaDistribution", BetaDistributionEvaluator.class)
+ .withFunctionName("polyfit", PolyFitEvaluator.class)
+ .withFunctionName("polyfitDerivative", PolyFitDerivativeEvaluator.class)
// Boolean Stream Evaluators
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1782dd9c/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/PolyFitDerivativeEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/PolyFitDerivativeEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/PolyFitDerivativeEvaluator.java
new file mode 100644
index 0000000..1aacb9e
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/PolyFitDerivativeEvaluator.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.ArrayList;
+
+import org.apache.commons.math3.analysis.UnivariateFunction;
+import org.apache.commons.math3.analysis.polynomials.PolynomialFunction;
+import org.apache.commons.math3.fitting.PolynomialCurveFitter;
+
+import org.apache.commons.math3.fitting.WeightedObservedPoints;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+
+public class PolyFitDerivativeEvaluator extends RecursiveNumericEvaluator implements ManyValueWorker {
+ protected static final long serialVersionUID = 1L;
+
+ public PolyFitDerivativeEvaluator(StreamExpression expression, StreamFactory factory) throws IOException{
+ super(expression, factory);
+ }
+
+ @Override
+ public Object doWork(Object... objects) throws IOException{
+
+ if(objects.length > 3) {
+ throw new IOException("polyfitDerivative function takes a maximum of 3 arguments.");
+ }
+
+ Object first = objects[0];
+
+ double[] x = null;
+ double[] y = null;
+ int degree = 3;
+
+ if(objects.length == 1) {
+ //Only the y values passed
+
+ y = ((List) first).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ x = new double[y.length];
+ for(int i=0; i<y.length; i++) {
+ x[i] = i;
+ }
+
+ } else if(objects.length == 3) {
+ // x, y and degree passed
+
+ Object second = objects[1];
+ x = ((List) first).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ y = ((List) second).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ degree = ((Number)objects[2]).intValue();
+ } else if(objects.length == 2) {
+ if(objects[1] instanceof List) {
+ // x and y passed
+ Object second = objects[1];
+ x = ((List) first).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ y = ((List) second).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ } else {
+ // y and degree passed
+ y = ((List) first).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ x = new double[y.length];
+ for(int i=0; i<y.length; i++) {
+ x[i] = i;
+ }
+
+ degree = ((Number)objects[1]).intValue();
+ }
+ }
+
+ PolynomialCurveFitter curveFitter = PolynomialCurveFitter.create(degree);
+ WeightedObservedPoints points = new WeightedObservedPoints();
+ for(int i=0; i<x.length; i++) {
+ points.add(x[i], y[i]);
+ }
+
+ double[] coef = curveFitter.fit(points.toList());
+ PolynomialFunction pf = new PolynomialFunction(coef);
+ UnivariateFunction univariateFunction = pf.derivative();
+
+ List list = new ArrayList();
+ for(double xvalue : x) {
+ double yvalue= univariateFunction.value(xvalue);
+ list.add(yvalue);
+ }
+
+ return list;
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1782dd9c/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/PolyFitEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/PolyFitEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/PolyFitEvaluator.java
new file mode 100644
index 0000000..2c0bf2a
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/PolyFitEvaluator.java
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.ArrayList;
+
+import org.apache.commons.math3.analysis.polynomials.PolynomialFunction;
+import org.apache.commons.math3.fitting.PolynomialCurveFitter;
+
+import org.apache.commons.math3.fitting.WeightedObservedPoints;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+
+public class PolyFitEvaluator extends RecursiveNumericEvaluator implements ManyValueWorker {
+ protected static final long serialVersionUID = 1L;
+
+ public PolyFitEvaluator(StreamExpression expression, StreamFactory factory) throws IOException{
+ super(expression, factory);
+ }
+
+ @Override
+ public Object doWork(Object... objects) throws IOException{
+
+ if(objects.length > 3) {
+ throw new IOException("polyfit function takes a maximum of 3 arguments.");
+ }
+
+ Object first = objects[0];
+
+ double[] x = null;
+ double[] y = null;
+ int degree = 3;
+
+ if(objects.length == 1) {
+ //Only the y values passed
+
+ y = ((List) first).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ x = new double[y.length];
+ for(int i=0; i<y.length; i++) {
+ x[i] = i;
+ }
+
+ } else if(objects.length == 3) {
+ // x, y and degree passed
+
+ Object second = objects[1];
+ x = ((List) first).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ y = ((List) second).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ degree = ((Number)objects[2]).intValue();
+ } else if(objects.length == 2) {
+ if(objects[1] instanceof List) {
+ // x and y passed
+ Object second = objects[1];
+ x = ((List) first).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ y = ((List) second).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ } else {
+ // y and degree passed
+ y = ((List) first).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ x = new double[y.length];
+ for(int i=0; i<y.length; i++) {
+ x[i] = i;
+ }
+
+ degree = ((Number)objects[1]).intValue();
+ }
+ }
+
+ PolynomialCurveFitter curveFitter = PolynomialCurveFitter.create(degree);
+ WeightedObservedPoints points = new WeightedObservedPoints();
+ for(int i=0; i<x.length; i++) {
+ points.add(x[i], y[i]);
+ }
+
+ double[] coef = curveFitter.fit(points.toList());
+ PolynomialFunction pf = new PolynomialFunction(coef);
+
+ List list = new ArrayList();
+ for(double xvalue : x) {
+ double yvalue= pf.value(xvalue);
+ list.add(yvalue);
+ }
+
+ return list;
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1782dd9c/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 9e4c6c3..c73adf6 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6541,6 +6541,44 @@ public class StreamExpressionTest extends SolrCloudTestCase {
assertTrue(out.get(5).intValue() == -6);
}
+ @Test
+ public void testPolyfit() throws Exception {
+ String cexpr = "let(echo=true," +
+ " a=array(0,1,2,3,4,5,6,7)," +
+ " fit=polyfit(a, 1)," +
+ " deriv=polyfitDerivative(a, 1))";
+ ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ TupleStream solrStream = new SolrStream(url, paramsLoc);
+ StreamContext context = new StreamContext();
+ solrStream.setStreamContext(context);
+ List<Tuple> tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ List<Number> out = (List<Number>)tuples.get(0).get("fit");
+ assertTrue(out.size() == 8);
+ assertTrue(out.get(0).intValue() == 0);
+ assertTrue(out.get(1).intValue() == 1);
+ assertTrue(out.get(2).intValue() == 2);
+ assertTrue(out.get(3).intValue() == 3);
+ assertTrue(out.get(4).intValue() == 4);
+ assertTrue(out.get(5).intValue() == 5);
+ assertTrue(out.get(6).intValue() == 6);
+ assertTrue(out.get(7).intValue() == 7);
+
+ out = (List<Number>)tuples.get(0).get("deriv");
+ assertTrue(out.size() == 8);
+ assertTrue(out.get(0).intValue() == 1);
+ assertTrue(out.get(1).intValue() == 1);
+ assertTrue(out.get(2).intValue() == 1);
+ assertTrue(out.get(3).intValue() == 1);
+ assertTrue(out.get(4).intValue() == 1);
+ assertTrue(out.get(5).intValue() == 1);
+ assertTrue(out.get(6).intValue() == 1);
+ assertTrue(out.get(7).intValue() == 1);
+ }
+
@Test
public void testAnova() throws Exception {
[09/50] lucene-solr:feature/autoscaling_72: addBackcompatIndexes.py:
Create 'sorted' indexes;
on major releases, produce moreterms, dvupdates, and empty indexes. Add 7.0.0
indexes.
Posted by ab...@apache.org.
addBackcompatIndexes.py: Create 'sorted' indexes; on major releases, produce moreterms, dvupdates, and empty indexes. Add 7.0.0 indexes.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5afd41de
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5afd41de
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5afd41de
Branch: refs/heads/feature/autoscaling_72
Commit: 5afd41de114ff62a755634c4a49091f88bde22cd
Parents: dc47c6e
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 20:44:01 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 20:44:01 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/addBackcompatIndexes.py | 67 ++++++++++++++-----
.../index/TestBackwardsCompatibility.java | 38 ++++++++---
.../org/apache/lucene/index/dvupdates.7.0.0.zip | Bin 0 -> 13401 bytes
.../org/apache/lucene/index/empty.7.0.0.zip | Bin 0 -> 375 bytes
.../org/apache/lucene/index/index.7.0.0-cfs.zip | Bin 0 -> 15594 bytes
.../apache/lucene/index/index.7.0.0-nocfs.zip | Bin 0 -> 15586 bytes
.../org/apache/lucene/index/moreterms.7.0.0.zip | Bin 0 -> 175822 bytes
.../org/apache/lucene/index/sorted.7.0.0.zip | Bin 0 -> 74558 bytes
8 files changed, 79 insertions(+), 26 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/dev-tools/scripts/addBackcompatIndexes.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py
index 543fad6..1cada01 100644
--- a/dev-tools/scripts/addBackcompatIndexes.py
+++ b/dev-tools/scripts/addBackcompatIndexes.py
@@ -31,12 +31,24 @@ import re
import shutil
def create_and_add_index(source, indextype, index_version, current_version, temp_dir):
+ if not current_version.is_back_compat_with(index_version):
+ prefix = 'unsupported'
+ else:
+ prefix = {
+ 'cfs': 'index',
+ 'nocfs': 'index',
+ 'sorted': 'sorted',
+ 'moreterms': 'moreterms',
+ 'dvupdates': 'dvupdates',
+ 'emptyIndex': 'empty'
+ }[indextype]
if indextype in ('cfs', 'nocfs'):
dirname = 'index.%s' % indextype
+ filename = '%s.%s-%s.zip' % (prefix, index_version, indextype)
else:
dirname = indextype
- prefix = 'index' if current_version.is_back_compat_with(index_version) else 'unsupported'
- filename = '%s.%s-%s.zip' % (prefix, index_version, indextype)
+ filename = '%s.%s.zip' % (prefix, index_version)
+
print(' creating %s...' % filename, end='', flush=True)
module = 'backward-codecs'
index_dir = os.path.join('lucene', module, 'src/test/org/apache/lucene/index')
@@ -47,7 +59,11 @@ def create_and_add_index(source, indextype, index_version, current_version, temp
test = {
'cfs': 'testCreateCFS',
- 'nocfs': 'testCreateNoCFS'
+ 'nocfs': 'testCreateNoCFS',
+ 'sorted': 'testCreateSortedIndex',
+ 'moreterms': 'testCreateMoreTermsIndex',
+ 'dvupdates': 'testCreateIndexWithDocValuesUpdates',
+ 'emptyIndex': 'testCreateEmptyIndex'
}[indextype]
ant_args = ' '.join([
'-Dtests.bwcdir=%s' % temp_dir,
@@ -83,16 +99,22 @@ def create_and_add_index(source, indextype, index_version, current_version, temp
print('done')
def update_backcompat_tests(types, index_version, current_version):
- print(' adding new indexes to backcompat tests...', end='', flush=True)
+ print(' adding new indexes %s to backcompat tests...' % types, end='', flush=True)
module = 'lucene/backward-codecs'
filename = '%s/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java' % module
- matcher = re.compile(r'final static String\[\] oldNames = {|};' if current_version.is_back_compat_with(index_version)
- else r'final String\[\] unsupportedNames = {|};')
+ if not current_version.is_back_compat_with(index_version):
+ matcher = re.compile(r'final String\[\] unsupportedNames = {|};'),
+ elif 'sorted' in types:
+ matcher = re.compile(r'final static String\[\] oldSortedNames = {|};')
+ else:
+ matcher = re.compile(r'final static String\[\] oldNames = {|};')
+
+ strip_dash_suffix_re = re.compile(r'-.*')
def find_version(x):
x = x.strip()
- end = x.index("-")
- return scriptutil.Version.parse(x[1:end])
+ x = re.sub(strip_dash_suffix_re, '', x) # remove the -suffix if any
+ return scriptutil.Version.parse(x)
class Edit(object):
start = None
@@ -100,10 +122,12 @@ def update_backcompat_tests(types, index_version, current_version):
if self.start:
# find where this version should exist
i = len(buffer) - 1
- v = find_version(buffer[i])
- while i >= self.start and v.on_or_after(index_version):
- i -= 1
+ previous_version_exists = '};' not in line # Version list closure => there are no versions
+ if previous_version_exists: # Only look if there is a version here
v = find_version(buffer[i])
+ while i >= self.start and v.on_or_after(index_version):
+ i -= 1
+ v = find_version(buffer[i])
i += 1 # readjust since we skipped past by 1
# unfortunately python doesn't have a range remove from list...
@@ -111,14 +135,20 @@ def update_backcompat_tests(types, index_version, current_version):
while i < len(buffer) and index_version.on_or_after(find_version(buffer[i])):
buffer.pop(i)
- if i == len(buffer) and not buffer[-1].strip().endswith(","):
+ if i == len(buffer) and previous_version_exists and not buffer[-1].strip().endswith(","):
# add comma
buffer[-1] = buffer[-1].rstrip() + ",\n"
- last = buffer[-1]
- spaces = ' ' * (len(last) - len(last.lstrip()))
+ if previous_version_exists:
+ last = buffer[-1]
+ spaces = ' ' * (len(last) - len(last.lstrip()))
+ else:
+ spaces = ' '
for (j, t) in enumerate(types):
- newline = spaces + ('"%s-%s"' % (index_version, t))
+ if t == 'sorted':
+ newline = spaces + ('"sorted.%s"') % index_version
+ else:
+ newline = spaces + ('"%s-%s"' % (index_version, t))
if j < len(types) - 1 or i < len(buffer):
newline += ','
buffer.insert(i, newline + '\n')
@@ -215,9 +245,16 @@ def main():
current_version = scriptutil.Version.parse(scriptutil.find_current_version())
create_and_add_index(source, 'cfs', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'nocfs', c.version, current_version, c.temp_dir)
+ create_and_add_index(source, 'sorted', c.version, current_version, c.temp_dir)
+ if c.version.minor == 0 and c.version.bugfix == 0:
+ create_and_add_index(source, 'moreterms', c.version, current_version, c.temp_dir)
+ create_and_add_index(source, 'dvupdates', c.version, current_version, c.temp_dir)
+ create_and_add_index(source, 'emptyIndex', c.version, current_version, c.temp_dir)
+ print ('\nMANUAL UPDATE REQUIRED: edit TestBackwardsCompatibility to enable moreterms, dvupdates, and empty index testing')
print('\nAdding backwards compatibility tests')
update_backcompat_tests(['cfs', 'nocfs'], c.version, current_version)
+ update_backcompat_tests(['sorted'], c.version, current_version)
print('\nTesting changes')
check_backcompat_tests()
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index c202973..88900d4 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -156,7 +156,7 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
IndexWriterConfig conf = new IndexWriterConfig(analyzer)
.setMergePolicy(mp).setUseCompoundFile(false);
IndexWriter writer = new IndexWriter(dir, conf);
- LineFileDocs docs = new LineFileDocs(null);
+ LineFileDocs docs = new LineFileDocs(random());
for(int i=0;i<50;i++) {
writer.addDocument(docs.nextDoc());
}
@@ -275,12 +275,33 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
dir.close();
}
+ public void testCreateEmptyIndex() throws Exception {
+ Path indexDir = getIndexDir().resolve("emptyIndex");
+ Files.deleteIfExists(indexDir);
+ IndexWriterConfig conf = new IndexWriterConfig(new MockAnalyzer(random()))
+ .setUseCompoundFile(false).setMergePolicy(NoMergePolicy.INSTANCE);
+ try (Directory dir = newFSDirectory(indexDir);
+ IndexWriter writer = new IndexWriter(dir, conf)) {
+ writer.flush();
+ }
+ }
+
final static String[] oldNames = {
+ "7.0.0-cfs",
+ "7.0.0-nocfs"
};
public static String[] getOldNames() {
return oldNames;
}
+
+ final static String[] oldSortedNames = {
+ "sorted.7.0.0"
+ };
+
+ public static String[] getOldSortedNames() {
+ return oldSortedNames;
+ }
final String[] unsupportedNames = {
"1.9.0-cfs",
@@ -765,7 +786,7 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
Directory targetDir2 = newDirectory();
IndexWriter w = new IndexWriter(targetDir2, newIndexWriterConfig(new MockAnalyzer(random())));
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> TestUtil.addIndexesSlowly(w, reader));
- assertEquals(e.getMessage(), "Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7");
+ assertEquals(e.getMessage(), "Cannot merge a segment that has been created with major version 7 into this index which has been created by major version 8");
w.close();
targetDir2.close();
@@ -1449,14 +1470,13 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
public static final String emptyIndex = "empty.7.0.0.zip";
public void testUpgradeEmptyOldIndex() throws Exception {
- assumeTrue("Reenable when 7.0 is released", false);
Path oldIndexDir = createTempDir("emptyIndex");
TestUtil.unzip(getDataInputStream(emptyIndex), oldIndexDir);
Directory dir = newFSDirectory(oldIndexDir);
newIndexUpgrader(dir).upgrade();
- checkAllSegmentsUpgraded(dir, 6);
+ checkAllSegmentsUpgraded(dir, 7);
dir.close();
}
@@ -1464,7 +1484,6 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
public static final String moreTermsIndex = "moreterms.7.0.0.zip";
public void testMoreTerms() throws Exception {
- assumeTrue("Reenable when 7.0 is released", false);
Path oldIndexDir = createTempDir("moreterms");
TestUtil.unzip(getDataInputStream(moreTermsIndex), oldIndexDir);
Directory dir = newFSDirectory(oldIndexDir);
@@ -1509,7 +1528,6 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
}
public void testDocValuesUpdates() throws Exception {
- assumeTrue("Reenable when 7.0 is released", false);
Path oldIndexDir = createTempDir("dvupdates");
TestUtil.unzip(getDataInputStream(dvUpdatesIndex), oldIndexDir);
Directory dir = newFSDirectory(oldIndexDir);
@@ -1572,12 +1590,10 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
}
public void testSortedIndex() throws Exception {
- assumeTrue("Reenable when 7.0 is released", false);
- String[] versions = new String[] {};
- for(String version : versions) {
+ for(String name : oldSortedNames) {
Path path = createTempDir("sorted");
- InputStream resource = TestBackwardsCompatibility.class.getResourceAsStream("sorted." + version + ".zip");
- assertNotNull("Sorted index index " + version + " not found", resource);
+ InputStream resource = TestBackwardsCompatibility.class.getResourceAsStream(name + ".zip");
+ assertNotNull("Sorted index index " + name + " not found", resource);
TestUtil.unzip(resource, path);
// TODO: more tests
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip
new file mode 100644
index 0000000..6c94eae
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip
new file mode 100644
index 0000000..d8dac24
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip
new file mode 100644
index 0000000..1419931
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip
new file mode 100644
index 0000000..83c3242
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip
new file mode 100644
index 0000000..5366c09
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip
new file mode 100644
index 0000000..13948dc
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip differ
[45/50] lucene-solr:feature/autoscaling_72: SOLR-11285: Fix a bug in
Policy modifier methods.
Posted by ab...@apache.org.
SOLR-11285: Fix a bug in Policy modifier methods.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5c62fb56
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5c62fb56
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5c62fb56
Branch: refs/heads/feature/autoscaling_72
Commit: 5c62fb56f7a7a86214cbff7e171f461603b9b0fe
Parents: 070d6d3
Author: Andrzej Bialecki <ab...@apache.org>
Authored: Wed Oct 4 18:53:32 2017 +0200
Committer: Andrzej Bialecki <ab...@apache.org>
Committed: Wed Oct 4 20:53:20 2017 +0200
----------------------------------------------------------------------
.../client/solrj/cloud/autoscaling/Policy.java | 29 +++++++++++++++-----
1 file changed, 22 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5c62fb56/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
index 12024e4..78961d6 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
@@ -113,29 +113,44 @@ public class Policy implements MapWriter {
}
- private Policy(Map<String, List<Clause>> policies, List<Clause> clusterPolicy, List<Preference> clusterPreferences,
- List<String> params) {
+ private Policy(Map<String, List<Clause>> policies, List<Clause> clusterPolicy, List<Preference> clusterPreferences) {
this.policies = policies != null ? Collections.unmodifiableMap(policies) : Collections.emptyMap();
this.clusterPolicy = clusterPolicy != null ? Collections.unmodifiableList(clusterPolicy) : Collections.emptyList();
this.clusterPreferences = clusterPreferences != null ? Collections.unmodifiableList(clusterPreferences) :
Collections.singletonList(DEFAULT_PREFERENCE);
- this.params = params != null ? Collections.unmodifiableList(params) : Collections.emptyList();
+ this.params = Collections.unmodifiableList(buildParams(this.clusterPreferences, this.clusterPolicy, this.policies));
+ }
+
+ private List<String> buildParams(List<Preference> preferences, List<Clause> policy, Map<String, List<Clause>> policies) {
+ final SortedSet<String> paramsOfInterest = new TreeSet<>();
+ preferences.forEach(p -> {
+ if (paramsOfInterest.contains(p.name.name())) {
+ throw new RuntimeException(p.name + " is repeated");
+ }
+ paramsOfInterest.add(p.name.toString());
+ });
+ List<String> newParams = new ArrayList<>(paramsOfInterest);
+ policy.forEach(c -> {
+ c.addTags(newParams);
+ });
+ policies.values().forEach(clauses -> clauses.forEach(c -> c.addTags(newParams)));
+ return newParams;
}
public Policy withPolicies(Map<String, List<Clause>> policies) {
- return new Policy(policies, clusterPolicy, clusterPreferences, params);
+ return new Policy(policies, clusterPolicy, clusterPreferences);
}
public Policy withClusterPreferences(List<Preference> clusterPreferences) {
- return new Policy(policies, clusterPolicy, clusterPreferences, params);
+ return new Policy(policies, clusterPolicy, clusterPreferences);
}
public Policy withClusterPolicy(List<Clause> clusterPolicy) {
- return new Policy(policies, clusterPolicy, clusterPreferences, params);
+ return new Policy(policies, clusterPolicy, clusterPreferences);
}
public Policy withParams(List<String> params) {
- return new Policy(policies, clusterPolicy, clusterPreferences, params);
+ return new Policy(policies, clusterPolicy, clusterPreferences);
}
public List<Clause> getClusterPolicy() {
[25/50] lucene-solr:feature/autoscaling_72: Merge branch 'master' of
https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Posted by ab...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2c873cd8
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/2c873cd8
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/2c873cd8
Branch: refs/heads/feature/autoscaling_72
Commit: 2c873cd8c6375c6f54a0309cd45e6dfe009bc7c1
Parents: 309ed64
Author: Noble Paul <no...@apache.org>
Authored: Wed Oct 4 00:08:14 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Wed Oct 4 00:08:14 2017 +1030
----------------------------------------------------------------------
.../org/apache/solr/cloud/autoscaling/TriggerEventQueue.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2c873cd8/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
index 99f641c..36a6d5f 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
@@ -4,8 +4,8 @@ import java.lang.invoke.MethodHandles;
import java.util.Map;
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventType;
-import org.apache.solr.cloud.DistributedQueue;
import org.apache.solr.cloud.Overseer;
+import org.apache.solr.cloud.ZkDistributedQueue;
import org.apache.solr.common.cloud.SolrZkClient;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.util.Utils;
@@ -17,7 +17,7 @@ import org.slf4j.LoggerFactory;
/**
*
*/
-public class TriggerEventQueue extends DistributedQueue {
+public class TriggerEventQueue extends ZkDistributedQueue {
private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
public static final String ENQUEUE_TIME = "_enqueue_time_";
[22/50] lucene-solr:feature/autoscaling_72: SOLR-10285: Skip LEADER
messages when there are leader only shards
Posted by ab...@apache.org.
SOLR-10285: Skip LEADER messages when there are leader only shards
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/fd2b4f3f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/fd2b4f3f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/fd2b4f3f
Branch: refs/heads/feature/autoscaling_72
Commit: fd2b4f3f868276a3b3e3b0b6845d1a7309c9cad0
Parents: 5b3a515
Author: Cao Manh Dat <da...@apache.org>
Authored: Tue Oct 3 09:00:02 2017 +0700
Committer: Cao Manh Dat <da...@apache.org>
Committed: Tue Oct 3 09:00:02 2017 +0700
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +
.../org/apache/solr/cloud/ElectionContext.java | 38 ++++++---
.../solr/cloud/TestShardsWithSingleReplica.java | 84 ++++++++++++++++++++
3 files changed, 112 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fd2b4f3f/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 90ada02..177cdf7 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -171,6 +171,8 @@ Optimizations
* SOLR-8344: Decide default when requested fields are both column and row stored. (Cao Manh Dat, David Smiley)
+* SOLR-10285: Skip LEADER messages when there are leader only shards (Cao Manh Dat, Joshua Humphries)
+
Other Changes
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fd2b4f3f/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java b/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
index 491ae00..ba88e75 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
@@ -115,6 +115,7 @@ class ShardLeaderElectionContextBase extends ElectionContext {
protected String shardId;
protected String collection;
protected LeaderElector leaderElector;
+ protected ZkStateReader zkStateReader;
private Integer leaderZkNodeParentVersion;
// Prevents a race between cancelling and becoming leader.
@@ -128,6 +129,7 @@ class ShardLeaderElectionContextBase extends ElectionContext {
collection, shardId), props, zkStateReader.getZkClient());
this.leaderElector = leaderElector;
this.zkClient = zkStateReader.getZkClient();
+ this.zkStateReader = zkStateReader;
this.shardId = shardId;
this.collection = collection;
}
@@ -216,14 +218,24 @@ class ShardLeaderElectionContextBase extends ElectionContext {
}
assert shardId != null;
- ZkNodeProps m = ZkNodeProps.fromKeyVals(Overseer.QUEUE_OPERATION,
- OverseerAction.LEADER.toLower(), ZkStateReader.SHARD_ID_PROP, shardId,
- ZkStateReader.COLLECTION_PROP, collection, ZkStateReader.BASE_URL_PROP,
- leaderProps.getProperties().get(ZkStateReader.BASE_URL_PROP),
- ZkStateReader.CORE_NAME_PROP,
- leaderProps.getProperties().get(ZkStateReader.CORE_NAME_PROP),
- ZkStateReader.STATE_PROP, Replica.State.ACTIVE.toString());
- Overseer.getStateUpdateQueue(zkClient).offer(Utils.toJSON(m));
+ boolean isAlreadyLeader = false;
+ if (zkStateReader.getClusterState() != null &&
+ zkStateReader.getClusterState().getCollection(collection).getSlice(shardId).getReplicas().size() < 2) {
+ Replica leader = zkStateReader.getLeader(collection, shardId);
+ if (leader != null
+ && leader.getBaseUrl().equals(leaderProps.get(ZkStateReader.BASE_URL_PROP))
+ && leader.getCoreName().equals(leaderProps.get(ZkStateReader.CORE_NAME_PROP))) {
+ isAlreadyLeader = true;
+ }
+ }
+ if (!isAlreadyLeader) {
+ ZkNodeProps m = ZkNodeProps.fromKeyVals(Overseer.QUEUE_OPERATION, OverseerAction.LEADER.toLower(),
+ ZkStateReader.SHARD_ID_PROP, shardId,
+ ZkStateReader.COLLECTION_PROP, collection,
+ ZkStateReader.BASE_URL_PROP, leaderProps.get(ZkStateReader.BASE_URL_PROP),
+ ZkStateReader.CORE_NAME_PROP, leaderProps.get(ZkStateReader.CORE_NAME_PROP));
+ Overseer.getStateUpdateQueue(zkClient).offer(Utils.toJSON(m));
+ }
}
public LeaderElector getLeaderElector() {
@@ -309,10 +321,12 @@ final class ShardLeaderElectionContext extends ShardLeaderElectionContextBase {
int leaderVoteWait = cc.getZkController().getLeaderVoteWait();
log.debug("Running the leader process for shard={} and weAreReplacement={} and leaderVoteWait={}", shardId, weAreReplacement, leaderVoteWait);
- // clear the leader in clusterstate
- ZkNodeProps m = new ZkNodeProps(Overseer.QUEUE_OPERATION, OverseerAction.LEADER.toLower(),
- ZkStateReader.SHARD_ID_PROP, shardId, ZkStateReader.COLLECTION_PROP, collection);
- Overseer.getStateUpdateQueue(zkClient).offer(Utils.toJSON(m));
+ if (zkController.getClusterState().getCollection(collection).getSlice(shardId).getReplicas().size() > 1) {
+ // Clear the leader in clusterstate. We only need to worry about this if there is actually more than one replica.
+ ZkNodeProps m = new ZkNodeProps(Overseer.QUEUE_OPERATION, OverseerAction.LEADER.toLower(),
+ ZkStateReader.SHARD_ID_PROP, shardId, ZkStateReader.COLLECTION_PROP, collection);
+ Overseer.getStateUpdateQueue(zkClient).offer(Utils.toJSON(m));
+ }
boolean allReplicasInLine = false;
if (!weAreReplacement) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fd2b4f3f/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java b/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
new file mode 100644
index 0000000..a4c51ba
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cloud;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.embedded.JettySolrRunner;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.response.CollectionAdminResponse;
+import org.junit.BeforeClass;
+
+public class TestShardsWithSingleReplica extends SolrCloudTestCase {
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ System.setProperty("solr.directoryFactory", "solr.StandardDirectoryFactory");
+ System.setProperty("solr.ulog.numRecordsToKeep", "1000");
+
+ configureCluster(3)
+ .addConfig("config", TEST_PATH().resolve("configsets").resolve("cloud-minimal").resolve("conf"))
+ .configure();
+ }
+
+ public void testSkipLeaderOperations() throws Exception {
+ String overseerLeader = getOverseerLeader();
+ List<JettySolrRunner> notOverseerNodes = cluster.getJettySolrRunners()
+ .stream()
+ .filter(solrRunner -> !solrRunner.getNodeName().equals(overseerLeader))
+ .collect(Collectors.toList());
+ String collection = "collection1";
+ CollectionAdminRequest
+ .createCollection(collection, 2, 1)
+ .setCreateNodeSet(notOverseerNodes
+ .stream()
+ .map(JettySolrRunner::getNodeName)
+ .collect(Collectors.joining(","))
+ )
+ .process(cluster.getSolrClient());
+
+ for (JettySolrRunner solrRunner : notOverseerNodes) {
+ cluster.stopJettySolrRunner(solrRunner);
+ }
+ waitForState("Expected empty liveNodes", collection,
+ (liveNodes, collectionState) -> liveNodes.size() == 1);
+
+ CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ for (JettySolrRunner solrRunner : notOverseerNodes) {
+ cluster.startJettySolrRunner(solrRunner);
+ }
+
+ waitForState("Expected 2x1 for collection: " + collection, collection,
+ clusterShape(2, 1));
+ CollectionAdminResponse resp2 = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ assertEquals(getNumLeaderOpeations(resp), getNumLeaderOpeations(resp2));
+ }
+
+ private int getNumLeaderOpeations(CollectionAdminResponse resp) {
+ return (int) resp.getResponse().findRecursive("overseer_operations", "leader", "requests");
+ }
+
+ private String getOverseerLeader() throws IOException, SolrServerException {
+ CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ return (String) resp.getResponse().get("leader");
+ }
+
+}
[05/50] lucene-solr:feature/autoscaling_72: SOLR-11420: sql Streaming
Expression should use the default collection if none is provided
Posted by ab...@apache.org.
SOLR-11420: sql Streaming Expression should use the default collection if none is provided
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/71eb59e0
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/71eb59e0
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/71eb59e0
Branch: refs/heads/feature/autoscaling_72
Commit: 71eb59e0435f744db2d5cf8c58b1fca70af9b824
Parents: 633af79
Author: Joel Bernstein <jb...@apache.org>
Authored: Fri Sep 29 10:36:26 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Fri Sep 29 10:36:26 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/client/solrj/io/stream/SqlStream.java | 7 +------
.../solr/client/solrj/io/stream/expr/StreamFactory.java | 6 ++++++
.../solr/client/solrj/io/stream/StreamExpressionTest.java | 10 ++++++++++
3 files changed, 17 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/71eb59e0/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java
index d2296b7..93345e7 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java
@@ -73,12 +73,7 @@ public class SqlStream extends TupleStream implements Expressible {
// Collection Name
if(null == collectionName){
- throw new IOException(String.format(Locale.ROOT,"invalid expression %s - collectionName expected as first operand",expression));
- }
-
- // Validate there are no unknown parameters - zkHost and alias are namedParameter so we don't need to count it twice
- if(expression.getParameters().size() != 1 + namedParams.size()){
- throw new IOException(String.format(Locale.ROOT,"invalid expression %s - unknown operands found",expression));
+ collectionName = factory.getDefaultCollection();
}
// Named parameters - passed directly to solr as solrparams
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/71eb59e0/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java
index f4c9b58..321545b 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java
@@ -46,6 +46,7 @@ public class StreamFactory implements Serializable {
private transient HashMap<String,String> collectionZkHosts;
private transient HashMap<String,Class<? extends Expressible>> functionNames;
private transient String defaultZkHost;
+ private transient String defaultCollection;
public StreamFactory(){
collectionZkHosts = new HashMap<>();
@@ -54,9 +55,14 @@ public class StreamFactory implements Serializable {
public StreamFactory withCollectionZkHost(String collectionName, String zkHost){
this.collectionZkHosts.put(collectionName, zkHost);
+ this.defaultCollection = collectionName;
return this;
}
+ public String getDefaultCollection() {
+ return defaultCollection;
+ }
+
public StreamFactory withDefaultZkHost(String zkHost) {
this.defaultZkHost = zkHost;
return this;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/71eb59e0/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 949d224..d84120f 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -279,6 +279,16 @@ public class StreamExpressionTest extends SolrCloudTestCase {
assert (tuples.size() == 5);
assertOrder(tuples, 0, 1, 2, 3, 4);
+ //Test with using the default collection
+ solrParams = new ModifiableSolrParams();
+ solrParams.add("qt", "/stream");
+ solrParams.add("expr", "sql(stmt=\"select id from collection1 order by a_i asc\")");
+ solrStream = new SolrStream(shardUrls.get(0), solrParams);
+ solrStream.setStreamContext(streamContext);
+ tuples = getTuples(solrStream);
+ assert (tuples.size() == 5);
+ assertOrder(tuples, 0, 1, 2, 3, 4);
+
} finally {
solrClientCache.close();
}
[34/50] lucene-solr:feature/autoscaling_72: LUCENE-7983:
IndexWriter.IndexReaderWarmer is now a functional interface instead of an
abstract class with a single method.
Posted by ab...@apache.org.
LUCENE-7983: IndexWriter.IndexReaderWarmer is now a functional interface instead of an abstract class with a single method.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/46cd679e
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/46cd679e
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/46cd679e
Branch: refs/heads/feature/autoscaling_72
Commit: 46cd679e91f0ba03d1965dff45b00f7a54a5351e
Parents: b09c99f
Author: Dawid Weiss <dw...@apache.org>
Authored: Wed Oct 4 10:54:13 2017 +0200
Committer: Dawid Weiss <dw...@apache.org>
Committed: Wed Oct 4 10:59:16 2017 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 4 ++
.../org/apache/lucene/index/IndexWriter.java | 23 +++++-----
.../lucene/index/SimpleMergedSegmentWarmer.java | 6 +--
.../index/TestDemoParallelLeafReader.java | 19 ++++-----
.../lucene/index/TestIndexWriterReader.java | 44 +++++++-------------
.../nrt/PreCopyMergedSegmentWarmer.java | 4 +-
.../ThreadedIndexingAndSearchingTestCase.java | 42 +++++++++----------
7 files changed, 62 insertions(+), 80 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46cd679e/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index d82889b..f3f04ed 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -106,6 +106,10 @@ Other
* LUCENE-7978: Add some more documentation about setting up build
environment. (Anton R. Yuste via Uwe Schindler)
+* LUCENE-7983: IndexWriter.IndexReaderWarmer is now a functional interface
+ instead of an abstract class with a single method (Dawid Weiss)
+
+
======================= Lucene 7.0.1 =======================
(No Changes)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46cd679e/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
index 05608b4..1e6d792 100644
--- a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
@@ -4804,19 +4804,16 @@ public class IndexWriter implements Closeable, TwoPhaseCommit, Accountable {
*
* @lucene.experimental
*
- * <p><b>NOTE</b>: warm is called before any deletes have
- * been carried over to the merged segment. */
- public static abstract class IndexReaderWarmer {
-
- /** Sole constructor. (For invocation by subclass
- * constructors, typically implicit.) */
- protected IndexReaderWarmer() {
- }
-
- /** Invoked on the {@link LeafReader} for the newly
- * merged segment, before that segment is made visible
- * to near-real-time readers. */
- public abstract void warm(LeafReader reader) throws IOException;
+ * <p><b>NOTE</b>: {@link #warm(LeafReader)} is called before any
+ * deletes have been carried over to the merged segment. */
+ @FunctionalInterface
+ public static interface IndexReaderWarmer {
+ /**
+ * Invoked on the {@link LeafReader} for the newly
+ * merged segment, before that segment is made visible
+ * to near-real-time readers.
+ */
+ void warm(LeafReader reader) throws IOException;
}
void tragicEvent(Throwable tragedy, String location) throws IOException {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46cd679e/lucene/core/src/java/org/apache/lucene/index/SimpleMergedSegmentWarmer.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/SimpleMergedSegmentWarmer.java b/lucene/core/src/java/org/apache/lucene/index/SimpleMergedSegmentWarmer.java
index cf092ac..51f0600 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SimpleMergedSegmentWarmer.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SimpleMergedSegmentWarmer.java
@@ -26,9 +26,9 @@ import org.apache.lucene.util.InfoStream;
* A very simple merged segment warmer that just ensures
* data structures are initialized.
*/
-public class SimpleMergedSegmentWarmer extends IndexReaderWarmer {
+public class SimpleMergedSegmentWarmer implements IndexReaderWarmer {
private final InfoStream infoStream;
-
+
/**
* Creates a new SimpleMergedSegmentWarmer
* @param infoStream InfoStream to log statistics about warming.
@@ -53,7 +53,7 @@ public class SimpleMergedSegmentWarmer extends IndexReaderWarmer {
normsCount++;
}
}
-
+
if (info.getDocValuesType() != DocValuesType.NONE) {
switch(info.getDocValuesType()) {
case NUMERIC:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46cd679e/lucene/core/src/test/org/apache/lucene/index/TestDemoParallelLeafReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestDemoParallelLeafReader.java b/lucene/core/src/test/org/apache/lucene/index/TestDemoParallelLeafReader.java
index 2f0121a..5b576d9 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestDemoParallelLeafReader.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestDemoParallelLeafReader.java
@@ -129,17 +129,14 @@ public class TestDemoParallelLeafReader extends LuceneTestCase {
}
w = new IndexWriter(indexDir, iwc);
- w.getConfig().setMergedSegmentWarmer(new IndexWriter.IndexReaderWarmer() {
- @Override
- public void warm(LeafReader reader) throws IOException {
- // This will build the parallel index for the merged segment before the merge becomes visible, so reopen delay is only due to
- // newly flushed segments:
- if (DEBUG) System.out.println(Thread.currentThread().getName() +": TEST: now warm " + reader);
- // TODO: it's not great that we pass false here; it means we close the reader & reopen again for NRT reader; still we did "warm" by
- // building the parallel index, if necessary
- getParallelLeafReader(reader, false, getCurrentSchemaGen());
- }
- });
+ w.getConfig().setMergedSegmentWarmer((reader) -> {
+ // This will build the parallel index for the merged segment before the merge becomes visible, so reopen delay is only due to
+ // newly flushed segments:
+ if (DEBUG) System.out.println(Thread.currentThread().getName() +": TEST: now warm " + reader);
+ // TODO: it's not great that we pass false here; it means we close the reader & reopen again for NRT reader; still we did "warm" by
+ // building the parallel index, if necessary
+ getParallelLeafReader(reader, false, getCurrentSchemaGen());
+ });
// start with empty commit:
w.commit();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46cd679e/lucene/core/src/test/org/apache/lucene/index/TestIndexWriterReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestIndexWriterReader.java b/lucene/core/src/test/org/apache/lucene/index/TestIndexWriterReader.java
index db95f5f..86881ac 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestIndexWriterReader.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestIndexWriterReader.java
@@ -612,24 +612,15 @@ public class TestIndexWriterReader extends LuceneTestCase {
}
}
- private static class MyWarmer extends IndexWriter.IndexReaderWarmer {
- int warmCount;
- @Override
- public void warm(LeafReader reader) throws IOException {
- warmCount++;
- }
- }
-
public void testMergeWarmer() throws Exception {
-
Directory dir1 = getAssertNoDeletesDirectory(newDirectory());
// Enroll warmer
- MyWarmer warmer = new MyWarmer();
+ AtomicInteger warmCount = new AtomicInteger();
IndexWriter writer = new IndexWriter(
dir1,
newIndexWriterConfig(new MockAnalyzer(random()))
.setMaxBufferedDocs(2)
- .setMergedSegmentWarmer(warmer)
+ .setMergedSegmentWarmer((leafReader) -> warmCount.incrementAndGet())
.setMergeScheduler(new ConcurrentMergeScheduler())
.setMergePolicy(newLogMergePolicy())
);
@@ -648,12 +639,12 @@ public class TestIndexWriterReader extends LuceneTestCase {
}
((ConcurrentMergeScheduler) writer.getConfig().getMergeScheduler()).sync();
- assertTrue(warmer.warmCount > 0);
- final int count = warmer.warmCount;
+ assertTrue(warmCount.get() > 0);
+ final int count = warmCount.get();
writer.addDocument(DocHelper.createDocument(17, "test", 4));
writer.forceMerge(1);
- assertTrue(warmer.warmCount > count);
+ assertTrue(warmCount.get() > count);
writer.close();
r1.close();
@@ -969,24 +960,21 @@ public class TestIndexWriterReader extends LuceneTestCase {
final AtomicBoolean didWarm = new AtomicBoolean();
IndexWriter w = new IndexWriter(
dir,
- newIndexWriterConfig(new MockAnalyzer(random())).
- setMaxBufferedDocs(2).
- setReaderPooling(true).
- setMergedSegmentWarmer(new IndexWriter.IndexReaderWarmer() {
- @Override
- public void warm(LeafReader r) throws IOException {
- IndexSearcher s = newSearcher(r);
- TopDocs hits = s.search(new TermQuery(new Term("foo", "bar")), 10);
- assertEquals(20, hits.totalHits);
- didWarm.set(true);
- }
- }).
- setMergePolicy(newLogMergePolicy(10))
+ newIndexWriterConfig(new MockAnalyzer(random()))
+ .setMaxBufferedDocs(2)
+ .setReaderPooling(true)
+ .setMergedSegmentWarmer((r) -> {
+ IndexSearcher s = newSearcher(r);
+ TopDocs hits = s.search(new TermQuery(new Term("foo", "bar")), 10);
+ assertEquals(20, hits.totalHits);
+ didWarm.set(true);
+ })
+ .setMergePolicy(newLogMergePolicy(10))
);
Document doc = new Document();
doc.add(newStringField("foo", "bar", Field.Store.NO));
- for(int i=0;i<20;i++) {
+ for (int i = 0; i < 20; i++) {
w.addDocument(doc);
}
w.waitForMerges();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46cd679e/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/PreCopyMergedSegmentWarmer.java
----------------------------------------------------------------------
diff --git a/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/PreCopyMergedSegmentWarmer.java b/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/PreCopyMergedSegmentWarmer.java
index 0b33f17..15b9474 100644
--- a/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/PreCopyMergedSegmentWarmer.java
+++ b/lucene/replicator/src/java/org/apache/lucene/replicator/nrt/PreCopyMergedSegmentWarmer.java
@@ -34,7 +34,7 @@ import org.apache.lucene.index.SegmentReader;
// TODO: or ... replica node can do merging locally? tricky to keep things in sync, when one node merges more slowly than others...
-class PreCopyMergedSegmentWarmer extends IndexReaderWarmer {
+class PreCopyMergedSegmentWarmer implements IndexReaderWarmer {
private final PrimaryNode primary;
@@ -48,7 +48,7 @@ class PreCopyMergedSegmentWarmer extends IndexReaderWarmer {
final SegmentCommitInfo info = ((SegmentReader) reader).getSegmentInfo();
//System.out.println("TEST: warm merged segment files " + info);
Map<String,FileMetaData> filesMetaData = new HashMap<>();
- for(String fileName : info.files()) {
+ for (String fileName : info.files()) {
FileMetaData metaData = primary.readLocalFileMetaData(fileName);
assert metaData != null;
assert filesMetaData.containsKey(fileName) == false;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46cd679e/lucene/test-framework/src/java/org/apache/lucene/index/ThreadedIndexingAndSearchingTestCase.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/index/ThreadedIndexingAndSearchingTestCase.java b/lucene/test-framework/src/java/org/apache/lucene/index/ThreadedIndexingAndSearchingTestCase.java
index 3d97004..3850429 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/index/ThreadedIndexingAndSearchingTestCase.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/index/ThreadedIndexingAndSearchingTestCase.java
@@ -16,7 +16,6 @@
*/
package org.apache.lucene.index;
-import java.io.IOException;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Collections;
@@ -468,32 +467,29 @@ public abstract class ThreadedIndexingAndSearchingTestCase extends LuceneTestCas
mp.setNoCFSRatio(Math.max(0.25d, mp.getNoCFSRatio()));
}
- conf.setMergedSegmentWarmer(new IndexWriter.IndexReaderWarmer() {
- @Override
- public void warm(LeafReader reader) throws IOException {
- if (VERBOSE) {
- System.out.println("TEST: now warm merged reader=" + reader);
- }
- warmed.put(((SegmentReader) reader).core, Boolean.TRUE);
- final int maxDoc = reader.maxDoc();
- final Bits liveDocs = reader.getLiveDocs();
- int sum = 0;
- final int inc = Math.max(1, maxDoc/50);
- for(int docID=0;docID<maxDoc;docID += inc) {
- if (liveDocs == null || liveDocs.get(docID)) {
- final Document doc = reader.document(docID);
- sum += doc.getFields().size();
- }
+ conf.setMergedSegmentWarmer((reader) -> {
+ if (VERBOSE) {
+ System.out.println("TEST: now warm merged reader=" + reader);
+ }
+ warmed.put(((SegmentReader) reader).core, Boolean.TRUE);
+ final int maxDoc = reader.maxDoc();
+ final Bits liveDocs = reader.getLiveDocs();
+ int sum = 0;
+ final int inc = Math.max(1, maxDoc/50);
+ for(int docID=0;docID<maxDoc;docID += inc) {
+ if (liveDocs == null || liveDocs.get(docID)) {
+ final Document doc = reader.document(docID);
+ sum += doc.getFields().size();
}
+ }
- IndexSearcher searcher = newSearcher(reader, false);
- sum += searcher.search(new TermQuery(new Term("body", "united")), 10).totalHits;
+ IndexSearcher searcher = newSearcher(reader, false);
+ sum += searcher.search(new TermQuery(new Term("body", "united")), 10).totalHits;
- if (VERBOSE) {
- System.out.println("TEST: warm visited " + sum + " fields");
- }
+ if (VERBOSE) {
+ System.out.println("TEST: warm visited " + sum + " fields");
}
- });
+ });
if (VERBOSE) {
conf.setInfoStream(new PrintStreamInfoStream(System.out) {
[42/50] lucene-solr:feature/autoscaling_72: SOLR-11372: addemdum,
refactor refinement randomization to be reproducible when re-using
the same TermFacet instance
Posted by ab...@apache.org.
SOLR-11372: addemdum, refactor refinement randomization to be reproducible when re-using the same TermFacet instance
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/b10eb117
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/b10eb117
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/b10eb117
Branch: refs/heads/feature/autoscaling_72
Commit: b10eb1172a76ad877dece87893fec80895562968
Parents: 4392500
Author: Chris Hostetter <ho...@apache.org>
Authored: Wed Oct 4 09:47:28 2017 -0700
Committer: Chris Hostetter <ho...@apache.org>
Committed: Wed Oct 4 09:47:28 2017 -0700
----------------------------------------------------------------------
.../apache/solr/search/facet/FacetField.java | 3 +-
.../cloud/TestCloudJSONFacetJoinDomain.java | 220 +++++++++++++++----
2 files changed, 176 insertions(+), 47 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b10eb117/solr/core/src/java/org/apache/solr/search/facet/FacetField.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/search/facet/FacetField.java b/solr/core/src/java/org/apache/solr/search/facet/FacetField.java
index f4f9b14..578994f 100644
--- a/solr/core/src/java/org/apache/solr/search/facet/FacetField.java
+++ b/solr/core/src/java/org/apache/solr/search/facet/FacetField.java
@@ -49,6 +49,7 @@ abstract class FacetRequestSorted extends FacetRequest {
public class FacetField extends FacetRequestSorted {
+ public static final int DEFAULT_FACET_LIMIT = 10;
String field;
boolean missing;
boolean allBuckets; // show cumulative stats across all buckets (this can be different than non-bucketed stats across all docs because of multi-valued docs)
@@ -63,7 +64,7 @@ public class FacetField extends FacetRequestSorted {
{
// defaults for FacetRequestSorted
mincount = 1;
- limit = 10;
+ limit = DEFAULT_FACET_LIMIT;
}
public enum FacetMethod {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b10eb117/solr/core/src/test/org/apache/solr/cloud/TestCloudJSONFacetJoinDomain.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestCloudJSONFacetJoinDomain.java b/solr/core/src/test/org/apache/solr/cloud/TestCloudJSONFacetJoinDomain.java
index b104c92..042a9f5 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestCloudJSONFacetJoinDomain.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestCloudJSONFacetJoinDomain.java
@@ -44,6 +44,7 @@ import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
+import org.apache.solr.search.facet.FacetField;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.slf4j.Logger;
@@ -60,15 +61,18 @@ import org.slf4j.LoggerFactory;
* @see TestCloudPivotFacet
*/
public class TestCloudJSONFacetJoinDomain extends SolrCloudTestCase {
-
+
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final String DEBUG_LABEL = MethodHandles.lookup().lookupClass().getName();
private static final String COLLECTION_NAME = DEBUG_LABEL + "_collection";
+ private static final int DEFAULT_LIMIT = FacetField.DEFAULT_FACET_LIMIT;
private static final int MAX_FIELD_NUM = 15;
private static final int UNIQUE_FIELD_VALS = 20;
- private static final int FACET_LIMIT = UNIQUE_FIELD_VALS + 1;
+
+ // NOTE: set to 'true' to see if refinement testing is adequate (should get fails occasionally)
+ private static final boolean FORCE_DISABLE_REFINEMENT = false;
/** Multivalued string field suffixes that can be randomized for testing diff facet/join code paths */
private static final String[] STR_FIELD_SUFFIXES = new String[] { "_ss", "_sds", "_sdsS" };
@@ -88,8 +92,6 @@ public class TestCloudJSONFacetJoinDomain extends SolrCloudTestCase {
@BeforeClass
private static void createMiniSolrCloudCluster() throws Exception {
// sanity check constants
- assertTrue("bad test constants: must have UNIQUE_FIELD_VALS < FACET_LIMIT to get accurate counts without refinements",
- UNIQUE_FIELD_VALS < FACET_LIMIT);
assertTrue("bad test constants: some suffixes will never be tested",
(STR_FIELD_SUFFIXES.length < MAX_FIELD_NUM) && (INT_FIELD_SUFFIXES.length < MAX_FIELD_NUM));
@@ -170,14 +172,14 @@ public class TestCloudJSONFacetJoinDomain extends SolrCloudTestCase {
/**
* Given a (random) field number, returns a random (integer based) value for that field.
* NOTE: The number of unique values in each field is constant acording to {@link #UNIQUE_FIELD_VALS}
- * but the cise pr<em>range</em> of values will vary for each unique field number, such that cross field joins
+ * but the precise <em>range</em> of values will vary for each unique field number, such that cross field joins
* will match fewer documents based on how far apart the field numbers are.
*
* @see #UNIQUE_FIELD_VALS
* @see #field
*/
private static String randFieldValue(final int fieldNum) {
- return "" + (fieldNum + TestUtil.nextInt(random(), 0, UNIQUE_FIELD_VALS));
+ return "" + (fieldNum + TestUtil.nextInt(random(), 1, UNIQUE_FIELD_VALS));
}
@@ -301,6 +303,64 @@ public class TestCloudJSONFacetJoinDomain extends SolrCloudTestCase {
assertFacetCountsAreCorrect(facets, "("+strfield(7)+":6 OR "+strfield(9)+":6 OR "+strfield(6)+":19 OR "+strfield(0)+":11)");
}
+
+ { // low limits, explicit refinement
+ Map<String,TermFacet> facets = new LinkedHashMap<>();
+ TermFacet top = new TermFacet(strfield(9),
+ new JoinDomain(strfield(5), strfield(9), strfield(9)+":[* TO *]"),
+ 5, 0, true);
+ top.subFacets.put("facet_5", new TermFacet(strfield(11),
+ new JoinDomain(strfield(8), strfield(8), null),
+ 10, 0, true));
+ facets.put("facet_4", top);
+ assertFacetCountsAreCorrect(facets, "("+strfield(7)+":6 OR "+strfield(9)+":6 OR "+strfield(6)+":19 OR "+strfield(0)+":11)");
+ }
+
+ { // low limit, high overrequest
+ Map<String,TermFacet> facets = new LinkedHashMap<>();
+ TermFacet top = new TermFacet(strfield(9),
+ new JoinDomain(strfield(5), strfield(9), strfield(9)+":[* TO *]"),
+ 5, UNIQUE_FIELD_VALS + 10, false);
+ top.subFacets.put("facet_5", new TermFacet(strfield(11),
+ new JoinDomain(strfield(8), strfield(8), null),
+ 10, UNIQUE_FIELD_VALS + 10, false));
+ facets.put("facet_4", top);
+ assertFacetCountsAreCorrect(facets, "("+strfield(7)+":6 OR "+strfield(9)+":6 OR "+strfield(6)+":19 OR "+strfield(0)+":11)");
+ }
+
+ { // low limit, low overrequest, explicit refinement
+ Map<String,TermFacet> facets = new LinkedHashMap<>();
+ TermFacet top = new TermFacet(strfield(9),
+ new JoinDomain(strfield(5), strfield(9), strfield(9)+":[* TO *]"),
+ 5, 7, true);
+ top.subFacets.put("facet_5", new TermFacet(strfield(11),
+ new JoinDomain(strfield(8), strfield(8), null),
+ 10, 7, true));
+ facets.put("facet_4", top);
+ assertFacetCountsAreCorrect(facets, "("+strfield(7)+":6 OR "+strfield(9)+":6 OR "+strfield(6)+":19 OR "+strfield(0)+":11)");
+ }
+
+ }
+
+ public void testTheTestRandomRefineParam() {
+ // sanity check that randomRefineParam never violates isRefinementNeeded
+ // (should be imposisble ... unless someone changes/breaks the randomization logic in the future)
+ final int numIters = atLeast(100);
+ for (int iter = 0; iter < numIters; iter++) {
+ final Integer limit = TermFacet.randomLimitParam(random());
+ final Integer overrequest = TermFacet.randomOverrequestParam(random());
+ final Boolean refine = TermFacet.randomRefineParam(random(), limit, overrequest);
+ if (TermFacet.isRefinementNeeded(limit, overrequest)) {
+ assertEquals("limit: " + limit + ", overrequest: " + overrequest + ", refine: " + refine,
+ Boolean.TRUE, refine);
+ }
+ }
+ }
+
+ public void testTheTestTermFacetShouldFreakOutOnBadRefineOptions() {
+ expectThrows(AssertionError.class, () -> {
+ final TermFacet bogus = new TermFacet("foo", null, 5, 0, false);
+ });
}
public void testRandom() throws Exception {
@@ -423,10 +483,25 @@ public class TestCloudJSONFacetJoinDomain extends SolrCloudTestCase {
public final String field;
public final Map<String,TermFacet> subFacets = new LinkedHashMap<>();
public final JoinDomain domain; // may be null
+ public final Integer limit; // may be null
+ public final Integer overrequest; // may be null
+ public final Boolean refine; // may be null
+
+ /** Simplified constructor asks for limit = # unique vals */
public TermFacet(String field, JoinDomain domain) {
+ this(field, domain, UNIQUE_FIELD_VALS, 0, false);
+ }
+ public TermFacet(String field, JoinDomain domain, Integer limit, Integer overrequest, Boolean refine) {
assert null != field;
this.field = field;
this.domain = domain;
+ this.limit = limit;
+ this.overrequest = overrequest;
+ this.refine = refine;
+ if (isRefinementNeeded(limit, overrequest)) {
+ assertEquals("Invalid refine param based on limit & overrequest: " + this.toString(),
+ Boolean.TRUE, refine);
+ }
}
/**
@@ -455,45 +530,10 @@ public class TestCloudJSONFacetJoinDomain extends SolrCloudTestCase {
* recursively generates the <code>json.facet</code> param value to use for testing this facet
*/
private CharSequence toJSONFacetParamValue() {
- int limit = random().nextInt(FACET_LIMIT*2);
- String limitStr = ", limit:" + limit;
- if (limit >= FACET_LIMIT && random().nextBoolean()) {
- limitStr = ", limit:-1"; // unlimited
- } else if (limit == 10 && random().nextBoolean()) {
- limitStr=""; // don't specify limit since it's the default
- }
-
- int overrequest = -1;
- switch(random().nextInt(10)) {
- case 0:
- case 1:
- case 2:
- case 3:
- overrequest = 0; // 40% of the time, no overrequest to better stress refinement
- break;
- case 4:
- case 5:
- overrequest = random().nextInt(FACET_LIMIT);
- break;
- case 6:
- overrequest = random().nextInt(Integer.MAX_VALUE);
- break;
- default: break;
- }
- String overrequestStr = overrequest==-1 ? "" : ", overrequest:"+overrequest;
-
- boolean refine = (overrequest >= 0 && (long)limit + overrequest < FACET_LIMIT)
- || (overrequest < 0 && limit < FACET_LIMIT) // don't assume how much overrequest we do by default, just check the limit
- || random().nextInt(10)==0; // once in a while, turn on refinement even when it isn't needed.
-
- // refine = false; // NOTE: Uncomment this line to see if refinement testing is adequate (should get fails occasionally)
- String refineStr=", refine:" + refine;
- if (!refine) {
- // if refine==false, don't specify it sometimes (it's the default)
- if (random().nextBoolean()) refineStr="";
- }
-
- StringBuilder sb = new StringBuilder("{ type:terms, field:" + field + limitStr + overrequestStr + refineStr);
+ final String limitStr = (null == limit) ? "" : (", limit:" + limit);
+ final String overrequestStr = (null == overrequest) ? "" : (", overrequest:" + overrequest);
+ final String refineStr = (null == refine) ? "" : ", refine:" + refine;
+ final StringBuilder sb = new StringBuilder("{ type:terms, field:" + field + limitStr + overrequestStr + refineStr);
if (! subFacets.isEmpty()) {
sb.append(", facet:");
sb.append(toJSONFacetParamValue(subFacets));
@@ -539,6 +579,91 @@ public class TestCloudJSONFacetJoinDomain extends SolrCloudTestCase {
return buildRandomFacets(keyCounter, maxDepth);
}
+ /**
+ * picks a random value for the "limit" param, biased in favor of interesting test cases
+ *
+ * @return a number to specify in the request, or null to specify nothing (trigger default behavior)
+ * @see #UNIQUE_FIELD_VALS
+ */
+ public static Integer randomLimitParam(Random r) {
+ final int limit = r.nextInt(UNIQUE_FIELD_VALS * 2);
+ if (limit >= UNIQUE_FIELD_VALS && r.nextBoolean()) {
+ return -1; // unlimited
+ } else if (limit == DEFAULT_LIMIT && r.nextBoolean()) {
+ return null; // sometimes, don't specify limit if it's the default
+ }
+ return limit;
+ }
+
+ /**
+ * picks a random value for the "overrequest" param, biased in favor of interesting test cases
+ *
+ * @return a number to specify in the request, or null to specify nothing (trigger default behavior)
+ * @see #UNIQUE_FIELD_VALS
+ */
+ public static Integer randomOverrequestParam(Random r) {
+ switch(r.nextInt(10)) {
+ case 0:
+ case 1:
+ case 2:
+ case 3:
+ return 0; // 40% of the time, no overrequest to better stress refinement
+ case 4:
+ case 5:
+ return r.nextInt(UNIQUE_FIELD_VALS); // 20% ask for less them what's needed
+ case 6:
+ return r.nextInt(Integer.MAX_VALUE); // 10%: completley random value, statisticaly more then enough
+ default: break;
+ }
+ // else.... either leave param unspecified (or redundently specify the -1 default)
+ return r.nextBoolean() ? null : -1;
+ }
+
+ /**
+ * picks a random value for the "refine" param, that is garunteed to be suitable for
+ * the specified limit & overrequest params.
+ *
+ * @return a value to specify in the request, or null to specify nothing (trigger default behavior)
+ * @see #randomLimitParam
+ * @see #randomOverrequestParam
+ * @see #UNIQUE_FIELD_VALS
+ */
+ public static Boolean randomRefineParam(Random r, Integer limitParam, Integer overrequestParam) {
+ if (isRefinementNeeded(limitParam, overrequestParam)) {
+ return true;
+ }
+
+ // refinement is not required
+ if (0 == r.nextInt(10)) { // once in a while, turn on refinement even if it isn't needed.
+ return true;
+ }
+ // explicitly or implicitly indicate refinement is not needed
+ return r.nextBoolean() ? false : null;
+ }
+
+ /**
+ * Deterministicly identifies if the specified limit & overrequest params <b>require</b>
+ * a "refine:true" param be used in the the request, in order for the counts to be 100% accurate.
+ *
+ * @see #UNIQUE_FIELD_VALS
+ */
+ public static boolean isRefinementNeeded(Integer limitParam, Integer overrequestParam) {
+
+ if (FORCE_DISABLE_REFINEMENT) {
+ return false;
+ }
+
+ // use the "effective" values if the params are null
+ final int limit = null == limitParam ? DEFAULT_LIMIT : limitParam;
+ final int overrequest = null == overrequestParam ? 0 : overrequestParam;
+
+ return
+ // don't presume how much overrequest will be done by default, just check the limit
+ (overrequest < 0 && limit < UNIQUE_FIELD_VALS)
+ // if the user specified overrequest is not "enough" to get all unique values
+ || (overrequest >= 0 && (long)limit + overrequest < UNIQUE_FIELD_VALS);
+ }
+
/**
* recursive helper method for building random facets
*
@@ -551,9 +676,12 @@ public class TestCloudJSONFacetJoinDomain extends SolrCloudTestCase {
for (int i = 0; i < numFacets; i++) {
final JoinDomain domain = JoinDomain.buildRandomDomain();
assert null != domain;
+ final Integer limit = randomLimitParam(random());
+ final Integer overrequest = randomOverrequestParam(random());
final TermFacet facet = new TermFacet(field(random().nextBoolean() ? STR_FIELD_SUFFIXES : INT_FIELD_SUFFIXES,
random().nextInt(MAX_FIELD_NUM)),
- domain);
+ domain, limit, overrequest,
+ randomRefineParam(random(), limit, overrequest));
results.put("facet_" + keyCounter.incrementAndGet(), facet);
if (0 < maxDepth) {
// if we're going wide, don't go deep
[03/50] lucene-solr:feature/autoscaling_72: no Jira - minor generics
/ code constructs fixes in MLT
Posted by ab...@apache.org.
no Jira - minor generics / code constructs fixes in MLT
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/f9f6d3a3
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/f9f6d3a3
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/f9f6d3a3
Branch: refs/heads/feature/autoscaling_72
Commit: f9f6d3a3ac43be3a8bd09a5ff131f99e4b5df145
Parents: f1bb914
Author: Tommaso Teofili <to...@apache.org>
Authored: Fri Sep 29 15:32:27 2017 +0200
Committer: Tommaso Teofili <to...@apache.org>
Committed: Fri Sep 29 15:34:52 2017 +0200
----------------------------------------------------------------------
.../org/apache/lucene/queries/mlt/MoreLikeThis.java | 16 ++++------------
1 file changed, 4 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f9f6d3a3/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java
----------------------------------------------------------------------
diff --git a/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java b/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java
index ea02af3..9afa687 100644
--- a/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java
+++ b/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java
@@ -782,11 +782,7 @@ public final class MoreLikeThis {
* @param vector List of terms and their frequencies for a doc/field
*/
private void addTermFrequencies(Map<String, Map<String, Int>> field2termFreqMap, Terms vector, String fieldName) throws IOException {
- Map<String, Int> termFreqMap = field2termFreqMap.get(fieldName);
- if (termFreqMap == null) {
- termFreqMap = new HashMap<>();
- field2termFreqMap.put(fieldName, termFreqMap);
- }
+ Map<String, Int> termFreqMap = field2termFreqMap.computeIfAbsent(fieldName, k -> new HashMap<>());
final TermsEnum termsEnum = vector.iterator();
final CharsRefBuilder spare = new CharsRefBuilder();
BytesRef text;
@@ -823,11 +819,7 @@ public final class MoreLikeThis {
throw new UnsupportedOperationException("To use MoreLikeThis without " +
"term vectors, you must provide an Analyzer");
}
- Map<String, Int> termFreqMap = perFieldTermFrequencies.get(fieldName);
- if (termFreqMap == null) {
- termFreqMap = new HashMap<>();
- perFieldTermFrequencies.put(fieldName, termFreqMap);
- }
+ Map<String, Int> termFreqMap = perFieldTermFrequencies.computeIfAbsent(fieldName, k -> new HashMap<>());
try (TokenStream ts = analyzer.tokenStream(fieldName, r)) {
int tokenCount = 0;
// for every token
@@ -906,7 +898,7 @@ public final class MoreLikeThis {
* @see #retrieveInterestingTerms(java.io.Reader, String)
*/
public String[] retrieveInterestingTerms(int docNum) throws IOException {
- ArrayList<Object> al = new ArrayList<>(maxQueryTerms);
+ ArrayList<String> al = new ArrayList<>(maxQueryTerms);
PriorityQueue<ScoreTerm> pq = retrieveTerms(docNum);
ScoreTerm scoreTerm;
int lim = maxQueryTerms; // have to be careful, retrieveTerms returns all words but that's probably not useful to our caller...
@@ -929,7 +921,7 @@ public final class MoreLikeThis {
* @see #setMaxQueryTerms
*/
public String[] retrieveInterestingTerms(Reader r, String fieldName) throws IOException {
- ArrayList<Object> al = new ArrayList<>(maxQueryTerms);
+ ArrayList<String> al = new ArrayList<>(maxQueryTerms);
PriorityQueue<ScoreTerm> pq = retrieveTerms(r, fieldName);
ScoreTerm scoreTerm;
int lim = maxQueryTerms; // have to be careful, retrieveTerms returns all words but that's probably not useful to our caller...
[06/50] lucene-solr:feature/autoscaling_72: SOLR-11422: Add
probabilities parameter to the enumeratedDistribution Stream Evaluator
Posted by ab...@apache.org.
SOLR-11422: Add probabilities parameter to the enumeratedDistribution Stream Evaluator
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/c322e369
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/c322e369
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/c322e369
Branch: refs/heads/feature/autoscaling_72
Commit: c322e36926576ad8cda1379d070598bdd9b43ea1
Parents: 71eb59e
Author: Joel Bernstein <jb...@apache.org>
Authored: Fri Sep 29 13:10:26 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Fri Sep 29 13:10:26 2017 -0400
----------------------------------------------------------------------
.../eval/EnumeratedDistributionEvaluator.java | 19 ++++++++++++++-----
.../solrj/io/stream/StreamExpressionTest.java | 20 ++++++++++++++++++++
2 files changed, 34 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c322e369/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java
index a14e54b..4a8b7f7 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java
@@ -25,7 +25,7 @@ import org.apache.commons.math3.distribution.EnumeratedIntegerDistribution;
import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
-public class EnumeratedDistributionEvaluator extends RecursiveNumericEvaluator implements OneValueWorker {
+public class EnumeratedDistributionEvaluator extends RecursiveNumericEvaluator implements ManyValueWorker {
private static final long serialVersionUID = 1;
@@ -34,12 +34,21 @@ public class EnumeratedDistributionEvaluator extends RecursiveNumericEvaluator i
}
@Override
- public Object doWork(Object first) throws IOException{
- if(null == first){
+ public Object doWork(Object... values) throws IOException{
+ if(values.length == 0){
throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the first value",toExpression(constructingFactory)));
}
- int[] samples = ((List)first).stream().mapToInt(value -> ((BigDecimal) value).intValue()).toArray();
- return new EnumeratedIntegerDistribution(samples);
+ if(values.length == 1) {
+ List<Number> first = (List<Number>)values[0];
+ int[] samples = ((List) first).stream().mapToInt(value -> ((BigDecimal) value).intValue()).toArray();
+ return new EnumeratedIntegerDistribution(samples);
+ } else {
+ List<Number> first = (List<Number>)values[0];
+ List<Number> second = (List<Number>)values[1];
+ int[] singletons = ((List) first).stream().mapToInt(value -> ((BigDecimal) value).intValue()).toArray();
+ double[] probs = ((List) second).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ return new EnumeratedIntegerDistribution(singletons, probs);
+ }
}
}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c322e369/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index d84120f..9e4c6c3 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6480,6 +6480,26 @@ public class StreamExpressionTest extends SolrCloudTestCase {
assertEquals(prob.doubleValue(), 0.1, 0.07);
Number cprob = (Number)tuples.get(0).get("c");
assertEquals(cprob.doubleValue(), 0.5, 0.07);
+
+
+ cexpr = "let(a=sample(enumeratedDistribution(array(1,2,3,4), array(40, 30, 20, 10)), 50000),"+
+ "b=freqTable(a),"+
+ "y=col(b, pct))";
+
+ paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ solrStream = new SolrStream(url, paramsLoc);
+ context = new StreamContext();
+ solrStream.setStreamContext(context);
+ tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ List<Number> freqs = (List<Number>)tuples.get(0).get("y");
+ assertEquals(freqs.get(0).doubleValue(), .40, .03);
+ assertEquals(freqs.get(1).doubleValue(), .30, .03);
+ assertEquals(freqs.get(2).doubleValue(), .20, .03);
+ assertEquals(freqs.get(3).doubleValue(), .10, .03);
}
@Test
[31/50] lucene-solr:feature/autoscaling_72: SOLR-11424: When
legacyCloud=false, cores should not publish itself as DOWN on startup
Posted by ab...@apache.org.
SOLR-11424: When legacyCloud=false, cores should not publish itself as DOWN on startup
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/99057368
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/99057368
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/99057368
Branch: refs/heads/feature/autoscaling_72
Commit: 990573689887cd0e6df0313984b3051aa74ba76f
Parents: fb19528
Author: Cao Manh Dat <da...@apache.org>
Authored: Wed Oct 4 15:25:17 2017 +0700
Committer: Cao Manh Dat <da...@apache.org>
Committed: Wed Oct 4 15:25:17 2017 +0700
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +
.../org/apache/solr/cloud/ZkController.java | 32 +++--
.../org/apache/solr/core/CoreContainer.java | 2 +-
.../solr/cloud/TestShardsWithSingleReplica.java | 84 -------------
.../solr/cloud/TestSkipOverseerOperations.java | 125 +++++++++++++++++++
5 files changed, 152 insertions(+), 93 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/99057368/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index c14c9b6..0d61cc2 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -174,6 +174,8 @@ Optimizations
* SOLR-10285: Skip LEADER messages when there are leader only shards (Cao Manh Dat, Joshua Humphries)
+* SOLR-11424: When legacyCloud=false, cores should not publish itself as DOWN on startup. (Cao Manh Dat)
+
Other Changes
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/99057368/solr/core/src/java/org/apache/solr/cloud/ZkController.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/ZkController.java b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
index dcce2c8..068a130 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ZkController.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
@@ -1471,7 +1471,7 @@ public class ZkController {
return coreNodeName;
}
- public void preRegister(CoreDescriptor cd) {
+ public void preRegister(CoreDescriptor cd, boolean publishState) {
String coreNodeName = getCoreNodeName(cd);
@@ -1487,7 +1487,10 @@ public class ZkController {
cloudDesc.setCoreNodeName(coreNodeName);
}
- publish(cd, Replica.State.DOWN, false, true);
+ // publishState == false on startup
+ if (publishState || isPublishAsDownOnStartup(cloudDesc)) {
+ publish(cd, Replica.State.DOWN, false, true);
+ }
String collectionName = cd.getCloudDescriptor().getCollectionName();
DocCollection collection = zkStateReader.getClusterState().getCollectionOrNull(collectionName);
log.debug(collection == null ?
@@ -1504,15 +1507,28 @@ public class ZkController {
throw new ZooKeeperException(SolrException.ErrorCode.SERVER_ERROR, "", e);
}
- if (cd.getCloudDescriptor().getShardId() == null && needsToBeAssignedShardId(cd, zkStateReader.getClusterState(), coreNodeName)) {
- doGetShardIdAndNodeNameProcess(cd);
- } else {
- // still wait till we see us in local state
- doGetShardIdAndNodeNameProcess(cd);
- }
+ doGetShardIdAndNodeNameProcess(cd);
}
+ /**
+ * On startup, the node already published all of its replicas as DOWN,
+ * so in case of legacyCloud=false ( the replica must already present on Zk )
+ * we can skip publish the replica as down
+ * @return Should publish the replica as down on startup
+ */
+ private boolean isPublishAsDownOnStartup(CloudDescriptor cloudDesc) {
+ if (!Overseer.isLegacy(zkStateReader)) {
+ Replica replica = zkStateReader.getClusterState().getCollection(cloudDesc.getCollectionName())
+ .getSlice(cloudDesc.getShardId())
+ .getReplica(cloudDesc.getCoreNodeName());
+ if (replica.getNodeName().equals(getNodeName())) {
+ return false;
+ }
+ }
+ return true;
+ }
+
private void checkStateInZk(CoreDescriptor cd) throws InterruptedException {
if (!Overseer.isLegacy(zkStateReader)) {
CloudDescriptor cloudDesc = cd.getCloudDescriptor();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/99057368/solr/core/src/java/org/apache/solr/core/CoreContainer.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/CoreContainer.java b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
index f3f3c49..4467bc0 100644
--- a/solr/core/src/java/org/apache/solr/core/CoreContainer.java
+++ b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
@@ -1018,7 +1018,7 @@ public class CoreContainer {
MDCLoggingContext.setCoreDescriptor(this, dcore);
SolrIdentifierValidator.validateCoreName(dcore.getName());
if (zkSys.getZkController() != null) {
- zkSys.getZkController().preRegister(dcore);
+ zkSys.getZkController().preRegister(dcore, publishState);
}
ConfigSet coreConfig = coreConfigService.getConfig(dcore);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/99057368/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java b/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
deleted file mode 100644
index a4c51ba..0000000
--- a/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
+++ /dev/null
@@ -1,84 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.solr.cloud;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.stream.Collectors;
-
-import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.embedded.JettySolrRunner;
-import org.apache.solr.client.solrj.request.CollectionAdminRequest;
-import org.apache.solr.client.solrj.response.CollectionAdminResponse;
-import org.junit.BeforeClass;
-
-public class TestShardsWithSingleReplica extends SolrCloudTestCase {
-
- @BeforeClass
- public static void setupCluster() throws Exception {
- System.setProperty("solr.directoryFactory", "solr.StandardDirectoryFactory");
- System.setProperty("solr.ulog.numRecordsToKeep", "1000");
-
- configureCluster(3)
- .addConfig("config", TEST_PATH().resolve("configsets").resolve("cloud-minimal").resolve("conf"))
- .configure();
- }
-
- public void testSkipLeaderOperations() throws Exception {
- String overseerLeader = getOverseerLeader();
- List<JettySolrRunner> notOverseerNodes = cluster.getJettySolrRunners()
- .stream()
- .filter(solrRunner -> !solrRunner.getNodeName().equals(overseerLeader))
- .collect(Collectors.toList());
- String collection = "collection1";
- CollectionAdminRequest
- .createCollection(collection, 2, 1)
- .setCreateNodeSet(notOverseerNodes
- .stream()
- .map(JettySolrRunner::getNodeName)
- .collect(Collectors.joining(","))
- )
- .process(cluster.getSolrClient());
-
- for (JettySolrRunner solrRunner : notOverseerNodes) {
- cluster.stopJettySolrRunner(solrRunner);
- }
- waitForState("Expected empty liveNodes", collection,
- (liveNodes, collectionState) -> liveNodes.size() == 1);
-
- CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
- for (JettySolrRunner solrRunner : notOverseerNodes) {
- cluster.startJettySolrRunner(solrRunner);
- }
-
- waitForState("Expected 2x1 for collection: " + collection, collection,
- clusterShape(2, 1));
- CollectionAdminResponse resp2 = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
- assertEquals(getNumLeaderOpeations(resp), getNumLeaderOpeations(resp2));
- }
-
- private int getNumLeaderOpeations(CollectionAdminResponse resp) {
- return (int) resp.getResponse().findRecursive("overseer_operations", "leader", "requests");
- }
-
- private String getOverseerLeader() throws IOException, SolrServerException {
- CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
- return (String) resp.getResponse().get("leader");
- }
-
-}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/99057368/solr/core/src/test/org/apache/solr/cloud/TestSkipOverseerOperations.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestSkipOverseerOperations.java b/solr/core/src/test/org/apache/solr/cloud/TestSkipOverseerOperations.java
new file mode 100644
index 0000000..2039249
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/cloud/TestSkipOverseerOperations.java
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cloud;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.embedded.JettySolrRunner;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.response.CollectionAdminResponse;
+import org.junit.BeforeClass;
+
+public class TestSkipOverseerOperations extends SolrCloudTestCase {
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ System.setProperty("solr.directoryFactory", "solr.StandardDirectoryFactory");
+ System.setProperty("solr.ulog.numRecordsToKeep", "1000");
+
+ configureCluster(3)
+ .addConfig("config", TEST_PATH().resolve("configsets").resolve("cloud-minimal").resolve("conf"))
+ .configure();
+ }
+
+ public void testSkipLeaderOperations() throws Exception {
+ String overseerLeader = getOverseerLeader();
+ List<JettySolrRunner> notOverseerNodes = cluster.getJettySolrRunners()
+ .stream()
+ .filter(solrRunner -> !solrRunner.getNodeName().equals(overseerLeader))
+ .collect(Collectors.toList());
+ String collection = "collection1";
+ CollectionAdminRequest
+ .createCollection(collection, 2, 1)
+ .setCreateNodeSet(notOverseerNodes
+ .stream()
+ .map(JettySolrRunner::getNodeName)
+ .collect(Collectors.joining(","))
+ )
+ .process(cluster.getSolrClient());
+
+ for (JettySolrRunner solrRunner : notOverseerNodes) {
+ solrRunner.stop();
+ }
+ waitForState("Expected single liveNode", collection,
+ (liveNodes, collectionState) -> liveNodes.size() == 1);
+
+ CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ for (JettySolrRunner solrRunner : notOverseerNodes) {
+ solrRunner.start();
+ }
+
+ waitForState("Expected 2x1 for collection: " + collection, collection,
+ clusterShape(2, 1));
+ CollectionAdminResponse resp2 = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ assertEquals(getNumLeaderOpeations(resp), getNumLeaderOpeations(resp2));
+ CollectionAdminRequest.deleteCollection(collection).process(cluster.getSolrClient());
+ }
+
+ public void testSkipDownOperations() throws Exception {
+ String overseerLeader = getOverseerLeader();
+ List<JettySolrRunner> notOverseerNodes = cluster.getJettySolrRunners()
+ .stream()
+ .filter(solrRunner -> !solrRunner.getNodeName().equals(overseerLeader))
+ .collect(Collectors.toList());
+ String collection = "collection2";
+ CollectionAdminRequest
+ .createCollection(collection, 2, 2)
+ .setCreateNodeSet(notOverseerNodes
+ .stream()
+ .map(JettySolrRunner::getNodeName)
+ .collect(Collectors.joining(","))
+ )
+ .setMaxShardsPerNode(2)
+ .process(cluster.getSolrClient());
+
+ for (JettySolrRunner solrRunner : notOverseerNodes) {
+ solrRunner.stop();
+ }
+ waitForState("Expected single liveNode", collection,
+ (liveNodes, collectionState) -> liveNodes.size() == 1);
+
+ CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ for (JettySolrRunner solrRunner : notOverseerNodes) {
+ solrRunner.start();
+ }
+
+ waitForState("Expected 2x2 for collection: " + collection, collection,
+ clusterShape(2, 2));
+ CollectionAdminResponse resp2 = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ // 2 for recovering state, 4 for active state
+ assertEquals(getNumStateOpeations(resp) + 6, getNumStateOpeations(resp2));
+ CollectionAdminRequest.deleteCollection(collection).process(cluster.getSolrClient());
+ }
+
+ private int getNumLeaderOpeations(CollectionAdminResponse resp) {
+ return (int) resp.getResponse().findRecursive("overseer_operations", "leader", "requests");
+ }
+
+ private int getNumStateOpeations(CollectionAdminResponse resp) {
+ return (int) resp.getResponse().findRecursive("overseer_operations", "state", "requests");
+ }
+
+ private String getOverseerLeader() throws IOException, SolrServerException {
+ CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ return (String) resp.getResponse().get("leader");
+ }
+
+}
[14/50] lucene-solr:feature/autoscaling_72: LUCENE-7974: Add
N-dimensional FloatPoint K-nearest-neighbor implementation
Posted by ab...@apache.org.
LUCENE-7974: Add N-dimensional FloatPoint K-nearest-neighbor implementation
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/d52564c0
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/d52564c0
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/d52564c0
Branch: refs/heads/feature/autoscaling_72
Commit: d52564c079bb7ca842a1041bc8baff468e1938d4
Parents: 472d520
Author: Steve Rowe <sa...@apache.org>
Authored: Sun Oct 1 18:50:06 2017 -0400
Committer: Steve Rowe <sa...@apache.org>
Committed: Sun Oct 1 18:50:55 2017 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +
.../document/FloatPointNearestNeighbor.java | 382 +++++++++++++++++++
.../document/TestFloatPointNearestNeighbor.java | 239 ++++++++++++
3 files changed, 624 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d52564c0/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 16482e9..e0c8124 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -34,6 +34,9 @@ New Features
* LUCENE-7973: Update dictionary version for Ukrainian analyzer to 3.9.0 (Andriy
Rysin via Dawid Weiss)
+
+* LUCENE-7974: Add FloatPointNearestNeighbor, an N-dimensional FloatPoint
+ K-nearest-neighbor search implementation. (Steve Rowe)
Optimizations
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d52564c0/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
new file mode 100644
index 0000000..d3360a8
--- /dev/null
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.document;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.PriorityQueue;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.search.FieldDoc;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopFieldDocs;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.bkd.BKDReader;
+
+/**
+ * KNN search on top of N dimensional indexed float points.
+ *
+ * @lucene.experimental
+ */
+public class FloatPointNearestNeighbor {
+
+ static class Cell implements Comparable<Cell> {
+ final int readerIndex;
+ final byte[] minPacked;
+ final byte[] maxPacked;
+ final BKDReader.IndexTree index;
+
+ /** The closest possible distance^2 of all points in this cell */
+ final double distanceSquared;
+
+ Cell(BKDReader.IndexTree index, int readerIndex, byte[] minPacked, byte[] maxPacked, double distanceSquared) {
+ this.index = index;
+ this.readerIndex = readerIndex;
+ this.minPacked = minPacked.clone();
+ this.maxPacked = maxPacked.clone();
+ this.distanceSquared = distanceSquared;
+ }
+
+ public int compareTo(Cell other) {
+ return Double.compare(distanceSquared, other.distanceSquared);
+ }
+
+ @Override
+ public String toString() {
+ return "Cell(readerIndex=" + readerIndex + " nodeID=" + index.getNodeID()
+ + " isLeaf=" + index.isLeafNode() + " distanceSquared=" + distanceSquared + ")";
+ }
+ }
+
+ private static class NearestVisitor implements PointValues.IntersectVisitor {
+ int curDocBase;
+ Bits curLiveDocs;
+ final int topN;
+ final PriorityQueue<NearestHit> hitQueue;
+ final float[] origin;
+ private int dims;
+ private int updateMinMaxCounter;
+ private float[] min;
+ private float[] max;
+
+
+ public NearestVisitor(PriorityQueue<NearestHit> hitQueue, int topN, float[] origin) {
+ this.hitQueue = hitQueue;
+ this.topN = topN;
+ this.origin = origin;
+ dims = origin.length;
+ min = new float[dims];
+ max = new float[dims];
+ Arrays.fill(min, Float.NEGATIVE_INFINITY);
+ Arrays.fill(max, Float.POSITIVE_INFINITY);
+ }
+
+ @Override
+ public void visit(int docID) {
+ throw new AssertionError();
+ }
+
+ private static final int MANTISSA_BITS = 23;
+
+ /**
+ * Returns the minimum value that will change the given distance when added to it.
+ *
+ * This value is calculated from the distance exponent reduced by (at most) 23,
+ * the number of bits in a float mantissa. This is necessary when the result of
+ * subtracting/adding the distance in a single dimension has an exponent that
+ * differs significantly from that of the distance value. Without this fudge
+ * factor (i.e. only subtracting/adding the distance), cells and values can be
+ * inappropriately judged as outside the search radius.
+ */
+ private float getMinDelta(float distance) {
+ int exponent = Float.floatToIntBits(distance) >> MANTISSA_BITS; // extract biased exponent (distance is positive)
+ if (exponent == 0) {
+ return Float.MIN_VALUE;
+ } else {
+ exponent = exponent <= MANTISSA_BITS ? 1 : exponent - MANTISSA_BITS; // Avoid underflow
+ return Float.intBitsToFloat(exponent << MANTISSA_BITS);
+ }
+ }
+
+ private void maybeUpdateMinMax() {
+ if (updateMinMaxCounter < 1024 || (updateMinMaxCounter & 0x3F) == 0x3F) {
+ NearestHit hit = hitQueue.peek();
+ float distance = (float)Math.sqrt(hit.distanceSquared);
+ float minDelta = getMinDelta(distance);
+ // String oldMin = Arrays.toString(min);
+ // String oldMax = Arrays.toString(max);
+ for (int d = 0 ; d < dims ; ++d) {
+ min[d] = (origin[d] - distance) - minDelta;
+ max[d] = (origin[d] + distance) + minDelta;
+ // System.out.println("origin[" + d + "] (" + origin[d] + ") - distance (" + distance + ") - minDelta (" + minDelta + ") = min[" + d + "] (" + min[d] + ")");
+ // System.out.println("origin[" + d + "] (" + origin[d] + ") + distance (" + distance + ") + minDelta (" + minDelta + ") = max[" + d + "] (" + max[d] + ")");
+ }
+ // System.out.println("maybeUpdateMinMax: min: " + oldMin + " -> " + Arrays.toString(min) + " max: " + oldMax + " -> " + Arrays.toString(max));
+ }
+ ++updateMinMaxCounter;
+ }
+
+ @Override
+ public void visit(int docID, byte[] packedValue) {
+ // System.out.println("visit docID=" + docID + " liveDocs=" + curLiveDocs);
+
+ if (curLiveDocs != null && curLiveDocs.get(docID) == false) {
+ return;
+ }
+
+ float[] docPoint = new float[dims];
+ for (int d = 0, offset = 0 ; d < dims ; ++d, offset += Float.BYTES) {
+ docPoint[d] = FloatPoint.decodeDimension(packedValue, offset);
+ if (docPoint[d] > max[d] || docPoint[d] < min[d]) {
+
+ // if (docPoint[d] > max[d]) {
+ // System.out.println(" skipped because docPoint[" + d + "] (" + docPoint[d] + ") > max[" + d + "] (" + max[d] + ")");
+ // } else {
+ // System.out.println(" skipped because docPoint[" + d + "] (" + docPoint[d] + ") < min[" + d + "] (" + min[d] + ")");
+ // }
+
+ return;
+ }
+ }
+
+ double distanceSquared = euclideanDistanceSquared(origin, docPoint);
+
+ // System.out.println(" visit docID=" + docID + " distanceSquared=" + distanceSquared + " value: " + Arrays.toString(docPoint));
+
+ int fullDocID = curDocBase + docID;
+
+ if (hitQueue.size() == topN) { // queue already full
+ NearestHit bottom = hitQueue.peek();
+ // System.out.println(" bottom distanceSquared=" + bottom.distanceSquared);
+ if (distanceSquared < bottom.distanceSquared
+ // we don't collect docs in order here, so we must also test the tie-break case ourselves:
+ || (distanceSquared == bottom.distanceSquared && fullDocID < bottom.docID)) {
+ hitQueue.poll();
+ bottom.docID = fullDocID;
+ bottom.distanceSquared = distanceSquared;
+ hitQueue.offer(bottom);
+ // System.out.println(" ** keep1, now bottom=" + bottom);
+ maybeUpdateMinMax();
+ }
+ } else {
+ NearestHit hit = new NearestHit();
+ hit.docID = fullDocID;
+ hit.distanceSquared = distanceSquared;
+ hitQueue.offer(hit);
+ // System.out.println(" ** keep2, new addition=" + hit);
+ }
+ }
+
+ @Override
+ public PointValues.Relation compare(byte[] minPackedValue, byte[] maxPackedValue) {
+ throw new AssertionError();
+ }
+ }
+
+ static class NearestHit {
+ public int docID;
+ public double distanceSquared;
+
+ @Override
+ public String toString() {
+ return "NearestHit(docID=" + docID + " distanceSquared=" + distanceSquared + ")";
+ }
+ }
+
+ public static NearestHit[] nearest(List<BKDReader> readers, List<Bits> liveDocs, List<Integer> docBases, final int topN, float[] origin) throws IOException {
+
+ // System.out.println("NEAREST: readers=" + readers + " liveDocs=" + liveDocs + " origin: " + Arrays.toString(origin));
+
+ // Holds closest collected points seen so far:
+ // TODO: if we used lucene's PQ we could just updateTop instead of poll/offer:
+ final PriorityQueue<NearestHit> hitQueue = new PriorityQueue<>(topN, (a, b) -> {
+ // sort by opposite distance natural order
+ int cmp = Double.compare(a.distanceSquared, b.distanceSquared);
+ return cmp != 0 ? -cmp : b.docID - a.docID; // tie-break by higher docID
+ });
+
+ // Holds all cells, sorted by closest to the point:
+ PriorityQueue<Cell> cellQueue = new PriorityQueue<>();
+
+ NearestVisitor visitor = new NearestVisitor(hitQueue, topN, origin);
+ List<BKDReader.IntersectState> states = new ArrayList<>();
+
+ // Add root cell for each reader into the queue:
+ int bytesPerDim = -1;
+
+ for (int i = 0 ; i < readers.size() ; ++i) {
+ BKDReader reader = readers.get(i);
+ if (bytesPerDim == -1) {
+ bytesPerDim = reader.getBytesPerDimension();
+ } else if (bytesPerDim != reader.getBytesPerDimension()) {
+ throw new IllegalStateException("bytesPerDim changed from " + bytesPerDim
+ + " to " + reader.getBytesPerDimension() + " across readers");
+ }
+ byte[] minPackedValue = reader.getMinPackedValue();
+ byte[] maxPackedValue = reader.getMaxPackedValue();
+ BKDReader.IntersectState state = reader.getIntersectState(visitor);
+ states.add(state);
+
+ cellQueue.offer(new Cell(state.index, i, reader.getMinPackedValue(), reader.getMaxPackedValue(),
+ approxBestDistanceSquared(minPackedValue, maxPackedValue, origin)));
+ }
+
+ LOOP_OVER_CELLS: while (cellQueue.size() > 0) {
+ Cell cell = cellQueue.poll();
+ // System.out.println(" visit " + cell);
+
+ // TODO: if we replace approxBestDistance with actualBestDistance, we can put an opto here to break once this "best" cell is fully outside of the hitQueue bottom's radius:
+ BKDReader reader = readers.get(cell.readerIndex);
+
+ if (cell.index.isLeafNode()) {
+ // System.out.println(" leaf");
+ // Leaf block: visit all points and possibly collect them:
+ visitor.curDocBase = docBases.get(cell.readerIndex);
+ visitor.curLiveDocs = liveDocs.get(cell.readerIndex);
+ reader.visitLeafBlockValues(cell.index, states.get(cell.readerIndex));
+ // System.out.println(" now " + hitQueue.size() + " hits");
+ } else {
+ // System.out.println(" non-leaf");
+ // Non-leaf block: split into two cells and put them back into the queue:
+
+ if (hitQueue.size() == topN) {
+ for (int d = 0, offset = 0; d < visitor.dims; ++d, offset += Float.BYTES) {
+ float cellMaxAtDim = FloatPoint.decodeDimension(cell.maxPacked, offset);
+ float cellMinAtDim = FloatPoint.decodeDimension(cell.minPacked, offset);
+ if (cellMaxAtDim < visitor.min[d] || cellMinAtDim > visitor.max[d]) {
+ // this cell is outside our search radius; don't bother exploring any more
+
+ // if (cellMaxAtDim < visitor.min[d]) {
+ // System.out.println(" skipped because cell max at " + d + " (" + cellMaxAtDim + ") < visitor.min[" + d + "] (" + visitor.min[d] + ")");
+ // } else {
+ // System.out.println(" skipped because cell min at " + d + " (" + cellMinAtDim + ") > visitor.max[" + d + "] (" + visitor.max[d] + ")");
+ // }
+
+ continue LOOP_OVER_CELLS;
+ }
+ }
+ }
+ BytesRef splitValue = BytesRef.deepCopyOf(cell.index.getSplitDimValue());
+ int splitDim = cell.index.getSplitDim();
+
+ // we must clone the index so that we we can recurse left and right "concurrently":
+ BKDReader.IndexTree newIndex = cell.index.clone();
+ byte[] splitPackedValue = cell.maxPacked.clone();
+ System.arraycopy(splitValue.bytes, splitValue.offset, splitPackedValue, splitDim * bytesPerDim, bytesPerDim);
+
+ cell.index.pushLeft();
+ cellQueue.offer(new Cell(cell.index, cell.readerIndex, cell.minPacked, splitPackedValue,
+ approxBestDistanceSquared(cell.minPacked, splitPackedValue, origin)));
+
+ splitPackedValue = cell.minPacked.clone();
+ System.arraycopy(splitValue.bytes, splitValue.offset, splitPackedValue, splitDim * bytesPerDim, bytesPerDim);
+
+ newIndex.pushRight();
+ cellQueue.offer(new Cell(newIndex, cell.readerIndex, splitPackedValue, cell.maxPacked,
+ approxBestDistanceSquared(splitPackedValue, cell.maxPacked, origin)));
+ }
+ }
+
+ NearestHit[] hits = new NearestHit[hitQueue.size()];
+ int downTo = hitQueue.size()-1;
+ while (hitQueue.size() != 0) {
+ hits[downTo] = hitQueue.poll();
+ downTo--;
+ }
+ return hits;
+ }
+
+ private static double approxBestDistanceSquared(byte[] minPackedValue, byte[] maxPackedValue, float[] value) {
+ boolean insideCell = true;
+ float[] min = new float[value.length];
+ float[] max = new float[value.length];
+ double[] closest = new double[value.length];
+ for (int i = 0, offset = 0 ; i < value.length ; ++i, offset += Float.BYTES) {
+ min[i] = FloatPoint.decodeDimension(minPackedValue, offset);
+ max[i] = FloatPoint.decodeDimension(maxPackedValue, offset);
+ if (insideCell) {
+ if (value[i] < min[i] || value[i] > max[i]) {
+ insideCell = false;
+ }
+ }
+ double minDiff = Math.abs((double)value[i] - (double)min[i]);
+ double maxDiff = Math.abs((double)value[i] - (double)max[i]);
+ closest[i] = minDiff < maxDiff ? minDiff : maxDiff;
+ }
+ if (insideCell) {
+ return 0.0f;
+ }
+ double sumOfSquaredDiffs = 0.0d;
+ for (int d = 0 ; d < value.length ; ++d) {
+ sumOfSquaredDiffs += closest[d] * closest[d];
+ }
+ return sumOfSquaredDiffs;
+ }
+
+ static double euclideanDistanceSquared(float[] a, float[] b) {
+ double sumOfSquaredDifferences = 0.0d;
+ for (int d = 0 ; d < a.length ; ++d) {
+ double diff = (double)a[d] - (double)b[d];
+ sumOfSquaredDifferences += diff * diff;
+ }
+ return sumOfSquaredDifferences;
+ }
+
+ public static TopFieldDocs nearest(IndexSearcher searcher, String field, int topN, float... origin) throws IOException {
+ if (topN < 1) {
+ throw new IllegalArgumentException("topN must be at least 1; got " + topN);
+ }
+ if (field == null) {
+ throw new IllegalArgumentException("field must not be null");
+ }
+ if (searcher == null) {
+ throw new IllegalArgumentException("searcher must not be null");
+ }
+ List<BKDReader> readers = new ArrayList<>();
+ List<Integer> docBases = new ArrayList<>();
+ List<Bits> liveDocs = new ArrayList<>();
+ int totalHits = 0;
+ for (LeafReaderContext leaf : searcher.getIndexReader().leaves()) {
+ PointValues points = leaf.reader().getPointValues(field);
+ if (points != null) {
+ if (points instanceof BKDReader == false) {
+ throw new IllegalArgumentException("can only run on Lucene60PointsReader points implementation, but got " + points);
+ }
+ totalHits += points.getDocCount();
+ readers.add((BKDReader)points);
+ docBases.add(leaf.docBase);
+ liveDocs.add(leaf.reader().getLiveDocs());
+ }
+ }
+
+ NearestHit[] hits = nearest(readers, liveDocs, docBases, topN, origin);
+
+ // Convert to TopFieldDocs:
+ ScoreDoc[] scoreDocs = new ScoreDoc[hits.length];
+ for(int i=0;i<hits.length;i++) {
+ NearestHit hit = hits[i];
+ scoreDocs[i] = new FieldDoc(hit.docID, 0.0f, new Object[] { (float)Math.sqrt(hit.distanceSquared) });
+ }
+ return new TopFieldDocs(totalHits, scoreDocs, null, 0.0f);
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d52564c0/lucene/sandbox/src/test/org/apache/lucene/document/TestFloatPointNearestNeighbor.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/test/org/apache/lucene/document/TestFloatPointNearestNeighbor.java b/lucene/sandbox/src/test/org/apache/lucene/document/TestFloatPointNearestNeighbor.java
new file mode 100644
index 0000000..9e3c3c1
--- /dev/null
+++ b/lucene/sandbox/src/test/org/apache/lucene/document/TestFloatPointNearestNeighbor.java
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.document;
+
+import java.util.Arrays;
+
+import org.apache.lucene.codecs.Codec;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.index.RandomIndexWriter;
+import org.apache.lucene.index.SerialMergeScheduler;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.FieldDoc;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+public class TestFloatPointNearestNeighbor extends LuceneTestCase {
+
+ public void testNearestNeighborWithDeletedDocs() throws Exception {
+ Directory dir = newDirectory();
+ RandomIndexWriter w = new RandomIndexWriter(random(), dir, getIndexWriterConfig());
+ Document doc = new Document();
+ doc.add(new FloatPoint("point", 40.0f, 50.0f));
+ doc.add(new StringField("id", "0", Field.Store.YES));
+ w.addDocument(doc);
+
+ doc = new Document();
+ doc.add(new FloatPoint("point", 45.0f, 55.0f));
+ doc.add(new StringField("id", "1", Field.Store.YES));
+ w.addDocument(doc);
+
+ DirectoryReader r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ IndexSearcher s = newSearcher(r, false);
+ FieldDoc hit = (FieldDoc)FloatPointNearestNeighbor.nearest(s, "point", 1, 40.0f, 50.0f).scoreDocs[0];
+ assertEquals("0", r.document(hit.doc).getField("id").stringValue());
+ r.close();
+
+ w.deleteDocuments(new Term("id", "0"));
+ r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ s = newSearcher(r, false);
+ hit = (FieldDoc)LatLonPoint.nearest(s, "point", 40.0, 50.0, 1).scoreDocs[0];
+ assertEquals("1", r.document(hit.doc).getField("id").stringValue());
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ public void testNearestNeighborWithAllDeletedDocs() throws Exception {
+ Directory dir = newDirectory();
+ RandomIndexWriter w = new RandomIndexWriter(random(), dir, getIndexWriterConfig());
+ Document doc = new Document();
+ doc.add(new FloatPoint("point", 40.0f, 50.0f));
+ doc.add(new StringField("id", "0", Field.Store.YES));
+ w.addDocument(doc);
+ doc = new Document();
+ doc.add(new FloatPoint("point", 45.0f, 55.0f));
+ doc.add(new StringField("id", "1", Field.Store.YES));
+ w.addDocument(doc);
+
+ DirectoryReader r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ IndexSearcher s = newSearcher(r, false);
+ FieldDoc hit = (FieldDoc)FloatPointNearestNeighbor.nearest(s, "point", 1, 40.0f, 50.0f).scoreDocs[0];
+ assertEquals("0", r.document(hit.doc).getField("id").stringValue());
+ r.close();
+
+ w.deleteDocuments(new Term("id", "0"));
+ w.deleteDocuments(new Term("id", "1"));
+ r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ s = newSearcher(r, false);
+ assertEquals(0, FloatPointNearestNeighbor.nearest(s, "point", 1, 40.0f, 50.0f).scoreDocs.length);
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ public void testTieBreakByDocID() throws Exception {
+ Directory dir = newDirectory();
+ IndexWriter w = new IndexWriter(dir, getIndexWriterConfig());
+ Document doc = new Document();
+ doc.add(new FloatPoint("point", 40.0f, 50.0f));
+ doc.add(new StringField("id", "0", Field.Store.YES));
+ w.addDocument(doc);
+ doc = new Document();
+ doc.add(new FloatPoint("point", 40.0f, 50.0f));
+ doc.add(new StringField("id", "1", Field.Store.YES));
+ w.addDocument(doc);
+
+ DirectoryReader r = DirectoryReader.open(w);
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ ScoreDoc[] hits = FloatPointNearestNeighbor.nearest(newSearcher(r, false), "point", 2, 45.0f, 50.0f).scoreDocs;
+ assertEquals("0", r.document(hits[0].doc).getField("id").stringValue());
+ assertEquals("1", r.document(hits[1].doc).getField("id").stringValue());
+
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ public void testNearestNeighborWithNoDocs() throws Exception {
+ Directory dir = newDirectory();
+ RandomIndexWriter w = new RandomIndexWriter(random(), dir, getIndexWriterConfig());
+ DirectoryReader r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ assertEquals(0, FloatPointNearestNeighbor.nearest(newSearcher(r, false), "point", 1, 40.0f, 50.0f).scoreDocs.length);
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ public void testNearestNeighborRandom() throws Exception {
+ Directory dir;
+ int numPoints = atLeast(5000);
+ if (numPoints > 100000) {
+ dir = newFSDirectory(createTempDir(getClass().getSimpleName()));
+ } else {
+ dir = newDirectory();
+ }
+ IndexWriterConfig iwc = getIndexWriterConfig();
+ iwc.setMergePolicy(newLogMergePolicy());
+ iwc.setMergeScheduler(new SerialMergeScheduler());
+ RandomIndexWriter w = new RandomIndexWriter(random(), dir, iwc);
+
+ int dims = TestUtil.nextInt(random(), 1, PointValues.MAX_DIMENSIONS);
+ float[][] values = new float[numPoints][dims];
+ for (int id = 0 ; id < numPoints ; ++id) {
+ for (int dim = 0 ; dim < dims ; ++dim) {
+ Float f = Float.NaN;
+ while (f.isNaN()) {
+ f = Float.intBitsToFloat(random().nextInt());
+ }
+ values[id][dim] = f;
+ }
+ Document doc = new Document();
+ doc.add(new FloatPoint("point", values[id]));
+ doc.add(new StoredField("id", id));
+ w.addDocument(doc);
+ }
+
+ if (random().nextBoolean()) {
+ w.forceMerge(1);
+ }
+
+ DirectoryReader r = w.getReader();
+ if (VERBOSE) {
+ System.out.println("TEST: reader=" + r);
+ }
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ IndexSearcher s = newSearcher(r, false);
+ int iters = atLeast(100);
+ for (int iter = 0 ; iter < iters ; ++iter) {
+ if (VERBOSE) {
+ System.out.println("\nTEST: iter=" + iter);
+ }
+ float[] origin = new float[dims];
+ for (int dim = 0 ; dim < dims ; ++dim) {
+ Float f = Float.NaN;
+ while (f.isNaN()) {
+ f = Float.intBitsToFloat(random().nextInt());
+ }
+ origin[dim] = f;
+ }
+
+ // dumb brute force search to get the expected result:
+ FloatPointNearestNeighbor.NearestHit[] expectedHits = new FloatPointNearestNeighbor.NearestHit[numPoints];
+ for (int id = 0 ; id < numPoints ; ++id) {
+ FloatPointNearestNeighbor.NearestHit hit = new FloatPointNearestNeighbor.NearestHit();
+ hit.distanceSquared = FloatPointNearestNeighbor.euclideanDistanceSquared(origin, values[id]);
+ hit.docID = id;
+ expectedHits[id] = hit;
+ }
+
+ Arrays.sort(expectedHits, (a, b) -> {
+ int cmp = Double.compare(a.distanceSquared, b.distanceSquared);
+ return cmp != 0 ? cmp : a.docID - b.docID; // tie break by smaller id
+ });
+
+ int topK = TestUtil.nextInt(random(), 1, numPoints);
+
+ if (VERBOSE) {
+ System.out.println("\nhits for origin=" + Arrays.toString(origin));
+ }
+
+ ScoreDoc[] hits = FloatPointNearestNeighbor.nearest(s, "point", topK, origin).scoreDocs;
+ assertEquals("fewer than expected hits: ", topK, hits.length);
+
+ if (VERBOSE) {
+ for (int i = 0 ; i < topK ; ++i) {
+ FloatPointNearestNeighbor.NearestHit expected = expectedHits[i];
+ FieldDoc actual = (FieldDoc)hits[i];
+ Document actualDoc = r.document(actual.doc);
+ System.out.println("hit " + i);
+ System.out.println(" expected id=" + expected.docID + " " + Arrays.toString(values[expected.docID])
+ + " distance=" + (float)Math.sqrt(expected.distanceSquared) + " distanceSquared=" + expected.distanceSquared);
+ System.out.println(" actual id=" + actualDoc.getField("id") + " distance=" + actual.fields[0]);
+ }
+ }
+
+ for (int i = 0 ; i < topK ; ++i) {
+ FloatPointNearestNeighbor.NearestHit expected = expectedHits[i];
+ FieldDoc actual = (FieldDoc)hits[i];
+ assertEquals("hit " + i + ":", expected.docID, actual.doc);
+ assertEquals("hit " + i + ":", (float)Math.sqrt(expected.distanceSquared), (Float)actual.fields[0], 0.000001);
+ }
+ }
+
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ private IndexWriterConfig getIndexWriterConfig() {
+ IndexWriterConfig iwc = newIndexWriterConfig();
+ iwc.setCodec(Codec.forName("Lucene70"));
+ return iwc;
+ }
+}
[47/50] lucene-solr:feature/autoscaling_72: LUCENE-7985: maven build:
update forbiddenapis to 2.4.1
Posted by ab...@apache.org.
LUCENE-7985: maven build: update forbiddenapis to 2.4.1
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/080232f3
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/080232f3
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/080232f3
Branch: refs/heads/feature/autoscaling_72
Commit: 080232f3d1766da66bb1883378692409c89e986b
Parents: 93d8e42
Author: Steve Rowe <sa...@gmail.com>
Authored: Wed Oct 4 23:12:32 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Wed Oct 4 23:12:32 2017 -0400
----------------------------------------------------------------------
dev-tools/maven/pom.xml.template | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/080232f3/dev-tools/maven/pom.xml.template
----------------------------------------------------------------------
diff --git a/dev-tools/maven/pom.xml.template b/dev-tools/maven/pom.xml.template
index c676f6e..832d6b5 100644
--- a/dev-tools/maven/pom.xml.template
+++ b/dev-tools/maven/pom.xml.template
@@ -161,7 +161,7 @@
<plugin>
<groupId>de.thetaphi</groupId>
<artifactId>forbiddenapis</artifactId>
- <version>2.2</version>
+ <version>2.4.1</version>
<configuration>
<!--
This is the default setting, we don't support too new Java versions.
[04/50] lucene-solr:feature/autoscaling_72: SOLR-11415: Add
betaDistribution Stream Evaluator
Posted by ab...@apache.org.
SOLR-11415: Add betaDistribution Stream Evaluator
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/633af794
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/633af794
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/633af794
Branch: refs/heads/feature/autoscaling_72
Commit: 633af79406566b2323f589c83640f13b196a0312
Parents: 97f246b
Author: Joel Bernstein <jb...@apache.org>
Authored: Fri Sep 29 09:57:23 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Fri Sep 29 09:57:45 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/handler/StreamHandler.java | 2 +
.../io/eval/BetaDistributionEvaluator.java | 48 ++++++++++++++++++++
.../solrj/io/stream/StreamExpressionTest.java | 45 ++++++++++++++++++
3 files changed, 95 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/633af794/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index 127c44a..16a31fb 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -307,6 +307,8 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
.withFunctionName("logNormalDistribution", LogNormalDistributionEvaluator.class)
.withFunctionName("zipFDistribution", ZipFDistributionEvaluator.class)
.withFunctionName("gammaDistribution", GammaDistributionEvaluator.class)
+ .withFunctionName("betaDistribution", BetaDistributionEvaluator.class)
+
// Boolean Stream Evaluators
.withFunctionName("and", AndEvaluator.class)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/633af794/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/BetaDistributionEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/BetaDistributionEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/BetaDistributionEvaluator.java
new file mode 100644
index 0000000..288ecb9
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/BetaDistributionEvaluator.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.util.Locale;
+
+import org.apache.commons.math3.distribution.BetaDistribution;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+
+public class BetaDistributionEvaluator extends RecursiveNumericEvaluator implements TwoValueWorker {
+
+ private static final long serialVersionUID = 1;
+
+ public BetaDistributionEvaluator(StreamExpression expression, StreamFactory factory) throws IOException {
+ super(expression, factory);
+ }
+
+ @Override
+ public Object doWork(Object first, Object second) throws IOException{
+ if(null == first){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the first value",toExpression(constructingFactory)));
+ }
+ if(null == second){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the second value",toExpression(constructingFactory)));
+ }
+
+ Number shape1 = (Number)first;
+ Number shape2 = (Number)second;
+
+ return new BetaDistribution(shape1.doubleValue(), shape2.doubleValue());
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/633af794/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 6f11382..949d224 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6399,6 +6399,51 @@ public class StreamExpressionTest extends SolrCloudTestCase {
}
+ @Test
+ public void testBetaDistribution() throws Exception {
+ String cexpr = "let(a=sample(betaDistribution(1, 5), 50000), b=hist(a, 11), c=col(b, N))";
+ ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ TupleStream solrStream = new SolrStream(url, paramsLoc);
+ StreamContext context = new StreamContext();
+ solrStream.setStreamContext(context);
+ List<Tuple> tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ List<Number> counts = (List<Number>)tuples.get(0).get("c");
+
+ int lastCount = Integer.MAX_VALUE;
+ for(Number number : counts) {
+ int current = number.intValue();
+ if(current > lastCount) {
+ throw new Exception("This beta distribution should be descending");
+ } else {
+ lastCount = current;
+ }
+ }
+
+ cexpr = "let(a=sample(betaDistribution(5, 1), 50000), b=hist(a, 11), c=col(b, N))";
+ paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ solrStream = new SolrStream(url, paramsLoc);
+ context = new StreamContext();
+ solrStream.setStreamContext(context);
+ tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ counts = (List<Number>)tuples.get(0).get("c");
+
+ lastCount = Integer.MIN_VALUE;
+ for(Number number : counts) {
+ int current = number.intValue();
+ if(current < lastCount) {
+ throw new Exception("This beta distribution should be ascending");
+ } else {
+ lastCount = current;
+ }
+ }
+ }
@Test
public void testEnumeratedDistribution() throws Exception {
[46/50] lucene-solr:feature/autoscaling_72: SOLR-10842: Convert all
remaining {{quickstart.html}} links to {{guide/solr-tutorial.html}};
remove all references to quickstart from the build;
and version the link to the ref guide's tutorial in Solr's versio
Posted by ab...@apache.org.
SOLR-10842: Convert all remaining {{quickstart.html}} links to {{guide/solr-tutorial.html}}; remove all references to quickstart from the build; and version the link to the ref guide's tutorial in Solr's versioned top-level documentation page.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/93d8e428
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/93d8e428
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/93d8e428
Branch: refs/heads/feature/autoscaling_72
Commit: 93d8e428ea4643c6641d8a9b2c73827683b831a4
Parents: 5c62fb5
Author: Steve Rowe <sa...@apache.org>
Authored: Wed Oct 4 14:47:11 2017 -0400
Committer: Steve Rowe <sa...@apache.org>
Committed: Wed Oct 4 15:25:20 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/checkJavadocLinks.py | 3 ---
solr/CHANGES.txt | 2 +-
solr/README.txt | 4 ++--
solr/build.xml | 29 ++++++++++++-----------------
solr/contrib/ltr/README.md | 2 +-
solr/example/README.txt | 4 ++--
solr/site/index.xsl | 3 ++-
solr/site/quickstart.mdtext | 6 ------
8 files changed, 20 insertions(+), 33 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/93d8e428/dev-tools/scripts/checkJavadocLinks.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/checkJavadocLinks.py b/dev-tools/scripts/checkJavadocLinks.py
index d3ba89a..8ae0f4c 100644
--- a/dev-tools/scripts/checkJavadocLinks.py
+++ b/dev-tools/scripts/checkJavadocLinks.py
@@ -210,9 +210,6 @@ def checkAll(dirName):
elif link.find('lucene.apache.org/solr/mirrors-solr-latest-redir.html') != -1:
# OK
pass
- elif link.find('lucene.apache.org/solr/quickstart.html') != -1:
- # OK
- pass
elif link.find('lucene.apache.org/solr/guide/') != -1:
# OK
pass
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/93d8e428/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 928d37f..4996a13 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -14,7 +14,7 @@ Getting Started
You need a Java 1.8 VM or later installed.
In this release, there is an example Solr server including a bundled
servlet container in the directory named "example".
-See the Quick Start guide at http://lucene.apache.org/solr/quickstart.html
+See the Solr tutorial at https://lucene.apache.org/solr/guide/solr-tutorial.html
================== 8.0.0 ==================
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/93d8e428/solr/README.txt
----------------------------------------------------------------------
diff --git a/solr/README.txt b/solr/README.txt
index 6af0cc6..0de8b57 100644
--- a/solr/README.txt
+++ b/solr/README.txt
@@ -87,8 +87,8 @@ For more information about Solr examples please read...
* example/README.txt
For more information about the "Solr Home" and Solr specific configuration
- * http://lucene.apache.org/solr/quickstart.html
- For a Quick Start guide
+ * https://lucene.apache.org/solr/guide/solr-tutorial.html
+ For a Solr tutorial
* http://lucene.apache.org/solr/resources.html
For a list of other tutorials and introductory articles.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/93d8e428/solr/build.xml
----------------------------------------------------------------------
diff --git a/solr/build.xml b/solr/build.xml
index 16d4201..9b0bdab 100644
--- a/solr/build.xml
+++ b/solr/build.xml
@@ -191,22 +191,6 @@
depends="javadocs,changes-to-html,process-webpages"/>
<target name="compile-core" depends="compile-solr-core" unless="solr.core.compiled"/>
- <target name="generate-website-quickstart"
- description="Generate a version of the quickstart tutorial suitable for the website, at build/website/quickstart.mdtext">
- <copy file="${common-solr.dir}/site/quickstart.mdtext" tofile="${common-solr.dir}/build/website/quickstart.mdtext"
- overwrite="false" encoding="UTF-8">
- <filterchain>
- <tokenfilter>
- <filetokenizer/>
- <!-- Website images are under /solr/assets/images/ -->
- <replaceregex pattern="src\s*=\s*"images/" replace="src="/solr/assets/images/" flags="gs"/>
- <!-- Redirect to the website's version-specific system requirements page -->
- <replaceregex pattern="\(SYSTEM_REQUIREMENTS.html\)" replace="(/solr/api/SYSTEM_REQUIREMENTS.html)" flags="gs"/>
- </tokenfilter>
- </filterchain>
- </copy>
- </target>
-
<target name="documentation-online" description="Generate a link to the online documentation"
depends="define-solr-javadoc-url">
<xslt in="${ant.file}" out="${javadoc-online.dir}/index.html" style="site/online-link.xsl" force="true">
@@ -226,6 +210,16 @@
<makeurl property="process-webpages.buildfiles" separator="|">
<fileset dir="." includes="core/build.xml,test-framework/build.xml,solrj/build.xml,contrib/**/build.xml"/>
</makeurl>
+
+ <loadresource property="doc-solr-guide-version-path">
+ <propertyresource name="version"/>
+ <filterchain>
+ <tokenfilter>
+ <filetokenizer/>
+ <replaceregex pattern="^(\d+)\.(\d+).*" replace="\1_\2"/>
+ </tokenfilter>
+ </filterchain>
+ </loadresource>
<!--
The XSL input file is ignored completely, but XSL expects one to be given,
so we pass ourself (${ant.file}) here. The list of module build.xmls is given
@@ -239,6 +233,7 @@
<param name="buildfiles" expression="${process-webpages.buildfiles}"/>
<param name="version" expression="${version}"/>
<param name="luceneJavadocUrl" expression="${lucene.javadoc.url}"/>
+ <param name="solrGuideVersion" expression="${doc-solr-guide-version-path}"/>
</xslt>
<markdown todir="${javadoc.dir}">
@@ -676,7 +671,7 @@
<!-- NOTE: must currently exclude deprecated-list due to a javadocs bug (as of 1.7.0_09)
javadocs generates invalid XML if you deprecate a method that takes a parameter
with a generic type -->
- <fileset dir="build/docs" includes="**/*.html" excludes="**/deprecated-list.html,quickstart.html"/>
+ <fileset dir="build/docs" includes="**/*.html" excludes="**/deprecated-list.html"/>
</jtidy-macro>
<echo message="Checking for broken links..."/>
<check-broken-links dir="${javadoc.dir}"/>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/93d8e428/solr/contrib/ltr/README.md
----------------------------------------------------------------------
diff --git a/solr/contrib/ltr/README.md b/solr/contrib/ltr/README.md
index 6b56cdf..6324ecf 100644
--- a/solr/contrib/ltr/README.md
+++ b/solr/contrib/ltr/README.md
@@ -14,7 +14,7 @@ For information on how to get started with solr ltr please see:
For information on how to get started with solr please see:
* [solr/README.txt](../../README.txt)
- * [Solr Quick Start](http://lucene.apache.org/solr/quickstart.html)
+ * [Solr Tutorial](https://lucene.apache.org/solr/guide/solr-tutorial.html)
# How To Contribute
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/93d8e428/solr/example/README.txt
----------------------------------------------------------------------
diff --git a/solr/example/README.txt b/solr/example/README.txt
index 4c8cca1..562c256 100644
--- a/solr/example/README.txt
+++ b/solr/example/README.txt
@@ -48,8 +48,8 @@ For more information about this example please read...
* example/solr/README.txt
For more information about the "Solr Home" and Solr specific configuration
- * http://lucene.apache.org/solr/quickstart.html
- For a Tutorial using this example configuration
+ * https://lucene.apache.org/solr/guide/solr-tutorial.html
+ For a Solr tutorial
* http://wiki.apache.org/solr/SolrResources
For a list of other tutorials and introductory articles.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/93d8e428/solr/site/index.xsl
----------------------------------------------------------------------
diff --git a/solr/site/index.xsl b/solr/site/index.xsl
index b75fb9c..20eeea7 100644
--- a/solr/site/index.xsl
+++ b/solr/site/index.xsl
@@ -23,6 +23,7 @@
<xsl:param name="buildfiles"/>
<xsl:param name="version"/>
<xsl:param name="luceneJavadocUrl"/>
+ <xsl:param name="solrGuideVersion"/>
<!--
NOTE: This template matches the root element of any given input XML document!
@@ -74,7 +75,7 @@
<li><a href="http://wiki.apache.org/solr">Wiki</a>: Additional documentation, especially focused on using Solr.</li>
<li><a href="changes/Changes.html">Changes</a>: List of changes in this release.</li>
<li><a href="SYSTEM_REQUIREMENTS.html">System Requirements</a>: Minimum and supported Java versions.</li>
- <li><a href="quickstart.html">Solr Quick Start</a>: This document covers the basics of running Solr using an example schema, and some sample data.</li>
+ <li><a href="https://lucene.apache.org/solr/guide/{$solrGuideVersion}/solr-tutorial.html">Solr Tutorial</a>: This document covers the basics of running Solr using an example schema, and some sample data.</li>
<li><a href="{$luceneJavadocUrl}index.html">Lucene Documentation</a></li>
</ul>
<h2>API Javadocs</h2>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/93d8e428/solr/site/quickstart.mdtext
----------------------------------------------------------------------
diff --git a/solr/site/quickstart.mdtext b/solr/site/quickstart.mdtext
deleted file mode 100644
index 92f593d..0000000
--- a/solr/site/quickstart.mdtext
+++ /dev/null
@@ -1,6 +0,0 @@
-<!-- As of 2 Oct 2017, this page now automatically redirects to
- the Solr Reference Guide page solr/guide/solr-tutorial.html -->
-
-# Solr Quick Start
-
-Please see the [Solr Tutorial](https://lucene.apache.org/solr/guide/solr-tutorial.html) or additional [Resources](http://lucene.apache.org/solr/resources.html).
[30/50] lucene-solr:feature/autoscaling_72: Merge branch
'feature/autoscaling' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Posted by ab...@apache.org.
Merge branch 'feature/autoscaling' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/17f5a4e4
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/17f5a4e4
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/17f5a4e4
Branch: refs/heads/feature/autoscaling_72
Commit: 17f5a4e47650158575a0abc63253f8eb25f143b1
Parents: 0e679e4
Author: Noble Paul <no...@apache.org>
Authored: Wed Oct 4 18:54:00 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Wed Oct 4 18:54:00 2017 +1030
----------------------------------------------------------------------
solr/CHANGES.txt | 66 ++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 63 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/17f5a4e4/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index c14c9b6..f3382ca 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -51,6 +51,15 @@ Detailed Change List
Upgrade Notes
----------------------
+* 'autoAddReplicas' feature is ported to autoscaling framework. Existing users of this feature should not have
+ to change anything. Note these changes:
+ 1. Behaviour: Changing the autoAddReplicas property from disabled to enabled using MODIFYCOLLECTION API
+ no longer replaces down replicas for the collection immediately. Instead, replicas are only added
+ if a node containing them went down while autoAddReplicas was enabled. The params
+ autoReplicaFailoverBadNodeExpiration and autoReplicaFailoverWorkLoopDelay are no longer used.
+ 2. Deprecations: Enabling/disabling autoAddReplicas cluster wide with the API will be deprecated; use
+ suspend/resume trigger APIs with name='.auto_add_replicas' instead.
+
* SOLR-11195: shard and cluster metric reporter configuration now requires a class attribute.
If a reporter configures the group="shard" attribute then please also configure the
class="org.apache.solr.metrics.reporters.solr.SolrShardReporter" attribute.
@@ -60,7 +69,7 @@ Upgrade Notes
* SOLR-11254: the abstract DocTransformer class now has an abstract score-less transform method variant.
* SOLR-11283: all Stream Evaluators in solrj.io.eval have been refactored to have a simplier and more
- robust structure. This simplifies and condenses the code required to implement a new Evaluator and
+ robust structure. This simplifies and condenses the code required to implement a new Evaluator and
makes it much easier for evaluators to handle differing data types (primitives, objects, arrays,
lists, and so forth). (Dennis Gove)
@@ -69,15 +78,36 @@ Upgrade Notes
New Features
----------------------
+* SOLR-10339: New set-trigger and remove-trigger APIs for autoscaling. (shalin)
+
+* SOLR-10340: New set-listener and remove-listener API for autoscaling. (shalin)
+
+* SOLR-10358: New suspend-trigger and resume-trigger APIs for autoscaling. (shalin)
+
+* SOLR-10376: Implement autoscaling trigger for nodeAdded event. (shalin)
+
+* SOLR-10396: Implement trigger support for nodeLost event type (Cao Manh Dat, shalin)
+
+* SOLR-10496: New ComputePlanAction for autoscaling which uses the policy framework to compute cluster
+ operations upon a trigger fire. (Noble Paul, shalin)
+
+* SOLR-10965: New ExecutePlanAction for autoscaling which executes the operations computed by ComputePlanAction
+ against the cluster. (shalin)
* SOLR-11019: Add addAll Stream Evaluator (Joel Bernstein)
+* SOLR-10996: Implement TriggerListener API (ab, shalin)
+
* SOLR-11046: Add residuals Stream Evaluator (Joel Bernstein)
* SOLR-10858: Make UUIDUpdateProcessorFactory as Runtime URP (Amit Sarkar, noble)
* SOLR-11126: Node level health check handler (Anshum Gupta)
+* SOLR-11031: Implement SystemLogListener for autoscaling (ab)
+
+* SOLR-11205: Any metrics value can be directly accessed in autoscaling policies (noble)
+
* SOLR-11199: Payloads supports an "operator" param. Supported operators are 'or', "phrase" ( default ).
A new "sum" function is also added. Example :
{!payload_score f=payload_field func=sum operator=or}A B C" (Varun Thacker)
@@ -102,6 +132,13 @@ New Features
Bug Fixes
----------------------
+* SOLR-10602: Triggers should be able to restore state from old instances when taking over. (shalin)
+
+* SOLR-10714: OverseerTriggerThread does not start triggers on overseer start until autoscaling
+ config watcher is fired. (shalin)
+
+* SOLR-10738: TriggerAction is initialised even if the trigger is never scheduled. (shalin)
+
* SOLR-10668: fix NPE at sort=childfield(..) .. on absent values (Mikhail Khludnev)
* SOLR-8984: EnumField's error reporting to now indicate the field name in failure log (Lanny Ripple,
@@ -109,6 +146,11 @@ Bug Fixes
* SOLR-11012: Fix three (JavaBinCodec not being closed) Resource Leak warnings. (Christine Poerschke)
+* SOLR-11026: MoveReplicaSuggester must check if the target becomes more 'loaded' than the source
+ if an operation is performed (noble)
+
+* SOLR-10994: CREATE, CREATESHARD & ADDREPLICA to use policy framework & support replica types (noble)
+
* SOLR-11011: Assign.buildCoreName can lead to error in creating a new core when legacyCloud=false (Cao Manh Dat)
* SOLR-10944: Get expression fails to return EOF tuple (Susheel Kumar, Joel Bernstein)
@@ -145,6 +187,8 @@ Bug Fixes
* SOLR-11363: JSON Facet API: repeated values in a numeric points field with docValues enabled
were double counted. (Hossman, yonik)
+* SOLR-11085: Improve resiliency of autoscaling actions against overseer restarts and operation failures. (shalin)
+
* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
(Luiz Armesto, Shawn Heisey, Erick Erickson)
@@ -164,8 +208,12 @@ Optimizations
* SOLR-10985: Remove unnecessary toString() calls in solr-core's search package's debug logging.
(Michael Braun via Christine Poerschke)
+* SOLR-11000: Changes made via AutoScalingHandler should be atomic. (ab, shalin)
+
* SOLR-11124: MoveReplicaCmd should skip deleting old replica in case of its node is not live (Cao Manh Dat)
+* SOLR-10769: Allow nodeAdded / nodeLost events to report multiple nodes in one event. (ab)
+
* SOLR-11242: QueryParser: re-use the LookaheadSuccess exception. (Michael Braun via David Smiley)
* SOLR-11314: FastCharStream: re-use the READ_PAST_EOF exception. (Michael Braun via David Smiley)
@@ -176,6 +224,9 @@ Optimizations
Other Changes
----------------------
+* SOLR-10643: Throttling strategy for triggers and policy executions. (shalin)
+
+* SOLR-10764: AutoScalingHandler should validate policy and preferences before updating zookeeper. (shalin)
* SOLR-10827: Factor out abstract FilteringSolrMetricReporter class. (Christine Poerschke)
@@ -193,10 +244,10 @@ Other Changes
* SOLR-11106: TestLBHttpSolrClient.testReliablity takes 30 seconds because of the wrong server name
(Kensho Hirasawa via Erick Erickson)
-
+
* SOLR-10338: Configure SecureRandom non blocking for tests. (Mihaly Toth, hossman, Ishan Chattopadhyaya, via Mark Miller)
-* SOLR-10916: Convert tests that extend LuceneTestCase and use MiniSolrCloudCluster
+* SOLR-10916: Convert tests that extend LuceneTestCase and use MiniSolrCloudCluster
to instead extend SolrCloudTestCase. (Steve Rowe)
* SOLR-11131: Document 'assert' as a command option in bin/solr, and bin/solr.cmd scripts.
@@ -207,6 +258,11 @@ Other Changes
* SOLR-11187: contrib/ltr TestModelManagerPersistence improvements. (Yuki Yano via Christine Poerschke)
+* SOLR-10397: Port 'autoAddReplicas' feature to the autoscaling framework and make it work with non-shared filesystems
+ (Cao Manh Dat, shalin)
+
+* SOLR-11157: remove-policy must fail if a policy to be deleted is used by a collection (noble)
+
* SOLR-11090: Add Replica.getProperty accessor. (Christine Poerschke)
* SOLR-11061: Add a spins metric for data directory paths. (ab)
@@ -215,8 +271,12 @@ Other Changes
* SOLR-11195: Require class attribute for shard and cluster metric reporter configuration. (Christine Poerschke)
+* SOLR:10822: Share a Policy.session object between multiple collection admin calls to get the correct computations (noble)
+
* SOLR-11249: Upgrade Jetty from 9.3.14.v20161028 to 9.3.20.v20170531 (Michael Braun via David Smiley)
+* SOLR-11270: Automatically added SystemLogListener should not prevent deleting a trigger. (ab)
+
* SOLR-5129: Timeout property for waiting ZK get started. (Cao Manh Dat, Hrishikesh Gadre, Varun Thacker)
* SOLR-10628: Less verbose output from bin/solr commands. (Jason Gerlowski, janhoy)
[29/50] lucene-solr:feature/autoscaling_72: Merge branch
'feature/autoscaling' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Posted by ab...@apache.org.
Merge branch 'feature/autoscaling' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/0e679e4e
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/0e679e4e
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/0e679e4e
Branch: refs/heads/feature/autoscaling_72
Commit: 0e679e4edf6a0ac5788876c4df28a7ef5143531c
Parents: fb19528 2c873cd
Author: Noble Paul <no...@apache.org>
Authored: Wed Oct 4 18:53:51 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Wed Oct 4 18:53:51 2017 +1030
----------------------------------------------------------------------
.../morfologik-ukrainian-search-3.7.5.jar.sha1 | 1 +
.../org/apache/solr/cloud/ActionThrottle.java | 25 +-
.../org/apache/solr/cloud/AddReplicaCmd.java | 36 +-
.../src/java/org/apache/solr/cloud/Assign.java | 107 +-
.../java/org/apache/solr/cloud/CloudUtil.java | 7 +-
.../apache/solr/cloud/CreateCollectionCmd.java | 23 +-
.../org/apache/solr/cloud/CreateShardCmd.java | 148 ++-
.../apache/solr/cloud/DeleteCollectionCmd.java | 3 +
.../java/org/apache/solr/cloud/Overseer.java | 33 +-
.../OverseerAutoReplicaFailoverThread.java | 531 --------
.../cloud/OverseerCollectionMessageHandler.java | 10 +-
.../java/org/apache/solr/cloud/RestoreCmd.java | 218 ++--
.../org/apache/solr/cloud/SplitShardCmd.java | 7 +-
.../org/apache/solr/cloud/ZkController.java | 71 +-
.../solr/cloud/autoscaling/ActionContext.java | 68 ++
.../autoscaling/AutoAddReplicasPlanAction.java | 51 +
.../solr/cloud/autoscaling/AutoScaling.java | 170 +++
.../cloud/autoscaling/AutoScalingHandler.java | 573 +++++++--
.../cloud/autoscaling/ComputePlanAction.java | 115 ++
.../cloud/autoscaling/ExecutePlanAction.java | 177 +++
.../cloud/autoscaling/HttpTriggerListener.java | 178 +++
.../solr/cloud/autoscaling/LoggingListener.java | 38 +
.../cloud/autoscaling/NodeAddedTrigger.java | 320 +++++
.../solr/cloud/autoscaling/NodeLostTrigger.java | 318 +++++
.../autoscaling/OverseerTriggerThread.java | 369 ++++++
.../cloud/autoscaling/ScheduledTriggers.java | 638 ++++++++++
.../cloud/autoscaling/SystemLogListener.java | 219 ++++
.../solr/cloud/autoscaling/TriggerAction.java | 31 +
.../cloud/autoscaling/TriggerActionBase.java | 47 +
.../solr/cloud/autoscaling/TriggerBase.java | 107 ++
.../solr/cloud/autoscaling/TriggerEvent.java | 155 +++
.../cloud/autoscaling/TriggerEventQueue.java | 101 ++
.../solr/cloud/autoscaling/TriggerListener.java | 47 +
.../cloud/autoscaling/TriggerListenerBase.java | 47 +
.../solr/cloud/overseer/ReplicaMutator.java | 1 +
.../org/apache/solr/core/BlobRepository.java | 12 +-
.../java/org/apache/solr/core/CloudConfig.java | 37 +-
.../org/apache/solr/core/CoreContainer.java | 23 +-
.../org/apache/solr/core/MemClassLoader.java | 3 +-
.../src/java/org/apache/solr/core/SolrCore.java | 4 +-
.../org/apache/solr/core/SolrXmlConfig.java | 8 +-
.../org/apache/solr/handler/BlobHandler.java | 1 +
.../admin/AutoscalingHistoryHandler.java | 132 ++
.../solr/handler/admin/CollectionsHandler.java | 24 +-
.../solr/handler/admin/MetricsHandler.java | 11 +-
.../solr/security/PermissionNameProvider.java | 1 +
.../org/apache/solr/servlet/HttpSolrCall.java | 8 +-
.../src/java/org/apache/solr/util/IdUtils.java | 39 +
.../src/java/org/apache/solr/util/SolrCLI.java | 3 +-
.../java/org/apache/solr/util/TimeSource.java | 57 +
.../src/resources/SystemCollectionSchema.xml | 27 +-
.../apache/solr/cloud/ActionThrottleTest.java | 21 +-
.../apache/solr/cloud/ClusterStateMockUtil.java | 48 +-
.../solr/cloud/MoveReplicaHDFSFailoverTest.java | 207 ++++
.../solr/cloud/MoveReplicaHDFSUlogDirTest.java | 142 ---
.../org/apache/solr/cloud/NodeMutatorTest.java | 15 +-
...verseerCollectionConfigSetProcessorTest.java | 3 +
.../org/apache/solr/cloud/OverseerTest.java | 26 +-
.../org/apache/solr/cloud/ShardSplitTest.java | 1 -
.../cloud/SharedFSAutoReplicaFailoverTest.java | 215 ++--
.../SharedFSAutoReplicaFailoverUtilsTest.java | 191 ---
.../AutoAddReplicasIntegrationTest.java | 182 +++
.../AutoAddReplicasPlanActionTest.java | 195 +++
.../autoscaling/AutoScalingHandlerTest.java | 522 +++++++-
.../autoscaling/ComputePlanActionTest.java | 372 ++++++
.../autoscaling/ExecutePlanActionTest.java | 221 ++++
.../HdfsAutoAddReplicasIntegrationTest.java | 56 +
.../autoscaling/HttpTriggerListenerTest.java | 209 ++++
.../cloud/autoscaling/NodeAddedTriggerTest.java | 306 +++++
.../cloud/autoscaling/NodeLostTriggerTest.java | 348 ++++++
.../autoscaling/SystemLogListenerTest.java | 239 ++++
.../solr/cloud/autoscaling/TestPolicyCloud.java | 113 +-
.../autoscaling/TriggerIntegrationTest.java | 1139 ++++++++++++++++++
.../solr/core/BlobRepositoryCloudTest.java | 3 +-
.../admin/AutoscalingHistoryHandlerTest.java | 294 +++++
.../solr/handler/admin/MetricsHandlerTest.java | 9 -
solr/solr-ref-guide/src/collections-api.adoc | 1 +
.../src/fonts/Noto_Sans/LICENSE.txt | 404 +++----
.../src/solrcloud-autoscaling-api.adoc | 28 +
.../cloud/autoscaling/AddReplicaSuggester.java | 53 +-
.../cloud/autoscaling/AutoScalingConfig.java | 531 ++++++++
.../client/solrj/cloud/autoscaling/Clause.java | 247 ++--
.../cloud/autoscaling/ClusterDataProvider.java | 2 +-
.../cloud/autoscaling/MoveReplicaSuggester.java | 29 +-
.../solrj/cloud/autoscaling/NoneSuggester.java | 32 +
.../client/solrj/cloud/autoscaling/Operand.java | 23 +-
.../client/solrj/cloud/autoscaling/Policy.java | 305 +++--
.../solrj/cloud/autoscaling/PolicyHelper.java | 157 ++-
.../solrj/cloud/autoscaling/Preference.java | 42 +-
.../solrj/cloud/autoscaling/ReplicaCount.java | 92 ++
.../solrj/cloud/autoscaling/ReplicaInfo.java | 64 +
.../client/solrj/cloud/autoscaling/Row.java | 45 +-
.../autoscaling/TriggerEventProcessorStage.java | 30 +
.../cloud/autoscaling/TriggerEventType.java | 31 +
.../solr/client/solrj/impl/HttpClientUtil.java | 4 +-
.../solr/client/solrj/impl/HttpSolrClient.java | 1 -
.../solrj/impl/SolrClientDataProvider.java | 74 +-
.../solrj/request/CollectionAdminRequest.java | 20 +-
.../java/org/apache/solr/common/MapWriter.java | 21 +
.../apache/solr/common/cloud/ClusterState.java | 15 +
.../solr/common/cloud/ClusterStateUtil.java | 13 +-
.../apache/solr/common/cloud/DocCollection.java | 7 +-
.../solr/common/cloud/LiveNodesListener.java | 38 +
.../org/apache/solr/common/cloud/Replica.java | 14 +-
.../solr/common/cloud/ReplicaPosition.java | 2 +-
.../apache/solr/common/cloud/ZkStateReader.java | 71 +-
.../solr/common/params/AutoScalingParams.java | 54 +
.../common/params/CollectionAdminParams.java | 4 +-
.../apache/solr/common/params/CommonParams.java | 5 +
.../solr/common/util/CommandOperation.java | 2 +-
.../resources/apispec/autoscaling.Commands.json | 147 +++
.../resources/apispec/collections.Commands.json | 5 +
.../collections.collection.Commands.modify.json | 5 +
.../solrj/cloud/autoscaling/TestPolicy.java | 786 +++++++++++-
114 files changed, 11687 insertions(+), 2174 deletions(-)
----------------------------------------------------------------------
[27/50] lucene-solr:feature/autoscaling_72: SOLR-11425:
SolrClientBuilder does not allow infinite timeout (value 0).
Posted by ab...@apache.org.
SOLR-11425: SolrClientBuilder does not allow infinite timeout (value 0).
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/fc429399
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/fc429399
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/fc429399
Branch: refs/heads/feature/autoscaling_72
Commit: fc429399ecd917bd2fd1902f8a47e32d9093a786
Parents: a9fb4dd
Author: markrmiller <ma...@apache.org>
Authored: Mon Oct 2 22:55:37 2017 -0500
Committer: markrmiller <ma...@apache.org>
Committed: Tue Oct 3 09:08:26 2017 -0500
----------------------------------------------------------------------
solr/CHANGES.txt | 1 +
.../solr/client/solrj/impl/SolrClientBuilder.java | 14 ++++++++++----
.../client/solrj/impl/CloudSolrClientBuilderTest.java | 11 +++++++++++
3 files changed, 22 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fc429399/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 177cdf7..c14c9b6 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -156,6 +156,7 @@ Bug Fixes
* SOLR-11278: Fix a race condition in the CDCR bootstrap process which could lead to bootstraps cancelling itself
(Amrit Sarkar, shalin, Varun Thacker)
+* SOLR-11425: SolrClientBuilder does not allow infinite timeout (value 0). (Peter Szantai-Kis via Mark Miller)
Optimizations
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fc429399/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java
index ec149bd..74e981d 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java
@@ -48,10 +48,13 @@ public abstract class SolrClientBuilder<B extends SolrClientBuilder<B>> {
/**
* Tells {@link Builder} that created clients should obey the following timeout when connecting to Solr servers.
+ * <p>
+ * For valid values see {@link org.apache.http.client.config.RequestConfig#getConnectTimeout()}
+ * </p>
*/
public B withConnectionTimeout(int connectionTimeoutMillis) {
- if (connectionTimeoutMillis <= 0) {
- throw new IllegalArgumentException("connectionTimeoutMillis must be a positive integer.");
+ if (connectionTimeoutMillis < 0) {
+ throw new IllegalArgumentException("connectionTimeoutMillis must be a non-negative integer.");
}
this.connectionTimeoutMillis = connectionTimeoutMillis;
@@ -60,10 +63,13 @@ public abstract class SolrClientBuilder<B extends SolrClientBuilder<B>> {
/**
* Tells {@link Builder} that created clients should set the following read timeout on all sockets.
+ * <p>
+ * For valid values see {@link org.apache.http.client.config.RequestConfig#getSocketTimeout()}
+ * </p>
*/
public B withSocketTimeout(int socketTimeoutMillis) {
- if (socketTimeoutMillis <= 0) {
- throw new IllegalArgumentException("socketTimeoutMillis must be a positive integer.");
+ if (socketTimeoutMillis < 0) {
+ throw new IllegalArgumentException("socketTimeoutMillis must be a non-negative integer.");
}
this.socketTimeoutMillis = socketTimeoutMillis;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fc429399/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientBuilderTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientBuilderTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientBuilderTest.java
index 4c38554..97b79ef 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientBuilderTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientBuilderTest.java
@@ -97,4 +97,15 @@ public class CloudSolrClientBuilderTest extends LuceneTestCase {
assertFalse(createdClient.isDirectUpdatesToLeadersOnly());
}
}
+
+ @Test
+ public void test0Timeouts() throws IOException {
+ try(CloudSolrClient createdClient = new Builder()
+ .withZkHost(ANY_ZK_HOST)
+ .withZkChroot(ANY_CHROOT)
+ .withSocketTimeout(0)
+ .withConnectionTimeout(0)
+ .build()) {
+ }
+ }
}
[33/50] lucene-solr:feature/autoscaling_72: license header missing
Posted by ab...@apache.org.
license header missing
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/b09c99f5
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/b09c99f5
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/b09c99f5
Branch: refs/heads/feature/autoscaling_72
Commit: b09c99f516973c10c011c8ad11d2763fee2bea0b
Parents: 34c176d
Author: Noble Paul <no...@apache.org>
Authored: Wed Oct 4 19:05:23 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Wed Oct 4 19:05:23 2017 +1030
----------------------------------------------------------------------
.../cloud/autoscaling/TriggerEventQueue.java | 19 ++++++++++++++++---
1 file changed, 16 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b09c99f5/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
index 36a6d5f..4030cd2 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.solr.cloud.autoscaling;
import java.lang.invoke.MethodHandles;
@@ -14,9 +30,6 @@ import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-/**
- *
- */
public class TriggerEventQueue extends ZkDistributedQueue {
private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
[26/50] lucene-solr:feature/autoscaling_72: LUCENE-7975: change the
default taxonomy facets cache to a faster UTF-8 cache
Posted by ab...@apache.org.
LUCENE-7975: change the default taxonomy facets cache to a faster UTF-8 cache
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a9fb4ddf
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a9fb4ddf
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a9fb4ddf
Branch: refs/heads/feature/autoscaling_72
Commit: a9fb4ddf80f28c5de36459569f1c94a261a70e8e
Parents: fd2b4f3
Author: Mike McCandless <mi...@apache.org>
Authored: Tue Oct 3 09:58:34 2017 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Tue Oct 3 09:58:34 2017 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +
.../directory/DirectoryTaxonomyWriter.java | 15 +-
.../taxonomy/writercache/CategoryPathUtils.java | 81 ----
.../writercache/Cl2oTaxonomyWriterCache.java | 98 ----
.../taxonomy/writercache/CollisionMap.java | 235 ----------
.../writercache/CompactLabelToOrdinal.java | 467 -------------------
.../writercache/UTF8TaxonomyWriterCache.java | 158 +++++++
.../TestConcurrentFacetedIndexing.java | 8 +-
.../directory/TestDirectoryTaxonomyWriter.java | 12 +-
.../writercache/TestCompactLabelToOrdinal.java | 137 ------
.../TestUTF8TaxonomyWriterCache.java | 120 +++++
11 files changed, 298 insertions(+), 1036 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 9cc0fbd..24885d6 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -38,6 +38,9 @@ New Features
* LUCENE-7974: Add FloatPointNearestNeighbor, an N-dimensional FloatPoint
K-nearest-neighbor search implementation. (Steve Rowe)
+* LUCENE-7975: Change the default taxonomy facets cache to a faster
+ byte[] (UTF-8) based cache.
+
Optimizations
* LUCENE-7905: Optimize how OrdinalMap (used by
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyWriter.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyWriter.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyWriter.java
index ffcf47e..00f9b72 100644
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyWriter.java
+++ b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyWriter.java
@@ -39,19 +39,19 @@ import org.apache.lucene.facet.FacetsConfig;
import org.apache.lucene.facet.taxonomy.FacetLabel;
import org.apache.lucene.facet.taxonomy.TaxonomyReader;
import org.apache.lucene.facet.taxonomy.TaxonomyWriter;
-import org.apache.lucene.facet.taxonomy.writercache.Cl2oTaxonomyWriterCache;
+import org.apache.lucene.facet.taxonomy.writercache.UTF8TaxonomyWriterCache;
import org.apache.lucene.facet.taxonomy.writercache.LruTaxonomyWriterCache;
import org.apache.lucene.facet.taxonomy.writercache.TaxonomyWriterCache;
import org.apache.lucene.index.CorruptIndexException; // javadocs
import org.apache.lucene.index.DirectoryReader;
-import org.apache.lucene.index.PostingsEnum;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
-import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
+import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.LeafReader;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.LogByteSizeMergePolicy;
+import org.apache.lucene.index.PostingsEnum;
import org.apache.lucene.index.ReaderManager;
import org.apache.lucene.index.SegmentInfos;
import org.apache.lucene.index.Terms;
@@ -149,7 +149,7 @@ public class DirectoryTaxonomyWriter implements TaxonomyWriter {
* @param cache
* A {@link TaxonomyWriterCache} implementation which determines
* the in-memory caching policy. See for example
- * {@link LruTaxonomyWriterCache} and {@link Cl2oTaxonomyWriterCache}.
+ * {@link LruTaxonomyWriterCache} and {@link UTF8TaxonomyWriterCache}.
* If null or missing, {@link #defaultTaxonomyWriterCache()} is used.
* @throws CorruptIndexException
* if the taxonomy is corrupted.
@@ -291,12 +291,11 @@ public class DirectoryTaxonomyWriter implements TaxonomyWriter {
* Defines the default {@link TaxonomyWriterCache} to use in constructors
* which do not specify one.
* <P>
- * The current default is {@link Cl2oTaxonomyWriterCache} constructed
- * with the parameters (1024, 0.15f, 3), i.e., the entire taxonomy is
- * cached in memory while building it.
+ * The current default is {@link UTF8TaxonomyWriterCache}, i.e.,
+ * the entire taxonomy is cached in memory while building it.
*/
public static TaxonomyWriterCache defaultTaxonomyWriterCache() {
- return new Cl2oTaxonomyWriterCache(1024, 0.15f, 3);
+ return new UTF8TaxonomyWriterCache();
}
/** Create this with {@code OpenMode.CREATE_OR_APPEND}. */
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CategoryPathUtils.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CategoryPathUtils.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CategoryPathUtils.java
deleted file mode 100644
index 497440e..0000000
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CategoryPathUtils.java
+++ /dev/null
@@ -1,81 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.lucene.facet.taxonomy.writercache;
-
-import org.apache.lucene.facet.taxonomy.FacetLabel;
-
-/** Utilities for use of {@link FacetLabel} by {@link CompactLabelToOrdinal}. */
-class CategoryPathUtils {
-
- /** Serializes the given {@link FacetLabel} to the {@link CharBlockArray}. */
- public static void serialize(FacetLabel cp, CharBlockArray charBlockArray) {
- charBlockArray.append((char) cp.length);
- if (cp.length == 0) {
- return;
- }
- for (int i = 0; i < cp.length; i++) {
- charBlockArray.append((char) cp.components[i].length());
- charBlockArray.append(cp.components[i]);
- }
- }
-
- /**
- * Calculates a hash function of a path that was serialized with
- * {@link #serialize(FacetLabel, CharBlockArray)}.
- */
- public static int hashCodeOfSerialized(CharBlockArray charBlockArray, int offset) {
- int length = charBlockArray.charAt(offset++);
- if (length == 0) {
- return 0;
- }
-
- int hash = length;
- for (int i = 0; i < length; i++) {
- int len = charBlockArray.charAt(offset++);
- hash = hash * 31 + charBlockArray.subSequence(offset, offset + len).hashCode();
- offset += len;
- }
- return hash;
- }
-
- /**
- * Check whether the {@link FacetLabel} is equal to the one serialized in
- * {@link CharBlockArray}.
- */
- public static boolean equalsToSerialized(FacetLabel cp, CharBlockArray charBlockArray, int offset) {
- int n = charBlockArray.charAt(offset++);
- if (cp.length != n) {
- return false;
- }
- if (cp.length == 0) {
- return true;
- }
-
- for (int i = 0; i < cp.length; i++) {
- int len = charBlockArray.charAt(offset++);
- if (len != cp.components[i].length()) {
- return false;
- }
- if (!cp.components[i].equals(charBlockArray.subSequence(offset, offset + len))) {
- return false;
- }
- offset += len;
- }
- return true;
- }
-
-}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/Cl2oTaxonomyWriterCache.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/Cl2oTaxonomyWriterCache.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/Cl2oTaxonomyWriterCache.java
deleted file mode 100644
index 03b0a3c..0000000
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/Cl2oTaxonomyWriterCache.java
+++ /dev/null
@@ -1,98 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.lucene.facet.taxonomy.writercache;
-
-import java.util.concurrent.locks.ReadWriteLock;
-import java.util.concurrent.locks.ReentrantReadWriteLock;
-
-import org.apache.lucene.facet.taxonomy.FacetLabel;
-import org.apache.lucene.facet.taxonomy.writercache.TaxonomyWriterCache;
-
-/**
- * {@link TaxonomyWriterCache} using {@link CompactLabelToOrdinal}. Although
- * called cache, it maintains in memory all the mappings from category to
- * ordinal, relying on that {@link CompactLabelToOrdinal} is an efficient
- * mapping for this purpose.
- *
- * @lucene.experimental
- */
-public class Cl2oTaxonomyWriterCache implements TaxonomyWriterCache {
-
- private final ReadWriteLock lock = new ReentrantReadWriteLock();
- private final int initialCapcity, numHashArrays;
- private final float loadFactor;
-
- private volatile CompactLabelToOrdinal cache;
-
- /** Sole constructor. */
- public Cl2oTaxonomyWriterCache(int initialCapcity, float loadFactor, int numHashArrays) {
- this.cache = new CompactLabelToOrdinal(initialCapcity, loadFactor, numHashArrays);
- this.initialCapcity = initialCapcity;
- this.numHashArrays = numHashArrays;
- this.loadFactor = loadFactor;
- }
-
- @Override
- public void clear() {
- lock.writeLock().lock();
- try {
- cache = new CompactLabelToOrdinal(initialCapcity, loadFactor, numHashArrays);
- } finally {
- lock.writeLock().unlock();
- }
- }
-
- @Override
- public synchronized void close() {
- cache = null;
- }
-
- @Override
- public boolean isFull() {
- // This cache is never full
- return false;
- }
-
- @Override
- public int get(FacetLabel categoryPath) {
- lock.readLock().lock();
- try {
- return cache.getOrdinal(categoryPath);
- } finally {
- lock.readLock().unlock();
- }
- }
-
- @Override
- public boolean put(FacetLabel categoryPath, int ordinal) {
- lock.writeLock().lock();
- try {
- cache.addLabel(categoryPath, ordinal);
- // Tell the caller we didn't clear part of the cache, so it doesn't
- // have to flush its on-disk index now
- return false;
- } finally {
- lock.writeLock().unlock();
- }
- }
-
- /** Returns the number of bytes in memory used by this object. */
- public int getMemoryUsage() {
- return cache == null ? 0 : cache.getMemoryUsage();
- }
-
-}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CollisionMap.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CollisionMap.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CollisionMap.java
deleted file mode 100644
index b80f2bd..0000000
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CollisionMap.java
+++ /dev/null
@@ -1,235 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.lucene.facet.taxonomy.writercache;
-
-import java.util.Iterator;
-import java.util.NoSuchElementException;
-
-import org.apache.lucene.facet.taxonomy.FacetLabel;
-
-/**
- * HashMap to store colliding labels. See {@link CompactLabelToOrdinal} for
- * details.
- *
- * @lucene.experimental
- */
-public class CollisionMap {
-
- private int capacity;
- private float loadFactor;
- private int size;
- private int threshold;
-
- static class Entry {
- int offset;
- int cid;
- Entry next;
- int hash;
-
- Entry(int offset, int cid, int h, Entry e) {
- this.offset = offset;
- this.cid = cid;
- this.next = e;
- this.hash = h;
- }
- }
-
- private CharBlockArray labelRepository;
-
- private Entry[] entries;
-
- CollisionMap(CharBlockArray labelRepository) {
- this(16 * 1024, 0.75f, labelRepository);
- }
-
- CollisionMap(int initialCapacity, CharBlockArray labelRepository) {
- this(initialCapacity, 0.75f, labelRepository);
- }
-
- private CollisionMap(int initialCapacity, float loadFactor, CharBlockArray labelRepository) {
- this.labelRepository = labelRepository;
- this.loadFactor = loadFactor;
- this.capacity = CompactLabelToOrdinal.determineCapacity(2, initialCapacity);
-
- this.entries = new Entry[this.capacity];
- this.threshold = (int) (this.capacity * this.loadFactor);
- }
-
- /** How many mappings. */
- public int size() {
- return this.size;
- }
-
- /** How many slots are allocated. */
- public int capacity() {
- return this.capacity;
- }
-
- private void grow() {
- int newCapacity = this.capacity * 2;
- Entry[] newEntries = new Entry[newCapacity];
- Entry[] src = this.entries;
-
- for (int j = 0; j < src.length; j++) {
- Entry e = src[j];
- if (e != null) {
- src[j] = null;
- do {
- Entry next = e.next;
- int hash = e.hash;
- int i = indexFor(hash, newCapacity);
- e.next = newEntries[i];
- newEntries[i] = e;
- e = next;
- } while (e != null);
- }
- }
-
- this.capacity = newCapacity;
- this.entries = newEntries;
- this.threshold = (int) (this.capacity * this.loadFactor);
- }
-
- /** Return the mapping, or {@link
- * LabelToOrdinal#INVALID_ORDINAL} if the label isn't
- * recognized. */
- public int get(FacetLabel label, int hash) {
- int bucketIndex = indexFor(hash, this.capacity);
- Entry e = this.entries[bucketIndex];
-
- while (e != null && !(hash == e.hash && CategoryPathUtils.equalsToSerialized(label, labelRepository, e.offset))) {
- e = e.next;
- }
- if (e == null) {
- return LabelToOrdinal.INVALID_ORDINAL;
- }
-
- return e.cid;
- }
-
- /** Add another mapping. */
- public int addLabel(FacetLabel label, int hash, int cid) {
- int bucketIndex = indexFor(hash, this.capacity);
- for (Entry e = this.entries[bucketIndex]; e != null; e = e.next) {
- if (e.hash == hash && CategoryPathUtils.equalsToSerialized(label, labelRepository, e.offset)) {
- return e.cid;
- }
- }
-
- // new string; add to label repository
- int offset = labelRepository.length();
- CategoryPathUtils.serialize(label, labelRepository);
- addEntry(offset, cid, hash, bucketIndex);
- return cid;
- }
-
- /**
- * This method does not check if the same value is already in the map because
- * we pass in an char-array offset, so so we now that we're in resize-mode
- * here.
- */
- public void addLabelOffset(int hash, int offset, int cid) {
- int bucketIndex = indexFor(hash, this.capacity);
- addEntry(offset, cid, hash, bucketIndex);
- }
-
- private void addEntry(int offset, int cid, int hash, int bucketIndex) {
- Entry e = this.entries[bucketIndex];
- this.entries[bucketIndex] = new Entry(offset, cid, hash, e);
- if (this.size++ >= this.threshold) {
- grow();
- }
- }
-
- Iterator<CollisionMap.Entry> entryIterator() {
- return new EntryIterator(entries, size);
- }
-
- /**
- * Returns index for hash code h.
- */
- static int indexFor(int h, int length) {
- return h & (length - 1);
- }
-
- /**
- * Returns an estimate of the memory usage of this CollisionMap.
- * @return The approximate number of bytes used by this structure.
- */
- int getMemoryUsage() {
- int memoryUsage = 0;
- if (this.entries != null) {
- for (Entry e : this.entries) {
- if (e != null) {
- memoryUsage += (4 * 4);
- for (Entry ee = e.next; ee != null; ee = ee.next) {
- memoryUsage += (4 * 4);
- }
- }
- }
- }
- return memoryUsage;
- }
-
- private static class EntryIterator implements Iterator<Entry> {
- Entry next; // next entry to return
- int index; // current slot
- Entry[] ents;
-
- EntryIterator(Entry[] entries, int size) {
- this.ents = entries;
- Entry[] t = entries;
- int i = t.length;
- Entry n = null;
- if (size != 0) { // advance to first entry
- while (i > 0 && (n = t[--i]) == null) {
- // advance
- }
- }
- this.next = n;
- this.index = i;
- }
-
- @Override
- public boolean hasNext() {
- return this.next != null;
- }
-
- @Override
- public Entry next() {
- Entry e = this.next;
- if (e == null) throw new NoSuchElementException();
-
- Entry n = e.next;
- Entry[] t = ents;
- int i = this.index;
- while (n == null && i > 0) {
- n = t[--i];
- }
- this.index = i;
- this.next = n;
- return e;
- }
-
- @Override
- public void remove() {
- throw new UnsupportedOperationException();
- }
-
- }
-
-}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CompactLabelToOrdinal.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CompactLabelToOrdinal.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CompactLabelToOrdinal.java
deleted file mode 100644
index 4e6173e..0000000
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/CompactLabelToOrdinal.java
+++ /dev/null
@@ -1,467 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.lucene.facet.taxonomy.writercache;
-
-import java.io.BufferedInputStream;
-import java.io.BufferedOutputStream;
-import java.io.DataInputStream;
-import java.io.DataOutputStream;
-import java.io.IOException;
-import java.io.OutputStream;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.Iterator;
-
-import org.apache.lucene.facet.taxonomy.FacetLabel;
-
-/**
- * This is a very efficient LabelToOrdinal implementation that uses a
- * CharBlockArray to store all labels and a configurable number of HashArrays to
- * reference the labels.
- * <p>
- * Since the HashArrays don't handle collisions, a {@link CollisionMap} is used
- * to store the colliding labels.
- * <p>
- * This data structure grows by adding a new HashArray whenever the number of
- * collisions in the {@link CollisionMap} exceeds {@code loadFactor} *
- * {@link #getMaxOrdinal()}. Growing also includes reinserting all colliding
- * labels into the HashArrays to possibly reduce the number of collisions.
- *
- * For setting the {@code loadFactor} see
- * {@link #CompactLabelToOrdinal(int, float, int)}.
- *
- * <p>
- * This data structure has a much lower memory footprint (~30%) compared to a
- * Java HashMap<String, Integer>. It also only uses a small fraction of objects
- * a HashMap would use, thus limiting the GC overhead. Ingestion speed was also
- * ~50% faster compared to a HashMap for 3M unique labels.
- *
- * @lucene.experimental
- */
-public class CompactLabelToOrdinal extends LabelToOrdinal {
-
- /** Default maximum load factor. */
- public static final float DefaultLoadFactor = 0.15f;
-
- static final char TERMINATOR_CHAR = 0xffff;
- private static final int COLLISION = -5;
-
- private HashArray[] hashArrays;
- private CollisionMap collisionMap;
- private CharBlockArray labelRepository;
-
- private int capacity;
- private int threshold;
- private float loadFactor;
-
- /** How many labels. */
- public int sizeOfMap() {
- return this.collisionMap.size();
- }
-
- private CompactLabelToOrdinal() {
- }
-
- /** Sole constructor. */
- public CompactLabelToOrdinal(int initialCapacity, float loadFactor,
- int numHashArrays) {
-
- this.hashArrays = new HashArray[numHashArrays];
-
- this.capacity = determineCapacity((int) Math.pow(2, numHashArrays),
- initialCapacity);
- init();
- this.collisionMap = new CollisionMap(this.labelRepository);
-
- this.counter = 0;
- this.loadFactor = loadFactor;
-
- this.threshold = (int) (this.loadFactor * this.capacity);
- }
-
- static int determineCapacity(int minCapacity, int initialCapacity) {
- int capacity = minCapacity;
- while (capacity < initialCapacity) {
- capacity <<= 1;
- }
- return capacity;
- }
-
- private void init() {
- labelRepository = new CharBlockArray();
- CategoryPathUtils.serialize(new FacetLabel(), labelRepository);
-
- int c = this.capacity;
- for (int i = 0; i < this.hashArrays.length; i++) {
- this.hashArrays[i] = new HashArray(c);
- c /= 2;
- }
- }
-
- @Override
- public void addLabel(FacetLabel label, int ordinal) {
- if (collisionMap.size() > threshold) {
- grow();
- }
-
- int hash = CompactLabelToOrdinal.stringHashCode(label);
- for (int i = 0; i < this.hashArrays.length; i++) {
- if (addLabel(this.hashArrays[i], label, hash, ordinal)) {
- return;
- }
- }
-
- int prevVal = collisionMap.addLabel(label, hash, ordinal);
- if (prevVal != ordinal) {
- throw new IllegalArgumentException("Label already exists: " + label + " prev ordinal " + prevVal);
- }
- }
-
- @Override
- public int getOrdinal(FacetLabel label) {
- if (label == null) {
- return LabelToOrdinal.INVALID_ORDINAL;
- }
-
- int hash = CompactLabelToOrdinal.stringHashCode(label);
- for (int i = 0; i < this.hashArrays.length; i++) {
- int ord = getOrdinal(this.hashArrays[i], label, hash);
- if (ord != COLLISION) {
- return ord;
- }
- }
-
- return this.collisionMap.get(label, hash);
- }
-
- private void grow() {
- HashArray temp = this.hashArrays[this.hashArrays.length - 1];
-
- for (int i = this.hashArrays.length - 1; i > 0; i--) {
- this.hashArrays[i] = this.hashArrays[i - 1];
- }
-
- this.capacity *= 2;
- this.hashArrays[0] = new HashArray(this.capacity);
-
- for (int i = 1; i < this.hashArrays.length; i++) {
- int[] sourceOffsetArray = this.hashArrays[i].offsets;
- int[] sourceCidsArray = this.hashArrays[i].cids;
-
- for (int k = 0; k < sourceOffsetArray.length; k++) {
-
- for (int j = 0; j < i && sourceOffsetArray[k] != 0; j++) {
- int[] targetOffsetArray = this.hashArrays[j].offsets;
- int[] targetCidsArray = this.hashArrays[j].cids;
-
- int newIndex = indexFor(stringHashCode(
- this.labelRepository, sourceOffsetArray[k]),
- targetOffsetArray.length);
- if (targetOffsetArray[newIndex] == 0) {
- targetOffsetArray[newIndex] = sourceOffsetArray[k];
- targetCidsArray[newIndex] = sourceCidsArray[k];
- sourceOffsetArray[k] = 0;
- }
- }
- }
- }
-
- for (int i = 0; i < temp.offsets.length; i++) {
- int offset = temp.offsets[i];
- if (offset > 0) {
- int hash = stringHashCode(this.labelRepository, offset);
- addLabelOffset(hash, temp.cids[i], offset);
- }
- }
-
- CollisionMap oldCollisionMap = this.collisionMap;
- this.collisionMap = new CollisionMap(oldCollisionMap.capacity(),
- this.labelRepository);
- this.threshold = (int) (this.capacity * this.loadFactor);
-
- Iterator<CollisionMap.Entry> it = oldCollisionMap.entryIterator();
- while (it.hasNext()) {
- CollisionMap.Entry e = it.next();
- addLabelOffset(stringHashCode(this.labelRepository, e.offset),
- e.cid, e.offset);
- }
- }
-
- private boolean addLabel(HashArray a, FacetLabel label, int hash, int ordinal) {
- int index = CompactLabelToOrdinal.indexFor(hash, a.offsets.length);
- int offset = a.offsets[index];
-
- if (offset == 0) {
- a.offsets[index] = this.labelRepository.length();
- CategoryPathUtils.serialize(label, labelRepository);
- a.cids[index] = ordinal;
- return true;
- }
-
- return false;
- }
-
- private void addLabelOffset(int hash, int cid, int knownOffset) {
- for (int i = 0; i < this.hashArrays.length; i++) {
- if (addLabelOffsetToHashArray(this.hashArrays[i], hash, cid,
- knownOffset)) {
- return;
- }
- }
-
- this.collisionMap.addLabelOffset(hash, knownOffset, cid);
-
- if (this.collisionMap.size() > this.threshold) {
- grow();
- }
- }
-
- private boolean addLabelOffsetToHashArray(HashArray a, int hash, int ordinal,
- int knownOffset) {
-
- int index = CompactLabelToOrdinal.indexFor(hash, a.offsets.length);
- int offset = a.offsets[index];
-
- if (offset == 0) {
- a.offsets[index] = knownOffset;
- a.cids[index] = ordinal;
- return true;
- }
-
- return false;
- }
-
- private int getOrdinal(HashArray a, FacetLabel label, int hash) {
- if (label == null) {
- return LabelToOrdinal.INVALID_ORDINAL;
- }
-
- int index = indexFor(hash, a.offsets.length);
- int offset = a.offsets[index];
- if (offset == 0) {
- return LabelToOrdinal.INVALID_ORDINAL;
- }
-
- if (CategoryPathUtils.equalsToSerialized(label, labelRepository, offset)) {
- return a.cids[index];
- }
-
- return COLLISION;
- }
-
- /** Returns index for hash code h. */
- static int indexFor(int h, int length) {
- return h & (length - 1);
- }
-
- // static int stringHashCode(String label) {
- // int len = label.length();
- // int hash = 0;
- // int i;
- // for (i = 0; i < len; ++i)
- // hash = 33 * hash + label.charAt(i);
- //
- // hash = hash ^ ((hash >>> 20) ^ (hash >>> 12));
- // hash = hash ^ (hash >>> 7) ^ (hash >>> 4);
- //
- // return hash;
- //
- // }
-
- static int stringHashCode(FacetLabel label) {
- int hash = label.hashCode();
-
- hash = hash ^ ((hash >>> 20) ^ (hash >>> 12));
- hash = hash ^ (hash >>> 7) ^ (hash >>> 4);
-
- return hash;
-
- }
-
- static int stringHashCode(CharBlockArray labelRepository, int offset) {
- int hash = CategoryPathUtils.hashCodeOfSerialized(labelRepository, offset);
- hash = hash ^ ((hash >>> 20) ^ (hash >>> 12));
- hash = hash ^ (hash >>> 7) ^ (hash >>> 4);
- return hash;
- }
-
- // public static boolean equals(CharSequence label, CharBlockArray array,
- // int offset) {
- // // CONTINUE HERE
- // int len = label.length();
- // int bi = array.blockIndex(offset);
- // CharBlockArray.Block b = array.blocks.get(bi);
- // int index = array.indexInBlock(offset);
- //
- // for (int i = 0; i < len; i++) {
- // if (label.charAt(i) != b.chars[index]) {
- // return false;
- // }
- // index++;
- // if (index == b.length) {
- // b = array.blocks.get(++bi);
- // index = 0;
- // }
- // }
- //
- // return b.chars[index] == TerminatorChar;
- // }
-
- /**
- * Returns an estimate of the amount of memory used by this table. Called only in
- * this package. Memory is consumed mainly by three structures: the hash arrays,
- * label repository and collision map.
- */
- int getMemoryUsage() {
- int memoryUsage = 0;
- if (this.hashArrays != null) {
- // HashArray capacity is instance-specific.
- for (HashArray ha : this.hashArrays) {
- // Each has 2 capacity-length arrays of ints.
- memoryUsage += ( ha.capacity * 2 * 4 ) + 4;
- }
- }
- if (this.labelRepository != null) {
- // All blocks are the same size.
- int blockSize = this.labelRepository.blockSize;
- // Each block has room for blockSize UTF-16 chars.
- int actualBlockSize = ( blockSize * 2 ) + 4;
- memoryUsage += this.labelRepository.blocks.size() * actualBlockSize;
- memoryUsage += 8; // Two int values for array as a whole.
- }
- if (this.collisionMap != null) {
- memoryUsage += this.collisionMap.getMemoryUsage();
- }
- return memoryUsage;
- }
-
- /**
- * Opens the file and reloads the CompactLabelToOrdinal. The file it expects
- * is generated from the {@link #flush(Path)} command.
- */
- static CompactLabelToOrdinal open(Path file, float loadFactor,
- int numHashArrays) throws IOException {
- /**
- * Part of the file is the labelRepository, which needs to be rehashed
- * and label offsets re-added to the object. I am unsure as to why we
- * can't just store these off in the file as well, but in keeping with
- * the spirit of the original code, I did it this way. (ssuppe)
- */
- CompactLabelToOrdinal l2o = new CompactLabelToOrdinal();
- l2o.loadFactor = loadFactor;
- l2o.hashArrays = new HashArray[numHashArrays];
-
- DataInputStream dis = null;
- try {
- dis = new DataInputStream(new BufferedInputStream(
- Files.newInputStream(file)));
-
- // TaxiReader needs to load the "counter" or occupancy (L2O) to know
- // the next unique facet. we used to load the delimiter too, but
- // never used it.
- l2o.counter = dis.readInt();
-
- l2o.capacity = determineCapacity((int) Math.pow(2,
- l2o.hashArrays.length), l2o.counter);
- l2o.init();
-
- // now read the chars
- l2o.labelRepository = CharBlockArray.open(dis);
-
- l2o.collisionMap = new CollisionMap(l2o.labelRepository);
-
- // Calculate hash on the fly based on how CategoryPath hashes
- // itself. Maybe in the future we can call some static based methods
- // in CategoryPath so that this doesn't break again? I don't like
- // having code in two different places...
- int cid = 0;
- // Skip the initial offset, it's the CategoryPath(0,0), which isn't
- // a hashed value.
- int offset = 1;
- int lastStartOffset = offset;
- // This loop really relies on a well-formed input (assumes pretty blindly
- // that array offsets will work). Since the initial file is machine
- // generated, I think this should be OK.
- while (offset < l2o.labelRepository.length()) {
- // identical code to CategoryPath.hashFromSerialized. since we need to
- // advance offset, we cannot call the method directly. perhaps if we
- // could pass a mutable Integer or something...
- int length = (short) l2o.labelRepository.charAt(offset++);
- int hash = length;
- if (length != 0) {
- for (int i = 0; i < length; i++) {
- int len = (short) l2o.labelRepository.charAt(offset++);
- hash = hash * 31 + l2o.labelRepository.subSequence(offset, offset + len).hashCode();
- offset += len;
- }
- }
- // Now that we've hashed the components of the label, do the
- // final part of the hash algorithm.
- hash = hash ^ ((hash >>> 20) ^ (hash >>> 12));
- hash = hash ^ (hash >>> 7) ^ (hash >>> 4);
- // Add the label, and let's keep going
- l2o.addLabelOffset(hash, cid, lastStartOffset);
- cid++;
- lastStartOffset = offset;
- }
-
- } catch (ClassNotFoundException cnfe) {
- throw new IOException("Invalid file format. Cannot deserialize.");
- } finally {
- if (dis != null) {
- dis.close();
- }
- }
-
- l2o.threshold = (int) (l2o.loadFactor * l2o.capacity);
- return l2o;
-
- }
-
- void flush(Path file) throws IOException {
- OutputStream fos = Files.newOutputStream(file);
-
- try {
- BufferedOutputStream os = new BufferedOutputStream(fos);
-
- DataOutputStream dos = new DataOutputStream(os);
- dos.writeInt(this.counter);
-
- // write the labelRepository
- this.labelRepository.flush(dos);
-
- // Closes the data output stream
- dos.close();
-
- } finally {
- fos.close();
- }
- }
-
- private static final class HashArray {
- int[] offsets;
- int[] cids;
-
- int capacity;
-
- HashArray(int c) {
- this.capacity = c;
- this.offsets = new int[this.capacity];
- this.cids = new int[this.capacity];
- }
- }
-}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/UTF8TaxonomyWriterCache.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/UTF8TaxonomyWriterCache.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/UTF8TaxonomyWriterCache.java
new file mode 100644
index 0000000..70bb6ba
--- /dev/null
+++ b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/writercache/UTF8TaxonomyWriterCache.java
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.facet.taxonomy.writercache;
+
+import org.apache.lucene.facet.taxonomy.FacetLabel;
+import org.apache.lucene.facet.taxonomy.writercache.TaxonomyWriterCache;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.ByteBlockPool.DirectTrackingAllocator;
+import org.apache.lucene.util.ByteBlockPool;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.BytesRefBuilder;
+import org.apache.lucene.util.BytesRefHash;
+import org.apache.lucene.util.Counter;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.lucene.util.UnicodeUtil;
+
+/** A "cache" that never frees memory, and stores labels in a BytesRefHash (utf-8 encoding). */
+public final class UTF8TaxonomyWriterCache implements TaxonomyWriterCache, Accountable {
+ private final ThreadLocal<BytesRefBuilder> bytes = new ThreadLocal<BytesRefBuilder>() {
+ @Override
+ protected BytesRefBuilder initialValue() {
+ return new BytesRefBuilder();
+ }
+ };
+ private final Counter bytesUsed = Counter.newCounter();
+ private final BytesRefHash map = new BytesRefHash(new ByteBlockPool(new DirectTrackingAllocator(bytesUsed)));
+
+ private final static int ORDINALS_PAGE_SIZE = 65536;
+ private final static int ORDINALS_PAGE_MASK = ORDINALS_PAGE_SIZE - 1;
+
+ private volatile int[][] ordinals;
+
+ // How many labels we are storing:
+ private int count;
+
+ // How many pages in ordinals we've allocated:
+ private int pageCount;
+
+ /** Sole constructor. */
+ public UTF8TaxonomyWriterCache() {
+ ordinals = new int[1][];
+ ordinals[0] = new int[ORDINALS_PAGE_SIZE];
+ }
+
+ @Override
+ public int get(FacetLabel label) {
+ BytesRef bytes = toBytes(label);
+ int id;
+ synchronized (this) {
+ id = map.find(bytes);
+ }
+ if (id == -1) {
+ return LabelToOrdinal.INVALID_ORDINAL;
+ }
+ int page = id / ORDINALS_PAGE_SIZE;
+ int offset = id % ORDINALS_PAGE_MASK;
+ return ordinals[page][offset];
+ }
+
+ // Called only from assert
+ private boolean assertSameOrdinal(FacetLabel label, int id, int ord) {
+ id = -id - 1;
+ int page = id / ORDINALS_PAGE_SIZE;
+ int offset = id % ORDINALS_PAGE_MASK;
+ int oldOrd = ordinals[page][offset];
+ if (oldOrd != ord) {
+ throw new IllegalArgumentException("label " + label + " was already cached, with old ord=" + oldOrd + " versus new ord=" + ord);
+ }
+ return true;
+ }
+
+ @Override
+ public boolean put(FacetLabel label, int ord) {
+ BytesRef bytes = toBytes(label);
+ int id;
+ synchronized (this) {
+ id = map.add(bytes);
+ if (id < 0) {
+ assert assertSameOrdinal(label, id, ord);
+ return false;
+ }
+ assert id == count;
+ int page = id / ORDINALS_PAGE_SIZE;
+ int offset = id % ORDINALS_PAGE_MASK;
+ if (page == pageCount) {
+ if (page == ordinals.length) {
+ int[][] newOrdinals = new int[ArrayUtil.oversize(page+1, RamUsageEstimator.NUM_BYTES_OBJECT_REF)][];
+ System.arraycopy(ordinals, 0, newOrdinals, 0, ordinals.length);
+ ordinals = newOrdinals;
+ }
+ ordinals[page] = new int[ORDINALS_PAGE_MASK];
+ pageCount++;
+ }
+ ordinals[page][offset] = ord;
+ count++;
+
+ // we never prune from the cache
+ return false;
+ }
+ }
+
+ @Override
+ public boolean isFull() {
+ // we are never full
+ return false;
+ }
+
+ @Override
+ public synchronized void clear() {
+ map.clear();
+ map.reinit();
+ ordinals = new int[1][];
+ ordinals[0] = new int[ORDINALS_PAGE_SIZE];
+ count = 0;
+ pageCount = 0;
+ assert bytesUsed.get() == 0;
+ }
+
+ @Override
+ public synchronized long ramBytesUsed() {
+ return bytesUsed.get() + pageCount * ORDINALS_PAGE_SIZE * RamUsageEstimator.NUM_BYTES_INT;
+ }
+
+ @Override
+ public void close() {
+ }
+
+ private static final byte DELIM_CHAR = (byte) 0x1F;
+
+ private BytesRef toBytes(FacetLabel label) {
+ BytesRefBuilder bytes = this.bytes.get();
+ bytes.clear();
+ for (int i = 0; i < label.length; i++) {
+ String part = label.components[i];
+ if (i > 0) {
+ bytes.append(DELIM_CHAR);
+ }
+ bytes.grow(bytes.length() + UnicodeUtil.maxUTF8Length(part.length()));
+ bytes.setLength(UnicodeUtil.UTF16toUTF8(part, 0, part.length(), bytes.bytes(), bytes.length()));
+ }
+ return bytes.get();
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestConcurrentFacetedIndexing.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestConcurrentFacetedIndexing.java b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestConcurrentFacetedIndexing.java
index d80b410..bf10a3b 100644
--- a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestConcurrentFacetedIndexing.java
+++ b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestConcurrentFacetedIndexing.java
@@ -26,9 +26,9 @@ import org.apache.lucene.facet.FacetField;
import org.apache.lucene.facet.FacetTestCase;
import org.apache.lucene.facet.FacetsConfig;
import org.apache.lucene.facet.taxonomy.FacetLabel;
-import org.apache.lucene.facet.taxonomy.writercache.TaxonomyWriterCache;
-import org.apache.lucene.facet.taxonomy.writercache.Cl2oTaxonomyWriterCache;
import org.apache.lucene.facet.taxonomy.writercache.LruTaxonomyWriterCache;
+import org.apache.lucene.facet.taxonomy.writercache.TaxonomyWriterCache;
+import org.apache.lucene.facet.taxonomy.writercache.UTF8TaxonomyWriterCache;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
import org.apache.lucene.store.Directory;
@@ -66,13 +66,13 @@ public class TestConcurrentFacetedIndexing extends FacetTestCase {
final double d = random().nextDouble();
if (d < 0.7) {
// this is the fastest, yet most memory consuming
- return new Cl2oTaxonomyWriterCache(1024, 0.15f, 3);
+ return new UTF8TaxonomyWriterCache();
} else if (TEST_NIGHTLY && d > 0.98) {
// this is the slowest, but tests the writer concurrency when no caching is done.
// only pick it during NIGHTLY tests, and even then, with very low chances.
return NO_OP_CACHE;
} else {
- // this is slower than CL2O, but less memory consuming, and exercises finding categories on disk too.
+ // this is slower than UTF8, but less memory consuming, and exercises finding categories on disk too.
return new LruTaxonomyWriterCache(ndocs / 10);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyWriter.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyWriter.java b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyWriter.java
index 416b823..1f330e0 100644
--- a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyWriter.java
+++ b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyWriter.java
@@ -25,16 +25,16 @@ import java.util.concurrent.atomic.AtomicInteger;
import org.apache.lucene.analysis.MockAnalyzer;
import org.apache.lucene.document.Document;
+import org.apache.lucene.facet.DrillDownQuery;
import org.apache.lucene.facet.FacetField;
import org.apache.lucene.facet.FacetTestCase;
import org.apache.lucene.facet.FacetsConfig;
-import org.apache.lucene.facet.DrillDownQuery;
import org.apache.lucene.facet.taxonomy.FacetLabel;
import org.apache.lucene.facet.taxonomy.TaxonomyReader;
import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.MemoryOrdinalMap;
-import org.apache.lucene.facet.taxonomy.writercache.TaxonomyWriterCache;
-import org.apache.lucene.facet.taxonomy.writercache.Cl2oTaxonomyWriterCache;
import org.apache.lucene.facet.taxonomy.writercache.LruTaxonomyWriterCache;
+import org.apache.lucene.facet.taxonomy.writercache.TaxonomyWriterCache;
+import org.apache.lucene.facet.taxonomy.writercache.UTF8TaxonomyWriterCache;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
@@ -252,13 +252,13 @@ public class TestDirectoryTaxonomyWriter extends FacetTestCase {
final TaxonomyWriterCache cache;
if (d < 0.7) {
// this is the fastest, yet most memory consuming
- cache = new Cl2oTaxonomyWriterCache(1024, 0.15f, 3);
+ cache = new UTF8TaxonomyWriterCache();
} else if (TEST_NIGHTLY && d > 0.98) {
// this is the slowest, but tests the writer concurrency when no caching is done.
// only pick it during NIGHTLY tests, and even then, with very low chances.
cache = NO_OP_CACHE;
} else {
- // this is slower than CL2O, but less memory consuming, and exercises finding categories on disk too.
+ // this is slower than UTF8, but less memory consuming, and exercises finding categories on disk too.
cache = new LruTaxonomyWriterCache(ncats / 10);
}
if (VERBOSE) {
@@ -441,7 +441,7 @@ public class TestDirectoryTaxonomyWriter extends FacetTestCase {
public void testHugeLabel() throws Exception {
Directory indexDir = newDirectory(), taxoDir = newDirectory();
IndexWriter indexWriter = new IndexWriter(indexDir, newIndexWriterConfig(new MockAnalyzer(random())));
- DirectoryTaxonomyWriter taxoWriter = new DirectoryTaxonomyWriter(taxoDir, OpenMode.CREATE, new Cl2oTaxonomyWriterCache(2, 1f, 1));
+ DirectoryTaxonomyWriter taxoWriter = new DirectoryTaxonomyWriter(taxoDir, OpenMode.CREATE, new UTF8TaxonomyWriterCache());
FacetsConfig config = new FacetsConfig();
// Add one huge label:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/writercache/TestCompactLabelToOrdinal.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/writercache/TestCompactLabelToOrdinal.java b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/writercache/TestCompactLabelToOrdinal.java
deleted file mode 100644
index 801b46f..0000000
--- a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/writercache/TestCompactLabelToOrdinal.java
+++ /dev/null
@@ -1,137 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.lucene.facet.taxonomy.writercache;
-
-import java.nio.ByteBuffer;
-import java.nio.charset.CharsetDecoder;
-import java.nio.charset.CodingErrorAction;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Random;
-
-import org.apache.lucene.facet.FacetTestCase;
-import org.apache.lucene.facet.taxonomy.FacetLabel;
-import org.junit.Test;
-
-public class TestCompactLabelToOrdinal extends FacetTestCase {
-
- @Test
- public void testL2O() throws Exception {
- LabelToOrdinal map = new LabelToOrdinalMap();
-
- CompactLabelToOrdinal compact = new CompactLabelToOrdinal(2000000, 0.15f, 3);
-
- final int n = atLeast(10 * 1000);
- final int numUniqueValues = 50 * 1000;
-
- String[] uniqueValues = new String[numUniqueValues];
- byte[] buffer = new byte[50];
-
- Random random = random();
- for (int i = 0; i < numUniqueValues;) {
- random.nextBytes(buffer);
- int size = 1 + random.nextInt(buffer.length);
-
- // This test is turning random bytes into a string,
- // this is asking for trouble.
- CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder()
- .onUnmappableCharacter(CodingErrorAction.REPLACE)
- .onMalformedInput(CodingErrorAction.REPLACE);
- uniqueValues[i] = decoder.decode(ByteBuffer.wrap(buffer, 0, size)).toString();
- // we cannot have empty path components, so eliminate all prefix as well
- // as middle consecutive delimiter chars.
- uniqueValues[i] = uniqueValues[i].replaceAll("/+", "/");
- if (uniqueValues[i].startsWith("/")) {
- uniqueValues[i] = uniqueValues[i].substring(1);
- }
- if (uniqueValues[i].indexOf(CompactLabelToOrdinal.TERMINATOR_CHAR) == -1) {
- i++;
- }
- }
-
- Path tmpDir = createTempDir("testLableToOrdinal");
- Path f = tmpDir.resolve("CompactLabelToOrdinalTest.tmp");
- int flushInterval = 10;
-
- for (int i = 0; i < n; i++) {
- if (i > 0 && i % flushInterval == 0) {
- compact.flush(f);
- compact = CompactLabelToOrdinal.open(f, 0.15f, 3);
- Files.delete(f);
- if (flushInterval < (n / 10)) {
- flushInterval *= 10;
- }
- }
-
- int index = random.nextInt(numUniqueValues);
- FacetLabel label;
- String s = uniqueValues[index];
- if (s.length() == 0) {
- label = new FacetLabel();
- } else {
- label = new FacetLabel(s.split("/"));
- }
-
- int ord1 = map.getOrdinal(label);
- int ord2 = compact.getOrdinal(label);
-
- assertEquals(ord1, ord2);
-
- if (ord1 == LabelToOrdinal.INVALID_ORDINAL) {
- ord1 = compact.getNextOrdinal();
- map.addLabel(label, ord1);
- compact.addLabel(label, ord1);
- }
- }
-
- for (int i = 0; i < numUniqueValues; i++) {
- FacetLabel label;
- String s = uniqueValues[i];
- if (s.length() == 0) {
- label = new FacetLabel();
- } else {
- label = new FacetLabel(s.split("/"));
- }
- int ord1 = map.getOrdinal(label);
- int ord2 = compact.getOrdinal(label);
- assertEquals(ord1, ord2);
- }
- }
-
- private static class LabelToOrdinalMap extends LabelToOrdinal {
- private Map<FacetLabel, Integer> map = new HashMap<>();
-
- LabelToOrdinalMap() { }
-
- @Override
- public void addLabel(FacetLabel label, int ordinal) {
- map.put(label, ordinal);
- }
-
- @Override
- public int getOrdinal(FacetLabel label) {
- Integer value = map.get(label);
- return (value != null) ? value.intValue() : LabelToOrdinal.INVALID_ORDINAL;
- }
-
- }
-
-}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a9fb4ddf/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/writercache/TestUTF8TaxonomyWriterCache.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/writercache/TestUTF8TaxonomyWriterCache.java b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/writercache/TestUTF8TaxonomyWriterCache.java
new file mode 100644
index 0000000..31a5e49
--- /dev/null
+++ b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/writercache/TestUTF8TaxonomyWriterCache.java
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.facet.taxonomy.writercache;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+
+import org.apache.lucene.facet.FacetTestCase;
+import org.apache.lucene.facet.taxonomy.FacetLabel;
+import org.apache.lucene.util.TestUtil;
+
+public class TestUTF8TaxonomyWriterCache extends FacetTestCase {
+
+ public void testRandom() throws Exception {
+ LabelToOrdinal map = new LabelToOrdinalMap();
+
+ UTF8TaxonomyWriterCache cache = new UTF8TaxonomyWriterCache();
+
+ final int n = atLeast(10 * 1000);
+ final int numUniqueValues = 50 * 1000;
+
+ byte[] buffer = new byte[50];
+
+ Random random = random();
+ Set<String> uniqueValuesSet = new HashSet<>();
+ while (uniqueValuesSet.size() < numUniqueValues) {
+ int numParts = TestUtil.nextInt(random(), 1, 5);
+ StringBuilder b = new StringBuilder();
+ for (int i=0;i<numParts;i++) {
+ String part = null;
+ while (true) {
+ part = TestUtil.randomRealisticUnicodeString(random(), 16);
+ part = part.replace("/", "");
+ if (part.length() > 0) {
+ break;
+ }
+ }
+
+ if (i > 0) {
+ b.append('/');
+ }
+ b.append(part);
+ }
+ uniqueValuesSet.add(b.toString());
+ }
+ String[] uniqueValues = uniqueValuesSet.toArray(new String[0]);
+
+ int ordUpto = 0;
+ for (int i = 0; i < n; i++) {
+
+ int index = random.nextInt(numUniqueValues);
+ FacetLabel label;
+ String s = uniqueValues[index];
+ if (s.length() == 0) {
+ label = new FacetLabel();
+ } else {
+ label = new FacetLabel(s.split("/"));
+ }
+
+ int ord1 = map.getOrdinal(label);
+ int ord2 = cache.get(label);
+
+ assertEquals(ord1, ord2);
+
+ if (ord1 == LabelToOrdinal.INVALID_ORDINAL) {
+ ord1 = ordUpto++;
+ map.addLabel(label, ord1);
+ cache.put(label, ord1);
+ }
+ }
+
+ for (int i = 0; i < numUniqueValues; i++) {
+ FacetLabel label;
+ String s = uniqueValues[i];
+ if (s.length() == 0) {
+ label = new FacetLabel();
+ } else {
+ label = new FacetLabel(s.split("/"));
+ }
+ int ord1 = map.getOrdinal(label);
+ int ord2 = cache.get(label);
+ assertEquals(ord1, ord2);
+ }
+ }
+
+ private static class LabelToOrdinalMap extends LabelToOrdinal {
+ private Map<FacetLabel, Integer> map = new HashMap<>();
+
+ LabelToOrdinalMap() { }
+
+ @Override
+ public void addLabel(FacetLabel label, int ordinal) {
+ map.put(label, ordinal);
+ }
+
+ @Override
+ public int getOrdinal(FacetLabel label) {
+ Integer value = map.get(label);
+ return (value != null) ? value.intValue() : LabelToOrdinal.INVALID_ORDINAL;
+ }
+ }
+}
[24/50] lucene-solr:feature/autoscaling_72: Merge branch 'master' of
https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Posted by ab...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/309ed641
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/309ed641
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/309ed641
Branch: refs/heads/feature/autoscaling_72
Commit: 309ed641ee7ec464ca36883c1c5fbc9a05501eb5
Parents: 310bcd7
Author: Noble Paul <no...@apache.org>
Authored: Tue Oct 3 23:57:29 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Tue Oct 3 23:57:29 2017 +1030
----------------------------------------------------------------------
solr/CHANGES.txt | 46 +++++++++++++++++++-
.../cloud/OverseerCollectionMessageHandler.java | 5 ++-
.../java/org/apache/solr/cloud/RestoreCmd.java | 2 +-
.../org/apache/solr/cloud/SplitShardCmd.java | 1 +
4 files changed, 50 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/309ed641/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 0383f8b..9451d45 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -73,6 +73,9 @@ Upgrade Notes
makes it much easier for evaluators to handle differing data types (primitives, objects, arrays,
lists, and so forth). (Dennis Gove)
+* SOLR-10962: in the ReplicationHandler the master.commitReserveDuration sub-element is deprecated. Instead
+ please configure a direct commitReserveDuration element for use in all modes (master, slave, cloud).
+
New Features
----------------------
* SOLR-10339: New set-trigger and remove-trigger APIs for autoscaling. (shalin)
@@ -119,6 +122,13 @@ New Features
* SOLR-11316: JSON Facet API: min/max aggregations are now supported on single-valued date fields.
(yonik)
+* SOLR-10962: Make ReplicationHandler's commitReserveDuration configurable in SolrCloud mode.
+ (Ramsey Haddad, Christine Poerschke, hossman)
+
+* SOLR-11382: Lucene's Geo3D (surface of sphere & ellipsoid) is now supported on spatial RPT fields by
+ setting spatialContextFactory="Geo3D". Furthermore, this is the first time Solr has out of the box
+ support for polygons. (David Smiley)
+
Bug Fixes
----------------------
@@ -166,7 +176,7 @@ Bug Fixes
* SOLR-11278: Stopping CDCR should cancel a running bootstrap operation. (Amrit Sarkar, shalin)
-* SOLR-11293: Potential data loss in TLOG replicas after replication failures (noble)
+* SOLR-11293: Potential data loss in TLOG replicas when masterVersion equals zero (noble, Cao Manh Dat)
* SOLR-10101: TestLazyCores hangs (Erick Erickson)
@@ -179,6 +189,17 @@ Bug Fixes
* SOLR-11085: Improve resiliency of autoscaling actions against overseer restarts and operation failures. (shalin)
+* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
+ (Luiz Armesto, Shawn Heisey, Erick Erickson)
+
+* SOLR-11399: The UnifiedHighlighter was ignoring the hl.fragsize parameter when hl.bs.type=SEPARATOR
+ (Marc Morissette via David Smiley)
+
+* SOLR-11224: SolrStream.close can hit an NPE (Erick Erickson)
+
+* SOLR-11278: Fix a race condition in the CDCR bootstrap process which could lead to bootstraps cancelling itself
+ (Amrit Sarkar, shalin, Varun Thacker)
+
Optimizations
----------------------
@@ -197,6 +218,8 @@ Optimizations
* SOLR-8344: Decide default when requested fields are both column and row stored. (Cao Manh Dat, David Smiley)
+* SOLR-10285: Skip LEADER messages when there are leader only shards (Cao Manh Dat, Joshua Humphries)
+
Other Changes
----------------------
* SOLR-10643: Throttling strategy for triggers and policy executions. (shalin)
@@ -273,6 +296,27 @@ Other Changes
* SOLR-10451: Remove contrib/ltr/lib from lib includes in the techproducts example config (sungjunyoung via janhoy)
+================== 7.0.1 ==================
+
+Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
+
+Versions of Major Components
+---------------------
+Apache Tika 1.13
+Carrot2 3.15.0
+Velocity 1.7 and Velocity Tools 2.0
+Apache UIMA 2.3.1
+Apache ZooKeeper 3.4.10
+Jetty 9.3.14.v20161028
+
+Bug Fixes
+----------------------
+
+* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
+ (Luiz Armesto, Shawn Heisey, Erick Erickson)
+
+* SOLR-11406: Solr 7.0 cannot read indexes from 6.x versions. (Shawn Heisey, Steve Rowe)
+
================== 7.0.0 ==================
Versions of Major Components
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/309ed641/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java b/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
index 577cf64..509d17c 100644
--- a/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
+++ b/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
@@ -37,6 +37,7 @@ import org.apache.commons.lang.StringUtils;
import org.apache.solr.client.solrj.SolrResponse;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper;
+import org.apache.solr.common.cloud.DistributedQueue;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.HttpSolrClient.RemoteSolrException;
import org.apache.solr.client.solrj.request.AbstractUpdateRequest;
@@ -321,7 +322,7 @@ public class OverseerCollectionMessageHandler implements OverseerMessageHandler
}
private void processReplicaDeletePropertyCommand(ClusterState clusterState, ZkNodeProps message, NamedList results)
- throws KeeperException, InterruptedException {
+ throws Exception {
checkRequired(message, COLLECTION_PROP, SHARD_ID_PROP, REPLICA_PROP, PROPERTY_PROP);
SolrZkClient zkClient = zkStateReader.getZkClient();
DistributedQueue inQueue = Overseer.getStateUpdateQueue(zkClient);
@@ -332,7 +333,7 @@ public class OverseerCollectionMessageHandler implements OverseerMessageHandler
inQueue.offer(Utils.toJSON(m));
}
- private void balanceProperty(ClusterState clusterState, ZkNodeProps message, NamedList results) throws KeeperException, InterruptedException {
+ private void balanceProperty(ClusterState clusterState, ZkNodeProps message, NamedList results) throws Exception {
if (StringUtils.isBlank(message.getStr(COLLECTION_PROP)) || StringUtils.isBlank(message.getStr(PROPERTY_PROP))) {
throw new SolrException(ErrorCode.BAD_REQUEST,
"The '" + COLLECTION_PROP + "' and '" + PROPERTY_PROP +
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/309ed641/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java b/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java
index 9edf729..f2d3794 100644
--- a/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java
@@ -32,8 +32,8 @@ import java.util.Objects;
import java.util.Optional;
import java.util.Properties;
import java.util.Set;
-import java.util.concurrent.atomic.AtomicReference;
+import org.apache.solr.common.cloud.DistributedQueue;
import org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper;
import org.apache.solr.cloud.overseer.OverseerAction;
import org.apache.solr.common.cloud.ReplicaPosition;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/309ed641/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java b/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java
index 3bd6d89..bde92d6 100644
--- a/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java
@@ -27,6 +27,7 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
+import org.apache.solr.common.cloud.DistributedQueue;
import org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper;
import org.apache.solr.client.solrj.request.CoreAdminRequest;
import org.apache.solr.cloud.OverseerCollectionMessageHandler.Cmd;
[32/50] lucene-solr:feature/autoscaling_72: Merge branch 'master' of
https://git-wip-us.apache.org/repos/asf/lucene-solr
Posted by ab...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/34c176da
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/34c176da
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/34c176da
Branch: refs/heads/feature/autoscaling_72
Commit: 34c176da2971230ea1057ea5629bd6138bac0211
Parents: 9905736 17f5a4e
Author: Cao Manh Dat <da...@apache.org>
Authored: Wed Oct 4 15:25:32 2017 +0700
Committer: Cao Manh Dat <da...@apache.org>
Committed: Wed Oct 4 15:25:32 2017 +0700
----------------------------------------------------------------------
.../morfologik-ukrainian-search-3.7.5.jar.sha1 | 1 +
solr/CHANGES.txt | 66 +-
.../org/apache/solr/cloud/ActionThrottle.java | 25 +-
.../org/apache/solr/cloud/AddReplicaCmd.java | 36 +-
.../src/java/org/apache/solr/cloud/Assign.java | 107 +-
.../java/org/apache/solr/cloud/CloudUtil.java | 7 +-
.../apache/solr/cloud/CreateCollectionCmd.java | 23 +-
.../org/apache/solr/cloud/CreateShardCmd.java | 148 ++-
.../apache/solr/cloud/DeleteCollectionCmd.java | 3 +
.../java/org/apache/solr/cloud/Overseer.java | 33 +-
.../OverseerAutoReplicaFailoverThread.java | 531 --------
.../cloud/OverseerCollectionMessageHandler.java | 10 +-
.../java/org/apache/solr/cloud/RestoreCmd.java | 218 ++--
.../org/apache/solr/cloud/SplitShardCmd.java | 7 +-
.../org/apache/solr/cloud/ZkController.java | 71 +-
.../solr/cloud/autoscaling/ActionContext.java | 68 ++
.../autoscaling/AutoAddReplicasPlanAction.java | 51 +
.../solr/cloud/autoscaling/AutoScaling.java | 170 +++
.../cloud/autoscaling/AutoScalingHandler.java | 573 +++++++--
.../cloud/autoscaling/ComputePlanAction.java | 115 ++
.../cloud/autoscaling/ExecutePlanAction.java | 177 +++
.../cloud/autoscaling/HttpTriggerListener.java | 178 +++
.../solr/cloud/autoscaling/LoggingListener.java | 38 +
.../cloud/autoscaling/NodeAddedTrigger.java | 320 +++++
.../solr/cloud/autoscaling/NodeLostTrigger.java | 318 +++++
.../autoscaling/OverseerTriggerThread.java | 369 ++++++
.../cloud/autoscaling/ScheduledTriggers.java | 638 ++++++++++
.../cloud/autoscaling/SystemLogListener.java | 219 ++++
.../solr/cloud/autoscaling/TriggerAction.java | 31 +
.../cloud/autoscaling/TriggerActionBase.java | 47 +
.../solr/cloud/autoscaling/TriggerBase.java | 107 ++
.../solr/cloud/autoscaling/TriggerEvent.java | 155 +++
.../cloud/autoscaling/TriggerEventQueue.java | 101 ++
.../solr/cloud/autoscaling/TriggerListener.java | 47 +
.../cloud/autoscaling/TriggerListenerBase.java | 47 +
.../solr/cloud/overseer/ReplicaMutator.java | 1 +
.../org/apache/solr/core/BlobRepository.java | 12 +-
.../java/org/apache/solr/core/CloudConfig.java | 37 +-
.../org/apache/solr/core/CoreContainer.java | 23 +-
.../org/apache/solr/core/MemClassLoader.java | 3 +-
.../src/java/org/apache/solr/core/SolrCore.java | 4 +-
.../org/apache/solr/core/SolrXmlConfig.java | 8 +-
.../org/apache/solr/handler/BlobHandler.java | 1 +
.../admin/AutoscalingHistoryHandler.java | 132 ++
.../solr/handler/admin/CollectionsHandler.java | 24 +-
.../solr/handler/admin/MetricsHandler.java | 11 +-
.../solr/security/PermissionNameProvider.java | 1 +
.../org/apache/solr/servlet/HttpSolrCall.java | 8 +-
.../src/java/org/apache/solr/util/IdUtils.java | 39 +
.../src/java/org/apache/solr/util/SolrCLI.java | 3 +-
.../java/org/apache/solr/util/TimeSource.java | 57 +
.../src/resources/SystemCollectionSchema.xml | 27 +-
.../apache/solr/cloud/ActionThrottleTest.java | 21 +-
.../apache/solr/cloud/ClusterStateMockUtil.java | 48 +-
.../solr/cloud/MoveReplicaHDFSFailoverTest.java | 207 ++++
.../solr/cloud/MoveReplicaHDFSUlogDirTest.java | 142 ---
.../org/apache/solr/cloud/NodeMutatorTest.java | 15 +-
...verseerCollectionConfigSetProcessorTest.java | 3 +
.../org/apache/solr/cloud/OverseerTest.java | 26 +-
.../org/apache/solr/cloud/ShardSplitTest.java | 1 -
.../cloud/SharedFSAutoReplicaFailoverTest.java | 215 ++--
.../SharedFSAutoReplicaFailoverUtilsTest.java | 191 ---
.../AutoAddReplicasIntegrationTest.java | 182 +++
.../AutoAddReplicasPlanActionTest.java | 195 +++
.../autoscaling/AutoScalingHandlerTest.java | 522 +++++++-
.../autoscaling/ComputePlanActionTest.java | 372 ++++++
.../autoscaling/ExecutePlanActionTest.java | 221 ++++
.../HdfsAutoAddReplicasIntegrationTest.java | 56 +
.../autoscaling/HttpTriggerListenerTest.java | 209 ++++
.../cloud/autoscaling/NodeAddedTriggerTest.java | 306 +++++
.../cloud/autoscaling/NodeLostTriggerTest.java | 348 ++++++
.../autoscaling/SystemLogListenerTest.java | 239 ++++
.../solr/cloud/autoscaling/TestPolicyCloud.java | 113 +-
.../autoscaling/TriggerIntegrationTest.java | 1139 ++++++++++++++++++
.../solr/core/BlobRepositoryCloudTest.java | 3 +-
.../admin/AutoscalingHistoryHandlerTest.java | 294 +++++
.../solr/handler/admin/MetricsHandlerTest.java | 9 -
solr/solr-ref-guide/src/collections-api.adoc | 1 +
.../src/fonts/Noto_Sans/LICENSE.txt | 404 +++----
.../src/solrcloud-autoscaling-api.adoc | 28 +
.../cloud/autoscaling/AddReplicaSuggester.java | 53 +-
.../cloud/autoscaling/AutoScalingConfig.java | 531 ++++++++
.../client/solrj/cloud/autoscaling/Clause.java | 247 ++--
.../cloud/autoscaling/ClusterDataProvider.java | 2 +-
.../cloud/autoscaling/MoveReplicaSuggester.java | 29 +-
.../solrj/cloud/autoscaling/NoneSuggester.java | 32 +
.../client/solrj/cloud/autoscaling/Operand.java | 23 +-
.../client/solrj/cloud/autoscaling/Policy.java | 305 +++--
.../solrj/cloud/autoscaling/PolicyHelper.java | 157 ++-
.../solrj/cloud/autoscaling/Preference.java | 42 +-
.../solrj/cloud/autoscaling/ReplicaCount.java | 92 ++
.../solrj/cloud/autoscaling/ReplicaInfo.java | 64 +
.../client/solrj/cloud/autoscaling/Row.java | 45 +-
.../autoscaling/TriggerEventProcessorStage.java | 30 +
.../cloud/autoscaling/TriggerEventType.java | 31 +
.../solr/client/solrj/impl/HttpClientUtil.java | 4 +-
.../solr/client/solrj/impl/HttpSolrClient.java | 1 -
.../solrj/impl/SolrClientDataProvider.java | 74 +-
.../solrj/request/CollectionAdminRequest.java | 20 +-
.../java/org/apache/solr/common/MapWriter.java | 21 +
.../apache/solr/common/cloud/ClusterState.java | 15 +
.../solr/common/cloud/ClusterStateUtil.java | 13 +-
.../apache/solr/common/cloud/DocCollection.java | 7 +-
.../solr/common/cloud/LiveNodesListener.java | 38 +
.../org/apache/solr/common/cloud/Replica.java | 14 +-
.../solr/common/cloud/ReplicaPosition.java | 2 +-
.../apache/solr/common/cloud/ZkStateReader.java | 71 +-
.../solr/common/params/AutoScalingParams.java | 54 +
.../common/params/CollectionAdminParams.java | 4 +-
.../apache/solr/common/params/CommonParams.java | 5 +
.../solr/common/util/CommandOperation.java | 2 +-
.../resources/apispec/autoscaling.Commands.json | 147 +++
.../resources/apispec/collections.Commands.json | 5 +
.../collections.collection.Commands.modify.json | 5 +
.../solrj/cloud/autoscaling/TestPolicy.java | 786 +++++++++++-
115 files changed, 11750 insertions(+), 2177 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/34c176da/solr/CHANGES.txt
----------------------------------------------------------------------
diff --cc solr/CHANGES.txt
index 0d61cc2,f3382ca..928d37f
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@@ -174,10 -222,11 +222,13 @@@ Optimization
* SOLR-10285: Skip LEADER messages when there are leader only shards (Cao Manh Dat, Joshua Humphries)
+* SOLR-11424: When legacyCloud=false, cores should not publish itself as DOWN on startup. (Cao Manh Dat)
+
Other Changes
----------------------
+ * SOLR-10643: Throttling strategy for triggers and policy executions. (shalin)
+
+ * SOLR-10764: AutoScalingHandler should validate policy and preferences before updating zookeeper. (shalin)
* SOLR-10827: Factor out abstract FilteringSolrMetricReporter class. (Christine Poerschke)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/34c176da/solr/core/src/java/org/apache/solr/cloud/ZkController.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/34c176da/solr/core/src/java/org/apache/solr/core/CoreContainer.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/core/CoreContainer.java
index 4467bc0,dd9e592..c3ce0ed
--- a/solr/core/src/java/org/apache/solr/core/CoreContainer.java
+++ b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
@@@ -1018,10 -1024,10 +1024,10 @@@ public class CoreContainer
MDCLoggingContext.setCoreDescriptor(this, dcore);
SolrIdentifierValidator.validateCoreName(dcore.getName());
if (zkSys.getZkController() != null) {
- zkSys.getZkController().preRegister(dcore);
+ zkSys.getZkController().preRegister(dcore, publishState);
}
- ConfigSet coreConfig = coreConfigService.getConfig(dcore);
+ ConfigSet coreConfig = getConfigSet(dcore);
dcore.setConfigSetTrusted(coreConfig.isTrusted());
log.info("Creating SolrCore '{}' using configuration from {}, trusted={}", dcore.getName(), coreConfig.getName(), dcore.isConfigSetTrusted());
try {
[12/50] lucene-solr:feature/autoscaling_72: addBackcompatIndexes.py:
fix random seed when creating moreterms index
Posted by ab...@apache.org.
addBackcompatIndexes.py: fix random seed when creating moreterms index
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/90842fa0
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/90842fa0
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/90842fa0
Branch: refs/heads/feature/autoscaling_72
Commit: 90842fa0b1bcada723f75420e5f2cc23fdd1f767
Parents: ae5caf3
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 23:37:16 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 23:37:16 2017 -0400
----------------------------------------------------------------------
.../test/org/apache/lucene/index/TestBackwardsCompatibility.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/90842fa0/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index 7437e75..eb6a3e1 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -152,11 +152,10 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
MockAnalyzer analyzer = new MockAnalyzer(random());
analyzer.setMaxTokenLength(TestUtil.nextInt(random(), 1, IndexWriter.MAX_TERM_LENGTH));
- // TODO: remove randomness
IndexWriterConfig conf = new IndexWriterConfig(analyzer)
.setMergePolicy(mp).setUseCompoundFile(false);
IndexWriter writer = new IndexWriter(dir, conf);
- LineFileDocs docs = new LineFileDocs(random());
+ LineFileDocs docs = new LineFileDocs(new Random(0));
for(int i=0;i<50;i++) {
writer.addDocument(docs.nextDoc());
}
[10/50] lucene-solr:feature/autoscaling_72: addBackcompatIndexes.py:
4 space indent instead of 6 for new index names;
check previous line when deciding whether there are previous versions.
Posted by ab...@apache.org.
addBackcompatIndexes.py: 4 space indent instead of 6 for new index names; check previous line when deciding whether there are previous versions.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/0e8d9411
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/0e8d9411
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/0e8d9411
Branch: refs/heads/feature/autoscaling_72
Commit: 0e8d9411b0714e581793689530c5e43ef5939950
Parents: 5afd41d
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 23:20:51 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 23:20:51 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/addBackcompatIndexes.py | 6 +++---
.../org/apache/lucene/index/TestBackwardsCompatibility.java | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0e8d9411/dev-tools/scripts/addBackcompatIndexes.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py
index 1cada01..03e858c 100644
--- a/dev-tools/scripts/addBackcompatIndexes.py
+++ b/dev-tools/scripts/addBackcompatIndexes.py
@@ -121,8 +121,8 @@ def update_backcompat_tests(types, index_version, current_version):
def __call__(self, buffer, match, line):
if self.start:
# find where this version should exist
- i = len(buffer) - 1
- previous_version_exists = '};' not in line # Version list closure => there are no versions
+ i = len(buffer) - 1
+ previous_version_exists = not ('};' in line and buffer[-1].strip().endswith("{"))
if previous_version_exists: # Only look if there is a version here
v = find_version(buffer[i])
while i >= self.start and v.on_or_after(index_version):
@@ -143,7 +143,7 @@ def update_backcompat_tests(types, index_version, current_version):
last = buffer[-1]
spaces = ' ' * (len(last) - len(last.lstrip()))
else:
- spaces = ' '
+ spaces = ' '
for (j, t) in enumerate(types):
if t == 'sorted':
newline = spaces + ('"sorted.%s"') % index_version
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0e8d9411/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index 88900d4..7437e75 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -287,8 +287,8 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
}
final static String[] oldNames = {
- "7.0.0-cfs",
- "7.0.0-nocfs"
+ "7.0.0-cfs",
+ "7.0.0-nocfs"
};
public static String[] getOldNames() {
@@ -296,7 +296,7 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
}
final static String[] oldSortedNames = {
- "sorted.7.0.0"
+ "sorted.7.0.0"
};
public static String[] getOldSortedNames() {
[49/50] lucene-solr:feature/autoscaling_72: SOLR-11440: Potential fix
for better handling of exceptions thrown during closing.
Posted by ab...@apache.org.
SOLR-11440: Potential fix for better handling of exceptions thrown during closing.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/77fc1885
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/77fc1885
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/77fc1885
Branch: refs/heads/feature/autoscaling_72
Commit: 77fc1885c7a35740fdc5f591ecd54e75702f1612
Parents: 6b327fa
Author: Andrzej Bialecki <ab...@apache.org>
Authored: Thu Oct 5 10:45:28 2017 +0200
Committer: Andrzej Bialecki <ab...@apache.org>
Committed: Thu Oct 5 10:45:28 2017 +0200
----------------------------------------------------------------------
.../solr/cloud/autoscaling/ScheduledTriggers.java | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/77fc1885/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
index 7bbbcd9..9dcec35 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java
@@ -37,7 +37,6 @@ import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
@@ -147,18 +146,22 @@ public class ScheduledTriggers implements Closeable {
if (isClosed) {
throw new AlreadyClosedException("ScheduledTriggers has been closed and cannot be used anymore");
}
- ScheduledTrigger st = null;
+ ScheduledTrigger st;
try {
st = new ScheduledTrigger(newTrigger, zkClient, queueStats);
} catch (Exception e) {
- //TODO to decide what to do
- log.error("Failed to add trigger", e);
+ if (isClosed) {
+ throw new AlreadyClosedException("ScheduledTriggers has been closed and cannot be used anymore");
+ }
+ if (!zkClient.isConnected() || zkClient.isClosed()) {
+ log.error("Failed to add trigger " + newTrigger.getName() + " - closing or disconnected from ZK", e);
+ } else {
+ log.error("Failed to add trigger " + newTrigger.getName(), e);
+ }
return;
}
-
ScheduledTrigger scheduledTrigger = st;
-
ScheduledTrigger old = scheduledTriggers.putIfAbsent(newTrigger.getName(), scheduledTrigger);
if (old != null) {
if (old.trigger.equals(newTrigger)) {
[50/50] lucene-solr:feature/autoscaling_72: Merge branch 'master'
into feature/autoscaling_72
Posted by ab...@apache.org.
Merge branch 'master' into feature/autoscaling_72
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/c0abcff1
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/c0abcff1
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/c0abcff1
Branch: refs/heads/feature/autoscaling_72
Commit: c0abcff115108ff032d6026ea2b5271b6f0ceb38
Parents: f28904b 77fc188
Author: Andrzej Bialecki <ab...@apache.org>
Authored: Thu Oct 5 13:55:29 2017 +0200
Committer: Andrzej Bialecki <ab...@apache.org>
Committed: Thu Oct 5 13:55:29 2017 +0200
----------------------------------------------------------------------
.gitignore | 1 +
README.md | 8 +
dev-tools/doap/lucene.rdf | 7 +
dev-tools/doap/solr.rdf | 7 +
.../lucene/backward-codecs/pom.xml.template | 13 +
.../lucene/spatial-extras/pom.xml.template | 7 +
.../maven/lucene/spatial3d/pom.xml.template | 13 +
dev-tools/maven/pom.xml.template | 7 +-
.../maven/solr/core/src/test/pom.xml.template | 7 +
dev-tools/scripts/addBackcompatIndexes.py | 69 +-
dev-tools/scripts/buildAndPushRelease.py | 11 +-
dev-tools/scripts/checkJavadocLinks.py | 2 +-
lucene/CHANGES.txt | 41 +
.../lucene/analysis/synonym/SynonymMap.java | 3 +-
.../index/TestBackwardsCompatibility.java | 49 +-
.../org/apache/lucene/index/dvupdates.7.0.0.zip | Bin 0 -> 13401 bytes
.../org/apache/lucene/index/empty.7.0.0.zip | Bin 0 -> 375 bytes
.../org/apache/lucene/index/index.7.0.0-cfs.zip | Bin 0 -> 15594 bytes
.../apache/lucene/index/index.7.0.0-nocfs.zip | Bin 0 -> 15586 bytes
.../org/apache/lucene/index/moreterms.7.0.0.zip | Bin 0 -> 175822 bytes
.../org/apache/lucene/index/sorted.7.0.0.zip | Bin 0 -> 74558 bytes
.../classification/ClassificationTestBase.java | 1 +
lucene/common-build.xml | 4 +-
.../org/apache/lucene/index/IndexWriter.java | 23 +-
.../lucene/index/SimpleMergedSegmentWarmer.java | 6 +-
.../apache/lucene/search/ConjunctionScorer.java | 4 +-
.../lucene/search/NormsFieldExistsQuery.java | 80 ++
.../java/org/apache/lucene/util/Version.java | 7 +
.../index/TestDemoParallelLeafReader.java | 19 +-
.../lucene/index/TestIndexWriterReader.java | 44 +-
.../apache/lucene/search/TestConjunctions.java | 56 +-
.../lucene/search/TestIntRangeFieldQueries.java | 2 +-
.../search/TestNormsFieldExistsQuery.java | 197 +++
.../directory/DirectoryTaxonomyReader.java | 58 +-
.../directory/DirectoryTaxonomyWriter.java | 17 +-
.../taxonomy/directory/TaxonomyIndexArrays.java | 40 +-
.../taxonomy/writercache/CategoryPathUtils.java | 81 --
.../writercache/Cl2oTaxonomyWriterCache.java | 98 --
.../taxonomy/writercache/CollisionMap.java | 235 ----
.../writercache/CompactLabelToOrdinal.java | 467 -------
.../writercache/UTF8TaxonomyWriterCache.java | 158 +++
.../TestConcurrentFacetedIndexing.java | 8 +-
.../directory/TestDirectoryTaxonomyReader.java | 34 +-
.../directory/TestDirectoryTaxonomyWriter.java | 12 +-
.../writercache/TestCompactLabelToOrdinal.java | 137 --
.../TestUTF8TaxonomyWriterCache.java | 120 ++
lucene/ivy-versions.properties | 8 +-
.../morfologik-ukrainian-search-3.7.6.jar.sha1 | 1 -
.../morfologik-ukrainian-search-3.9.0.jar.sha1 | 1 +
.../apache/lucene/queries/mlt/MoreLikeThis.java | 16 +-
.../nrt/PreCopyMergedSegmentWarmer.java | 4 +-
.../document/FloatPointNearestNeighbor.java | 383 ++++++
.../lucene/document/LatLonBoundingBox.java | 231 ++++
.../org/apache/lucene/document/LatLonPoint.java | 3 +-
.../document/TestFloatPointNearestNeighbor.java | 239 ++++
.../search/TestLatLonBoundingBoxQueries.java | 299 +++++
lucene/site/xsl/index.xsl | 2 +-
lucene/spatial-extras/build.xml | 9 +
.../spatial/spatial4j/Geo3dBinaryCodec.java | 130 ++
.../spatial/spatial4j/Geo3dCircleShape.java | 82 ++
.../spatial4j/Geo3dDistanceCalculator.java | 154 +++
.../spatial/spatial4j/Geo3dPointShape.java | 78 ++
.../spatial/spatial4j/Geo3dRectangleShape.java | 163 +++
.../lucene/spatial/spatial4j/Geo3dShape.java | 164 ++-
.../spatial/spatial4j/Geo3dShapeFactory.java | 396 ++++++
.../spatial4j/Geo3dSpatialContextFactory.java | 94 ++
.../apache/lucene/spatial/SpatialArgsTest.java | 12 +-
.../lucene/spatial/spatial4j/Geo3dRptTest.java | 193 +--
.../Geo3dShapeRectRelationTestCase.java | 20 +-
.../Geo3dShapeSphereModelRectRelationTest.java | 29 +-
.../Geo3dShapeWGS84ModelRectRelationTest.java | 21 +
.../apache/lucene/spatial3d/geom/Bounded.java | 35 +
.../lucene/spatial3d/geom/GeoCircleFactory.java | 16 +
.../lucene/spatial3d/geom/GeoExactCircle.java | 373 ++++++
.../apache/lucene/spatial3d/geom/GeoShape.java | 10 +-
.../lucene/spatial3d/geom/PlanetModel.java | 61 +-
.../lucene/spatial3d/geom/StandardObjects.java | 1 +
.../lucene/spatial3d/geom/GeoCircleTest.java | 83 ++
.../geom/RandomGeo3dShapeGenerator.java | 40 +-
.../java/org/apache/lucene/geo/GeoTestUtil.java | 5 +
.../ThreadedIndexingAndSearchingTestCase.java | 42 +-
.../search/BaseRangeFieldQueryTestCase.java | 10 +-
solr/CHANGES.txt | 142 +-
solr/README.txt | 4 +-
solr/build.xml | 29 +-
solr/common-build.xml | 9 +-
solr/contrib/ltr/README.md | 2 +-
solr/core/build.xml | 3 +-
.../org/apache/solr/cloud/AddReplicaCmd.java | 1 -
.../org/apache/solr/cloud/DeleteShardCmd.java | 1 +
.../org/apache/solr/cloud/DistributedQueue.java | 481 -------
.../org/apache/solr/cloud/ElectionContext.java | 38 +-
.../java/org/apache/solr/cloud/Overseer.java | 100 +-
.../cloud/OverseerCollectionMessageHandler.java | 5 +-
.../apache/solr/cloud/OverseerTaskQueue.java | 4 +-
.../org/apache/solr/cloud/RecoveryStrategy.java | 1 +
.../apache/solr/cloud/ReplicateFromLeader.java | 7 +-
.../java/org/apache/solr/cloud/RestoreCmd.java | 2 +-
.../org/apache/solr/cloud/SplitShardCmd.java | 1 +
.../org/apache/solr/cloud/ZkController.java | 38 +-
.../apache/solr/cloud/ZkDistributedQueue.java | 489 +++++++
.../solr/cloud/autoscaling/AutoScaling.java | 29 +-
.../cloud/autoscaling/AutoScalingHandler.java | 6 +-
.../cloud/autoscaling/ComputePlanAction.java | 55 +-
.../cloud/autoscaling/ExecutePlanAction.java | 105 +-
.../cloud/autoscaling/NodeAddedTrigger.java | 128 +-
.../solr/cloud/autoscaling/NodeLostTrigger.java | 116 +-
.../autoscaling/OverseerTriggerThread.java | 54 +-
.../cloud/autoscaling/ScheduledTriggers.java | 139 +-
.../cloud/autoscaling/SearchRateTrigger.java | 290 -----
.../solr/cloud/autoscaling/TriggerBase.java | 129 +-
.../cloud/autoscaling/TriggerEventQueue.java | 28 +-
.../solr/cloud/overseer/ReplicaMutator.java | 1 -
.../org/apache/solr/core/CoreContainer.java | 45 +-
.../solr/handler/CdcrReplicatorManager.java | 8 +-
.../apache/solr/handler/CdcrRequestHandler.java | 9 +-
.../org/apache/solr/handler/IndexFetcher.java | 16 +-
.../apache/solr/handler/ReplicationHandler.java | 31 +-
.../org/apache/solr/handler/StreamHandler.java | 17 +-
.../admin/AutoscalingHistoryHandler.java | 19 +-
.../solr/handler/admin/MetricsHandler.java | 11 +-
.../solr/handler/admin/PrepRecoveryOp.java | 33 -
.../solr/highlight/UnifiedSolrHighlighter.java | 18 +-
.../solr/index/SlowCompositeReaderWrapper.java | 16 +-
.../apache/solr/metrics/SolrMetricManager.java | 2 +-
.../org/apache/solr/response/DocsStreamer.java | 87 +-
.../solr/response/RetrieveFieldsOptimizer.java | 130 ++
.../solr/schema/AbstractSpatialFieldType.java | 5 +
.../apache/solr/schema/ExternalFileField.java | 1 -
.../apache/solr/search/SolrDocumentFetcher.java | 25 +-
.../apache/solr/search/facet/FacetField.java | 3 +-
.../facet/FacetFieldProcessorByHashDV.java | 11 +-
.../org/apache/solr/search/facet/HLLAgg.java | 1 +
.../org/apache/solr/search/facet/UniqueAgg.java | 1 +
.../solr/search/join/GraphPointsCollector.java | 1 +
.../java/org/apache/solr/update/UpdateLog.java | 12 +-
.../ClassificationUpdateProcessorParams.java | 2 +-
.../processor/DistributedUpdateProcessor.java | 2 +-
.../processor/UpdateRequestProcessor.java | 1 +
.../src/java/org/apache/solr/util/IdUtils.java | 16 +
.../solr/collection1/conf/schema-spatial.xml | 4 +
.../collection1/conf/solrconfig-spatial.xml | 6 +
.../solr/configsets/backcompat/conf/schema.xml | 22 +
.../configsets/backcompat/conf/solrconfig.xml | 43 +
.../backcompat/TestLuceneIndexBackCompat.java | 96 ++
.../apache/solr/cloud/CdcrBootstrapTest.java | 1 -
.../org/apache/solr/cloud/DeleteShardTest.java | 4 +-
.../apache/solr/cloud/DistributedQueueTest.java | 15 +-
.../org/apache/solr/cloud/ForceLeaderTest.java | 4 +-
.../apache/solr/cloud/MultiThreadedOCPTest.java | 5 +-
.../org/apache/solr/cloud/OverseerTest.java | 22 +-
.../cloud/SharedFSAutoReplicaFailoverTest.java | 25 +-
.../cloud/TestCloudJSONFacetJoinDomain.java | 215 +++-
.../cloud/TestRandomRequestDistribution.java | 2 +-
.../solr/cloud/TestSkipOverseerOperations.java | 125 ++
.../org/apache/solr/cloud/TestTlogReplica.java | 50 +-
.../AutoAddReplicasIntegrationTest.java | 2 +
.../AutoAddReplicasPlanActionTest.java | 2 +
.../autoscaling/ComputePlanActionTest.java | 59 +-
.../autoscaling/ExecutePlanActionTest.java | 44 +-
.../cloud/autoscaling/NodeAddedTriggerTest.java | 14 +-
.../cloud/autoscaling/NodeLostTriggerTest.java | 15 +-
.../autoscaling/SearchRateTriggerTest.java | 167 ---
.../admin/AutoscalingHistoryHandlerTest.java | 40 +-
.../solr/handler/admin/MetricsHandlerTest.java | 9 -
.../highlight/TestUnifiedSolrHighlighter.java | 3 +
.../response/TestRetrieveFieldsOptimizer.java | 56 +
.../solr/schema/SpatialRPTFieldTypeTest.java | 12 +-
.../apache/solr/search/TestSolr4Spatial2.java | 21 +-
.../solr/search/facet/TestJsonFacets.java | 26 +
solr/example/README.txt | 4 +-
.../asciidoctor-ant-1.6.0-alpha.3.jar.sha1 | 1 -
.../asciidoctor-ant-1.6.0-alpha.5.jar.sha1 | 1 +
.../conf/solrconfig.xml | 1 -
solr/site/index.xsl | 3 +-
solr/site/quickstart.mdtext | 612 ---------
.../src/basic-authentication-plugin.adoc | 4 +-
solr/solr-ref-guide/src/blob-store-api.adoc | 5 +-
.../src/collapse-and-expand-results.adoc | 20 +-
solr/solr-ref-guide/src/enabling-ssl.adoc | 18 +-
solr/solr-ref-guide/src/errata.adoc | 4 +-
solr/solr-ref-guide/src/faceting.adoc | 4 +-
.../solr-tutorial/tutorial-add-copy-field.png | Bin 123194 -> 93190 bytes
.../images/solr-tutorial/tutorial-add-field.png | Bin 212142 -> 110529 bytes
.../src/implicit-requesthandlers.adoc | 19 +-
solr/solr-ref-guide/src/index-replication.adoc | 4 +-
solr/solr-ref-guide/src/installing-solr.adoc | 2 +-
.../src/major-changes-in-solr-7.adoc | 12 +-
.../src/making-and-restoring-backups.adoc | 5 +-
solr/solr-ref-guide/src/managed-resources.adoc | 2 +-
solr/solr-ref-guide/src/managing-solr.adoc | 2 +-
.../src/mbean-request-handler.adoc | 13 +-
.../src/meta-docs/asciidoc-syntax.adoc | 10 +-
solr/solr-ref-guide/src/metrics-reporting.adoc | 21 +-
.../src/near-real-time-searching.adoc | 8 +-
.../src/pdf/SolrRefGuide-all.adoc | 1 +
.../src/performance-statistics-reference.adoc | 160 ++-
solr/solr-ref-guide/src/query-re-ranking.adoc | 2 +-
solr/solr-ref-guide/src/response-writers.adoc | 28 +-
.../src/rule-based-authorization-plugin.adoc | 23 +-
.../src/running-solr-on-hdfs.adoc | 3 +-
solr/solr-ref-guide/src/solr-tutorial.adoc | 10 +-
.../src/solrcloud-autoscaling-actions.adoc | 28 +
.../src/solrcloud-autoscaling-api.adoc | 4 +-
...solrcloud-autoscaling-auto-add-replicas.adoc | 21 +
.../solrcloud-autoscaling-fault-tolerance.adoc | 60 +
.../src/solrcloud-autoscaling-listeners.adoc | 179 +++
.../src/solrcloud-autoscaling-overview.adoc | 43 +-
.../src/solrcloud-autoscaling-triggers.adoc | 106 ++
.../src/solrcloud-autoscaling.adoc | 7 +-
...rcloud-query-routing-and-read-tolerance.adoc | 6 +-
solr/solr-ref-guide/src/spatial-search.adoc | 71 +-
.../src/statistical-programming.adoc | 190 ++-
.../src/stream-decorator-reference.adoc | 1216 ++++++++++++++++++
solr/solr-ref-guide/src/stream-decorators.adoc | 1216 ------------------
.../src/stream-evaluator-reference.adoc | 966 ++++++++++++++
solr/solr-ref-guide/src/stream-evaluators.adoc | 1030 ---------------
.../src/stream-source-reference.adoc | 539 ++++++++
solr/solr-ref-guide/src/stream-sources.adoc | 540 --------
.../src/streaming-expressions.adoc | 9 +-
.../src/the-extended-dismax-query-parser.adoc | 3 +-
.../src/the-query-elevation-component.adoc | 24 +-
.../solr-ref-guide/src/the-stats-component.adoc | 8 +-
.../src/the-term-vector-component.adoc | 7 +-
.../solr-ref-guide/src/the-terms-component.adoc | 18 +-
.../src/updating-parts-of-documents.adoc | 6 +-
.../src/upgrading-a-solr-cluster.adoc | 18 +-
...g-data-with-solr-cell-using-apache-tika.adoc | 2 -
.../solr-ref-guide/src/using-jmx-with-solr.adoc | 70 +-
solr/solr-ref-guide/src/v2-api.adoc | 5 -
solr/solr-ref-guide/src/working-with-dates.adoc | 14 +-
.../cloud/autoscaling/ClusterDataProvider.java | 4 -
.../solrj/cloud/autoscaling/NoneSuggester.java | 4 +-
.../client/solrj/cloud/autoscaling/Policy.java | 69 +-
.../solrj/cloud/autoscaling/PolicyHelper.java | 6 -
.../solrj/cloud/autoscaling/Preference.java | 8 +
.../solrj/cloud/autoscaling/ReplicaInfo.java | 24 -
.../client/solrj/impl/SolrClientBuilder.java | 14 +-
.../solrj/impl/SolrClientDataProvider.java | 24 +-
.../io/eval/BetaDistributionEvaluator.java | 48 +
.../io/eval/ConstantDistributionEvaluator.java | 44 +
.../eval/EnumeratedDistributionEvaluator.java | 19 +-
.../solrj/io/eval/FactorialEvaluator.java | 54 +
.../io/eval/GammaDistributionEvaluator.java | 48 +
.../io/eval/LogNormalDistributionEvaluator.java | 48 +
.../client/solrj/io/eval/MeanEvaluator.java | 57 +
.../client/solrj/io/eval/ModeEvaluator.java | 64 +
.../solrj/io/eval/MonteCarloEvaluator.java | 76 ++
.../solrj/io/eval/MovingMedianEvaluator.java | 68 +
.../io/eval/PolyFitDerivativeEvaluator.java | 104 ++
.../client/solrj/io/eval/PolyFitEvaluator.java | 102 ++
.../client/solrj/io/eval/SampleEvaluator.java | 30 +-
.../io/eval/WeibullDistributionEvaluator.java | 48 +
.../io/eval/ZipFDistributionEvaluator.java | 48 +
.../solr/client/solrj/io/stream/LetStream.java | 27 +-
.../solr/client/solrj/io/stream/SolrStream.java | 4 +-
.../solr/client/solrj/io/stream/SqlStream.java | 7 +-
.../io/stream/expr/StreamExpressionParser.java | 29 +
.../solrj/io/stream/expr/StreamFactory.java | 6 +
.../java/org/apache/solr/common/MapWriter.java | 1 -
.../solr/common/cloud/DistributedQueue.java | 41 +
.../apache/solr/common/cloud/ZkStateReader.java | 3 +-
.../solr/common/params/AutoScalingParams.java | 7 -
.../solrj/cloud/autoscaling/TestPolicy.java | 96 +-
.../solrj/impl/CloudSolrClientBuilderTest.java | 11 +
.../solrj/io/stream/StreamExpressionTest.java | 394 +++++-
266 files changed, 11604 insertions(+), 7071 deletions(-)
----------------------------------------------------------------------
[02/50] lucene-solr:feature/autoscaling_72: LUCENE-7981 - CTB should
check that CR is not null
Posted by ab...@apache.org.
LUCENE-7981 - CTB should check that CR is not null
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/97f246b4
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/97f246b4
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/97f246b4
Branch: refs/heads/feature/autoscaling_72
Commit: 97f246b4ba3f77f87751a374c587a2c84d3406ab
Parents: f9f6d3a
Author: Tommaso Teofili <to...@apache.org>
Authored: Fri Sep 29 15:34:43 2017 +0200
Committer: Tommaso Teofili <to...@apache.org>
Committed: Fri Sep 29 15:34:52 2017 +0200
----------------------------------------------------------------------
.../org/apache/lucene/classification/ClassificationTestBase.java | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/97f246b4/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java
----------------------------------------------------------------------
diff --git a/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java b/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java
index 6c8f7fd..b3220b0 100644
--- a/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java
+++ b/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java
@@ -88,6 +88,7 @@ public abstract class ClassificationTestBase<T> extends LuceneTestCase {
protected ClassificationResult<T> checkCorrectClassification(Classifier<T> classifier, String inputDoc, T expectedResult) throws Exception {
ClassificationResult<T> classificationResult = classifier.assignClass(inputDoc);
+ assertNotNull(classificationResult);
T assignedClass = classificationResult.getAssignedClass();
assertNotNull(assignedClass);
assertEquals("got an assigned class of " + assignedClass, expectedResult instanceof BytesRef ? ((BytesRef) expectedResult).utf8ToString() : expectedResult, assignedClass instanceof BytesRef ? ((BytesRef) assignedClass).utf8ToString() : assignedClass);
[13/50] lucene-solr:feature/autoscaling_72: Fix my name spelling in
the CHANGES file
Posted by ab...@apache.org.
Fix my name spelling in the CHANGES file
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/472d5202
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/472d5202
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/472d5202
Branch: refs/heads/feature/autoscaling_72
Commit: 472d52022f00774783a611aed9eba6f6b424ae1b
Parents: 90842fa
Author: Alexandre Rafalovitch <ar...@apache.org>
Authored: Sun Oct 1 18:42:21 2017 -0400
Committer: Alexandre Rafalovitch <ar...@apache.org>
Committed: Sun Oct 1 18:42:21 2017 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/472d5202/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index aed0c22..90ada02 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -469,7 +469,7 @@ New Features
* SOLR-10406: v2 API error messages list the URL request path as /solr/____v2/... when the original path was /v2/... (Cao Manh Dat, noble)
* SOLR-10574: New _default config set replacing basic_configs and data_driven_schema_configs.
- (Ishan Chattopadhyaya, noble, shalin, hossman, David Smiley, janhoy, Alexandre Rafalovich)
+ (Ishan Chattopadhyaya, noble, shalin, hossman, David Smiley, janhoy, Alexandre Rafalovitch)
* SOLR-10272: Use _default config set if no collection.configName is specified with CREATE (Ishan Chattopadhyaya)
@@ -477,7 +477,7 @@ New Features
to facilitate both search and faceting. AddSchemaFieldsUpdateProcessor now has the ability to add a "copyField" to
the type mappings, with an optional maxChars limitation. You can also define one typeMappings as default.
This also solves issues SOLR-8495, SOLR-6966, and SOLR-7058
- (janhoy, Steve Rowe, hossman, Alexandre Rafalovich, Shawn Heisey, Cao Manh Dat)
+ (janhoy, Steve Rowe, hossman, Alexandre Rafalovitch, Shawn Heisey, Cao Manh Dat)
* SOLR-10123: Upgraded the Analytics Component to version 2.0 which now supports distributed collections, expressions over
multivalued fields, a new JSON request language, and more. DocValues are now required for any field used in the analytics
[08/50] lucene-solr:feature/autoscaling_72: Add version 7.0.1
Posted by ab...@apache.org.
Add version 7.0.1
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/dc47c6e3
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/dc47c6e3
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/dc47c6e3
Branch: refs/heads/feature/autoscaling_72
Commit: dc47c6e3502e1f8da99cce1f49f05355a395fa77
Parents: 39ba1c3
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 06:58:43 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 06:58:43 2017 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +++
lucene/core/src/java/org/apache/lucene/util/Version.java | 7 +++++++
2 files changed, 10 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/dc47c6e3/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index ae61d86..16482e9 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -95,6 +95,9 @@ Other
* LUCENE-7978: Add some more documentation about setting up build
environment. (Anton R. Yuste via Uwe Schindler)
+======================= Lucene 7.0.1 =======================
+(No Changes)
+
======================= Lucene 7.0.0 =======================
New Features
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/dc47c6e3/lucene/core/src/java/org/apache/lucene/util/Version.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/Version.java b/lucene/core/src/java/org/apache/lucene/util/Version.java
index 0fe4f30..65134fb 100644
--- a/lucene/core/src/java/org/apache/lucene/util/Version.java
+++ b/lucene/core/src/java/org/apache/lucene/util/Version.java
@@ -40,6 +40,13 @@ public final class Version {
public static final Version LUCENE_7_0_0 = new Version(7, 0, 0);
/**
+ * Match settings and bugs in Lucene's 7.0.1 release.
+ * @deprecated Use latest
+ */
+ @Deprecated
+ public static final Version LUCENE_7_0_1 = new Version(7, 0, 1);
+
+ /**
* Match settings and bugs in Lucene's 7.1.0 release.
* @deprecated Use latest
*/
[21/50] lucene-solr:feature/autoscaling_72: SOLR-10842: replace
quickstart.mdtext content with link to Ref Guide tutorial
Posted by ab...@apache.org.
SOLR-10842: replace quickstart.mdtext content with link to Ref Guide tutorial
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5b3a5152
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5b3a5152
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5b3a5152
Branch: refs/heads/feature/autoscaling_72
Commit: 5b3a5152bdf11758b5a02efb60568098fe825d45
Parents: 266cc82
Author: Cassandra Targett <ct...@apache.org>
Authored: Mon Oct 2 17:14:11 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Mon Oct 2 17:14:11 2017 -0500
----------------------------------------------------------------------
solr/site/quickstart.mdtext | 614 +--------------------------------------
1 file changed, 4 insertions(+), 610 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5b3a5152/solr/site/quickstart.mdtext
----------------------------------------------------------------------
diff --git a/solr/site/quickstart.mdtext b/solr/site/quickstart.mdtext
index fcd9be0..92f593d 100644
--- a/solr/site/quickstart.mdtext
+++ b/solr/site/quickstart.mdtext
@@ -1,612 +1,6 @@
-# Solr Quick Start
-
-## Overview
-
-This document covers getting Solr up and running, ingesting a variety of data sources into multiple collections,
-and getting a feel for the Solr administrative and search interfaces.
-
-## Requirements
-
-To follow along with this tutorial, you will need...
-
-1. To meet the [system requirements](SYSTEM_REQUIREMENTS.html)
-2. An Apache Solr release ([download](http://lucene.apache.org/solr/downloads.html)). This tutorial was written using Apache Solr 6.2.0.
-
-## Getting Started
-
-Please run the browser showing this tutorial and the Solr server on the same machine so tutorial links will correctly
-point to your Solr server.
-
-Begin by unzipping the Solr release and changing your working directory to the subdirectory where Solr was installed.
-Note that the base directory name may vary with the version of Solr downloaded. For example, with a shell in UNIX,
-Cygwin, or MacOS:
-
- /:$ ls solr*
- solr-X.Y.Z.zip
- /:$ unzip -q solr-X.Y.Z.zip
- /:$ cd solr-X.Y.Z/
-
-Note that "X.Y.Z" will be replaced by an official Solr version (i.e. 6.4.3, 7.0.0, etc.)
-
-To launch Solr, run: `bin/solr start -e cloud -noprompt`
-
- /solr-X.Y.Z:$ bin/solr start -e cloud -noprompt
-
- Welcome to the SolrCloud example!
-
- Starting up 2 Solr nodes for your example SolrCloud cluster.
- ...
-
- Started Solr server on port 8983 (pid=8404). Happy searching!
- ...
-
- Started Solr server on port 7574 (pid=8549). Happy searching!
- ...
-
- SolrCloud example running, please visit http://localhost:8983/solr
-
- /solr-X.Y.Z:$ _
-
-You can see that the Solr is running by loading the Solr Admin UI in your web browser: <http://localhost:8983/solr/>.
-This is the main starting point for administering Solr.
-
-Solr will now be running two "nodes", one on port 7574 and one on port 8983. There is one collection created
-automatically, `gettingstarted`, a two shard collection, each with two replicas.
-The [Cloud tab](http://localhost:8983/solr/#/~cloud) in the Admin UI diagrams the collection nicely:
-
-<img alt="Solr Quick Start: SolrCloud diagram" style="width:800px" src="images/quickstart-solrcloud.png" />
-
-## Indexing Data
-
-Your Solr server is up and running, but it doesn't contain any data. The Solr install includes the `bin/post` tool in
-order to facilitate getting various types of documents easily into Solr from the start. We'll be
-using this tool for the indexing examples below.
-
-You'll need a command shell to run these examples, rooted in the Solr install directory; the shell from where you
-launched Solr works just fine.
-
-* NOTE: Currently the `bin/post` tool does not have a comparable Windows script, but the underlying Java program invoked
-is available. See the
-[Post Tool, Windows section](https://cwiki.apache.org/confluence/display/solr/Post+Tool#PostTool-Windows)
-for details.
-
-### Indexing a directory of "rich" files
-
-Let's first index local "rich" files including HTML, PDF, Microsoft Office formats (such as MS Word), plain text and
-many other formats. `bin/post` features the ability to crawl a directory of files, optionally recursively even,
-sending the raw content of each file into Solr for extraction and indexing. A Solr install includes a `docs/`
-subdirectory, so that makes a convenient set of (mostly) HTML files built-in to start with.
-
- bin/post -c gettingstarted docs/
-
-Here's what it'll look like:
-
- /solr-X.Y.Z:$ bin/post -c gettingstarted docs/
- java -classpath /solr-X.Y.Z/dist/solr-core-X.Y.Z.jar -Dauto=yes -Dc=gettingstarted -Ddata=files -Drecursive=yes org.apache.solr.util.SimplePostTool docs/
- SimplePostTool version 5.0.0
- Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
- Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
- Entering recursive mode, max depth=999, delay=0s
- Indexing directory docs (3 files, depth=0)
- POSTing file index.html (text/html) to [base]/extract
- POSTing file quickstart.html (text/html) to [base]/extract
- POSTing file SYSTEM_REQUIREMENTS.html (text/html) to [base]/extract
- Indexing directory docs/changes (1 files, depth=1)
- POSTing file Changes.html (text/html) to [base]/extract
- ...
- 4329 files indexed.
- COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
- Time spent: 0:01:16.252
-
-The command-line breaks down as follows:
-
- * `-c gettingstarted`: name of the collection to index into
- * `docs/`: a relative path of the Solr install `docs/` directory
-
-You have now indexed thousands of documents into the `gettingstarted` collection in Solr and committed these changes.
-You can search for "solr" by loading the Admin UI [Query tab](http://localhost:8983/solr/#/gettingstarted/query),
-enter "solr" in the `q` param (replacing `*:*`, which matches all documents), and "Execute Query".
-See the [Searching](#searching) section below for more information.
-
-To index your own data, re-run the directory indexing command pointed to your own directory of documents. For example,
-on a Mac instead of `docs/` try `~/Documents/` or `~/Desktop/`! You may want to start from a clean, empty system
-again rather than have your content in addition to the Solr `docs/` directory; see the Cleanup section [below](#cleanup)
-for how to get back to a clean starting point.
-
-### Indexing Solr XML
-
-Solr supports indexing structured content in a variety of incoming formats. The historically predominant format for
-getting structured content into Solr has been
-[Solr XML](https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-XMLFormattedIndexUpdates).
-Many Solr indexers have been coded to process domain content into Solr XML output, generally HTTP POSTed directly to
-Solr's `/update` endpoint.
-
-<a name="techproducts"></a>
-Solr's install includes a handful of Solr XML formatted files with example data (mostly mocked tech product data).
-NOTE: This tech product data has a more domain-specific configuration, including schema and browse UI. The `bin/solr`
-script includes built-in support for this by running `bin/solr start -e techproducts` which not only starts Solr but
-also then indexes this data too (be sure to `bin/solr stop -all` before trying it out).
-However, the example below assumes Solr was started with `bin/solr start -e cloud` to stay consistent with all examples
-on this page, and thus the collection used is "gettingstarted", not "techproducts".
-
-Using `bin/post`, index the example Solr XML files in `example/exampledocs/`:
-
- bin/post -c gettingstarted example/exampledocs/*.xml
-
-Here's what you'll see:
-
- /solr-X.Y.Z:$ bin/post -c gettingstarted example/exampledocs/*.xml
- java -classpath /solr-X.Y.Z/dist/solr-core-X.Y.Z.jar -Dauto=yes -Dc=gettingstarted -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/gb18030-example.xml ...
- SimplePostTool version 5.0.0
- Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
- Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
- POSTing file gb18030-example.xml (application/xml) to [base]
- POSTing file hd.xml (application/xml) to [base]
- POSTing file ipod_other.xml (application/xml) to [base]
- POSTing file ipod_video.xml (application/xml) to [base]
- POSTing file manufacturers.xml (application/xml) to [base]
- POSTing file mem.xml (application/xml) to [base]
- POSTing file money.xml (application/xml) to [base]
- POSTing file monitor.xml (application/xml) to [base]
- POSTing file monitor2.xml (application/xml) to [base]
- POSTing file mp500.xml (application/xml) to [base]
- POSTing file sd500.xml (application/xml) to [base]
- POSTing file solr.xml (application/xml) to [base]
- POSTing file utf8-example.xml (application/xml) to [base]
- POSTing file vidcard.xml (application/xml) to [base]
- 14 files indexed.
- COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
- Time spent: 0:00:02.077
-
-...and now you can search for all sorts of things using the default
-[Solr Query Syntax](https://cwiki.apache.org/confluence/display/solr/The+Standard+Query+Parser#TheStandardQueryParser-SpecifyingTermsfortheStandardQueryParser)
-(a superset of the Lucene query syntax)...
-
-NOTE:
-You can browse the documents indexed at <http://localhost:8983/solr/gettingstarted/browse>. The `/browse` UI allows
-getting a feel for how Solr's technical capabilities can be worked with in a familiar, though a bit rough and
-prototypical, interactive HTML view. (The `/browse` view defaults to assuming the `gettingstarted` schema and data
-are a catch-all mix of structured XML, JSON, CSV example data, and unstructured rich documents. Your own data may not
-look ideal at first, though the `/browse` templates are customizable.)
-
-### Indexing JSON
-
-Solr supports indexing JSON, either arbitrary structured JSON or "Solr JSON" (which is similar to Solr XML).
-
-Solr includes a small sample Solr JSON file to illustrate this capability. Again using `bin/post`, index the
-sample JSON file:
-
- bin/post -c gettingstarted example/exampledocs/books.json
-
-You'll see:
-
- /solr-X.Y.Z:$ bin/post -c gettingstarted example/exampledocs/books.json
- java -classpath /solr-X.Y.Z/dist/solr-core-X.Y.Z.jar -Dauto=yes -Dc=gettingstarted -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/books.json
- SimplePostTool version 5.0.0
- Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
- Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
- POSTing file books.json (application/json) to [base]/json/docs
- 1 files indexed.
- COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
- Time spent: 0:00:00.493
-
-For more information on indexing Solr JSON, see the Solr Reference Guide section
-[Solr-Style JSON](https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-Solr-StyleJSON)
-
-To flatten (and/or split) and index arbitrary structured JSON, a topic beyond this quick start guide, check out
-[Transforming and Indexing Custom JSON data](https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-TransformingandIndexingCustomJSON).
-
-### Indexing CSV (Comma/Column Separated Values)
-
-A great conduit of data into Solr is via CSV, especially when the documents are homogeneous by all having the
-same set of fields. CSV can be conveniently exported from a spreadsheet such as Excel, or exported from databases such
-as MySQL. When getting started with Solr, it can often be easiest to get your structured data into CSV format and then
-index that into Solr rather than a more sophisticated single step operation.
-
-Using `bin/post` index the included example CSV file:
-
- bin/post -c gettingstarted example/exampledocs/books.csv
-
-In your terminal you'll see:
-
- /solr-X.Y.Z:$ bin/post -c gettingstarted example/exampledocs/books.csv
- java -classpath /solr-X.Y.Z/dist/solr-core-X.Y.Z.jar -Dauto=yes -Dc=gettingstarted -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/books.csv
- SimplePostTool version 5.0.0
- Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
- Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
- POSTing file books.csv (text/csv) to [base]
- 1 files indexed.
- COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
- Time spent: 0:00:00.109
-
-For more information, see the Solr Reference Guide section
-[CSV Formatted Index Updates](https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-CSVFormattedIndexUpdates)
-
-### Other indexing techniques
-
-* Import records from a database using the
-[Data Import Handler (DIH)](https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler).
-
-* Use [SolrJ](https://cwiki.apache.org/confluence/display/solr/Using+SolrJ) from JVM-based languages or
-other [Solr clients](https://cwiki.apache.org/confluence/display/solr/Client+APIs) to programmatically create documents
-to send to Solr.
-
-* Use the Admin UI [Documents tab](http://localhost:8983/solr/#/gettingstarted/documents) to paste in a document to be
-indexed, or select `Document Builder` from the `Document Type` dropdown to build a document one field at a time.
-Click on the `Submit Document` button below the form to index your document.
-
-***
-
-## Updating Data
-
-You may notice that even if you index content in this guide more than once, it does not duplicate the results found.
-This is because the example `schema.xml` specifies a "`uniqueKey`" field called "`id`". Whenever you POST commands to
-Solr to add a document with the same value for the `uniqueKey` as an existing document, it automatically replaces it
-for you. You can see that that has happened by looking at the values for `numDocs` and `maxDoc` in the core-specific
-Overview section of the Solr Admin UI.
-
-`numDocs` represents the number of searchable documents in the index (and will be larger than the number of XML, JSON,
-or CSV files since some files contained more than one document). The maxDoc value may be larger as the maxDoc count
-includes logically deleted documents that have not yet been physically removed from the index. You can re-post the
-sample files over and over again as much as you want and `numDocs` will never increase, because the new documents will
-constantly be replacing the old.
-
-Go ahead and edit any of the existing example data files, change some of the data, and re-run the SimplePostTool
-command. You'll see your changes reflected in subsequent searches.
-
-## Deleting Data
-
-You can delete data by POSTing a delete command to the update URL and specifying the value of the document's unique key
-field, or a query that matches multiple documents (be careful with that one!). Since these commands are smaller, we
-specify them right on the command line rather than reference a JSON or XML file.
-
-Execute the following command to delete a specific document:
-
- bin/post -c gettingstarted -d "<delete><id>SP2514N</id></delete>"
-
-
-## Searching
-
-Solr can be queried via REST clients, cURL, wget, Chrome POSTMAN, etc., as well as via the native clients available for
-many programming languages.
+<!-- As of 2 Oct 2017, this page now automatically redirects to
+ the Solr Reference Guide page solr/guide/solr-tutorial.html -->
-The Solr Admin UI includes a query builder interface - see the `gettingstarted` query tab at
-<http://localhost:8983/solr/#/gettingstarted/query>. If you click the `Execute Query` button without changing anything
-in the form, you'll get 10 documents in JSON format (`*:*` in the `q` param matches all documents):
-
-<img style="border:1px solid #ccc; width:800px" src="images/quickstart-query-screen.png" alt="Solr Quick Start: gettingstarted Query tab"/>
-
-The URL sent by the Admin UI to Solr is shown in light grey near the top right of the above screenshot - if you click on
-it, your browser will show you the raw response. To use cURL, give the same URL in quotes on the `curl` command line:
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=*:*"
-
-
-### Basics
-
-#### Search for a single term
-
-To search for a term, give it as the `q` param value in the core-specific Solr Admin UI Query section, replace `*:*`
-with the term you want to find. To search for "foundation":
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=foundation"
-
-You'll see:
-
- $ curl "http://localhost:8983/solr/gettingstarted/select?q=foundation"
- {
- "responseHeader":{
- "zkConnected":true,
- "status":0,
- "QTime":527,
- "params":{
- "q":"foundation",
- "indent":"true",
- "wt":"json"}},
- "response":{"numFound":4156,"start":0,"maxScore":0.10203234,"docs":[
- {
- "id":"0553293354",
- "cat":["book"],
- "name":["Foundation"],
- ...
-
-The response indicates that there are 4,156 hits (`"numFound":4156`), of which the first 10 were returned, since by
-default `start=0` and `rows=10`. You can specify these params to page through results, where `start` is the
-(zero-based) position of the first result to return, and `rows` is the page size.
-
-To restrict fields returned in the response, use the `fl` param, which takes a comma-separated list of field names.
-E.g. to only return the `id` field:
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=foundation&fl=id"
-
-`q=foundation` matches nearly all of the docs we've indexed, since most of the files under `docs/` contain
-"The Apache Software Foundation". To restrict search to a particular field, use the syntax "`q=field:value`",
-e.g. to search for `Foundation` only in the `name` field:
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=name:Foundation"
-
-The above request returns only one document (`"numFound":1`) - from the response:
-
- ...
- "response":{"numFound":1,"start":0,"maxScore":2.5902672,"docs":[
- {
- "id":"0553293354",
- "cat":["book"],
- "name":["Foundation"],
- ...
-
-#### Phrase search
-
-To search for a multi-term phrase, enclose it in double quotes: `q="multiple terms here"`. E.g. to search for
-"CAS latency" - note that the space between terms must be converted to "`+`" in a URL (the Admin UI will handle URL
-encoding for you automatically):
-
- curl "http://localhost:8983/solr/gettingstarted/select?indent=true&q=\"CAS+latency\""
-
-You'll get back:
-
- {
- "responseHeader":{
- "zkConnected":true,
- "status":0,
- "QTime":391,
- "params":{
- "q":"\"CAS latency\"",
- "indent":"true",
- "wt":"json"}},
- "response":{"numFound":3,"start":0,"maxScore":22.027056,"docs":[
- {
- "id":"TWINX2048-3200PRO",
- "name":["CORSAIR XMS 2GB (2 x 1GB) 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) Dual Channel Kit System Memory - Retail"],
- "manu":["Corsair Microsystems Inc."],
- "manu_id_s":"corsair",
- "cat":["electronics", "memory"],
- "features":["CAS latency 2, 2-3-3-6 timing, 2.75v, unbuffered, heat-spreader"],
- ...
-
-#### Combining searches
-
-By default, when you search for multiple terms and/or phrases in a single query, Solr will only require that one of them
-is present in order for a document to match. Documents containing more terms will be sorted higher in the results list.
-
-You can require that a term or phrase is present by prefixing it with a "`+`"; conversely, to disallow the presence of a
-term or phrase, prefix it with a "`-`".
-
-To find documents that contain both terms "`one`" and "`three`", enter `+one +three` in the `q` param in the
-Admin UI Query tab. Because the "`+`" character has a reserved purpose in URLs (encoding the space character),
-you must URL encode it for `curl` as "`%2B`":
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=%2Bone+%2Bthree"
-
-To search for documents that contain the term "`two`" but **don't** contain the term "`one`", enter `+two -one` in the
-`q` param in the Admin UI. Again, URL encode "`+`" as "`%2B`":
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=%2Btwo+-one"
-
-#### In depth
-
-For more Solr search options, see the Solr Reference Guide's
-[Searching](https://cwiki.apache.org/confluence/display/solr/Searching) section.
-
-
-### Faceting
-
-One of Solr's most popular features is faceting. Faceting allows the search results to be arranged into subsets (or
-buckets or categories), providing a count for each subset. There are several types of faceting: field values, numeric
-and date ranges, pivots (decision tree), and arbitrary query faceting.
-
-#### Field facets
-
-In addition to providing search results, a Solr query can return the number of documents that contain each unique value
-in the whole result set.
-
-From the core-specific Admin UI Query tab, if you check the "`facet`" checkbox, you'll see a few facet-related options
-appear:
-
-<img style="border:1px solid #ccc" src="images/quickstart-admin-ui-facet-options.png" alt="Solr Quick Start: Query tab facet options"/>
-
-To see facet counts from all documents (`q=*:*`): turn on faceting (`facet=true`), and specify the field to facet on via
-the `facet.field` param. If you only want facets, and no document contents, specify `rows=0`. The `curl` command below
-will return facet counts for the `manu_id_s` field:
-
- curl 'http://localhost:8983/solr/gettingstarted/select?q=*:*&rows=0'\
- '&facet=true&facet.field=manu_id_s'
-
-In your terminal, you'll see:
-
- {
- "responseHeader":{
- "zkConnected":true,
- "status":0,
- "QTime":201,
- "params":{
- "q":"*:*",
- "facet.field":"manu_id_s",
- "indent":"true",
- "rows":"0",
- "wt":"json",
- "facet":"true"}},
- "response":{"numFound":4374,"start":0,"maxScore":1.0,"docs":[]
- },
- "facet_counts":{
- "facet_queries":{},
- "facet_fields":{
- "manu_id_s":[
- "corsair",3,
- "belkin",2,
- "canon",2,
- "apple",1,
- "asus",1,
- "ati",1,
- "boa",1,
- "dell",1,
- "eu",1,
- "maxtor",1,
- "nor",1,
- "uk",1,
- "viewsonic",1,
- "samsung",0]},
- "facet_ranges":{},
- "facet_intervals":{},
- "facet_heatmaps":{}}}
-
-#### Range facets
-
-For numerics or dates, it's often desirable to partition the facet counts into ranges rather than discrete values.
-A prime example of numeric range faceting, using the example product data, is `price`. In the `/browse` UI, it looks
-like this:
-
-<img style="border:1px solid #ccc" src="images/quickstart-range-facet.png" alt="Solr Quick Start: Range facets"/>
-
-The data for these price range facets can be seen in JSON format with this command:
-
- curl 'http://localhost:8983/solr/gettingstarted/select?q=*:*&rows=0'\
- '&facet=true'\
- '&facet.range=price'\
- '&f.price.facet.range.start=0'\
- '&f.price.facet.range.end=600'\
- '&f.price.facet.range.gap=50'\
- '&facet.range.other=after'
-
-In your terminal you will see:
-
- {
- "responseHeader":{
- "zkConnected":true,
- "status":0,
- "QTime":248,
- "params":{
- "facet.range":"price",
- "q":"*:*",
- "f.price.facet.range.start":"0",
- "facet.range.other":"after",
- "indent":"on",
- "f.price.facet.range.gap":"50",
- "rows":"0",
- "wt":"json",
- "facet":"true",
- "f.price.facet.range.end":"600"}},
- "response":{"numFound":4374,"start":0,"maxScore":1.0,"docs":[]
- },
- "facet_counts":{
- "facet_queries":{},
- "facet_fields":{},
- "facet_ranges":{
- "price":{
- "counts":[
- "0.0",19,
- "50.0",1,
- "100.0",0,
- "150.0",2,
- "200.0",0,
- "250.0",1,
- "300.0",1,
- "350.0",2,
- "400.0",0,
- "450.0",1,
- "500.0",0,
- "550.0",0],
- "gap":50.0,
- "after":2,
- "start":0.0,
- "end":600.0}},
- "facet_intervals":{},
- "facet_heatmaps":{}}}
-
-#### Pivot facets
-
-Another faceting type is pivot facets, also known as "decision trees", allowing two or more fields to be nested for all
-the various possible combinations. Using the example technical product data, pivot facets can be used to see how many
-of the products in the "book" category (the `cat` field) are in stock or not in stock. Here's how to get at the raw
-data for this scenario:
-
- curl 'http://localhost:8983/solr/gettingstarted/select?q=*:*&rows=0&facet=on&facet.pivot=cat,inStock'
-
-This results in the following response (trimmed to just the book category output), which says out of 14 items in the
-"book" category, 12 are in stock and 2 are not in stock:
-
- ...
- "facet_pivot":{
- "cat,inStock":[{
- "field":"cat",
- "value":"book",
- "count":14,
- "pivot":[{
- "field":"inStock",
- "value":true,
- "count":12},
- {
- "field":"inStock",
- "value":false,
- "count":2}]},
- ...
-
-#### More faceting options
-
-For the full scoop on Solr faceting, visit the Solr Reference Guide's
-[Faceting](https://cwiki.apache.org/confluence/display/solr/Faceting) section.
-
-
-### Spatial
-
-Solr has sophisticated geospatial support, including searching within a specified distance range of a given location
-(or within a bounding box), sorting by distance, or even boosting results by the distance. Some of the example tech
-products documents in `example/exampledocs/*.xml` have locations associated with them to illustrate the spatial
-capabilities. To run the tech products example, see the [techproducts example section](#techproducts). Spatial queries
-can be combined with any other types of queries, such as in this example of querying for "ipod" within 10 kilometers
-from San Francisco:
-
-<img style="border:1px solid #ccc; width:800px" src="images/quickstart-spatial.png" alt="Solr Quick Start: spatial search"/>
-
-The URL to this example is
-<http://localhost:8983/solr/techproducts/browse?q=ipod&pt=37.7752%2C-122.4232&d=10&sfield=store&fq=%7B%21bbox%7D&queryOpts=spatial&queryOpts=spatial>,
-leveraging the `/browse` UI to show a map for each item and allow easy selection of the location to search near.
-
-To learn more about Solr's spatial capabilities, see the Solr Reference Guide's
-[Spatial Search](https://cwiki.apache.org/confluence/display/solr/Spatial+Search) section.
-
-## Wrapping up
-
-If you've run the full set of commands in this quick start guide you have done the following:
-
-* Launched Solr into SolrCloud mode, two nodes, two collections including shards and replicas
-* Indexed a directory of rich text files
-* Indexed Solr XML files
-* Indexed Solr JSON files
-* Indexed CSV content
-* Opened the admin console, used its query interface to get JSON formatted results
-* Opened the /browse interface to explore Solr's features in a more friendly and familiar interface
-
-Nice work! The script (see below) to run all of these items took under two minutes! (Your run time may vary, depending
-on your computer's power and resources available.)
-
-Here's a Unix script for convenient copying and pasting in order to run the key commands for this quick start guide:
-
- date
- bin/solr start -e cloud -noprompt
- open http://localhost:8983/solr
- bin/post -c gettingstarted docs/
- open http://localhost:8983/solr/gettingstarted/browse
- bin/post -c gettingstarted example/exampledocs/*.xml
- bin/post -c gettingstarted example/exampledocs/books.json
- bin/post -c gettingstarted example/exampledocs/books.csv
- bin/post -c gettingstarted -d "<delete><id>SP2514N</id></delete>"
- bin/solr healthcheck -c gettingstarted
- date
-
-## Cleanup
-
-As you work through this guide, you may want to stop Solr and reset the environment back to the starting point.
-The following command line will stop Solr and remove the directories for each of the two nodes that the start script
-created:
-
- bin/solr stop -all ; rm -Rf example/cloud/
-
-## Where to next?
-
-For more information on Solr, check out the following resources:
-
- * [Solr Reference Guide](https://cwiki.apache.org/confluence/display/solr/Apache+Solr+Reference+Guide) (ensure you
- match the version of the reference guide with your version of Solr)
- * See also additional [Resources](http://lucene.apache.org/solr/resources.html)
+# Solr Quick Start
+Please see the [Solr Tutorial](https://lucene.apache.org/solr/guide/solr-tutorial.html) or additional [Resources](http://lucene.apache.org/solr/resources.html).
[17/50] lucene-solr:feature/autoscaling_72: LUCENE-7974: make
NearestHit public, and add javadocs, to make precommit happy
Posted by ab...@apache.org.
LUCENE-7974: make NearestHit public, and add javadocs, to make precommit happy
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/73f34033
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/73f34033
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/73f34033
Branch: refs/heads/feature/autoscaling_72
Commit: 73f3403381b95a908543cf859b2f43f28cb9a34a
Parents: 0062e99
Author: Mike McCandless <mi...@apache.org>
Authored: Mon Oct 2 09:42:17 2017 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Mon Oct 2 09:42:17 2017 -0400
----------------------------------------------------------------------
.../org/apache/lucene/document/FloatPointNearestNeighbor.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/73f34033/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
index d3360a8..b7859d0 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
@@ -193,7 +193,8 @@ public class FloatPointNearestNeighbor {
}
}
- static class NearestHit {
+ /** Holds one hit from {@link FloatPointNearestNeighbor#nearest} */
+ public static class NearestHit {
public int docID;
public double distanceSquared;
[19/50] lucene-solr:feature/autoscaling_72: buildAndPushRelease.py:
use wait() instead of poll() to check for process completion
Posted by ab...@apache.org.
buildAndPushRelease.py: use wait() instead of poll() to check for process completion
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2a13e783
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/2a13e783
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/2a13e783
Branch: refs/heads/feature/autoscaling_72
Commit: 2a13e783f94be23c3fa2d795e36dd903a7d24585
Parents: 74050a3
Author: Steve Rowe <sa...@gmail.com>
Authored: Mon Oct 2 13:34:30 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Mon Oct 2 13:34:56 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/buildAndPushRelease.py | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2a13e783/dev-tools/scripts/buildAndPushRelease.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/buildAndPushRelease.py b/dev-tools/scripts/buildAndPushRelease.py
index 1143ebf..cd2fdd1 100644
--- a/dev-tools/scripts/buildAndPushRelease.py
+++ b/dev-tools/scripts/buildAndPushRelease.py
@@ -53,9 +53,14 @@ def runAndSendGPGPassword(command, password):
p.stdin.write((password + '\n').encode('UTF-8'))
p.stdin.write('\n'.encode('UTF-8'))
- result = p.poll()
- if result is not None:
- msg = ' FAILED: %s [see log %s]' % (command, LOG)
+ try:
+ result = p.wait(timeout=120)
+ if result != 0:
+ msg = ' FAILED: %s [see log %s]' % (command, LOG)
+ print(msg)
+ raise RuntimeError(msg)
+ except TimeoutExpired:
+ msg = ' FAILED: %s [timed out after 2 minutes; see log %s]' % (command, LOG)
print(msg)
raise RuntimeError(msg)
[35/50] lucene-solr:feature/autoscaling_72: precommit errors
Posted by ab...@apache.org.
precommit errors
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/30122391
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/30122391
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/30122391
Branch: refs/heads/feature/autoscaling_72
Commit: 301223915d2517aed1407b49b6ae5590914cddb8
Parents: 46cd679
Author: Noble Paul <no...@apache.org>
Authored: Wed Oct 4 19:54:44 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Wed Oct 4 19:54:44 2017 +1030
----------------------------------------------------------------------
.../solr/cloud/autoscaling/AutoScaling.java | 5 +++--
.../solr/cloud/autoscaling/TriggerBase.java | 2 --
.../cloud/autoscaling/TriggerEventQueue.java | 5 +++--
.../solr/cloud/overseer/ReplicaMutator.java | 1 -
.../java/org/apache/solr/core/CoreContainer.java | 1 -
.../handler/admin/AutoscalingHistoryHandler.java | 19 ++++++++++++++++++-
.../src/java/org/apache/solr/util/IdUtils.java | 16 ++++++++++++++++
.../AutoAddReplicasPlanActionTest.java | 2 ++
.../cloud/autoscaling/ComputePlanActionTest.java | 4 ++--
.../admin/AutoscalingHistoryHandlerTest.java | 19 ++++++++++++++++---
10 files changed, 60 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
index bd74c84..ef0cfda 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
@@ -21,6 +21,7 @@ import java.io.Closeable;
import java.io.IOException;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import com.google.common.base.Preconditions;
import org.apache.lucene.store.AlreadyClosedException;
@@ -120,8 +121,8 @@ public class AutoScaling {
private boolean isClosed = false;
public TriggerFactory(CoreContainer coreContainer, ZkController zkController) {
- Preconditions.checkNotNull(coreContainer);
- Preconditions.checkNotNull(zkController);
+ Objects.requireNonNull(coreContainer);
+ Objects.requireNonNull(zkController);
this.coreContainer = coreContainer;
this.zkController = zkController;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerBase.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerBase.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerBase.java
index 7aff846..45d0f89 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerBase.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerBase.java
@@ -17,9 +17,7 @@
package org.apache.solr.cloud.autoscaling;
import java.lang.invoke.MethodHandles;
-import java.util.Arrays;
import java.util.Map;
-import java.util.TreeMap;
import org.apache.solr.common.cloud.SolrZkClient;
import org.apache.solr.common.cloud.ZkStateReader;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
index 4030cd2..3b22a3c 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
@@ -17,6 +17,7 @@
package org.apache.solr.cloud.autoscaling;
import java.lang.invoke.MethodHandles;
+import java.nio.charset.StandardCharsets;
import java.util.Map;
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventType;
@@ -69,7 +70,7 @@ public class TriggerEventQueue extends ZkDistributedQueue {
Map<String, Object> map = (Map<String, Object>) Utils.fromJSON(data);
return fromMap(map);
} catch (Exception e) {
- LOG.warn("Invalid event data, ignoring: " + new String(data));
+ LOG.warn("Invalid event data, ignoring: " + new String(data, StandardCharsets.UTF_8));
continue;
}
}
@@ -91,7 +92,7 @@ public class TriggerEventQueue extends ZkDistributedQueue {
Map<String, Object> map = (Map<String, Object>) Utils.fromJSON(data);
return fromMap(map);
} catch (Exception e) {
- LOG.warn("Invalid event data, ignoring: " + new String(data));
+ LOG.warn("Invalid event data, ignoring: " + new String(data, StandardCharsets.UTF_8));
continue;
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/java/org/apache/solr/cloud/overseer/ReplicaMutator.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/overseer/ReplicaMutator.java b/solr/core/src/java/org/apache/solr/cloud/overseer/ReplicaMutator.java
index fad5d96..82d106d 100644
--- a/solr/core/src/java/org/apache/solr/cloud/overseer/ReplicaMutator.java
+++ b/solr/core/src/java/org/apache/solr/cloud/overseer/ReplicaMutator.java
@@ -37,7 +37,6 @@ import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.Slice;
import org.apache.solr.common.cloud.ZkNodeProps;
import org.apache.solr.common.cloud.ZkStateReader;
-import org.apache.solr.common.params.CoreAdminParams;
import org.apache.solr.common.util.Utils;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/java/org/apache/solr/core/CoreContainer.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/CoreContainer.java b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
index c3ce0ed..e1e5a01 100644
--- a/solr/core/src/java/org/apache/solr/core/CoreContainer.java
+++ b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
@@ -105,7 +105,6 @@ import org.apache.solr.security.HttpClientBuilderPlugin;
import org.apache.solr.security.PKIAuthenticationPlugin;
import org.apache.solr.security.SecurityPluginHolder;
import org.apache.solr.update.SolrCoreState;
-import org.apache.solr.update.UpdateLog;
import org.apache.solr.update.UpdateShardHandler;
import org.apache.solr.util.DefaultSolrThreadFactory;
import org.apache.solr.util.stats.MetricUtils;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/java/org/apache/solr/handler/admin/AutoscalingHistoryHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/admin/AutoscalingHistoryHandler.java b/solr/core/src/java/org/apache/solr/handler/admin/AutoscalingHistoryHandler.java
index 0e3440b..1c568d6 100644
--- a/solr/core/src/java/org/apache/solr/handler/admin/AutoscalingHistoryHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/admin/AutoscalingHistoryHandler.java
@@ -1,7 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.solr.handler.admin;
import java.lang.invoke.MethodHandles;
import java.util.HashMap;
+import java.util.Locale;
import java.util.Map;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
@@ -99,7 +116,7 @@ public class AutoscalingHistoryHandler extends RequestHandlerBase implements Per
String[] values = params.remove(e.getKey());
if (values != null) {
for (String value : values) {
- params.add(CommonParams.FQ, String.format(e.getValue(), value));
+ params.add(CommonParams.FQ, String.format(Locale.ROOT, e.getValue(), value));
}
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/java/org/apache/solr/util/IdUtils.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/util/IdUtils.java b/solr/core/src/java/org/apache/solr/util/IdUtils.java
index 1ad9cf8..a6ea7d2 100644
--- a/solr/core/src/java/org/apache/solr/util/IdUtils.java
+++ b/solr/core/src/java/org/apache/solr/util/IdUtils.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.solr.util;
import java.util.concurrent.TimeUnit;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoAddReplicasPlanActionTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoAddReplicasPlanActionTest.java b/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoAddReplicasPlanActionTest.java
index 4a0495d..2259c40 100644
--- a/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoAddReplicasPlanActionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoAddReplicasPlanActionTest.java
@@ -36,6 +36,7 @@ import org.apache.solr.common.params.CollectionParams;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SuppressForbidden;
import org.apache.solr.util.TimeOut;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -147,6 +148,7 @@ public class AutoAddReplicasPlanActionTest extends SolrCloudTestCase{
}
}
+ @SuppressForbidden(reason = "Needs currentTimeMillis to create unique id")
private List<SolrRequest> getOperations(JettySolrRunner actionJetty, String lostNodeName) {
AutoAddReplicasPlanAction action = new AutoAddReplicasPlanAction();
TriggerEvent lostNode = new NodeLostTrigger.NodeLostEvent(TriggerEventType.NODELOST, ".auto_add_replicas", Collections.singletonList(System.currentTimeMillis()), Collections.singletonList(lostNodeName));
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java b/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java
index d9b88b5..9f1e5de 100644
--- a/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java
@@ -26,7 +26,6 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
-import com.google.common.base.Charsets;
import org.apache.solr.client.solrj.SolrRequest;
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventType;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
@@ -47,6 +46,7 @@ import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.createAutoScalingRequest;
import static org.apache.solr.common.params.CollectionParams.CollectionAction.MOVEREPLICA;
@@ -79,7 +79,7 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
actionContextPropsRef.set(null);
// remove everything from autoscaling.json in ZK
- zkClient().setData(ZkStateReader.SOLR_AUTOSCALING_CONF_PATH, "{}".getBytes(Charsets.UTF_8), true);
+ zkClient().setData(ZkStateReader.SOLR_AUTOSCALING_CONF_PATH, "{}".getBytes(UTF_8), true);
if (cluster.getJettySolrRunners().size() > NODE_COUNT) {
// stop some to get to original state
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/30122391/solr/core/src/test/org/apache/solr/handler/admin/AutoscalingHistoryHandlerTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/handler/admin/AutoscalingHistoryHandlerTest.java b/solr/core/src/test/org/apache/solr/handler/admin/AutoscalingHistoryHandlerTest.java
index 56c920e..09f5edc 100644
--- a/solr/core/src/test/org/apache/solr/handler/admin/AutoscalingHistoryHandlerTest.java
+++ b/solr/core/src/test/org/apache/solr/handler/admin/AutoscalingHistoryHandlerTest.java
@@ -1,3 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
package org.apache.solr.handler.admin;
import java.lang.invoke.MethodHandles;
@@ -34,9 +50,6 @@ import org.slf4j.LoggerFactory;
import static org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.createAutoScalingRequest;
-/**
- *
- */
@LogLevel("org.apache.solr.cloud.autoscaling=DEBUG;org.apache.solr.cloud.Overseer=DEBUG;org.apache.solr.cloud.overseer=DEBUG;")
public class AutoscalingHistoryHandlerTest extends SolrCloudTestCase {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
[16/50] lucene-solr:feature/autoscaling_72: policy should compare the
approx val instead of actual val when moving replicas around
Posted by ab...@apache.org.
policy should compare the approx val instead of actual val when moving replicas around
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/942b6715
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/942b6715
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/942b6715
Branch: refs/heads/feature/autoscaling_72
Commit: 942b6715b39bf25832fde255f95de36e5fe2e993
Parents: fba51f3
Author: Noble Paul <no...@apache.org>
Authored: Mon Oct 2 13:05:50 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Mon Oct 2 13:05:50 2017 +1030
----------------------------------------------------------------------
.../client/solrj/cloud/autoscaling/Policy.java | 38 +++++++------
.../solrj/cloud/autoscaling/TestPolicy.java | 59 +++++++++++++++++---
2 files changed, 73 insertions(+), 24 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/942b6715/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
index 761f1b8..12024e4 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
@@ -253,20 +253,7 @@ public class Policy implements MapWriter {
* Apply the preferences and conditions
*/
private void applyRules() {
- if (!clusterPreferences.isEmpty()) {
- //this is to set the approximate value according to the precision
- ArrayList<Row> tmpMatrix = new ArrayList<>(matrix);
- for (Preference p : clusterPreferences) {
- Collections.sort(tmpMatrix, (r1, r2) -> p.compare(r1, r2, false));
- p.setApproxVal(tmpMatrix);
- }
- //approximate values are set now. Let's do recursive sorting
- Collections.sort(matrix, (Row r1, Row r2) -> {
- int result = clusterPreferences.get(0).compare(r1, r2, true);
- if (result == 0) result = clusterPreferences.get(0).compare(r1, r2, false);
- return result;
- });
- }
+ setApproxValuesAndSortNodes(clusterPreferences, matrix);
for (Clause clause : expandedClauses) {
List<Violation> errs = clause.test(matrix);
@@ -274,6 +261,8 @@ public class Policy implements MapWriter {
}
}
+
+
public List<Violation> getViolations() {
return violations;
}
@@ -303,6 +292,22 @@ public class Policy implements MapWriter {
}
}
+ static void setApproxValuesAndSortNodes(List<Preference> clusterPreferences, List<Row> matrix) {
+ if (!clusterPreferences.isEmpty()) {
+ //this is to set the approximate value according to the precision
+ ArrayList<Row> tmpMatrix = new ArrayList<>(matrix);
+ for (Preference p : clusterPreferences) {
+ Collections.sort(tmpMatrix, (r1, r2) -> p.compare(r1, r2, false));
+ p.setApproxVal(tmpMatrix);
+ }
+ //approximate values are set now. Let's do recursive sorting
+ Collections.sort(matrix, (Row r1, Row r2) -> {
+ int result = clusterPreferences.get(0).compare(r1, r2, true);
+ if (result == 0) result = clusterPreferences.get(0).compare(r1, r2, false);
+ return result;
+ });
+ }
+ }
public Session createSession(ClusterDataProvider dataProvider) {
return new Session(dataProvider);
@@ -470,7 +475,8 @@ public class Policy implements MapWriter {
}
}
- protected List<Violation> testChangedMatrix(boolean strict, List<Row> rows) {
+ List<Violation> testChangedMatrix(boolean strict, List<Row> rows) {
+ setApproxValuesAndSortNodes(session.getPolicy().clusterPreferences,rows);
List<Violation> errors = new ArrayList<>();
for (Clause clause : session.expandedClauses) {
if (strict || clause.strict) {
@@ -615,6 +621,6 @@ public class Policy implements MapWriter {
* a value {@code 1} if r1 is less loaded than r2
*/
static int compareRows(Row r1, Row r2, Policy policy) {
- return policy.clusterPreferences.get(0).compare(r1, r2, false);
+ return policy.clusterPreferences.get(0).compare(r1, r2, true);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/942b6715/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java b/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
index 96e2257..a91698d 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
@@ -19,6 +19,7 @@ package org.apache.solr.client.solrj.cloud.autoscaling;
import java.io.IOException;
+import java.lang.invoke.MethodHandles;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
@@ -34,19 +35,24 @@ import org.apache.solr.client.solrj.SolrRequest;
import org.apache.solr.client.solrj.cloud.autoscaling.Clause.Violation;
import org.apache.solr.client.solrj.cloud.autoscaling.Policy.Suggester.Hint;
import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.cloud.autoscaling.TriggerEvent;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.ReplicaPosition;
import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.params.CollectionParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.Pair;
import org.apache.solr.common.util.Utils;
import org.apache.solr.common.util.ValidatingJsonMap;
import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import static org.apache.solr.common.params.CollectionParams.CollectionAction.ADDREPLICA;
import static org.apache.solr.common.params.CollectionParams.CollectionAction.MOVEREPLICA;
public class TestPolicy extends SolrTestCaseJ4 {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
public static String clusterState = "{'gettingstarted':{" +
" 'router':{'name':'compositeId'}," +
@@ -1185,19 +1191,19 @@ public class TestPolicy extends SolrTestCaseJ4 {
public void testMultiReplicaPlacement() {
String autoScaleJson = "{" +
- " 'cluster-preferences': [" +
+ " cluster-preferences: [" +
" { maximize : freedisk , precision: 50}," +
" { minimize : cores, precision: 2}" +
" ]," +
- " 'cluster-policy': [" +
- " { replica : '0' , 'nodeRole': 'overseer'}," +
- " { 'replica': '<2', 'shard': '#ANY', 'node': '#ANY'" +
+ " cluster-policy: [" +
+ " { replica : '0' , nodeRole: overseer}," +
+ " { replica: '<2', shard: '#ANY', node: '#ANY'" +
" }" +
" ]," +
- " 'policies': {" +
- " 'policy1': [" +
- " { 'replica': '<2', 'shard': '#EACH', 'node': '#ANY'}," +
- " { 'replica': '<2', 'shard': '#EACH', 'sysprop.rack': 'rack1'}" +
+ " policies: {" +
+ " policy1: [" +
+ " { replica: '<2', shard: '#EACH', node: '#ANY'}," +
+ " { replica: '<2', shard: '#EACH', sysprop.rack: rack1}" +
" ]" +
" }" +
"}";
@@ -1271,5 +1277,42 @@ public class TestPolicy extends SolrTestCaseJ4 {
assertNull(op);
}
+ public void testComputePlanAfterNodeAdded() {
+
+ String dataproviderdata = "{" +
+ " liveNodes:[" +
+ " '127.0.0.1:51078_solr'," +
+ " '127.0.0.1:51147_solr']," +
+ " replicaInfo:{" +
+ " '127.0.0.1:51147_solr':{}," +
+ " '127.0.0.1:51078_solr':{testNodeAdded:{shard1:[" +
+ " { core_node3 : { type : NRT}}," +
+ " { core_node4 : { type : NRT}}]}}}," +
+ " nodeValues:{" +
+ " '127.0.0.1:51147_solr':{" +
+ " node:'127.0.0.1:51147_solr'," +
+ " cores:0," +
+ " freedisk : 880.5428657531738}," +
+ " '127.0.0.1:51078_solr':{" +
+ " node:'127.0.0.1:51078_solr'," +
+ " cores:2," +
+ " freedisk:880.5428695678711}}}";
+
+ String autoScalingjson = "cluster-preferences:[" +
+ " {minimize : cores}," +
+ " {'maximize':freedisk , precision:100}], " +
+ " cluster-policy:[{cores:'<10',node:'#ANY'}," +
+ " {replica:'<2', shard:'#EACH',node:'#ANY'}," +
+ " { nodeRole:overseer,replica:0}]}";
+ Policy policy = new Policy((Map<String, Object>) Utils.fromJSONString(autoScalingjson));
+ Policy.Session session = policy.createSession(dataProviderWithData(dataproviderdata));
+ Policy.Suggester suggester = session.getSuggester(CollectionParams.CollectionAction.MOVEREPLICA)
+ .hint(Hint.TARGET_NODE, "127.0.0.1:51147_solr");
+ SolrRequest op = suggester.getOperation();
+ log.info("" + op);
+ assertNotNull(op);
+ }
+
+
}
[38/50] lucene-solr:feature/autoscaling_72: LUCENE-7972:
DirectoryTaxonomyReader now implements Accountable
Posted by ab...@apache.org.
LUCENE-7972: DirectoryTaxonomyReader now implements Accountable
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/b9a51a16
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/b9a51a16
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/b9a51a16
Branch: refs/heads/feature/autoscaling_72
Commit: b9a51a16869cf516853fabb6b4904aa6e2332586
Parents: a6052ba
Author: Mike McCandless <mi...@apache.org>
Authored: Wed Oct 4 10:52:53 2017 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Wed Oct 4 10:52:53 2017 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 6 +-
.../directory/DirectoryTaxonomyReader.java | 58 +++++++++++++++++++-
.../taxonomy/directory/TaxonomyIndexArrays.java | 40 ++++++++++++--
.../directory/TestDirectoryTaxonomyReader.java | 34 +++++++++++-
4 files changed, 129 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b9a51a16/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 7c35503..53a8ee9 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -39,7 +39,11 @@ New Features
K-nearest-neighbor search implementation. (Steve Rowe)
* LUCENE-7975: Change the default taxonomy facets cache to a faster
- byte[] (UTF-8) based cache.
+ byte[] (UTF-8) based cache. (Mike McCandless)
+
+* LUCENE-7972: DirectoryTaxonomyReader, in Lucene's facet module, now
+ implements Accountable, so you can more easily track how much heap
+ it's using. (Mike McCandless)
* LUCENE-7982: A new NormsFieldExistsQuery matches documents that have
norms in a specified field (Colin Goodheart-Smithe via Mike McCandless)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b9a51a16/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyReader.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyReader.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyReader.java
index c72e60b..cde56e1 100644
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyReader.java
+++ b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/DirectoryTaxonomyReader.java
@@ -17,6 +17,10 @@
package org.apache.lucene.facet.taxonomy.directory;
import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;
@@ -30,12 +34,17 @@ import org.apache.lucene.facet.taxonomy.TaxonomyReader;
import org.apache.lucene.index.CorruptIndexException; // javadocs
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.MultiFields;
import org.apache.lucene.index.PostingsEnum;
+import org.apache.lucene.index.SegmentReader;
import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.Accountables;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.IOUtils;
+import org.apache.lucene.util.RamUsageEstimator;
/**
* A {@link TaxonomyReader} which retrieves stored taxonomy information from a
@@ -49,11 +58,14 @@ import org.apache.lucene.util.IOUtils;
*
* @lucene.experimental
*/
-public class DirectoryTaxonomyReader extends TaxonomyReader {
+public class DirectoryTaxonomyReader extends TaxonomyReader implements Accountable {
private static final Logger logger = Logger.getLogger(DirectoryTaxonomyReader.class.getName());
private static final int DEFAULT_CACHE_VALUE = 4000;
+
+ // NOTE: very coarse estimate!
+ private static final int BYTES_PER_CACHE_ENTRY = 4 * RamUsageEstimator.NUM_BYTES_OBJECT_REF + 4 * RamUsageEstimator.NUM_BYTES_OBJECT_HEADER + 8 * RamUsageEstimator.NUM_BYTES_CHAR;
private final DirectoryTaxonomyWriter taxoWriter;
private final long taxoEpoch; // used in doOpenIfChanged
@@ -325,7 +337,51 @@ public class DirectoryTaxonomyReader extends TaxonomyReader {
ensureOpen();
return indexReader.numDocs();
}
+
+ @Override
+ public synchronized long ramBytesUsed() {
+ ensureOpen();
+ long ramBytesUsed = 0;
+ for (LeafReaderContext ctx : indexReader.leaves()) {
+ ramBytesUsed += ((SegmentReader) ctx.reader()).ramBytesUsed();
+ }
+ if (taxoArrays != null) {
+ ramBytesUsed += taxoArrays.ramBytesUsed();
+ }
+ synchronized (categoryCache) {
+ ramBytesUsed += BYTES_PER_CACHE_ENTRY * categoryCache.size();
+ }
+
+ synchronized (ordinalCache) {
+ ramBytesUsed += BYTES_PER_CACHE_ENTRY * ordinalCache.size();
+ }
+
+ return ramBytesUsed;
+ }
+ @Override
+ public synchronized Collection<Accountable> getChildResources() {
+ final List<Accountable> resources = new ArrayList<>();
+ long ramBytesUsed = 0;
+ for (LeafReaderContext ctx : indexReader.leaves()) {
+ ramBytesUsed += ((SegmentReader) ctx.reader()).ramBytesUsed();
+ }
+ resources.add(Accountables.namedAccountable("indexReader", ramBytesUsed));
+ if (taxoArrays != null) {
+ resources.add(Accountables.namedAccountable("taxoArrays", taxoArrays));
+ }
+
+ synchronized (categoryCache) {
+ resources.add(Accountables.namedAccountable("categoryCache", BYTES_PER_CACHE_ENTRY * categoryCache.size()));
+ }
+
+ synchronized (ordinalCache) {
+ resources.add(Accountables.namedAccountable("ordinalCache", BYTES_PER_CACHE_ENTRY * ordinalCache.size()));
+ }
+
+ return Collections.unmodifiableList(resources);
+ }
+
/**
* setCacheSize controls the maximum allowed size of each of the caches
* used by {@link #getPath(int)} and {@link #getOrdinal(FacetLabel)}.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b9a51a16/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/TaxonomyIndexArrays.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/TaxonomyIndexArrays.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/TaxonomyIndexArrays.java
index bdd5c80..dc4d18a 100644
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/TaxonomyIndexArrays.java
+++ b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/directory/TaxonomyIndexArrays.java
@@ -16,16 +16,23 @@
*/
package org.apache.lucene.facet.taxonomy.directory;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
import org.apache.lucene.facet.taxonomy.ParallelTaxonomyArrays;
import org.apache.lucene.facet.taxonomy.TaxonomyReader;
import org.apache.lucene.index.CorruptIndexException;
-import org.apache.lucene.index.PostingsEnum;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.MultiFields;
+import org.apache.lucene.index.PostingsEnum;
import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.util.Accountable;
+import org.apache.lucene.util.Accountables;
import org.apache.lucene.util.ArrayUtil;
-
-import java.io.IOException;
+import org.apache.lucene.util.RamUsageEstimator;
/**
* A {@link ParallelTaxonomyArrays} that are initialized from the taxonomy
@@ -33,7 +40,7 @@ import java.io.IOException;
*
* @lucene.experimental
*/
-class TaxonomyIndexArrays extends ParallelTaxonomyArrays {
+class TaxonomyIndexArrays extends ParallelTaxonomyArrays implements Accountable {
private final int[] parents;
@@ -214,4 +221,29 @@ class TaxonomyIndexArrays extends ParallelTaxonomyArrays {
return siblings;
}
+ @Override
+ public synchronized long ramBytesUsed() {
+ long ramBytesUsed = RamUsageEstimator.NUM_BYTES_OBJECT_HEADER + 3 * RamUsageEstimator.NUM_BYTES_OBJECT_REF + RamUsageEstimator.NUM_BYTES_BOOLEAN;
+ ramBytesUsed += RamUsageEstimator.shallowSizeOf(parents);
+ if (children != null) {
+ ramBytesUsed += RamUsageEstimator.shallowSizeOf(children);
+ }
+ if (siblings != null) {
+ ramBytesUsed += RamUsageEstimator.shallowSizeOf(siblings);
+ }
+ return ramBytesUsed;
+ }
+
+ @Override
+ public synchronized Collection<Accountable> getChildResources() {
+ final List<Accountable> resources = new ArrayList<>();
+ resources.add(Accountables.namedAccountable("parents", RamUsageEstimator.shallowSizeOf(parents)));
+ if (children != null) {
+ resources.add(Accountables.namedAccountable("children", RamUsageEstimator.shallowSizeOf(children)));
+ }
+ if (siblings != null) {
+ resources.add(Accountables.namedAccountable("siblings", RamUsageEstimator.shallowSizeOf(siblings)));
+ }
+ return Collections.unmodifiableList(resources);
+ }
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b9a51a16/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyReader.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyReader.java b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyReader.java
index 1982048..de20230 100644
--- a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyReader.java
+++ b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/directory/TestDirectoryTaxonomyReader.java
@@ -25,12 +25,12 @@ import java.util.Set;
import org.apache.lucene.analysis.MockAnalyzer;
import org.apache.lucene.facet.FacetTestCase;
import org.apache.lucene.facet.taxonomy.FacetLabel;
-import org.apache.lucene.facet.taxonomy.TaxonomyReader;
import org.apache.lucene.facet.taxonomy.TaxonomyReader.ChildrenIterator;
+import org.apache.lucene.facet.taxonomy.TaxonomyReader;
import org.apache.lucene.facet.taxonomy.TaxonomyWriter;
import org.apache.lucene.index.IndexWriter;
-import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
+import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.LogByteSizeMergePolicy;
import org.apache.lucene.index.LogMergePolicy;
import org.apache.lucene.store.AlreadyClosedException;
@@ -529,5 +529,33 @@ public class TestDirectoryTaxonomyReader extends FacetTestCase {
dir.close();
}
-
+
+ public void testAccountable() throws Exception {
+ Directory dir = newDirectory();
+ DirectoryTaxonomyWriter taxoWriter = new DirectoryTaxonomyWriter(dir);
+ int numCategories = atLeast(10);
+ int numA = 0, numB = 0;
+ Random random = random();
+ // add the two categories for which we'll also add children (so asserts are simpler)
+ taxoWriter.addCategory(new FacetLabel("a"));
+ taxoWriter.addCategory(new FacetLabel("b"));
+ for (int i = 0; i < numCategories; i++) {
+ if (random.nextBoolean()) {
+ taxoWriter.addCategory(new FacetLabel("a", Integer.toString(i)));
+ ++numA;
+ } else {
+ taxoWriter.addCategory(new FacetLabel("b", Integer.toString(i)));
+ ++numB;
+ }
+ }
+ // add category with no children
+ taxoWriter.addCategory(new FacetLabel("c"));
+ taxoWriter.close();
+
+ DirectoryTaxonomyReader taxoReader = new DirectoryTaxonomyReader(dir);
+ assertTrue(taxoReader.ramBytesUsed() > 0);
+ assertTrue(taxoReader.getChildResources().size() > 0);
+ taxoReader.close();
+ dir.close();
+ }
}
[11/50] lucene-solr:feature/autoscaling_72: addBackcompatIndexes.py:
only create moreterms, dvupdates, and empty indexes on unstable branch
Posted by ab...@apache.org.
addBackcompatIndexes.py: only create moreterms, dvupdates, and empty indexes on unstable branch
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ae5caf3f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ae5caf3f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ae5caf3f
Branch: refs/heads/feature/autoscaling_72
Commit: ae5caf3fa67df0abcc6ab5190d0b380f94c2dc40
Parents: 0e8d941
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 23:22:46 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 23:22:46 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/addBackcompatIndexes.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ae5caf3f/dev-tools/scripts/addBackcompatIndexes.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py
index 03e858c..e64dbfc 100644
--- a/dev-tools/scripts/addBackcompatIndexes.py
+++ b/dev-tools/scripts/addBackcompatIndexes.py
@@ -246,7 +246,7 @@ def main():
create_and_add_index(source, 'cfs', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'nocfs', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'sorted', c.version, current_version, c.temp_dir)
- if c.version.minor == 0 and c.version.bugfix == 0:
+ if c.version.minor == 0 and c.version.bugfix == 0 and c.version.major < current_version.major:
create_and_add_index(source, 'moreterms', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'dvupdates', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'emptyIndex', c.version, current_version, c.temp_dir)
[39/50] lucene-solr:feature/autoscaling_72: precommit errors
Posted by ab...@apache.org.
precommit errors
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/15d17ed8
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/15d17ed8
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/15d17ed8
Branch: refs/heads/feature/autoscaling_72
Commit: 15d17ed833a3bb03ad8c266bc795e6174f2410d1
Parents: b9a51a1
Author: Noble Paul <no...@apache.org>
Authored: Thu Oct 5 02:21:45 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Thu Oct 5 02:21:45 2017 +1030
----------------------------------------------------------------------
.../src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/15d17ed8/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
index ef0cfda..dedaebe 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java
@@ -23,7 +23,6 @@ import java.util.List;
import java.util.Map;
import java.util.Objects;
-import com.google.common.base.Preconditions;
import org.apache.lucene.store.AlreadyClosedException;
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventType;
import org.apache.solr.common.util.Utils;
[07/50] lucene-solr:feature/autoscaling_72: SOLR-11406: Solr 7.0
cannot read indexes from 6.x versions
Posted by ab...@apache.org.
SOLR-11406: Solr 7.0 cannot read indexes from 6.x versions
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/39ba1c3d
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/39ba1c3d
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/39ba1c3d
Branch: refs/heads/feature/autoscaling_72
Commit: 39ba1c3da56c5ac3e9476830efe1f4c396ccf7c0
Parents: c322e36
Author: Steve Rowe <sa...@gmail.com>
Authored: Fri Sep 29 16:26:59 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Fri Sep 29 16:33:07 2017 -0400
----------------------------------------------------------------------
.../lucene/backward-codecs/pom.xml.template | 13 +++
.../maven/solr/core/src/test/pom.xml.template | 7 ++
.../index/TestBackwardsCompatibility.java | 12 ++-
solr/CHANGES.txt | 21 +++++
solr/common-build.xml | 6 ++
solr/core/build.xml | 3 +-
.../solr/index/SlowCompositeReaderWrapper.java | 16 ++--
.../solr/configsets/backcompat/conf/schema.xml | 22 +++++
.../configsets/backcompat/conf/solrconfig.xml | 43 +++++++++
.../backcompat/TestLuceneIndexBackCompat.java | 96 ++++++++++++++++++++
10 files changed, 230 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/dev-tools/maven/lucene/backward-codecs/pom.xml.template
----------------------------------------------------------------------
diff --git a/dev-tools/maven/lucene/backward-codecs/pom.xml.template b/dev-tools/maven/lucene/backward-codecs/pom.xml.template
index e40148f..f6e6680 100644
--- a/dev-tools/maven/lucene/backward-codecs/pom.xml.template
+++ b/dev-tools/maven/lucene/backward-codecs/pom.xml.template
@@ -71,5 +71,18 @@
</excludes>
</testResource>
</testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
</build>
</project>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/dev-tools/maven/solr/core/src/test/pom.xml.template
----------------------------------------------------------------------
diff --git a/dev-tools/maven/solr/core/src/test/pom.xml.template b/dev-tools/maven/solr/core/src/test/pom.xml.template
index a4e979f..fd6b8e7 100644
--- a/dev-tools/maven/solr/core/src/test/pom.xml.template
+++ b/dev-tools/maven/solr/core/src/test/pom.xml.template
@@ -61,6 +61,13 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.lucene</groupId>
+ <artifactId>lucene-backward-codecs</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
@solr-core.internal.test.dependencies@
@solr-core.external.test.dependencies@
</dependencies>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index abb189a..c202973 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -277,7 +277,11 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
final static String[] oldNames = {
};
-
+
+ public static String[] getOldNames() {
+ return oldNames;
+ }
+
final String[] unsupportedNames = {
"1.9.0-cfs",
"1.9.0-nocfs",
@@ -440,7 +444,11 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
// TODO: on 6.0.0 release, gen the single segment indices and add here:
final static String[] oldSingleSegmentNames = {
};
-
+
+ public static String[] getOldSingleSegmentNames() {
+ return oldSingleSegmentNames;
+ }
+
static Map<String,Directory> oldIndexDirs;
/**
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 13cee84..aed0c22 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -235,6 +235,27 @@ Other Changes
* SOLR-10451: Remove contrib/ltr/lib from lib includes in the techproducts example config (sungjunyoung via janhoy)
+================== 7.0.1 ==================
+
+Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
+
+Versions of Major Components
+---------------------
+Apache Tika 1.13
+Carrot2 3.15.0
+Velocity 1.7 and Velocity Tools 2.0
+Apache UIMA 2.3.1
+Apache ZooKeeper 3.4.10
+Jetty 9.3.14.v20161028
+
+Bug Fixes
+----------------------
+
+* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
+ (Luiz Armesto, Shawn Heisey, Erick Erickson)
+
+* SOLR-11406: Solr 7.0 cannot read indexes from 6.x versions. (Shawn Heisey, Steve Rowe)
+
================== 7.0.0 ==================
Versions of Major Components
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/common-build.xml
----------------------------------------------------------------------
diff --git a/solr/common-build.xml b/solr/common-build.xml
index 88f71ef..ec37848 100644
--- a/solr/common-build.xml
+++ b/solr/common-build.xml
@@ -493,6 +493,12 @@
</ant>
</target>
+ <target name="-compile-test-lucene-backward-codecs">
+ <ant dir="${common.dir}/backward-codecs" target="compile-test" inheritAll="false">
+ <propertyset refid="uptodate.and.compiled.properties"/>
+ </ant>
+ </target>
+
<!-- Solr contrib targets -->
<target name="-compile-analysis-extras">
<ant dir="${common-solr.dir}/contrib/analysis-extras" target="compile" inheritAll="false">
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/build.xml
----------------------------------------------------------------------
diff --git a/solr/core/build.xml b/solr/core/build.xml
index f7a2a59..45bd1b4 100644
--- a/solr/core/build.xml
+++ b/solr/core/build.xml
@@ -31,7 +31,7 @@
<target name="compile-core" depends="compile-solrj,common-solr.compile-core"/>
- <target name="compile-test" depends="jar-analyzers-icu,-compile-test-lucene-queryparser,-compile-analysis-extras,common-solr.compile-test"/>
+ <target name="compile-test" depends="jar-analyzers-icu,-compile-test-lucene-queryparser,-compile-test-lucene-backward-codecs,-compile-analysis-extras,common-solr.compile-test"/>
<path id="test.classpath">
<path refid="solr.test.base.classpath"/>
@@ -39,6 +39,7 @@
<pathelement location="${analyzers-icu.jar}"/>
<pathelement location="${common-solr.dir}/build/contrib/solr-analysis-extras/classes/java"/>
<pathelement location="${common.dir}/build/queryparser/classes/test"/>
+ <pathelement location="${common.dir}/build/backward-codecs/classes/test"/>
<fileset dir="${common-solr.dir}/contrib/analysis-extras/lib" includes="icu4j*.jar"/>
</path>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java b/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
index 2d612e9..283d245 100644
--- a/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
+++ b/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
@@ -69,12 +69,16 @@ public final class SlowCompositeReaderWrapper extends LeafReader {
if (reader.leaves().isEmpty()) {
metaData = new LeafMetaData(Version.LATEST.major, Version.LATEST, null);
} else {
- Version minVersion = reader.leaves().stream()
- .map(LeafReaderContext::reader)
- .map(LeafReader::getMetaData)
- .map(LeafMetaData::getMinVersion)
- .reduce((v1, v2) -> v1 == null ? null : v2 == null ? null : v2.onOrAfter(v1) ? v1 : v2)
- .get();
+ Version minVersion = Version.LATEST;
+ for (LeafReaderContext leafReaderContext : reader.leaves()) {
+ Version leafVersion = leafReaderContext.reader().getMetaData().getMinVersion();
+ if (leafVersion == null) {
+ minVersion = null;
+ break;
+ } else if (minVersion.onOrAfter(leafVersion)) {
+ minVersion = leafVersion;
+ }
+ }
metaData = new LeafMetaData(reader.leaves().get(0).reader().getMetaData().getCreatedVersionMajor(), minVersion, null);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/src/test-files/solr/configsets/backcompat/conf/schema.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/backcompat/conf/schema.xml b/solr/core/src/test-files/solr/configsets/backcompat/conf/schema.xml
new file mode 100644
index 0000000..b49edd2
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/backcompat/conf/schema.xml
@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<schema name="minimal" version="1.1">
+ <fieldType name="string" class="solr.StrField"/>
+ <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
+ <uniquekey>id</uniquekey>
+</schema>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/src/test-files/solr/configsets/backcompat/conf/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/backcompat/conf/solrconfig.xml b/solr/core/src/test-files/solr/configsets/backcompat/conf/solrconfig.xml
new file mode 100644
index 0000000..f82d5f4
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/backcompat/conf/solrconfig.xml
@@ -0,0 +1,43 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- Minimal solrconfig.xml with /select, /admin and /update only -->
+
+<config>
+ <dataDir>${solr.data.dir:}</dataDir>
+
+ <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>
+
+ <schemaFactory class="ClassicIndexSchemaFactory"/>
+
+ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
+
+ <updateHandler class="solr.DirectUpdateHandler2">
+ <commitWithin>
+ <softCommit>${solr.commitwithin.softcommit:true}</softCommit>
+ </commitWithin>
+ </updateHandler>
+
+ <requestHandler name="/select" class="solr.SearchHandler">
+ <lst name="defaults">
+ <str name="echoParams">explicit</str>
+ <str name="indent">true</str>
+ </lst>
+ </requestHandler>
+</config>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/src/test/org/apache/solr/backcompat/TestLuceneIndexBackCompat.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/backcompat/TestLuceneIndexBackCompat.java b/solr/core/src/test/org/apache/solr/backcompat/TestLuceneIndexBackCompat.java
new file mode 100644
index 0000000..2541d1f
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/backcompat/TestLuceneIndexBackCompat.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.backcompat;
+
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.lucene.index.TestBackwardsCompatibility;
+import org.apache.lucene.util.TestUtil;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.util.TestHarness;
+import org.junit.Test;
+
+/** Verify we can read/write previous versions' Lucene indexes. */
+public class TestLuceneIndexBackCompat extends SolrTestCaseJ4 {
+ private static final String[] oldNames = TestBackwardsCompatibility.getOldNames();
+ private static final String[] oldSingleSegmentNames = TestBackwardsCompatibility.getOldSingleSegmentNames();
+
+ @Test
+ public void testOldIndexes() throws Exception {
+ List<String> names = new ArrayList<>(oldNames.length + oldSingleSegmentNames.length);
+ names.addAll(Arrays.asList(oldNames));
+ names.addAll(Arrays.asList(oldSingleSegmentNames));
+ for (String name : names) {
+ setupCore(name);
+
+ assertQ(req("q", "*:*", "rows", "0"), "//result[@numFound='35']");
+
+ assertU(adoc("id", "id_123456789"));
+ assertU(commit());
+
+ deleteCore();
+ }
+ }
+
+ private void setupCore(String coreName) throws Exception {
+ if (h != null) {
+ h.close();
+ }
+ Path solrHome = createTempDir(coreName).toAbsolutePath();
+ Files.createDirectories(solrHome);
+ Path coreDir = solrHome.resolve(coreName);
+ Path confDir = coreDir.resolve("conf");
+ Files.createDirectories(confDir);
+ Path dataDir = coreDir.resolve("data");
+ Path indexDir = dataDir.resolve("index");
+ Files.createDirectories(indexDir);
+
+ Files.copy(getFile("solr/solr.xml").toPath(), solrHome.resolve("solr.xml"));
+ FileUtils.copyDirectory(configset("backcompat").toFile(), confDir.toFile());
+
+ try (Writer writer = new OutputStreamWriter(Files.newOutputStream(coreDir.resolve("core.properties")), StandardCharsets.UTF_8)) {
+ Properties coreProps = new Properties();
+ coreProps.put("name", coreName);
+ coreProps.store(writer, null);
+ }
+
+ InputStream resource = TestBackwardsCompatibility.class.getResourceAsStream("index." + coreName + ".zip");
+ assertNotNull("Index name " + coreName + " not found", resource);
+ TestUtil.unzip(resource, indexDir);
+
+ configString = "solrconfig.xml";
+ schemaString = "schema.xml";
+ testSolrHome = solrHome;
+ System.setProperty("solr.solr.home", solrHome.toString());
+ ignoreException("ignore_exception");
+ solrConfig = TestHarness.createConfig(testSolrHome, coreName, getSolrConfigFile());
+ h = new TestHarness(coreName, dataDir.toString(), solrConfig, getSchemaFile());
+ lrf = h.getRequestFactory("",0,20, CommonParams.VERSION,"2.2");
+ }
+}
[36/50] lucene-solr:feature/autoscaling_72: LUCENE-7982: add
NormsFieldExistsQuery
Posted by ab...@apache.org.
LUCENE-7982: add NormsFieldExistsQuery
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/0b11ee55
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/0b11ee55
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/0b11ee55
Branch: refs/heads/feature/autoscaling_72
Commit: 0b11ee5578c7930137d32c424d1173e23e3e158c
Parents: 3012239
Author: Mike McCandless <mi...@apache.org>
Authored: Wed Oct 4 10:20:54 2017 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Wed Oct 4 10:20:54 2017 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +
.../lucene/search/NormsFieldExistsQuery.java | 79 ++++++++
.../search/TestNormsFieldExistsQuery.java | 197 +++++++++++++++++++
3 files changed, 279 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0b11ee55/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index f3f04ed..7c35503 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -41,6 +41,9 @@ New Features
* LUCENE-7975: Change the default taxonomy facets cache to a faster
byte[] (UTF-8) based cache.
+* LUCENE-7982: A new NormsFieldExistsQuery matches documents that have
+ norms in a specified field (Colin Goodheart-Smithe via Mike McCandless)
+
Optimizations
* LUCENE-7905: Optimize how OrdinalMap (used by
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0b11ee55/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java b/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java
new file mode 100644
index 0000000..be0311e
--- /dev/null
+++ b/lucene/core/src/java/org/apache/lucene/search/NormsFieldExistsQuery.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+
+import java.io.IOException;
+import java.util.Objects;
+
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.index.FieldInfos;
+import org.apache.lucene.index.LeafReader;
+import org.apache.lucene.index.LeafReaderContext;
+
+/**
+ * A {@link Query} that matches documents that have a value for a given field
+ * as reported by field norms. This will not work for fields that omit norms,
+ * e.g. {@link StringField}.
+ */
+public final class NormsFieldExistsQuery extends Query {
+
+ private final String field;
+
+ /** Create a query that will match that have a value for the given
+ * {@code field}. */
+ public NormsFieldExistsQuery(String field) {
+ this.field = Objects.requireNonNull(field);
+ }
+
+ public String getField() {
+ return field;
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return sameClassAs(other) &&
+ field.equals(((NormsFieldExistsQuery) other).field);
+ }
+
+ @Override
+ public int hashCode() {
+ return 31 * classHash() + field.hashCode();
+ }
+
+ @Override
+ public String toString(String field) {
+ return "NormsFieldExistsQuery [field=" + this.field + "]";
+ }
+
+ @Override
+ public Weight createWeight(IndexSearcher searcher, boolean needsScores, float boost) throws IOException {
+ return new ConstantScoreWeight(this, boost) {
+ @Override
+ public Scorer scorer(LeafReaderContext context) throws IOException {
+ FieldInfos fieldInfos = context.reader().getFieldInfos();
+ FieldInfo fieldInfo = fieldInfos.fieldInfo(field);
+ if (fieldInfo == null || fieldInfo.hasNorms() == false) {
+ return null;
+ }
+ LeafReader reader = context.reader();
+ DocIdSetIterator iterator = reader.getNormValues(field);
+ return new ConstantScoreScorer(this, score(), iterator);
+ }
+ };
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0b11ee55/lucene/core/src/test/org/apache/lucene/search/TestNormsFieldExistsQuery.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/search/TestNormsFieldExistsQuery.java b/lucene/core/src/test/org/apache/lucene/search/TestNormsFieldExistsQuery.java
new file mode 100644
index 0000000..fba9e2f
--- /dev/null
+++ b/lucene/core/src/test/org/apache/lucene/search/TestNormsFieldExistsQuery.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+
+import java.io.IOException;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field.Store;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.document.TextField;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.RandomIndexWriter;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.LuceneTestCase;
+
+public class TestNormsFieldExistsQuery extends LuceneTestCase {
+
+ public void testRandom() throws IOException {
+ final int iters = atLeast(10);
+ for (int iter = 0; iter < iters; ++iter) {
+ Directory dir = newDirectory();
+ RandomIndexWriter iw = new RandomIndexWriter(random(), dir);
+ final int numDocs = atLeast(100);
+ for (int i = 0; i < numDocs; ++i) {
+ Document doc = new Document();
+ final boolean hasValue = random().nextBoolean();
+ if (hasValue) {
+ doc.add(new TextField("text1", "value", Store.NO));
+ doc.add(new StringField("has_value", "yes", Store.NO));
+ }
+ doc.add(new StringField("f", random().nextBoolean() ? "yes" : "no", Store.NO));
+ iw.addDocument(doc);
+ }
+ if (random().nextBoolean()) {
+ iw.deleteDocuments(new TermQuery(new Term("f", "no")));
+ }
+ iw.commit();
+ final IndexReader reader = iw.getReader();
+ final IndexSearcher searcher = newSearcher(reader);
+ iw.close();
+
+ assertSameMatches(searcher, new TermQuery(new Term("has_value", "yes")), new NormsFieldExistsQuery("text1"), false);
+
+ reader.close();
+ dir.close();
+ }
+ }
+
+ public void testApproximation() throws IOException {
+ final int iters = atLeast(10);
+ for (int iter = 0; iter < iters; ++iter) {
+ Directory dir = newDirectory();
+ RandomIndexWriter iw = new RandomIndexWriter(random(), dir);
+ final int numDocs = atLeast(100);
+ for (int i = 0; i < numDocs; ++i) {
+ Document doc = new Document();
+ final boolean hasValue = random().nextBoolean();
+ if (hasValue) {
+ doc.add(new TextField("text1", "value", Store.NO));
+ doc.add(new StringField("has_value", "yes", Store.NO));
+ }
+ doc.add(new StringField("f", random().nextBoolean() ? "yes" : "no", Store.NO));
+ iw.addDocument(doc);
+ }
+ if (random().nextBoolean()) {
+ iw.deleteDocuments(new TermQuery(new Term("f", "no")));
+ }
+ iw.commit();
+ final IndexReader reader = iw.getReader();
+ final IndexSearcher searcher = newSearcher(reader);
+ iw.close();
+
+ BooleanQuery.Builder ref = new BooleanQuery.Builder();
+ ref.add(new TermQuery(new Term("f", "yes")), Occur.MUST);
+ ref.add(new TermQuery(new Term("has_value", "yes")), Occur.FILTER);
+
+ BooleanQuery.Builder bq1 = new BooleanQuery.Builder();
+ bq1.add(new TermQuery(new Term("f", "yes")), Occur.MUST);
+ bq1.add(new NormsFieldExistsQuery("text1"), Occur.FILTER);
+ assertSameMatches(searcher, ref.build(), bq1.build(), true);
+
+ reader.close();
+ dir.close();
+ }
+ }
+
+ public void testScore() throws IOException {
+ final int iters = atLeast(10);
+ for (int iter = 0; iter < iters; ++iter) {
+ Directory dir = newDirectory();
+ RandomIndexWriter iw = new RandomIndexWriter(random(), dir);
+ final int numDocs = atLeast(100);
+ for (int i = 0; i < numDocs; ++i) {
+ Document doc = new Document();
+ final boolean hasValue = random().nextBoolean();
+ if (hasValue) {
+ doc.add(new TextField("text1", "value", Store.NO));
+ doc.add(new StringField("has_value", "yes", Store.NO));
+ }
+ doc.add(new StringField("f", random().nextBoolean() ? "yes" : "no", Store.NO));
+ iw.addDocument(doc);
+ }
+ if (random().nextBoolean()) {
+ iw.deleteDocuments(new TermQuery(new Term("f", "no")));
+ }
+ iw.commit();
+ final IndexReader reader = iw.getReader();
+ final IndexSearcher searcher = newSearcher(reader);
+ iw.close();
+
+ final float boost = random().nextFloat() * 10;
+ final Query ref = new BoostQuery(new ConstantScoreQuery(new TermQuery(new Term("has_value", "yes"))), boost);
+
+ final Query q1 = new BoostQuery(new NormsFieldExistsQuery("text1"), boost);
+ assertSameMatches(searcher, ref, q1, true);
+
+ reader.close();
+ dir.close();
+ }
+ }
+
+ public void testMissingField() throws IOException {
+ Directory dir = newDirectory();
+ RandomIndexWriter iw = new RandomIndexWriter(random(), dir);
+ iw.addDocument(new Document());
+ iw.commit();
+ final IndexReader reader = iw.getReader();
+ final IndexSearcher searcher = newSearcher(reader);
+ iw.close();
+ assertEquals(0, searcher.search(new NormsFieldExistsQuery("f"), 1).totalHits);
+ reader.close();
+ dir.close();
+ }
+
+ public void testAllDocsHaveField() throws IOException {
+ Directory dir = newDirectory();
+ RandomIndexWriter iw = new RandomIndexWriter(random(), dir);
+ Document doc = new Document();
+ doc.add(new TextField("f", "value", Store.NO));
+ iw.addDocument(doc);
+ iw.commit();
+ final IndexReader reader = iw.getReader();
+ final IndexSearcher searcher = newSearcher(reader);
+ iw.close();
+ assertEquals(1, searcher.search(new NormsFieldExistsQuery("f"), 1).totalHits);
+ reader.close();
+ dir.close();
+ }
+
+ public void testFieldExistsButNoDocsHaveField() throws IOException {
+ Directory dir = newDirectory();
+ RandomIndexWriter iw = new RandomIndexWriter(random(), dir);
+ // 1st segment has the field, but 2nd one does not
+ Document doc = new Document();
+ doc.add(new TextField("f", "value", Store.NO));
+ iw.addDocument(doc);
+ iw.commit();
+ iw.addDocument(new Document());
+ iw.commit();
+ final IndexReader reader = iw.getReader();
+ final IndexSearcher searcher = newSearcher(reader);
+ iw.close();
+ assertEquals(1, searcher.search(new NormsFieldExistsQuery("f"), 1).totalHits);
+ reader.close();
+ dir.close();
+ }
+
+ private void assertSameMatches(IndexSearcher searcher, Query q1, Query q2, boolean scores) throws IOException {
+ final int maxDoc = searcher.getIndexReader().maxDoc();
+ final TopDocs td1 = searcher.search(q1, maxDoc, scores ? Sort.RELEVANCE : Sort.INDEXORDER);
+ final TopDocs td2 = searcher.search(q2, maxDoc, scores ? Sort.RELEVANCE : Sort.INDEXORDER);
+ assertEquals(td1.totalHits, td2.totalHits);
+ for (int i = 0; i < td1.scoreDocs.length; ++i) {
+ assertEquals(td1.scoreDocs[i].doc, td2.scoreDocs[i].doc);
+ if (scores) {
+ assertEquals(td1.scoreDocs[i].score, td2.scoreDocs[i].score, 10e-7);
+ }
+ }
+ }
+}
[44/50] lucene-solr:feature/autoscaling_72: SOLR-11392: Change
collection names in test case
Posted by ab...@apache.org.
SOLR-11392: Change collection names in test case
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/070d6d37
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/070d6d37
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/070d6d37
Branch: refs/heads/feature/autoscaling_72
Commit: 070d6d3748341a955d807570d96896068a933f3e
Parents: 1782dd9
Author: Joel Bernstein <jb...@apache.org>
Authored: Wed Oct 4 14:29:15 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Wed Oct 4 14:29:15 2017 -0400
----------------------------------------------------------------------
.../solrj/io/stream/StreamExpressionTest.java | 38 ++++++++++----------
1 file changed, 19 insertions(+), 19 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/070d6d37/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index c73adf6..36333eb 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -7556,14 +7556,14 @@ public class StreamExpressionTest extends SolrCloudTestCase {
@Test
public void testParallelExecutorStream() throws Exception {
- CollectionAdminRequest.createCollection("workQueue", "conf", 2, 1).processAndWait(cluster.getSolrClient(),DEFAULT_TIMEOUT);
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("workQueue", cluster.getSolrClient().getZkStateReader(),
+ CollectionAdminRequest.createCollection("workQueue1", "conf", 2, 1).processAndWait(cluster.getSolrClient(),DEFAULT_TIMEOUT);
+ AbstractDistribZkTestBase.waitForRecoveriesToFinish("workQueue1", cluster.getSolrClient().getZkStateReader(),
false, true, TIMEOUT);
- CollectionAdminRequest.createCollection("mainCorpus", "conf", 2, 1).processAndWait(cluster.getSolrClient(), DEFAULT_TIMEOUT);
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("mainCorpus", cluster.getSolrClient().getZkStateReader(),
+ CollectionAdminRequest.createCollection("mainCorpus1", "conf", 2, 1).processAndWait(cluster.getSolrClient(), DEFAULT_TIMEOUT);
+ AbstractDistribZkTestBase.waitForRecoveriesToFinish("mainCorpus1", cluster.getSolrClient().getZkStateReader(),
false, true, TIMEOUT);
- CollectionAdminRequest.createCollection("destination", "conf", 2, 1).processAndWait(cluster.getSolrClient(), DEFAULT_TIMEOUT);
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("destination", cluster.getSolrClient().getZkStateReader(),
+ CollectionAdminRequest.createCollection("destination1", "conf", 2, 1).processAndWait(cluster.getSolrClient(), DEFAULT_TIMEOUT);
+ AbstractDistribZkTestBase.waitForRecoveriesToFinish("destination1", cluster.getSolrClient().getZkStateReader(),
false, true, TIMEOUT);
UpdateRequest workRequest = new UpdateRequest();
@@ -7571,27 +7571,27 @@ public class StreamExpressionTest extends SolrCloudTestCase {
for (int i = 0; i < 500; i++) {
- workRequest.add(id, String.valueOf(i), "expr_s", "update(destination, batchSize=50, search(mainCorpus, q=id:"+i+", rows=1, sort=\"id asc\", fl=\"id, body_t, field_i\"))");
+ workRequest.add(id, String.valueOf(i), "expr_s", "update(destination1, batchSize=50, search(mainCorpus1, q=id:"+i+", rows=1, sort=\"id asc\", fl=\"id, body_t, field_i\"))");
dataRequest.add(id, String.valueOf(i), "body_t", "hello world "+i, "field_i", Integer.toString(i));
}
- workRequest.commit(cluster.getSolrClient(), "workQueue");
- dataRequest.commit(cluster.getSolrClient(), "mainCorpus");
+ workRequest.commit(cluster.getSolrClient(), "workQueue1");
+ dataRequest.commit(cluster.getSolrClient(), "mainCorpus1");
- String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString() + "/destination";
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString() + "/destination1";
TupleStream executorStream;
ModifiableSolrParams paramsLoc;
StreamFactory factory = new StreamFactory()
- .withCollectionZkHost("workQueue", cluster.getZkServer().getZkAddress())
- .withCollectionZkHost("mainCorpus", cluster.getZkServer().getZkAddress())
- .withCollectionZkHost("destination", cluster.getZkServer().getZkAddress())
+ .withCollectionZkHost("workQueue1", cluster.getZkServer().getZkAddress())
+ .withCollectionZkHost("mainCorpus1", cluster.getZkServer().getZkAddress())
+ .withCollectionZkHost("destination1", cluster.getZkServer().getZkAddress())
.withFunctionName("search", CloudSolrStream.class)
.withFunctionName("executor", ExecutorStream.class)
.withFunctionName("parallel", ParallelStream.class)
.withFunctionName("update", UpdateStream.class);
- String executorExpression = "parallel(workQueue, workers=2, sort=\"EOF asc\", executor(threads=3, queueSize=100, search(workQueue, q=\"*:*\", fl=\"id, expr_s\", rows=1000, partitionKeys=id, sort=\"id desc\")))";
+ String executorExpression = "parallel(workQueue1, workers=2, sort=\"EOF asc\", executor(threads=3, queueSize=100, search(workQueue1, q=\"*:*\", fl=\"id, expr_s\", rows=1000, partitionKeys=id, sort=\"id desc\")))";
executorStream = factory.constructStream(executorExpression);
StreamContext context = new StreamContext();
@@ -7600,9 +7600,9 @@ public class StreamExpressionTest extends SolrCloudTestCase {
executorStream.setStreamContext(context);
getTuples(executorStream);
//Destination collection should now contain all the records in the main corpus.
- cluster.getSolrClient().commit("destination");
+ cluster.getSolrClient().commit("destination1");
paramsLoc = new ModifiableSolrParams();
- paramsLoc.set("expr", "search(destination, q=\"*:*\", fl=\"id, body_t, field_i\", rows=1000, sort=\"field_i asc\")");
+ paramsLoc.set("expr", "search(destination1, q=\"*:*\", fl=\"id, body_t, field_i\", rows=1000, sort=\"field_i asc\")");
paramsLoc.set("qt", "/stream");
SolrStream solrStream = new SolrStream(url, paramsLoc);
@@ -7618,9 +7618,9 @@ public class StreamExpressionTest extends SolrCloudTestCase {
solrStream.close();
clientCache.close();
- CollectionAdminRequest.deleteCollection("workQueue").process(cluster.getSolrClient());
- CollectionAdminRequest.deleteCollection("mainCorpus").process(cluster.getSolrClient());
- CollectionAdminRequest.deleteCollection("destination").process(cluster.getSolrClient());
+ CollectionAdminRequest.deleteCollection("workQueue1").process(cluster.getSolrClient());
+ CollectionAdminRequest.deleteCollection("mainCorpus1").process(cluster.getSolrClient());
+ CollectionAdminRequest.deleteCollection("destination1").process(cluster.getSolrClient());
}
[18/50] lucene-solr:feature/autoscaling_72: LUCENE-7974: nearest()
method returning NearestHit should be private,
and NearestHit class should be package private
Posted by ab...@apache.org.
LUCENE-7974: nearest() method returning NearestHit should be private, and NearestHit class should be package private
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/74050a3f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/74050a3f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/74050a3f
Branch: refs/heads/feature/autoscaling_72
Commit: 74050a3f159eca393ccb8e0b28c4fc4f974f6d5e
Parents: 73f3403
Author: Steve Rowe <sa...@gmail.com>
Authored: Mon Oct 2 10:39:31 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Mon Oct 2 10:39:31 2017 -0400
----------------------------------------------------------------------
.../org/apache/lucene/document/FloatPointNearestNeighbor.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/74050a3f/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
index b7859d0..90a80b6 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
@@ -194,7 +194,7 @@ public class FloatPointNearestNeighbor {
}
/** Holds one hit from {@link FloatPointNearestNeighbor#nearest} */
- public static class NearestHit {
+ static class NearestHit {
public int docID;
public double distanceSquared;
@@ -204,7 +204,7 @@ public class FloatPointNearestNeighbor {
}
}
- public static NearestHit[] nearest(List<BKDReader> readers, List<Bits> liveDocs, List<Integer> docBases, final int topN, float[] origin) throws IOException {
+ private static NearestHit[] nearest(List<BKDReader> readers, List<Bits> liveDocs, List<Integer> docBases, final int topN, float[] origin) throws IOException {
// System.out.println("NEAREST: readers=" + readers + " liveDocs=" + liveDocs + " origin: " + Arrays.toString(origin));
[41/50] lucene-solr:feature/autoscaling_72: LUCENE-7911: allow
javadoc links containing 'lucene.apache.org/solr/guide/'
Posted by ab...@apache.org.
LUCENE-7911: allow javadoc links containing 'lucene.apache.org/solr/guide/'
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/4392500a
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/4392500a
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/4392500a
Branch: refs/heads/feature/autoscaling_72
Commit: 4392500a3b19b2cf7111f2914daf6f23fce985d5
Parents: fe38dca
Author: Steve Rowe <sa...@apache.org>
Authored: Wed Oct 4 12:28:24 2017 -0400
Committer: Steve Rowe <sa...@apache.org>
Committed: Wed Oct 4 12:30:48 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/checkJavadocLinks.py | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/4392500a/dev-tools/scripts/checkJavadocLinks.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/checkJavadocLinks.py b/dev-tools/scripts/checkJavadocLinks.py
index 2e3cdea..d3ba89a 100644
--- a/dev-tools/scripts/checkJavadocLinks.py
+++ b/dev-tools/scripts/checkJavadocLinks.py
@@ -213,6 +213,9 @@ def checkAll(dirName):
elif link.find('lucene.apache.org/solr/quickstart.html') != -1:
# OK
pass
+ elif link.find('lucene.apache.org/solr/guide/') != -1:
+ # OK
+ pass
elif link.find('lucene.apache.org/solr/downloads.html') != -1:
# OK
pass