You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by kr...@apache.org on 2016/10/20 19:31:10 UTC
[01/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7491: also allow
for merging segment with points before one without points for a given field
(thank you MockRandomMergePolicy!)
Repository: lucene-solr
Updated Branches:
refs/heads/jira/solr-8593 d5d478e11 -> 2e7f5f928
LUCENE-7491: also allow for merging segment with points before one without points for a given field (thank you MockRandomMergePolicy!)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/f4d3ca89
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/f4d3ca89
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/f4d3ca89
Branch: refs/heads/jira/solr-8593
Commit: f4d3ca89802d06087780d7b7a1e7516c6b0f94c7
Parents: 4ae1643
Author: Mike McCandless <mi...@apache.org>
Authored: Thu Oct 13 10:20:12 2016 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Thu Oct 13 10:20:38 2016 -0400
----------------------------------------------------------------------
lucene/core/src/java/org/apache/lucene/index/FieldInfo.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f4d3ca89/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java b/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java
index 1118f27..4e02320 100644
--- a/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java
+++ b/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java
@@ -145,7 +145,7 @@ public final class FieldInfo {
if (this.pointDimensionCount == 0 && dimensionCount != 0) {
this.pointDimensionCount = dimensionCount;
this.pointNumBytes = dimensionNumBytes;
- } else if (this.pointDimensionCount != dimensionCount || this.pointNumBytes != dimensionNumBytes) {
+ } else if (dimensionCount != 0 && (this.pointDimensionCount != dimensionCount || this.pointNumBytes != dimensionNumBytes)) {
throw new IllegalArgumentException("cannot change field \"" + name + "\" from points dimensionCount=" + this.pointDimensionCount + ", numBytes=" + this.pointNumBytes + " to inconsistent dimensionCount=" + dimensionCount + ", numBytes=" + dimensionNumBytes);
}
[41/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-7580: Move defaults
in bin/solr.in.sh into bin/solr (incl. Windows)
Posted by kr...@apache.org.
SOLR-7580: Move defaults in bin/solr.in.sh into bin/solr (incl. Windows)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/8ae3304c
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/8ae3304c
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/8ae3304c
Branch: refs/heads/jira/solr-8593
Commit: 8ae3304c8631e1ba9a14aa0fc576e13b9198dfe5
Parents: 67ba19a
Author: David Smiley <ds...@apache.org>
Authored: Wed Oct 19 16:38:06 2016 -0400
Committer: David Smiley <ds...@apache.org>
Committed: Wed Oct 19 16:38:06 2016 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 10 +++++++--
solr/bin/solr | 54 +++++++++++++++++++++++++++++++---------------
solr/bin/solr.cmd | 55 ++++++++++++++++++++++++++++++++++-------------
solr/bin/solr.in.cmd | 55 +++++++++++++++++------------------------------
solr/bin/solr.in.sh | 30 +++++++-------------------
5 files changed, 113 insertions(+), 91 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8ae3304c/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 880718c..56f3b80 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -80,10 +80,10 @@ Upgrade Notes
----------------------
* If you use the JSON Facet API (json.facet) with method=stream, you must now set sort='index asc' to get the streaming
-behavior; otherwise it won't stream. Reminder: "method" is a hint that doesn't change defaults of other parameters.
+ behavior; otherwise it won't stream. Reminder: "method" is a hint that doesn't change defaults of other parameters.
* If you use the JSON Facet API (json.facet) to facet on a numeric field and if you use mincount=0 or if you set the
-prefix, then you will now get an error as these options are incompatible with numeric faceting.
+ prefix, then you will now get an error as these options are incompatible with numeric faceting.
* Solr's logging verbosity at the INFO level has been greatly reduced, and
you may need to update the log configs to use the DEBUG level to get the
@@ -94,6 +94,9 @@ prefix, then you will now get an error as these options are incompatible with nu
addition, MiniSolrCloudCluster#uploadConfigSet(File, String) has been
deprecated in favour of #uploadConfigSet(Path, String)
+* The bin/solr.in.sh (bin/solr.in.cmd on Windows) is now completely commented by default. Previously, this wasn't so,
+ which had the effect of masking existing environment variables.
+
New Features
----------------------
* SOLR-5725: facet.method=enum can bypass exact counts calculation with facet.exists=true, it just returns 1 for
@@ -298,6 +301,9 @@ Other Changes
* SOLR-9634: Deprecate collection methods on MiniSolrCloudCluster (Alan Woodward)
+* SOLR-7580: Moved defaults within bin/solr.in.sh (and bin/solr.in.cmd on Windows) to bin/solr (and bin/solr.cmd)
+ such that the default state of these files is to set nothing. This makes Solr work better with Docker. (David Smiley)
+
================== 6.2.1 ==================
Bug Fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8ae3304c/solr/bin/solr
----------------------------------------------------------------------
diff --git a/solr/bin/solr b/solr/bin/solr
index 90fffa1..df6b4d0 100755
--- a/solr/bin/solr
+++ b/solr/bin/solr
@@ -1409,15 +1409,21 @@ if [ "`echo $java_ver_out | grep -i "IBM J9"`" != "" ]; then
JAVA_VENDOR="IBM J9"
fi
+# Establish default opts no env var set (otherwise init to empty)
+if [ -z ${GC_LOG_OPTS+x} ]; then
+ GC_LOG_OPTS=('-verbose:gc' '-XX:+PrintHeapAtGC' '-XX:+PrintGCDetails' \
+ '-XX:+PrintGCDateStamps' '-XX:+PrintGCTimeStamps' '-XX:+PrintTenuringDistribution' \
+ '-XX:+PrintGCApplicationStoppedTime')
+else
+ GC_LOG_OPTS=($GC_LOG_OPTS)
+fi
# if verbose gc logging enabled, setup the location of the log file
if [ "$GC_LOG_OPTS" != "" ]; then
gc_log_flag="-Xloggc"
if [ "$JAVA_VENDOR" == "IBM J9" ]; then
gc_log_flag="-Xverbosegclog"
fi
- GC_LOG_OPTS=($GC_LOG_OPTS "$gc_log_flag:$SOLR_LOGS_DIR/solr_gc.log")
-else
- GC_LOG_OPTS=()
+ GC_LOG_OPTS+=("$gc_log_flag:$SOLR_LOGS_DIR/solr_gc.log")
fi
# If ZK_HOST is defined, the assume SolrCloud mode
@@ -1484,6 +1490,12 @@ else
JAVA_MEM_OPTS=("-Xms$SOLR_HEAP" "-Xmx$SOLR_HEAP")
fi
+# Pick default for Java thread stack size, and then add to SOLR_OPTS
+if [ -z ${SOLR_JAVA_STACK_SIZE+x} ]; then
+ SOLR_JAVA_STACK_SIZE='-Xss256k'
+fi
+SOLR_OPTS+=($SOLR_JAVA_STACK_SIZE)
+
if [ -z "$SOLR_TIMEZONE" ]; then
SOLR_TIMEZONE='UTC'
fi
@@ -1496,20 +1508,28 @@ function launch_solr() {
SOLR_ADDL_ARGS="$2"
- GC_TUNE=($GC_TUNE)
- # deal with Java version specific GC and other flags
- if [ "${JAVA_VERSION:0:3}" == "1.7" ]; then
- # Specific Java version hacking
- GC_TUNE+=('-XX:CMSFullGCsBeforeCompaction=1' '-XX:CMSTriggerPermRatio=80')
- if [ "$JAVA_VENDOR" != "IBM J9" ]; then
- JAVA_MINOR_VERSION=${JAVA_VERSION:(-2)}
- if [[ $JAVA_MINOR_VERSION -ge 40 && $JAVA_MINOR_VERSION -le 51 ]]; then
- GC_TUNE+=('-XX:-UseSuperWord')
- echo -e "\nWARNING: Java version $JAVA_VERSION has known bugs with Lucene and requires the -XX:-UseSuperWord flag. Please consider upgrading your JVM.\n"
- fi
- fi
+ # define default GC_TUNE
+ if [ -z ${GC_TUNE+x} ]; then
+ GC_TUNE=('-XX:NewRatio=3' \
+ '-XX:SurvivorRatio=4' \
+ '-XX:TargetSurvivorRatio=90' \
+ '-XX:MaxTenuringThreshold=8' \
+ '-XX:+UseConcMarkSweepGC' \
+ '-XX:+UseParNewGC' \
+ '-XX:ConcGCThreads=4' '-XX:ParallelGCThreads=4' \
+ '-XX:+CMSScavengeBeforeRemark' \
+ '-XX:PretenureSizeThreshold=64m' \
+ '-XX:+UseCMSInitiatingOccupancyOnly' \
+ '-XX:CMSInitiatingOccupancyFraction=50' \
+ '-XX:CMSMaxAbortablePrecleanTime=6000' \
+ '-XX:+CMSParallelRemarkEnabled' \
+ '-XX:+ParallelRefProcEnabled' \
+ '-XX:-OmitStackTraceInFastThrow')
+ else
+ GC_TUNE=($GC_TUNE)
fi
+
# If SSL-related system props are set, add them to SOLR_OPTS
if [ -n "$SOLR_SSL_OPTS" ]; then
# If using SSL and solr.jetty.https.port not set explicitly, use the jetty.port
@@ -1540,11 +1560,11 @@ function launch_solr() {
fi
if [ "$SOLR_OPTS" != "" ]; then
- echo -e " SOLR_OPTS = ${SOLR_OPTS[@]}"
+ echo -e " SOLR_OPTS = ${SOLR_OPTS[@]}"
fi
if [ "$SOLR_ADDL_ARGS" != "" ]; then
- echo -e " SOLR_ADDL_ARGS = $SOLR_ADDL_ARGS"
+ echo -e " SOLR_ADDL_ARGS = $SOLR_ADDL_ARGS"
fi
if [ "$ENABLE_REMOTE_JMX_OPTS" == "true" ]; then
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8ae3304c/solr/bin/solr.cmd
----------------------------------------------------------------------
diff --git a/solr/bin/solr.cmd b/solr/bin/solr.cmd
index 5376143..10ea6d6 100644
--- a/solr/bin/solr.cmd
+++ b/solr/bin/solr.cmd
@@ -62,6 +62,9 @@ IF NOT "%SOLR_HOST%"=="" (
) ELSE (
set "SOLR_TOOL_HOST=localhost"
)
+IF "%SOLR_JETTY_HOST%"=="" (
+ set SOLR_JETTY_HOST=0.0.0.0
+)
REM Verify Java is available
IF DEFINED SOLR_JAVA_HOME set "JAVA_HOME=%SOLR_JAVA_HOME%"
@@ -910,16 +913,36 @@ IF "%ENABLE_REMOTE_JMX_OPTS%"=="true" (
IF NOT "%SOLR_HEAP%"=="" set SOLR_JAVA_MEM=-Xms%SOLR_HEAP% -Xmx%SOLR_HEAP%
IF "%SOLR_JAVA_MEM%"=="" set SOLR_JAVA_MEM=-Xms512m -Xmx512m
+IF "%SOLR_JAVA_STACK_SIZE%"=="" set SOLR_JAVA_STACK_SIZE=-Xss256k
+set SOLR_OPTS=%SOLR_JAVA_STACK_SIZE% %SOLR_OPTS%
IF "%SOLR_TIMEZONE%"=="" set SOLR_TIMEZONE=UTC
-IF "!JAVA_MAJOR_VERSION!"=="7" (
- set "GC_TUNE=%GC_TUNE% -XX:CMSFullGCsBeforeCompaction=1 -XX:CMSTriggerPermRatio=80"
- IF !JAVA_BUILD! GEQ 40 (
- IF !JAVA_BUILD! LEQ 51 (
- set "GC_TUNE=!GC_TUNE! -XX:-UseSuperWord"
- @echo WARNING: Java version !JAVA_VERSION_INFO! has known bugs with Lucene and requires the -XX:-UseSuperWord flag. Please consider upgrading your JVM.
- )
- )
+IF "%GC_TUNE%"=="" (
+ set GC_TUNE=-XX:NewRatio=3 ^
+ -XX:SurvivorRatio=4 ^
+ -XX:TargetSurvivorRatio=90 ^
+ -XX:MaxTenuringThreshold=8 ^
+ -XX:+UseConcMarkSweepGC ^
+ -XX:+UseParNewGC ^
+ -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 ^
+ -XX:+CMSScavengeBeforeRemark ^
+ -XX:PretenureSizeThreshold=64m ^
+ -XX:+UseCMSInitiatingOccupancyOnly ^
+ -XX:CMSInitiatingOccupancyFraction=50 ^
+ -XX:CMSMaxAbortablePrecleanTime=6000 ^
+ -XX:+CMSParallelRemarkEnabled ^
+ -XX:+ParallelRefProcEnabled ^
+ -XX:-OmitStackTraceInFastThrow
+)
+
+IF "%GC_LOG_OPTS%"=="" (
+ set GC_LOG_OPTS=-verbose:gc ^
+ -XX:+PrintHeapAtGC ^
+ -XX:+PrintGCDetails ^
+ -XX:+PrintGCDateStamps ^
+ -XX:+PrintGCTimeStamps ^
+ -XX:+PrintTenuringDistribution ^
+ -XX:+PrintGCApplicationStoppedTime
)
IF "%verbose%"=="1" (
@@ -1009,15 +1032,17 @@ IF "%FG%"=="1" (
echo %SOLR_PORT%>"%SOLR_TIP%"\bin\solr-%SOLR_PORT%.port
"%JAVA%" %SERVEROPT% %SOLR_JAVA_MEM% %START_OPTS% %GCLOG_OPT%:"!SOLR_LOGS_DIR!/solr_gc.log" ^
-Dlog4j.configuration="%LOG4J_CONFIG%" -DSTOP.PORT=!STOP_PORT! -DSTOP.KEY=%STOP_KEY% ^
- -Djetty.port=%SOLR_PORT% -Dsolr.solr.home="%SOLR_HOME%" -Dsolr.install.dir="%SOLR_TIP%" ^
- -Djetty.home="%SOLR_SERVER_DIR%" -Djava.io.tmpdir="%SOLR_SERVER_DIR%\tmp" -jar start.jar "%SOLR_JETTY_CONFIG%"
+ -Dsolr.solr.home="%SOLR_HOME%" -Dsolr.install.dir="%SOLR_TIP%" ^
+ -Djetty.host=%SOLR_JETTY_HOST% -Djetty.port=%SOLR_PORT% -Djetty.home="%SOLR_SERVER_DIR%" ^
+ -Djava.io.tmpdir="%SOLR_SERVER_DIR%\tmp" -jar start.jar "%SOLR_JETTY_CONFIG%"
) ELSE (
- START /B "Solr-%SOLR_PORT%" /D "%SOLR_SERVER_DIR%" "%JAVA%" %SERVEROPT% %SOLR_JAVA_MEM% %START_OPTS% ^
- %GCLOG_OPT%:"!SOLR_LOGS_DIR!/solr_gc.log" -Dlog4j.configuration="%LOG4J_CONFIG%" -DSTOP.PORT=!STOP_PORT! ^
+ START /B "Solr-%SOLR_PORT%" /D "%SOLR_SERVER_DIR%" ^
+ "%JAVA%" %SERVEROPT% %SOLR_JAVA_MEM% %START_OPTS% %GCLOG_OPT%:"!SOLR_LOGS_DIR!/solr_gc.log" ^
+ -Dlog4j.configuration="%LOG4J_CONFIG%" -DSTOP.PORT=!STOP_PORT! -DSTOP.KEY=%STOP_KEY% ^
-Dsolr.log.muteconsole ^
- -DSTOP.KEY=%STOP_KEY% -Djetty.port=%SOLR_PORT% -Dsolr.solr.home="%SOLR_HOME%" -Dsolr.install.dir="%SOLR_TIP%" ^
- -Djetty.home="%SOLR_SERVER_DIR%" -Djava.io.tmpdir="%SOLR_SERVER_DIR%\tmp" -jar start.jar ^
- "%SOLR_JETTY_CONFIG%" > "!SOLR_LOGS_DIR!\solr-%SOLR_PORT%-console.log"
+ -Dsolr.solr.home="%SOLR_HOME%" -Dsolr.install.dir="%SOLR_TIP%" ^
+ -Djetty.host=%SOLR_JETTY_HOST% -Djetty.port=%SOLR_PORT% -Djetty.home="%SOLR_SERVER_DIR%" ^
+ -Djava.io.tmpdir="%SOLR_SERVER_DIR%\tmp" -jar start.jar "%SOLR_JETTY_CONFIG%" > "!SOLR_LOGS_DIR!\solr-%SOLR_PORT%-console.log"
echo %SOLR_PORT%>"%SOLR_TIP%"\bin\solr-%SOLR_PORT%.port
REM now wait to see Solr come online ...
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8ae3304c/solr/bin/solr.in.cmd
----------------------------------------------------------------------
diff --git a/solr/bin/solr.in.cmd b/solr/bin/solr.in.cmd
index 5a2d00f..06da233 100644
--- a/solr/bin/solr.in.cmd
+++ b/solr/bin/solr.in.cmd
@@ -16,40 +16,23 @@
@echo off
+REM Settings here will override settings in existing env vars or in bin/solr. The default shipped state
+REM of this file is completely commented.
+
REM By default the script will use JAVA_HOME to determine which java
REM to use, but you can set a specific path for Solr to use without
REM affecting other Java applications on your server/workstation.
REM set SOLR_JAVA_HOME=
REM Increase Java Min/Max Heap as needed to support your indexing / query needs
-set SOLR_JAVA_MEM=-Xms512m -Xmx512m
+REM set SOLR_JAVA_MEM=-Xms512m -Xmx512m
REM Enable verbose GC logging
-set GC_LOG_OPTS=-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-
-REM Changes the logging level. Valid values: ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF. Default is INFO
-REM This is an alternative to changing the rootLogger in log4j.properties
-REM set SOLR_LOG_LEVEL=INFO
-
-REM Location where Solr should write logs to. Absolute or relative to solr start dir
-REM set SOLR_LOGS_DIR=logs
+REM set GC_LOG_OPTS=-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-REM These GC settings have shown to work well for a number of common Solr workloads
-set GC_TUNE=-XX:NewRatio=3 ^
- -XX:SurvivorRatio=4 ^
- -XX:TargetSurvivorRatio=90 ^
- -XX:MaxTenuringThreshold=8 ^
- -XX:+UseConcMarkSweepGC ^
- -XX:+UseParNewGC ^
- -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 ^
- -XX:+CMSScavengeBeforeRemark ^
- -XX:PretenureSizeThreshold=64m ^
- -XX:+UseCMSInitiatingOccupancyOnly ^
- -XX:CMSInitiatingOccupancyFraction=50 ^
- -XX:CMSMaxAbortablePrecleanTime=6000 ^
- -XX:+CMSParallelRemarkEnabled ^
- -XX:+ParallelRefProcEnabled ^
- -XX:-OmitStackTraceInFastThrow
+REM Various GC settings have shown to work well for a number of common Solr workloads.
+REM See solr.cmd GC_TUNE for the default list.
+REM set GC_TUNE=-XX:NewRatio=3 -XX:SurvivorRatio=4 etc.
REM Set the ZooKeeper connection string if using an external ZooKeeper ensemble
REM e.g. host1:2181,host2:2181/chroot
@@ -69,20 +52,11 @@ REM set SOLR_TIMEZONE=UTC
REM Set to true to activate the JMX RMI connector to allow remote JMX client applications
REM to monitor the JVM hosting Solr; set to "false" to disable that behavior
REM (false is recommended in production environments)
-set ENABLE_REMOTE_JMX_OPTS=false
+REM set ENABLE_REMOTE_JMX_OPTS=false
REM The script will use SOLR_PORT+10000 for the RMI_PORT or you can set it here
REM set RMI_PORT=18983
-REM Set the host interface to listen on. Jetty will listen on all interfaces (0.0.0.0) by default.
-REM This must be an IPv4 ("a.b.c.d") or bracketed IPv6 ("[x::y]") address, not a hostname!
-set SOLR_JETTY_HOST=0.0.0.0
-
-set SOLR_OPTS=%SOLR_OPTS% -Djetty.host=%SOLR_JETTY_HOST%
-
-REM Set the thread stack size
-set SOLR_OPTS=%SOLR_OPTS% -Xss256k
-
REM Anything you add to the SOLR_OPTS variable will be included in the java
REM start command line as-is, in ADDITION to other options. If you specify the
REM -a option on start script, those options will be appended as well. Examples:
@@ -94,6 +68,17 @@ REM Path to a directory for Solr to store cores and their data. By default, Solr
REM If solr.xml is not stored in ZooKeeper, this directory needs to contain solr.xml
REM set SOLR_HOME=
+REM Changes the logging level. Valid values: ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF. Default is INFO
+REM This is an alternative to changing the rootLogger in log4j.properties
+REM set SOLR_LOG_LEVEL=INFO
+
+REM Location where Solr should write logs to. Absolute or relative to solr start dir
+REM set SOLR_LOGS_DIR=logs
+
+REM Set the host interface to listen on. Jetty will listen on all interfaces (0.0.0.0) by default.
+REM This must be an IPv4 ("a.b.c.d") or bracketed IPv6 ("[x::y]") address, not a hostname!
+REM set SOLR_JETTY_HOST=0.0.0.0
+
REM Sets the port Solr binds to, default is 8983
REM set SOLR_PORT=8983
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8ae3304c/solr/bin/solr.in.sh
----------------------------------------------------------------------
diff --git a/solr/bin/solr.in.sh b/solr/bin/solr.in.sh
index dd41f46..2fcaabb 100644
--- a/solr/bin/solr.in.sh
+++ b/solr/bin/solr.in.sh
@@ -13,38 +13,27 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+# Settings here will override settings in existing env vars or in bin/solr. The default shipped state
+# of this file is completely commented.
+
# By default the script will use JAVA_HOME to determine which java
# to use, but you can set a specific path for Solr to use without
# affecting other Java applications on your server/workstation.
#SOLR_JAVA_HOME=""
# Increase Java Heap as needed to support your indexing / query needs
-SOLR_HEAP="512m"
+#SOLR_HEAP="512m"
# Expert: If you want finer control over memory options, specify them directly
# Comment out SOLR_HEAP if you are using this though, that takes precedence
#SOLR_JAVA_MEM="-Xms512m -Xmx512m"
# Enable verbose GC logging
-GC_LOG_OPTS="-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails \
--XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime"
+#GC_LOG_OPTS="-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails \
+#-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime"
# These GC settings have shown to work well for a number of common Solr workloads
-GC_TUNE="-XX:NewRatio=3 \
--XX:SurvivorRatio=4 \
--XX:TargetSurvivorRatio=90 \
--XX:MaxTenuringThreshold=8 \
--XX:+UseConcMarkSweepGC \
--XX:+UseParNewGC \
--XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
--XX:+CMSScavengeBeforeRemark \
--XX:PretenureSizeThreshold=64m \
--XX:+UseCMSInitiatingOccupancyOnly \
--XX:CMSInitiatingOccupancyFraction=50 \
--XX:CMSMaxAbortablePrecleanTime=6000 \
--XX:+CMSParallelRemarkEnabled \
--XX:+ParallelRefProcEnabled \
--XX:-OmitStackTraceInFastThrow"
+#GC_TUNE="-XX:NewRatio=3 -XX:SurvivorRatio=4 etc.
# Set the ZooKeeper connection string if using an external ZooKeeper ensemble
# e.g. host1:2181,host2:2181/chroot
@@ -64,14 +53,11 @@ GC_TUNE="-XX:NewRatio=3 \
# Set to true to activate the JMX RMI connector to allow remote JMX client applications
# to monitor the JVM hosting Solr; set to "false" to disable that behavior
# (false is recommended in production environments)
-ENABLE_REMOTE_JMX_OPTS="false"
+#ENABLE_REMOTE_JMX_OPTS="false"
# The script will use SOLR_PORT+10000 for the RMI_PORT or you can set it here
# RMI_PORT=18983
-# Set the thread stack size
-SOLR_OPTS="$SOLR_OPTS -Xss256k"
-
# Anything you add to the SOLR_OPTS variable will be included in the java
# start command line as-is, in ADDITION to other options. If you specify the
# -a option on start script, those options will be appended as well. Examples:
[50/50] [abbrv] lucene-solr:jira/solr-8593: Merge branch
'apache-https-master' into jira/solr-8593
Posted by kr...@apache.org.
Merge branch 'apache-https-master' into jira/solr-8593
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2e7f5f92
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/2e7f5f92
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/2e7f5f92
Branch: refs/heads/jira/solr-8593
Commit: 2e7f5f9283519d8aefd31e241b9307ff61d72173
Parents: d5d478e 9776196
Author: Kevin Risden <kr...@apache.org>
Authored: Thu Oct 20 14:24:16 2016 -0500
Committer: Kevin Risden <kr...@apache.org>
Committed: Thu Oct 20 14:24:16 2016 -0500
----------------------------------------------------------------------
build.xml | 41 +-
.../dot.settings/org.eclipse.jdt.core.prefs | 3 +
.../idea/.idea/copyright/profiles_settings.xml | 1 -
.../lucene/spatial-extras/spatial-extras.iml | 1 +
.../test-framework/lucene-test-framework.iml | 1 +
.../idea/solr/contrib/analytics/analytics.iml | 1 +
dev-tools/idea/solr/core/src/java/solr-core.iml | 1 +
.../idea/solr/core/src/solr-core-tests.iml | 1 +
.../solr/test-framework/solr-test-framework.iml | 3 +
.../lucene/analysis/common/pom.xml.template | 13 +
dev-tools/maven/lucene/pom.xml.template | 6 +-
dev-tools/maven/pom.xml.template | 10 +-
.../contrib/analysis-extras/pom.xml.template | 7 +
dev-tools/maven/solr/pom.xml.template | 8 +-
.../maven/solr/test-framework/pom.xml.template | 10 +-
dev-tools/scripts/addBackcompatIndexes.py | 30 +-
dev-tools/scripts/addVersion.py | 118 +-
dev-tools/scripts/buildAndPushRelease.py | 89 +-
dev-tools/scripts/checkJavaDocs.py | 74 +-
dev-tools/scripts/checkJavadocLinks.py | 3 +
dev-tools/scripts/poll-mirrors.pl | 155 -
dev-tools/scripts/poll-mirrors.py | 162 +
dev-tools/scripts/releasedJirasRegex.py | 93 +
dev-tools/scripts/scriptutil.py | 24 +-
dev-tools/scripts/smokeTestRelease.py | 42 +-
lucene/CHANGES.txt | 505 +-
lucene/MIGRATE.txt | 104 +-
lucene/analysis/common/build.xml | 39 +-
.../lucene/analysis/ar/ArabicAnalyzer.java | 21 +-
.../lucene/analysis/bg/BulgarianAnalyzer.java | 17 +-
.../lucene/analysis/br/BrazilianAnalyzer.java | 17 +-
.../lucene/analysis/ca/CatalanAnalyzer.java | 18 +-
.../charfilter/HTMLStripCharFilter.java | 6 +-
.../charfilter/HTMLStripCharFilter.jflex | 1 +
.../analysis/charfilter/NormalizeCharMap.java | 2 -
.../apache/lucene/analysis/cjk/CJKAnalyzer.java | 15 +-
.../lucene/analysis/ckb/SoraniAnalyzer.java | 24 +-
.../analysis/commongrams/CommonGramsFilter.java | 4 +-
.../commongrams/CommonGramsFilterFactory.java | 3 +-
.../compound/CompoundWordTokenFilterBase.java | 8 +-
.../DictionaryCompoundWordTokenFilter.java | 2 +-
...ictionaryCompoundWordTokenFilterFactory.java | 8 +-
.../HyphenationCompoundWordTokenFilter.java | 8 +-
...phenationCompoundWordTokenFilterFactory.java | 12 +-
.../analysis/core/DecimalDigitFilter.java | 2 +-
.../lucene/analysis/core/LowerCaseFilter.java | 22 +-
.../analysis/core/LowerCaseFilterFactory.java | 1 -
.../lucene/analysis/core/SimpleAnalyzer.java | 7 +
.../lucene/analysis/core/StopAnalyzer.java | 32 +-
.../apache/lucene/analysis/core/StopFilter.java | 80 +-
.../lucene/analysis/core/StopFilterFactory.java | 10 +-
.../lucene/analysis/core/TypeTokenFilter.java | 2 +-
.../lucene/analysis/core/UpperCaseFilter.java | 6 +-
.../lucene/analysis/custom/CustomAnalyzer.java | 28 +-
.../lucene/analysis/cz/CzechAnalyzer.java | 25 +-
.../lucene/analysis/da/DanishAnalyzer.java | 19 +-
.../lucene/analysis/de/GermanAnalyzer.java | 20 +-
.../lucene/analysis/el/GreekAnalyzer.java | 13 +-
.../analysis/el/GreekLowerCaseFilter.java | 4 +-
.../apache/lucene/analysis/el/GreekStemmer.java | 2 +-
.../lucene/analysis/en/EnglishAnalyzer.java | 15 +-
.../org/apache/lucene/analysis/en/KStemmer.java | 2 +-
.../lucene/analysis/en/PorterStemmer.java | 4 -
.../lucene/analysis/es/SpanishAnalyzer.java | 19 +-
.../lucene/analysis/eu/BasqueAnalyzer.java | 17 +-
.../lucene/analysis/fa/PersianAnalyzer.java | 23 +-
.../lucene/analysis/fi/FinnishAnalyzer.java | 19 +-
.../lucene/analysis/fr/FrenchAnalyzer.java | 32 +-
.../lucene/analysis/ga/IrishAnalyzer.java | 16 +-
.../lucene/analysis/gl/GalicianAnalyzer.java | 19 +-
.../lucene/analysis/hi/HindiAnalyzer.java | 24 +-
.../lucene/analysis/hu/HungarianAnalyzer.java | 19 +-
.../lucene/analysis/hunspell/Stemmer.java | 4 +-
.../lucene/analysis/hy/ArmenianAnalyzer.java | 17 +-
.../lucene/analysis/id/IndonesianAnalyzer.java | 17 +-
.../lucene/analysis/it/ItalianAnalyzer.java | 20 +-
.../lucene/analysis/lt/LithuanianAnalyzer.java | 17 +-
.../lucene/analysis/lv/LatvianAnalyzer.java | 19 +-
.../lucene/analysis/minhash/MinHashFilter.java | 515 ++
.../analysis/minhash/MinHashFilterFactory.java | 58 +
.../lucene/analysis/minhash/package-info.java | 21 +
.../miscellaneous/ASCIIFoldingFilter.java | 24 +-
.../miscellaneous/CapitalizationFilter.java | 2 +-
.../CapitalizationFilterFactory.java | 8 +-
.../miscellaneous/CodepointCountFilter.java | 2 +-
.../miscellaneous/DateRecognizerFilter.java | 2 +-
.../miscellaneous/FingerprintFilter.java | 2 +-
.../analysis/miscellaneous/KeepWordFilter.java | 4 +-
.../miscellaneous/KeepWordFilterFactory.java | 8 +-
.../KeywordMarkerFilterFactory.java | 2 +-
.../analysis/miscellaneous/LengthFilter.java | 2 +-
.../RemoveDuplicatesTokenFilter.java | 2 +-
.../ScandinavianFoldingFilterFactory.java | 10 +-
.../ScandinavianNormalizationFilterFactory.java | 10 +-
.../miscellaneous/SetKeywordMarkerFilter.java | 2 +-
.../miscellaneous/WordDelimiterFilter.java | 10 +-
.../WordDelimiterFilterFactory.java | 18 +-
.../analysis/ngram/EdgeNGramTokenFilter.java | 7 +-
.../analysis/ngram/EdgeNGramTokenizer.java | 1 -
.../lucene/analysis/ngram/NGramTokenFilter.java | 13 +-
.../lucene/analysis/ngram/NGramTokenizer.java | 10 +-
.../lucene/analysis/nl/DutchAnalyzer.java | 30 +-
.../lucene/analysis/no/NorwegianAnalyzer.java | 19 +-
.../pattern/PatternCaptureGroupTokenFilter.java | 1 -
.../lucene/analysis/pt/PortugueseAnalyzer.java | 19 +-
.../lucene/analysis/pt/RSLPStemmerBase.java | 2 +-
.../query/QueryAutoStopWordAnalyzer.java | 5 +-
.../lucene/analysis/ro/RomanianAnalyzer.java | 17 +-
.../lucene/analysis/ru/RussianAnalyzer.java | 23 +-
.../analysis/snowball/SnowballFilter.java | 4 +-
.../snowball/SnowballPorterFilterFactory.java | 6 +-
.../analysis/standard/ASCIITLD.jflex-macro | 2 +-
.../analysis/standard/ClassicAnalyzer.java | 15 +-
.../analysis/standard/ClassicTokenizer.java | 1 +
.../analysis/standard/ClassicTokenizerImpl.java | 1 -
.../standard/ClassicTokenizerImpl.jflex | 1 +
.../analysis/standard/StandardAnalyzer.java | 98 -
.../analysis/standard/StandardFilter.java | 38 -
.../analysis/standard/StandardTokenizer.java | 201 -
.../standard/StandardTokenizerImpl.java | 818 ---
.../standard/StandardTokenizerImpl.jflex | 201 -
.../standard/UAX29URLEmailAnalyzer.java | 19 +-
.../standard/UAX29URLEmailTokenizer.java | 1 +
.../standard/UAX29URLEmailTokenizerFactory.java | 1 -
.../standard/UAX29URLEmailTokenizerImpl.jflex | 1 +
.../lucene/analysis/standard/package-info.java | 63 -
.../lucene/analysis/standard/package.html | 54 +
.../lucene/analysis/sv/SwedishAnalyzer.java | 19 +-
.../analysis/synonym/SynonymFilterFactory.java | 4 +-
.../lucene/analysis/synonym/SynonymMap.java | 2 -
.../apache/lucene/analysis/th/ThaiAnalyzer.java | 16 +-
.../lucene/analysis/tr/TurkishAnalyzer.java | 16 +-
.../analysis/util/AbstractAnalysisFactory.java | 4 +-
.../lucene/analysis/util/CharArrayMap.java | 672 ---
.../lucene/analysis/util/CharArraySet.java | 193 -
.../lucene/analysis/util/CharTokenizer.java | 15 +-
.../lucene/analysis/util/CharacterUtils.java | 392 --
.../lucene/analysis/util/ElisionFilter.java | 2 +-
.../analysis/util/ElisionFilterFactory.java | 1 +
.../analysis/util/FilteringTokenFilter.java | 76 -
.../analysis/util/StopwordAnalyzerBase.java | 138 -
.../lucene/analysis/util/UnicodeProps.java | 4 +-
.../lucene/analysis/util/WordlistLoader.java | 244 -
.../analysis/wikipedia/WikipediaTokenizer.java | 1 -
.../wikipedia/WikipediaTokenizerImpl.java | 1 +
.../wikipedia/WikipediaTokenizerImpl.jflex | 1 +
.../lucene/collation/CollationKeyAnalyzer.java | 7 +-
.../apache/lucene/collation/package-info.java | 2 +-
...ache.lucene.analysis.util.TokenFilterFactory | 1 +
.../lucene/analysis/ar/TestArabicAnalyzer.java | 3 +-
.../ar/TestArabicNormalizationFilter.java | 1 -
.../analysis/ar/TestArabicStemFilter.java | 2 +-
.../analysis/bg/TestBulgarianAnalyzer.java | 2 +-
.../bg/TestBulgarianStemFilterFactory.java | 1 -
.../analysis/bg/TestBulgarianStemmer.java | 2 +-
.../analysis/br/TestBrazilianAnalyzer.java | 3 +-
.../lucene/analysis/ca/TestCatalanAnalyzer.java | 2 +-
.../TestHTMLStripCharFilterFactory.java | 1 -
.../lucene/analysis/cjk/TestCJKAnalyzer.java | 4 +-
.../analysis/cjk/TestCJKWidthFilterFactory.java | 1 -
.../lucene/analysis/ckb/TestSoraniAnalyzer.java | 3 +-
.../TestSoraniNormalizationFilterFactory.java | 1 -
.../commongrams/CommonGramsFilterTest.java | 3 +-
.../TestCommonGramsFilterFactory.java | 10 +-
.../TestCommonGramsQueryFilterFactory.java | 10 +-
.../compound/TestCompoundWordTokenFilter.java | 2 +-
.../core/TestAllAnalyzersHaveFactories.java | 8 +-
.../lucene/analysis/core/TestAnalyzers.java | 5 +
.../analysis/core/TestBugInSomething.java | 2 +-
.../analysis/core/TestDecimalDigitFilter.java | 149 +-
.../analysis/core/TestKeywordAnalyzer.java | 1 -
.../lucene/analysis/core/TestRandomChains.java | 22 +-
.../lucene/analysis/core/TestStopAnalyzer.java | 14 +-
.../lucene/analysis/core/TestStopFilter.java | 176 -
.../analysis/core/TestStopFilterFactory.java | 2 +-
.../analysis/custom/TestCustomAnalyzer.java | 143 +
.../lucene/analysis/cz/TestCzechAnalyzer.java | 4 +-
.../lucene/analysis/cz/TestCzechStemmer.java | 2 +-
.../lucene/analysis/da/TestDanishAnalyzer.java | 2 +-
.../lucene/analysis/de/TestGermanAnalyzer.java | 2 +-
.../analysis/de/TestGermanLightStemFilter.java | 2 +-
.../de/TestGermanLightStemFilterFactory.java | 1 -
.../de/TestGermanMinimalStemFilter.java | 2 +-
.../analysis/de/TestGermanStemFilter.java | 5 +-
.../el/TestGreekLowerCaseFilterFactory.java | 1 -
.../analysis/el/TestGreekStemFilterFactory.java | 1 -
.../lucene/analysis/en/TestEnglishAnalyzer.java | 2 +-
.../en/TestEnglishMinimalStemFilterFactory.java | 1 -
.../analysis/en/TestKStemFilterFactory.java | 1 -
.../analysis/en/TestPorterStemFilter.java | 8 +-
.../en/TestPorterStemFilterFactory.java | 1 -
.../lucene/analysis/es/TestSpanishAnalyzer.java | 2 +-
.../lucene/analysis/eu/TestBasqueAnalyzer.java | 2 +-
.../lucene/analysis/fa/TestPersianAnalyzer.java | 3 +-
.../TestPersianNormalizationFilterFactory.java | 1 -
.../lucene/analysis/fi/TestFinnishAnalyzer.java | 2 +-
.../analysis/fi/TestFinnishLightStemFilter.java | 2 +-
.../lucene/analysis/fr/TestFrenchAnalyzer.java | 2 +-
.../analysis/fr/TestFrenchLightStemFilter.java | 2 +-
.../fr/TestFrenchLightStemFilterFactory.java | 1 -
.../fr/TestFrenchMinimalStemFilter.java | 2 +-
.../fr/TestFrenchMinimalStemFilterFactory.java | 1 -
.../lucene/analysis/ga/TestIrishAnalyzer.java | 2 +-
.../ga/TestIrishLowerCaseFilterFactory.java | 1 -
.../analysis/gl/TestGalicianAnalyzer.java | 2 +-
.../gl/TestGalicianMinimalStemFilter.java | 2 +-
.../TestGalicianMinimalStemFilterFactory.java | 1 -
.../gl/TestGalicianStemFilterFactory.java | 1 -
.../lucene/analysis/hi/TestHindiAnalyzer.java | 3 +-
.../lucene/analysis/hi/TestHindiFilters.java | 2 -
.../lucene/analysis/hi/TestHindiNormalizer.java | 4 -
.../analysis/hu/TestHungarianAnalyzer.java | 2 +-
.../hu/TestHungarianLightStemFilter.java | 2 +-
.../hu/TestHungarianLightStemFilterFactory.java | 1 -
.../analysis/hunspell/Test64kAffixes.java | 2 -
.../analysis/hunspell/TestAllDictionaries.java | 1 -
.../analysis/hunspell/TestAllDictionaries2.java | 1 -
.../analysis/hunspell/TestDictionary.java | 1 -
.../hunspell/TestHunspellStemFilter.java | 3 +-
.../analysis/hy/TestArmenianAnalyzer.java | 2 +-
.../analysis/id/TestIndonesianAnalyzer.java | 2 +-
.../lucene/analysis/it/TestItalianAnalyzer.java | 2 +-
.../it/TestItalianLightStemFilterFactory.java | 1 -
.../analysis/lt/TestLithuanianAnalyzer.java | 2 +-
.../lucene/analysis/lv/TestLatvianAnalyzer.java | 2 +-
.../analysis/minhash/MinHashFilterTest.java | 330 ++
.../miscellaneous/TestASCIIFoldingFilter.java | 13 +
.../miscellaneous/TestCapitalizationFilter.java | 2 +-
.../miscellaneous/TestKeepFilterFactory.java | 4 +-
.../miscellaneous/TestKeepWordFilter.java | 2 +-
.../miscellaneous/TestKeywordMarkerFilter.java | 6 +-
.../TestStemmerOverrideFilter.java | 6 +-
.../analysis/miscellaneous/TestTrimFilter.java | 46 +-
.../miscellaneous/TestTrimFilterFactory.java | 1 -
.../miscellaneous/TestWordDelimiterFilter.java | 12 +-
.../lucene/analysis/nl/TestDutchAnalyzer.java | 6 +-
.../analysis/no/TestNorwegianAnalyzer.java | 2 +-
.../no/TestNorwegianLightStemFilter.java | 2 +-
.../no/TestNorwegianLightStemFilterFactory.java | 1 -
.../no/TestNorwegianMinimalStemFilter.java | 2 +-
.../TestPatternReplaceCharFilterFactory.java | 1 -
.../TestPatternReplaceFilterFactory.java | 1 -
.../DelimitedPayloadTokenFilterTest.java | 4 -
.../analysis/pt/TestPortugueseAnalyzer.java | 2 +-
.../pt/TestPortugueseLightStemFilter.java | 2 +-
.../pt/TestPortugueseMinimalStemFilter.java | 2 +-
.../analysis/pt/TestPortugueseStemFilter.java | 6 +-
.../pt/TestPortugueseStemFilterFactory.java | 1 -
.../reverse/TestReverseStringFilterFactory.java | 1 -
.../analysis/ro/TestRomanianAnalyzer.java | 2 +-
.../lucene/analysis/ru/TestRussianAnalyzer.java | 4 +-
.../analysis/ru/TestRussianLightStemFilter.java | 2 +-
.../ru/TestRussianLightStemFilterFactory.java | 1 -
.../shingle/ShingleAnalyzerWrapperTest.java | 4 +-
.../analysis/shingle/ShingleFilterTest.java | 1 -
.../shingle/TestShingleFilterFactory.java | 1 -
.../analysis/sinks/TestTeeSinkTokenFilter.java | 4 +-
.../analysis/standard/TestStandardAnalyzer.java | 390 --
.../standard/TestStandardFactories.java | 1 -
.../standard/WordBreakTestUnicode_6_3_0.java | 5537 ------------------
.../generateJavaUnicodeWordBreakTest.pl | 232 -
.../lucene/analysis/sv/TestSwedishAnalyzer.java | 2 +-
.../analysis/sv/TestSwedishLightStemFilter.java | 2 +-
.../lucene/analysis/th/TestThaiAnalyzer.java | 3 +-
.../lucene/analysis/tr/TestTurkishAnalyzer.java | 2 +-
.../lucene/analysis/util/TestCharArrayMap.java | 244 -
.../lucene/analysis/util/TestCharArraySet.java | 429 --
.../analysis/util/TestCharacterUtils.java | 222 -
.../lucene/analysis/util/TestElision.java | 2 +-
.../util/TestFilesystemResourceLoader.java | 2 +
.../analysis/util/TestWordlistLoader.java | 79 -
.../collation/TestCollationDocValuesField.java | 3 +-
.../tools/groovy/generate-unicode-data.groovy | 4 +-
.../icu/src/data/uax29/MyanmarSyllable.rbbi | 50 +
.../segmentation/DefaultICUTokenizerConfig.java | 14 +-
.../analysis/icu/segmentation/ICUTokenizer.java | 2 +-
.../icu/segmentation/ICUTokenizerFactory.java | 6 +-
.../collation/ICUCollationKeyAnalyzer.java | 3 -
lucene/analysis/icu/src/java/overview.html | 2 +-
.../icu/segmentation/MyanmarSyllable.brk | Bin 0 -> 19776 bytes
.../icu/TestICUFoldingFilterFactory.java | 1 -
.../icu/TestICUNormalizer2FilterFactory.java | 1 -
.../icu/TestICUTransformFilterFactory.java | 1 -
.../icu/segmentation/TestICUTokenizer.java | 6 +-
.../icu/segmentation/TestICUTokenizerCJK.java | 2 +-
.../segmentation/TestICUTokenizerFactory.java | 1 -
.../icu/segmentation/TestMyanmarSyllable.java | 156 +
.../segmentation/TestWithCJKBigramFilter.java | 8 +-
.../TestICUCollationDocValuesField.java | 3 +-
.../lucene/analysis/ja/JapaneseAnalyzer.java | 15 +-
.../analysis/ja/JapaneseNumberFilter.java | 15 +-
.../ja/JapanesePartOfSpeechStopFilter.java | 4 +-
.../JapanesePartOfSpeechStopFilterFactory.java | 2 +-
.../lucene/analysis/ja/JapaneseTokenizer.java | 11 +-
.../analysis/ja/dict/ConnectionCosts.java | 1 -
.../lucene/analysis/ja/dict/UserDictionary.java | 1 -
.../apache/lucene/analysis/ja/util/CSVUtil.java | 2 +-
.../apache/lucene/analysis/ja/TestCSVUtil.java | 52 +
.../lucene/analysis/ja/TestFactories.java | 203 +
.../analysis/ja/TestJapaneseBaseFormFilter.java | 2 +-
.../ja/TestJapaneseKatakanaStemFilter.java | 6 +-
.../analysis/ja/TestJapaneseNumberFilter.java | 2 +-
.../analysis/ja/TestJapaneseTokenizer.java | 9 +
.../analysis/morfologik/MorfologikAnalyzer.java | 6 +
.../analysis/morfologik/MorfologikFilter.java | 16 +-
.../uk/UkrainianMorfologikAnalyzer.java | 153 +
.../apache/lucene/analysis/uk/package-info.java | 21 +
.../apache/lucene/analysis/uk/mapping_uk.txt | 19 +
.../org/apache/lucene/analysis/uk/stopwords.txt | 1269 ++++
.../apache/lucene/analysis/uk/ukrainian.dict | Bin 0 -> 1989243 bytes
.../apache/lucene/analysis/uk/ukrainian.info | 10 +
.../morfologik/TestMorfologikAnalyzer.java | 2 +-
.../analysis/uk/TestUkrainianAnalyzer.java | 72 +
.../TestDoubleMetaphoneFilterFactory.java | 3 -
.../analysis/cn/smart/SmartChineseAnalyzer.java | 12 +-
.../analysis/cn/smart/hhmm/WordDictionary.java | 18 +-
.../lucene/analysis/pl/PolishAnalyzer.java | 21 +-
.../src/java/org/egothor/stemmer/Compile.java | 43 +-
.../src/java/org/egothor/stemmer/DiffIt.java | 28 +-
.../lucene/analysis/pl/TestPolishAnalyzer.java | 2 +-
.../TestStempelPolishStemFilterFactory.java | 1 -
.../uima/UIMAAnnotationsTokenizerFactory.java | 1 -
.../lucene/analysis/uima/UIMABaseAnalyzer.java | 1 -
...IMATypeAwareAnnotationsTokenizerFactory.java | 1 -
.../uima/UIMATypeAwareAnalyzerTest.java | 1 -
.../org/apache/lucene/codecs/Placeholder.java | 1 -
.../lucene/codecs/lucene50/Lucene50Codec.java | 170 -
.../lucene50/Lucene50DocValuesConsumer.java | 658 ---
.../lucene50/Lucene50DocValuesFormat.java | 115 -
.../lucene50/Lucene50DocValuesProducer.java | 1299 ----
.../codecs/lucene50/Lucene50NormsFormat.java | 62 -
.../codecs/lucene50/Lucene50NormsProducer.java | 481 --
.../lucene50/Lucene50SegmentInfoFormat.java | 90 +
.../lucene/codecs/lucene53/Lucene53Codec.java | 176 -
.../codecs/lucene53/Lucene53NormsFormat.java | 91 +
.../codecs/lucene53/Lucene53NormsProducer.java | 230 +
.../lucene/codecs/lucene53/package-info.java | 23 +
.../apache/lucene/codecs/lucene53/package.html | 25 -
.../lucene/codecs/lucene54/Lucene54Codec.java | 176 -
.../lucene54/Lucene54DocValuesConsumer.java | 797 +++
.../lucene54/Lucene54DocValuesFormat.java | 186 +
.../lucene54/Lucene54DocValuesProducer.java | 1740 ++++++
.../lucene/codecs/lucene54/package-info.java | 403 ++
.../apache/lucene/codecs/lucene54/package.html | 25 -
.../lucene/codecs/lucene60/Lucene60Codec.java | 177 +
.../apache/lucene/codecs/lucene60/package.html | 25 +
.../lucene/codecs/lucene62/Lucene62Codec.java | 176 +
.../apache/lucene/codecs/lucene62/package.html | 25 +
.../apache/lucene/legacy/LegacyDoubleField.java | 174 +
.../org/apache/lucene/legacy/LegacyField.java | 90 +
.../apache/lucene/legacy/LegacyFieldType.java | 149 +
.../apache/lucene/legacy/LegacyFloatField.java | 174 +
.../apache/lucene/legacy/LegacyIntField.java | 175 +
.../apache/lucene/legacy/LegacyLongField.java | 184 +
.../lucene/legacy/LegacyNumericRangeQuery.java | 537 ++
.../lucene/legacy/LegacyNumericTokenStream.java | 357 ++
.../apache/lucene/legacy/LegacyNumericType.java | 34 +
.../lucene/legacy/LegacyNumericUtils.java | 510 ++
.../org/apache/lucene/legacy/package-info.java | 21 +
.../services/org.apache.lucene.codecs.Codec | 5 +-
.../org.apache.lucene.codecs.DocValuesFormat | 2 +-
.../codecs/lucene50/Lucene50NormsConsumer.java | 403 --
.../lucene/codecs/lucene50/Lucene50RWCodec.java | 34 -
.../codecs/lucene50/Lucene50RWNormsFormat.java | 36 -
.../lucene50/Lucene50RWSegmentInfoFormat.java | 125 +
.../lucene50/TestLucene50DocValuesFormat.java | 281 -
.../lucene50/TestLucene50NormsFormat.java | 130 -
.../lucene50/TestLucene50SegmentInfoFormat.java | 40 +
.../codecs/lucene53/Lucene53NormsConsumer.java | 159 +
.../codecs/lucene53/Lucene53RWNormsFormat.java | 31 +
.../lucene53/TestLucene53NormsFormat.java | 38 +
.../lucene54/TestLucene54DocValuesFormat.java | 612 ++
.../lucene/codecs/lucene60/Lucene60RWCodec.java | 38 +
.../lucene/codecs/lucene62/Lucene62RWCodec.java | 32 +
.../index/TestBackwardsCompatibility.java | 121 +-
.../org/apache/lucene/index/index.6.0.0-cfs.zip | Bin 13744 -> 15807 bytes
.../apache/lucene/index/index.6.0.0-nocfs.zip | Bin 13749 -> 15806 bytes
.../org/apache/lucene/index/index.6.0.1-cfs.zip | Bin 0 -> 15820 bytes
.../apache/lucene/index/index.6.0.1-nocfs.zip | Bin 0 -> 15823 bytes
.../org/apache/lucene/index/index.6.1.0-cfs.zip | Bin 0 -> 15803 bytes
.../apache/lucene/index/index.6.1.0-nocfs.zip | Bin 0 -> 15829 bytes
.../org/apache/lucene/index/index.6.2.0-cfs.zip | Bin 0 -> 15880 bytes
.../apache/lucene/index/index.6.2.0-nocfs.zip | Bin 0 -> 15867 bytes
.../org/apache/lucene/index/index.6.2.1-cfs.zip | Bin 0 -> 15851 bytes
.../apache/lucene/index/index.6.2.1-nocfs.zip | Bin 0 -> 15845 bytes
.../lucene/index/unsupported.5.5.1-cfs.zip | Bin 0 -> 15553 bytes
.../lucene/index/unsupported.5.5.1-nocfs.zip | Bin 0 -> 13729 bytes
.../lucene/index/unsupported.5.5.2-cfs.zip | Bin 0 -> 13712 bytes
.../lucene/index/unsupported.5.5.2-nocfs.zip | Bin 0 -> 13720 bytes
.../lucene/index/unsupported.5.5.3-cfs.zip | Bin 0 -> 13724 bytes
.../lucene/index/unsupported.5.5.3-nocfs.zip | Bin 0 -> 13725 bytes
.../apache/lucene/legacy/TestLegacyField.java | 196 +
.../lucene/legacy/TestLegacyFieldReuse.java | 81 +
.../lucene/legacy/TestLegacyNumericUtils.java | 571 ++
.../apache/lucene/legacy/TestLegacyTerms.java | 164 +
.../TestMultiValuedNumericRangeQuery.java | 84 +
.../lucene/legacy/TestNumericRangeQuery32.java | 461 ++
.../lucene/legacy/TestNumericRangeQuery64.java | 490 ++
.../lucene/legacy/TestNumericTokenStream.java | 188 +
lucene/benchmark/.gitignore | 4 +-
lucene/benchmark/README.enwiki | 11 +-
lucene/benchmark/conf/highlight-profile.alg | 68 -
.../conf/highlight-vs-vector-highlight.alg | 80 -
lucene/benchmark/conf/highlighters-postings.alg | 65 +
lucene/benchmark/conf/highlighters-tv.alg | 64 +
lucene/benchmark/conf/highlights.alg | 69 +
lucene/benchmark/conf/query-phrases.txt | 10 +
lucene/benchmark/conf/query-terms.txt | 10 +
lucene/benchmark/conf/query-wildcards.txt | 7 +
.../benchmark/conf/standard-highlights-notv.alg | 69 -
.../benchmark/conf/standard-highlights-tv.alg | 69 -
.../benchmark/conf/vector-highlight-profile.alg | 68 -
.../lucene/benchmark/byTask/PerfRunData.java | 2 +
.../lucene/benchmark/byTask/feeds/DocMaker.java | 7 +
.../benchmark/byTask/tasks/AddIndexesTask.java | 3 -
.../byTask/tasks/BenchmarkHighlighter.java | 30 -
.../byTask/tasks/CloseTaxonomyReaderTask.java | 10 +-
.../benchmark/byTask/tasks/CommitIndexTask.java | 2 +-
.../benchmark/byTask/tasks/CreateIndexTask.java | 4 +-
.../byTask/tasks/NearRealtimeReaderTask.java | 1 -
.../benchmark/byTask/tasks/OpenReaderTask.java | 1 -
.../lucene/benchmark/byTask/tasks/ReadTask.java | 121 +-
.../byTask/tasks/ReopenReaderTask.java | 1 -
.../byTask/tasks/RollbackIndexTask.java | 2 -
.../tasks/SearchTravRetHighlightTask.java | 283 +-
.../tasks/SearchTravRetVectorHighlightTask.java | 147 -
.../byTask/tasks/WriteEnwikiLineDocTask.java | 1 -
.../benchmark/quality/utils/SimpleQQParser.java | 1 -
.../benchmark/byTask/TestPerfTasksLogic.java | 106 -
.../tasks/CountingHighlighterTestTask.java | 68 -
.../tasks/WriteEnwikiLineDocTaskTest.java | 4 -
.../benchmark/quality/TestQualityRun.java | 3 -
.../lucene/benchmark/quality/trecQRels.txt | 428 +-
lucene/build.xml | 17 +-
lucene/classification/build.xml | 6 +-
.../BooleanPerceptronClassifier.java | 20 +-
.../CachingNaiveBayesClassifier.java | 18 +-
.../lucene/classification/Classifier.java | 4 -
.../KNearestNeighborClassifier.java | 9 +-
.../SimpleNaiveBayesClassifier.java | 50 +-
.../KNearestNeighborDocumentClassifier.java | 10 +-
.../SimpleNaiveBayesDocumentClassifier.java | 40 +-
.../utils/ConfusionMatrixGenerator.java | 57 +-
.../classification/utils/DatasetSplitter.java | 50 +-
.../SimpleNaiveBayesClassifierTest.java | 1 -
.../DocumentClassificationTestBase.java | 11 +-
.../KNearestNeighborDocumentClassifierTest.java | 40 +-
.../SimpleNaiveBayesDocumentClassifierTest.java | 32 +-
.../classification/utils/DataSplitterTest.java | 6 +-
.../utils/DocToDoubleVectorUtilsTest.java | 2 -
.../autoprefix/AutoPrefixPostingsFormat.java | 125 -
.../lucene/codecs/autoprefix/package-info.java | 22 -
.../codecs/blockterms/BlockTermsReader.java | 1 -
.../blocktreeords/OrdsIntersectTermsEnum.java | 4 +-
.../blocktreeords/OrdsSegmentTermsEnum.java | 2 -
.../codecs/memory/DirectDocValuesConsumer.java | 33 +-
.../codecs/memory/DirectDocValuesProducer.java | 100 +-
.../codecs/memory/DirectPostingsFormat.java | 2 +-
.../lucene/codecs/memory/FSTOrdTermsReader.java | 3 +-
.../lucene/codecs/memory/FSTTermsReader.java | 2 +-
.../codecs/memory/MemoryDocValuesConsumer.java | 45 +-
.../codecs/memory/MemoryDocValuesProducer.java | 165 +-
.../codecs/simpletext/SimpleTextBKDReader.java | 1 -
.../simpletext/SimpleTextCompoundFormat.java | 5 +-
.../simpletext/SimpleTextDocValuesFormat.java | 2 +-
.../simpletext/SimpleTextDocValuesReader.java | 408 +-
.../simpletext/SimpleTextDocValuesWriter.java | 181 +-
.../simpletext/SimpleTextFieldsReader.java | 1 -
.../simpletext/SimpleTextFieldsWriter.java | 2 +
.../simpletext/SimpleTextLiveDocsFormat.java | 2 -
.../simpletext/SimpleTextNormsFormat.java | 10 +-
.../simpletext/SimpleTextPointsReader.java | 98 +-
.../simpletext/SimpleTextPointsWriter.java | 26 +-
.../simpletext/SimpleTextPostingsFormat.java | 2 +-
.../simpletext/SimpleTextSegmentInfoFormat.java | 174 +-
.../SimpleTextStoredFieldsWriter.java | 1 -
.../simpletext/SimpleTextTermVectorsReader.java | 1 -
.../org.apache.lucene.codecs.DocValuesFormat | 1 -
.../org.apache.lucene.codecs.PostingsFormat | 2 -
.../TestAutoPrefixPostingsFormat.java | 38 -
.../codecs/autoprefix/TestAutoPrefixTerms.java | 941 ---
.../TestSimpleTextCompoundFormat.java | 10 +
lucene/common-build.xml | 136 +-
lucene/core/build.xml | 18 +-
.../org/apache/lucene/analysis/Analyzer.java | 135 +-
.../apache/lucene/analysis/CharArrayMap.java | 669 +++
.../apache/lucene/analysis/CharArraySet.java | 196 +
.../apache/lucene/analysis/CharacterUtils.java | 251 +
.../lucene/analysis/FilteringTokenFilter.java | 76 +
.../analysis/LegacyNumericTokenStream.java | 357 --
.../apache/lucene/analysis/LowerCaseFilter.java | 50 +
.../org/apache/lucene/analysis/StopFilter.java | 111 +
.../lucene/analysis/StopwordAnalyzerBase.java | 138 +
.../java/org/apache/lucene/analysis/Token.java | 1 -
.../org/apache/lucene/analysis/TokenStream.java | 7 +-
.../apache/lucene/analysis/WordlistLoader.java | 244 +
.../apache/lucene/analysis/package-info.java | 2 +-
.../analysis/standard/StandardAnalyzer.java | 122 +
.../analysis/standard/StandardFilter.java | 39 +
.../analysis/standard/StandardTokenizer.java | 214 +
.../standard/StandardTokenizerImpl.java | 823 +++
.../standard/StandardTokenizerImpl.jflex | 207 +
.../lucene/analysis/standard/package-info.java | 33 +
.../PackedTokenAttributeImpl.java | 11 +
.../PositionIncrementAttributeImpl.java | 5 +
.../apache/lucene/codecs/BlockTermState.java | 14 +-
.../java/org/apache/lucene/codecs/Codec.java | 2 +-
.../org/apache/lucene/codecs/CodecUtil.java | 55 +-
.../apache/lucene/codecs/CompoundFormat.java | 4 +-
.../apache/lucene/codecs/DocValuesConsumer.java | 1180 ++--
.../apache/lucene/codecs/DocValuesProducer.java | 7 -
.../lucene/codecs/LegacyDocValuesIterables.java | 534 ++
.../lucene/codecs/MultiLevelSkipListReader.java | 9 +-
.../lucene/codecs/MutablePointValues.java | 42 +
.../org/apache/lucene/codecs/NormsConsumer.java | 160 +-
.../org/apache/lucene/codecs/PointsFormat.java | 35 +-
.../org/apache/lucene/codecs/PointsReader.java | 5 +-
.../org/apache/lucene/codecs/PointsWriter.java | 205 +-
.../lucene/codecs/StoredFieldsWriter.java | 61 +-
.../apache/lucene/codecs/TermVectorsWriter.java | 78 +-
.../codecs/blocktree/AutoPrefixTermsWriter.java | 438 --
.../codecs/blocktree/BlockTreeTermsReader.java | 11 +-
.../codecs/blocktree/BlockTreeTermsWriter.java | 222 +-
.../lucene/codecs/blocktree/FieldReader.java | 2 +-
.../codecs/blocktree/IntersectTermsEnum.java | 4 +-
.../CompressingStoredFieldsWriter.java | 6 +
.../CompressingTermVectorsWriter.java | 5 +
.../codecs/lucene50/Lucene50CompoundFormat.java | 26 +-
.../codecs/lucene50/Lucene50CompoundReader.java | 12 +-
.../lucene50/Lucene50FieldInfosFormat.java | 12 +-
.../codecs/lucene50/Lucene50PostingsFormat.java | 1 -
.../lucene50/Lucene50SegmentInfoFormat.java | 166 -
.../lucene/codecs/lucene50/package-info.java | 2 +-
.../codecs/lucene53/Lucene53NormsConsumer.java | 153 -
.../codecs/lucene53/Lucene53NormsFormat.java | 91 -
.../codecs/lucene53/Lucene53NormsProducer.java | 208 -
.../lucene/codecs/lucene53/package-info.java | 23 -
.../lucene54/Lucene54DocValuesConsumer.java | 776 ---
.../lucene54/Lucene54DocValuesFormat.java | 183 -
.../lucene54/Lucene54DocValuesProducer.java | 1488 -----
.../lucene/codecs/lucene54/package-info.java | 403 --
.../lucene/codecs/lucene60/Lucene60Codec.java | 176 -
.../codecs/lucene60/Lucene60PointsReader.java | 84 +-
.../codecs/lucene60/Lucene60PointsWriter.java | 42 +-
.../lucene/codecs/lucene60/package-info.java | 397 +-
.../lucene62/Lucene62SegmentInfoFormat.java | 318 +
.../lucene/codecs/lucene62/package-info.java | 24 +
.../lucene/codecs/lucene70/IndexedDISI.java | 269 +
.../lucene/codecs/lucene70/Lucene70Codec.java | 176 +
.../lucene70/Lucene70DocValuesConsumer.java | 522 ++
.../lucene70/Lucene70DocValuesFormat.java | 160 +
.../lucene70/Lucene70DocValuesProducer.java | 1140 ++++
.../codecs/lucene70/Lucene70NormsConsumer.java | 159 +
.../codecs/lucene70/Lucene70NormsFormat.java | 99 +
.../codecs/lucene70/Lucene70NormsProducer.java | 267 +
.../lucene/codecs/lucene70/package-info.java | 392 ++
.../perfield/PerFieldDocValuesFormat.java | 56 +-
.../codecs/perfield/PerFieldMergeState.java | 274 +
.../codecs/perfield/PerFieldPostingsFormat.java | 85 +-
.../lucene/document/BinaryDocValuesField.java | 1 -
.../org/apache/lucene/document/BinaryPoint.java | 5 +-
.../lucene/document/CompressionTools.java | 150 -
.../java/org/apache/lucene/document/Field.java | 57 +-
.../org/apache/lucene/document/FieldType.java | 100 +-
.../lucene/document/LegacyDoubleField.java | 172 -
.../lucene/document/LegacyFloatField.java | 174 -
.../apache/lucene/document/LegacyIntField.java | 174 -
.../apache/lucene/document/LegacyLongField.java | 182 -
.../lucene/document/SortedDocValuesField.java | 4 +-
.../document/SortedSetDocValuesField.java | 2 +
.../java/org/apache/lucene/geo/GeoUtils.java | 43 +-
.../src/java/org/apache/lucene/geo/Polygon.java | 13 +-
.../java/org/apache/lucene/geo/Polygon2D.java | 18 +-
.../java/org/apache/lucene/geo/Rectangle.java | 33 +-
.../lucene/geo/SimpleGeoJSONPolygonParser.java | 440 ++
.../apache/lucene/index/AutomatonTermsEnum.java | 6 +-
.../apache/lucene/index/BinaryDocValues.java | 19 +-
.../lucene/index/BinaryDocValuesWriter.java | 102 +-
.../apache/lucene/index/BufferedUpdates.java | 7 +-
.../lucene/index/BufferedUpdatesStream.java | 30 +-
.../org/apache/lucene/index/CheckIndex.java | 296 +-
.../apache/lucene/index/CoalescedUpdates.java | 14 +-
.../org/apache/lucene/index/CodecReader.java | 161 +-
.../lucene/index/DefaultIndexingChain.java | 19 +-
.../org/apache/lucene/index/DocIDMerger.java | 180 +
.../java/org/apache/lucene/index/DocValues.java | 329 +-
.../apache/lucene/index/DocValuesUpdate.java | 2 +-
.../apache/lucene/index/DocumentsWriter.java | 112 +-
.../index/DocumentsWriterDeleteQueue.java | 142 +-
.../index/DocumentsWriterFlushControl.java | 51 +-
.../lucene/index/DocumentsWriterPerThread.java | 46 +-
.../index/DocumentsWriterPerThreadPool.java | 21 +-
.../index/DocumentsWriterStallControl.java | 26 +-
.../lucene/index/EmptyDocValuesProducer.java | 71 +
.../java/org/apache/lucene/index/FieldInfo.java | 4 +
.../lucene/index/FilterBinaryDocValues.java | 61 +
.../apache/lucene/index/FilterCodecReader.java | 11 +-
.../apache/lucene/index/FilterLeafReader.java | 9 +-
.../lucene/index/FilterNumericDocValues.java | 59 +
.../org/apache/lucene/index/IndexCommit.java | 2 +-
.../org/apache/lucene/index/IndexUpgrader.java | 2 +-
.../org/apache/lucene/index/IndexWriter.java | 466 +-
.../apache/lucene/index/IndexWriterConfig.java | 43 +-
.../org/apache/lucene/index/LeafReader.java | 24 +-
.../lucene/index/LegacyBinaryDocValues.java | 39 +
.../index/LegacyBinaryDocValuesWrapper.java | 82 +
.../lucene/index/LegacyNumericDocValues.java | 38 +
.../index/LegacyNumericDocValuesWrapper.java | 87 +
.../lucene/index/LegacySortedDocValues.java | 110 +
.../index/LegacySortedDocValuesWrapper.java | 92 +
.../index/LegacySortedNumericDocValues.java | 49 +
.../LegacySortedNumericDocValuesWrapper.java | 88 +
.../lucene/index/LegacySortedSetDocValues.java | 111 +
.../index/LegacySortedSetDocValuesWrapper.java | 102 +
.../lucene/index/LiveIndexWriterConfig.java | 13 +
.../org/apache/lucene/index/LogMergePolicy.java | 6 +-
.../lucene/index/MappingMultiPostingsEnum.java | 124 +-
.../org/apache/lucene/index/MergePolicy.java | 63 +-
.../apache/lucene/index/MergeReaderWrapper.java | 251 +
.../org/apache/lucene/index/MergeState.java | 234 +-
.../org/apache/lucene/index/MultiDocValues.java | 646 +-
.../apache/lucene/index/MultiPostingsEnum.java | 8 +-
.../org/apache/lucene/index/MultiSorter.java | 404 ++
.../apache/lucene/index/NormValuesWriter.java | 98 +-
.../apache/lucene/index/NumericDocValues.java | 11 +-
.../index/NumericDocValuesFieldUpdates.java | 3 +-
.../lucene/index/NumericDocValuesWriter.java | 93 +-
.../apache/lucene/index/ParallelLeafReader.java | 125 +-
.../org/apache/lucene/index/PointValues.java | 64 +-
.../apache/lucene/index/PointValuesWriter.java | 170 +-
.../apache/lucene/index/RandomAccessOrds.java | 53 -
.../apache/lucene/index/ReadersAndUpdates.java | 216 +-
.../lucene/index/SegmentDocValuesProducer.java | 8 -
.../org/apache/lucene/index/SegmentInfo.java | 21 +-
.../org/apache/lucene/index/SegmentInfos.java | 59 +-
.../org/apache/lucene/index/SegmentMerger.java | 5 +
.../org/apache/lucene/index/SegmentReader.java | 45 +-
.../index/SingletonSortedNumericDocValues.java | 55 +-
.../index/SingletonSortedSetDocValues.java | 51 +-
.../lucene/index/SlowCodecReaderWrapper.java | 64 +-
.../apache/lucene/index/SortedDocValues.java | 41 +-
.../lucene/index/SortedDocValuesWriter.java | 129 +-
.../lucene/index/SortedNumericDocValues.java | 25 +-
.../index/SortedNumericDocValuesWriter.java | 123 +-
.../apache/lucene/index/SortedSetDocValues.java | 28 +-
.../lucene/index/SortedSetDocValuesWriter.java | 207 +-
.../java/org/apache/lucene/index/Sorter.java | 454 ++
.../apache/lucene/index/SortingLeafReader.java | 1213 ++++
.../lucene/index/StandardDirectoryReader.java | 56 +-
.../org/apache/lucene/index/TermContext.java | 12 -
.../java/org/apache/lucene/index/TermState.java | 6 -
.../index/TermVectorsConsumerPerField.java | 1 -
.../apache/lucene/index/TermsHashPerField.java | 1 -
.../lucene/index/TrackingIndexWriter.java | 168 -
.../org/apache/lucene/index/TwoPhaseCommit.java | 5 +-
.../apache/lucene/search/BlendedTermQuery.java | 28 +-
.../org/apache/lucene/search/BooleanQuery.java | 101 +-
.../org/apache/lucene/search/BooleanScorer.java | 32 +-
.../lucene/search/BooleanTopLevelScorers.java | 182 -
.../org/apache/lucene/search/BooleanWeight.java | 180 +-
.../org/apache/lucene/search/BoostQuery.java | 61 +-
.../apache/lucene/search/ConjunctionDISI.java | 288 +-
.../apache/lucene/search/ConjunctionScorer.java | 14 +-
.../lucene/search/ConstantScoreQuery.java | 21 +-
.../lucene/search/ConstantScoreWeight.java | 35 +-
.../search/ControlledRealTimeReopenThread.java | 17 +-
.../lucene/search/DisjunctionMaxQuery.java | 49 +-
.../lucene/search/DisjunctionMaxScorer.java | 2 +-
.../lucene/search/DisjunctionSumScorer.java | 11 +-
.../apache/lucene/search/DocValuesDocIdSet.java | 157 -
.../lucene/search/DocValuesRewriteMethod.java | 48 +-
.../EarlyTerminatingSortingCollector.java | 129 +
.../apache/lucene/search/ExactPhraseScorer.java | 5 +-
.../org/apache/lucene/search/Explanation.java | 33 +-
.../apache/lucene/search/FieldComparator.java | 290 +-
.../apache/lucene/search/FieldValueQuery.java | 58 +-
.../lucene/search/FilterLeafCollector.java | 7 +-
.../org/apache/lucene/search/FilterWeight.java | 73 +
.../org/apache/lucene/search/FuzzyQuery.java | 2 +-
.../apache/lucene/search/FuzzyTermsEnum.java | 368 +-
.../org/apache/lucene/search/IndexSearcher.java | 28 +-
.../org/apache/lucene/search/LRUQueryCache.java | 59 +-
.../lucene/search/LeafFieldComparator.java | 9 +-
.../lucene/search/LegacyNumericRangeQuery.java | 536 --
.../apache/lucene/search/MatchAllDocsQuery.java | 14 +-
.../apache/lucene/search/MatchNoDocsQuery.java | 59 +-
.../lucene/search/MinShouldMatchSumScorer.java | 6 +-
.../apache/lucene/search/MultiPhraseQuery.java | 40 +-
.../apache/lucene/search/MultiTermQuery.java | 35 +-
.../MultiTermQueryConstantScoreWrapper.java | 22 +-
.../java/org/apache/lucene/search/Multiset.java | 2 -
.../apache/lucene/search/NGramPhraseQuery.java | 16 +-
.../org/apache/lucene/search/PhraseQuery.java | 39 +-
.../apache/lucene/search/PointInSetQuery.java | 105 +-
.../apache/lucene/search/PointRangeQuery.java | 90 +-
.../org/apache/lucene/search/PrefixQuery.java | 5 +-
.../java/org/apache/lucene/search/Query.java | 52 +-
.../lucene/search/RandomAccessWeight.java | 4 +-
.../org/apache/lucene/search/RegexpQuery.java | 1 -
.../apache/lucene/search/ScoringRewrite.java | 6 +-
.../lucene/search/SloppyPhraseScorer.java | 5 +-
.../src/java/org/apache/lucene/search/Sort.java | 3 +
.../org/apache/lucene/search/SortField.java | 3 -
.../lucene/search/SortedNumericSelector.java | 101 +-
.../lucene/search/SortedNumericSortField.java | 2 +-
.../apache/lucene/search/SortedSetSelector.java | 265 +-
.../lucene/search/SortedSetSortField.java | 5 +-
.../org/apache/lucene/search/SynonymQuery.java | 35 +-
.../org/apache/lucene/search/TermQuery.java | 95 +-
.../apache/lucene/search/TopFieldCollector.java | 2 +-
.../apache/lucene/search/TopTermsRewrite.java | 4 +-
.../apache/lucene/search/TwoPhaseIterator.java | 79 +-
.../java/org/apache/lucene/search/Weight.java | 15 +-
.../org/apache/lucene/search/package-info.java | 31 +-
.../search/similarities/BM25Similarity.java | 56 +-
.../lucene/search/similarities/BasicStats.java | 35 +-
.../search/similarities/ClassicSimilarity.java | 12 -
.../search/similarities/LMSimilarity.java | 8 +-
.../search/similarities/MultiSimilarity.java | 24 +-
.../similarities/PerFieldSimilarityWrapper.java | 14 +-
.../lucene/search/similarities/Similarity.java | 68 +-
.../search/similarities/SimilarityBase.java | 34 +-
.../search/similarities/TFIDFSimilarity.java | 203 +-
.../search/spans/FieldMaskingSpanQuery.java | 28 +-
.../search/spans/ScoringWrapperSpans.java | 95 -
.../lucene/search/spans/SpanBoostQuery.java | 72 +-
.../lucene/search/spans/SpanContainQuery.java | 20 +-
.../search/spans/SpanContainingQuery.java | 12 +-
.../search/spans/SpanMultiTermQueryWrapper.java | 16 +-
.../lucene/search/spans/SpanNearQuery.java | 60 +-
.../lucene/search/spans/SpanNotQuery.java | 35 +-
.../apache/lucene/search/spans/SpanOrQuery.java | 25 +-
.../search/spans/SpanPositionCheckQuery.java | 23 +-
.../apache/lucene/search/spans/SpanQuery.java | 2 +-
.../apache/lucene/search/spans/SpanScorer.java | 10 +-
.../lucene/search/spans/SpanTermQuery.java | 22 +-
.../apache/lucene/search/spans/SpanWeight.java | 20 +-
.../lucene/search/spans/SpanWithinQuery.java | 12 +-
.../lucene/store/ByteArrayIndexInput.java | 6 +-
.../apache/lucene/store/ByteBufferGuard.java | 136 +
.../lucene/store/ByteBufferIndexInput.java | 104 +-
.../java/org/apache/lucene/store/DataInput.java | 33 -
.../org/apache/lucene/store/DataOutput.java | 45 -
.../java/org/apache/lucene/store/Directory.java | 8 +-
.../org/apache/lucene/store/FSDirectory.java | 15 +-
.../lucene/store/FileSwitchDirectory.java | 10 +-
.../apache/lucene/store/FilterDirectory.java | 9 +-
.../org/apache/lucene/store/IndexInput.java | 5 +
.../store/LockValidatingDirectoryWrapper.java | 10 +-
.../org/apache/lucene/store/MMapDirectory.java | 8 +-
.../lucene/store/NRTCachingDirectory.java | 4 +-
.../org/apache/lucene/store/RAMDirectory.java | 22 +-
.../lucene/store/TrackingDirectoryWrapper.java | 4 +-
.../java/org/apache/lucene/util/ArrayUtil.java | 296 +-
.../org/apache/lucene/util/AttributeImpl.java | 11 +
.../org/apache/lucene/util/AttributeSource.java | 10 +
.../org/apache/lucene/util/BitSetIterator.java | 10 +
.../src/java/org/apache/lucene/util/Bits.java | 4 +-
.../org/apache/lucene/util/ByteBlockPool.java | 31 +-
.../apache/lucene/util/BytesRefComparator.java | 55 +
.../org/apache/lucene/util/BytesRefHash.java | 37 +-
.../org/apache/lucene/util/CharsRefBuilder.java | 1 -
.../org/apache/lucene/util/DocIdSetBuilder.java | 249 +-
.../lucene/util/FixedLengthBytesRefArray.java | 29 +
.../apache/lucene/util/InPlaceMergeSorter.java | 4 +-
.../org/apache/lucene/util/IntroSelector.java | 128 +
.../org/apache/lucene/util/IntroSorter.java | 20 +-
.../apache/lucene/util/LegacyNumericUtils.java | 508 --
.../java/org/apache/lucene/util/LongValues.java | 14 +-
.../org/apache/lucene/util/MSBRadixSorter.java | 282 +
.../org/apache/lucene/util/QueryBuilder.java | 1 -
.../org/apache/lucene/util/RadixSelector.java | 278 +
.../java/org/apache/lucene/util/Selector.java | 41 +
.../java/org/apache/lucene/util/SloppyMath.java | 20 +-
.../src/java/org/apache/lucene/util/Sorter.java | 59 +-
.../lucene/util/StringMSBRadixSorter.java | 76 +
.../java/org/apache/lucene/util/Version.java | 34 +-
.../apache/lucene/util/automaton/Automaton.java | 4 +-
.../lucene/util/automaton/ByteRunAutomaton.java | 2 +-
.../util/automaton/CharacterRunAutomaton.java | 4 +-
.../util/automaton/CompiledAutomaton.java | 2 +-
.../automaton/DaciukMihovAutomatonBuilder.java | 1 -
.../lucene/util/automaton/RunAutomaton.java | 15 +-
.../org/apache/lucene/util/bkd/BKDReader.java | 128 +-
.../org/apache/lucene/util/bkd/BKDWriter.java | 775 ++-
.../apache/lucene/util/bkd/DocIdsWriter.java | 170 +
.../apache/lucene/util/bkd/HeapPointWriter.java | 21 +-
.../util/bkd/MutablePointsReaderUtils.java | 186 +
.../org/apache/lucene/util/fst/Builder.java | 9 +-
.../lucene/util/mutable/MutableValueStr.java | 1 -
.../org/apache/lucene/util/packed/Direct16.java | 2 +-
.../org/apache/lucene/util/packed/Direct32.java | 2 +-
.../org/apache/lucene/util/packed/Direct64.java | 2 +-
.../org/apache/lucene/util/packed/Direct8.java | 2 +-
.../util/packed/DirectMonotonicReader.java | 2 -
.../util/packed/MonotonicBlockPackedWriter.java | 1 -
.../lucene/util/packed/MonotonicLongValues.java | 1 -
.../lucene/util/packed/Packed16ThreeBlocks.java | 2 +-
.../lucene/util/packed/Packed64SingleBlock.java | 2 +-
.../lucene/util/packed/Packed8ThreeBlocks.java | 2 +-
.../apache/lucene/util/packed/PackedInts.java | 4 +-
lucene/core/src/java/overview.html | 2 +-
.../services/org.apache.lucene.codecs.Codec | 2 +-
.../org.apache.lucene.codecs.DocValuesFormat | 2 +-
.../lucene/analysis/TestCharArrayMap.java | 244 +
.../lucene/analysis/TestCharArraySet.java | 430 ++
.../lucene/analysis/TestCharacterUtils.java | 107 +
.../lucene/analysis/TestNumericTokenStream.java | 169 -
.../apache/lucene/analysis/TestStopFilter.java | 139 +
.../lucene/analysis/TestWordlistLoader.java | 79 +
.../analysis/standard/TestStandardAnalyzer.java | 396 ++
.../org/apache/lucene/codecs/TestCodecUtil.java | 306 +
.../TestGrowableByteArrayDataOutput.java | 2 +-
.../lucene50/TestLucene50FieldInfoFormat.java | 33 -
.../lucene50/TestLucene50SegmentInfoFormat.java | 39 -
...cene50StoredFieldsFormatHighCompression.java | 8 +-
.../lucene50/TestLucene60FieldInfoFormat.java | 33 +
.../lucene53/TestLucene53NormsFormat.java | 34 -
.../lucene54/TestLucene54DocValuesFormat.java | 609 --
.../lucene62/TestLucene62SegmentInfoFormat.java | 39 +
.../lucene/codecs/lucene70/TestIndexedDISI.java | 223 +
.../lucene70/TestLucene70DocValuesFormat.java | 537 ++
.../lucene70/TestLucene70NormsFormat.java | 34 +
.../perfield/TestPerFieldDocValuesFormat.java | 134 +-
.../perfield/TestPerFieldPostingsFormat2.java | 122 +-
.../lucene/document/TestBinaryDocument.java | 29 -
.../org/apache/lucene/document/TestField.java | 94 -
.../apache/lucene/document/TestFieldType.java | 11 -
.../org/apache/lucene/geo/TestGeoUtils.java | 11 +-
.../test/org/apache/lucene/geo/TestPolygon.java | 241 +
.../org/apache/lucene/geo/TestPolygon2D.java | 9 +-
.../lucene/index/Test2BBinaryDocValues.java | 6 +-
.../org/apache/lucene/index/Test2BDocs.java | 135 +
.../lucene/index/Test2BNumericDocValues.java | 3 +-
.../org/apache/lucene/index/Test2BPoints.java | 14 +-
.../index/Test2BSortedDocValuesFixedSorted.java | 5 +-
.../lucene/index/Test2BSortedDocValuesOrds.java | 5 +-
.../org/apache/lucene/index/Test2BTerms.java | 2 +-
.../org/apache/lucene/index/TestAddIndexes.java | 55 +-
.../index/TestAllFilesCheckIndexHeader.java | 11 +-
.../apache/lucene/index/TestBagOfPositions.java | 4 +-
.../index/TestBinaryDocValuesUpdates.java | 141 +-
.../org/apache/lucene/index/TestCodecUtil.java | 305 -
.../org/apache/lucene/index/TestCodecs.java | 9 +-
.../index/TestConcurrentMergeScheduler.java | 3 -
.../test/org/apache/lucene/index/TestCrash.java | 1 -
.../apache/lucene/index/TestCustomNorms.java | 15 +-
.../apache/lucene/index/TestDeletionPolicy.java | 4 +-
.../index/TestDemoParallelLeafReader.java | 96 +-
.../lucene/index/TestDirectoryReader.java | 12 +-
.../lucene/index/TestDirectoryReaderReopen.java | 243 +-
.../test/org/apache/lucene/index/TestDoc.java | 2 +-
.../apache/lucene/index/TestDocIDMerger.java | 203 +
.../org/apache/lucene/index/TestDocValues.java | 11 +-
.../lucene/index/TestDocValuesIndexing.java | 28 +-
.../lucene/index/TestDocsAndPositions.java | 1 -
.../index/TestDocumentsWriterDeleteQueue.java | 6 +-
.../index/TestDocumentsWriterStallControl.java | 6 +-
.../index/TestExitableDirectoryReader.java | 5 +-
.../org/apache/lucene/index/TestFieldReuse.java | 53 +-
.../apache/lucene/index/TestFieldsReader.java | 1 -
.../lucene/index/TestIndexFileDeleter.java | 7 -
.../org/apache/lucene/index/TestIndexInput.java | 4 -
.../apache/lucene/index/TestIndexSorting.java | 1403 +++++
.../apache/lucene/index/TestIndexWriter.java | 49 +-
.../lucene/index/TestIndexWriterCommit.java | 58 +-
.../lucene/index/TestIndexWriterConfig.java | 2 -
.../lucene/index/TestIndexWriterDelete.java | 5 +-
.../lucene/index/TestIndexWriterExceptions.java | 8 +-
.../index/TestIndexWriterExceptions2.java | 1 -
.../lucene/index/TestIndexWriterForceMerge.java | 4 -
.../lucene/index/TestIndexWriterFromReader.java | 10 -
.../index/TestIndexWriterLockRelease.java | 2 -
.../lucene/index/TestIndexWriterMerging.java | 5 -
.../lucene/index/TestIndexWriterOnDiskFull.java | 3 -
.../TestIndexWriterOutOfFileDescriptors.java | 1 -
.../index/TestIndexWriterWithThreads.java | 3 -
.../index/TestIndexingSequenceNumbers.java | 644 ++
.../lucene/index/TestMaxTermFrequency.java | 5 +-
.../lucene/index/TestMergeRateLimiter.java | 2 -
.../lucene/index/TestMixedDocValuesUpdates.java | 27 +-
.../apache/lucene/index/TestMultiDocValues.java | 142 +-
.../apache/lucene/index/TestMultiTermsEnum.java | 1 -
.../lucene/index/TestNRTReaderCleanup.java | 3 -
.../apache/lucene/index/TestNeverDelete.java | 2 -
.../apache/lucene/index/TestNoMergePolicy.java | 1 -
.../test/org/apache/lucene/index/TestNorms.java | 45 +-
.../index/TestNumericDocValuesUpdates.java | 253 +-
.../org/apache/lucene/index/TestOmitNorms.java | 19 +-
.../apache/lucene/index/TestOmitPositions.java | 1 -
.../org/apache/lucene/index/TestOmitTf.java | 6 -
.../org/apache/lucene/index/TestOrdinalMap.java | 10 +-
.../index/TestParallelCompositeReader.java | 2 -
.../lucene/index/TestParallelLeafReader.java | 59 +-
.../lucene/index/TestParallelTermEnum.java | 1 -
.../org/apache/lucene/index/TestPayloads.java | 1 -
.../lucene/index/TestPerSegmentDeletes.java | 11 +-
.../TestPersistentSnapshotDeletionPolicy.java | 1 -
.../apache/lucene/index/TestPointValues.java | 36 +-
.../lucene/index/TestPostingsOffsets.java | 20 +-
.../apache/lucene/index/TestReaderClosed.java | 1 -
.../apache/lucene/index/TestRollingUpdates.java | 2 +-
.../apache/lucene/index/TestSegmentInfos.java | 7 +-
.../apache/lucene/index/TestSegmentMerger.java | 24 +-
.../index/TestSnapshotDeletionPolicy.java | 2 -
.../test/org/apache/lucene/index/TestTerms.java | 134 -
.../org/apache/lucene/index/TestTermsEnum.java | 13 +-
.../lucene/index/TestTransactionRollback.java | 4 +-
.../apache/lucene/index/TestTransactions.java | 2 -
.../org/apache/lucene/index/TestTryDelete.java | 17 +-
.../lucene/index/TestTwoPhaseCommitTool.java | 14 +-
.../lucene/index/TestUniqueTermCount.java | 5 +-
.../search/FuzzyTermOnShortTermsTest.java | 15 +-
.../apache/lucene/search/JustCompileSearch.java | 45 +-
.../org/apache/lucene/search/TestBoolean2.java | 218 +-
.../apache/lucene/search/TestBooleanCoord.java | 860 ---
.../search/TestBooleanMinShouldMatch.java | 58 +-
.../org/apache/lucene/search/TestBooleanOr.java | 2 +-
.../apache/lucene/search/TestBooleanQuery.java | 37 +-
.../search/TestBooleanQueryVisitSubscorers.java | 5 +-
.../lucene/search/TestBooleanRewrites.java | 120 +-
.../apache/lucene/search/TestBooleanScorer.java | 38 +-
.../apache/lucene/search/TestBoostQuery.java | 4 +-
.../lucene/search/TestComplexExplanations.java | 17 +-
.../lucene/search/TestConjunctionDISI.java | 110 +-
.../apache/lucene/search/TestConjunctions.java | 13 +-
.../lucene/search/TestConstantScoreQuery.java | 24 +-
.../TestControlledRealTimeReopenThread.java | 60 +-
.../lucene/search/TestDisjunctionMaxQuery.java | 16 +
.../lucene/search/TestDocValuesScoring.java | 40 +-
.../TestEarlyTerminatingSortingCollector.java | 254 +
.../lucene/search/TestElevationComparator.java | 20 +-
.../apache/lucene/search/TestFilterWeight.java | 67 +
.../apache/lucene/search/TestFuzzyQuery.java | 213 +
.../apache/lucene/search/TestIndexSearcher.java | 1 -
.../apache/lucene/search/TestLRUQueryCache.java | 50 +-
.../lucene/search/TestMatchNoDocsQuery.java | 49 +-
.../lucene/search/TestMinShouldMatch2.java | 20 +-
.../lucene/search/TestMultiPhraseQuery.java | 27 -
.../search/TestMultiTermConstantScore.java | 74 -
.../search/TestMultiTermQueryRewrites.java | 12 +-
.../TestMultiValuedNumericRangeQuery.java | 80 -
.../apache/lucene/search/TestNeedsScores.java | 39 +-
.../lucene/search/TestNumericRangeQuery32.java | 589 --
.../lucene/search/TestNumericRangeQuery64.java | 623 --
.../apache/lucene/search/TestPhraseQuery.java | 1 -
.../apache/lucene/search/TestPointQueries.java | 38 +-
.../lucene/search/TestPositionIncrement.java | 4 +-
.../search/TestPositiveScoresOnlyCollector.java | 2 +-
.../apache/lucene/search/TestQueryRescorer.java | 42 +-
.../apache/lucene/search/TestRegexpRandom2.java | 1 -
.../search/TestScoreCachingWrappingScorer.java | 2 +-
.../apache/lucene/search/TestScorerPerf.java | 17 +-
.../lucene/search/TestSearcherManager.java | 144 +
.../apache/lucene/search/TestSimilarity.java | 4 -
.../lucene/search/TestSimilarityProvider.java | 25 +-
.../lucene/search/TestSimpleExplanations.java | 42 +-
.../TestSimpleExplanationsWithFillerDocs.java | 128 +
.../apache/lucene/search/TestSortRandom.java | 23 +-
.../org/apache/lucene/search/TestTermQuery.java | 154 +
.../org/apache/lucene/search/TestWildcard.java | 3 +-
.../search/similarities/TestSimilarity2.java | 5 -
.../search/similarities/TestSimilarityBase.java | 8 +-
.../search/spans/JustCompileSearchSpans.java | 16 +-
.../search/spans/TestFieldMaskingSpanQuery.java | 14 +-
.../search/spans/TestNearSpansOrdered.java | 50 +-
.../lucene/search/spans/TestSpanCollection.java | 17 +-
.../search/spans/TestSpanContainQuery.java | 2 +-
.../apache/lucene/search/spans/TestSpans.java | 10 +-
.../lucene/store/TestBufferedIndexInput.java | 1 -
.../org/apache/lucene/store/TestDirectory.java | 1 -
.../lucene/store/TestFileSwitchDirectory.java | 1 -
.../apache/lucene/store/TestMmapDirectory.java | 38 +
.../apache/lucene/store/TestRateLimiter.java | 1 -
.../lucene/store/TestSimpleFSLockFactory.java | 1 -
.../store/TestTrackingDirectoryWrapper.java | 2 +-
.../apache/lucene/util/Test2BPagedBytes.java | 1 -
.../org/apache/lucene/util/TestArrayUtil.java | 64 +-
.../apache/lucene/util/TestByteBlockPool.java | 21 +-
.../apache/lucene/util/TestDocIdSetBuilder.java | 14 +-
.../org/apache/lucene/util/TestIOUtils.java | 1 -
.../apache/lucene/util/TestIntroSelector.java | 86 +
.../lucene/util/TestLegacyNumericUtils.java | 564 --
.../apache/lucene/util/TestMSBRadixSorter.java | 185 +
.../apache/lucene/util/TestOfflineSorter.java | 13 -
.../apache/lucene/util/TestQueryBuilder.java | 43 +-
.../apache/lucene/util/TestRadixSelector.java | 106 +
.../util/TestRecyclingByteBlockAllocator.java | 1 -
.../util/TestRecyclingIntBlockAllocator.java | 1 -
.../lucene/util/TestStringMSBRadixSorter.java | 97 +
.../automaton/FiniteStringsIteratorTest.java | 1 -
.../LimitedFiniteStringsIteratorTest.java | 1 -
.../lucene/util/automaton/TestUTF32ToUTF8.java | 4 +-
.../org/apache/lucene/util/bkd/TestBKD.java | 58 +-
.../lucene/util/bkd/TestDocIdsWriter.java | 101 +
.../util/bkd/TestMutablePointsReaderUtils.java | 255 +
.../org/apache/lucene/util/fst/TestFSTs.java | 1 -
.../lucene/util/packed/TestDirectPacked.java | 6 +-
lucene/default-nested-ivy-settings.xml | 60 +
.../demo/facet/DistanceFacetsExample.java | 27 +-
.../test/org/apache/lucene/demo/TestDemo.java | 2 -
.../TestExpressionAggregationFacetsExample.java | 3 -
.../demo/facet/TestRangeFacetsExample.java | 2 -
.../facet/TestSimpleSortedSetFacetsExample.java | 1 -
.../lucene/expressions/SimpleBindings.java | 2 +-
.../expressions/js/JavascriptCompiler.java | 2 +-
.../lucene/expressions/js/JavascriptLexer.java | 3 -
.../lucene/expressions/js/JavascriptParser.java | 3 -
.../expressions/TestExpressionValueSource.java | 1 +
.../org/apache/lucene/facet/DrillDownQuery.java | 22 +-
.../org/apache/lucene/facet/DrillSideways.java | 2 +-
.../apache/lucene/facet/DrillSidewaysQuery.java | 45 +-
.../apache/lucene/facet/FacetsCollector.java | 42 +-
.../apache/lucene/facet/range/DoubleRange.java | 25 +-
.../apache/lucene/facet/range/LongRange.java | 25 +-
.../DefaultSortedSetDocValuesReaderState.java | 9 +-
.../SortedSetDocValuesFacetCounts.java | 58 +-
.../facet/taxonomy/DocValuesOrdinalsReader.java | 19 +-
.../facet/taxonomy/FastTaxonomyFacetCounts.java | 12 +-
.../taxonomy/OrdinalMappingLeafReader.java | 18 +-
.../facet/taxonomy/TaxonomyFacetCounts.java | 2 +-
.../TaxonomyFacetSumFloatAssociations.java | 39 +-
.../TaxonomyFacetSumIntAssociations.java | 39 +-
.../lucene/facet/taxonomy/TaxonomyReader.java | 2 +-
.../lucene/facet/taxonomy/TaxonomyWriter.java | 16 +-
.../directory/DirectoryTaxonomyWriter.java | 70 +-
.../apache/lucene/facet/TestDrillDownQuery.java | 11 +
.../apache/lucene/facet/TestDrillSideways.java | 30 +-
.../lucene/facet/TestMultipleIndexFields.java | 2 +-
.../facet/range/TestRangeFacetCounts.java | 25 +-
.../lucene/facet/taxonomy/TestFacetLabel.java | 2 -
.../taxonomy/TestOrdinalMappingLeafReader.java | 6 +-
.../taxonomy/directory/TestAddTaxonomy.java | 16 +-
.../directory/TestDirectoryTaxonomyWriter.java | 20 +-
.../AbstractFirstPassGroupingCollector.java | 6 +-
.../grouping/AbstractGroupFacetCollector.java | 1 -
.../search/grouping/BlockGroupingCollector.java | 2 +-
.../lucene/search/grouping/SearchGroup.java | 8 +-
.../FunctionFirstPassGroupingCollector.java | 2 +-
.../term/TermAllGroupHeadsCollector.java | 170 +-
.../grouping/term/TermAllGroupsCollector.java | 22 +-
.../term/TermDistinctValuesCollector.java | 37 +-
.../term/TermFirstPassGroupingCollector.java | 16 +-
.../grouping/term/TermGroupFacetCollector.java | 63 +-
.../term/TermSecondPassGroupingCollector.java | 17 +-
.../grouping/AllGroupHeadsCollectorTest.java | 11 +-
.../grouping/DistinctValuesCollectorTest.java | 2 +-
.../lucene/search/grouping/TestGrouping.java | 43 +-
.../search/highlight/GradientFormatter.java | 2 +-
.../search/highlight/TermVectorLeafReader.java | 14 +-
.../highlight/WeightedSpanTermExtractor.java | 85 +-
.../lucene/search/highlight/package-info.java | 4 +-
.../CustomSeparatorBreakIterator.java | 4 +-
.../postingshighlight/WholeBreakIterator.java | 4 +-
.../uhighlight/AnalysisOffsetStrategy.java | 190 +
.../uhighlight/DefaultPassageFormatter.java | 138 +
.../search/uhighlight/FieldHighlighter.java | 276 +
.../search/uhighlight/FieldOffsetStrategy.java | 122 +
.../uhighlight/MultiTermHighlighting.java | 379 ++
.../uhighlight/MultiValueTokenStream.java | 148 +
.../search/uhighlight/NoOpOffsetStrategy.java | 50 +
.../lucene/search/uhighlight/OffsetsEnum.java | 97 +
.../lucene/search/uhighlight/Passage.java | 161 +
.../search/uhighlight/PassageFormatter.java | 40 +
.../lucene/search/uhighlight/PassageScorer.java | 113 +
.../lucene/search/uhighlight/PhraseHelper.java | 581 ++
.../uhighlight/PostingsOffsetStrategy.java | 61 +
.../PostingsWithTermVectorsOffsetStrategy.java | 71 +
.../uhighlight/SplittingBreakIterator.java | 244 +
.../TermVectorFilteredLeafReader.java | 130 +
.../uhighlight/TermVectorOffsetStrategy.java | 68 +
.../uhighlight/TokenStreamFromTermVector.java | 395 ++
.../search/uhighlight/UnifiedHighlighter.java | 1021 ++++
.../lucene/search/uhighlight/package-info.java | 22 +
.../search/vectorhighlight/FieldQuery.java | 13 +
.../search/highlight/HighlighterTest.java | 72 +-
.../highlight/OffsetLimitTokenFilterTest.java | 1 -
.../custom/HighlightCustomQueryTest.java | 25 +-
.../lucene/search/uhighlight/CambridgeMA.utf8 | 1 +
.../uhighlight/TestSplittingBreakIterator.java | 192 +
.../uhighlight/TestUnifiedHighlighter.java | 962 +++
.../uhighlight/TestUnifiedHighlighterMTQ.java | 936 +++
.../TestUnifiedHighlighterRanking.java | 339 ++
.../TestUnifiedHighlighterReanalysis.java | 74 +
.../TestUnifiedHighlighterStrictPhrases.java | 404 ++
.../TestUnifiedHighlighterTermVec.java | 182 +
.../lucene/search/uhighlight/UHTestHelper.java | 69 +
.../TestUnifiedHighlighterExtensibility.java | 182 +
.../FastVectorHighlighterTest.java | 38 +
.../search/vectorhighlight/FieldQueryTest.java | 116 +-
.../vectorhighlight/IndexTimeSynonymTest.java | 4 -
lucene/ivy-settings.xml | 63 -
lucene/ivy-versions.properties | 46 +-
.../search/join/BaseGlobalOrdinalScorer.java | 6 +-
.../lucene/search/join/BlockJoinSelector.java | 200 +-
.../search/join/DocValuesTermsCollector.java | 88 +-
.../search/join/GenericTermsCollector.java | 32 +-
.../search/join/GlobalOrdinalsCollector.java | 21 +-
.../lucene/search/join/GlobalOrdinalsQuery.java | 52 +-
.../join/GlobalOrdinalsWithScoreCollector.java | 36 +-
.../join/GlobalOrdinalsWithScoreQuery.java | 82 +-
.../org/apache/lucene/search/join/JoinUtil.java | 93 +-
.../join/PointInSetIncludingScoreQuery.java | 80 +-
.../lucene/search/join/TermsCollector.java | 23 +-
.../search/join/TermsIncludingScoreQuery.java | 49 +-
.../search/join/TermsWithScoreCollector.java | 89 +-
.../search/join/ToChildBlockJoinQuery.java | 57 +-
.../search/join/ToParentBlockJoinCollector.java | 4 +-
.../search/join/ToParentBlockJoinQuery.java | 99 +-
.../search/join/ToParentBlockJoinSortField.java | 54 +-
.../lucene/search/join/TestBlockJoin.java | 201 +-
.../search/join/TestBlockJoinSelector.java | 197 +-
.../search/join/TestBlockJoinSorting.java | 9 +-
.../search/join/TestBlockJoinValidation.java | 4 +-
.../apache/lucene/search/join/TestJoinUtil.java | 118 +-
lucene/licenses/asm-5.0.4.jar.sha1 | 1 -
lucene/licenses/asm-5.1.jar.sha1 | 1 +
lucene/licenses/asm-commons-5.0.4.jar.sha1 | 1 -
lucene/licenses/asm-commons-5.1.jar.sha1 | 1 +
lucene/licenses/commons-compress-1.11.jar.sha1 | 1 +
lucene/licenses/commons-compress-1.8.1.jar.sha1 | 1 -
lucene/licenses/javax.servlet-LICENSE-CDDL.txt | 139 -
.../jcl-over-slf4j-LICENSE-BSD_LIKE.txt | 21 -
lucene/licenses/jcl-over-slf4j-LICENSE-MIT.txt | 21 +
lucene/licenses/morfologik-fsa-2.1.0.jar.sha1 | 1 -
lucene/licenses/morfologik-fsa-2.1.1.jar.sha1 | 1 +
.../licenses/morfologik-polish-2.1.0.jar.sha1 | 1 -
.../licenses/morfologik-polish-2.1.1.jar.sha1 | 1 +
.../licenses/morfologik-stemming-2.1.0.jar.sha1 | 1 -
.../licenses/morfologik-stemming-2.1.1.jar.sha1 | 1 +
lucene/licenses/servlet-api-LICENSE-CDDL.txt | 137 -
lucene/licenses/slf4j-LICENSE-BSD_LIKE.txt | 21 -
lucene/licenses/slf4j-LICENSE-MIT.txt | 21 +
.../apache/lucene/index/memory/MemoryIndex.java | 233 +-
.../lucene/index/memory/TestMemoryIndex.java | 90 +-
.../memory/TestMemoryIndexAgainstRAMDir.java | 66 +-
.../org/apache/lucene/index/IndexSplitter.java | 2 +-
.../apache/lucene/index/MergeReaderWrapper.java | 259 -
.../lucene/index/MultiPassIndexSplitter.java | 1 -
.../index/SlowCompositeReaderWrapper.java | 275 -
.../java/org/apache/lucene/index/Sorter.java | 287 -
.../apache/lucene/index/SortingLeafReader.java | 940 ---
.../apache/lucene/index/SortingMergePolicy.java | 264 -
.../java/org/apache/lucene/index/package.html | 18 -
.../org/apache/lucene/misc/IndexMergeTool.java | 4 +-
.../apache/lucene/misc/SweetSpotSimilarity.java | 16 +
.../search/BlockJoinComparatorSource.java | 224 -
.../search/DiversifiedTopDocsCollector.java | 14 +-
.../EarlyTerminatingSortingCollector.java | 146 -
.../store/HardlinkCopyDirectoryWrapper.java | 95 +
.../apache/lucene/uninverting/DocTermOrds.java | 886 ---
.../apache/lucene/uninverting/FieldCache.java | 466 --
.../lucene/uninverting/FieldCacheImpl.java | 1085 ----
.../uninverting/FieldCacheSanityChecker.java | 425 --
.../lucene/uninverting/UninvertingReader.java | 391 --
.../apache/lucene/uninverting/package-info.java | 21 -
.../apache/lucene/index/IndexSortingTest.java | 89 -
.../org/apache/lucene/index/SorterTestBase.java | 405 --
.../lucene/index/SortingLeafReaderTest.java | 73 -
.../lucene/index/TestBlockJoinSorter.java | 128 -
.../index/TestMultiPassIndexSplitter.java | 1 -
.../index/TestSlowCompositeReaderWrapper.java | 91 -
.../lucene/index/TestSortingMergePolicy.java | 201 -
.../search/TestDiversifiedTopDocsCollector.java | 85 +-
.../TestEarlyTerminatingSortingCollector.java | 305 -
.../store/TestHardLinkCopyDirectoryWrapper.java | 89 +
.../lucene/uninverting/TestDocTermOrds.java | 681 ---
.../lucene/uninverting/TestFieldCache.java | 731 ---
.../uninverting/TestFieldCacheReopen.java | 70 -
.../TestFieldCacheSanityChecker.java | 164 -
.../lucene/uninverting/TestFieldCacheSort.java | 1814 ------
.../uninverting/TestFieldCacheSortRandom.java | 315 -
.../uninverting/TestFieldCacheVsDocValues.java | 592 --
.../uninverting/TestFieldCacheWithThreads.java | 227 -
.../uninverting/TestLegacyFieldCache.java | 498 --
.../lucene/uninverting/TestNumericTerms32.java | 156 -
.../lucene/uninverting/TestNumericTerms64.java | 166 -
.../uninverting/TestUninvertingReader.java | 395 --
.../apache/lucene/util/fst/TestFSTsMisc.java | 2 -
.../apache/lucene/queries/BoostingQuery.java | 45 +-
.../apache/lucene/queries/CommonTermsQuery.java | 93 +-
.../apache/lucene/queries/CustomScoreQuery.java | 106 +-
.../org/apache/lucene/queries/TermsQuery.java | 27 +-
.../lucene/queries/function/BoostedQuery.java | 34 +-
.../lucene/queries/function/FunctionQuery.java | 47 +-
.../queries/function/FunctionRangeQuery.java | 36 +-
.../lucene/queries/function/FunctionValues.java | 60 +-
.../lucene/queries/function/ValueSource.java | 10 +-
.../queries/function/ValueSourceScorer.java | 2 +-
.../function/docvalues/BoolDocValues.java | 24 +-
.../docvalues/DocTermsIndexDocValues.java | 78 +-
.../function/docvalues/DoubleDocValues.java | 32 +-
.../function/docvalues/FloatDocValues.java | 22 +-
.../function/docvalues/IntDocValues.java | 24 +-
.../function/docvalues/LongDocValues.java | 26 +-
.../function/docvalues/StrDocValues.java | 12 +-
.../valuesource/BytesRefFieldSource.java | 56 +-
.../valuesource/ComparisonBoolFunction.java | 105 +
.../function/valuesource/DefFunction.java | 26 +-
.../function/valuesource/DivFloatFunction.java | 4 +-
.../function/valuesource/DoubleFieldSource.java | 44 +-
.../function/valuesource/DualFloatFunction.java | 8 +-
.../function/valuesource/EnumFieldSource.java | 37 +-
.../function/valuesource/FloatFieldSource.java | 40 +-
.../function/valuesource/IfFunction.java | 24 +-
.../function/valuesource/IntFieldSource.java | 40 +-
.../valuesource/JoinDocFreqValueSource.java | 27 +-
.../valuesource/LinearFloatFunction.java | 6 +-
.../function/valuesource/LongFieldSource.java | 56 +-
.../function/valuesource/MaxFloatFunction.java | 6 +-
.../function/valuesource/MinFloatFunction.java | 6 +-
.../function/valuesource/MultiBoolFunction.java | 6 +-
.../valuesource/MultiFloatFunction.java | 10 +-
.../function/valuesource/MultiFunction.java | 13 +-
.../function/valuesource/NormValueSource.java | 23 +-
.../function/valuesource/PowFloatFunction.java | 4 +-
.../valuesource/ProductFloatFunction.java | 4 +-
.../valuesource/RangeMapFloatFunction.java | 4 +-
.../valuesource/ReciprocalFloatFunction.java | 6 +-
.../valuesource/ScaleFloatFunction.java | 6 +-
.../valuesource/SimpleBoolFunction.java | 6 +-
.../valuesource/SimpleFloatFunction.java | 6 +-
.../valuesource/SortedSetFieldSource.java | 6 +-
.../function/valuesource/SumFloatFunction.java | 6 +-
.../SumTotalTermFreqValueSource.java | 1 -
.../function/valuesource/VectorValueSource.java | 33 +-
.../lucene/queries/mlt/MoreLikeThisQuery.java | 56 +-
.../queries/payloads/PayloadScoreQuery.java | 49 +-
.../queries/payloads/SpanPayloadCheckQuery.java | 23 +-
.../lucene/queries/BoostingQueryTest.java | 4 +-
.../lucene/queries/CommonTermsQueryTest.java | 10 +-
.../apache/lucene/queries/TermsQueryTest.java | 2 -
.../queries/TestCustomScoreExplanations.java | 13 +-
.../lucene/queries/TestCustomScoreQuery.java | 15 +-
.../function/TestLongNormValueSource.java | 12 -
.../queries/function/TestValueSources.java | 4 +-
.../lucene/queries/mlt/TestMoreLikeThis.java | 6 +-
.../queries/payloads/TestPayloadScoreQuery.java | 10 -
.../queries/payloads/TestPayloadSpans.java | 30 +-
.../queries/payloads/TestPayloadTermQuery.java | 14 +-
lucene/queryparser/build.xml | 2 +-
.../analyzing/AnalyzingQueryParser.java | 202 -
.../queryparser/analyzing/package-info.java | 22 -
.../lucene/queryparser/classic/CharStream.java | 2 +-
.../classic/MultiFieldQueryParser.java | 51 +-
.../queryparser/classic/ParseException.java | 2 +-
.../lucene/queryparser/classic/QueryParser.java | 402 +-
.../lucene/queryparser/classic/QueryParser.jj | 267 +-
.../queryparser/classic/QueryParserBase.java | 179 +-
.../classic/QueryParserTokenManager.java | 37 +-
.../lucene/queryparser/classic/Token.java | 2 +-
.../queryparser/classic/TokenMgrError.java | 2 +-
.../complexPhrase/ComplexPhraseQueryParser.java | 74 +-
.../queryparser/ext/ExtendableQueryParser.java | 1 -
.../flexible/core/nodes/QueryNodeImpl.java | 2 +-
.../CommonQueryParserConfiguration.java | 12 -
.../flexible/standard/StandardQueryParser.java | 51 +-
.../builders/DummyQueryNodeBuilder.java | 1 -
.../LegacyNumericRangeQueryNodeBuilder.java | 93 -
.../StandardBooleanQueryNodeBuilder.java | 109 -
.../builders/StandardQueryTreeBuilder.java | 10 +-
.../builders/SynonymQueryNodeBuilder.java | 48 +
.../standard/config/LegacyNumericConfig.java | 166 -
.../LegacyNumericFieldConfigListener.java | 75 -
.../config/StandardQueryConfigHandler.java | 38 +-
.../standard/nodes/LegacyNumericQueryNode.java | 153 -
.../nodes/LegacyNumericRangeQueryNode.java | 153 -
.../nodes/StandardBooleanQueryNode.java | 49 -
.../standard/nodes/SynonymQueryNode.java | 30 +
.../StandardSyntaxParserTokenManager.java | 22 -
.../processors/AnalyzerQueryNodeProcessor.java | 8 +-
.../processors/FuzzyQueryNodeProcessor.java | 11 +-
.../LegacyNumericQueryNodeProcessor.java | 154 -
.../LegacyNumericRangeQueryNodeProcessor.java | 170 -
...owercaseExpandedTermsQueryNodeProcessor.java | 100 -
.../MultiFieldQueryNodeProcessor.java | 1 -
.../processors/RegexpQueryNodeProcessor.java | 56 +
.../StandardQueryNodeProcessorPipeline.java | 9 +-
.../processors/TermRangeQueryNodeProcessor.java | 11 +-
.../processors/WildcardQueryNodeProcessor.java | 58 +-
.../queryparser/simple/SimpleQueryParser.java | 20 +-
.../surround/parser/QueryParser.java | 2 -
.../parser/QueryParserTokenManager.java | 13 -
.../surround/query/RewriteQuery.java | 33 +-
.../lucene/queryparser/xml/CoreParser.java | 3 +-
.../xml/builders/BooleanQueryBuilder.java | 1 -
.../xml/builders/FuzzyLikeThisQueryBuilder.java | 4 +-
.../LegacyNumericRangeQueryBuilder.java | 135 -
.../xml/builders/TermsQueryBuilder.java | 1 -
.../analyzing/TestAnalyzingQueryParser.java | 268 -
.../queryparser/classic/TestMultiAnalyzer.java | 51 +-
.../classic/TestMultiFieldQueryParser.java | 22 +-
.../classic/TestMultiPhraseQueryParsing.java | 1 -
.../queryparser/classic/TestQueryParser.java | 485 +-
.../ext/TestExtendableQueryParser.java | 1 +
.../precedence/TestPrecedenceQueryParser.java | 62 +-
.../standard/TestLegacyNumericQueryParser.java | 535 --
.../standard/TestMultiAnalyzerQPHelper.java | 3 -
.../standard/TestMultiFieldQPHelper.java | 24 +-
.../flexible/standard/TestQPHelper.java | 93 +-
.../flexible/standard/TestStandardQP.java | 29 +-
.../simple/TestSimpleQueryParser.java | 11 +-
.../queryparser/util/QueryParserTestBase.java | 149 +-
.../xml/CoreParserTestIndexData.java | 2 -
.../queryparser/xml/LegacyNumericRangeQuery.xml | 31 -
.../LegacyNumericRangeQueryWithoutLowerTerm.xml | 31 -
.../xml/LegacyNumericRangeQueryWithoutRange.xml | 31 -
.../LegacyNumericRangeQueryWithoutUpperTerm.xml | 31 -
.../lucene/queryparser/xml/TestCoreParser.java | 20 -
.../xml/TestQueryTemplateManager.java | 1 -
.../builders/TestNumericRangeQueryBuilder.java | 179 -
.../IndexAndTaxonomyReplicationHandler.java | 6 +-
.../replicator/IndexReplicationHandler.java | 3 +-
.../apache/lucene/replicator/nrt/CopyJob.java | 4 +-
.../lucene/replicator/nrt/CopyOneFile.java | 6 +-
.../apache/lucene/replicator/nrt/CopyState.java | 5 +-
.../lucene/replicator/nrt/FileMetaData.java | 9 +-
.../org/apache/lucene/replicator/nrt/Node.java | 31 +-
.../nrt/NodeCommunicationException.java | 4 +-
.../nrt/PreCopyMergedSegmentWarmer.java | 4 +-
.../lucene/replicator/nrt/PrimaryNode.java | 35 +-
.../replicator/nrt/ReplicaFileDeleter.java | 6 +-
.../lucene/replicator/nrt/ReplicaNode.java | 8 +-
.../nrt/SegmentInfosSearcherManager.java | 5 +-
.../IndexAndTaxonomyReplicationClientTest.java | 16 +-
.../IndexAndTaxonomyRevisionTest.java | 1 -
.../replicator/IndexReplicationClientTest.java | 15 +-
.../lucene/replicator/IndexRevisionTest.java | 1 -
.../lucene/replicator/LocalReplicatorTest.java | 5 +-
.../lucene/replicator/ReplicatorTestCase.java | 2 -
.../replicator/http/HttpReplicatorTest.java | 2 +-
.../lucene/replicator/nrt/Connection.java | 6 +-
.../org/apache/lucene/replicator/nrt/Jobs.java | 6 +-
.../lucene/replicator/nrt/NodeProcess.java | 4 +-
.../lucene/replicator/nrt/SimpleCopyJob.java | 43 +-
.../replicator/nrt/SimplePrimaryNode.java | 4 +-
.../replicator/nrt/SimpleReplicaNode.java | 4 +-
.../lucene/replicator/nrt/SimpleServer.java | 4 +-
.../lucene/replicator/nrt/SimpleTransLog.java | 7 +-
.../replicator/nrt/TestNRTReplication.java | 16 +-
.../nrt/TestStressNRTReplication.java | 5 +-
.../lucene/replicator/nrt/ThreadPumper.java | 4 +-
.../idversion/IDVersionPostingsReader.java | 1 -
.../idversion/IDVersionSegmentTermsEnum.java | 1 -
.../lucene/document/DoubleRangeField.java | 262 +
.../apache/lucene/document/FloatRangeField.java | 262 +
.../apache/lucene/document/HalfFloatPoint.java | 425 ++
.../lucene/document/InetAddressPoint.java | 2 +-
.../apache/lucene/document/IntRangeField.java | 262 +
.../org/apache/lucene/document/LatLonPoint.java | 17 +-
.../lucene/document/LatLonPointBoxQuery.java | 292 -
.../document/LatLonPointDistanceComparator.java | 58 +-
.../document/LatLonPointDistanceQuery.java | 79 +-
.../document/LatLonPointInPolygonQuery.java | 40 +-
.../apache/lucene/document/LongRangeField.java | 260 +
.../apache/lucene/document/MatchingPoints.java | 102 -
.../apache/lucene/document/RangeFieldQuery.java | 315 +
.../sandbox/queries/FuzzyLikeThisQuery.java | 332 +-
.../lucene/sandbox/queries/SlowFuzzyQuery.java | 201 -
.../sandbox/queries/SlowFuzzyTermsEnum.java | 264 -
.../lucene/search/DocValuesNumbersQuery.java | 55 +-
.../lucene/search/DocValuesRangeQuery.java | 64 +-
.../lucene/search/DocValuesTermsQuery.java | 39 +-
.../lucene/search/TermAutomatonQuery.java | 58 +-
.../lucene/search/TermAutomatonScorer.java | 2 +-
.../search/TokenStreamToTermAutomatonQuery.java | 2 -
.../lucene/document/TestDoubleRangeField.java | 67 +
.../lucene/document/TestHalfFloatPoint.java | 243 +
.../org/apache/lucene/document/TestNearest.java | 2 +-
.../sandbox/queries/FuzzyLikeThisQueryTest.java | 22 +-
.../sandbox/queries/TestSlowFuzzyQuery.java | 487 --
.../sandbox/queries/TestSlowFuzzyQuery2.java | 184 -
.../lucene/sandbox/queries/fuzzyTestData.txt | 3721 ------------
.../search/BaseRangeFieldQueryTestCase.java | 336 ++
.../search/TestDoubleRangeFieldQueries.java | 244 +
.../search/TestFloatRangeFieldQueries.java | 244 +
.../lucene/search/TestIntRangeFieldQueries.java | 244 +
.../search/TestLongRangeFieldQueries.java | 244 +
.../lucene/search/TestTermAutomatonQuery.java | 22 +-
lucene/site/changes/changes2html.pl | 4 +-
lucene/site/xsl/index.xsl | 14 +-
lucene/spatial-extras/build.xml | 8 +-
.../spatial/bbox/BBoxSimilarityValueSource.java | 9 +-
.../lucene/spatial/bbox/BBoxStrategy.java | 39 +-
.../lucene/spatial/bbox/BBoxValueSource.java | 54 +-
.../spatial/composite/CompositeVerifyQuery.java | 27 +-
.../composite/IntersectsRPTVerifyQuery.java | 21 +-
.../spatial/prefix/AbstractPrefixTreeQuery.java | 22 +-
.../prefix/BytesRefIteratorTokenStream.java | 2 +-
.../spatial/prefix/HeatmapFacetCounter.java | 34 +-
.../prefix/tree/DateRangePrefixTree.java | 229 +-
.../serialized/SerializedDVStrategy.java | 53 +-
.../spatial/util/CachingDoubleValueSource.java | 6 +-
.../util/DistanceToShapeValueSource.java | 4 +-
.../spatial/util/ShapeAreaValueSource.java | 6 +-
.../spatial/util/ShapePredicateValueSource.java | 4 +-
.../spatial/vector/DistanceValueSource.java | 33 +-
.../spatial/vector/PointVectorStrategy.java | 37 +-
.../lucene/spatial/bbox/TestBBoxStrategy.java | 8 +-
.../spatial/prefix/DateNRStrategyTest.java | 4 +-
.../spatial/prefix/HeatmapFacetCounterTest.java | 27 +-
.../prefix/tree/DateRangePrefixTreeTest.java | 48 +-
.../geopoint/document/GeoPointField.java | 122 +-
.../geopoint/document/GeoPointTokenStream.java | 31 +-
.../geopoint/search/GeoPointDistanceQuery.java | 37 +-
.../search/GeoPointDistanceQueryImpl.java | 68 +-
.../geopoint/search/GeoPointInBBoxQuery.java | 52 +-
.../search/GeoPointInBBoxQueryImpl.java | 37 +-
.../geopoint/search/GeoPointInPolygonQuery.java | 38 +-
.../search/GeoPointInPolygonQueryImpl.java | 19 +-
.../geopoint/search/GeoPointMultiTermQuery.java | 56 +-
.../search/GeoPointNumericTermsEnum.java | 162 -
.../search/GeoPointPrefixTermsEnum.java | 235 -
.../GeoPointTermQueryConstantScoreWrapper.java | 31 +-
.../geopoint/search/GeoPointTermsEnum.java | 187 +-
.../geopoint/search/GeoPointTestUtil.java | 287 -
.../geopoint/search/TestGeoPointQuery.java | 46 +-
.../search/TestLegacyGeoPointQuery.java | 103 -
.../lucene/spatial/util/TestGeoPointField.java | 4 +-
.../lucene/spatial3d/Geo3DDocValuesField.java | 486 ++
.../org/apache/lucene/spatial3d/Geo3DPoint.java | 157 +-
.../spatial3d/Geo3DPointDistanceComparator.java | 171 +
.../Geo3DPointOutsideDistanceComparator.java | 142 +
.../spatial3d/Geo3DPointOutsideSortField.java | 96 +
.../lucene/spatial3d/Geo3DPointSortField.java | 96 +
.../org/apache/lucene/spatial3d/Geo3DUtil.java | 241 +-
.../spatial3d/PointInGeo3DShapeQuery.java | 34 +-
.../spatial3d/PointInShapeIntersectVisitor.java | 30 +-
.../apache/lucene/spatial3d/geom/Bounds.java | 18 +
.../spatial3d/geom/GeoBaseDistanceShape.java | 6 +-
.../spatial3d/geom/GeoComplexPolygon.java | 111 +-
.../geom/GeoCompositeMembershipShape.java | 2 +-
.../spatial3d/geom/GeoConcavePolygon.java | 42 +-
.../lucene/spatial3d/geom/GeoConvexPolygon.java | 42 +-
.../spatial3d/geom/GeoDegeneratePoint.java | 2 +-
.../lucene/spatial3d/geom/GeoDistance.java | 6 +-
.../spatial3d/geom/GeoLongitudeSlice.java | 1 +
.../spatial3d/geom/GeoNorthRectangle.java | 1 +
.../lucene/spatial3d/geom/GeoRectangle.java | 1 +
.../spatial3d/geom/GeoSouthRectangle.java | 1 +
.../lucene/spatial3d/geom/GeoStandardPath.java | 47 +-
.../spatial3d/geom/GeoWideLongitudeSlice.java | 1 +
.../spatial3d/geom/GeoWideNorthRectangle.java | 1 +
.../lucene/spatial3d/geom/GeoWideRectangle.java | 1 +
.../spatial3d/geom/GeoWideSouthRectangle.java | 1 +
.../lucene/spatial3d/geom/LatLonBounds.java | 11 +
.../org/apache/lucene/spatial3d/geom/Plane.java | 651 +-
.../lucene/spatial3d/geom/SidedPlane.java | 34 +
.../lucene/spatial3d/geom/StandardXYZSolid.java | 2 -
.../apache/lucene/spatial3d/geom/Vector.java | 48 +-
.../apache/lucene/spatial3d/geom/XYZBounds.java | 20 +-
.../lucene/spatial3d/TestGeo3DDocValues.java | 53 +
.../apache/lucene/spatial3d/TestGeo3DPoint.java | 102 +-
.../lucene/spatial3d/geom/GeoBBoxTest.java | 26 +
.../lucene/spatial3d/geom/GeoCircleTest.java | 24 +-
.../lucene/spatial3d/geom/GeoPathTest.java | 45 +-
.../lucene/spatial3d/geom/GeoPolygonTest.java | 157 +-
.../lucene/search/spell/DirectSpellChecker.java | 22 +-
.../lucene/search/spell/LuceneDictionary.java | 1 -
.../search/spell/WordBreakSpellChecker.java | 1 -
.../search/suggest/DocumentDictionary.java | 12 +-
.../suggest/DocumentValueSourceDictionary.java | 8 +-
.../analyzing/AnalyzingInfixSuggester.java | 52 +-
.../analyzing/BlendedInfixSuggester.java | 26 +-
.../search/suggest/analyzing/FSTUtil.java | 1 -
.../suggest/analyzing/SuggestStopFilter.java | 5 +-
.../analyzing/SuggestStopFilterFactory.java | 10 +-
.../suggest/document/CompletionWeight.java | 8 -
.../search/suggest/document/ContextQuery.java | 17 +-
.../suggest/document/FuzzyCompletionQuery.java | 2 +-
.../suggest/document/PrefixCompletionQuery.java | 12 +-
.../suggest/document/RegexCompletionQuery.java | 11 +-
.../suggest/document/SuggestIndexSearcher.java | 2 +-
.../suggest/fst/FSTCompletionBuilder.java | 1 -
.../search/suggest/jaspell/JaspellLookup.java | 1 -
.../jaspell/JaspellTernarySearchTrie.java | 83 +-
.../search/suggest/FileDictionaryTest.java | 1 -
.../lucene/search/suggest/PersistenceTest.java | 1 -
.../analyzing/AnalyzingInfixSuggesterTest.java | 84 +-
.../analyzing/AnalyzingSuggesterTest.java | 1 -
.../analyzing/BlendedInfixSuggesterTest.java | 2 +-
.../suggest/analyzing/FuzzySuggesterTest.java | 1 -
.../analyzing/TestFreeTextSuggester.java | 6 +-
.../analyzing/TestSuggestStopFilter.java | 4 +-
.../analyzing/TestSuggestStopFilterFactory.java | 2 +-
.../document/TestPrefixCompletionQuery.java | 33 +-
.../suggest/document/TestSuggestField.java | 4 +-
.../search/suggest/fst/FSTCompletionTest.java | 1 -
.../search/suggest/fst/WFSTCompletionTest.java | 1 -
.../analysis/BaseTokenStreamTestCase.java | 8 +-
.../lucene/analysis/CollationTestBase.java | 1 -
.../apache/lucene/analysis/MockAnalyzer.java | 12 +-
.../lucene/analysis/MockBytesAnalyzer.java | 7 +
.../apache/lucene/analysis/MockCharFilter.java | 1 -
.../lucene/analysis/MockLowerCaseFilter.java | 40 +
.../lucene/analysis/MockPayloadAnalyzer.java | 1 -
.../lucene/analysis/MockSynonymAnalyzer.java | 28 +
.../lucene/analysis/MockSynonymFilter.java | 97 +
.../apache/lucene/analysis/MockTokenizer.java | 6 +-
.../standard/WordBreakTestUnicode_6_3_0.java | 5537 ++++++++++++++++++
.../generateJavaUnicodeWordBreakTest.pl | 232 +
.../lucene/analysis/standard/package.html | 26 +
.../asserting/AssertingDocValuesFormat.java | 173 +-
.../asserting/AssertingLiveDocsFormat.java | 2 +-
.../codecs/asserting/AssertingNormsFormat.java | 22 +-
.../codecs/asserting/AssertingPointsFormat.java | 139 +-
.../codecs/cranky/CrankyDocValuesFormat.java | 21 +-
.../lucene/codecs/cranky/CrankyNormsFormat.java | 4 +-
.../codecs/cranky/CrankyPointsFormat.java | 113 +-
.../mockrandom/MockRandomPostingsFormat.java | 14 +-
.../apache/lucene/geo/BaseGeoPointTestCase.java | 23 +-
.../org/apache/lucene/geo/EarthDebugger.java | 8 +-
.../java/org/apache/lucene/geo/GeoTestUtil.java | 8 +-
.../lucene/index/AssertingLeafReader.java | 521 +-
.../index/BaseCompoundFormatTestCase.java | 165 +-
.../index/BaseDocValuesFormatTestCase.java | 724 +--
.../index/BaseFieldInfoFormatTestCase.java | 2 +-
.../index/BaseIndexFileFormatTestCase.java | 104 +-
.../lucene/index/BaseNormsFormatTestCase.java | 411 +-
.../lucene/index/BasePointsFormatTestCase.java | 109 +-
.../index/BasePostingsFormatTestCase.java | 283 +-
.../index/BaseSegmentInfoFormatTestCase.java | 77 +-
.../index/BaseStoredFieldsFormatTestCase.java | 3 +-
.../index/BaseTermVectorsFormatTestCase.java | 2 -
.../java/org/apache/lucene/index/DocHelper.java | 1 -
.../lucene/index/FieldFilterLeafReader.java | 13 -
.../apache/lucene/index/ForceMergePolicy.java | 3 -
.../lucene/index/MockRandomMergePolicy.java | 50 +-
.../org/apache/lucene/index/RandomCodec.java | 24 +-
.../apache/lucene/index/RandomIndexWriter.java | 64 +-
.../lucene/index/RandomPostingsTester.java | 2 +-
.../ThreadedIndexingAndSearchingTestCase.java | 1 -
.../lucene/mockfile/MockFileSystemTestCase.java | 1 -
.../apache/lucene/mockfile/VirusCheckingFS.java | 4 +-
.../org/apache/lucene/mockfile/WindowsFS.java | 4 -
.../lucene/search/AssertingIndexSearcher.java | 23 +-
.../apache/lucene/search/AssertingQuery.java | 13 +-
.../apache/lucene/search/AssertingWeight.java | 29 +-
.../lucene/search/BaseExplanationTestCase.java | 43 +-
.../org/apache/lucene/search/CheckHits.java | 11 +-
.../org/apache/lucene/search/QueryUtils.java | 71 +-
.../lucene/search/RandomApproximationQuery.java | 49 +-
.../search/similarities/RandomSimilarity.java | 32 +-
.../lucene/search/spans/AssertingSpanQuery.java | 28 +-
.../search/spans/AssertingSpanWeight.java | 12 +-
.../lucene/store/BaseDirectoryTestCase.java | 93 +-
.../lucene/store/BaseLockFactoryTestCase.java | 1 -
.../lucene/store/MockDirectoryWrapper.java | 140 +-
.../lucene/store/MockIndexInputWrapper.java | 47 +-
.../store/SlowClosingMockIndexInputWrapper.java | 2 +-
.../store/SlowOpeningMockIndexInputWrapper.java | 2 +-
.../apache/lucene/util/CloseableDirectory.java | 1 -
.../org/apache/lucene/util/LineFileDocs.java | 1 -
.../org/apache/lucene/util/LuceneTestCase.java | 87 +-
.../util/TestRuleSetupAndRestoreClassEnv.java | 6 +-
.../java/org/apache/lucene/util/TestUtil.java | 24 +-
.../util/automaton/AutomatonTestUtil.java | 10 +-
.../analysis/TestLookaheadTokenFilter.java | 1 -
.../lucene/analysis/TestMockSynonymFilter.java | 151 +
.../TestCompressingStoredFieldsFormat.java | 4 -
.../TestCompressingTermVectorsFormat.java | 1 -
.../lucene/index/TestAssertingLeafReader.java | 8 -
.../lucene/mockfile/TestHandleTrackingFS.java | 4 -
.../lucene/mockfile/TestVirusCheckingFS.java | 4 +-
.../apache/lucene/mockfile/TestWindowsFS.java | 2 -
.../search/TestBaseExplanationTestCase.java | 104 +
.../lucene/store/TestMockDirectoryWrapper.java | 122 +-
lucene/tools/custom-tasks.xml | 4 +-
lucene/tools/forbiddenApis/base.txt | 4 -
lucene/tools/forbiddenApis/lucene.txt | 3 +
lucene/tools/javadoc/ecj.javadocs.prefs | 2 +-
lucene/tools/junit4/solr-tests.policy | 1 +
lucene/tools/junit4/tests.policy | 9 +-
.../dependencies/GetMavenDependenciesTask.java | 2 +-
.../lucene/validation/LibVersionsCheckTask.java | 15 +-
lucene/top-level-ivy-settings.xml | 24 +
solr/CHANGES.txt | 1109 +++-
solr/NOTICE.txt | 5 +-
solr/bin/install_solr_service.sh | 88 +-
solr/bin/solr | 432 +-
solr/bin/solr.cmd | 423 +-
solr/bin/solr.in.cmd | 49 +-
solr/bin/solr.in.sh | 38 +-
solr/build.xml | 29 +-
solr/common-build.xml | 25 +-
solr/contrib/analysis-extras/build.xml | 10 +
.../apache/solr/schema/ICUCollationField.java | 6 +-
.../collection1/conf/schema-folding-extra.xml | 64 +-
.../collection1/conf/schema-icucollate-dv.xml | 53 +-
.../solr/collection1/conf/schema-icucollate.xml | 53 +-
.../conf/schema-icucollateoptions.xml | 70 +-
.../solr/schema/TestICUCollationField.java | 17 +-
.../accumulator/FacetingAccumulator.java | 2 +-
.../facet/FieldFacetAccumulator.java | 44 +-
.../AbstractDelegatingStatsCollector.java | 2 +-
.../statistics/MedianStatsCollector.java | 3 +-
.../statistics/MinMaxStatsCollector.java | 2 +-
.../statistics/NumericStatsCollector.java | 3 +-
.../statistics/PercentileStatsCollector.java | 3 +-
.../analytics/statistics/StatsCollector.java | 2 +-
.../statistics/UniqueStatsCollector.java | 3 +-
.../solr/analytics/util/AnalyticsParsers.java | 2 +-
.../AbsoluteValueDoubleFunction.java | 4 +-
.../util/valuesource/AddDoubleFunction.java | 4 +-
.../util/valuesource/ConcatStringFunction.java | 4 +-
.../util/valuesource/DateFieldSource.java | 38 +-
.../util/valuesource/DateMathFunction.java | 3 +-
.../util/valuesource/DivDoubleFunction.java | 4 +-
.../util/valuesource/DualDoubleFunction.java | 8 +-
.../util/valuesource/FilterFieldSource.java | 22 +-
.../util/valuesource/LogDoubleFunction.java | 4 +-
.../util/valuesource/MultiDateFunction.java | 10 +-
.../util/valuesource/MultiDoubleFunction.java | 8 +-
.../util/valuesource/MultiStringFunction.java | 12 +-
.../valuesource/MultiplyDoubleFunction.java | 4 +-
.../util/valuesource/NegateDoubleFunction.java | 4 +-
.../util/valuesource/PowDoubleFunction.java | 4 +-
.../util/valuesource/ReverseStringFunction.java | 4 +-
.../util/valuesource/SingleDoubleFunction.java | 8 +-
.../util/valuesource/SingleStringFunction.java | 14 +-
.../solr/collection1/conf/schema-analytics.xml | 119 +-
.../analytics/facet/FieldFacetExtrasTest.java | 1 -
.../solr/analytics/facet/FieldFacetTest.java | 1 -
.../SolrStopwordsCarrot2LexicalDataFactory.java | 4 +-
.../clustering/solr/collection1/conf/schema.xml | 553 +-
.../solr/collection1/conf/solrconfig.xml | 8 -
.../clustering/src/test-files/log4j.properties | 2 +-
.../conf/dataimport-schema-no-unique-key.xml | 336 +-
.../dataimport/TestTikaEntityProcessor.java | 2 -
.../handler/dataimport/DIHCacheSupport.java | 3 -
.../handler/dataimport/DataImportHandler.java | 6 -
.../dataimport/DataImportHandlerException.java | 4 +-
.../solr/handler/dataimport/DataImporter.java | 7 +-
.../handler/dataimport/DateFormatEvaluator.java | 45 +-
.../solr/handler/dataimport/DebugInfo.java | 1 -
.../solr/handler/dataimport/Evaluator.java | 5 -
.../dataimport/FileListEntityProcessor.java | 18 +-
.../solr/handler/dataimport/JdbcDataSource.java | 154 +-
.../dataimport/SimplePropertiesWriter.java | 33 +-
.../handler/dataimport/TemplateTransformer.java | 30 +-
.../solr/handler/dataimport/UrlEvaluator.java | 1 -
.../handler/dataimport/VariableResolver.java | 66 +-
.../dataimport/XPathEntityProcessor.java | 1 -
.../handler/dataimport/ZKPropertiesWriter.java | 20 +-
.../apache/solr/handler/dataimport/Zipper.java | 1 -
.../dataimport/config/ConfigParseUtil.java | 6 -
.../solr/collection1/conf/dataimport-schema.xml | 112 +-
.../conf/dataimport-solr_id-schema.xml | 512 +-
.../src/test-files/log4j.properties | 2 +-
.../solr/handler/dataimport/TestDataConfig.java | 1 -
.../solr/handler/dataimport/TestDocBuilder.java | 1 -
.../handler/dataimport/TestDocBuilder2.java | 2 +-
.../handler/dataimport/TestEphemeralCache.java | 1 -
.../dataimport/TestFileListEntityProcessor.java | 22 +-
.../TestFileListWithLineEntityProcessor.java | 1 -
.../dataimport/TestHierarchicalDocBuilder.java | 1 -
.../handler/dataimport/TestJdbcDataSource.java | 345 +-
.../dataimport/TestNonWritablePersistFile.java | 1 -
.../TestPlainTextEntityProcessor.java | 1 -
.../dataimport/TestScriptTransformer.java | 2 -
.../dataimport/TestSimplePropertiesWriter.java | 2 -
.../dataimport/TestSqlEntityProcessorDelta.java | 1 -
.../dataimport/TestTemplateTransformer.java | 38 +
.../dataimport/TestVariableResolver.java | 21 +-
solr/contrib/extraction/ivy.xml | 3 +
.../extraction/ExtractingDocumentLoader.java | 4 +-
.../handler/extraction/XLSXResponseWriter.java | 414 ++
.../extraction/solr/collection1/conf/schema.xml | 827 +--
.../ExtractingRequestHandlerTest.java | 2 -
.../extraction/TestXLSXResponseWriter.java | 257 +
...anguageIdentifierUpdateProcessorFactory.java | 1 -
.../LanguageIdentifierUpdateProcessor.java | 1 -
.../langid/solr/collection1/conf/schema.xml | 91 +-
solr/contrib/map-reduce/ivy.xml | 4 +-
.../solr/hadoop/MapReduceIndexerTool.java | 3 -
.../apache/solr/hadoop/SolrOutputFormat.java | 1 -
.../apache/solr/hadoop/SolrRecordWriter.java | 7 +-
.../solr/hadoop/TreeMergeOutputFormat.java | 15 +-
.../apache/solr/hadoop/ZooKeeperInspector.java | 21 +-
.../solr/hadoop/MorphlineBasicMiniMRTest.java | 2 +
.../solr/hadoop/MorphlineGoLiveMiniMRTest.java | 2 +
.../apache/solr/hadoop/MorphlineMapperTest.java | 2 +-
.../morphlines/cell/SolrCellMorphlineTest.java | 3 +-
solr/contrib/morphlines-core/ivy.xml | 5 +-
.../solr/collection1/conf/elevate.xml | 24 +-
.../test-files/solr/collection1/conf/schema.xml | 1649 +++---
.../solr/collection1/conf/solrconfig.xml | 65 -
.../src/test-files/solr/minimr/conf/elevate.xml | 24 +-
.../src/test-files/solr/minimr/conf/schema.xml | 1675 +++---
.../test-files/solr/minimr/conf/solrconfig.xml | 65 -
.../src/test-files/solr/mrunit/conf/elevate.xml | 24 +-
.../src/test-files/solr/mrunit/conf/schema.xml | 1674 +++---
.../test-files/solr/mrunit/conf/solrconfig.xml | 65 -
.../solrcelltest/collection1/conf/elevate.xml | 24 +-
.../solrcelltest/collection1/conf/schema.xml | 1576 ++---
.../collection1/conf/solrconfig.xml | 65 -
.../solr/solrcloud/conf/solrconfig.xml | 66 -
.../solr/AbstractSolrMorphlineZkTestBase.java | 114 +-
.../solr/SolrMorphlineZkAliasTest.java | 51 +-
.../solr/SolrMorphlineZkAvroTest.java | 48 +-
.../morphlines/solr/SolrMorphlineZkTest.java | 17 +-
.../uima/solr/collection1/conf/schema.xml | 6 +-
.../uima/solr/collection1/conf/solrconfig.xml | 55 -
.../test-files/uima/uima-tokenizers-schema.xml | 1098 ++--
.../uima/uima-tokenizers-solrconfig.xml | 53 -
.../velocity/solr/collection1/conf/schema.xml | 17 +-
solr/core/build.xml | 2 +-
solr/core/ivy.xml | 13 +-
.../apache/solr/analysis/TokenizerChain.java | 28 +-
.../solrj/embedded/EmbeddedSolrServer.java | 6 +-
.../client/solrj/embedded/JettySolrRunner.java | 18 +-
.../solr/client/solrj/embedded/SSLConfig.java | 2 +-
.../org/apache/solr/cloud/AddReplicaCmd.java | 192 +
.../src/java/org/apache/solr/cloud/Assign.java | 3 +-
.../java/org/apache/solr/cloud/BackupCmd.java | 132 +
.../org/apache/solr/cloud/CreateAliasCmd.java | 101 +
.../apache/solr/cloud/CreateCollectionCmd.java | 291 +
.../org/apache/solr/cloud/CreateShardCmd.java | 120 +
.../org/apache/solr/cloud/DeleteAliasCmd.java | 95 +
.../apache/solr/cloud/DeleteCollectionCmd.java | 121 +
.../org/apache/solr/cloud/DeleteNodeCmd.java | 98 +
.../org/apache/solr/cloud/DeleteReplicaCmd.java | 281 +
.../org/apache/solr/cloud/DeleteShardCmd.java | 180 +
.../org/apache/solr/cloud/DistributedMap.java | 1 -
.../org/apache/solr/cloud/DistributedQueue.java | 95 +-
.../org/apache/solr/cloud/ElectionContext.java | 117 +-
.../solr/cloud/ExclusiveSliceProperty.java | 345 ++
.../org/apache/solr/cloud/LeaderElector.java | 14 +-
.../java/org/apache/solr/cloud/LockTree.java | 182 +
.../java/org/apache/solr/cloud/MigrateCmd.java | 333 ++
.../java/org/apache/solr/cloud/Overseer.java | 349 +-
.../OverseerAutoReplicaFailoverThread.java | 39 +-
.../OverseerCollectionConfigSetProcessor.java | 11 +-
.../cloud/OverseerCollectionMessageHandler.java | 2044 +------
.../cloud/OverseerConfigSetMessageHandler.java | 35 +-
.../solr/cloud/OverseerMessageHandler.java | 33 +-
.../org/apache/solr/cloud/OverseerRoleCmd.java | 102 +
.../apache/solr/cloud/OverseerStatusCmd.java | 122 +
.../solr/cloud/OverseerTaskProcessor.java | 176 +-
.../apache/solr/cloud/OverseerTaskQueue.java | 54 +-
.../org/apache/solr/cloud/RecoveryStrategy.java | 6 +-
.../org/apache/solr/cloud/ReplaceNodeCmd.java | 166 +
.../java/org/apache/solr/cloud/RestoreCmd.java | 243 +
.../solr/cloud/SizeLimitedDistributedMap.java | 2 -
.../org/apache/solr/cloud/SplitShardCmd.java | 508 ++
.../org/apache/solr/cloud/SyncStrategy.java | 34 +-
.../src/java/org/apache/solr/cloud/ZkCLI.java | 51 +-
.../org/apache/solr/cloud/ZkController.java | 335 +-
.../apache/solr/cloud/ZkSolrResourceLoader.java | 16 +-
.../cloud/overseer/ClusterStateMutator.java | 2 +-
.../solr/cloud/overseer/CollectionMutator.java | 19 +-
.../apache/solr/cloud/overseer/NodeMutator.java | 5 +-
.../solr/cloud/overseer/OverseerAction.java | 2 -
.../solr/cloud/overseer/ReplicaMutator.java | 79 +-
.../solr/cloud/overseer/SliceMutator.java | 8 +-
.../solr/cloud/overseer/ZkStateWriter.java | 8 +-
.../apache/solr/cloud/rule/ImplicitSnitch.java | 134 +-
.../apache/solr/cloud/rule/RemoteCallback.java | 23 -
.../apache/solr/cloud/rule/ReplicaAssigner.java | 22 +-
.../java/org/apache/solr/cloud/rule/Rule.java | 12 +-
.../solr/cloud/rule/ServerSnitchContext.java | 108 +
.../java/org/apache/solr/cloud/rule/Snitch.java | 2 +-
.../apache/solr/cloud/rule/SnitchContext.java | 137 -
.../org/apache/solr/core/BlobRepository.java | 6 -
.../solr/core/CachingDirectoryFactory.java | 24 +-
.../src/java/org/apache/solr/core/Config.java | 4 +-
.../org/apache/solr/core/ConfigOverlay.java | 14 +-
.../apache/solr/core/ConfigSetProperties.java | 4 +-
.../org/apache/solr/core/ConfigSetService.java | 1 -
.../org/apache/solr/core/CoreContainer.java | 152 +-
.../org/apache/solr/core/CoreDescriptor.java | 2 +-
.../apache/solr/core/CorePropertiesLocator.java | 12 +-
.../java/org/apache/solr/core/CoreSorter.java | 185 +
.../org/apache/solr/core/DirectoryFactory.java | 27 +-
.../apache/solr/core/HdfsDirectoryFactory.java | 46 +-
.../solr/core/IndexDeletionPolicyWrapper.java | 27 +-
.../java/org/apache/solr/core/InitParams.java | 4 +-
.../org/apache/solr/core/JmxMonitoredMap.java | 20 +-
.../org/apache/solr/core/MapSerializable.java | 23 -
.../java/org/apache/solr/core/NodeConfig.java | 38 +-
.../java/org/apache/solr/core/PluginBag.java | 9 +-
.../java/org/apache/solr/core/PluginInfo.java | 31 +-
.../apache/solr/core/QuerySenderListener.java | 10 +-
.../org/apache/solr/core/RequestHandlers.java | 2 +-
.../org/apache/solr/core/RequestParams.java | 15 +-
.../apache/solr/core/SchemaCodecFactory.java | 10 +-
.../solr/core/SimpleFSDirectoryFactory.java | 1 -
.../java/org/apache/solr/core/SolrConfig.java | 116 +-
.../src/java/org/apache/solr/core/SolrCore.java | 275 +-
.../java/org/apache/solr/core/SolrCores.java | 13 -
.../apache/solr/core/SolrDeletionPolicy.java | 20 +-
.../apache/solr/core/SolrResourceLoader.java | 54 +-
.../org/apache/solr/core/SolrXmlConfig.java | 12 +
.../java/org/apache/solr/core/ZkContainer.java | 46 +-
.../apache/solr/core/backup/BackupManager.java | 249 +
.../apache/solr/core/backup/package-info.java | 22 +
.../backup/repository/BackupRepository.java | 184 +
.../repository/BackupRepositoryFactory.java | 89 +
.../backup/repository/HdfsBackupRepository.java | 180 +
.../repository/LocalFileSystemRepository.java | 157 +
.../core/backup/repository/package-info.java | 23 +
.../java/org/apache/solr/core/package-info.java | 2 +-
.../core/snapshots/SolrSnapshotManager.java | 120 +
.../snapshots/SolrSnapshotMetaDataManager.java | 416 ++
.../solr/core/snapshots/package-info.java | 22 +
.../handler/AnalysisRequestHandlerBase.java | 4 +-
.../org/apache/solr/handler/BlobHandler.java | 2 +-
.../org/apache/solr/handler/CdcrParams.java | 10 +-
.../org/apache/solr/handler/CdcrReplicator.java | 8 +-
.../solr/handler/CdcrReplicatorManager.java | 242 +-
.../solr/handler/CdcrReplicatorScheduler.java | 6 +-
.../solr/handler/CdcrReplicatorState.java | 23 +
.../apache/solr/handler/CdcrRequestHandler.java | 233 +-
.../org/apache/solr/handler/ClassifyStream.java | 229 +
.../solr/handler/ContentStreamHandlerBase.java | 1 -
.../org/apache/solr/handler/GraphHandler.java | 282 +
.../org/apache/solr/handler/IndexFetcher.java | 50 +-
.../solr/handler/MoreLikeThisHandler.java | 7 +-
.../solr/handler/NestedRequestHandler.java | 2 +-
.../apache/solr/handler/OldBackupDirectory.java | 55 +-
.../apache/solr/handler/RealTimeGetHandler.java | 1 -
.../apache/solr/handler/ReplicationHandler.java | 114 +-
.../apache/solr/handler/RequestHandlerBase.java | 72 +-
.../org/apache/solr/handler/RestoreCore.java | 30 +-
.../org/apache/solr/handler/SQLHandler.java | 13 +-
.../org/apache/solr/handler/SchemaHandler.java | 2 +-
.../org/apache/solr/handler/SnapShooter.java | 214 +-
.../apache/solr/handler/SolrConfigHandler.java | 150 +-
.../solr/handler/StandardRequestHandler.java | 2 +-
.../org/apache/solr/handler/StreamHandler.java | 76 +-
.../apache/solr/handler/admin/BackupCoreOp.java | 84 +
.../solr/handler/admin/ClusterStatus.java | 2 +-
.../solr/handler/admin/CollectionsHandler.java | 867 ++-
.../solr/handler/admin/ConfigSetsHandler.java | 1 -
.../solr/handler/admin/CoreAdminHandler.java | 11 +-
.../solr/handler/admin/CoreAdminOperation.java | 1022 +---
.../solr/handler/admin/CreateSnapshotOp.java | 62 +
.../solr/handler/admin/DeleteSnapshotOp.java | 47 +
.../org/apache/solr/handler/admin/InvokeOp.java | 58 +
.../solr/handler/admin/LukeRequestHandler.java | 3 +-
.../solr/handler/admin/MergeIndexesOp.java | 142 +
.../solr/handler/admin/PrepRecoveryOp.java | 217 +
.../handler/admin/RequestApplyUpdatesOp.java | 71 +
.../solr/handler/admin/RequestSyncShardOp.java | 100 +
.../solr/handler/admin/RestoreCoreOp.java | 71 +
.../handler/admin/ShowFileRequestHandler.java | 2 -
.../handler/admin/SolrInfoMBeanHandler.java | 1 -
.../org/apache/solr/handler/admin/SplitOp.java | 142 +
.../org/apache/solr/handler/admin/StatusOp.java | 64 +
.../solr/handler/component/DebugComponent.java | 5 +
.../solr/handler/component/ExpandComponent.java | 100 +-
.../solr/handler/component/FacetComponent.java | 88 +-
.../solr/handler/component/FieldFacetStats.java | 24 +-
.../handler/component/HighlightComponent.java | 1 -
.../component/HttpShardHandlerFactory.java | 2 +-
.../component/MoreLikeThisComponent.java | 19 +-
.../solr/handler/component/QueryComponent.java | 15 +-
.../component/QueryElevationComponent.java | 1 -
.../handler/component/RealTimeGetComponent.java | 373 +-
.../solr/handler/component/ResponseBuilder.java | 4 +-
.../solr/handler/component/SearchComponent.java | 2 +
.../solr/handler/component/SearchHandler.java | 2 +
.../handler/component/SpellCheckComponent.java | 7 +-
.../handler/component/SpellCheckMergeData.java | 1 -
.../solr/handler/component/StatsField.java | 19 +-
.../solr/handler/component/StatsValues.java | 2 +-
.../handler/component/StatsValuesFactory.java | 15 +-
.../solr/handler/component/TermsComponent.java | 125 +-
.../apache/solr/handler/loader/JsonLoader.java | 40 +-
.../apache/solr/handler/loader/XMLLoader.java | 2 +-
.../solr/highlight/DefaultSolrHighlighter.java | 112 +-
.../solr/highlight/PostingsSolrHighlighter.java | 1 -
.../apache/solr/highlight/SolrHighlighter.java | 2 +-
.../org/apache/solr/highlight/package-info.java | 2 +-
.../solr/index/SlowCompositeReaderWrapper.java | 279 +
.../apache/solr/index/SortingMergePolicy.java | 44 +
.../solr/index/SortingMergePolicyFactory.java | 1 -
.../org/apache/solr/internal/csv/CSVParser.java | 4 +-
.../apache/solr/internal/csv/CSVPrinter.java | 2 -
.../internal/csv/ExtendedBufferedReader.java | 2 +-
.../org/apache/solr/logging/LogWatcher.java | 8 +-
.../org/apache/solr/logging/LoggerInfo.java | 2 +-
.../org/apache/solr/parser/QueryParser.java | 6 -
.../solr/parser/QueryParserTokenManager.java | 13 -
.../apache/solr/parser/SolrQueryParserBase.java | 1 -
.../java/org/apache/solr/query/FilterQuery.java | 6 +-
.../org/apache/solr/query/SolrRangeQuery.java | 20 +-
.../apache/solr/request/DocValuesFacets.java | 86 +-
.../org/apache/solr/request/DocValuesStats.java | 52 +-
.../org/apache/solr/request/IntervalFacets.java | 62 +-
.../org/apache/solr/request/NumericFacets.java | 40 +-
.../request/PerSegmentSingleValuedFaceting.java | 40 +-
.../org/apache/solr/request/SimpleFacets.java | 248 +-
.../apache/solr/request/SolrRequestInfo.java | 7 +-
.../org/apache/solr/request/json/JSONUtil.java | 6 -
.../apache/solr/request/json/ObjectUtil.java | 3 -
.../solr/request/macro/MacroExpander.java | 13 +-
.../response/BinaryQueryResponseWriter.java | 2 +-
.../solr/response/BinaryResponseWriter.java | 6 +-
.../solr/response/GeoJSONResponseWriter.java | 1 -
.../solr/response/GraphMLResponseWriter.java | 167 +
.../solr/response/JSONResponseWriter.java | 30 +-
.../response/PHPSerializedResponseWriter.java | 1 -
.../solr/response/QueryResponseWriter.java | 2 +-
.../solr/response/QueryResponseWriterUtil.java | 2 +-
.../org/apache/solr/response/ResultContext.java | 5 +-
.../solr/response/SortingResponseWriter.java | 277 +-
.../solr/response/TextResponseWriter.java | 8 +-
.../org/apache/solr/response/package-info.java | 2 +-
.../transform/ChildDocTransformerFactory.java | 4 +-
.../transform/DocIdAugmenterFactory.java | 10 +-
.../solr/response/transform/DocTransformer.java | 26 +-
.../response/transform/DocTransformers.java | 12 +
.../transform/ExcludedMarkerFactory.java | 2 -
.../transform/ExplainAugmenterFactory.java | 22 +-
.../transform/SubQueryAugmenterFactory.java | 376 ++
.../response/transform/TransformerFactory.java | 3 +-
.../transform/ValueSourceAugmenter.java | 21 +-
.../solr/rest/ManagedResourceStorage.java | 19 +-
.../java/org/apache/solr/rest/RestManager.java | 12 +-
.../org/apache/solr/rest/SolrSchemaRestApi.java | 2 -
.../analysis/ManagedStopFilterFactory.java | 4 +-
.../schema/analysis/ManagedWordSetResource.java | 1 -
.../solr/schema/AbstractSpatialFieldType.java | 97 +-
.../java/org/apache/solr/schema/BBoxField.java | 7 +-
.../org/apache/solr/schema/BinaryField.java | 2 +-
.../java/org/apache/solr/schema/BoolField.java | 75 +-
.../org/apache/solr/schema/CollationField.java | 4 +-
.../org/apache/solr/schema/CurrencyField.java | 44 +-
.../org/apache/solr/schema/DateRangeField.java | 40 +-
.../java/org/apache/solr/schema/EnumField.java | 35 +-
.../solr/schema/ExchangeRateProvider.java | 2 +-
.../apache/solr/schema/ExternalFileField.java | 8 +-
.../java/org/apache/solr/schema/FieldType.java | 43 +-
.../org/apache/solr/schema/GeoHashField.java | 10 +-
.../org/apache/solr/schema/IndexSchema.java | 50 +-
.../solr/schema/JsonPreAnalyzedParser.java | 1 -
.../java/org/apache/solr/schema/LatLonType.java | 34 +-
.../solr/schema/ManagedIndexSchemaFactory.java | 3 +-
.../schema/OpenExchangeRatesOrgProvider.java | 4 +-
.../java/org/apache/solr/schema/PointType.java | 12 +-
.../apache/solr/schema/PreAnalyzedField.java | 8 +-
.../org/apache/solr/schema/RandomSortField.java | 4 +-
.../schema/RptWithGeometrySpatialField.java | 21 +-
.../org/apache/solr/schema/SchemaField.java | 63 +-
.../org/apache/solr/schema/SchemaManager.java | 71 +-
.../schema/SpatialPointVectorFieldType.java | 9 +-
.../java/org/apache/solr/schema/StrField.java | 2 +-
.../org/apache/solr/schema/StrFieldSource.java | 8 +-
.../java/org/apache/solr/schema/TextField.java | 14 +-
.../org/apache/solr/schema/TrieDoubleField.java | 51 +-
.../java/org/apache/solr/schema/TrieField.java | 51 +-
.../org/apache/solr/schema/TrieFloatField.java | 51 +-
.../org/apache/solr/schema/TrieIntField.java | 52 +-
.../org/apache/solr/schema/TrieLongField.java | 52 +-
.../java/org/apache/solr/schema/UUIDField.java | 2 +-
.../apache/solr/schema/ZkIndexSchemaReader.java | 10 +-
.../apache/solr/search/AbstractReRankQuery.java | 83 +
.../java/org/apache/solr/search/BitDocSet.java | 16 +-
.../org/apache/solr/search/CacheConfig.java | 28 +-
.../solr/search/CollapsingQParserPlugin.java | 301 +-
.../java/org/apache/solr/search/CursorMark.java | 2 +-
.../org/apache/solr/search/DisMaxQParser.java | 1 -
.../apache/solr/search/DisMaxQParserPlugin.java | 1 -
.../src/java/org/apache/solr/search/DocSet.java | 2 +-
.../java/org/apache/solr/search/DocSetBase.java | 19 +-
.../java/org/apache/solr/search/DocSetUtil.java | 33 +
.../java/org/apache/solr/search/DocSlice.java | 1 -
.../apache/solr/search/ExportQParserPlugin.java | 29 +-
.../solr/search/ExtendedDismaxQParser.java | 53 +-
.../search/ExtendedDismaxQParserPlugin.java | 1 -
.../apache/solr/search/ExtendedQueryBase.java | 2 +-
.../apache/solr/search/FieldQParserPlugin.java | 1 -
.../src/java/org/apache/solr/search/Filter.java | 10 +-
.../solr/search/FunctionQParserPlugin.java | 1 -
.../solr/search/FunctionRangeQParserPlugin.java | 1 -
.../solr/search/GraphTermsQParserPlugin.java | 21 +-
.../java/org/apache/solr/search/Grouping.java | 2 +-
.../apache/solr/search/HashQParserPlugin.java | 71 +-
.../solr/search/IGainTermsQParserPlugin.java | 251 +
.../java/org/apache/solr/search/Insanity.java | 6 +-
.../apache/solr/search/JoinQParserPlugin.java | 45 +-
.../java/org/apache/solr/search/LFUCache.java | 10 +-
.../java/org/apache/solr/search/LRUCache.java | 35 +-
.../search/LegacyNumericRangeQueryBuilder.java | 136 +
.../apache/solr/search/LuceneQParserPlugin.java | 10 +-
.../solr/search/OldLuceneQParserPlugin.java | 1 -
.../apache/solr/search/PrefixQParserPlugin.java | 3 -
.../java/org/apache/solr/search/QParser.java | 22 +-
.../org/apache/solr/search/QParserPlugin.java | 12 +-
.../org/apache/solr/search/QueryContext.java | 1 -
.../org/apache/solr/search/QueryParsing.java | 10 +-
.../org/apache/solr/search/QueryResultKey.java | 8 +-
.../java/org/apache/solr/search/QueryUtils.java | 2 -
.../apache/solr/search/QueryWrapperFilter.java | 14 +-
.../apache/solr/search/RawQParserPlugin.java | 1 -
.../org/apache/solr/search/ReRankCollector.java | 175 +
.../apache/solr/search/ReRankQParserPlugin.java | 361 +-
.../org/apache/solr/search/ReRankWeight.java | 48 +
.../org/apache/solr/search/ReturnFields.java | 7 +
.../apache/solr/search/SimpleQParserPlugin.java | 3 -
.../solr/search/SolrConstantScoreQuery.java | 19 +-
.../org/apache/solr/search/SolrCoreParser.java | 42 +-
.../apache/solr/search/SolrFieldCacheMBean.java | 12 +-
.../apache/solr/search/SolrIndexSearcher.java | 226 +-
.../apache/solr/search/SolrQueryBuilder.java | 34 +
.../org/apache/solr/search/SolrQueryParser.java | 18 -
.../apache/solr/search/SolrReturnFields.java | 14 +-
.../org/apache/solr/search/SortedIntDocSet.java | 16 +-
.../solr/search/SpatialBoxQParserPlugin.java | 1 -
.../solr/search/SpatialFilterQParserPlugin.java | 1 -
.../solr/search/SurroundQParserPlugin.java | 1 -
.../apache/solr/search/TermQParserPlugin.java | 1 -
.../apache/solr/search/TermsQParserPlugin.java | 2 -
.../TextLogisticRegressionQParserPlugin.java | 292 +
.../apache/solr/search/ValueSourceParser.java | 154 +-
.../org/apache/solr/search/WrappedQuery.java | 4 +-
.../apache/solr/search/XmlQParserPlugin.java | 10 +
.../solr/search/facet/AggValueSource.java | 1 -
.../org/apache/solr/search/facet/BlockJoin.java | 1 -
.../apache/solr/search/facet/FacetBucket.java | 189 +
.../solr/search/facet/FacetDebugInfo.java | 6 +-
.../apache/solr/search/facet/FacetField.java | 1063 +---
.../solr/search/facet/FacetFieldMerger.java | 211 +
.../solr/search/facet/FacetFieldProcessor.java | 501 ++
.../facet/FacetFieldProcessorByArray.java | 95 +
.../facet/FacetFieldProcessorByArrayDV.java | 322 +
.../facet/FacetFieldProcessorByArrayUIF.java | 71 +
.../FacetFieldProcessorByEnumTermsStream.java | 356 ++
.../facet/FacetFieldProcessorByHashDV.java | 449 ++
.../search/facet/FacetFieldProcessorDV.java | 291 -
.../facet/FacetFieldProcessorNumeric.java | 444 --
.../apache/solr/search/facet/FacetMerger.java | 126 +-
.../apache/solr/search/facet/FacetModule.java | 582 +-
.../solr/search/facet/FacetProcessor.java | 203 +-
.../apache/solr/search/facet/FacetQuery.java | 5 -
.../apache/solr/search/facet/FacetRange.java | 13 +-
.../solr/search/facet/FacetRangeMerger.java | 123 +
.../apache/solr/search/facet/FacetRequest.java | 73 +-
.../search/facet/FacetRequestSortedMerger.java | 247 +
.../org/apache/solr/search/facet/FieldUtil.java | 6 +-
.../org/apache/solr/search/facet/HLLAgg.java | 20 +-
.../apache/solr/search/facet/LegacyFacet.java | 1 -
.../apache/solr/search/facet/PercentileAgg.java | 4 +-
.../solr/search/facet/SimpleAggValueSource.java | 3 -
.../org/apache/solr/search/facet/SlotAcc.java | 35 +-
.../solr/search/facet/StrAggValueSource.java | 7 -
.../solr/search/facet/UnInvertedField.java | 15 +-
.../org/apache/solr/search/facet/UniqueAgg.java | 14 +-
.../solr/search/facet/UniqueMultiDvSlotAcc.java | 91 +
.../search/facet/UniqueMultivaluedSlotAcc.java | 69 +
.../search/facet/UniqueSinglevaluedSlotAcc.java | 85 +
.../apache/solr/search/facet/UniqueSlotAcc.java | 165 -
.../solr/search/function/FileFloatSource.java | 3 +-
.../solr/search/function/OrdFieldSource.java | 41 +-
.../search/function/ReverseOrdFieldSource.java | 21 +-
.../function/SolrComparisonBoolFunction.java | 60 +
.../function/distance/GeohashFunction.java | 4 +-
.../distance/GeohashHaversineFunction.java | 6 +-
.../distance/HaversineConstFunction.java | 4 +-
.../function/distance/HaversineFunction.java | 6 +-
.../distance/SquaredEuclideanFunction.java | 4 +-
.../distance/StringDistanceFunction.java | 6 +-
.../distance/VectorDistanceFunction.java | 6 +-
.../apache/solr/search/grouping/Command.java | 4 +-
.../distributed/command/QueryCommand.java | 2 +-
.../command/SearchGroupsFieldCommand.java | 2 +-
.../TopGroupsShardRequestFactory.java | 2 +-
.../TopGroupsShardResponseProcessor.java | 2 +-
.../SearchGroupsResultTransformer.java | 2 -
.../TopGroupsResultTransformer.java | 2 +-
.../join/BlockJoinDocSetFacetComponent.java | 34 +-
.../search/join/BlockJoinFacetAccsHolder.java | 97 +
.../search/join/BlockJoinFacetCollector.java | 131 -
.../search/join/BlockJoinFacetComponent.java | 165 +-
.../join/BlockJoinFacetComponentSupport.java | 156 +
.../solr/search/join/BlockJoinFacetFilter.java | 17 +
.../join/BlockJoinFieldFacetAccumulator.java | 37 +-
.../search/join/BlockJoinParentQParser.java | 11 +-
.../join/BlockJoinParentQParserPlugin.java | 1 -
.../solr/search/join/GraphQParserPlugin.java | 1 -
.../org/apache/solr/search/join/GraphQuery.java | 89 +-
.../solr/search/join/GraphQueryParser.java | 1 -
.../solr/search/join/GraphTermsCollector.java | 20 +-
.../search/join/ScoreJoinQParserPlugin.java | 45 +-
.../apache/solr/search/mlt/CloudMLTQParser.java | 23 +-
.../solr/search/mlt/MLTQParserPlugin.java | 1 -
.../solr/search/mlt/SimpleMLTQParser.java | 4 +-
.../similarities/SchemaSimilarityFactory.java | 43 +-
.../SweetSpotSimilarityFactory.java | 1 -
.../apache/solr/search/stats/LRUStatsCache.java | 5 -
.../solr/security/AuthenticationPlugin.java | 32 +-
.../solr/security/AuthorizationContext.java | 1 -
.../security/AutorizationEditOperation.java | 31 +-
.../apache/solr/security/BasicAuthPlugin.java | 6 +-
.../security/DelegationTokenKerberosFilter.java | 215 +
.../apache/solr/security/KerberosFilter.java | 14 +
.../apache/solr/security/KerberosPlugin.java | 289 +-
.../solr/security/PKIAuthenticationPlugin.java | 33 +-
.../org/apache/solr/security/Permission.java | 46 +-
.../solr/security/PermissionNameProvider.java | 4 +-
.../solr/security/PrintWriterWrapper.java | 215 +
.../security/RuleBasedAuthorizationPlugin.java | 2 -
.../solr/servlet/DirectSolrConnection.java | 6 -
.../org/apache/solr/servlet/HttpSolrCall.java | 8 +-
.../apache/solr/servlet/SolrDispatchFilter.java | 70 +-
.../solr/servlet/StartupLoggingUtils.java | 116 +
.../solr/servlet/cache/HttpCacheHeaderUtil.java | 2 -
.../spelling/AbstractLuceneSpellChecker.java | 6 -
.../solr/spelling/DirectSolrSpellChecker.java | 6 -
.../solr/spelling/IndexBasedSpellChecker.java | 2 +-
.../solr/spelling/SpellCheckCollator.java | 3 +
.../solr/spelling/suggest/SolrSuggester.java | 20 +-
.../solr/store/blockcache/BlockCache.java | 1 -
.../solr/store/blockcache/BlockDirectory.java | 5 +-
.../store/blockcache/BlockDirectoryCache.java | 5 +-
.../solr/store/blockcache/BufferStore.java | 2 +-
.../blockcache/CustomBufferedIndexInput.java | 2 +-
.../apache/solr/store/blockcache/Metrics.java | 1 -
.../apache/solr/store/hdfs/HdfsDirectory.java | 33 +-
.../apache/solr/store/hdfs/HdfsFileReader.java | 105 -
.../apache/solr/uninverting/DocTermOrds.java | 889 +++
.../org/apache/solr/uninverting/FieldCache.java | 461 ++
.../apache/solr/uninverting/FieldCacheImpl.java | 1246 ++++
.../uninverting/FieldCacheSanityChecker.java | 425 ++
.../solr/uninverting/UninvertingReader.java | 389 ++
.../apache/solr/uninverting/package-info.java | 21 +
.../apache/solr/update/AddUpdateCommand.java | 1 -
.../org/apache/solr/update/CdcrUpdateLog.java | 9 +-
.../solr/update/DefaultSolrCoreState.java | 25 +-
.../solr/update/DeleteByQueryWrapper.java | 40 +-
.../apache/solr/update/DeleteUpdateCommand.java | 2 -
.../solr/update/DirectUpdateHandler2.java | 139 +-
.../apache/solr/update/HdfsTransactionLog.java | 141 +-
.../org/apache/solr/update/HdfsUpdateLog.java | 15 +-
.../apache/solr/update/IndexFingerprint.java | 54 +-
.../java/org/apache/solr/update/PeerSync.java | 256 +-
.../org/apache/solr/update/SolrCoreState.java | 4 +-
.../org/apache/solr/update/SolrIndexConfig.java | 29 +-
.../apache/solr/update/SolrIndexSplitter.java | 11 +-
.../org/apache/solr/update/SolrIndexWriter.java | 30 +-
.../org/apache/solr/update/TransactionLog.java | 7 +-
.../java/org/apache/solr/update/UpdateLog.java | 13 +-
.../apache/solr/update/UpdateShardHandler.java | 17 +-
.../org/apache/solr/update/VersionInfo.java | 10 +-
...tractDefaultValueUpdateProcessorFactory.java | 2 -
.../AddSchemaFieldsUpdateProcessorFactory.java | 35 +-
...aluesOrNoneFieldMutatingUpdateProcessor.java | 2 +-
.../update/processor/CdcrUpdateProcessor.java | 10 +-
.../ClassificationUpdateProcessor.java | 40 +-
.../ClassificationUpdateProcessorFactory.java | 26 +-
.../ConcatFieldUpdateProcessorFactory.java | 68 +-
.../CountFieldValuesUpdateProcessorFactory.java | 25 +-
.../DefaultValueUpdateProcessorFactory.java | 4 -
.../processor/DistributedUpdateProcessor.java | 34 +-
.../DistributingUpdateProcessorFactory.java | 2 -
...BasedVersionConstraintsProcessorFactory.java | 4 +-
.../DocExpirationUpdateProcessorFactory.java | 16 +-
.../FieldLengthUpdateProcessorFactory.java | 29 +-
.../processor/FieldMutatingUpdateProcessor.java | 72 +-
.../FieldMutatingUpdateProcessorFactory.java | 17 +-
.../FieldValueMutatingUpdateProcessor.java | 12 +
.../FieldValueSubsetUpdateProcessorFactory.java | 23 +-
.../FirstFieldValueUpdateProcessorFactory.java | 14 +-
.../HTMLStripFieldUpdateProcessorFactory.java | 51 +-
.../IgnoreFieldUpdateProcessorFactory.java | 33 +-
.../LastFieldValueUpdateProcessorFactory.java | 10 +-
.../solr/update/processor/MD5Signature.java | 1 -
.../MaxFieldValueUpdateProcessorFactory.java | 16 +-
.../MinFieldValueUpdateProcessorFactory.java | 16 +-
.../NoOpDistributingUpdateProcessorFactory.java | 2 -
...ParseBooleanFieldUpdateProcessorFactory.java | 28 +-
.../ParseDateFieldUpdateProcessorFactory.java | 27 +-
...ParseNumericFieldUpdateProcessorFactory.java | 16 +-
.../processor/RegexReplaceProcessorFactory.java | 25 +-
.../update/processor/RegexpBoostProcessor.java | 1 -
.../RemoveBlankFieldUpdateProcessorFactory.java | 18 +-
.../processor/RunUpdateProcessorFactory.java | 3 +-
.../processor/SimpleUpdateProcessorFactory.java | 46 +-
.../TemplateUpdateProcessorFactory.java | 111 +
.../TimestampUpdateProcessorFactory.java | 7 -
.../processor/TolerantUpdateProcessor.java | 22 +-
.../TolerantUpdateProcessorFactory.java | 1 -
.../TrimFieldUpdateProcessorFactory.java | 17 +-
.../TruncateFieldUpdateProcessorFactory.java | 28 +-
.../UniqFieldsUpdateProcessorFactory.java | 16 +-
.../processor/UpdateRequestProcessorChain.java | 61 +-
.../org/apache/solr/util/CommandOperation.java | 3 -
.../apache/solr/util/ConcurrentLFUCache.java | 15 +-
.../apache/solr/util/ConcurrentLRUCache.java | 40 +-
.../org/apache/solr/util/DateMathParser.java | 168 +-
.../org/apache/solr/util/PivotListEntry.java | 4 +-
.../apache/solr/util/RecordingJSONParser.java | 80 +-
.../org/apache/solr/util/SimplePostTool.java | 6 +-
.../src/java/org/apache/solr/util/SolrCLI.java | 927 ++-
.../org/apache/solr/util/SolrPluginUtils.java | 36 +-
.../org/apache/solr/util/TestInjection.java | 143 +-
.../org/apache/solr/util/VersionedFile.java | 4 +-
.../solr/util/xslt/TransformerProvider.java | 1 -
.../resources/EditableSolrConfigAttributes.json | 5 +-
solr/core/src/resources/ImplicitPlugins.json | 74 +-
.../resources/SystemCollectionSolrConfig.xml | 1 -
.../solr/analysisconfs/analysis-err-schema.xml | 31 +-
.../bad-schema-analyzer-class-and-nested.xml | 34 +-
.../bad-schema-bogus-analysis-parameters.xml | 16 +-
.../conf/bad-schema-bogus-field-parameters.xml | 8 +-
.../bad-schema-codec-global-vs-ft-mismatch.xml | 22 +-
.../bad-schema-currency-dynamic-multivalued.xml | 21 +-
...bad-schema-currency-ft-bogus-code-in-xml.xml | 26 +-
...ad-schema-currency-ft-bogus-default-code.xml | 26 +-
.../conf/bad-schema-currency-ft-multivalued.xml | 18 +-
.../conf/bad-schema-currency-ft-oer-norates.xml | 24 +-
.../conf/bad-schema-currency-multivalued.xml | 20 +-
.../conf/bad-schema-dup-dynamicField.xml | 27 +-
.../collection1/conf/bad-schema-dup-field.xml | 28 +-
.../conf/bad-schema-dup-fieldType.xml | 40 +-
.../bad-schema-dynamicfield-default-val.xml | 18 +-
.../conf/bad-schema-dynamicfield-required.xml | 18 +-
.../conf/bad-schema-external-filefield.xml | 10 +-
.../collection1/conf/bad-schema-init-error.xml | 8 +-
...asterisk-copyfield-dest-should-fail-test.xml | 8 +-
...terisk-copyfield-source-should-fail-test.xml | 8 +-
...asterisk-copyfield-dest-should-fail-test.xml | 8 +-
...terisk-copyfield-source-should-fail-test.xml | 8 +-
...source-matching-nothing-should-fail-test.xml | 13 +-
.../conf/bad-schema-nontext-analyzer.xml | 25 +-
.../conf/bad-schema-not-indexed-but-norms.xml | 24 +-
.../conf/bad-schema-not-indexed-but-pos.xml | 24 +-
.../conf/bad-schema-not-indexed-but-tf.xml | 23 +-
.../conf/bad-schema-omit-tf-but-not-pos.xml | 25 +-
.../bad-schema-sim-global-vs-ft-mismatch.xml | 32 +-
.../conf/bad-schema-sweetspot-both-tf.xml | 50 +-
.../bad-schema-sweetspot-partial-baseline.xml | 32 +-
.../bad-schema-sweetspot-partial-hyperbolic.xml | 46 +-
.../conf/bad-schema-sweetspot-partial-norms.xml | 44 +-
.../bad-schema-uniquekey-is-copyfield-dest.xml | 20 +-
.../conf/bad-schema-uniquekey-multivalued.xml | 16 +-
.../conf/bad-schema-uniquekey-uses-default.xml | 16 +-
.../conf/bad-schema-unsupported-docValues.xml | 10 +-
.../conf/schema-HighlighterMaxOffsetTest.xml | 122 +-
...chema-add-schema-fields-update-processor.xml | 22 +-
.../solr/collection1/conf/schema-behavior.xml | 218 +-
.../collection1/conf/schema-binaryfield.xml | 78 +-
.../conf/schema-blockjoinfacetcomponent.xml | 41 +-
.../solr/collection1/conf/schema-bm25.xml | 48 +-
.../collection1/conf/schema-charfilters.xml | 52 +-
...a-class-name-shortening-on-serialization.xml | 42 +-
.../solr/collection1/conf/schema-collate-dv.xml | 56 +-
.../solr/collection1/conf/schema-collate.xml | 55 +-
.../collection1/conf/schema-complex-phrase.xml | 20 +-
.../collection1/conf/schema-copyfield-test.xml | 841 +--
.../collection1/conf/schema-custom-field.xml | 40 +-
.../collection1/conf/schema-customfield.xml | 78 +-
.../solr/collection1/conf/schema-dfi.xml | 51 +-
.../solr/collection1/conf/schema-dfr.xml | 82 +-
.../conf/schema-distrib-interval-faceting.xml | 113 +-
.../conf/schema-distributed-missing-sort.xml | 87 +-
.../solr/collection1/conf/schema-docValues.xml | 80 +-
.../conf/schema-docValuesFaceting.xml | 93 +-
.../collection1/conf/schema-docValuesJoin.xml | 135 +-
.../conf/schema-docValuesMissing.xml | 154 +-
.../collection1/conf/schema-docValuesMulti.xml | 43 +-
.../solr/collection1/conf/schema-eff.xml | 25 +-
.../solr/collection1/conf/schema-enums.xml | 38 +-
.../conf/schema-field-sort-values.xml | 32 +-
.../solr/collection1/conf/schema-folding.xml | 482 +-
.../solr/collection1/conf/schema-hash.xml | 1070 ++--
.../solr/collection1/conf/schema-ib.xml | 57 +-
.../conf/schema-id-and-version-fields-only.xml | 12 +-
.../collection1/conf/schema-lmdirichlet.xml | 43 +-
.../collection1/conf/schema-lmjelinekmercer.xml | 43 +-
.../conf/schema-luceneMatchVersion.xml | 22 +-
.../conf/schema-minimal-atomic-stress.xml | 38 +
.../solr/collection1/conf/schema-minimal.xml | 6 +-
.../conf/schema-not-required-unique-key.xml | 33 +-
.../solr/collection1/conf/schema-numeric.xml | 84 +-
...ma-one-field-no-dynamic-field-unique-key.xml | 14 +-
.../conf/schema-one-field-no-dynamic-field.xml | 14 +-
.../collection1/conf/schema-phrasesuggest.xml | 34 +-
.../conf/schema-postingshighlight.xml | 48 +-
.../collection1/conf/schema-preanalyzed.xml | 43 +-
.../collection1/conf/schema-psuedo-fields.xml | 74 +
.../collection1/conf/schema-replication1.xml | 15 +-
.../collection1/conf/schema-replication2.xml | 20 +-
.../collection1/conf/schema-required-fields.xml | 737 +--
.../conf/schema-rest-lucene-match-version.xml | 38 +-
.../solr/collection1/conf/schema-rest.xml | 1205 ++--
.../solr/collection1/conf/schema-reversed.xml | 118 +-
.../solr/collection1/conf/schema-sim.xml | 46 +-
.../collection1/conf/schema-simpleqpplugin.xml | 74 +-
.../conf/schema-single-dynamic-copy-field.xml | 1152 ++--
.../collection1/conf/schema-sortingresponse.xml | 67 +-
.../solr/collection1/conf/schema-sorts.xml | 535 +-
.../solr/collection1/conf/schema-spatial.xml | 81 +-
.../collection1/conf/schema-spellchecker.xml | 50 +-
.../solr/collection1/conf/schema-sql.xml | 1087 ++--
.../solr/collection1/conf/schema-stop-keep.xml | 51 +-
.../solr/collection1/conf/schema-sweetspot.xml | 93 +-
.../conf/schema-synonym-tokenizer.xml | 35 +-
.../solr/collection1/conf/schema-tiny.xml | 14 +-
.../solr/collection1/conf/schema-trie.xml | 526 +-
.../solr/collection1/conf/schema-version-dv.xml | 30 +-
.../collection1/conf/schema-version-indexed.xml | 30 +-
.../solr/collection1/conf/schema-xinclude.xml | 12 +-
.../test-files/solr/collection1/conf/schema.xml | 1274 ++--
.../solr/collection1/conf/schema11.xml | 2 +-
.../solr/collection1/conf/schema12.xml | 1087 ++--
.../solr/collection1/conf/schema15.xml | 1076 ++--
.../solr/collection1/conf/schema_codec.xml | 54 +-
.../solr/collection1/conf/schema_latest.xml | 1324 ++---
.../solr/collection1/conf/schemasurround.xml | 1075 ++--
.../conf/solrconfig-analytics-query.xml | 8 -
.../conf/solrconfig-collapseqparser.xml | 8 -
.../conf/solrconfig-plugcollector.xml | 9 -
.../conf/solrconfig-testxmlparser.xml | 33 +
.../solr/collection1/conf/solrconfig-tlog.xml | 8 +-
.../solr/collection1/conf/solrconfig.xml | 13 +-
.../configsets/bad-mergepolicy/conf/schema.xml | 6 +-
.../cdcr-source-disabled/conf/schema.xml | 29 +
.../cdcr-source-disabled/conf/solrconfig.xml | 60 +
.../solr/configsets/cdcr-source/conf/schema.xml | 29 +
.../configsets/cdcr-source/conf/solrconfig.xml | 76 +
.../solr/configsets/cdcr-target/conf/schema.xml | 29 +
.../configsets/cdcr-target/conf/solrconfig.xml | 63 +
.../configsets/cloud-dynamic/conf/schema.xml | 275 +
.../cloud-dynamic/conf/solrconfig.xml | 48 +
.../cloud-managed-upgrade/conf/schema.xml | 27 +
.../cloud-managed-upgrade/conf/solrconfig.xml | 50 +
.../cloud-managed/conf/managed-schema | 6 +-
.../configsets/cloud-minimal/conf/schema.xml | 8 +-
.../configsets/cloud-subdirs/conf/schema.xml | 28 +
.../cloud-subdirs/conf/solrconfig.xml | 48 +
.../conf/stopwords/stopwords-en.txt | 62 +
.../solr/configsets/configset-2/conf/schema.xml | 12 +-
.../solr/configsets/doc-expiry/conf/schema.xml | 271 +
.../solrconfig.snippet.randomindexconfig.xml | 47 +
.../configsets/doc-expiry/conf/solrconfig.xml | 96 +
.../exitable-directory/conf/schema.xml | 28 +
.../exitable-directory/conf/solrconfig.xml | 117 +
.../solr/configsets/minimal/conf/schema.xml | 6 +-
.../solr/configsets/resource-sharing/schema.xml | 6 +-
solr/core/src/test-files/solr/solr-50-all.xml | 4 +
solr/core/src/test-files/solr/solr-no-core.xml | 45 -
solr/core/src/test-files/solr/solr.xml | 3 +
.../org/apache/solr/BasicFunctionalityTest.java | 2 +-
.../apache/solr/DisMaxRequestHandlerTest.java | 4 +-
.../solr/DistributedIntervalFacetingTest.java | 3 +-
.../solr/HelloWorldSolrCloudTestCase.java | 94 +
.../test/org/apache/solr/MinimalSchemaTest.java | 6 +-
.../test/org/apache/solr/SolrInfoMBeanTest.java | 1 -
.../apache/solr/TestDistributedGrouping.java | 34 +-
.../org/apache/solr/TestDistributedSearch.java | 4 +-
.../org/apache/solr/TestRandomFaceting.java | 261 +-
.../solr/TestSimpleTrackingShardHandler.java | 1 -
.../core/src/test/org/apache/solr/TestTrie.java | 4 +-
.../PathHierarchyTokenizerFactoryTest.java | 3 -
.../TestReversedWildcardFilterFactory.java | 2 +-
.../solr/client/solrj/ConnectionReuseTest.java | 198 -
.../client/solrj/impl/ConnectionReuseTest.java | 196 +
.../AbstractCloudBackupRestoreTestCase.java | 270 +
.../apache/solr/cloud/AliasIntegrationTest.java | 251 +-
.../AsyncCallRequestStatusResponseTest.java | 44 +-
.../solr/cloud/AsyncMigrateRouteKeyTest.java | 121 -
.../solr/cloud/BaseCdcrDistributedZkTest.java | 28 +-
.../solr/cloud/BasicDistributedZk2Test.java | 27 +-
.../solr/cloud/BasicDistributedZkTest.java | 5 -
.../apache/solr/cloud/CdcrBootstrapTest.java | 372 ++
.../cloud/CdcrReplicationDistributedZkTest.java | 31 +
.../solr/cloud/ChaosMonkeyShardSplitTest.java | 13 +-
.../apache/solr/cloud/CleanupOldIndexTest.java | 105 +-
.../cloud/CloudExitableDirectoryReaderTest.java | 47 +-
.../org/apache/solr/cloud/ClusterStateTest.java | 1 -
.../solr/cloud/ClusterStateUpdateTest.java | 5 -
.../apache/solr/cloud/CollectionReloadTest.java | 81 +-
.../solr/cloud/CollectionStateFormat2Test.java | 73 +-
.../cloud/CollectionsAPIDistributedZkTest.java | 4 +-
.../solr/cloud/CollectionsAPISolrJTest.java | 5 +-
...ConcurrentDeleteAndCreateCollectionTest.java | 13 +-
.../apache/solr/cloud/ConfigSetsAPITest.java | 39 +-
.../solr/cloud/ConnectionManagerTest.java | 5 -
.../solr/cloud/CreateCollectionCleanupTest.java | 4 +-
.../apache/solr/cloud/CustomCollectionTest.java | 7 +-
.../solr/cloud/DeleteInactiveReplicaTest.java | 186 +-
.../DeleteLastCustomShardedReplicaTest.java | 3 +-
.../org/apache/solr/cloud/DeleteNodeTest.java | 75 +
.../apache/solr/cloud/DeleteReplicaTest.java | 242 +-
.../solr/cloud/DistribCursorPagingTest.java | 15 +-
...DistribDocExpirationUpdateProcessorTest.java | 141 +-
.../cloud/DistribJoinFromCollectionTest.java | 137 +-
.../apache/solr/cloud/DistributedQueueTest.java | 47 +-
.../solr/cloud/DistributedVersionInfoTest.java | 102 +-
.../apache/solr/cloud/HttpPartitionTest.java | 20 +-
.../apache/solr/cloud/KerberosTestServices.java | 229 +
.../org/apache/solr/cloud/KerberosTestUtil.java | 147 -
.../cloud/LeaderElectionIntegrationTest.java | 10 +-
.../apache/solr/cloud/LeaderElectionTest.java | 1 +
.../cloud/LeaderFailureAfterFreshStartTest.java | 295 +
.../apache/solr/cloud/MigrateRouteKeyTest.java | 160 +-
.../apache/solr/cloud/MultiThreadedOCPTest.java | 88 +-
...utOfBoxZkACLAndCredentialsProvidersTest.java | 7 +-
...rriddenZkACLAndCredentialsProvidersTest.java | 71 +-
...verseerCollectionConfigSetProcessorTest.java | 220 +-
.../cloud/OverseerModifyCollectionTest.java | 96 +
.../apache/solr/cloud/OverseerRolesTest.java | 2 +-
.../solr/cloud/OverseerTaskQueueTest.java | 3 +-
.../org/apache/solr/cloud/OverseerTest.java | 20 +-
.../solr/cloud/PeerSyncReplicationTest.java | 363 ++
.../solr/cloud/RecoveryAfterSoftCommitTest.java | 1 -
.../apache/solr/cloud/RemoteQueryErrorTest.java | 2 +-
.../org/apache/solr/cloud/ReplaceNodeTest.java | 104 +
.../solr/cloud/ReplicationFactorTest.java | 1 -
.../org/apache/solr/cloud/SSLMigrationTest.java | 1 -
.../solr/cloud/SaslZkACLProviderTest.java | 39 +-
.../org/apache/solr/cloud/ShardSplitTest.java | 421 ++
.../cloud/SharedFSAutoReplicaFailoverTest.java | 150 +-
.../SharedFSAutoReplicaFailoverUtilsTest.java | 99 +-
.../cloud/SimpleCollectionCreateDeleteTest.java | 2 -
.../apache/solr/cloud/SolrCLIZkUtilsTest.java | 632 ++
.../apache/solr/cloud/SolrCloudExampleTest.java | 6 +-
.../org/apache/solr/cloud/SyncSliceTest.java | 2 -
.../solr/cloud/TestAuthenticationFramework.java | 176 +-
.../solr/cloud/TestCloudBackupRestore.java | 219 -
.../solr/cloud/TestCloudDeleteByQuery.java | 20 +-
.../apache/solr/cloud/TestCloudInspectUtil.java | 2 -
.../apache/solr/cloud/TestCloudPivotFacet.java | 11 +-
.../solr/cloud/TestCloudPseudoReturnFields.java | 841 +++
.../apache/solr/cloud/TestCollectionAPI.java | 1 -
.../apache/solr/cloud/TestConfigSetsAPI.java | 18 +-
.../cloud/TestConfigSetsAPIExclusivity.java | 11 +-
.../solr/cloud/TestConfigSetsAPIZkFailure.java | 5 +-
.../org/apache/solr/cloud/TestCryptoKeys.java | 1 -
.../solr/cloud/TestHdfsCloudBackupRestore.java | 147 +
.../TestLeaderElectionWithEmptyReplica.java | 125 +
.../TestLeaderRecoverFromLogOnStartup.java | 77 +
.../cloud/TestLocalFSCloudBackupRestore.java | 57 +
.../org/apache/solr/cloud/TestLockTree.java | 123 +
.../solr/cloud/TestMiniSolrCloudCluster.java | 123 +-
.../cloud/TestMiniSolrCloudClusterBase.java | 207 -
.../cloud/TestMiniSolrCloudClusterKerberos.java | 46 +-
.../solr/cloud/TestMiniSolrCloudClusterSSL.java | 57 +-
.../cloud/TestOnReconnectListenerSupport.java | 4 +-
.../apache/solr/cloud/TestRandomFlRTGCloud.java | 968 +++
.../cloud/TestRandomRequestDistribution.java | 2 +-
.../solr/cloud/TestRequestForwarding.java | 4 +-
.../apache/solr/cloud/TestSSLRandomization.java | 202 +-
.../cloud/TestSizeLimitedDistributedMap.java | 4 +-
.../TestSolrCloudWithDelegationTokens.java | 414 ++
.../cloud/TestSolrCloudWithKerberosAlt.java | 43 +-
.../TestSolrCloudWithSecureImpersonation.java | 362 ++
.../TestStressCloudBlindAtomicUpdates.java | 491 ++
.../apache/solr/cloud/TestStressLiveNodes.java | 8 +-
.../cloud/TestTolerantUpdateProcessorCloud.java | 51 +-
.../TestTolerantUpdateProcessorRandomCloud.java | 37 +-
.../solr/cloud/UnloadDistributedZkTest.java | 6 +-
...MParamsZkACLAndCredentialsProvidersTest.java | 25 +-
.../test/org/apache/solr/cloud/ZkCLITest.java | 58 +-
.../org/apache/solr/cloud/ZkControllerTest.java | 142 +-
.../org/apache/solr/cloud/ZkSolrClientTest.java | 16 -
.../solr/cloud/hdfs/HdfsNNFailoverTest.java | 2 +-
.../solr/cloud/hdfs/HdfsThreadLeakTest.java | 2 -
.../HdfsWriteToMultipleCollectionsTest.java | 21 +-
.../apache/solr/cloud/hdfs/StressHdfsTest.java | 2 +-
.../solr/cloud/overseer/ZkStateReaderTest.java | 59 +-
.../solr/cloud/overseer/ZkStateWriterTest.java | 403 +-
.../solr/cloud/rule/ImplicitSnitchTest.java | 51 +-
.../apache/solr/cloud/rule/RuleEngineTest.java | 56 +-
.../org/apache/solr/cloud/rule/RulesTest.java | 272 +-
.../solr/core/BlobRepositoryCloudTest.java | 53 +-
.../solr/core/BlobRepositoryMockingTest.java | 37 +-
.../org/apache/solr/core/CoreSorterTest.java | 240 +
.../solr/core/CountUsageValueSourceParser.java | 2 +-
.../solr/core/ExitableDirectoryReaderTest.java | 2 +-
.../solr/core/HdfsDirectoryFactoryTest.java | 5 -
.../test/org/apache/solr/core/HelloStream.java | 100 +
.../org/apache/solr/core/MockInfoMBean.java | 1 -
.../solr/core/OpenCloseCoreStressTest.java | 16 +-
.../apache/solr/core/QueryResultKeyTest.java | 4 +-
.../test/org/apache/solr/core/SolrCoreTest.java | 66 +-
.../apache/solr/core/TestArbitraryIndexDir.java | 113 -
.../solr/core/TestBackupRepositoryFactory.java | 149 +
.../org/apache/solr/core/TestCodecSupport.java | 7 +-
.../test/org/apache/solr/core/TestConfig.java | 5 +-
.../solr/core/TestConfigSetImmutable.java | 3 +-
.../org/apache/solr/core/TestCustomStream.java | 94 +
.../apache/solr/core/TestDynamicLoading.java | 1 -
.../apache/solr/core/TestInfoStreamLogging.java | 1 -
.../apache/solr/core/TestJmxMonitoredMap.java | 3 -
.../org/apache/solr/core/TestLazyCores.java | 6 +-
.../apache/solr/core/TestMergePolicyConfig.java | 2 +-
.../solr/core/TestQuerySenderListener.java | 18 +-
.../apache/solr/core/TestSolrConfigHandler.java | 80 +-
.../test/org/apache/solr/core/TestSolrXml.java | 15 +-
.../core/snapshots/TestSolrCoreSnapshots.java | 312 +
.../apache/solr/handler/BackupRestoreUtils.java | 104 +
.../apache/solr/handler/CheckBackupStatus.java | 10 +-
.../FieldAnalysisRequestHandlerTest.java | 4 -
.../org/apache/solr/handler/JsonLoaderTest.java | 68 +-
.../solr/handler/PingRequestHandlerTest.java | 12 +-
.../handler/StandardRequestHandlerTest.java | 8 -
.../apache/solr/handler/TestBlobHandler.java | 1 -
.../org/apache/solr/handler/TestCSVLoader.java | 1 -
.../apache/solr/handler/TestConfigReload.java | 1 -
.../solr/handler/TestHdfsBackupRestoreCore.java | 239 +
.../solr/handler/TestReplicationHandler.java | 102 +-
.../handler/TestReplicationHandlerBackup.java | 39 +-
.../apache/solr/handler/TestReqParamsAPI.java | 91 +-
.../apache/solr/handler/TestRestoreCore.java | 52 +-
.../org/apache/solr/handler/TestSQLHandler.java | 696 +--
.../solr/handler/TestSQLHandlerNonCloud.java | 92 +
.../handler/TestSolrConfigHandlerCloud.java | 3 +-
.../TestSolrConfigHandlerConcurrent.java | 2 -
.../handler/ThrowErrorOnInitRequestHandler.java | 1 -
.../handler/admin/CoreAdminHandlerTest.java | 2 +-
.../admin/CoreMergeIndexesAdminHandlerTest.java | 3 -
.../solr/handler/admin/StatsReloadRaceTest.java | 99 +
.../DistributedFacetExistsSmallTest.java | 236 +
.../DistributedFacetPivotLargeTest.java | 2 +-
.../component/DistributedMLTComponentTest.java | 1 -
...DistributedQueryComponentCustomSortTest.java | 15 +-
...stributedQueryComponentOptimizationTest.java | 236 +-
.../DistributedTermsComponentTest.java | 13 +-
.../component/ResourceSharingTestComponent.java | 34 +-
.../handler/component/SearchHandlerTest.java | 20 +-
.../handler/component/TermsComponentTest.java | 58 +-
.../handler/component/TestExpandComponent.java | 1 -
.../handler/component/TestPivotHelperCode.java | 3 -
.../highlight/HighlighterMaxOffsetTest.java | 1 -
.../apache/solr/highlight/HighlighterTest.java | 82 +-
.../index/TestSlowCompositeReaderWrapper.java | 95 +
.../solr/index/hdfs/CheckHdfsIndexTest.java | 3 +-
.../solr/internal/csv/CSVPrinterTest.java | 3 -
.../solr/internal/csv/CSVStrategyTest.java | 2 -
.../csv/ExtendedBufferedReaderTest.java | 2 -
.../org/apache/solr/request/JSONWriterTest.java | 142 -
.../apache/solr/request/SimpleFacetsTest.java | 319 +-
.../apache/solr/request/SmileWriterTest.java | 253 -
.../solr/request/TestBinaryResponseWriter.java | 106 -
.../apache/solr/request/TestFacetMethods.java | 207 +
.../org/apache/solr/request/TestFaceting.java | 73 +-
.../solr/request/TestIntervalFaceting.java | 1 -
.../solr/request/macro/TestMacroExpander.java | 116 +
.../apache/solr/response/JSONWriterTest.java | 144 +
.../apache/solr/response/SmileWriterTest.java | 253 +
.../solr/response/TestBinaryResponseWriter.java | 104 +
.../solr/response/TestChildDocTransformer.java | 244 -
.../solr/response/TestCustomDocTransformer.java | 4 +-
.../response/TestGraphMLResponseWriter.java | 155 +
.../TestPHPSerializedResponseWriter.java | 1 -
.../solr/response/TestRawResponseWriter.java | 10 +-
.../transform/TestChildDocTransformer.java | 403 ++
.../transform/TestSubQueryTransformer.java | 570 ++
.../TestSubQueryTransformerCrossCore.java | 131 +
.../TestSubQueryTransformerDistrib.java | 217 +
.../solr/rest/schema/TestBulkSchemaAPI.java | 69 +-
.../solr/schema/AbstractCurrencyFieldTest.java | 2 +-
.../apache/solr/schema/BooleanFieldTest.java | 78 +
.../solr/schema/ChangedSchemaMergeTest.java | 8 -
.../schema/CurrencyFieldOpenExchangeTest.java | 10 -
.../solr/schema/CurrencyFieldXmlFileTest.java | 9 -
.../apache/solr/schema/DateRangeFieldTest.java | 29 +
.../solr/schema/DocValuesMissingTest.java | 129 +
.../apache/solr/schema/DocValuesMultiTest.java | 113 +-
.../org/apache/solr/schema/DocValuesTest.java | 145 +-
.../org/apache/solr/schema/EnumFieldTest.java | 1 -
.../apache/solr/schema/MyCrazyCustomField.java | 2 +-
.../org/apache/solr/schema/PolyFieldTest.java | 1 -
.../solr/schema/SpatialRPTFieldTypeTest.java | 32 +-
.../solr/schema/TestBulkSchemaConcurrent.java | 1 -
.../TestCloudManagedSchemaConcurrent.java | 1 -
.../apache/solr/schema/TestCloudSchemaless.java | 2 -
.../apache/solr/schema/TestCollationField.java | 1 -
.../solr/schema/TestManagedSchemaAPI.java | 37 +-
.../schema/TestManagedSchemaThreadSafety.java | 164 +
.../TrieIntPrefixActsAsRangeQueryFieldType.java | 2 +-
.../apache/solr/search/AnalyticsQueryTest.java | 1 -
.../solr/search/DelayingSearchComponent.java | 1 -
.../apache/solr/search/FooQParserPlugin.java | 1 -
.../apache/solr/search/GoodbyeQueryBuilder.java | 39 +
.../apache/solr/search/HandyQueryBuilder.java | 53 +
.../apache/solr/search/HelloQueryBuilder.java | 39 +
.../apache/solr/search/QueryEqualityTest.java | 45 +-
.../apache/solr/search/QueryParsingTest.java | 8 -
.../org/apache/solr/search/RankQueryTest.java | 1 -
.../apache/solr/search/SortSpecParsingTest.java | 1 -
.../solr/search/TestCollapseQParserPlugin.java | 18 +-
.../test/org/apache/solr/search/TestDocSet.java | 15 +-
.../solr/search/TestExtendedDismaxParser.java | 155 +
.../solr/search/TestFilteredDocIdSet.java | 21 +
.../search/TestGraphTermsQParserPlugin.java | 14 +-
.../solr/search/TestHashQParserPlugin.java | 9 +-
.../apache/solr/search/TestIndexSearcher.java | 6 +-
.../org/apache/solr/search/TestLRUCache.java | 5 -
.../TestLegacyNumericRangeQueryBuilder.java | 179 +
.../solr/search/TestMaxScoreQueryParser.java | 1 +
.../apache/solr/search/TestMissingGroups.java | 4 +-
...OverriddenPrefixQueryForCustomFieldType.java | 2 +-
.../solr/search/TestPseudoReturnFields.java | 632 +-
.../org/apache/solr/search/TestQueryUtils.java | 2 -
.../solr/search/TestQueryWrapperFilter.java | 14 +-
.../search/TestRandomCollapseQParserPlugin.java | 7 +-
.../org/apache/solr/search/TestRangeQuery.java | 146 +-
.../apache/solr/search/TestRankQueryPlugin.java | 22 +-
.../solr/search/TestReRankQParserPlugin.java | 1 -
.../org/apache/solr/search/TestRealTimeGet.java | 1 -
.../org/apache/solr/search/TestRecovery.java | 2 +-
.../apache/solr/search/TestRecoveryHdfs.java | 1 -
.../test/org/apache/solr/search/TestReload.java | 2 -
.../apache/solr/search/TestReloadDeadlock.java | 4 +-
.../org/apache/solr/search/TestSearchPerf.java | 9 +-
.../apache/solr/search/TestSearcherReuse.java | 2 +-
.../solr/search/TestSimpleQParserPlugin.java | 3 -
.../apache/solr/search/TestSmileRequest.java | 2 +-
.../apache/solr/search/TestSolr4Spatial2.java | 2 +-
.../apache/solr/search/TestSolrQueryParser.java | 5 +-
.../test/org/apache/solr/search/TestSort.java | 15 +-
.../solr/search/TestStandardQParsers.java | 9 +
.../solr/search/TestXmlQParserPlugin.java | 78 +
.../search/facet/TestJsonFacetRefinement.java | 214 +
.../solr/search/facet/TestJsonFacets.java | 60 +-
.../search/function/NvlValueSourceParser.java | 8 +-
.../solr/search/function/TestFunctionQuery.java | 72 +-
.../function/TestMinMaxOnMultiValuedField.java | 4 -
.../solr/search/function/TestOrdValues.java | 8 +-
.../function/TestSortByMinMaxFunction.java | 2 -
.../apache/solr/search/join/BJQParserTest.java | 1 -
.../search/join/BlockJoinFacetDistribTest.java | 94 +-
.../search/join/BlockJoinFacetSimpleTest.java | 24 +
.../search/join/TestScoreJoinQPNoScore.java | 4 +-
.../solr/search/join/TestScoreJoinQPScore.java | 2 +-
.../solr/search/mlt/CloudMLTQParserTest.java | 279 +-
.../solr/search/mlt/SimpleMLTQParserTest.java | 4 -
.../TestClassicSimilarityFactory.java | 1 -
.../similarities/TestDFRSimilarityFactory.java | 2 +-
.../similarities/TestIBSimilarityFactory.java | 2 +-
.../TestNonDefinedSimilarityFactory.java | 3 +-
...stPerFieldSimilarityWithDefaultOverride.java | 2 -
.../TestSweetSpotSimilarityFactory.java | 1 -
.../solr/search/stats/TestDistribIDF.java | 8 +-
.../solr/security/BasicAuthIntegrationTest.java | 283 +-
.../HttpParamDelegationTokenPlugin.java | 272 +
.../solr/security/MockAuthenticationPlugin.java | 32 +-
.../security/TestPKIAuthenticationPlugin.java | 41 +-
.../apache/solr/servlet/ResponseHeaderTest.java | 4 -
.../spelling/FileBasedSpellCheckerTest.java | 1 -
.../solr/spelling/SimpleQueryConverter.java | 2 -
.../solr/spelling/SpellCheckCollatorTest.java | 30 +
.../spelling/TestSuggestSpellingConverter.java | 3 +-
.../suggest/TestFreeTextSuggestions.java | 1 -
.../solr/store/hdfs/HdfsDirectoryTest.java | 5 +-
.../solr/uninverting/TestDocTermOrds.java | 687 +++
.../apache/solr/uninverting/TestFieldCache.java | 730 +++
.../solr/uninverting/TestFieldCacheReopen.java | 72 +
.../TestFieldCacheSanityChecker.java | 164 +
.../solr/uninverting/TestFieldCacheSort.java | 1814 ++++++
.../uninverting/TestFieldCacheSortRandom.java | 319 +
.../uninverting/TestFieldCacheVsDocValues.java | 623 ++
.../uninverting/TestFieldCacheWithThreads.java | 256 +
.../solr/uninverting/TestLegacyFieldCache.java | 504 ++
.../solr/uninverting/TestNumericTerms32.java | 156 +
.../solr/uninverting/TestNumericTerms64.java | 166 +
.../solr/uninverting/TestUninvertingReader.java | 395 ++
.../apache/solr/update/AddBlockUpdateTest.java | 2 -
.../org/apache/solr/update/AutoCommitTest.java | 29 +
.../update/DirectUpdateHandlerOptimizeTest.java | 1 -
.../solr/update/DirectUpdateHandlerTest.java | 48 +-
.../apache/solr/update/DocumentBuilderTest.java | 22 +-
.../apache/solr/update/HardAutoCommitTest.java | 2 +
.../apache/solr/update/SoftAutoCommitTest.java | 16 +-
.../apache/solr/update/SolrIndexConfigTest.java | 14 +-
.../update/TestDocBasedVersionConstraints.java | 22 +-
.../solr/update/TestExceedMaxTermLength.java | 1 -
.../org/apache/solr/update/VersionInfoTest.java | 2 -
...lassificationUpdateProcessorFactoryTest.java | 4 +-
.../CloneFieldUpdateProcessorFactoryTest.java | 6 +-
.../processor/CustomUpdateRequestProcessor.java | 4 -
.../DefaultValueUpdateProcessorTest.java | 5 -
...DocExpirationUpdateProcessorFactoryTest.java | 1 -
...ommitOptimizeUpdateProcessorFactoryTest.java | 5 -
.../RecordingUpdateProcessorFactory.java | 7 +-
.../solr/update/processor/ScriptEngineTest.java | 2 +-
...atelessScriptUpdateProcessorFactoryTest.java | 5 -
.../processor/TemplateUpdateProcessorTest.java | 48 +
.../processor/TestNamedUpdateProcessors.java | 1 -
.../TestPartialUpdateDeduplication.java | 3 -
.../processor/TolerantUpdateProcessorTest.java | 2 -
.../UniqFieldsUpdateProcessorFactoryTest.java | 1 -
.../processor/UpdateProcessorTestBase.java | 1 -
.../UpdateRequestProcessorFactoryTest.java | 17 +-
.../apache/solr/util/DateMathParserTest.java | 226 +-
.../apache/solr/util/SolrPluginUtilsTest.java | 28 +
.../org/apache/solr/util/TestFastWriter.java | 3 -
.../solr/util/TestObjectReleaseTracker.java | 12 +-
.../org/apache/solr/util/TestTestInjection.java | 4 +
.../org/apache/solr/util/UtilsToolTest.java | 185 +
.../example-DIH/solr/db/conf/elevate.xml | 24 +-
.../example-DIH/solr/db/conf/managed-schema | 2 +-
.../example-DIH/solr/db/conf/solrconfig.xml | 84 +-
.../example-DIH/solr/mail/conf/elevate.xml | 24 +-
.../example-DIH/solr/mail/conf/managed-schema | 2 +-
.../example-DIH/solr/mail/conf/solrconfig.xml | 84 +-
.../example-DIH/solr/rss/conf/elevate.xml | 24 +-
.../example-DIH/solr/rss/conf/managed-schema | 2 +-
.../example-DIH/solr/rss/conf/solrconfig.xml | 84 +-
.../example-DIH/solr/solr/conf/elevate.xml | 24 +-
.../example-DIH/solr/solr/conf/managed-schema | 2 +-
.../example-DIH/solr/solr/conf/solrconfig.xml | 85 +-
.../example-DIH/solr/tika/conf/managed-schema | 2 +-
.../example-DIH/solr/tika/conf/solrconfig.xml | 85 +-
solr/example/files/conf/elevate.xml | 24 +-
solr/example/files/conf/solrconfig.xml | 66 -
solr/example/resources/log4j.properties | 2 +-
solr/licenses/asm-5.0.4.jar.sha1 | 1 -
solr/licenses/asm-5.1.jar.sha1 | 1 +
solr/licenses/asm-commons-5.0.4.jar.sha1 | 1 -
solr/licenses/asm-commons-5.1.jar.sha1 | 1 +
solr/licenses/bcpkix-jdk15on-1.47.jar.sha1 | 1 +
.../bcpkix-jdk15on-LICENSE-BSD_LIKE.txt | 15 +
solr/licenses/bcpkix-jdk15on-NOTICE.txt | 2 +
solr/licenses/commons-compress-1.11.jar.sha1 | 1 +
solr/licenses/commons-compress-1.8.1.jar.sha1 | 1 -
solr/licenses/commons-io-2.4.jar.sha1 | 1 -
solr/licenses/commons-io-2.5.jar.sha1 | 1 +
solr/licenses/curator-recipes-2.8.0.jar.sha1 | 1 +
solr/licenses/curator-recipes-LICENSE-ASL.txt | 202 +
solr/licenses/curator-recipes-NOTICE.txt | 5 +
solr/licenses/fontbox-1.8.8.jar.sha1 | 1 -
solr/licenses/fontbox-2.0.1.jar.sha1 | 1 +
solr/licenses/gimap-LICENSE-CDDL.txt | 262 +-
solr/licenses/hadoop-annotations-2.6.0.jar.sha1 | 1 -
solr/licenses/hadoop-annotations-2.7.2.jar.sha1 | 1 +
solr/licenses/hadoop-auth-2.6.0.jar.sha1 | 1 -
solr/licenses/hadoop-auth-2.7.2.jar.sha1 | 1 +
.../licenses/hadoop-common-2.6.0-tests.jar.sha1 | 1 -
solr/licenses/hadoop-common-2.6.0.jar.sha1 | 1 -
.../licenses/hadoop-common-2.7.2-tests.jar.sha1 | 1 +
solr/licenses/hadoop-common-2.7.2.jar.sha1 | 1 +
solr/licenses/hadoop-hdfs-2.6.0-tests.jar.sha1 | 1 -
solr/licenses/hadoop-hdfs-2.6.0.jar.sha1 | 1 -
solr/licenses/hadoop-hdfs-2.7.2-tests.jar.sha1 | 1 +
solr/licenses/hadoop-hdfs-2.7.2.jar.sha1 | 1 +
.../hadoop-mapreduce-client-app-2.6.0.jar.sha1 | 1 -
.../hadoop-mapreduce-client-app-2.7.2.jar.sha1 | 1 +
...adoop-mapreduce-client-common-2.6.0.jar.sha1 | 1 -
...adoop-mapreduce-client-common-2.7.2.jar.sha1 | 1 +
.../hadoop-mapreduce-client-core-2.6.0.jar.sha1 | 1 -
.../hadoop-mapreduce-client-core-2.7.2.jar.sha1 | 1 +
.../hadoop-mapreduce-client-hs-2.6.0.jar.sha1 | 1 -
.../hadoop-mapreduce-client-hs-2.7.2.jar.sha1 | 1 +
...reduce-client-jobclient-2.6.0-tests.jar.sha1 | 1 -
...op-mapreduce-client-jobclient-2.6.0.jar.sha1 | 1 -
...reduce-client-jobclient-2.7.2-tests.jar.sha1 | 1 +
...op-mapreduce-client-jobclient-2.7.2.jar.sha1 | 1 +
...doop-mapreduce-client-shuffle-2.6.0.jar.sha1 | 1 -
...doop-mapreduce-client-shuffle-2.7.2.jar.sha1 | 1 +
solr/licenses/hadoop-minikdc-2.6.0.jar.sha1 | 1 -
solr/licenses/hadoop-minikdc-2.7.2.jar.sha1 | 1 +
solr/licenses/hadoop-yarn-api-2.6.0.jar.sha1 | 1 -
solr/licenses/hadoop-yarn-api-2.7.2.jar.sha1 | 1 +
solr/licenses/hadoop-yarn-client-2.6.0.jar.sha1 | 1 -
solr/licenses/hadoop-yarn-client-2.7.2.jar.sha1 | 1 +
solr/licenses/hadoop-yarn-common-2.6.0.jar.sha1 | 1 -
solr/licenses/hadoop-yarn-common-2.7.2.jar.sha1 | 1 +
...ver-applicationhistoryservice-2.6.0.jar.sha1 | 1 -
...ver-applicationhistoryservice-2.7.2.jar.sha1 | 1 +
.../hadoop-yarn-server-common-2.6.0.jar.sha1 | 1 -
.../hadoop-yarn-server-common-2.7.2.jar.sha1 | 1 +
...adoop-yarn-server-nodemanager-2.6.0.jar.sha1 | 1 -
...adoop-yarn-server-nodemanager-2.7.2.jar.sha1 | 1 +
...p-yarn-server-resourcemanager-2.6.0.jar.sha1 | 1 -
...p-yarn-server-resourcemanager-2.7.2.jar.sha1 | 1 +
...adoop-yarn-server-tests-2.6.0-tests.jar.sha1 | 1 -
...adoop-yarn-server-tests-2.7.2-tests.jar.sha1 | 1 +
.../hadoop-yarn-server-web-proxy-2.6.0.jar.sha1 | 1 -
.../hadoop-yarn-server-web-proxy-2.7.2.jar.sha1 | 1 +
solr/licenses/htrace-core-3.0.4.jar.sha1 | 1 -
.../htrace-core-3.2.0-incubating.jar.sha1 | 1 +
solr/licenses/isoparser-1.0.2.jar.sha1 | 1 -
solr/licenses/isoparser-1.1.18.jar.sha1 | 1 +
solr/licenses/jackcess-2.1.3.jar.sha1 | 1 +
solr/licenses/jackcess-LICENSE-ASL.txt | 507 ++
solr/licenses/jackcess-NOTICE.txt | 2 +
solr/licenses/javax.mail-LICENSE-CDDL.txt | 262 +-
.../licenses/javax.servlet-api-LICENSE-CDDL.txt | 137 -
solr/licenses/jaxb-impl-LICENSE-CDDL.txt | 262 +-
.../jcl-over-slf4j-LICENSE-BSD_LIKE.txt | 21 -
solr/licenses/jcl-over-slf4j-LICENSE-MIT.txt | 21 +
solr/licenses/jempbox-1.8.12.jar.sha1 | 1 +
solr/licenses/jempbox-1.8.8.jar.sha1 | 1 -
solr/licenses/jul-to-slf4j-LICENSE-BSD_LIKE.txt | 21 -
solr/licenses/jul-to-slf4j-LICENSE-MIT.txt | 21 +
solr/licenses/mail-LICENSE-CDDL.txt | 263 -
solr/licenses/metadata-extractor-2.6.2.jar.sha1 | 1 -
solr/licenses/metadata-extractor-2.8.1.jar.sha1 | 1 +
solr/licenses/morfologik-fsa-2.1.0.jar.sha1 | 1 -
solr/licenses/morfologik-fsa-2.1.1.jar.sha1 | 1 +
solr/licenses/morfologik-polish-2.1.0.jar.sha1 | 1 -
solr/licenses/morfologik-polish-2.1.1.jar.sha1 | 1 +
.../licenses/morfologik-stemming-2.1.0.jar.sha1 | 1 -
.../licenses/morfologik-stemming-2.1.1.jar.sha1 | 1 +
solr/licenses/netty-3.2.4.Final.jar.sha1 | 1 +
solr/licenses/netty-3.7.0.Final.jar.sha1 | 1 -
solr/licenses/netty-NOTICE.txt | 85 +-
solr/licenses/netty-all-4.0.36.Final.jar.sha1 | 1 +
solr/licenses/netty-all-LICENSE-ASL.txt | 202 +
solr/licenses/netty-all-NOTICE.txt | 121 +
solr/licenses/pdfbox-1.8.8.jar.sha1 | 1 -
solr/licenses/pdfbox-2.0.1.jar.sha1 | 1 +
solr/licenses/pdfbox-tools-2.0.1.jar.sha1 | 1 +
solr/licenses/pdfbox-tools-LICENSE-ASL.txt | 314 +
solr/licenses/pdfbox-tools-NOTICE.txt | 14 +
solr/licenses/poi-3.11.jar.sha1 | 1 -
solr/licenses/poi-3.15-beta1.jar.sha1 | 1 +
solr/licenses/poi-ooxml-3.11.jar.sha1 | 1 -
solr/licenses/poi-ooxml-3.15-beta1.jar.sha1 | 1 +
solr/licenses/poi-ooxml-schemas-3.11.jar.sha1 | 1 -
.../poi-ooxml-schemas-3.15-beta1.jar.sha1 | 1 +
solr/licenses/poi-scratchpad-3.11.jar.sha1 | 1 -
.../licenses/poi-scratchpad-3.15-beta1.jar.sha1 | 1 +
solr/licenses/rome-1.6.1.jar.sha1 | 1 +
solr/licenses/servlet-api-LICENSE-CDDL.txt | 137 -
solr/licenses/slf4j-LICENSE-BSD_LIKE.txt | 21 -
solr/licenses/slf4j-LICENSE-MIT.txt | 21 +
solr/licenses/tika-core-1.13.jar.sha1 | 1 +
solr/licenses/tika-core-1.7.jar.sha1 | 1 -
solr/licenses/tika-java7-1.13.jar.sha1 | 1 +
solr/licenses/tika-java7-1.7.jar.sha1 | 1 -
solr/licenses/tika-parsers-1.13.jar.sha1 | 1 +
solr/licenses/tika-parsers-1.7.jar.sha1 | 1 -
solr/licenses/tika-xmp-1.13.jar.sha1 | 1 +
solr/licenses/tika-xmp-1.7.jar.sha1 | 1 -
solr/licenses/vorbis-java-core-0.6.jar.sha1 | 1 -
solr/licenses/vorbis-java-core-0.8.jar.sha1 | 1 +
solr/licenses/vorbis-java-tika-0.6.jar.sha1 | 1 -
solr/licenses/vorbis-java-tika-0.8.jar.sha1 | 1 +
solr/server/resources/log4j.properties | 15 +-
.../scripts/cloud-scripts/log4j.properties | 2 +-
.../basic_configs/conf/_rest_managed.json | 1 -
.../configsets/basic_configs/conf/elevate.xml | 42 +
.../basic_configs/conf/lang/contractions_ca.txt | 8 +
.../basic_configs/conf/lang/contractions_fr.txt | 15 +
.../basic_configs/conf/lang/contractions_ga.txt | 5 +
.../basic_configs/conf/lang/contractions_it.txt | 23 +
.../basic_configs/conf/lang/hyphenations_ga.txt | 5 +
.../basic_configs/conf/lang/stemdict_nl.txt | 6 +
.../basic_configs/conf/lang/stoptags_ja.txt | 420 ++
.../basic_configs/conf/lang/stopwords_ar.txt | 125 +
.../basic_configs/conf/lang/stopwords_bg.txt | 193 +
.../basic_configs/conf/lang/stopwords_ca.txt | 220 +
.../basic_configs/conf/lang/stopwords_cz.txt | 172 +
.../basic_configs/conf/lang/stopwords_da.txt | 110 +
.../basic_configs/conf/lang/stopwords_de.txt | 294 +
.../basic_configs/conf/lang/stopwords_el.txt | 78 +
.../basic_configs/conf/lang/stopwords_es.txt | 356 ++
.../basic_configs/conf/lang/stopwords_eu.txt | 99 +
.../basic_configs/conf/lang/stopwords_fa.txt | 313 +
.../basic_configs/conf/lang/stopwords_fi.txt | 97 +
.../basic_configs/conf/lang/stopwords_fr.txt | 186 +
.../basic_configs/conf/lang/stopwords_ga.txt | 110 +
.../basic_configs/conf/lang/stopwords_gl.txt | 161 +
.../basic_configs/conf/lang/stopwords_hi.txt | 235 +
.../basic_configs/conf/lang/stopwords_hu.txt | 211 +
.../basic_configs/conf/lang/stopwords_hy.txt | 46 +
.../basic_configs/conf/lang/stopwords_id.txt | 359 ++
.../basic_configs/conf/lang/stopwords_it.txt | 303 +
.../basic_configs/conf/lang/stopwords_ja.txt | 127 +
.../basic_configs/conf/lang/stopwords_lv.txt | 172 +
.../basic_configs/conf/lang/stopwords_nl.txt | 119 +
.../basic_configs/conf/lang/stopwords_no.txt | 194 +
.../basic_configs/conf/lang/stopwords_pt.txt | 253 +
.../basic_configs/conf/lang/stopwords_ro.txt | 233 +
.../basic_configs/conf/lang/stopwords_ru.txt | 243 +
.../basic_configs/conf/lang/stopwords_sv.txt | 133 +
.../basic_configs/conf/lang/stopwords_th.txt | 119 +
.../basic_configs/conf/lang/stopwords_tr.txt | 212 +
.../basic_configs/conf/lang/userdict_ja.txt | 29 +
.../basic_configs/conf/managed-schema | 765 ++-
.../configsets/basic_configs/conf/params.json | 20 +
.../basic_configs/conf/solrconfig.xml | 1116 +++-
.../data_driven_schema_configs/conf/elevate.xml | 24 +-
.../conf/managed-schema | 9 +-
.../conf/solrconfig.xml | 65 -
.../conf/elevate.xml | 24 +-
.../conf/managed-schema | 5 +-
.../conf/params.json | 11 +
.../conf/solrconfig.xml | 78 -
.../conf/velocity/cluster.vm | 2 +-
solr/site/SYSTEM_REQUIREMENTS.mdtext | 2 +-
.../quickstart-admin-ui-facet-options.png | Bin 8787 -> 10646 bytes
.../assets/images/quickstart-query-screen.png | Bin 400716 -> 278938 bytes
solr/site/quickstart.mdtext | 279 +-
solr/solrj/ivy.xml | 1 +
.../org/apache/solr/client/solrj/SolrQuery.java | 11 +-
.../solr/client/solrj/impl/CloudSolrClient.java | 249 +-
.../solrj/impl/ConcurrentUpdateSolrClient.java | 3 +-
.../impl/DelegationTokenHttpSolrClient.java | 73 +
.../solr/client/solrj/impl/HttpClientUtil.java | 23 +-
.../solr/client/solrj/impl/HttpSolrClient.java | 44 +-
.../solrj/impl/InputStreamResponseParser.java | 4 -
.../solrj/impl/Krb5HttpClientBuilder.java | 24 +-
.../client/solrj/impl/LBHttpSolrClient.java | 116 +-
.../impl/SolrHttpClientContextBuilder.java | 12 +-
.../solrj/io/ClassificationEvaluation.java | 85 +
.../apache/solr/client/solrj/io/ModelCache.java | 143 +
.../org/apache/solr/client/solrj/io/Tuple.java | 53 +-
.../solr/client/solrj/io/comp/HashKey.java | 2 -
.../solrj/io/comp/MultipleFieldComparator.java | 1 -
.../solrj/io/graph/GatherNodesStream.java | 45 +-
.../solrj/io/graph/ShortestPathStream.java | 63 +-
.../solr/client/solrj/io/sql/ResultSetImpl.java | 2 +
.../solr/client/solrj/io/sql/StatementImpl.java | 11 +-
.../client/solrj/io/stream/BiJoinStream.java | 8 -
.../client/solrj/io/stream/CloudSolrStream.java | 135 +-
.../client/solrj/io/stream/CommitStream.java | 260 +
.../client/solrj/io/stream/DaemonStream.java | 44 +-
.../client/solrj/io/stream/FacetStream.java | 77 +-
.../io/stream/FeaturesSelectionStream.java | 436 ++
.../client/solrj/io/stream/FetchStream.java | 314 +
.../client/solrj/io/stream/HashJoinStream.java | 48 +-
.../solr/client/solrj/io/stream/JDBCStream.java | 4 +
.../solrj/io/stream/LeftOuterJoinStream.java | 4 -
.../client/solrj/io/stream/ModelStream.java | 200 +
.../solrj/io/stream/OuterHashJoinStream.java | 18 +-
.../client/solrj/io/stream/ParallelStream.java | 25 +-
.../solrj/io/stream/ScoreNodesStream.java | 279 +
.../solr/client/solrj/io/stream/SolrStream.java | 33 +-
.../client/solrj/io/stream/StatsStream.java | 58 +-
.../client/solrj/io/stream/StreamContext.java | 11 +-
.../client/solrj/io/stream/TextLogitStream.java | 656 +++
.../client/solrj/io/stream/TopicStream.java | 149 +-
.../client/solrj/io/stream/UpdateStream.java | 3 +-
.../solrj/io/stream/expr/Explanation.java | 1 +
.../solrj/io/stream/expr/StreamExplanation.java | 2 -
.../solrj/io/stream/expr/StreamFactory.java | 79 +-
.../solrj/request/CollectionAdminRequest.java | 214 +-
.../client/solrj/request/CoreAdminRequest.java | 63 +
.../solr/client/solrj/request/CoreStatus.java | 48 +
.../solrj/request/DelegationTokenRequest.java | 152 +
.../client/solrj/request/UpdateRequest.java | 6 +
.../solrj/response/DelegationTokenResponse.java | 109 +
.../solr/client/solrj/response/FacetField.java | 1 -
.../client/solrj/response/LukeResponse.java | 15 +-
.../solr/client/solrj/response/PivotField.java | 2 -
.../org/apache/solr/common/MapSerializable.java | 30 +
.../solr/common/ToleratedUpdateError.java | 2 +-
.../solr/common/cloud/ClusterProperties.java | 126 +
.../apache/solr/common/cloud/ClusterState.java | 13 +-
.../solr/common/cloud/ClusterStateUtil.java | 2 -
.../common/cloud/CollectionStatePredicate.java | 42 +
.../common/cloud/CollectionStateWatcher.java | 42 +
.../solr/common/cloud/ConnectionManager.java | 25 +-
.../common/cloud/DefaultConnectionStrategy.java | 4 +-
.../apache/solr/common/cloud/DocCollection.java | 70 +-
.../org/apache/solr/common/cloud/Replica.java | 15 +-
.../solr/common/cloud/SaslZkACLProvider.java | 21 +-
.../cloud/SecurityAwareZkACLProvider.java | 79 +
.../org/apache/solr/common/cloud/Slice.java | 27 +-
.../apache/solr/common/cloud/SolrZkClient.java | 147 +-
...ParamsAllAndReadonlyDigestZkACLProvider.java | 53 +-
.../cloud/ZkClientConnectionStrategy.java | 4 +-
.../solr/common/cloud/ZkConfigManager.java | 118 +-
.../solr/common/cloud/ZkMaintenanceUtils.java | 368 ++
.../apache/solr/common/cloud/ZkNodeProps.java | 12 +
.../apache/solr/common/cloud/ZkStateReader.java | 551 +-
.../solr/common/cloud/rule/ImplicitSnitch.java | 160 +
.../solr/common/cloud/rule/RemoteCallback.java | 23 +
.../apache/solr/common/cloud/rule/Snitch.java | 36 +
.../solr/common/cloud/rule/SnitchContext.java | 103 +
.../solr/common/cloud/rule/package-info.java | 23 +
.../common/params/CollectionAdminParams.java | 9 +-
.../solr/common/params/CollectionParams.java | 155 +-
.../apache/solr/common/params/CommonParams.java | 150 +-
.../solr/common/params/CoreAdminParams.java | 25 +-
.../apache/solr/common/params/FacetParams.java | 25 +
.../solr/common/params/HighlightParams.java | 1 +
.../apache/solr/common/params/SolrParams.java | 46 +-
.../apache/solr/common/params/TermsParams.java | 12 +
.../apache/solr/common/util/ExecutorUtil.java | 14 +-
.../apache/solr/common/util/JavaBinCodec.java | 7 +
.../solr/common/util/JsonRecordReader.java | 155 +-
.../org/apache/solr/common/util/NamedList.java | 31 +-
.../solr/common/util/ObjectReleaseTracker.java | 21 +-
.../java/org/apache/solr/common/util/Pair.java | 41 +-
.../org/apache/solr/common/util/RetryUtil.java | 19 +-
.../org/apache/solr/common/util/StrUtils.java | 3 +-
.../java/org/apache/solr/common/util/Utils.java | 1 +
.../collection1/conf/schema-replication1.xml | 15 +-
.../solrj/solr/collection1/conf/schema-sql.xml | 1087 ++--
.../solrj/solr/collection1/conf/schema.xml | 975 +--
.../configset-1/conf/schema-minimal.xml | 6 +-
.../solr/configsets/configset-2/conf/schema.xml | 6 +-
.../solrj/solr/configsets/ml/conf/schema.xml | 77 +
.../solr/configsets/ml/conf/solrconfig.xml | 51 +
.../solr/configsets/shared/conf/schema.xml | 76 +-
.../solr/configsets/streaming/conf/schema.xml | 19 +-
.../solrj/solr/crazy-path-to-schema.xml | 9 +-
.../solr/multicore/core0/conf/solrconfig.xml | 1 -
.../solr/multicore/core1/conf/solrconfig.xml | 2 -
.../apache/solr/client/solrj/GetByIdTest.java | 2 -
.../solrj/MergeIndexesExampleTestBase.java | 1 -
.../solr/client/solrj/SolrExampleTests.java | 39 +-
.../client/solrj/SolrSchemalessExampleTest.java | 17 +-
.../solrj/beans/TestDocumentObjectBinder.java | 2 -
.../embedded/LargeVolumeBinaryJettyTest.java | 1 -
.../solrj/embedded/LargeVolumeEmbeddedTest.java | 1 -
.../solrj/embedded/LargeVolumeJettyTest.java | 1 -
.../solrj/embedded/SolrExampleEmbeddedTest.java | 1 -
.../solrj/embedded/SolrExampleJettyTest.java | 6 +-
.../solrj/impl/CloudSolrClientBuilderTest.java | 28 +-
.../client/solrj/impl/CloudSolrClientTest.java | 92 +-
.../ConcurrentUpdateSolrClientBuilderTest.java | 12 +-
.../solrj/impl/HttpSolrClientBuilderTest.java | 22 +-
.../solrj/impl/HttpSolrClientConPoolTest.java | 189 +
.../impl/HttpSolrClientSSLAuthConPoolTest.java | 40 +
.../solrj/impl/LBHttpSolrClientBuilderTest.java | 18 +-
.../solrj/io/graph/GraphExpressionTest.java | 382 +-
.../solr/client/solrj/io/graph/GraphTest.java | 29 +-
.../client/solrj/io/sql/JdbcDriverTest.java | 7 +-
.../solr/client/solrj/io/sql/JdbcTest.java | 24 +-
.../client/solrj/io/stream/JDBCStreamTest.java | 16 +
.../solrj/io/stream/RecordCountStream.java | 1 -
.../solrj/io/stream/StreamExpressionTest.java | 1432 ++++-
.../stream/StreamExpressionToExpessionTest.java | 37 +-
.../StreamExpressionToExplanationTest.java | 2 -
.../client/solrj/io/stream/StreamingTest.java | 433 +-
.../io/stream/ops/ConcatOperationTest.java | 1 -
.../solr/client/solrj/request/SchemaTest.java | 11 +-
.../request/TestDelegationTokenRequest.java | 70 +
.../solrj/request/TestUpdateRequestCodec.java | 2 -
.../response/TestDelegationTokenResponse.java | 138 +
.../cloud/TestCollectionStateWatchers.java | 301 +
.../solr/common/cloud/TestZkConfigManager.java | 1 -
.../apache/solr/common/util/NamedListTest.java | 18 +
.../solr/common/util/TestJavaBinCodec.java | 104 +-
.../solr/common/util/TestJsonRecordReader.java | 115 +-
.../apache/solr/common/util/TestRetryUtil.java | 37 +-
solr/test-framework/build.xml | 9 +-
.../src/java/org/apache/solr/JSONTestUtil.java | 13 +
.../java/org/apache/solr/SolrTestCaseJ4.java | 165 +-
.../solr/analysis/MockTokenizerFactory.java | 1 -
.../solr/cloud/AbstractDistribZkTestBase.java | 10 +-
.../cloud/AbstractFullDistribZkTestBase.java | 18 +-
.../apache/solr/cloud/AbstractZkTestCase.java | 6 -
.../java/org/apache/solr/cloud/ChaosMonkey.java | 1 -
.../apache/solr/cloud/MiniSolrCloudCluster.java | 147 +-
.../org/apache/solr/cloud/MockSolrZkClient.java | 1 -
.../apache/solr/cloud/SolrCloudTestCase.java | 144 +
.../apache/solr/core/MockDirectoryFactory.java | 5 -
.../solr/core/MockFSDirectoryFactory.java | 1 -
.../org/apache/solr/util/BaseTestHarness.java | 1 -
.../org/apache/solr/util/ExternalPaths.java | 4 -
.../src/java/org/apache/solr/util/LogLevel.java | 91 +
.../org/apache/solr/util/RandomMergePolicy.java | 8 -
.../java/org/apache/solr/util/RandomizeSSL.java | 174 +
.../org/apache/solr/util/RestTestHarness.java | 14 +-
.../org/apache/solr/util/SSLTestConfig.java | 148 +-
.../resources/SSLTestConfig.testing.keystore | Bin 0 -> 2208 bytes
.../src/test-files/log4j.properties | 11 +
.../apache/solr/TestLogLevelAnnotations.java | 47 +
.../apache/solr/cloud/JettySolrRunnerTest.java | 66 +
.../solr/cloud/MiniSolrCloudClusterTest.java | 105 +
solr/webapp/web/WEB-INF/web.xml | 17 +
solr/webapp/web/css/angular/cloud.css | 14 +
solr/webapp/web/css/angular/dashboard.css | 8 +-
solr/webapp/web/css/angular/index.css | 12 +-
solr/webapp/web/css/angular/schema.css | 23 +
solr/webapp/web/js/angular/app.js | 2 +-
solr/webapp/web/js/angular/controllers/cloud.js | 53 +-
.../web/js/angular/controllers/collections.js | 7 +-
.../web/js/angular/controllers/core-overview.js | 83 -
solr/webapp/web/js/angular/controllers/cores.js | 10 +-
.../web/js/angular/controllers/dataimport.js | 58 +-
solr/webapp/web/js/angular/controllers/files.js | 6 +-
solr/webapp/web/js/angular/controllers/query.js | 10 +-
.../webapp/web/js/angular/controllers/schema.js | 64 +-
solr/webapp/web/js/angular/services.js | 6 +-
solr/webapp/web/js/lib/jquery.blockUI.js | 2 +-
solr/webapp/web/partials/cloud.html | 3 +-
solr/webapp/web/partials/core_overview.html | 17 -
solr/webapp/web/partials/dataimport.html | 8 +-
solr/webapp/web/partials/schema.html | 15 +-
3080 files changed, 164644 insertions(+), 104586 deletions(-)
----------------------------------------------------------------------
[21/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9634: Deprecate
create/deleteCollection methods on MiniSolrCloudCluster
Posted by kr...@apache.org.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestStressCloudBlindAtomicUpdates.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestStressCloudBlindAtomicUpdates.java b/solr/core/src/test/org/apache/solr/cloud/TestStressCloudBlindAtomicUpdates.java
index fc6d18c..9f29d3d 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestStressCloudBlindAtomicUpdates.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestStressCloudBlindAtomicUpdates.java
@@ -19,29 +19,30 @@ package org.apache.solr.cloud;
import java.lang.invoke.MethodHandles;
import java.nio.file.Path;
import java.nio.file.Paths;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Future;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.lucene.util.LuceneTestCase.Slow;
+import org.apache.lucene.util.TestUtil;
import org.apache.solr.SolrTestCaseJ4.SuppressSSL;
-import org.apache.solr.cloud.SolrCloudTestCase;
import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.UpdateRequest;
-import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.client.solrj.request.schema.SchemaRequest.Field;
import org.apache.solr.client.solrj.request.schema.SchemaRequest.FieldType;
+import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.client.solrj.response.schema.SchemaResponse.FieldResponse;
import org.apache.solr.client.solrj.response.schema.SchemaResponse.FieldTypeResponse;
import org.apache.solr.common.SolrDocument;
@@ -52,13 +53,9 @@ import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.ExecutorUtil;
import org.apache.solr.util.DefaultSolrThreadFactory;
import org.apache.solr.util.TestInjection;
-import org.apache.lucene.util.LuceneTestCase.Slow;
-import org.apache.lucene.util.TestUtil;
-
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -125,17 +122,15 @@ public class TestStressCloudBlindAtomicUpdates extends SolrCloudTestCase {
final Path configDir = Paths.get(TEST_HOME(), "collection1", "conf");
configureCluster(numNodes).addConfig(configName, configDir).configure();
-
- Map<String, String> collectionProperties = new HashMap<>();
- collectionProperties.put("config", "solrconfig-tlog.xml");
- collectionProperties.put("schema", "schema-minimal-atomic-stress.xml");
- assertNotNull(cluster.createCollection(COLLECTION_NAME, numShards, repFactor,
- configName, null, null, collectionProperties));
-
CLOUD_CLIENT = cluster.getSolrClient();
CLOUD_CLIENT.setDefaultCollection(COLLECTION_NAME);
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, configName, numShards, repFactor)
+ .withProperty("config", "solrconfig-tlog.xml")
+ .withProperty("schema", "schema-minimal-atomic-stress.xml")
+ .process(CLOUD_CLIENT);
+
waitForRecoveriesToFinish(CLOUD_CLIENT);
for (JettySolrRunner jetty : cluster.getJettySolrRunners()) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorCloud.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorCloud.java b/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorCloud.java
index b599244..f5dead9 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorCloud.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorCloud.java
@@ -25,21 +25,20 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.List;
-import java.util.Map;
import java.util.Set;
-import org.apache.solr.cloud.SolrCloudTestCase;
import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.UpdateRequest;
import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.SolrInputField;
-import org.apache.solr.common.SolrException;
import org.apache.solr.common.ToleratedUpdateError;
import org.apache.solr.common.ToleratedUpdateError.CmdType;
import org.apache.solr.common.cloud.ClusterState;
@@ -50,11 +49,9 @@ import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.SimpleOrderedMap;
-
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -105,7 +102,7 @@ public class TestTolerantUpdateProcessorCloud extends SolrCloudTestCase {
private static final String S_TWO_PRE = "XYZ!";
@BeforeClass
- private static void createMiniSolrCloudCluster() throws Exception {
+ public static void createMiniSolrCloudCluster() throws Exception {
final String configName = "solrCloudCollectionConfig";
final File configDir = new File(TEST_HOME() + File.separator + "collection1" + File.separator + "conf");
@@ -114,16 +111,14 @@ public class TestTolerantUpdateProcessorCloud extends SolrCloudTestCase {
.addConfig(configName, configDir.toPath())
.configure();
assertSpinLoopAllJettyAreRunning(cluster);
-
- Map<String, String> collectionProperties = new HashMap<>();
- collectionProperties.put("config", "solrconfig-distrib-update-processor-chains.xml");
- collectionProperties.put("schema", "schema15.xml"); // string id for doc routing prefix
- assertNotNull(cluster.createCollection(COLLECTION_NAME, NUM_SHARDS, REPLICATION_FACTOR,
- configName, null, null, collectionProperties));
-
CLOUD_CLIENT = cluster.getSolrClient();
CLOUD_CLIENT.setDefaultCollection(COLLECTION_NAME);
+
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, configName, NUM_SHARDS, REPLICATION_FACTOR)
+ .withProperty("config", "solrconfig-distrib-update-processor-chains.xml")
+ .withProperty("schema", "schema15.xml") // string id for doc routing prefix
+ .process(CLOUD_CLIENT);
ZkStateReader zkStateReader = CLOUD_CLIENT.getZkStateReader();
AbstractDistribZkTestBase.waitForRecoveriesToFinish(COLLECTION_NAME, zkStateReader, true, true, 330);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorRandomCloud.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorRandomCloud.java b/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorRandomCloud.java
index 0e8e4dc..c60c22b 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorRandomCloud.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestTolerantUpdateProcessorRandomCloud.java
@@ -16,15 +16,6 @@
*/
package org.apache.solr.cloud;
-import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.addErr;
-import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertUpdateTolerantErrors;
-import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.delIErr;
-import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.delQErr;
-import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.f;
-import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.update;
-import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_PARAM;
-import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_START;
-
import java.io.File;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
@@ -42,6 +33,7 @@ import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.UpdateRequest;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.client.solrj.response.UpdateResponse;
@@ -57,6 +49,15 @@ import org.junit.BeforeClass;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.addErr;
+import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertUpdateTolerantErrors;
+import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.delIErr;
+import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.delQErr;
+import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.f;
+import static org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.update;
+import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_PARAM;
+import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_START;
+
/**
* Test of TolerantUpdateProcessor using a randomized MiniSolrCloud.
* Reuses some utility methods in {@link TestTolerantUpdateProcessorCloud}
@@ -81,7 +82,7 @@ public class TestTolerantUpdateProcessorRandomCloud extends SolrCloudTestCase {
private static List<HttpSolrClient> NODE_CLIENTS;
@BeforeClass
- private static void createMiniSolrCloudCluster() throws Exception {
+ public static void createMiniSolrCloudCluster() throws Exception {
final String configName = "solrCloudCollectionConfig";
final File configDir = new File(TEST_HOME() + File.separator + "collection1" + File.separator + "conf");
@@ -102,13 +103,13 @@ public class TestTolerantUpdateProcessorRandomCloud extends SolrCloudTestCase {
collectionProperties.put("config", "solrconfig-distrib-update-processor-chains.xml");
collectionProperties.put("schema", "schema15.xml"); // string id
-
- assertNotNull(cluster.createCollection(COLLECTION_NAME, numShards, repFactor,
- configName, null, null, collectionProperties));
-
CLOUD_CLIENT = cluster.getSolrClient();
CLOUD_CLIENT.setDefaultCollection(COLLECTION_NAME);
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, configName, numShards, repFactor)
+ .setProperties(collectionProperties)
+ .process(CLOUD_CLIENT);
+
if (NODE_CLIENTS != null) {
for (HttpSolrClient client : NODE_CLIENTS) {
client.close();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/core/BlobRepositoryCloudTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/core/BlobRepositoryCloudTest.java b/solr/core/src/test/org/apache/solr/core/BlobRepositoryCloudTest.java
index 40f5792..ad3c8db 100644
--- a/solr/core/src/test/org/apache/solr/core/BlobRepositoryCloudTest.java
+++ b/solr/core/src/test/org/apache/solr/core/BlobRepositoryCloudTest.java
@@ -25,6 +25,7 @@ import java.util.HashMap;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.cloud.SolrCloudTestCase;
import org.apache.solr.common.SolrDocumentList;
@@ -45,14 +46,17 @@ public class BlobRepositoryCloudTest extends SolrCloudTestCase {
.configure();
// Thread.sleep(2000);
HashMap<String, String> params = new HashMap<>();
- cluster.createCollection(".system", 1, 1, null, params);
+ CollectionAdminRequest.createCollection(".system", null, 1, 1)
+ .process(cluster.getSolrClient());
// test component will fail if it cant' find a blob with this data by this name
TestBlobHandler.postData(cluster.getSolrClient(), findLiveNodeURI(), "testResource", ByteBuffer.wrap("foo,bar\nbaz,bam".getBytes(StandardCharsets.UTF_8)));
// Thread.sleep(2000);
// if these don't load we probably failed to post the blob above
- cluster.createCollection("col1", 1, 1, "configname", params);
- cluster.createCollection("col2", 1, 1, "configname", params);
-// Thread.sleep(2000);
+ CollectionAdminRequest.createCollection("col1", "configname", 1, 1)
+ .process(cluster.getSolrClient());
+ CollectionAdminRequest.createCollection("col2", "configname", 1, 1)
+ .process(cluster.getSolrClient());
+
SolrInputDocument document = new SolrInputDocument();
document.addField("id", "1");
document.addField("text", "col1");
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/handler/PingRequestHandlerTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/handler/PingRequestHandlerTest.java b/solr/core/src/test/org/apache/solr/handler/PingRequestHandlerTest.java
index b95a5f6..96a01c3 100644
--- a/solr/core/src/test/org/apache/solr/handler/PingRequestHandlerTest.java
+++ b/solr/core/src/test/org/apache/solr/handler/PingRequestHandlerTest.java
@@ -24,6 +24,7 @@ import org.apache.commons.io.FileUtils;
import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.SolrPing;
import org.apache.solr.client.solrj.response.SolrPingResponse;
import org.apache.solr.cloud.MiniSolrCloudCluster;
@@ -189,10 +190,10 @@ public class PingRequestHandlerTest extends SolrTestCaseJ4 {
// create collection
String collectionName = "testSolrCloudCollection";
String configName = "solrCloudCollectionConfig";
- File configDir = new File(SolrTestCaseJ4.TEST_HOME() + File.separator + "collection1" + File.separator + "conf");
- miniCluster.uploadConfigDir(configDir, configName);
- miniCluster.createCollection(collectionName, NUM_SHARDS, REPLICATION_FACTOR, configName, null);
-
+ miniCluster.uploadConfigSet(SolrTestCaseJ4.TEST_PATH().resolve("collection1").resolve("conf"), configName);
+ CollectionAdminRequest.createCollection(collectionName, configName, NUM_SHARDS, REPLICATION_FACTOR)
+ .process(miniCluster.getSolrClient());
+
// Send distributed and non-distributed ping query
SolrPingWithDistrib reqDistrib = new SolrPingWithDistrib();
reqDistrib.setDistrib(true);
@@ -206,9 +207,6 @@ public class PingRequestHandlerTest extends SolrTestCaseJ4 {
assertEquals(0, rsp.getStatus());
assertTrue(rsp.getResponseHeader().getBooleanArg(("zkConnected")));
- // delete the collection we created earlier
- miniCluster.deleteCollection(collectionName);
-
}
finally {
miniCluster.shutdown();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/handler/TestReqParamsAPI.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/handler/TestReqParamsAPI.java b/solr/core/src/test/org/apache/solr/handler/TestReqParamsAPI.java
index 5c2f67d..d594e48 100644
--- a/solr/core/src/test/org/apache/solr/handler/TestReqParamsAPI.java
+++ b/solr/core/src/test/org/apache/solr/handler/TestReqParamsAPI.java
@@ -24,6 +24,7 @@ import java.util.function.Predicate;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.cloud.AbstractFullDistribZkTestBase;
import org.apache.solr.cloud.SolrCloudTestCase;
import org.apache.solr.common.cloud.DocCollection;
@@ -57,7 +58,8 @@ public class TestReqParamsAPI extends SolrCloudTestCase {
configureCluster(2)
.addConfig("conf1", TEST_PATH().resolve("configsets").resolve("cloud-managed").resolve("conf"))
.configure();
- cluster.createCollection(COLL_NAME, 1, 2, "conf1", null);
+ CollectionAdminRequest.createCollection(COLL_NAME, "conf1", 1, 2)
+ .process(cluster.getSolrClient());
}
@Test
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/handler/component/SearchHandlerTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/handler/component/SearchHandlerTest.java b/solr/core/src/test/org/apache/solr/handler/component/SearchHandlerTest.java
index 70744e2..7501651 100644
--- a/solr/core/src/test/org/apache/solr/handler/component/SearchHandlerTest.java
+++ b/solr/core/src/test/org/apache/solr/handler/component/SearchHandlerTest.java
@@ -16,13 +16,13 @@
*/
package org.apache.solr.handler.component;
-import java.io.File;
import java.util.ArrayList;
import java.util.List;
import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.QueryRequest;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.cloud.MiniSolrCloudCluster;
@@ -122,19 +122,14 @@ public class SearchHandlerTest extends SolrTestCaseJ4
// create collection
String collectionName = "testSolrCloudCollection";
String configName = "solrCloudCollectionConfig";
- File configDir = new File(SolrTestCaseJ4.TEST_HOME() + File.separator + "collection1" + File.separator + "conf");
- miniCluster.uploadConfigDir(configDir, configName);
- miniCluster.createCollection(collectionName, 2, 2, configName, null);
-
-
+ miniCluster.uploadConfigSet(SolrTestCaseJ4.TEST_PATH().resolve("collection1/conf"), configName);
+
+ CollectionAdminRequest.createCollection(collectionName, configName, 2, 2)
+ .process(miniCluster.getSolrClient());
QueryRequest req = new QueryRequest();
QueryResponse rsp = req.process(cloudSolrClient, collectionName);
assertTrue(rsp.getResponseHeader().getBooleanArg("zkConnected"));
-
-
- // delete the collection we created earlier
- miniCluster.deleteCollection(collectionName);
}
finally {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/response/transform/TestSubQueryTransformerDistrib.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/response/transform/TestSubQueryTransformerDistrib.java b/solr/core/src/test/org/apache/solr/response/transform/TestSubQueryTransformerDistrib.java
index 0273a85..c417272 100644
--- a/solr/core/src/test/org/apache/solr/response/transform/TestSubQueryTransformerDistrib.java
+++ b/solr/core/src/test/org/apache/solr/response/transform/TestSubQueryTransformerDistrib.java
@@ -29,6 +29,7 @@ import java.util.Random;
import org.apache.solr.SolrTestCaseJ4.SuppressSSL;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.ContentStreamUpdateRequest;
import org.apache.solr.client.solrj.request.QueryRequest;
import org.apache.solr.client.solrj.response.QueryResponse;
@@ -64,13 +65,16 @@ public class TestSubQueryTransformerDistrib extends SolrCloudTestCase {
int shards = 2;
int replicas = 2 ;
- assertNotNull(cluster.createCollection(people, shards, replicas,
- configName,
- collectionProperties));
-
- assertNotNull(cluster.createCollection(depts, shards, replicas,
- configName, collectionProperties));
-
+ CollectionAdminRequest.createCollection(people, configName, shards, replicas)
+ .withProperty("config", "solrconfig-doctransformers.xml")
+ .withProperty("schema", "schema-docValuesJoin.xml")
+ .process(cluster.getSolrClient());
+
+ CollectionAdminRequest.createCollection(depts, configName, shards, replicas)
+ .withProperty("config", "solrconfig-doctransformers.xml")
+ .withProperty("schema", "schema-docValuesJoin.xml")
+ .process(cluster.getSolrClient());
+
CloudSolrClient client = cluster.getSolrClient();
client.setDefaultCollection(people);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/schema/TestManagedSchemaAPI.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/schema/TestManagedSchemaAPI.java b/solr/core/src/test/org/apache/solr/schema/TestManagedSchemaAPI.java
index e426e3b..14f1525 100644
--- a/solr/core/src/test/org/apache/solr/schema/TestManagedSchemaAPI.java
+++ b/solr/core/src/test/org/apache/solr/schema/TestManagedSchemaAPI.java
@@ -50,7 +50,8 @@ public class TestManagedSchemaAPI extends SolrCloudTestCase {
@Test
public void test() throws Exception {
String collection = "testschemaapi";
- cluster.createCollection(collection, 1, 2, "conf1", null);
+ CollectionAdminRequest.createCollection(collection, "conf1", 1, 2)
+ .process(cluster.getSolrClient());
testReloadAndAddSimple(collection);
testAddFieldAndDocument(collection);
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/search/join/BlockJoinFacetDistribTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/search/join/BlockJoinFacetDistribTest.java b/solr/core/src/test/org/apache/solr/search/join/BlockJoinFacetDistribTest.java
index 70f95a7..2bd30fa 100644
--- a/solr/core/src/test/org/apache/solr/search/join/BlockJoinFacetDistribTest.java
+++ b/solr/core/src/test/org/apache/solr/search/join/BlockJoinFacetDistribTest.java
@@ -30,10 +30,10 @@ import java.util.Map;
import java.util.Set;
import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.response.FacetField;
import org.apache.solr.client.solrj.response.FacetField.Count;
import org.apache.solr.client.solrj.response.QueryResponse;
-import org.apache.solr.cloud.AbstractDistribZkTestBase;
import org.apache.solr.cloud.SolrCloudTestCase;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.ModifiableSolrParams;
@@ -64,13 +64,10 @@ public class BlockJoinFacetDistribTest extends SolrCloudTestCase{
int shards = 3;
int replicas = 2 ;
- assertNotNull(cluster.createCollection(collection, shards, replicas,
- configName,
- collectionProperties));
-
- AbstractDistribZkTestBase.waitForRecoveriesToFinish(collection,
- cluster.getSolrClient().getZkStateReader(), false, true, 30);
-
+ CollectionAdminRequest.createCollection(collection, configName, shards, replicas)
+ .setProperties(collectionProperties)
+ .process(cluster.getSolrClient());
+
}
final static List<String> colors = Arrays.asList("red","blue","brown","white","black","yellow","cyan","magenta","blur",
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/search/stats/TestDistribIDF.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/search/stats/TestDistribIDF.java b/solr/core/src/test/org/apache/solr/search/stats/TestDistribIDF.java
index f7f7f30..f3bcc16 100644
--- a/solr/core/src/test/org/apache/solr/search/stats/TestDistribIDF.java
+++ b/solr/core/src/test/org/apache/solr/search/stats/TestDistribIDF.java
@@ -16,7 +16,6 @@
*/
package org.apache.solr.search.stats;
-import java.io.File;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
@@ -59,10 +58,8 @@ public class TestDistribIDF extends SolrTestCaseJ4 {
// set some system properties for use by tests
System.setProperty("solr.test.sys.prop1", "propone");
System.setProperty("solr.test.sys.prop2", "proptwo");
- File configDir = getFile("solr").toPath().resolve("collection1/conf").toFile();
- solrCluster.uploadConfigDir(configDir, "conf1");
- configDir = getFile("solr").toPath().resolve("configsets/configset-2/conf").toFile();
- solrCluster.uploadConfigDir(configDir, "conf2");
+ solrCluster.uploadConfigSet(TEST_PATH().resolve("collection1/conf"), "conf1");
+ solrCluster.uploadConfigSet(configset("configset-2"), "conf2");
}
@Override
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/solrj/src/java/org/apache/solr/client/solrj/request/CollectionAdminRequest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/request/CollectionAdminRequest.java b/solr/solrj/src/java/org/apache/solr/client/solrj/request/CollectionAdminRequest.java
index af5d74b..c1f8261 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/request/CollectionAdminRequest.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/request/CollectionAdminRequest.java
@@ -18,6 +18,7 @@ package org.apache.solr.client.solrj.request;
import java.io.IOException;
import java.util.Collection;
+import java.util.Map;
import java.util.Optional;
import java.util.Properties;
import java.util.UUID;
@@ -460,6 +461,19 @@ public abstract class CollectionAdminRequest<T extends CollectionAdminResponse>
return this;
}
+ public Create setProperties(Map<String, String> properties) {
+ this.properties = new Properties();
+ this.properties.putAll(properties);
+ return this;
+ }
+
+ public Create withProperty(String key, String value) {
+ if (this.properties == null)
+ this.properties = new Properties();
+ this.properties.setProperty(key, value);
+ return this;
+ }
+
@Override
public SolrParams getParams() {
ModifiableSolrParams params = (ModifiableSolrParams) super.getParams();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/solrj/src/test/org/apache/solr/common/cloud/TestCollectionStateWatchers.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/common/cloud/TestCollectionStateWatchers.java b/solr/solrj/src/test/org/apache/solr/common/cloud/TestCollectionStateWatchers.java
index fca0e35..d8a1f0f 100644
--- a/solr/solrj/src/test/org/apache/solr/common/cloud/TestCollectionStateWatchers.java
+++ b/solr/solrj/src/test/org/apache/solr/common/cloud/TestCollectionStateWatchers.java
@@ -18,7 +18,6 @@
package org.apache.solr.common.cloud;
import java.lang.invoke.MethodHandles;
-import java.util.HashMap;
import java.util.concurrent.Callable;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;
@@ -269,7 +268,8 @@ public class TestCollectionStateWatchers extends SolrCloudTestCase {
@Test
public void testDeletionsTriggerWatches() throws Exception {
- cluster.createCollection("tobedeleted", 1, 1, "config", new HashMap<>());
+ CollectionAdminRequest.createCollection("tobedeleted", "config", 1, 1)
+ .process(cluster.getSolrClient());
Future<Boolean> future = waitInBackground("tobedeleted", MAX_WAIT_TIMEOUT, TimeUnit.SECONDS, (l, c) -> c == null);
CollectionAdminRequest.deleteCollection("tobedeleted").process(cluster.getSolrClient());
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
----------------------------------------------------------------------
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
index 7e809c7..9301fd0 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
@@ -393,12 +393,25 @@ public class MiniSolrCloudCluster {
jetty.stop();
return jetty;
}
-
+
+ /**
+ * @deprecated Use {@link #uploadConfigSet(Path, String)}
+ */
+ @Deprecated
public void uploadConfigDir(File configDir, String configName) throws IOException, KeeperException, InterruptedException {
+ uploadConfigSet(configDir.toPath(), configName);
+ }
+
+ /**
+ * Upload a config set
+ * @param configDir a path to the config set to upload
+ * @param configName the name to give the configset
+ */
+ public void uploadConfigSet(Path configDir, String configName) throws IOException, KeeperException, InterruptedException {
try(SolrZkClient zkClient = new SolrZkClient(zkServer.getZkAddress(),
AbstractZkTestCase.TIMEOUT, AbstractZkTestCase.TIMEOUT, null)) {
ZkConfigManager manager = new ZkConfigManager(zkClient);
- manager.uploadConfigDir(configDir.toPath(), configName);
+ manager.uploadConfigDir(configDir, configName);
}
}
@@ -410,13 +423,21 @@ public class MiniSolrCloudCluster {
}
}
}
-
- public NamedList<Object> createCollection(String name, int numShards, int replicationFactor,
+
+ /**
+ * @deprecated Use {@link CollectionAdminRequest#createCollection(String, String, int, int)}
+ */
+ @Deprecated
+ public NamedList<Object> createCollection(String name, int numShards, int replicationFactor,
String configName, Map<String, String> collectionProperties) throws SolrServerException, IOException {
return createCollection(name, numShards, replicationFactor, configName, null, null, collectionProperties);
}
- public NamedList<Object> createCollection(String name, int numShards, int replicationFactor,
+ /**
+ * @deprecated Use {@link CollectionAdminRequest#createCollection(String, String, int, int)}
+ */
+ @Deprecated
+ public NamedList<Object> createCollection(String name, int numShards, int replicationFactor,
String configName, String createNodeSet, String asyncId, Map<String, String> collectionProperties) throws SolrServerException, IOException {
final ModifiableSolrParams params = new ModifiableSolrParams();
params.set(CoreAdminParams.ACTION, CollectionAction.CREATE.name());
@@ -439,6 +460,10 @@ public class MiniSolrCloudCluster {
return makeCollectionsRequest(params);
}
+ /**
+ * @deprecated use {@link CollectionAdminRequest#deleteCollection(String)}
+ */
+ @Deprecated
public NamedList<Object> deleteCollection(String name) throws SolrServerException, IOException {
final ModifiableSolrParams params = new ModifiableSolrParams();
params.set(CoreAdminParams.ACTION, CollectionAction.DELETE.name());
[27/50] [abbrv] lucene-solr:jira/solr-8593: * SOLR-9506: cache
IndexFingerprint for each segment
Posted by kr...@apache.org.
* SOLR-9506: cache IndexFingerprint for each segment
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/bb907a29
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/bb907a29
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/bb907a29
Branch: refs/heads/jira/solr-8593
Commit: bb907a2983b4a7eba8cb4d527a859f1b312bdc79
Parents: b2188f4
Author: Noble Paul <no...@gmail.com>
Authored: Tue Oct 18 19:43:32 2016 +0530
Committer: Noble Paul <no...@gmail.com>
Committed: Tue Oct 18 19:43:32 2016 +0530
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +
.../src/java/org/apache/solr/core/SolrCore.java | 38 ++++++++++
.../apache/solr/search/SolrIndexSearcher.java | 62 +++++++++++++--
.../apache/solr/update/IndexFingerprint.java | 79 ++++++++++++--------
.../cloud/LeaderFailureAfterFreshStartTest.java | 32 --------
.../solr/cloud/PeerSyncReplicationTest.java | 2 +-
.../org/apache/solr/update/PeerSyncTest.java | 20 ++++-
.../solr/cloud/AbstractDistribZkTestBase.java | 31 ++++++++
8 files changed, 193 insertions(+), 73 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/bb907a29/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index be958d9..4d75f88 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -206,6 +206,8 @@ Optimizations
* SOLR-9566: Don't put replicas into recovery when first creating a Collection
(Alan Woodward)
+* SOLR-9506: cache IndexFingerprint for each segment (Pushkar Raste, noble)
+
Other Changes
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/bb907a29/solr/core/src/java/org/apache/solr/core/SolrCore.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/SolrCore.java b/solr/core/src/java/org/apache/solr/core/SolrCore.java
index 2827f03..e47c8b6 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrCore.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrCore.java
@@ -58,6 +58,7 @@ import org.apache.lucene.codecs.Codec;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexDeletionPolicy;
import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.IOContext;
@@ -127,6 +128,7 @@ import org.apache.solr.search.stats.LocalStatsCache;
import org.apache.solr.search.stats.StatsCache;
import org.apache.solr.update.DefaultSolrCoreState;
import org.apache.solr.update.DirectUpdateHandler2;
+import org.apache.solr.update.IndexFingerprint;
import org.apache.solr.update.SolrCoreState;
import org.apache.solr.update.SolrCoreState.IndexWriterCloser;
import org.apache.solr.update.SolrIndexWriter;
@@ -149,6 +151,8 @@ import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import com.google.common.collect.MapMaker;
+
import static com.google.common.base.Preconditions.checkNotNull;
import static org.apache.solr.common.params.CommonParams.NAME;
import static org.apache.solr.common.params.CommonParams.PATH;
@@ -200,6 +204,8 @@ public final class SolrCore implements SolrInfoMBean, Closeable {
private final ReentrantLock snapshotDelLock; // A lock instance to guard against concurrent deletions.
public Date getStartTimeStamp() { return startTime; }
+
+ private final Map<Object, IndexFingerprint> perSegmentFingerprintCache = new MapMaker().weakKeys().makeMap();
public long getStartNanoTime() {
return startNanoTime;
@@ -1589,6 +1595,38 @@ public final class SolrCore implements SolrInfoMBean, Closeable {
}
/**
+ * Computes fingerprint of a segment and caches it only if all the version in segment are included in the fingerprint.
+ * We can't use computeIfAbsent as caching is conditional (as described above)
+ * There is chance that two threads may compute fingerprint on the same segment. It might be OK to do so rather than locking entire map.
+ * @param searcher searcher that includes specified LeaderReaderContext
+ * @param ctx LeafReaderContext of a segment to compute fingerprint of
+ * @param maxVersion maximum version number to consider for fingerprint computation
+ * @return IndexFingerprint of the segment
+ * @throws IOException Can throw IOException
+ */
+ public IndexFingerprint getIndexFingerprint(SolrIndexSearcher searcher, LeafReaderContext ctx, long maxVersion)
+ throws IOException {
+ IndexFingerprint f = null;
+ f = perSegmentFingerprintCache.get(ctx.reader().getCoreCacheKey()) ;
+ // fingerprint is either not cached or
+ // we want fingerprint only up to a version less than maxVersionEncountered in the segment
+ if(f == null || (f.getMaxInHash() > maxVersion)) {
+ log.debug("IndexFingerprint cache miss for searcher:{} reader:{} readerHash:{} maxVersion:{}", searcher, ctx.reader() , ctx.reader().hashCode(), maxVersion);
+ f = IndexFingerprint.getFingerprint(searcher, ctx, maxVersion);
+ // cache fingerprint for the segment only if all the versions in the segment are included in the fingerprint
+ if(f.getMaxVersionEncountered() == f.getMaxInHash()) {
+ log.info("Caching fingerprint for searcher:{} leafReaderContext:{} mavVersion:{}", searcher, ctx, maxVersion);
+ perSegmentFingerprintCache.put(ctx.reader().getCoreCacheKey(), f);
+ }
+
+ } else {
+ log.debug("IndexFingerprint cache hit for searcher:{} reader:{} readerHash:{} maxVersion:{}", searcher, ctx.reader(), ctx.reader().hashCode(), maxVersion);
+ }
+ log.debug("Cache Size: {}, Segments Size:{}", perSegmentFingerprintCache.size(), searcher.getTopReaderContext().leaves().size());
+ return f;
+ }
+
+ /**
* Returns the current registered searcher with its reference count incremented, or null if none are registered.
*/
public RefCounted<SolrIndexSearcher> getRegisteredSearcher() {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/bb907a29/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java b/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
index 933477b..59797b9 100644
--- a/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
+++ b/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
@@ -42,22 +42,50 @@ import org.apache.lucene.document.Document;
import org.apache.lucene.document.DocumentStoredFieldVisitor;
import org.apache.lucene.document.LazyDocument;
import org.apache.lucene.index.*;
-import org.apache.lucene.index.NumericDocValues;
import org.apache.lucene.index.StoredFieldVisitor.Status;
-import org.apache.lucene.search.*;
+import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.CollectionStatistics;
+import org.apache.lucene.search.Collector;
+import org.apache.lucene.search.ConstantScoreQuery;
+import org.apache.lucene.search.DocIdSet;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.EarlyTerminatingSortingCollector;
+import org.apache.lucene.search.Explanation;
+import org.apache.lucene.search.FieldDoc;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.LeafCollector;
+import org.apache.lucene.search.MatchAllDocsQuery;
+import org.apache.lucene.search.MultiCollector;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.Scorer;
+import org.apache.lucene.search.SimpleCollector;
+import org.apache.lucene.search.Sort;
+import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.TermStatistics;
+import org.apache.lucene.search.TimeLimitingCollector;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.search.TopDocsCollector;
+import org.apache.lucene.search.TopFieldCollector;
+import org.apache.lucene.search.TopFieldDocs;
+import org.apache.lucene.search.TopScoreDocCollector;
+import org.apache.lucene.search.TotalHitCountCollector;
+import org.apache.lucene.search.Weight;
import org.apache.lucene.store.Directory;
import org.apache.lucene.util.Bits;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.FixedBitSet;
import org.apache.solr.common.SolrDocumentBase;
-import org.apache.solr.common.SolrException.ErrorCode;
import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.SimpleOrderedMap;
-import org.apache.solr.core.DirectoryFactory.DirContext;
import org.apache.solr.core.DirectoryFactory;
+import org.apache.solr.core.DirectoryFactory.DirContext;
import org.apache.solr.core.SolrConfig;
import org.apache.solr.core.SolrCore;
import org.apache.solr.core.SolrInfoMBean;
@@ -249,7 +277,7 @@ public class SolrIndexSearcher extends IndexSearcher implements Closeable, SolrI
this.path = path;
this.directoryFactory = directoryFactory;
- this.reader = (DirectoryReader) super.readerContext.reader();
+ this.reader = (DirectoryReader) super.getTopReaderContext().reader();
this.rawReader = r;
this.leafReader = SlowCompositeReaderWrapper.wrap(this.reader);
this.core = core;
@@ -2413,12 +2441,11 @@ public class SolrIndexSearcher extends IndexSearcher implements Closeable, SolrI
* gets a cached version of the IndexFingerprint for this searcher
**/
public IndexFingerprint getIndexFingerprint(long maxVersion) throws IOException {
- final SolrIndexSearcher searcher = this;
final AtomicReference<IOException> exception = new AtomicReference<>();
try {
return maxVersionFingerprintCache.computeIfAbsent(maxVersion, key -> {
try {
- return IndexFingerprint.getFingerprint(searcher, key);
+ return computeFromPerSegmentIndexFingerprint(maxVersion);
} catch (IOException e) {
exception.set(e);
return null;
@@ -2429,6 +2456,27 @@ public class SolrIndexSearcher extends IndexSearcher implements Closeable, SolrI
}
}
+ private IndexFingerprint computeFromPerSegmentIndexFingerprint(long maxVersion) throws IOException {
+ final SolrIndexSearcher searcher = this;
+ final AtomicReference<IOException> exception = new AtomicReference<>();
+ try {
+ return searcher.getTopReaderContext().leaves().stream()
+ .map(ctx -> {
+ try {
+ return searcher.getCore().getIndexFingerprint(searcher, ctx, maxVersion);
+ } catch (IOException e) {
+ exception.set(e);
+ return null;
+ }
+ })
+ .filter(java.util.Objects::nonNull)
+ .reduce(new IndexFingerprint(maxVersion), IndexFingerprint::reduce);
+ } finally {
+ if (exception.get() != null) throw exception.get();
+ }
+ }
+
+
/////////////////////////////////////////////////////////////////////
// SolrInfoMBean stuff: Statistics and Module Info
/////////////////////////////////////////////////////////////////////
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/bb907a29/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java b/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java
index 877ef03..31adf07 100644
--- a/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java
+++ b/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java
@@ -52,6 +52,14 @@ public class IndexFingerprint implements MapSerializable {
private long numDocs;
private long maxDoc;
+ public IndexFingerprint() {
+ // default constructor
+ }
+
+ public IndexFingerprint (long maxVersionSpecified) {
+ this.maxVersionSpecified = maxVersionSpecified;
+ }
+
public long getMaxVersionSpecified() {
return maxVersionSpecified;
}
@@ -82,53 +90,63 @@ public class IndexFingerprint implements MapSerializable {
/** Opens a new realtime searcher and returns it's (possibly cached) fingerprint */
public static IndexFingerprint getFingerprint(SolrCore core, long maxVersion) throws IOException {
+ RTimer timer = new RTimer();
core.getUpdateHandler().getUpdateLog().openRealtimeSearcher();
RefCounted<SolrIndexSearcher> newestSearcher = core.getUpdateHandler().getUpdateLog().uhandler.core.getRealtimeSearcher();
try {
- return newestSearcher.get().getIndexFingerprint(maxVersion);
+ IndexFingerprint f = newestSearcher.get().getIndexFingerprint(maxVersion);
+ final double duration = timer.stop();
+ log.debug("IndexFingerprint time : {} result:{}" ,duration, f);
+ return f;
} finally {
if (newestSearcher != null) {
newestSearcher.decref();
}
}
+
}
-
- /** Calculates an index fingerprint */
- public static IndexFingerprint getFingerprint(SolrIndexSearcher searcher, long maxVersion) throws IOException {
- RTimer timer = new RTimer();
-
+
+ public static IndexFingerprint getFingerprint(SolrIndexSearcher searcher, LeafReaderContext ctx, Long maxVersion)
+ throws IOException {
SchemaField versionField = VersionInfo.getAndCheckVersionField(searcher.getSchema());
-
- IndexFingerprint f = new IndexFingerprint();
- f.maxVersionSpecified = maxVersion;
- f.maxDoc = searcher.maxDoc();
-
- // TODO: this could be parallelized, or even cached per-segment if performance becomes an issue
ValueSource vs = versionField.getType().getValueSource(versionField, null);
Map funcContext = ValueSource.newContext(searcher);
vs.createWeight(funcContext, searcher);
- for (LeafReaderContext ctx : searcher.getTopReaderContext().leaves()) {
- int maxDoc = ctx.reader().maxDoc();
- f.numDocs += ctx.reader().numDocs();
- Bits liveDocs = ctx.reader().getLiveDocs();
- FunctionValues fv = vs.getValues(funcContext, ctx);
- for (int doc = 0; doc < maxDoc; doc++) {
- if (liveDocs != null && !liveDocs.get(doc)) continue;
- long v = fv.longVal(doc);
- f.maxVersionEncountered = Math.max(v, f.maxVersionEncountered);
- if (v <= f.maxVersionSpecified) {
- f.maxInHash = Math.max(v, f.maxInHash);
- f.versionsHash += Hash.fmix64(v);
- f.numVersions++;
- }
+
+ IndexFingerprint f = new IndexFingerprint();
+ f.maxVersionSpecified = maxVersion;
+ f.maxDoc = ctx.reader().maxDoc();
+ f.numDocs = ctx.reader().numDocs();
+
+ int maxDoc = ctx.reader().maxDoc();
+ Bits liveDocs = ctx.reader().getLiveDocs();
+ FunctionValues fv = vs.getValues(funcContext, ctx);
+ for (int doc = 0; doc < maxDoc; doc++) {
+ if (liveDocs != null && !liveDocs.get(doc)) continue;
+ long v = fv.longVal(doc);
+ f.maxVersionEncountered = Math.max(v, f.maxVersionEncountered);
+ if (v <= f.maxVersionSpecified) {
+ f.maxInHash = Math.max(v, f.maxInHash);
+ f.versionsHash += Hash.fmix64(v);
+ f.numVersions++;
}
}
-
- final double duration = timer.stop();
- log.info("IndexFingerprint millis:" + duration + " result:" + f);
-
+
return f;
}
+
+
+ public static IndexFingerprint reduce(IndexFingerprint acc, IndexFingerprint f2) {
+ // acc should have maxVersionSpecified already set in it using IndexFingerprint(long maxVersionSpecified) constructor
+ acc.maxDoc = Math.max(acc.maxDoc, f2.maxDoc);
+ acc.numDocs += f2.numDocs;
+ acc.maxVersionEncountered = Math.max(acc.maxVersionEncountered, f2.maxVersionEncountered);
+ acc.maxInHash = Math.max(acc.maxInHash, f2.maxInHash);
+ acc.versionsHash += f2.versionsHash;
+ acc.numVersions += f2.numVersions;
+
+ return acc;
+ }
/** returns 0 for equal, negative if f1 is less recent than f2, positive if more recent */
public static int compare(IndexFingerprint f1, IndexFingerprint f2) {
@@ -200,4 +218,5 @@ public class IndexFingerprint implements MapSerializable {
public String toString() {
return toMap(new LinkedHashMap<>()).toString();
}
+
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/bb907a29/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java b/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
index 348532c..ef21386 100644
--- a/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
@@ -29,7 +29,6 @@ import java.util.stream.Collectors;
import org.apache.commons.lang.RandomStringUtils;
import org.apache.lucene.util.LuceneTestCase.Slow;
import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.request.UpdateRequest;
import org.apache.solr.cloud.ZkTestServer.LimitViolationAction;
import org.apache.solr.common.SolrInputDocument;
@@ -37,10 +36,8 @@ import org.apache.solr.common.cloud.ClusterState;
import org.apache.solr.common.cloud.DocCollection;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.Slice;
-import org.apache.solr.common.cloud.Slice.State;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.params.ModifiableSolrParams;
-import org.apache.solr.core.Diagnostics;
import org.apache.solr.handler.ReplicationHandler;
import org.junit.Test;
import org.slf4j.Logger;
@@ -197,35 +194,6 @@ public class LeaderFailureAfterFreshStartTest extends AbstractFullDistribZkTestB
}
- static void waitForNewLeader(CloudSolrClient cloudClient, String shardName, Replica oldLeader, int maxWaitInSecs)
- throws Exception {
- log.info("Will wait for a node to become leader for {} secs", maxWaitInSecs);
- boolean waitForLeader = true;
- int i = 0;
- ZkStateReader zkStateReader = cloudClient.getZkStateReader();
- zkStateReader.forceUpdateCollection(DEFAULT_COLLECTION);
-
- while(waitForLeader) {
- ClusterState clusterState = zkStateReader.getClusterState();
- DocCollection coll = clusterState.getCollection("collection1");
- Slice slice = coll.getSlice(shardName);
- if(slice.getLeader() != oldLeader && slice.getState() == State.ACTIVE) {
- log.info("New leader got elected in {} secs", i);
- break;
- }
-
- if(i == maxWaitInSecs) {
- Diagnostics.logThreadDumps("Could not find new leader in specified timeout");
- zkStateReader.getZkClient().printLayoutToStdOut();
- fail("Could not find new leader even after waiting for " + maxWaitInSecs + "secs");
- }
-
- i++;
- Thread.sleep(1000);
- }
- }
-
-
private void waitTillNodesActive() throws Exception {
for (int i = 0; i < 60; i++) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/bb907a29/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java b/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
index 3ded7d2..e00ea3c 100644
--- a/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
@@ -149,7 +149,7 @@ public class PeerSyncReplicationTest extends AbstractFullDistribZkTestBase {
log.info("Now shutting down initial leader");
forceNodeFailures(singletonList(initialLeaderJetty));
log.info("Updating mappings from zk");
- LeaderFailureAfterFreshStartTest.waitForNewLeader(cloudClient, "shard1", (Replica) initialLeaderJetty.client.info, 15);
+ waitForNewLeader(cloudClient, "shard1", (Replica) initialLeaderJetty.client.info, 15);
updateMappingsFromZk(jettys, clients, true);
assertEquals("PeerSynced node did not become leader", nodePeerSynced, shardToLeaderJetty.get("shard1"));
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/bb907a29/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java b/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java
index 64edd21..60fed8b 100644
--- a/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java
+++ b/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java
@@ -16,6 +16,9 @@
*/
package org.apache.solr.update;
+import java.io.IOException;
+import java.util.Arrays;
+
import org.apache.solr.BaseDistributedSearchTestCase;
import org.apache.solr.SolrTestCaseJ4.SuppressSSL;
import org.apache.solr.client.solrj.SolrClient;
@@ -26,9 +29,6 @@ import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.StrUtils;
import org.junit.Test;
-import java.io.IOException;
-import java.util.Arrays;
-
import static org.apache.solr.update.processor.DistributedUpdateProcessor.DistribPhase;
import static org.apache.solr.update.processor.DistributingUpdateProcessorFactory.DISTRIB_UPDATE_PARAM;
@@ -145,6 +145,20 @@ public class PeerSyncTest extends BaseDistributedSearchTestCase {
assertSync(client1, numVersions, true, shardsArr[0]);
client0.commit(); client1.commit(); queryAndCompare(params("q", "*:*", "sort","_version_ desc"), client0, client1);
+ // Test PeerSync after replica misses delete
+ v = 2500;
+ add(client0, seenLeader, sdoc("id", "2500", "_version_", ++v));
+ add(client1, seenLeader, sdoc("id", "2500", "_version_", v));
+ client0.commit();
+ client1.commit();
+ del(client0, params(DISTRIB_UPDATE_PARAM, FROM_LEADER, "_version_", Long.toString(-++v)), "2500");
+ add(client0, seenLeader, sdoc("id", "2501", "_version_", ++v));
+ add(client1, seenLeader, sdoc("id", "2501", "_version_", v));
+ // Sync should be able to delete the document
+ assertSync(client1, numVersions, true, shardsArr[0]);
+ client0.commit();
+ client1.commit();
+ queryAndCompare(params("q", "*:*", "sort", "_version_ desc"), client0, client1);
//
// Test that handling reorders work when applying docs retrieved from peer
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/bb907a29/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
----------------------------------------------------------------------
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
index 03db71c..d04d996 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
@@ -24,11 +24,14 @@ import java.util.concurrent.atomic.AtomicInteger;
import org.apache.commons.io.FileUtils;
import org.apache.solr.BaseDistributedSearchTestCase;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.common.cloud.ClusterState;
+import org.apache.solr.common.cloud.DocCollection;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.Slice;
import org.apache.solr.common.cloud.SolrZkClient;
import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.cloud.Slice.State;
import org.apache.solr.core.Diagnostics;
import org.apache.solr.core.MockDirectoryFactory;
import org.apache.zookeeper.KeeperException;
@@ -222,6 +225,34 @@ public abstract class AbstractDistribZkTestBase extends BaseDistributedSearchTes
log.info("Collection has disappeared - collection: " + collection);
}
+
+ static void waitForNewLeader(CloudSolrClient cloudClient, String shardName, Replica oldLeader, int maxWaitInSecs)
+ throws Exception {
+ log.info("Will wait for a node to become leader for {} secs", maxWaitInSecs);
+ boolean waitForLeader = true;
+ int i = 0;
+ ZkStateReader zkStateReader = cloudClient.getZkStateReader();
+ zkStateReader.forceUpdateCollection(DEFAULT_COLLECTION);
+
+ while(waitForLeader) {
+ ClusterState clusterState = zkStateReader.getClusterState();
+ DocCollection coll = clusterState.getCollection("collection1");
+ Slice slice = coll.getSlice(shardName);
+ if(slice.getLeader() != oldLeader && slice.getState() == State.ACTIVE) {
+ log.info("New leader got elected in {} secs", i);
+ break;
+ }
+
+ if(i == maxWaitInSecs) {
+ Diagnostics.logThreadDumps("Could not find new leader in specified timeout");
+ zkStateReader.getZkClient().printLayoutToStdOut();
+ fail("Could not find new leader even after waiting for " + maxWaitInSecs + "secs");
+ }
+
+ i++;
+ Thread.sleep(1000);
+ }
+ }
public static void verifyReplicaStatus(ZkStateReader reader, String collection, String shard, String coreNodeName, Replica.State expectedState) throws InterruptedException {
int maxIterations = 100;
[37/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7506:
FastTaxonomyFacetCounts use ConjunctionDISI so cost is in proportion to size
of intersected set of documents
Posted by kr...@apache.org.
LUCENE-7506: FastTaxonomyFacetCounts use ConjunctionDISI so cost is in proportion to size of intersected set of documents
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/d03cc92b
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/d03cc92b
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/d03cc92b
Branch: refs/heads/jira/solr-8593
Commit: d03cc92b222681b5d701a0383d93c2ca5c1a186d
Parents: 731c5f9
Author: Mike McCandless <mi...@apache.org>
Authored: Wed Oct 19 10:04:39 2016 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Wed Oct 19 10:04:39 2016 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 5 +++
.../facet/taxonomy/FastTaxonomyFacetCounts.java | 41 +++++++++-----------
2 files changed, 24 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d03cc92b/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 21ded1a..7105330 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -40,6 +40,11 @@ Optimizations
in the sets of SHOULD and FILTER clauses, or both in MUST/FILTER and MUST_NOT
clauses. (Spyros Kapnissis via Adrien Grand, Uwe Schindler)
+* LUCENE-7506: FastTaxonomyFacetCounts should use CPU in proportion to
+ the size of the intersected set of hits from the query and documents
+ that have a facet value, so sparse faceting works as expected
+ (Adrien Grand via Mike McCandless)
+
Other
* LUCENE-7328: Remove LegacyNumericEncoding from GeoPointField. (Nick Knize)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d03cc92b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java
index 7ad5430..ef96073 100644
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java
+++ b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java
@@ -17,12 +17,14 @@
package org.apache.lucene.facet.taxonomy;
import java.io.IOException;
+import java.util.Arrays;
import java.util.List;
import org.apache.lucene.facet.FacetsCollector.MatchingDocs;
import org.apache.lucene.facet.FacetsCollector;
import org.apache.lucene.facet.FacetsConfig;
import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.search.ConjunctionDISI;
import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.util.BytesRef;
@@ -55,29 +57,24 @@ public class FastTaxonomyFacetCounts extends IntTaxonomyFacets {
continue;
}
- DocIdSetIterator docs = hits.bits.iterator();
+ DocIdSetIterator it = ConjunctionDISI.intersectIterators(Arrays.asList(
+ hits.bits.iterator(), dv));
- int doc;
- while ((doc = docs.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
- if (dv.docID() < doc) {
- dv.advance(doc);
- }
- if (dv.docID() == doc) {
- final BytesRef bytesRef = dv.binaryValue();
- byte[] bytes = bytesRef.bytes;
- int end = bytesRef.offset + bytesRef.length;
- int ord = 0;
- int offset = bytesRef.offset;
- int prev = 0;
- while (offset < end) {
- byte b = bytes[offset++];
- if (b >= 0) {
- prev = ord = ((ord << 7) | b) + prev;
- ++values[ord];
- ord = 0;
- } else {
- ord = (ord << 7) | (b & 0x7F);
- }
+ for (int doc = it.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = it.nextDoc()) {
+ final BytesRef bytesRef = dv.binaryValue();
+ byte[] bytes = bytesRef.bytes;
+ int end = bytesRef.offset + bytesRef.length;
+ int ord = 0;
+ int offset = bytesRef.offset;
+ int prev = 0;
+ while (offset < end) {
+ byte b = bytes[offset++];
+ if (b >= 0) {
+ prev = ord = ((ord << 7) | b) + prev;
+ ++values[ord];
+ ord = 0;
+ } else {
+ ord = (ord << 7) | (b & 0x7F);
}
}
}
[22/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9634: Deprecate
create/deleteCollection methods on MiniSolrCloudCluster
Posted by kr...@apache.org.
SOLR-9634: Deprecate create/deleteCollection methods on MiniSolrCloudCluster
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/67f3f433
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/67f3f433
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/67f3f433
Branch: refs/heads/jira/solr-8593
Commit: 67f3f433e2359aedd3a28bf4c3c01875e4c5d14a
Parents: 20ea535
Author: Alan Woodward <ro...@apache.org>
Authored: Thu Oct 6 11:39:02 2016 +0100
Committer: Alan Woodward <ro...@apache.org>
Committed: Tue Oct 18 12:34:13 2016 +0100
----------------------------------------------------------------------
solr/CHANGES.txt | 7 ++
.../cdcr-source-disabled/conf/schema.xml | 29 ++++++++
.../cdcr-source-disabled/conf/solrconfig.xml | 60 ++++++++++++++++
.../configsets/cdcr-source-disabled/schema.xml | 29 --------
.../cdcr-source-disabled/solrconfig.xml | 60 ----------------
.../solr/configsets/cdcr-source/conf/schema.xml | 29 ++++++++
.../configsets/cdcr-source/conf/solrconfig.xml | 76 ++++++++++++++++++++
.../solr/configsets/cdcr-source/schema.xml | 29 --------
.../solr/configsets/cdcr-source/solrconfig.xml | 76 --------------------
.../solr/configsets/cdcr-target/conf/schema.xml | 29 ++++++++
.../configsets/cdcr-target/conf/solrconfig.xml | 63 ++++++++++++++++
.../solr/configsets/cdcr-target/schema.xml | 29 --------
.../solr/configsets/cdcr-target/solrconfig.xml | 63 ----------------
.../apache/solr/cloud/CdcrBootstrapTest.java | 76 +++++++-------------
...ConcurrentDeleteAndCreateCollectionTest.java | 12 ++--
.../cloud/DistribJoinFromCollectionTest.java | 25 +++----
.../solr/cloud/TestAuthenticationFramework.java | 45 ++++++------
.../solr/cloud/TestCloudDeleteByQuery.java | 15 ++--
.../solr/cloud/TestCloudPseudoReturnFields.java | 21 ++----
.../apache/solr/cloud/TestConfigSetsAPI.java | 13 ++--
.../cloud/TestConfigSetsAPIExclusivity.java | 10 +--
.../solr/cloud/TestConfigSetsAPIZkFailure.java | 2 +-
.../solr/cloud/TestMiniSolrCloudCluster.java | 54 +++++---------
.../solr/cloud/TestMiniSolrCloudClusterSSL.java | 50 ++++++-------
.../apache/solr/cloud/TestRandomFlRTGCloud.java | 40 ++++-------
.../solr/cloud/TestRequestForwarding.java | 4 +-
.../cloud/TestSolrCloudWithKerberosAlt.java | 3 +-
.../TestSolrCloudWithSecureImpersonation.java | 8 +--
.../TestStressCloudBlindAtomicUpdates.java | 35 ++++-----
.../cloud/TestTolerantUpdateProcessorCloud.java | 23 +++---
.../TestTolerantUpdateProcessorRandomCloud.java | 29 ++++----
.../solr/core/BlobRepositoryCloudTest.java | 12 ++--
.../solr/handler/PingRequestHandlerTest.java | 12 ++--
.../apache/solr/handler/TestReqParamsAPI.java | 4 +-
.../handler/component/SearchHandlerTest.java | 15 ++--
.../TestSubQueryTransformerDistrib.java | 18 +++--
.../solr/schema/TestManagedSchemaAPI.java | 3 +-
.../search/join/BlockJoinFacetDistribTest.java | 13 ++--
.../solr/search/stats/TestDistribIDF.java | 7 +-
.../solrj/request/CollectionAdminRequest.java | 14 ++++
.../cloud/TestCollectionStateWatchers.java | 4 +-
.../apache/solr/cloud/MiniSolrCloudCluster.java | 35 +++++++--
42 files changed, 560 insertions(+), 621 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 281140a..e16ca7a 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -86,6 +86,11 @@ prefix, then you will now get an error as these options are incompatible with nu
you may need to update the log configs to use the DEBUG level to get the
same logging messages as before.
+* The create/deleteCollection methods on MiniSolrCloudCluster have been
+ deprecated. Clients should instead use the CollectionAdminRequest API. In
+ addition, MiniSolrCloudCluster#uploadConfigSet(File, String) has been
+ deprecated in favour of #uploadConfigSet(Path, String)
+
New Features
----------------------
* SOLR-5725: facet.method=enum can bypass exact counts calculation with facet.exists=true, it just returns 1 for
@@ -285,6 +290,8 @@ Other Changes
* SOLR-9632: Add MiniSolrCloudCluster#deleteAllCollections() method (Alan Woodward)
+* SOLR-9634: Deprecate collection methods on MiniSolrCloudCluster (Alan Woodward)
+
================== 6.2.1 ==================
Bug Fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/conf/schema.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/conf/schema.xml b/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/conf/schema.xml
new file mode 100644
index 0000000..2897315
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/conf/schema.xml
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<schema name="minimal" version="1.1">
+ <types>
+ <fieldType name="string" class="solr.StrField"/>
+ <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
+ </types>
+ <fields>
+ <field name="id" type="string" indexed="true" stored="true"/>
+ <field name="_version_" type="long" indexed="true" stored="true"/>
+ <dynamicField name="*" type="string" indexed="true" stored="true"/>
+ </fields>
+ <uniqueKey>id</uniqueKey>
+</schema>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/conf/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/conf/solrconfig.xml b/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/conf/solrconfig.xml
new file mode 100644
index 0000000..e63d9a6
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/conf/solrconfig.xml
@@ -0,0 +1,60 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- This is a "kitchen sink" config file that tests can use.
+ When writting a new test, feel free to add *new* items (plugins,
+ config options, etc...) as long as they don't break any existing
+ tests. if you need to test something esoteric please add a new
+ "solrconfig-your-esoteric-purpose.xml" config file.
+
+ Note in particular that this test is used by MinimalSchemaTest so
+ Anything added to this file needs to work correctly even if there
+ is now uniqueKey or defaultSearch Field.
+ -->
+
+<config>
+
+ <dataDir>${solr.data.dir:}</dataDir>
+
+ <directoryFactory name="DirectoryFactory"
+ class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
+ <schemaFactory class="ClassicIndexSchemaFactory"/>
+
+ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
+
+ <updateHandler class="solr.DirectUpdateHandler2">
+ <commitWithin>
+ <softCommit>${solr.commitwithin.softcommit:true}</softCommit>
+ </commitWithin>
+
+ <updateLog>
+ <str name="dir">${solr.ulog.dir:}</str>
+ </updateLog>
+
+ </updateHandler>
+ <requestHandler name="/select" class="solr.SearchHandler">
+ <lst name="defaults">
+ <str name="echoParams">explicit</str>
+ <str name="indent">true</str>
+ <str name="df">text</str>
+ </lst>
+
+ </requestHandler>
+</config>
+
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/schema.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/schema.xml b/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/schema.xml
deleted file mode 100644
index 2897315..0000000
--- a/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/schema.xml
+++ /dev/null
@@ -1,29 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" ?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<schema name="minimal" version="1.1">
- <types>
- <fieldType name="string" class="solr.StrField"/>
- <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
- </types>
- <fields>
- <field name="id" type="string" indexed="true" stored="true"/>
- <field name="_version_" type="long" indexed="true" stored="true"/>
- <dynamicField name="*" type="string" indexed="true" stored="true"/>
- </fields>
- <uniqueKey>id</uniqueKey>
-</schema>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/solrconfig.xml b/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/solrconfig.xml
deleted file mode 100644
index e63d9a6..0000000
--- a/solr/core/src/test-files/solr/configsets/cdcr-source-disabled/solrconfig.xml
+++ /dev/null
@@ -1,60 +0,0 @@
-<?xml version="1.0" ?>
-
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<!-- This is a "kitchen sink" config file that tests can use.
- When writting a new test, feel free to add *new* items (plugins,
- config options, etc...) as long as they don't break any existing
- tests. if you need to test something esoteric please add a new
- "solrconfig-your-esoteric-purpose.xml" config file.
-
- Note in particular that this test is used by MinimalSchemaTest so
- Anything added to this file needs to work correctly even if there
- is now uniqueKey or defaultSearch Field.
- -->
-
-<config>
-
- <dataDir>${solr.data.dir:}</dataDir>
-
- <directoryFactory name="DirectoryFactory"
- class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
- <schemaFactory class="ClassicIndexSchemaFactory"/>
-
- <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
-
- <updateHandler class="solr.DirectUpdateHandler2">
- <commitWithin>
- <softCommit>${solr.commitwithin.softcommit:true}</softCommit>
- </commitWithin>
-
- <updateLog>
- <str name="dir">${solr.ulog.dir:}</str>
- </updateLog>
-
- </updateHandler>
- <requestHandler name="/select" class="solr.SearchHandler">
- <lst name="defaults">
- <str name="echoParams">explicit</str>
- <str name="indent">true</str>
- <str name="df">text</str>
- </lst>
-
- </requestHandler>
-</config>
-
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-source/conf/schema.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-source/conf/schema.xml b/solr/core/src/test-files/solr/configsets/cdcr-source/conf/schema.xml
new file mode 100644
index 0000000..2897315
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/cdcr-source/conf/schema.xml
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<schema name="minimal" version="1.1">
+ <types>
+ <fieldType name="string" class="solr.StrField"/>
+ <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
+ </types>
+ <fields>
+ <field name="id" type="string" indexed="true" stored="true"/>
+ <field name="_version_" type="long" indexed="true" stored="true"/>
+ <dynamicField name="*" type="string" indexed="true" stored="true"/>
+ </fields>
+ <uniqueKey>id</uniqueKey>
+</schema>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-source/conf/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-source/conf/solrconfig.xml b/solr/core/src/test-files/solr/configsets/cdcr-source/conf/solrconfig.xml
new file mode 100644
index 0000000..f2528c3
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/cdcr-source/conf/solrconfig.xml
@@ -0,0 +1,76 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- This is a "kitchen sink" config file that tests can use.
+ When writting a new test, feel free to add *new* items (plugins,
+ config options, etc...) as long as they don't break any existing
+ tests. if you need to test something esoteric please add a new
+ "solrconfig-your-esoteric-purpose.xml" config file.
+
+ Note in particular that this test is used by MinimalSchemaTest so
+ Anything added to this file needs to work correctly even if there
+ is now uniqueKey or defaultSearch Field.
+ -->
+
+<config>
+
+ <dataDir>${solr.data.dir:}</dataDir>
+
+ <directoryFactory name="DirectoryFactory"
+ class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
+
+ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
+
+ <updateRequestProcessorChain name="cdcr-processor-chain">
+ <processor class="solr.CdcrUpdateProcessorFactory"/>
+ <processor class="solr.RunUpdateProcessorFactory"/>
+ </updateRequestProcessorChain>
+
+ <requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
+ <lst name="replica">
+ <str name="zkHost">${cdcr.target.zkHost}</str>
+ <str name="source">cdcr-source</str>
+ <str name="target">cdcr-target</str>
+ </lst>
+ <lst name="replicator">
+ <str name="threadPoolSize">1</str>
+ <str name="schedule">1000</str>
+ <str name="batchSize">1000</str>
+ </lst>
+ <lst name="updateLogSynchronizer">
+ <str name="schedule">1000</str>
+ </lst>
+ </requestHandler>
+
+ <updateHandler class="solr.DirectUpdateHandler2">
+ <updateLog class="solr.CdcrUpdateLog">
+ <str name="dir">${solr.ulog.dir:}</str>
+ </updateLog>
+ </updateHandler>
+
+ <requestHandler name="standard" class="solr.StandardRequestHandler">
+ </requestHandler>
+
+ <requestHandler name="/update" class="solr.UpdateRequestHandler">
+ <lst name="defaults">
+ <str name="update.chain">cdcr-processor-chain</str>
+ </lst>
+ </requestHandler>
+</config>
+
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-source/schema.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-source/schema.xml b/solr/core/src/test-files/solr/configsets/cdcr-source/schema.xml
deleted file mode 100644
index 2897315..0000000
--- a/solr/core/src/test-files/solr/configsets/cdcr-source/schema.xml
+++ /dev/null
@@ -1,29 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" ?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<schema name="minimal" version="1.1">
- <types>
- <fieldType name="string" class="solr.StrField"/>
- <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
- </types>
- <fields>
- <field name="id" type="string" indexed="true" stored="true"/>
- <field name="_version_" type="long" indexed="true" stored="true"/>
- <dynamicField name="*" type="string" indexed="true" stored="true"/>
- </fields>
- <uniqueKey>id</uniqueKey>
-</schema>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-source/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-source/solrconfig.xml b/solr/core/src/test-files/solr/configsets/cdcr-source/solrconfig.xml
deleted file mode 100644
index f2528c3..0000000
--- a/solr/core/src/test-files/solr/configsets/cdcr-source/solrconfig.xml
+++ /dev/null
@@ -1,76 +0,0 @@
-<?xml version="1.0" ?>
-
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<!-- This is a "kitchen sink" config file that tests can use.
- When writting a new test, feel free to add *new* items (plugins,
- config options, etc...) as long as they don't break any existing
- tests. if you need to test something esoteric please add a new
- "solrconfig-your-esoteric-purpose.xml" config file.
-
- Note in particular that this test is used by MinimalSchemaTest so
- Anything added to this file needs to work correctly even if there
- is now uniqueKey or defaultSearch Field.
- -->
-
-<config>
-
- <dataDir>${solr.data.dir:}</dataDir>
-
- <directoryFactory name="DirectoryFactory"
- class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
-
- <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
-
- <updateRequestProcessorChain name="cdcr-processor-chain">
- <processor class="solr.CdcrUpdateProcessorFactory"/>
- <processor class="solr.RunUpdateProcessorFactory"/>
- </updateRequestProcessorChain>
-
- <requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
- <lst name="replica">
- <str name="zkHost">${cdcr.target.zkHost}</str>
- <str name="source">cdcr-source</str>
- <str name="target">cdcr-target</str>
- </lst>
- <lst name="replicator">
- <str name="threadPoolSize">1</str>
- <str name="schedule">1000</str>
- <str name="batchSize">1000</str>
- </lst>
- <lst name="updateLogSynchronizer">
- <str name="schedule">1000</str>
- </lst>
- </requestHandler>
-
- <updateHandler class="solr.DirectUpdateHandler2">
- <updateLog class="solr.CdcrUpdateLog">
- <str name="dir">${solr.ulog.dir:}</str>
- </updateLog>
- </updateHandler>
-
- <requestHandler name="standard" class="solr.StandardRequestHandler">
- </requestHandler>
-
- <requestHandler name="/update" class="solr.UpdateRequestHandler">
- <lst name="defaults">
- <str name="update.chain">cdcr-processor-chain</str>
- </lst>
- </requestHandler>
-</config>
-
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-target/conf/schema.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-target/conf/schema.xml b/solr/core/src/test-files/solr/configsets/cdcr-target/conf/schema.xml
new file mode 100644
index 0000000..2897315
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/cdcr-target/conf/schema.xml
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<schema name="minimal" version="1.1">
+ <types>
+ <fieldType name="string" class="solr.StrField"/>
+ <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
+ </types>
+ <fields>
+ <field name="id" type="string" indexed="true" stored="true"/>
+ <field name="_version_" type="long" indexed="true" stored="true"/>
+ <dynamicField name="*" type="string" indexed="true" stored="true"/>
+ </fields>
+ <uniqueKey>id</uniqueKey>
+</schema>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-target/conf/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-target/conf/solrconfig.xml b/solr/core/src/test-files/solr/configsets/cdcr-target/conf/solrconfig.xml
new file mode 100644
index 0000000..ef24fa4
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/cdcr-target/conf/solrconfig.xml
@@ -0,0 +1,63 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- This is a "kitchen sink" config file that tests can use.
+ When writting a new test, feel free to add *new* items (plugins,
+ config options, etc...) as long as they don't break any existing
+ tests. if you need to test something esoteric please add a new
+ "solrconfig-your-esoteric-purpose.xml" config file.
+
+ Note in particular that this test is used by MinimalSchemaTest so
+ Anything added to this file needs to work correctly even if there
+ is now uniqueKey or defaultSearch Field.
+ -->
+
+<config>
+
+ <dataDir>${solr.data.dir:}</dataDir>
+
+ <directoryFactory name="DirectoryFactory"
+ class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
+
+ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
+
+ <updateRequestProcessorChain name="cdcr-processor-chain">
+ <processor class="solr.CdcrUpdateProcessorFactory"/>
+ <processor class="solr.RunUpdateProcessorFactory"/>
+ </updateRequestProcessorChain>
+
+ <requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
+ </requestHandler>
+
+ <updateHandler class="solr.DirectUpdateHandler2">
+ <updateLog class="solr.CdcrUpdateLog">
+ <str name="dir">${solr.ulog.dir:}</str>
+ </updateLog>
+ </updateHandler>
+
+ <requestHandler name="standard" class="solr.StandardRequestHandler">
+ </requestHandler>
+
+ <requestHandler name="/update" class="solr.UpdateRequestHandler">
+ <lst name="defaults">
+ <str name="update.chain">cdcr-processor-chain</str>
+ </lst>
+ </requestHandler>
+</config>
+
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-target/schema.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-target/schema.xml b/solr/core/src/test-files/solr/configsets/cdcr-target/schema.xml
deleted file mode 100644
index 2897315..0000000
--- a/solr/core/src/test-files/solr/configsets/cdcr-target/schema.xml
+++ /dev/null
@@ -1,29 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" ?>
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-<schema name="minimal" version="1.1">
- <types>
- <fieldType name="string" class="solr.StrField"/>
- <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
- </types>
- <fields>
- <field name="id" type="string" indexed="true" stored="true"/>
- <field name="_version_" type="long" indexed="true" stored="true"/>
- <dynamicField name="*" type="string" indexed="true" stored="true"/>
- </fields>
- <uniqueKey>id</uniqueKey>
-</schema>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test-files/solr/configsets/cdcr-target/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/cdcr-target/solrconfig.xml b/solr/core/src/test-files/solr/configsets/cdcr-target/solrconfig.xml
deleted file mode 100644
index ef24fa4..0000000
--- a/solr/core/src/test-files/solr/configsets/cdcr-target/solrconfig.xml
+++ /dev/null
@@ -1,63 +0,0 @@
-<?xml version="1.0" ?>
-
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<!-- This is a "kitchen sink" config file that tests can use.
- When writting a new test, feel free to add *new* items (plugins,
- config options, etc...) as long as they don't break any existing
- tests. if you need to test something esoteric please add a new
- "solrconfig-your-esoteric-purpose.xml" config file.
-
- Note in particular that this test is used by MinimalSchemaTest so
- Anything added to this file needs to work correctly even if there
- is now uniqueKey or defaultSearch Field.
- -->
-
-<config>
-
- <dataDir>${solr.data.dir:}</dataDir>
-
- <directoryFactory name="DirectoryFactory"
- class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
-
- <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
-
- <updateRequestProcessorChain name="cdcr-processor-chain">
- <processor class="solr.CdcrUpdateProcessorFactory"/>
- <processor class="solr.RunUpdateProcessorFactory"/>
- </updateRequestProcessorChain>
-
- <requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
- </requestHandler>
-
- <updateHandler class="solr.DirectUpdateHandler2">
- <updateLog class="solr.CdcrUpdateLog">
- <str name="dir">${solr.ulog.dir:}</str>
- </updateLog>
- </updateHandler>
-
- <requestHandler name="standard" class="solr.StandardRequestHandler">
- </requestHandler>
-
- <requestHandler name="/update" class="solr.UpdateRequestHandler">
- <lst name="defaults">
- <str name="update.chain">cdcr-processor-chain</str>
- </lst>
- </requestHandler>
-</config>
-
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java b/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java
index 1efdc6a..aca5e0b 100644
--- a/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java
@@ -17,13 +17,9 @@
package org.apache.solr.cloud;
-import java.io.File;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
-import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
import java.util.concurrent.TimeUnit;
import org.apache.solr.SolrTestCaseJ4;
@@ -32,6 +28,7 @@ import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.request.AbstractUpdateRequest;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.UpdateRequest;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrInputDocument;
@@ -71,17 +68,13 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
MiniSolrCloudCluster source = new MiniSolrCloudCluster(1, createTempDir("cdcr-source"), buildJettyConfig("/solr"));
try {
source.waitForAllNodes(30);
- final File configDir = getFile("solr").toPath().resolve("configsets/cdcr-source-disabled").toFile();
- System.out.println("config dir absolute path = " + configDir.getAbsolutePath());
- source.uploadConfigDir(configDir, "cdcr-source");
+ source.uploadConfigSet(configset("cdcr-source-disabled"), "cdcr-source");
// create a collection with the cdcr-source-disabled configset
- Map<String, String> collectionProperties = new HashMap<>();
- // todo investigate why this is necessary??? because by default it selects a ram directory which deletes the tlogs on reloads?
- collectionProperties.putIfAbsent("solr.directoryFactory", "solr.StandardDirectoryFactory");
- source.createCollection("cdcr-source", 1, 1, "cdcr-source", collectionProperties);
- source.getSolrClient().getZkStateReader().forceUpdateCollection("cdcr-source");
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("cdcr-source", source.getSolrClient().getZkStateReader(), true, true, 330);
+ CollectionAdminRequest.createCollection("cdcr-source", "cdcr-source", 1, 1)
+ // todo investigate why this is necessary??? because by default it selects a ram directory which deletes the tlogs on reloads?
+ .withProperty("solr.directoryFactory", "solr.StandardDirectoryFactory")
+ .process(source.getSolrClient());
// index 10000 docs with a hard commit every 1000 documents
CloudSolrClient sourceSolrClient = source.getSolrClient();
@@ -115,8 +108,7 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
}
// upload the cdcr-enabled config and restart source cluster
- final File cdcrEnabledSourceConfigDir = getFile("solr").toPath().resolve("configsets/cdcr-source").toFile();
- source.uploadConfigDir(cdcrEnabledSourceConfigDir, "cdcr-source");
+ source.uploadConfigSet(configset("cdcr-source"), "cdcr-source");
JettySolrRunner runner = source.stopJettySolrRunner(0);
source.startJettySolrRunner(runner);
assertTrue(runner.isRunning());
@@ -126,11 +118,9 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
assertEquals("Document mismatch on source after restart", numDocs, response.getResults().getNumFound());
// setup the target cluster
- final File targetConfigDir = getFile("solr").toPath().resolve("configsets/cdcr-target").toFile();
- target.uploadConfigDir(targetConfigDir, "cdcr-target");
- target.createCollection("cdcr-target", 1, 1, "cdcr-target", Collections.emptyMap());
- target.getSolrClient().getZkStateReader().forceUpdateCollection("cdcr-target");
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("cdcr-target", target.getSolrClient().getZkStateReader(), true, true, 330);
+ target.uploadConfigSet(configset("cdcr-target"), "cdcr-target");
+ CollectionAdminRequest.createCollection("cdcr-target", "cdcr-target", 1, 1)
+ .process(target.getSolrClient());
CloudSolrClient targetSolrClient = target.getSolrClient();
targetSolrClient.setDefaultCollection("cdcr-target");
Thread.sleep(1000);
@@ -174,16 +164,11 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
MiniSolrCloudCluster source = new MiniSolrCloudCluster(1, createTempDir("cdcr-source"), buildJettyConfig("/solr"));
try {
source.waitForAllNodes(30);
- final File configDir = getFile("solr").toPath().resolve("configsets/cdcr-source").toFile();
- System.out.println("config dir absolute path = " + configDir.getAbsolutePath());
- source.uploadConfigDir(configDir, "cdcr-source");
-
- Map<String, String> collectionProperties = new HashMap<>();
- // todo investigate why this is necessary???
- collectionProperties.putIfAbsent("solr.directoryFactory", "solr.StandardDirectoryFactory");
- source.createCollection("cdcr-source", 1, 1, "cdcr-source", collectionProperties);
- source.getSolrClient().getZkStateReader().forceUpdateCollection("cdcr-source");
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("cdcr-source", source.getSolrClient().getZkStateReader(), true, true, 330);
+ source.uploadConfigSet(configset("cdcr-source"), "cdcr-source");
+
+ CollectionAdminRequest.createCollection("cdcr-source", "cdcr-source", 1, 1)
+ .withProperty("solr.directoryFactory", "solr.StandardDirectoryFactory")
+ .process(source.getSolrClient());
// index 10000 docs with a hard commit every 1000 documents
CloudSolrClient sourceSolrClient = source.getSolrClient();
@@ -206,11 +191,9 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
assertEquals("", numDocs, response.getResults().getNumFound());
// setup the target cluster
- final File targetConfigDir = getFile("solr").toPath().resolve("configsets/cdcr-target").toFile();
- target.uploadConfigDir(targetConfigDir, "cdcr-target");
- target.createCollection("cdcr-target", 1, 1, "cdcr-target", Collections.emptyMap());
- target.getSolrClient().getZkStateReader().forceUpdateCollection("cdcr-target");
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("cdcr-target", target.getSolrClient().getZkStateReader(), true, true, 330);
+ target.uploadConfigSet(configset("cdcr-target"), "cdcr-target");
+ CollectionAdminRequest.createCollection("cdcr-target", "cdcr-target", 1, 1)
+ .process(target.getSolrClient());
CloudSolrClient targetSolrClient = target.getSolrClient();
targetSolrClient.setDefaultCollection("cdcr-target");
@@ -267,16 +250,11 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
MiniSolrCloudCluster source = new MiniSolrCloudCluster(1, createTempDir("cdcr-source"), buildJettyConfig("/solr"));
try {
source.waitForAllNodes(30);
- final File configDir = getFile("solr").toPath().resolve("configsets/cdcr-source").toFile();
- System.out.println("config dir absolute path = " + configDir.getAbsolutePath());
- source.uploadConfigDir(configDir, "cdcr-source");
-
- Map<String, String> collectionProperties = new HashMap<>();
- // todo investigate why this is necessary???
- collectionProperties.putIfAbsent("solr.directoryFactory", "solr.StandardDirectoryFactory");
- source.createCollection("cdcr-source", 1, 1, "cdcr-source", collectionProperties);
- source.getSolrClient().getZkStateReader().forceUpdateCollection("cdcr-source");
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("cdcr-source", source.getSolrClient().getZkStateReader(), true, true, 330);
+ source.uploadConfigSet(configset("cdcr-source"), "cdcr-source");
+
+ CollectionAdminRequest.createCollection("cdcr-source", "cdcr-source", 1, 1)
+ .withProperty("solr.directoryFactory", "solr.StandardDirectoryFactory")
+ .process(source.getSolrClient());
// index 10000 docs with a hard commit every 1000 documents
CloudSolrClient sourceSolrClient = source.getSolrClient();
@@ -299,11 +277,9 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
assertEquals("", numDocs, response.getResults().getNumFound());
// setup the target cluster
- final File targetConfigDir = getFile("solr").toPath().resolve("configsets/cdcr-target").toFile();
- target.uploadConfigDir(targetConfigDir, "cdcr-target");
- target.createCollection("cdcr-target", 1, 1, "cdcr-target", Collections.emptyMap());
- target.getSolrClient().getZkStateReader().forceUpdateCollection("cdcr-target");
- AbstractDistribZkTestBase.waitForRecoveriesToFinish("cdcr-target", target.getSolrClient().getZkStateReader(), true, true, 330);
+ target.uploadConfigSet(configset("cdcr-target"), "cdcr-target");
+ CollectionAdminRequest.createCollection("cdcr-target", "cdcr-target", 1, 1)
+ .process(target.getSolrClient());
CloudSolrClient targetSolrClient = target.getSolrClient();
targetSolrClient.setDefaultCollection("cdcr-target");
Thread.sleep(1000);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/ConcurrentDeleteAndCreateCollectionTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/ConcurrentDeleteAndCreateCollectionTest.java b/solr/core/src/test/org/apache/solr/cloud/ConcurrentDeleteAndCreateCollectionTest.java
index e9ec533..3e90759 100644
--- a/solr/core/src/test/org/apache/solr/cloud/ConcurrentDeleteAndCreateCollectionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/ConcurrentDeleteAndCreateCollectionTest.java
@@ -16,8 +16,8 @@
*/
package org.apache.solr.cloud;
-import java.io.File;
import java.io.IOException;
+import java.nio.file.Path;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
@@ -52,13 +52,12 @@ public class ConcurrentDeleteAndCreateCollectionTest extends SolrTestCaseJ4 {
}
public void testConcurrentCreateAndDeleteDoesNotFail() {
- final File configDir = getFile("solr").toPath().resolve("configsets/configset-2/conf").toFile();
final AtomicReference<Exception> failure = new AtomicReference<>();
final int timeToRunSec = 30;
final Thread[] threads = new Thread[10];
for (int i = 0; i < threads.length; i++) {
final String collectionName = "collection" + i;
- uploadConfig(configDir, collectionName);
+ uploadConfig(configset("configset-2"), collectionName);
final String baseUrl = solrCluster.getJettySolrRunners().get(0).getBaseUrl().toString();
final SolrClient solrClient = getHttpSolrClient(baseUrl);
threads[i] = new CreateDeleteSearchCollectionThread("create-delete-search-" + i, collectionName, collectionName,
@@ -73,8 +72,7 @@ public class ConcurrentDeleteAndCreateCollectionTest extends SolrTestCaseJ4 {
public void testConcurrentCreateAndDeleteOverTheSameConfig() {
final String configName = "testconfig";
- final File configDir = getFile("solr").toPath().resolve("configsets/configset-2/conf").toFile();
- uploadConfig(configDir, configName); // upload config once, to be used by all collections
+ uploadConfig(configset("configset-2"), configName); // upload config once, to be used by all collections
final String baseUrl = solrCluster.getJettySolrRunners().get(0).getBaseUrl().toString();
final SolrClient solrClient = getHttpSolrClient(baseUrl);
final AtomicReference<Exception> failure = new AtomicReference<>();
@@ -98,9 +96,9 @@ public class ConcurrentDeleteAndCreateCollectionTest extends SolrTestCaseJ4 {
}
}
- private void uploadConfig(File configDir, String configName) {
+ private void uploadConfig(Path configDir, String configName) {
try {
- solrCluster.uploadConfigDir(configDir, configName);
+ solrCluster.uploadConfigSet(configDir, configName);
} catch (IOException | KeeperException | InterruptedException e) {
throw new RuntimeException(e);
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/DistribJoinFromCollectionTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/DistribJoinFromCollectionTest.java b/solr/core/src/test/org/apache/solr/cloud/DistribJoinFromCollectionTest.java
index d0d79f6..cfebef2 100644
--- a/solr/core/src/test/org/apache/solr/cloud/DistribJoinFromCollectionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/DistribJoinFromCollectionTest.java
@@ -16,8 +16,6 @@
*/
package org.apache.solr.cloud;
-import static org.hamcrest.CoreMatchers.not;
-
import java.io.IOException;
import java.lang.invoke.MethodHandles;
import java.nio.file.Path;
@@ -49,6 +47,8 @@ import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static org.hamcrest.CoreMatchers.not;
+
/**
* Tests using fromIndex that points to a collection in SolrCloud mode.
*/
@@ -83,10 +83,10 @@ public class DistribJoinFromCollectionTest extends SolrCloudTestCase{
int shards = 2;
int replicas = 2 ;
- assertNotNull(cluster.createCollection(toColl, shards, replicas,
- configName,
- collectionProperties));
-
+ CollectionAdminRequest.createCollection(toColl, configName, shards, replicas)
+ .setProperties(collectionProperties)
+ .process(cluster.getSolrClient());
+
// get the set of nodes where replicas for the "to" collection exist
Set<String> nodeSet = new HashSet<>();
ZkStateReader zkStateReader = cluster.getSolrClient().getZkStateReader();
@@ -97,14 +97,11 @@ public class DistribJoinFromCollectionTest extends SolrCloudTestCase{
assertTrue(nodeSet.size() > 0);
// deploy the "from" collection to all nodes where the "to" collection exists
-
- assertNotNull(cluster.createCollection(fromColl, 1, 4,
- configName, StringUtils.join(nodeSet,","), null,
- collectionProperties));
-
- AbstractDistribZkTestBase.waitForRecoveriesToFinish(toColl, zkStateReader, false, true, 30);
- AbstractDistribZkTestBase.waitForRecoveriesToFinish(fromColl, zkStateReader, false, true, 30);
-
+ CollectionAdminRequest.createCollection(fromColl, configName, 1, 4)
+ .setCreateNodeSet(StringUtils.join(nodeSet, ","))
+ .setProperties(collectionProperties)
+ .process(cluster.getSolrClient());
+
toDocId = indexDoc(toColl, 1001, "a", null, "b");
indexDoc(fromColl, 2001, "a", "c", null);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestAuthenticationFramework.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestAuthenticationFramework.java b/solr/core/src/test/org/apache/solr/cloud/TestAuthenticationFramework.java
index 6b562a1..5fd8e42 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestAuthenticationFramework.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestAuthenticationFramework.java
@@ -16,19 +16,18 @@
*/
package org.apache.solr.cloud;
-import java.io.File;
-import java.io.IOException;
-import java.lang.invoke.MethodHandles;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
import javax.servlet.FilterChain;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule;
import org.apache.http.HttpException;
import org.apache.http.HttpRequest;
import org.apache.http.HttpRequestInterceptor;
@@ -43,8 +42,8 @@ import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.impl.HttpClientUtil;
import org.apache.solr.client.solrj.impl.SolrHttpClientBuilder;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.response.QueryResponse;
-import org.apache.solr.client.solrj.response.RequestStatusState;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.core.CoreDescriptor;
@@ -62,8 +61,6 @@ import org.junit.rules.TestRule;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule;
-
/**
* Test of the MiniSolrCloudCluster functionality with authentication enabled.
*/
@@ -143,8 +140,7 @@ public class TestAuthenticationFramework extends LuceneTestCase {
private void createCollection(MiniSolrCloudCluster miniCluster, String collectionName, String asyncId)
throws Exception {
String configName = "solrCloudCollectionConfig";
- File configDir = new File(SolrTestCaseJ4.TEST_HOME() + File.separator + "collection1" + File.separator + "conf");
- miniCluster.uploadConfigDir(configDir, configName);
+ miniCluster.uploadConfigSet(SolrTestCaseJ4.TEST_PATH().resolve("collection1/conf"), configName);
final boolean persistIndex = random().nextBoolean();
Map<String, String> collectionProperties = new HashMap<>();
@@ -165,7 +161,17 @@ public class TestAuthenticationFramework extends LuceneTestCase {
collectionProperties.putIfAbsent("solr.tests.mergeScheduler", "org.apache.lucene.index.ConcurrentMergeScheduler");
collectionProperties.putIfAbsent("solr.directoryFactory", (persistIndex ? "solr.StandardDirectoryFactory" : "solr.RAMDirectoryFactory"));
- miniCluster.createCollection(collectionName, NUM_SHARDS, REPLICATION_FACTOR, configName, null, asyncId, collectionProperties);
+ if (asyncId == null) {
+ CollectionAdminRequest.createCollection(collectionName, configName, NUM_SHARDS, REPLICATION_FACTOR)
+ .setProperties(collectionProperties)
+ .process(miniCluster.getSolrClient());
+ }
+ else {
+ CollectionAdminRequest.createCollection(collectionName, configName, NUM_SHARDS, REPLICATION_FACTOR)
+ .setProperties(collectionProperties)
+ .processAndWait(miniCluster.getSolrClient(), 30);
+ }
+
}
public void collectionCreateSearchDelete(MiniSolrCloudCluster miniCluster) throws Exception {
@@ -185,11 +191,6 @@ public class TestAuthenticationFramework extends LuceneTestCase {
log.info("#### Creating a collection");
final String asyncId = (random().nextBoolean() ? null : "asyncId("+collectionName+".create)="+random().nextInt());
createCollection(miniCluster, collectionName, asyncId);
- if (asyncId != null) {
- final RequestStatusState state = AbstractFullDistribZkTestBase.getRequestStateAfterCompletion(asyncId, 330,
- cloudSolrClient);
- assertSame("did not see async createCollection completion", RequestStatusState.COMPLETED, state);
- }
ZkStateReader zkStateReader = miniCluster.getSolrClient().getZkStateReader();
AbstractDistribZkTestBase.waitForRecoveriesToFinish(collectionName, zkStateReader, true, true, 330);
@@ -207,17 +208,11 @@ public class TestAuthenticationFramework extends LuceneTestCase {
assertEquals(1, rsp.getResults().getNumFound());
// delete the collection we created earlier
- miniCluster.deleteCollection(collectionName);
- AbstractDistribZkTestBase.waitForCollectionToDisappear(collectionName, zkStateReader, true, true, 330);
+ CollectionAdminRequest.deleteCollection(collectionName).process(miniCluster.getSolrClient());
// create it again
String asyncId2 = (random().nextBoolean() ? null : "asyncId("+collectionName+".create)="+random().nextInt());
createCollection(miniCluster, collectionName, asyncId2);
- if (asyncId2 != null) {
- final RequestStatusState state = AbstractFullDistribZkTestBase.getRequestStateAfterCompletion(asyncId2, 330,
- cloudSolrClient);
- assertSame("did not see async createCollection completion", RequestStatusState.COMPLETED, state);
- }
AbstractDistribZkTestBase.waitForRecoveriesToFinish(collectionName, zkStateReader, true, true, 330);
// check that there's no left-over state
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestCloudDeleteByQuery.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestCloudDeleteByQuery.java b/solr/core/src/test/org/apache/solr/cloud/TestCloudDeleteByQuery.java
index 91d5581..2f6c5e9 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestCloudDeleteByQuery.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestCloudDeleteByQuery.java
@@ -23,18 +23,19 @@ import java.nio.file.Paths;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
-import org.apache.solr.cloud.SolrCloudTestCase;
+
import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.UpdateRequest;
import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.SolrInputField;
-import org.apache.solr.common.SolrException;
import org.apache.solr.common.cloud.ClusterState;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.Slice;
@@ -44,7 +45,6 @@ import org.apache.solr.common.params.SolrParams;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -106,9 +106,10 @@ public class TestCloudDeleteByQuery extends SolrCloudTestCase {
collectionProperties.put("config", "solrconfig-tlog.xml");
collectionProperties.put("schema", "schema15.xml"); // string id for doc routing prefix
- assertNotNull(cluster.createCollection(COLLECTION_NAME, NUM_SHARDS, REPLICATION_FACTOR,
- configName, null, null, collectionProperties));
-
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, configName, NUM_SHARDS, REPLICATION_FACTOR)
+ .setProperties(collectionProperties)
+ .process(cluster.getSolrClient());
+
CLOUD_CLIENT = cluster.getSolrClient();
CLOUD_CLIENT.setDefaultCollection(COLLECTION_NAME);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestCloudPseudoReturnFields.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestCloudPseudoReturnFields.java b/solr/core/src/test/org/apache/solr/cloud/TestCloudPseudoReturnFields.java
index 6a15e68..f7e6756 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestCloudPseudoReturnFields.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestCloudPseudoReturnFields.java
@@ -17,7 +17,6 @@
package org.apache.solr.cloud;
import java.lang.invoke.MethodHandles;
-
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
@@ -28,28 +27,22 @@ import java.util.List;
import java.util.Map;
import java.util.Random;
+import org.apache.commons.lang.StringUtils;
+import org.apache.lucene.util.TestUtil;
import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.schema.SchemaRequest.Field;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.client.solrj.response.schema.SchemaResponse.FieldResponse;
-
-import org.apache.solr.cloud.SolrCloudTestCase;
-
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.params.SolrParams;
-
import org.apache.solr.search.TestPseudoReturnFields;
-
-import org.apache.lucene.util.TestUtil;
-
-import org.apache.commons.lang.StringUtils;
-
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
@@ -85,10 +78,10 @@ public class TestCloudPseudoReturnFields extends SolrCloudTestCase {
Map<String, String> collectionProperties = new HashMap<>();
collectionProperties.put("config", "solrconfig-tlog.xml");
collectionProperties.put("schema", "schema-psuedo-fields.xml");
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, configName, numShards, repFactor)
+ .setProperties(collectionProperties)
+ .process(cluster.getSolrClient());
- assertNotNull(cluster.createCollection(COLLECTION_NAME, numShards, repFactor,
- configName, null, null, collectionProperties));
-
CLOUD_CLIENT = cluster.getSolrClient();
CLOUD_CLIENT.setDefaultCollection(COLLECTION_NAME);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPI.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPI.java b/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPI.java
index e3870dc..2027a6d 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPI.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPI.java
@@ -80,8 +80,7 @@ public class TestConfigSetsAPI extends SolrTestCaseJ4 {
public void testCreateErrors() throws Exception {
final String baseUrl = solrCluster.getJettySolrRunners().get(0).getBaseUrl().toString();
final SolrClient solrClient = getHttpSolrClient(baseUrl);
- final File configDir = getFile("solr").toPath().resolve("configsets/configset-2/conf").toFile();
- solrCluster.uploadConfigDir(configDir, "configSet");
+ solrCluster.uploadConfigSet(configset("configset-2"), "configSet");
// no action
CreateNoErrorChecking createNoAction = new CreateNoErrorChecking();
@@ -137,7 +136,7 @@ public class TestConfigSetsAPI extends SolrTestCaseJ4 {
FileUtils.write(new File(tmpConfigDir, ConfigSetProperties.DEFAULT_FILENAME),
getConfigSetProps(oldProps), StandardCharsets.UTF_8);
}
- solrCluster.uploadConfigDir(tmpConfigDir, baseConfigSetName);
+ solrCluster.uploadConfigSet(tmpConfigDir.toPath(), baseConfigSetName);
}
private void verifyCreate(String baseConfigSetName, String configSetName,
@@ -243,7 +242,7 @@ public class TestConfigSetsAPI extends SolrTestCaseJ4 {
FileUtils.copyDirectory(configDir, tmpConfigDir);
FileUtils.write(new File(tmpConfigDir, "configsetprops.json"),
getConfigSetProps(ImmutableMap.<String, String>of("immutable", "true")), StandardCharsets.UTF_8);
- solrCluster.uploadConfigDir(tmpConfigDir, "configSet");
+ solrCluster.uploadConfigSet(tmpConfigDir.toPath(), "configSet");
// no ConfigSet name
DeleteNoErrorChecking delete = new DeleteNoErrorChecking();
@@ -275,9 +274,8 @@ public class TestConfigSetsAPI extends SolrTestCaseJ4 {
public void testDelete() throws Exception {
final String baseUrl = solrCluster.getJettySolrRunners().get(0).getBaseUrl().toString();
final SolrClient solrClient = getHttpSolrClient(baseUrl);
- final File configDir = getFile("solr").toPath().resolve("configsets/configset-2/conf").toFile();
final String configSet = "configSet";
- solrCluster.uploadConfigDir(configDir, configSet);
+ solrCluster.uploadConfigSet(configset("configset-2"), configSet);
SolrZkClient zkClient = new SolrZkClient(solrCluster.getZkServer().getZkAddress(),
AbstractZkTestCase.TIMEOUT, AbstractZkTestCase.TIMEOUT, null);
@@ -312,11 +310,10 @@ public class TestConfigSetsAPI extends SolrTestCaseJ4 {
assertEquals(0, actualConfigSets.size());
// test multiple
- final File configDir = getFile("solr").toPath().resolve("configsets/configset-2/conf").toFile();
Set<String> configSets = new HashSet<String>();
for (int i = 0; i < 5; ++i) {
String configSet = "configSet" + i;
- solrCluster.uploadConfigDir(configDir, configSet);
+ solrCluster.uploadConfigSet(configset("configset-2"), configSet);
configSets.add(configSet);
}
response = list.process(solrClient);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIExclusivity.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIExclusivity.java b/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIExclusivity.java
index d4a1177..a110727 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIExclusivity.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIExclusivity.java
@@ -16,23 +16,19 @@
*/
package org.apache.solr.cloud;
-import java.io.File;
import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;
-import org.apache.commons.io.FileUtils;
import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.request.ConfigSetAdminRequest;
import org.apache.solr.client.solrj.request.ConfigSetAdminRequest.Create;
import org.apache.solr.client.solrj.request.ConfigSetAdminRequest.Delete;
-
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -92,11 +88,7 @@ public class TestConfigSetsAPIExclusivity extends SolrTestCaseJ4 {
}
private void setupBaseConfigSet(String baseConfigSetName) throws Exception {
- final File configDir = getFile("solr").toPath().resolve("configsets/configset-2/conf").toFile();
- final File tmpConfigDir = createTempDir().toFile();
- tmpConfigDir.deleteOnExit();
- FileUtils.copyDirectory(configDir, tmpConfigDir);
- solrCluster.uploadConfigDir(tmpConfigDir, baseConfigSetName);
+ solrCluster.uploadConfigSet(configset("configset-2"), baseConfigSetName);
}
private Exception getFirstExceptionOrNull(List<Exception> list) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.java b/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.java
index 8fe9cb6..bcb0a1b 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.java
@@ -136,7 +136,7 @@ public class TestConfigSetsAPIZkFailure extends SolrTestCaseJ4 {
FileUtils.write(new File(tmpConfigDir, ConfigSetProperties.DEFAULT_FILENAME),
getConfigSetProps(oldProps), StandardCharsets.UTF_8);
}
- solrCluster.uploadConfigDir(tmpConfigDir, baseConfigSetName);
+ solrCluster.uploadConfigSet(tmpConfigDir.toPath(), baseConfigSetName);
}
private StringBuilder getConfigSetProps(Map<String, String> map) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudCluster.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudCluster.java b/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudCluster.java
index c2c99a1..97ecb67 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudCluster.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudCluster.java
@@ -37,8 +37,8 @@ import org.apache.solr.client.solrj.embedded.JettyConfig;
import org.apache.solr.client.solrj.embedded.JettyConfig.Builder;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.response.QueryResponse;
-import org.apache.solr.client.solrj.response.RequestStatusState;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.cloud.ClusterState;
import org.apache.solr.common.cloud.Replica;
@@ -93,8 +93,7 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
private void createCollection(MiniSolrCloudCluster miniCluster, String collectionName, String createNodeSet, String asyncId,
Boolean indexToPersist, Map<String,String> collectionProperties) throws Exception {
String configName = "solrCloudCollectionConfig";
- File configDir = new File(SolrTestCaseJ4.TEST_HOME() + File.separator + "collection1" + File.separator + "conf");
- miniCluster.uploadConfigDir(configDir, configName);
+ miniCluster.uploadConfigSet(SolrTestCaseJ4.TEST_PATH().resolve("collection1").resolve("conf"), configName);
final boolean persistIndex = (indexToPersist != null ? indexToPersist.booleanValue() : random().nextBoolean());
if (collectionProperties == null) {
@@ -115,8 +114,19 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
}
collectionProperties.putIfAbsent("solr.tests.mergeScheduler", "org.apache.lucene.index.ConcurrentMergeScheduler");
collectionProperties.putIfAbsent("solr.directoryFactory", (persistIndex ? "solr.StandardDirectoryFactory" : "solr.RAMDirectoryFactory"));
-
- miniCluster.createCollection(collectionName, NUM_SHARDS, REPLICATION_FACTOR, configName, createNodeSet, asyncId, collectionProperties);
+
+ if (asyncId == null) {
+ CollectionAdminRequest.createCollection(collectionName, configName, NUM_SHARDS, REPLICATION_FACTOR)
+ .setCreateNodeSet(createNodeSet)
+ .setProperties(collectionProperties)
+ .process(miniCluster.getSolrClient());
+ }
+ else {
+ CollectionAdminRequest.createCollection(collectionName, configName, NUM_SHARDS, REPLICATION_FACTOR)
+ .setCreateNodeSet(createNodeSet)
+ .setProperties(collectionProperties)
+ .processAndWait(miniCluster.getSolrClient(), 30);
+ }
}
@Test
@@ -151,11 +161,6 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
log.info("#### Creating a collection");
final String asyncId = (random().nextBoolean() ? null : "asyncId("+collectionName+".create)="+random().nextInt());
createCollection(miniCluster, collectionName, null, asyncId, null, null);
- if (asyncId != null) {
- final RequestStatusState state = AbstractFullDistribZkTestBase.getRequestStateAfterCompletion(asyncId, 330,
- cloudSolrClient);
- assertSame("did not see async createCollection completion", RequestStatusState.COMPLETED, state);
- }
ZkStateReader zkStateReader = miniCluster.getSolrClient().getZkStateReader();
AbstractDistribZkTestBase.waitForRecoveriesToFinish(collectionName, zkStateReader, true, true, 330);
@@ -205,19 +210,11 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
assertTrue(startedServer.isRunning());
assertEquals(NUM_SERVERS, miniCluster.getJettySolrRunners().size());
-
- // delete the collection we created earlier
- miniCluster.deleteCollection(collectionName);
- AbstractDistribZkTestBase.waitForCollectionToDisappear(collectionName, zkStateReader, true, true, 330);
+ CollectionAdminRequest.deleteCollection(collectionName).process(miniCluster.getSolrClient());
// create it again
String asyncId2 = (random().nextBoolean() ? null : "asyncId("+collectionName+".create)="+random().nextInt());
createCollection(miniCluster, collectionName, null, asyncId2, null, null);
- if (asyncId2 != null) {
- final RequestStatusState state = AbstractFullDistribZkTestBase.getRequestStateAfterCompletion(asyncId2, 330,
- cloudSolrClient);
- assertSame("did not see async createCollection completion", RequestStatusState.COMPLETED, state);
- }
AbstractDistribZkTestBase.waitForRecoveriesToFinish(collectionName, zkStateReader, true, true, 330);
// check that there's no left-over state
@@ -246,10 +243,6 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
// create collection
final String asyncId = (random().nextBoolean() ? null : "asyncId("+collectionName+".create)="+random().nextInt());
createCollection(miniCluster, collectionName, OverseerCollectionMessageHandler.CREATE_NODE_SET_EMPTY, asyncId, null, null);
- if (asyncId != null) {
- final RequestStatusState state = AbstractFullDistribZkTestBase.getRequestStateAfterCompletion(asyncId, 330, cloudSolrClient);
- assertSame("did not see async createCollection completion", RequestStatusState.COMPLETED, state);
- }
try (SolrZkClient zkClient = new SolrZkClient
(miniCluster.getZkServer().getZkAddress(), AbstractZkTestCase.TIMEOUT, AbstractZkTestCase.TIMEOUT, null);
@@ -267,10 +260,7 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
}
assertEquals(0, coreCount);
}
-
- // delete the collection we created earlier
- miniCluster.deleteCollection(collectionName);
- AbstractDistribZkTestBase.waitForCollectionToDisappear(collectionName, zkStateReader, true, true, 330);
+
}
}
finally {
@@ -387,9 +377,6 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
assertEquals(numDocs, rsp.getResults().getNumFound());
}
- // delete the collection we created earlier
- miniCluster.deleteCollection(collectionName);
- AbstractDistribZkTestBase.waitForCollectionToDisappear(collectionName, zkStateReader, true, true, 330);
}
}
finally {
@@ -418,10 +405,6 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
final Map<String, String> collectionProperties = new HashMap<>();
collectionProperties.put(CoreDescriptor.CORE_CONFIG, "solrconfig-sortingmergepolicyfactory.xml");
createCollection(miniCluster, collectionName, null, asyncId, Boolean.TRUE, collectionProperties);
- if (asyncId != null) {
- final RequestStatusState state = AbstractFullDistribZkTestBase.getRequestStateAfterCompletion(asyncId, 330, cloudSolrClient);
- assertSame("did not see async createCollection completion", RequestStatusState.COMPLETED, state);
- }
}
ZkStateReader zkStateReader = cloudSolrClient.getZkStateReader();
@@ -444,9 +427,6 @@ public class TestMiniSolrCloudCluster extends LuceneTestCase {
tstes.queryTimestampDescendingSegmentTerminateEarlyYesGrouped(cloudSolrClient);
tstes.queryTimestampAscendingSegmentTerminateEarlyYes(cloudSolrClient); // uses a sort order that is _not_ compatible with the merge sort order
- // delete the collection we created earlier
- miniCluster.deleteCollection(collectionName);
- AbstractDistribZkTestBase.waitForCollectionToDisappear(collectionName, zkStateReader, true, true, 330);
}
finally {
miniCluster.shutdown();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudClusterSSL.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudClusterSSL.java b/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudClusterSSL.java
index 4dc1d39..a53b39f 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudClusterSSL.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestMiniSolrCloudClusterSSL.java
@@ -16,43 +16,36 @@
*/
package org.apache.solr.cloud;
+import javax.net.ssl.SSLContext;
+import java.io.IOException;
import java.lang.invoke.MethodHandles;
-
-import java.util.Collections;
import java.util.List;
-import java.io.File;
-import java.io.IOException;
-
-import javax.net.ssl.SSLContext;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpHead;
+import org.apache.http.config.Registry;
+import org.apache.http.config.RegistryBuilder;
+import org.apache.http.conn.socket.ConnectionSocketFactory;
+import org.apache.http.conn.socket.PlainConnectionSocketFactory;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
+import org.apache.lucene.util.Constants;
import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.embedded.JettyConfig;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.HttpClientUtil;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.CoreAdminRequest;
-import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.params.CoreAdminParams.CoreAdminAction;
import org.apache.solr.util.SSLTestConfig;
-
-import org.apache.http.impl.client.CloseableHttpClient;
-import org.apache.http.impl.client.HttpClientBuilder;
-import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
-import org.apache.http.client.HttpClient;
-import org.apache.http.client.methods.HttpHead;
-import org.apache.http.config.RegistryBuilder;
-import org.apache.http.config.Registry;
-import org.apache.http.conn.socket.ConnectionSocketFactory;
-import org.apache.http.conn.socket.PlainConnectionSocketFactory;
-import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
-
-import org.apache.lucene.util.Constants;
-
import org.junit.After;
import org.junit.Before;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -174,9 +167,7 @@ public class TestMiniSolrCloudClusterSSL extends SolrTestCaseJ4 {
public static void checkClusterWithCollectionCreations(final MiniSolrCloudCluster cluster,
final SSLTestConfig sslConfig) throws Exception {
- cluster.uploadConfigDir(new File(SolrTestCaseJ4.TEST_HOME() + File.separator +
- "collection1" + File.separator + "conf"),
- CONF_NAME);
+ cluster.uploadConfigSet(SolrTestCaseJ4.TEST_PATH().resolve("collection1").resolve("conf"), CONF_NAME);
checkCreateCollection(cluster, "first_collection");
@@ -203,11 +194,10 @@ public class TestMiniSolrCloudClusterSSL extends SolrTestCaseJ4 {
*/
private static void checkCreateCollection(final MiniSolrCloudCluster cluster,
final String collection) throws Exception {
- assertNotNull(cluster.createCollection(collection,
- /* 1 shard/replica per server */ NUM_SERVERS, 1,
- CONF_NAME, null, null,
- Collections.singletonMap("config","solrconfig-tlog.xml")));
final CloudSolrClient cloudClient = cluster.getSolrClient();
+ CollectionAdminRequest.createCollection(collection, CONF_NAME, NUM_SERVERS, 1)
+ .withProperty("config", "solrconfig-tlog.xml")
+ .process(cloudClient);
ZkStateReader zkStateReader = cloudClient.getZkStateReader();
AbstractDistribZkTestBase.waitForRecoveriesToFinish(collection, zkStateReader, true, true, 330);
assertEquals("sanity query", 0, cloudClient.query(collection, params("q","*:*")).getStatus());
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestRandomFlRTGCloud.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestRandomFlRTGCloud.java b/solr/core/src/test/org/apache/solr/cloud/TestRandomFlRTGCloud.java
index 2e54679..966d8ef 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestRandomFlRTGCloud.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestRandomFlRTGCloud.java
@@ -16,9 +16,8 @@
*/
package org.apache.solr.cloud;
-import java.lang.invoke.MethodHandles;
-
import java.io.IOException;
+import java.lang.invoke.MethodHandles;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
@@ -33,36 +32,29 @@ import java.util.Random;
import java.util.Set;
import java.util.TreeSet;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.lucene.util.TestUtil;
import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.response.QueryResponse;
-
-import org.apache.solr.cloud.SolrCloudTestCase;
-
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
-import org.apache.solr.response.transform.DocTransformer; // jdoc
-import org.apache.solr.response.transform.RawValueTransformerFactory; // jdoc
+import org.apache.solr.response.transform.DocTransformer;
+import org.apache.solr.response.transform.RawValueTransformerFactory;
import org.apache.solr.response.transform.TransformerFactory;
-
-
import org.apache.solr.util.RandomizeSSL;
-import org.apache.lucene.util.TestUtil;
-
-import org.apache.commons.io.FilenameUtils;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import org.junit.AfterClass;
import org.junit.BeforeClass;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** @see TestCloudPseudoReturnFields */
@RandomizeSSL(clientAuth=0.0,reason="client auth uses too much RAM")
@@ -125,7 +117,7 @@ public class TestRandomFlRTGCloud extends SolrCloudTestCase {
new NotIncludedValidator("score","score_alias:score")));
@BeforeClass
- private static void createMiniSolrCloudCluster() throws Exception {
+ public static void createMiniSolrCloudCluster() throws Exception {
// 50% runs use single node/shard a FL_VALIDATORS with all validators known to work on single node
// 50% runs use multi node/shard with FL_VALIDATORS only containing stuff that works in cloud
@@ -142,17 +134,15 @@ public class TestRandomFlRTGCloud extends SolrCloudTestCase {
final Path configDir = Paths.get(TEST_HOME(), "collection1", "conf");
configureCluster(numNodes).addConfig(configName, configDir).configure();
-
- Map<String, String> collectionProperties = new HashMap<>();
- collectionProperties.put("config", "solrconfig-tlog.xml");
- collectionProperties.put("schema", "schema-psuedo-fields.xml");
- assertNotNull(cluster.createCollection(COLLECTION_NAME, numShards, repFactor,
- configName, null, null, collectionProperties));
-
CLOUD_CLIENT = cluster.getSolrClient();
CLOUD_CLIENT.setDefaultCollection(COLLECTION_NAME);
+ CollectionAdminRequest.createCollection(COLLECTION_NAME, configName, numShards, repFactor)
+ .withProperty("config", "solrconfig-tlog.xml")
+ .withProperty("schema", "schema-psuedo-fields.xml")
+ .process(CLOUD_CLIENT);
+
waitForRecoveriesToFinish(CLOUD_CLIENT);
for (JettySolrRunner jetty : cluster.getJettySolrRunners()) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestRequestForwarding.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestRequestForwarding.java b/solr/core/src/test/org/apache/solr/cloud/TestRequestForwarding.java
index e4737c4..8d85bf6 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestRequestForwarding.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestRequestForwarding.java
@@ -16,7 +16,6 @@
*/
package org.apache.solr.cloud;
-import java.io.File;
import java.net.URL;
import org.apache.solr.SolrTestCaseJ4;
@@ -38,8 +37,7 @@ public class TestRequestForwarding extends SolrTestCaseJ4 {
System.setProperty("solr.test.sys.prop1", "propone");
System.setProperty("solr.test.sys.prop2", "proptwo");
solrCluster = new MiniSolrCloudCluster(3, createTempDir(), buildJettyConfig("/solr"));
- File configDir = getFile("solr").toPath().resolve("collection1/conf").toFile();
- solrCluster.uploadConfigDir(configDir, "conf1");
+ solrCluster.uploadConfigSet(TEST_PATH().resolve("collection1/conf"), "conf1");
}
@Override
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithKerberosAlt.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithKerberosAlt.java b/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithKerberosAlt.java
index c505b51..d856011 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithKerberosAlt.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithKerberosAlt.java
@@ -172,8 +172,7 @@ public class TestSolrCloudWithKerberosAlt extends LuceneTestCase {
// create collection
String configName = "solrCloudCollectionConfig";
- File configDir = new File(SolrTestCaseJ4.TEST_HOME() + File.separator + "collection1" + File.separator + "conf");
- miniCluster.uploadConfigDir(configDir, configName);
+ miniCluster.uploadConfigSet(SolrTestCaseJ4.TEST_PATH().resolve("collection1/conf"), configName);
CollectionAdminRequest.Create createRequest = new CollectionAdminRequest.Create();
createRequest.setCollectionName(collectionName);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/67f3f433/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithSecureImpersonation.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithSecureImpersonation.java b/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithSecureImpersonation.java
index ef41e4d..53b7f5b 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithSecureImpersonation.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithSecureImpersonation.java
@@ -17,7 +17,6 @@
package org.apache.solr.cloud;
import javax.servlet.http.HttpServletRequest;
-import java.io.File;
import java.net.InetAddress;
import java.util.List;
import java.util.Map;
@@ -48,9 +47,9 @@ import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
-import static org.apache.solr.security.HttpParamDelegationTokenPlugin.USER_PARAM;
-import static org.apache.solr.security.HttpParamDelegationTokenPlugin.REMOTE_HOST_PARAM;
import static org.apache.solr.security.HttpParamDelegationTokenPlugin.REMOTE_ADDRESS_PARAM;
+import static org.apache.solr.security.HttpParamDelegationTokenPlugin.REMOTE_HOST_PARAM;
+import static org.apache.solr.security.HttpParamDelegationTokenPlugin.USER_PARAM;
public class TestSolrCloudWithSecureImpersonation extends SolrTestCaseJ4 {
private static final int NUM_SERVERS = 2;
@@ -343,8 +342,7 @@ public class TestSolrCloudWithSecureImpersonation extends SolrTestCaseJ4 {
@Test
public void testForwarding() throws Exception {
String collectionName = "forwardingCollection";
- File configDir = getFile("solr").toPath().resolve("collection1/conf").toFile();
- miniCluster.uploadConfigDir(configDir, "conf1");
+ miniCluster.uploadConfigSet(TEST_PATH().resolve("collection1/conf"), "conf1");
create1ShardCollection(collectionName, "conf1", miniCluster);
// try a command to each node, one of them must be forwarded
[13/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9627: Add
QParser.getSortSpec,
deprecate misleadingly named QParser.getSort (Judith Silverman,
Christine Poerschke)
Posted by kr...@apache.org.
SOLR-9627: Add QParser.getSortSpec, deprecate misleadingly named QParser.getSort (Judith Silverman, Christine Poerschke)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/c219525b
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/c219525b
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/c219525b
Branch: refs/heads/jira/solr-8593
Commit: c219525b1dfbe8ed0101d509754c9e5206f963de
Parents: 46aeb52
Author: Christine Poerschke <cp...@apache.org>
Authored: Mon Oct 17 17:13:32 2016 -0400
Committer: Christine Poerschke <cp...@apache.org>
Committed: Mon Oct 17 17:13:32 2016 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 2 ++
.../java/org/apache/solr/handler/MoreLikeThisHandler.java | 2 +-
.../org/apache/solr/handler/component/QueryComponent.java | 2 +-
.../java/org/apache/solr/search/LuceneQParserPlugin.java | 8 +++++++-
solr/core/src/java/org/apache/solr/search/QParser.java | 10 ++++++++++
5 files changed, 21 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c219525b/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 0ea8191..a3c14b0 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -281,6 +281,8 @@ Other Changes
* SOLR-9642: Refactor the core level snapshot cleanup mechanism to rely on Lucene (Hrishikesh Gadre via yonik)
+* SOLR-9627: Add QParser.getSortSpec, deprecate misleadingly named QParser.getSort (Judith Silverman, Christine Poerschke)
+
================== 6.2.1 ==================
Bug Fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c219525b/solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java b/solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java
index dcd70f4..9c86350 100644
--- a/solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java
@@ -117,7 +117,7 @@ public class MoreLikeThisHandler extends RequestHandlerBase
if (q != null) {
QParser parser = QParser.getParser(q, defType, req);
query = parser.getQuery();
- sortSpec = parser.getSort(true);
+ sortSpec = parser.getSortSpec(true);
}
String[] fqs = req.getParams().getParams(CommonParams.FQ);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c219525b/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java b/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
index 8e0f0ee..bc80629 100644
--- a/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
+++ b/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
@@ -184,7 +184,7 @@ public class QueryComponent extends SearchComponent
}
}
- rb.setSortSpec( parser.getSort(true) );
+ rb.setSortSpec( parser.getSortSpec(true) );
rb.setQparser(parser);
final String cursorStr = rb.req.getParams().get(CursorMarkParams.CURSOR_MARK_PARAM);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c219525b/solr/core/src/java/org/apache/solr/search/LuceneQParserPlugin.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/search/LuceneQParserPlugin.java b/solr/core/src/java/org/apache/solr/search/LuceneQParserPlugin.java
index 2aba02d..07b35ad 100644
--- a/solr/core/src/java/org/apache/solr/search/LuceneQParserPlugin.java
+++ b/solr/core/src/java/org/apache/solr/search/LuceneQParserPlugin.java
@@ -77,8 +77,14 @@ class OldLuceneQParser extends LuceneQParser {
}
@Override
+ @Deprecated
public SortSpec getSort(boolean useGlobal) throws SyntaxError {
- SortSpec sort = super.getSort(useGlobal);
+ return getSortSpec(useGlobal);
+ }
+
+ @Override
+ public SortSpec getSortSpec(boolean useGlobal) throws SyntaxError {
+ SortSpec sort = super.getSortSpec(useGlobal);
if (sortStr != null && sortStr.length()>0 && sort.getSort()==null) {
SortSpec oldSort = SortSpecParsing.parseSortSpec(sortStr, getReq());
if( oldSort.getSort() != null ) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c219525b/solr/core/src/java/org/apache/solr/search/QParser.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/search/QParser.java b/solr/core/src/java/org/apache/solr/search/QParser.java
index 898fdf1..40605be 100644
--- a/solr/core/src/java/org/apache/solr/search/QParser.java
+++ b/solr/core/src/java/org/apache/solr/search/QParser.java
@@ -209,8 +209,18 @@ public abstract class QParser {
/**
* @param useGlobalParams look up sort, start, rows in global params if not in local params
* @return the sort specification
+ * @deprecated Use the not misleadingly named getSortSpec() function instead.
*/
+ @Deprecated
public SortSpec getSort(boolean useGlobalParams) throws SyntaxError {
+ return getSortSpec(useGlobalParams);
+ }
+
+ /**
+ * @param useGlobalParams look up sort, start, rows in global params if not in local params
+ * @return the sort specification
+ */
+ public SortSpec getSortSpec(boolean useGlobalParams) throws SyntaxError {
getQuery(); // ensure query is parsed first
String sortStr = null;
[40/50] [abbrv] lucene-solr:jira/solr-8593: Merge remote-tracking
branch 'origin/master'
Posted by kr...@apache.org.
Merge remote-tracking branch 'origin/master'
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/67ba19a8
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/67ba19a8
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/67ba19a8
Branch: refs/heads/jira/solr-8593
Commit: 67ba19a810a6bce652622de67e57b6c33096060f
Parents: 6893595 f43742a
Author: Noble Paul <no...@gmail.com>
Authored: Thu Oct 20 01:26:31 2016 +0530
Committer: Noble Paul <no...@gmail.com>
Committed: Thu Oct 20 01:26:31 2016 +0530
----------------------------------------------------------------------
lucene/CHANGES.txt | 9 +
.../org/apache/lucene/facet/DrillSideways.java | 2 +-
.../facet/taxonomy/FastTaxonomyFacetCounts.java | 41 ++--
.../analyzing/AnalyzingInfixSuggester.java | 12 +-
.../analyzing/AnalyzingInfixSuggesterTest.java | 76 ++++++++
.../org/apache/solr/handler/StreamHandler.java | 4 +-
.../client/solrj/io/stream/DaemonStream.java | 44 ++++-
.../solrj/io/stream/StreamExpressionTest.java | 193 +++++++++++++++++--
8 files changed, 327 insertions(+), 54 deletions(-)
----------------------------------------------------------------------
[07/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-6203: in
QueryComponent rename groupSortStr to sortWithinGroupStr (so that name and
meaning match)
Posted by kr...@apache.org.
SOLR-6203: in QueryComponent rename groupSortStr to sortWithinGroupStr (so that name and meaning match)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a4a314d1
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a4a314d1
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a4a314d1
Branch: refs/heads/jira/solr-8593
Commit: a4a314d1602458cd7427b337d32eca60049c72da
Parents: 5261eb0
Author: Christine Poerschke <cp...@apache.org>
Authored: Sat Oct 15 20:56:05 2016 -0400
Committer: Christine Poerschke <cp...@apache.org>
Committed: Sat Oct 15 20:56:05 2016 -0400
----------------------------------------------------------------------
.../java/org/apache/solr/handler/component/QueryComponent.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a4a314d1/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java b/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
index ba6a68d..8e0f0ee 100644
--- a/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
+++ b/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
@@ -257,9 +257,9 @@ public class QueryComponent extends SearchComponent
}
// groupSort defaults to sort
- String groupSortStr = params.get(GroupParams.GROUP_SORT);
+ String sortWithinGroupStr = params.get(GroupParams.GROUP_SORT);
//TODO: move weighting of sort
- Sort sortWithinGroup = groupSortStr == null ? groupSort : searcher.weightSort(SortSpecParsing.parseSortSpec(groupSortStr, req).getSort());
+ Sort sortWithinGroup = sortWithinGroupStr == null ? groupSort : searcher.weightSort(SortSpecParsing.parseSortSpec(sortWithinGroupStr, req).getSort());
if (sortWithinGroup == null) {
sortWithinGroup = Sort.RELEVANCE;
}
[30/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7503: Undeprecate
o.o.l.util.LongValues.
Posted by kr...@apache.org.
LUCENE-7503: Undeprecate o.o.l.util.LongValues.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/3be6701f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/3be6701f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/3be6701f
Branch: refs/heads/jira/solr-8593
Commit: 3be6701f17d9a507e07e4a3f01bcfd702bdfc806
Parents: 9aa764a
Author: Adrien Grand <jp...@gmail.com>
Authored: Tue Oct 18 18:28:39 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Tue Oct 18 18:50:57 2016 +0200
----------------------------------------------------------------------
.../lucene54/Lucene54DocValuesProducer.java | 39 ++++++++++-
.../codecs/memory/MemoryDocValuesProducer.java | 69 +++++++-------------
.../java/org/apache/lucene/util/LongValues.java | 21 +-----
.../lucene/util/packed/TestDirectPacked.java | 6 +-
4 files changed, 67 insertions(+), 68 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3be6701f/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java b/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java
index 8a44c31..1f785fe 100644
--- a/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java
+++ b/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java
@@ -491,7 +491,44 @@ final class Lucene54DocValuesProducer extends DocValuesProducer implements Close
docsWithField = getLiveBits(entry.missingOffset, maxDoc);
}
}
- return new LegacyNumericDocValuesWrapper(docsWithField, getNumeric(entry));
+ final LongValues values = getNumeric(entry);
+ return new NumericDocValues() {
+
+ int doc = -1;
+ long value;
+
+ @Override
+ public long longValue() throws IOException {
+ return value;
+ }
+
+ @Override
+ public int docID() {
+ return doc;
+ }
+
+ @Override
+ public int nextDoc() throws IOException {
+ return advance(doc + 1);
+ }
+
+ @Override
+ public int advance(int target) throws IOException {
+ for (int doc = target; doc < maxDoc; ++doc) {
+ value = values.get(doc);
+ if (value != 0 || docsWithField.get(doc)) {
+ return this.doc = doc;
+ }
+ }
+ return doc = NO_MORE_DOCS;
+ }
+
+ @Override
+ public long cost() {
+ return maxDoc;
+ }
+
+ };
}
@Override
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3be6701f/lucene/codecs/src/java/org/apache/lucene/codecs/memory/MemoryDocValuesProducer.java
----------------------------------------------------------------------
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/memory/MemoryDocValuesProducer.java b/lucene/codecs/src/java/org/apache/lucene/codecs/memory/MemoryDocValuesProducer.java
index bff8c2d..b81e56e 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/memory/MemoryDocValuesProducer.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/memory/MemoryDocValuesProducer.java
@@ -42,7 +42,6 @@ import org.apache.lucene.util.FixedBitSet;
import org.apache.lucene.util.IOUtils;
import org.apache.lucene.util.IntsRef;
import org.apache.lucene.util.IntsRefBuilder;
-import org.apache.lucene.util.LongValues;
import org.apache.lucene.util.PagedBytes;
import org.apache.lucene.util.RamUsageEstimator;
import org.apache.lucene.util.fst.BytesRefFSTEnum.InputOutput;
@@ -371,7 +370,12 @@ class MemoryDocValuesProducer extends DocValuesProducer {
ramBytesUsed.addAndGet(reader.ramBytesUsed());
numericInfo.put(field.name, Accountables.namedAccountable("block compressed", reader));
}
- return reader;
+ return new LegacyNumericDocValues() {
+ @Override
+ public long get(int docID) {
+ return reader.get(docID);
+ }
+ };
case GCD_COMPRESSED:
final long min = data.readLong();
final long mult = data.readLong();
@@ -568,51 +572,26 @@ class MemoryDocValuesProducer extends DocValuesProducer {
}
addr = res;
}
- if (values instanceof LongValues) {
- // probably not the greatest codec choice for this situation, but we support it
- final LongValues longValues = (LongValues) values;
- return new LegacySortedNumericDocValuesWrapper(new LegacySortedNumericDocValues() {
- long startOffset;
- long endOffset;
-
- @Override
- public void setDocument(int doc) {
- startOffset = (int) addr.get(doc);
- endOffset = (int) addr.get(doc+1L);
- }
-
- @Override
- public long valueAt(int index) {
- return longValues.get(startOffset + index);
- }
-
- @Override
- public int count() {
- return (int) (endOffset - startOffset);
- }
- }, maxDoc);
- } else {
- return new LegacySortedNumericDocValuesWrapper(new LegacySortedNumericDocValues() {
- int startOffset;
- int endOffset;
-
- @Override
- public void setDocument(int doc) {
- startOffset = (int) addr.get(doc);
- endOffset = (int) addr.get(doc+1);
- }
+ return new LegacySortedNumericDocValuesWrapper(new LegacySortedNumericDocValues() {
+ int startOffset;
+ int endOffset;
+
+ @Override
+ public void setDocument(int doc) {
+ startOffset = (int) addr.get(doc);
+ endOffset = (int) addr.get(doc+1);
+ }
- @Override
- public long valueAt(int index) {
- return values.get(startOffset + index);
- }
+ @Override
+ public long valueAt(int index) {
+ return values.get(startOffset + index);
+ }
- @Override
- public int count() {
- return (endOffset - startOffset);
- }
- }, maxDoc);
- }
+ @Override
+ public int count() {
+ return (endOffset - startOffset);
+ }
+ }, maxDoc);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3be6701f/lucene/core/src/java/org/apache/lucene/util/LongValues.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/LongValues.java b/lucene/core/src/java/org/apache/lucene/util/LongValues.java
index aa3b7fa..23f4d32 100644
--- a/lucene/core/src/java/org/apache/lucene/util/LongValues.java
+++ b/lucene/core/src/java/org/apache/lucene/util/LongValues.java
@@ -16,21 +16,9 @@
*/
package org.apache.lucene.util;
-
-import org.apache.lucene.index.LegacyNumericDocValues;
-import org.apache.lucene.index.NumericDocValues;
-import org.apache.lucene.util.packed.PackedInts;
-
/** Abstraction over an array of longs.
- * This class extends NumericDocValues so that we don't need to add another
- * level of abstraction every time we want eg. to use the {@link PackedInts}
- * utility classes to represent a {@link LegacyNumericDocValues} instance.
- * @lucene.internal
- *
- * @deprecated Switch to {@link NumericDocValues} instead. */
- @Deprecated
-// TODO: cutover to iterator once codecs have all cutover?
-public abstract class LongValues extends LegacyNumericDocValues {
+ * @lucene.internal */
+public abstract class LongValues {
/** An instance that returns the provided value. */
public static final LongValues IDENTITY = new LongValues() {
@@ -45,9 +33,4 @@ public abstract class LongValues extends LegacyNumericDocValues {
/** Get value at <code>index</code>. */
public abstract long get(long index);
- @Override
- public long get(int idx) {
- return get((long) idx);
- }
-
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3be6701f/lucene/core/src/test/org/apache/lucene/util/packed/TestDirectPacked.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/util/packed/TestDirectPacked.java b/lucene/core/src/test/org/apache/lucene/util/packed/TestDirectPacked.java
index 4287125..3c5d5d8 100644
--- a/lucene/core/src/test/org/apache/lucene/util/packed/TestDirectPacked.java
+++ b/lucene/core/src/test/org/apache/lucene/util/packed/TestDirectPacked.java
@@ -19,12 +19,12 @@ package org.apache.lucene.util.packed;
import java.util.Random;
-import org.apache.lucene.index.LegacyNumericDocValues;
import org.apache.lucene.store.ByteArrayDataInput;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.IOContext;
import org.apache.lucene.store.IndexInput;
import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.LongValues;
import org.apache.lucene.util.LuceneTestCase;
import org.apache.lucene.util.TestUtil;
import org.apache.lucene.util.packed.DirectReader;
@@ -46,7 +46,7 @@ public class TestDirectPacked extends LuceneTestCase {
writer.finish();
output.close();
IndexInput input = dir.openInput("foo", IOContext.DEFAULT);
- LegacyNumericDocValues reader = DirectReader.getInstance(input.randomAccessSlice(0, input.length()), bitsPerValue, 0);
+ LongValues reader = DirectReader.getInstance(input.randomAccessSlice(0, input.length()), bitsPerValue, 0);
assertEquals(1, reader.get(0));
assertEquals(0, reader.get(1));
assertEquals(2, reader.get(2));
@@ -110,7 +110,7 @@ public class TestDirectPacked extends LuceneTestCase {
writer.finish();
output.close();
IndexInput input = directory.openInput(name, IOContext.DEFAULT);
- LegacyNumericDocValues reader = DirectReader.getInstance(input.randomAccessSlice(0, input.length()), bitsRequired, offset);
+ LongValues reader = DirectReader.getInstance(input.randomAccessSlice(0, input.length()), bitsRequired, offset);
for (int j = 0; j < original.length; j++) {
assertEquals("bpv=" + bpv, original[j], reader.get(j));
}
[23/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9634: Remove
deprecated methods
Posted by kr...@apache.org.
SOLR-9634: Remove deprecated methods
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/b2188f49
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/b2188f49
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/b2188f49
Branch: refs/heads/jira/solr-8593
Commit: b2188f4958c243dcdeca3d44ede5bc3c13542dcb
Parents: 67f3f43
Author: Alan Woodward <ro...@apache.org>
Authored: Tue Oct 18 12:03:42 2016 +0100
Committer: Alan Woodward <ro...@apache.org>
Committed: Tue Oct 18 12:34:18 2016 +0100
----------------------------------------------------------------------
solr/CHANGES.txt | 3 +
.../apache/solr/cloud/MiniSolrCloudCluster.java | 73 --------------------
2 files changed, 3 insertions(+), 73 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b2188f49/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index e16ca7a..be958d9 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -41,6 +41,9 @@ Upgrading from Solr 6.x
* Query time join with scoring {!join score=none} (even none) doesn't handle single value numeric fields.
Users are advised to convert these fields into string and reindex.
+* Deprecated collection and configset methods on MiniSolrCloudCluster have been
+ removed
+
Bug Fixes
----------------------
* SOLR-9262: Connection and read timeouts are being ignored by UpdateShardHandler after SOLR-4509.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b2188f49/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
----------------------------------------------------------------------
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
index 9301fd0..3c5aa16 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
@@ -17,7 +17,6 @@
package org.apache.solr.cloud;
import javax.servlet.Filter;
-import java.io.File;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
import java.nio.charset.Charset;
@@ -27,7 +26,6 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
-import java.util.Map;
import java.util.Objects;
import java.util.Random;
import java.util.SortedMap;
@@ -40,24 +38,17 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import com.google.common.base.Charsets;
-import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.embedded.JettyConfig;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.embedded.SSLConfig;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.impl.CloudSolrClient.Builder;
import org.apache.solr.client.solrj.request.CollectionAdminRequest;
-import org.apache.solr.client.solrj.request.QueryRequest;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.SolrZkClient;
import org.apache.solr.common.cloud.ZkConfigManager;
import org.apache.solr.common.cloud.ZkStateReader;
-import org.apache.solr.common.params.CollectionParams.CollectionAction;
-import org.apache.solr.common.params.CommonAdminParams;
-import org.apache.solr.common.params.CoreAdminParams;
-import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.util.ExecutorUtil;
-import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.SolrjNamedThreadFactory;
import org.apache.solr.core.CoreContainer;
import org.apache.zookeeper.KeeperException;
@@ -395,14 +386,6 @@ public class MiniSolrCloudCluster {
}
/**
- * @deprecated Use {@link #uploadConfigSet(Path, String)}
- */
- @Deprecated
- public void uploadConfigDir(File configDir, String configName) throws IOException, KeeperException, InterruptedException {
- uploadConfigSet(configDir.toPath(), configName);
- }
-
- /**
* Upload a config set
* @param configDir a path to the config set to upload
* @param configName the name to give the configset
@@ -425,62 +408,6 @@ public class MiniSolrCloudCluster {
}
/**
- * @deprecated Use {@link CollectionAdminRequest#createCollection(String, String, int, int)}
- */
- @Deprecated
- public NamedList<Object> createCollection(String name, int numShards, int replicationFactor,
- String configName, Map<String, String> collectionProperties) throws SolrServerException, IOException {
- return createCollection(name, numShards, replicationFactor, configName, null, null, collectionProperties);
- }
-
- /**
- * @deprecated Use {@link CollectionAdminRequest#createCollection(String, String, int, int)}
- */
- @Deprecated
- public NamedList<Object> createCollection(String name, int numShards, int replicationFactor,
- String configName, String createNodeSet, String asyncId, Map<String, String> collectionProperties) throws SolrServerException, IOException {
- final ModifiableSolrParams params = new ModifiableSolrParams();
- params.set(CoreAdminParams.ACTION, CollectionAction.CREATE.name());
- params.set(CoreAdminParams.NAME, name);
- params.set("numShards", numShards);
- params.set("replicationFactor", replicationFactor);
- params.set("collection.configName", configName);
- if (null != createNodeSet) {
- params.set(OverseerCollectionMessageHandler.CREATE_NODE_SET, createNodeSet);
- }
- if (null != asyncId) {
- params.set(CommonAdminParams.ASYNC, asyncId);
- }
- if(collectionProperties != null) {
- for(Map.Entry<String, String> property : collectionProperties.entrySet()){
- params.set(CoreAdminParams.PROPERTY_PREFIX + property.getKey(), property.getValue());
- }
- }
-
- return makeCollectionsRequest(params);
- }
-
- /**
- * @deprecated use {@link CollectionAdminRequest#deleteCollection(String)}
- */
- @Deprecated
- public NamedList<Object> deleteCollection(String name) throws SolrServerException, IOException {
- final ModifiableSolrParams params = new ModifiableSolrParams();
- params.set(CoreAdminParams.ACTION, CollectionAction.DELETE.name());
- params.set(CoreAdminParams.NAME, name);
-
- return makeCollectionsRequest(params);
- }
-
- private NamedList<Object> makeCollectionsRequest(final ModifiableSolrParams params) throws SolrServerException, IOException {
-
- final QueryRequest request = new QueryRequest(params);
- request.setPath("/admin/collections");
-
- return solrClient.request(request);
- }
-
- /**
* Shut down the cluster, including all Solr nodes and ZooKeeper
*/
public void shutdown() throws Exception {
[11/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9566: Don't put
replicas into recovery when collections are created
Posted by kr...@apache.org.
SOLR-9566: Don't put replicas into recovery when collections are created
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/65f55802
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/65f55802
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/65f55802
Branch: refs/heads/jira/solr-8593
Commit: 65f55802ee01b90a7f529de270d5d866a2282a40
Parents: 927fd51
Author: Alan Woodward <ro...@apache.org>
Authored: Mon Sep 26 22:51:39 2016 +0100
Committer: Alan Woodward <ro...@apache.org>
Committed: Mon Oct 17 17:18:25 2016 +0100
----------------------------------------------------------------------
solr/CHANGES.txt | 2 ++
.../apache/solr/cloud/CreateCollectionCmd.java | 1 +
.../org/apache/solr/cloud/ZkController.java | 23 +++++++++--------
.../org/apache/solr/core/CoreContainer.java | 26 ++++++++++----------
.../java/org/apache/solr/core/ZkContainer.java | 4 +--
.../solr/handler/admin/CoreAdminOperation.java | 4 ++-
.../org/apache/solr/core/TestCodecSupport.java | 2 +-
.../apache/solr/search/TestIndexSearcher.java | 6 ++---
.../solr/common/params/CoreAdminParams.java | 5 ++++
9 files changed, 41 insertions(+), 32 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 4f8d682..08abc76 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -195,6 +195,8 @@ Optimizations
reader rather than ask for a global view. In extreme cases, this leads to a 100x speedup.
(Takahiro Ishikawa, yonik)
+* SOLR-9566: Don't put replicas into recovery when first creating a Collection
+ (Alan Woodward)
Other Changes
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/core/src/java/org/apache/solr/cloud/CreateCollectionCmd.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/CreateCollectionCmd.java b/solr/core/src/java/org/apache/solr/cloud/CreateCollectionCmd.java
index 01e3fdf..a067b4a 100644
--- a/solr/core/src/java/org/apache/solr/cloud/CreateCollectionCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/CreateCollectionCmd.java
@@ -217,6 +217,7 @@ public class CreateCollectionCmd implements Cmd {
params.set(CoreAdminParams.COLLECTION, collectionName);
params.set(CoreAdminParams.SHARD, position.shard);
params.set(ZkStateReader.NUM_SHARDS_PROP, numSlices);
+ params.set(CoreAdminParams.NEW_COLLECTION, "true");
if (async != null) {
String coreAdminAsyncId = async + Math.abs(System.nanoTime());
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/core/src/java/org/apache/solr/cloud/ZkController.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/ZkController.java b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
index b56d1c9..9b0a90e 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ZkController.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
@@ -231,7 +231,7 @@ public class ZkController {
public Object call() throws Exception {
log.info("Registering core {} afterExpiration? {}", descriptor.getName(), afterExpiration);
- register(descriptor.getName(), descriptor, recoverReloadedCores, afterExpiration);
+ register(descriptor.getName(), descriptor, recoverReloadedCores, afterExpiration, false);
return descriptor;
}
}
@@ -354,7 +354,7 @@ public class ZkController {
if (executorService != null) {
executorService.submit(new RegisterCoreAsync(descriptor, true, true));
} else {
- register(descriptor.getName(), descriptor, true, true);
+ register(descriptor.getName(), descriptor, true, true, false);
}
} catch (Exception e) {
SolrException.log(log, "Error registering SolrCore", e);
@@ -840,8 +840,8 @@ public class ZkController {
*
* @return the shardId for the SolrCore
*/
- public String register(String coreName, final CoreDescriptor desc) throws Exception {
- return register(coreName, desc, false, false);
+ public String register(String coreName, final CoreDescriptor desc, boolean skipRecovery) throws Exception {
+ return register(coreName, desc, false, false, skipRecovery);
}
@@ -850,7 +850,8 @@ public class ZkController {
*
* @return the shardId for the SolrCore
*/
- public String register(String coreName, final CoreDescriptor desc, boolean recoverReloadedCores, boolean afterExpiration) throws Exception {
+ public String register(String coreName, final CoreDescriptor desc, boolean recoverReloadedCores,
+ boolean afterExpiration, boolean skipRecovery) throws Exception {
try (SolrCore core = cc.getCore(desc.getName())) {
MDCLoggingContext.setCore(core);
}
@@ -929,8 +930,8 @@ public class ZkController {
}
}
}
- boolean didRecovery = checkRecovery(coreName, desc, recoverReloadedCores, isLeader, cloudDesc, collection,
- coreZkNodeName, shardId, leaderProps, core, cc, afterExpiration);
+ boolean didRecovery
+ = checkRecovery(recoverReloadedCores, isLeader, skipRecovery, collection, coreZkNodeName, core, cc, afterExpiration);
if (!didRecovery) {
publish(desc, Replica.State.ACTIVE);
}
@@ -1080,10 +1081,8 @@ public class ZkController {
/**
* Returns whether or not a recovery was started
*/
- private boolean checkRecovery(String coreName, final CoreDescriptor desc,
- boolean recoverReloadedCores, final boolean isLeader,
- final CloudDescriptor cloudDesc, final String collection,
- final String shardZkNodeName, String shardId, ZkNodeProps leaderProps,
+ private boolean checkRecovery(boolean recoverReloadedCores, final boolean isLeader, boolean skipRecovery,
+ final String collection, String shardId,
SolrCore core, CoreContainer cc, boolean afterExpiration) {
if (SKIP_AUTO_RECOVERY) {
log.warn("Skipping recovery according to sys prop solrcloud.skip.autorecovery");
@@ -1092,7 +1091,7 @@ public class ZkController {
boolean doRecovery = true;
if (!isLeader) {
- if (!afterExpiration && core.isReloaded() && !recoverReloadedCores) {
+ if (skipRecovery || (!afterExpiration && core.isReloaded() && !recoverReloadedCores)) {
doRecovery = false;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/core/src/java/org/apache/solr/core/CoreContainer.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/CoreContainer.java b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
index fa8a8c0..a020b00 100644
--- a/solr/core/src/java/org/apache/solr/core/CoreContainer.java
+++ b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
@@ -517,14 +517,14 @@ public class CoreContainer {
zkSys.getZkController().throwErrorIfReplicaReplaced(cd);
}
- core = create(cd, false);
+ core = create(cd, false, false);
} finally {
if (asyncSolrCoreLoad) {
solrCores.markCoreAsNotLoading(cd);
}
}
try {
- zkSys.registerInZk(core, true);
+ zkSys.registerInZk(core, true, false);
} catch (RuntimeException e) {
SolrException.log(log, "Error registering SolrCore", e);
}
@@ -706,7 +706,7 @@ public class CoreContainer {
return coresLocator;
}
- protected SolrCore registerCore(String name, SolrCore core, boolean registerInZk) {
+ protected SolrCore registerCore(String name, SolrCore core, boolean registerInZk, boolean skipRecovery) {
if( core == null ) {
throw new RuntimeException( "Can not register a null core." );
}
@@ -744,7 +744,7 @@ public class CoreContainer {
if( old == null || old == core) {
log.debug( "registering core: "+name );
if (registerInZk) {
- zkSys.registerInZk(core, false);
+ zkSys.registerInZk(core, false, skipRecovery);
}
return null;
}
@@ -752,7 +752,7 @@ public class CoreContainer {
log.debug( "replacing core: "+name );
old.close();
if (registerInZk) {
- zkSys.registerInZk(core, false);
+ zkSys.registerInZk(core, false, skipRecovery);
}
return old;
}
@@ -765,7 +765,7 @@ public class CoreContainer {
* @return the newly created core
*/
public SolrCore create(String coreName, Map<String, String> parameters) {
- return create(coreName, cfg.getCoreRootDirectory().resolve(coreName), parameters);
+ return create(coreName, cfg.getCoreRootDirectory().resolve(coreName), parameters, false);
}
/**
@@ -775,7 +775,7 @@ public class CoreContainer {
* @param parameters the core parameters
* @return the newly created core
*/
- public SolrCore create(String coreName, Path instancePath, Map<String, String> parameters) {
+ public SolrCore create(String coreName, Path instancePath, Map<String, String> parameters, boolean newCollection) {
CoreDescriptor cd = new CoreDescriptor(this, coreName, instancePath, parameters);
@@ -798,7 +798,7 @@ public class CoreContainer {
preExisitingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
- SolrCore core = create(cd, true);
+ SolrCore core = create(cd, true, newCollection);
// only write out the descriptor if the core is successfully created
coresLocator.create(this, cd);
@@ -845,7 +845,7 @@ public class CoreContainer {
*
* @return the newly created core
*/
- private SolrCore create(CoreDescriptor dcore, boolean publishState) {
+ private SolrCore create(CoreDescriptor dcore, boolean publishState, boolean newCollection) {
if (isShutDown) {
throw new SolrException(ErrorCode.SERVICE_UNAVAILABLE, "Solr has been shutdown.");
@@ -868,7 +868,7 @@ public class CoreContainer {
core.getUpdateHandler().getUpdateLog().recoverFromLog();
}
- registerCore(dcore.getName(), core, publishState);
+ registerCore(dcore.getName(), core, publishState, newCollection);
return core;
} catch (Exception e) {
@@ -964,7 +964,7 @@ public class CoreContainer {
ConfigSet coreConfig = coreConfigService.getConfig(cd);
log.info("Reloading SolrCore '{}' using configuration from {}", cd.getName(), coreConfig.getName());
SolrCore newCore = core.reload(coreConfig);
- registerCore(name, newCore, false);
+ registerCore(name, newCore, false, false);
} catch (SolrCoreState.CoreIsClosedException e) {
throw e;
} catch (Exception e) {
@@ -1061,7 +1061,7 @@ public class CoreContainer {
SolrIdentifierValidator.validateCoreName(toName);
try (SolrCore core = getCore(name)) {
if (core != null) {
- registerCore(toName, core, true);
+ registerCore(toName, core, true, false);
SolrCore old = solrCores.remove(name);
coresLocator.rename(this, old.getCoreDescriptor(), core.getCoreDescriptor());
}
@@ -1132,7 +1132,7 @@ public class CoreContainer {
if (zkSys.getZkController() != null) {
zkSys.getZkController().throwErrorIfReplicaReplaced(desc);
}
- core = create(desc, true); // This should throw an error if it fails.
+ core = create(desc, true, false); // This should throw an error if it fails.
}
core.open();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/core/src/java/org/apache/solr/core/ZkContainer.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/ZkContainer.java b/solr/core/src/java/org/apache/solr/core/ZkContainer.java
index 22afe99..6665c4e 100644
--- a/solr/core/src/java/org/apache/solr/core/ZkContainer.java
+++ b/solr/core/src/java/org/apache/solr/core/ZkContainer.java
@@ -173,12 +173,12 @@ public class ZkContainer {
return zkRun.substring(0, zkRun.lastIndexOf('/'));
}
- public void registerInZk(final SolrCore core, boolean background) {
+ public void registerInZk(final SolrCore core, boolean background, boolean skipRecovery) {
Runnable r = () -> {
MDCLoggingContext.setCore(core);
try {
try {
- zkController.register(core.getName(), core.getCoreDescriptor());
+ zkController.register(core.getName(), core.getCoreDescriptor(), skipRecovery);
} catch (InterruptedException e) {
// Restore the interrupted status
Thread.currentThread().interrupt();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.java b/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.java
index e0d66a5..0b17d9e 100644
--- a/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.java
+++ b/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.java
@@ -82,7 +82,9 @@ enum CoreAdminOperation implements CoreAdminOp {
instancePath = coreContainer.getCoreRootDirectory().resolve(instanceDir).normalize();
}
- coreContainer.create(coreName, instancePath, coreParams);
+ boolean newCollection = params.getBool(CoreAdminParams.NEW_COLLECTION, false);
+
+ coreContainer.create(coreName, instancePath, coreParams, newCollection);
it.rsp.add("core", coreName);
}),
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/core/src/test/org/apache/solr/core/TestCodecSupport.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/core/TestCodecSupport.java b/solr/core/src/test/org/apache/solr/core/TestCodecSupport.java
index 0fe6a02..b6097ab 100644
--- a/solr/core/src/test/org/apache/solr/core/TestCodecSupport.java
+++ b/solr/core/src/test/org/apache/solr/core/TestCodecSupport.java
@@ -219,7 +219,7 @@ public class TestCodecSupport extends SolrTestCaseJ4 {
try {
c = new SolrCore(new CoreDescriptor(h.getCoreContainer(), newCoreName, testSolrHome.resolve(newCoreName)),
new ConfigSet("fakeConfigset", config, schema, null));
- assertNull(h.getCoreContainer().registerCore(newCoreName, c, false));
+ assertNull(h.getCoreContainer().registerCore(newCoreName, c, false, false));
h.coreName = newCoreName;
assertEquals("We are not using the correct core", "solrconfig_codec2.xml", h.getCore().getConfigResource());
assertU(add(doc("string_f", "foo")));
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/core/src/test/org/apache/solr/search/TestIndexSearcher.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/search/TestIndexSearcher.java b/solr/core/src/test/org/apache/solr/search/TestIndexSearcher.java
index f732f20..8fe3f97 100644
--- a/solr/core/src/test/org/apache/solr/search/TestIndexSearcher.java
+++ b/solr/core/src/test/org/apache/solr/search/TestIndexSearcher.java
@@ -230,7 +230,7 @@ public class TestIndexSearcher extends SolrTestCaseJ4 {
try {
// Create a new core, this should call all the firstSearcherListeners
- newCore = cores.create("core1", cd.getInstanceDir(), ImmutableMap.of("config", "solrconfig-searcher-listeners1.xml"));
+ newCore = cores.create("core1", cd.getInstanceDir(), ImmutableMap.of("config", "solrconfig-searcher-listeners1.xml"), false);
//validate that the new core was created with the correct solrconfig
assertNotNull(newCore.getSearchComponent("mock"));
@@ -280,7 +280,7 @@ public class TestIndexSearcher extends SolrTestCaseJ4 {
boolean coreCreated = false;
try {
// Create a new core, this should call all the firstSearcherListeners
- newCore = cores.create("core1", cd.getInstanceDir(), ImmutableMap.of("config", "solrconfig-searcher-listeners1.xml"));
+ newCore = cores.create("core1", cd.getInstanceDir(), ImmutableMap.of("config", "solrconfig-searcher-listeners1.xml"), false);
coreCreated = true;
//validate that the new core was created with the correct solrconfig
@@ -346,7 +346,7 @@ public class TestIndexSearcher extends SolrTestCaseJ4 {
try {
System.setProperty("tests.solr.useColdSearcher", "true");
// Create a new core, this should call all the firstSearcherListeners
- newCore = cores.create("core1", cd.getInstanceDir(), ImmutableMap.of("config", "solrconfig-searcher-listeners1.xml"));
+ newCore = cores.create("core1", cd.getInstanceDir(), ImmutableMap.of("config", "solrconfig-searcher-listeners1.xml"), false);
coreCreated = true;
//validate that the new core was created with the correct solrconfig
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/65f55802/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java b/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java
index 7f90a90..f3e0d7e 100644
--- a/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java
+++ b/solr/solrj/src/java/org/apache/solr/common/params/CoreAdminParams.java
@@ -123,6 +123,11 @@ public abstract class CoreAdminParams
*/
public static final String COMMIT_NAME = "commitName";
+ /**
+ * A boolean parameter specifying if a core is being created as part of a new collection
+ */
+ public static final String NEW_COLLECTION = "newCollection";
+
public enum CoreAdminAction {
STATUS(true),
UNLOAD,
[33/50] [abbrv] lucene-solr:jira/solr-8593: Merge remote-tracking
branch 'origin/master'
Posted by kr...@apache.org.
Merge remote-tracking branch 'origin/master'
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/53129ba6
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/53129ba6
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/53129ba6
Branch: refs/heads/jira/solr-8593
Commit: 53129ba670f8fb5c0e352fe6940736ee5b050631
Parents: ffa5c4b b78f221
Author: Noble Paul <no...@gmail.com>
Authored: Wed Oct 19 00:54:30 2016 +0530
Committer: Noble Paul <no...@gmail.com>
Committed: Wed Oct 19 00:54:30 2016 +0530
----------------------------------------------------------------------
.../lucene54/Lucene54DocValuesProducer.java | 39 ++++++++++-
.../codecs/memory/MemoryDocValuesProducer.java | 69 +++++++-------------
.../java/org/apache/lucene/util/LongValues.java | 21 +-----
.../lucene/util/packed/TestDirectPacked.java | 6 +-
.../search/join/ToParentBlockJoinQuery.java | 2 +-
.../lucene/search/join/TestBlockJoin.java | 66 +++++++++++++++++++
6 files changed, 134 insertions(+), 69 deletions(-)
----------------------------------------------------------------------
[24/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7501: BKDReader
should not store the split dimension explicitly in the 1D case.
Posted by kr...@apache.org.
LUCENE-7501: BKDReader should not store the split dimension explicitly in the 1D case.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/e8360714
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/e8360714
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/e8360714
Branch: refs/heads/jira/solr-8593
Commit: e8360714fae8bac3c705297641a735ed33cf47f9
Parents: b2188f4
Author: Adrien Grand <jp...@gmail.com>
Authored: Tue Oct 18 09:33:36 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Tue Oct 18 15:05:50 2016 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 ++
.../simpletext/SimpleTextPointsReader.java | 17 +++++++++--
.../org/apache/lucene/util/bkd/BKDReader.java | 32 +++++++++++---------
.../org/apache/lucene/util/bkd/BKDWriter.java | 14 ++++++---
4 files changed, 45 insertions(+), 21 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/e8360714/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index f3501cc..745d8fd 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -96,6 +96,9 @@ Improvements
Optimizations
+* LUCENE-7501: BKDReader should not store the split dimension explicitly in the
+ 1D case. (Adrien Grand)
+
Other
* LUCENE-7452: Block join query exception suggests how to find a doc, which
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/e8360714/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java
----------------------------------------------------------------------
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java
index e3b880a..f7ff16e 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java
@@ -139,15 +139,26 @@ class SimpleTextPointsReader extends PointsReader {
readLine(dataIn);
count = parseInt(SPLIT_COUNT);
- byte[] splitPackedValues = new byte[count * (1 + bytesPerDim)];
+ byte[] splitPackedValues;
+ int bytesPerIndexEntry;
+ if (numDims == 1) {
+ bytesPerIndexEntry = bytesPerDim;
+ } else {
+ bytesPerIndexEntry = 1 + bytesPerDim;
+ }
+ splitPackedValues = new byte[count * bytesPerIndexEntry];
for(int i=0;i<count;i++) {
readLine(dataIn);
- splitPackedValues[(1 + bytesPerDim) * i] = (byte) parseInt(SPLIT_DIM);
+ int address = bytesPerIndexEntry * i;
+ int splitDim = parseInt(SPLIT_DIM);
+ if (numDims != 1) {
+ splitPackedValues[address++] = (byte) splitDim;
+ }
readLine(dataIn);
assert startsWith(SPLIT_VALUE);
BytesRef br = SimpleTextUtil.fromBytesRefString(stripPrefix(SPLIT_VALUE));
assert br.length == bytesPerDim;
- System.arraycopy(br.bytes, br.offset, splitPackedValues, (1 + bytesPerDim) * i + 1, bytesPerDim);
+ System.arraycopy(br.bytes, br.offset, splitPackedValues, address, bytesPerDim);
}
return new SimpleTextBKDReader(dataIn, numDims, maxPointsInLeafNode, bytesPerDim, leafBlockFPs, splitPackedValues, minValue.bytes, maxValue.bytes, pointCount, docCount);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/e8360714/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java b/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java
index 0fe549a..1ddb566 100644
--- a/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java
+++ b/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java
@@ -25,6 +25,7 @@ import org.apache.lucene.index.PointValues;
import org.apache.lucene.store.IndexInput;
import org.apache.lucene.util.Accountable;
import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.RamUsageEstimator;
import org.apache.lucene.util.StringHelper;
/** Handles intersection of an multi-dimensional shape in byte[] space with a block KD-tree previously written with {@link BKDWriter}.
@@ -38,6 +39,7 @@ public class BKDReader extends PointValues implements Accountable {
final private int leafNodeOffset;
final int numDims;
final int bytesPerDim;
+ final int bytesPerIndexEntry;
final IndexInput in;
final int maxPointsInLeafNode;
final byte[] minPackedValue;
@@ -53,6 +55,7 @@ public class BKDReader extends PointValues implements Accountable {
numDims = in.readVInt();
maxPointsInLeafNode = in.readVInt();
bytesPerDim = in.readVInt();
+ bytesPerIndexEntry = numDims == 1 && version >= BKDWriter.VERSION_IMPLICIT_SPLIT_DIM_1D ? bytesPerDim : bytesPerDim + 1;
packedBytesLength = numDims * bytesPerDim;
// Read index:
@@ -69,7 +72,7 @@ public class BKDReader extends PointValues implements Accountable {
pointCount = in.readVLong();
docCount = in.readVInt();
- splitPackedValues = new byte[(1+bytesPerDim)*numLeaves];
+ splitPackedValues = new byte[bytesPerIndexEntry*numLeaves];
// TODO: don't write split packed values[0]!
in.readBytes(splitPackedValues, 0, splitPackedValues.length);
@@ -134,6 +137,7 @@ public class BKDReader extends PointValues implements Accountable {
this.numDims = numDims;
this.maxPointsInLeafNode = maxPointsInLeafNode;
this.bytesPerDim = bytesPerDim;
+ bytesPerIndexEntry = numDims == 1 ? bytesPerDim : bytesPerDim + 1;
packedBytesLength = numDims * bytesPerDim;
this.leafNodeOffset = leafBlockFPs.length;
this.leafBlockFPs = leafBlockFPs;
@@ -233,22 +237,22 @@ public class BKDReader extends PointValues implements Accountable {
} else {
// Non-leaf node:
- int address = nodeID * (bytesPerDim+1);
- int splitDim = splitPackedValues[address] & 0xff;
+ int address = nodeID * bytesPerIndexEntry;
+ int splitDim = numDims == 1 ? 0 : splitPackedValues[address++] & 0xff;
assert splitDim < numDims;
byte[] splitPackedValue = new byte[packedBytesLength];
// Recurse on left sub-tree:
System.arraycopy(cellMaxPacked, 0, splitPackedValue, 0, packedBytesLength);
- System.arraycopy(splitPackedValues, address+1, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
+ System.arraycopy(splitPackedValues, address, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
verify(state,
2*nodeID,
cellMinPacked, splitPackedValue);
// Recurse on right sub-tree:
System.arraycopy(cellMinPacked, 0, splitPackedValue, 0, packedBytesLength);
- System.arraycopy(splitPackedValues, address+1, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
+ System.arraycopy(splitPackedValues, address, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
verify(state,
2*nodeID+1,
splitPackedValue, cellMaxPacked);
@@ -456,8 +460,8 @@ public class BKDReader extends PointValues implements Accountable {
// Non-leaf node: recurse on the split left and right nodes
// TODO: save the unused 1 byte prefix (it's always 0) in the 1d case here:
- int address = nodeID * (bytesPerDim+1);
- int splitDim = splitPackedValues[address] & 0xff;
+ int address = nodeID * bytesPerIndexEntry;
+ int splitDim = numDims == 1 ? 0 : splitPackedValues[address++] & 0xff;
assert splitDim < numDims;
// TODO: can we alloc & reuse this up front?
@@ -467,14 +471,14 @@ public class BKDReader extends PointValues implements Accountable {
// Recurse on left sub-tree:
System.arraycopy(cellMaxPacked, 0, splitPackedValue, 0, packedBytesLength);
- System.arraycopy(splitPackedValues, address+1, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
+ System.arraycopy(splitPackedValues, address, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
intersect(state,
2*nodeID,
cellMinPacked, splitPackedValue);
// Recurse on right sub-tree:
System.arraycopy(cellMinPacked, 0, splitPackedValue, 0, packedBytesLength);
- System.arraycopy(splitPackedValues, address+1, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
+ System.arraycopy(splitPackedValues, address, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
intersect(state,
2*nodeID+1,
splitPackedValue, cellMaxPacked);
@@ -483,16 +487,16 @@ public class BKDReader extends PointValues implements Accountable {
/** Copies the split value for this node into the provided byte array */
public void copySplitValue(int nodeID, byte[] splitPackedValue) {
- int address = nodeID * (bytesPerDim+1);
- int splitDim = splitPackedValues[address] & 0xff;
+ int address = nodeID * bytesPerIndexEntry;
+ int splitDim = numDims == 1 ? 0 : splitPackedValues[address++] & 0xff;
assert splitDim < numDims;
- System.arraycopy(splitPackedValues, address+1, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
+ System.arraycopy(splitPackedValues, address, splitPackedValue, splitDim*bytesPerDim, bytesPerDim);
}
@Override
public long ramBytesUsed() {
- return splitPackedValues.length +
- leafBlockFPs.length * Long.BYTES;
+ return RamUsageEstimator.sizeOf(splitPackedValues) +
+ RamUsageEstimator.sizeOf(leafBlockFPs);
}
@Override
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/e8360714/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java b/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
index 6ee178b..5526624 100644
--- a/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
@@ -82,7 +82,8 @@ public class BKDWriter implements Closeable {
public static final int VERSION_START = 0;
public static final int VERSION_COMPRESSED_DOC_IDS = 1;
public static final int VERSION_COMPRESSED_VALUES = 2;
- public static final int VERSION_CURRENT = VERSION_COMPRESSED_VALUES;
+ public static final int VERSION_IMPLICIT_SPLIT_DIM_1D = 3;
+ public static final int VERSION_CURRENT = VERSION_IMPLICIT_SPLIT_DIM_1D;
/** How many bytes each docs takes in the fixed-width offline format */
private final int bytesPerDoc;
@@ -1033,10 +1034,15 @@ public class BKDWriter implements Closeable {
out.writeVLong(pointCount);
out.writeVInt(docsSeen.cardinality());
- // TODO: for 1D case, don't waste the first byte of each split value (it's always 0)
-
// NOTE: splitPackedValues[0] is unused, because nodeID is 1-based:
- out.writeBytes(splitPackedValues, 0, splitPackedValues.length);
+ if (numDims == 1) {
+ // write the index, skipping the byte used to store the split dim since it is always 0
+ for (int i = 1; i < splitPackedValues.length; i += 1 + bytesPerDim) {
+ out.writeBytes(splitPackedValues, i, bytesPerDim);
+ }
+ } else {
+ out.writeBytes(splitPackedValues, 0, splitPackedValues.length);
+ }
long lastFP = 0;
for (int i=0;i<leafBlockFPs.length;i++) {
[16/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7494: Give points
a per-field API.
Posted by kr...@apache.org.
LUCENE-7494: Give points a per-field API.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/c22725f0
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/c22725f0
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/c22725f0
Branch: refs/heads/jira/solr-8593
Commit: c22725f0b53a0d1a50cbc5a9d21ced29e6d7cd39
Parents: 19c7f8e
Author: Adrien Grand <jp...@gmail.com>
Authored: Mon Oct 17 18:09:23 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Tue Oct 18 10:14:27 2016 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 2 +
.../codecs/simpletext/SimpleTextBKDReader.java | 1 -
.../simpletext/SimpleTextPointsReader.java | 81 +-------
.../simpletext/SimpleTextPointsWriter.java | 10 +-
.../lucene/codecs/MutablePointValues.java | 42 ++++
.../lucene/codecs/MutablePointsReader.java | 41 ----
.../org/apache/lucene/codecs/PointsFormat.java | 35 +---
.../org/apache/lucene/codecs/PointsReader.java | 5 +-
.../org/apache/lucene/codecs/PointsWriter.java | 207 ++++++++++---------
.../codecs/lucene60/Lucene60PointsReader.java | 84 +-------
.../codecs/lucene60/Lucene60PointsWriter.java | 23 ++-
.../org/apache/lucene/index/CheckIndex.java | 19 +-
.../org/apache/lucene/index/CodecReader.java | 12 ++
.../apache/lucene/index/FilterCodecReader.java | 5 -
.../apache/lucene/index/FilterLeafReader.java | 4 +-
.../org/apache/lucene/index/LeafReader.java | 5 +-
.../apache/lucene/index/MergeReaderWrapper.java | 4 +-
.../apache/lucene/index/ParallelLeafReader.java | 97 +--------
.../org/apache/lucene/index/PointValues.java | 64 +++---
.../apache/lucene/index/PointValuesWriter.java | 66 +++---
.../org/apache/lucene/index/SegmentReader.java | 8 +-
.../lucene/index/SlowCodecReaderWrapper.java | 46 +----
.../apache/lucene/index/SortingLeafReader.java | 33 ++-
.../apache/lucene/search/PointInSetQuery.java | 24 +--
.../apache/lucene/search/PointRangeQuery.java | 28 ++-
.../org/apache/lucene/util/DocIdSetBuilder.java | 2 +-
.../org/apache/lucene/util/bkd/BKDReader.java | 13 +-
.../org/apache/lucene/util/bkd/BKDWriter.java | 24 +--
.../util/bkd/MutablePointsReaderUtils.java | 10 +-
.../org/apache/lucene/index/Test2BPoints.java | 4 +-
.../apache/lucene/index/TestPointValues.java | 25 ++-
.../apache/lucene/util/TestDocIdSetBuilder.java | 14 +-
.../util/bkd/TestMutablePointsReaderUtils.java | 33 +--
.../search/highlight/TermVectorLeafReader.java | 2 +-
.../join/PointInSetIncludingScoreQuery.java | 10 +-
.../apache/lucene/index/memory/MemoryIndex.java | 91 +++-----
.../lucene/index/memory/TestMemoryIndex.java | 6 +-
.../org/apache/lucene/document/LatLonPoint.java | 9 +-
.../document/LatLonPointDistanceQuery.java | 4 +-
.../document/LatLonPointInPolygonQuery.java | 4 +-
.../apache/lucene/document/RangeFieldQuery.java | 8 +-
.../spatial3d/PointInGeo3DShapeQuery.java | 4 +-
.../apache/lucene/spatial3d/TestGeo3DPoint.java | 4 +-
.../codecs/asserting/AssertingPointsFormat.java | 139 +------------
.../codecs/cranky/CrankyPointsFormat.java | 110 +++++-----
.../lucene/index/AssertingLeafReader.java | 155 +++++++++++++-
.../lucene/index/BasePointsFormatTestCase.java | 26 +--
.../org/apache/lucene/index/RandomCodec.java | 9 +-
.../org/apache/lucene/search/QueryUtils.java | 2 +-
.../org/apache/lucene/util/LuceneTestCase.java | 4 +-
.../solr/index/SlowCompositeReaderWrapper.java | 2 +-
.../apache/solr/uninverting/FieldCacheImpl.java | 22 +-
.../test/org/apache/solr/search/TestDocSet.java | 2 +-
53 files changed, 688 insertions(+), 996 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 54329ab..6defb27 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -25,6 +25,8 @@ API Changes
* LUCENE-7475: Norms now support sparsity, allowing to pay for what is
actually used. (Adrien Grand)
+* LUCENE-7494: Points now have a per-field API, like doc values. (Adrien Grand)
+
Bug Fixes
Improvements
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextBKDReader.java
----------------------------------------------------------------------
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextBKDReader.java b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextBKDReader.java
index 35e9448..a2b784a 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextBKDReader.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextBKDReader.java
@@ -20,7 +20,6 @@ package org.apache.lucene.codecs.simpletext;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
-import org.apache.lucene.index.PointValues.IntersectVisitor;
import org.apache.lucene.store.IndexInput;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.BytesRefBuilder;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java
----------------------------------------------------------------------
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java
index 1477f17..e3b880a 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsReader.java
@@ -26,6 +26,7 @@ import org.apache.lucene.codecs.PointsReader;
import org.apache.lucene.index.CorruptIndexException;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.IndexFileNames;
+import org.apache.lucene.index.PointValues;
import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.store.BufferedChecksumIndexInput;
import org.apache.lucene.store.ChecksumIndexInput;
@@ -174,7 +175,8 @@ class SimpleTextPointsReader extends PointsReader {
return new String(scratch.bytes(), prefix.length, scratch.length() - prefix.length, StandardCharsets.UTF_8);
}
- private BKDReader getBKDReader(String fieldName) {
+ @Override
+ public PointValues getValues(String fieldName) throws IOException {
FieldInfo fieldInfo = readState.fieldInfos.fieldInfo(fieldName);
if (fieldInfo == null) {
throw new IllegalArgumentException("field=\"" + fieldName + "\" is unrecognized");
@@ -185,18 +187,6 @@ class SimpleTextPointsReader extends PointsReader {
return readers.get(fieldName);
}
- /** Finds all documents and points matching the provided visitor */
- @Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this field were deleted in this segment:
- return;
- }
- bkdReader.intersect(visitor);
- }
-
@Override
public void checkIntegrity() throws IOException {
BytesRefBuilder scratch = new BytesRefBuilder();
@@ -234,69 +224,4 @@ class SimpleTextPointsReader extends PointsReader {
return "SimpleTextPointsReader(segment=" + readState.segmentInfo.name + " maxDoc=" + readState.segmentInfo.maxDoc() + ")";
}
- @Override
- public byte[] getMinPackedValue(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this field were deleted in this segment:
- return null;
- }
- return bkdReader.getMinPackedValue();
- }
-
- @Override
- public byte[] getMaxPackedValue(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this field were deleted in this segment:
- return null;
- }
- return bkdReader.getMaxPackedValue();
- }
-
- @Override
- public int getNumDimensions(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this field were deleted in this segment:
- return 0;
- }
- return bkdReader.getNumDimensions();
- }
-
- @Override
- public int getBytesPerDimension(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this field were deleted in this segment:
- return 0;
- }
- return bkdReader.getBytesPerDimension();
- }
-
- @Override
- public long size(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this field were deleted in this segment:
- return 0;
- }
- return bkdReader.getPointCount();
- }
-
- @Override
- public int getDocCount(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this field were deleted in this segment:
- return 0;
- }
- return bkdReader.getDocCount();
- }
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsWriter.java
----------------------------------------------------------------------
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsWriter.java b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsWriter.java
index 8d5c034..c06c128 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsWriter.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextPointsWriter.java
@@ -26,6 +26,7 @@ import org.apache.lucene.codecs.PointsReader;
import org.apache.lucene.codecs.PointsWriter;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.IndexFileNames;
+import org.apache.lucene.index.PointValues;
import org.apache.lucene.index.PointValues.IntersectVisitor;
import org.apache.lucene.index.PointValues.Relation;
import org.apache.lucene.index.SegmentWriteState;
@@ -68,9 +69,10 @@ class SimpleTextPointsWriter extends PointsWriter {
}
@Override
- public void writeField(FieldInfo fieldInfo, PointsReader values) throws IOException {
+ public void writeField(FieldInfo fieldInfo, PointsReader reader) throws IOException {
- boolean singleValuePerDoc = values.size(fieldInfo.name) == values.getDocCount(fieldInfo.name);
+ PointValues values = reader.getValues(fieldInfo.name);
+ boolean singleValuePerDoc = values.size() == values.getDocCount();
// We use the normal BKDWriter, but subclass to customize how it writes the index and blocks to disk:
try (BKDWriter writer = new BKDWriter(writeState.segmentInfo.maxDoc(),
@@ -80,7 +82,7 @@ class SimpleTextPointsWriter extends PointsWriter {
fieldInfo.getPointNumBytes(),
BKDWriter.DEFAULT_MAX_POINTS_IN_LEAF_NODE,
BKDWriter.DEFAULT_MAX_MB_SORT_IN_HEAP,
- values.size(fieldInfo.name),
+ values.size(),
singleValuePerDoc) {
@Override
@@ -173,7 +175,7 @@ class SimpleTextPointsWriter extends PointsWriter {
}
}) {
- values.intersect(fieldInfo.name, new IntersectVisitor() {
+ values.intersect(new IntersectVisitor() {
@Override
public void visit(int docID) {
throw new IllegalStateException();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/codecs/MutablePointValues.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/MutablePointValues.java b/lucene/core/src/java/org/apache/lucene/codecs/MutablePointValues.java
new file mode 100644
index 0000000..8f4d69c
--- /dev/null
+++ b/lucene/core/src/java/org/apache/lucene/codecs/MutablePointValues.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs;
+
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.util.BytesRef;
+
+/** {@link PointValues} whose order of points can be changed.
+ * This class is useful for codecs to optimize flush.
+ * @lucene.internal */
+public abstract class MutablePointValues extends PointValues {
+
+ /** Sole constructor. */
+ protected MutablePointValues() {}
+
+ /** Set {@code packedValue} with a reference to the packed bytes of the i-th value. */
+ public abstract void getValue(int i, BytesRef packedValue);
+
+ /** Get the k-th byte of the i-th value. */
+ public abstract byte getByteAt(int i, int k);
+
+ /** Return the doc ID of the i-th value. */
+ public abstract int getDocID(int i);
+
+ /** Swap the i-th and j-th values. */
+ public abstract void swap(int i, int j);
+
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/codecs/MutablePointsReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/MutablePointsReader.java b/lucene/core/src/java/org/apache/lucene/codecs/MutablePointsReader.java
deleted file mode 100644
index dccca26..0000000
--- a/lucene/core/src/java/org/apache/lucene/codecs/MutablePointsReader.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.lucene.codecs;
-
-import org.apache.lucene.util.BytesRef;
-
-/** {@link PointsReader} whose order of points can be changed.
- * This class is useful for codecs to optimize flush.
- * @lucene.internal */
-public abstract class MutablePointsReader extends PointsReader {
-
- /** Sole constructor. */
- protected MutablePointsReader() {}
-
- /** Set {@code packedValue} with a reference to the packed bytes of the i-th value. */
- public abstract void getValue(int i, BytesRef packedValue);
-
- /** Get the k-th byte of the i-th value. */
- public abstract byte getByteAt(int i, int k);
-
- /** Return the doc ID of the i-th value. */
- public abstract int getDocID(int i);
-
- /** Swap the i-th and j-th values. */
- public abstract void swap(int i, int j);
-
-}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/codecs/PointsFormat.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/PointsFormat.java b/lucene/core/src/java/org/apache/lucene/codecs/PointsFormat.java
index e49bf53..1723e42 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/PointsFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/PointsFormat.java
@@ -19,6 +19,7 @@ package org.apache.lucene.codecs;
import java.io.IOException;
+import org.apache.lucene.index.PointValues;
import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.index.SegmentWriteState;
@@ -72,38 +73,8 @@ public abstract class PointsFormat {
}
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) {
- throw new IllegalArgumentException("field=\"" + fieldName + "\" was not indexed with points");
- }
-
- @Override
- public byte[] getMinPackedValue(String fieldName) {
- throw new IllegalArgumentException("field=\"" + fieldName + "\" was not indexed with points");
- }
-
- @Override
- public byte[] getMaxPackedValue(String fieldName) {
- throw new IllegalArgumentException("field=\"" + fieldName + "\" was not indexed with points");
- }
-
- @Override
- public int getNumDimensions(String fieldName) {
- throw new IllegalArgumentException("field=\"" + fieldName + "\" was not indexed with points");
- }
-
- @Override
- public int getBytesPerDimension(String fieldName) {
- throw new IllegalArgumentException("field=\"" + fieldName + "\" was not indexed with points");
- }
-
- @Override
- public long size(String fieldName) {
- throw new IllegalArgumentException("field=\"" + fieldName + "\" was not indexed with points");
- }
-
- @Override
- public int getDocCount(String fieldName) {
- throw new IllegalArgumentException("field=\"" + fieldName + "\" was not indexed with points");
+ public PointValues getValues(String field) {
+ throw new IllegalArgumentException("field=\"" + field + "\" was not indexed with points");
}
};
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/codecs/PointsReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/PointsReader.java b/lucene/core/src/java/org/apache/lucene/codecs/PointsReader.java
index ab21431..b20614a 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/PointsReader.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/PointsReader.java
@@ -27,7 +27,7 @@ import org.apache.lucene.util.Accountable;
*
* @lucene.experimental
*/
-public abstract class PointsReader extends PointValues implements Closeable, Accountable {
+public abstract class PointsReader implements Closeable, Accountable {
/** Sole constructor. (For invocation by subclass constructors, typically implicit.) */
protected PointsReader() {}
@@ -41,6 +41,9 @@ public abstract class PointsReader extends PointValues implements Closeable, Acc
*/
public abstract void checkIntegrity() throws IOException;
+ /** Return {@link PointValues} for the given {@code field}. */
+ public abstract PointValues getValues(String field) throws IOException;
+
/**
* Returns an instance optimized for merging.
* <p>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/codecs/PointsWriter.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/PointsWriter.java b/lucene/core/src/java/org/apache/lucene/codecs/PointsWriter.java
index eac1716..38cd440 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/PointsWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/PointsWriter.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.MergeState;
+import org.apache.lucene.index.PointValues;
/** Abstract API to write points
*
@@ -48,109 +49,121 @@ public abstract class PointsWriter implements Closeable {
if (pointsReader != null) {
FieldInfo readerFieldInfo = mergeState.fieldInfos[i].fieldInfo(fieldInfo.name);
if (readerFieldInfo != null && readerFieldInfo.getPointDimensionCount() > 0) {
- maxPointCount += pointsReader.size(fieldInfo.name);
- docCount += pointsReader.getDocCount(fieldInfo.name);
+ PointValues values = pointsReader.getValues(fieldInfo.name);
+ if (values != null) {
+ maxPointCount += values.size();
+ docCount += values.getDocCount();
+ }
}
}
}
final long finalMaxPointCount = maxPointCount;
final int finalDocCount = docCount;
writeField(fieldInfo,
- new PointsReader() {
- @Override
- public void intersect(String fieldName, IntersectVisitor mergedVisitor) throws IOException {
- if (fieldName.equals(fieldInfo.name) == false) {
- throw new IllegalArgumentException("field name must match the field being merged");
- }
-
- for (int i=0;i<mergeState.pointsReaders.length;i++) {
- PointsReader pointsReader = mergeState.pointsReaders[i];
- if (pointsReader == null) {
- // This segment has no points
- continue;
- }
- FieldInfo readerFieldInfo = mergeState.fieldInfos[i].fieldInfo(fieldName);
- if (readerFieldInfo == null) {
- // This segment never saw this field
- continue;
- }
-
- if (readerFieldInfo.getPointDimensionCount() == 0) {
- // This segment saw this field, but the field did not index points in it:
- continue;
- }
-
- MergeState.DocMap docMap = mergeState.docMaps[i];
- pointsReader.intersect(fieldInfo.name,
- new IntersectVisitor() {
- @Override
- public void visit(int docID) {
- // Should never be called because our compare method never returns Relation.CELL_INSIDE_QUERY
- throw new IllegalStateException();
- }
-
- @Override
- public void visit(int docID, byte[] packedValue) throws IOException {
- int newDocID = docMap.get(docID);
- if (newDocID != -1) {
- // Not deleted:
- mergedVisitor.visit(newDocID, packedValue);
- }
- }
-
- @Override
- public Relation compare(byte[] minPackedValue, byte[] maxPackedValue) {
- // Forces this segment's PointsReader to always visit all docs + values:
- return Relation.CELL_CROSSES_QUERY;
- }
- });
- }
- }
-
- @Override
- public void checkIntegrity() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long ramBytesUsed() {
- return 0L;
- }
-
- @Override
- public void close() {
- }
-
- @Override
- public byte[] getMinPackedValue(String fieldName) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public byte[] getMaxPackedValue(String fieldName) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getNumDimensions(String fieldName) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getBytesPerDimension(String fieldName) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long size(String fieldName) {
- return finalMaxPointCount;
- }
-
- @Override
- public int getDocCount(String fieldName) {
- return finalDocCount;
- }
- });
+ new PointsReader() {
+
+ @Override
+ public long ramBytesUsed() {
+ return 0;
+ }
+
+ @Override
+ public void close() throws IOException {}
+
+ @Override
+ public PointValues getValues(String fieldName) {
+ if (fieldName.equals(fieldInfo.name) == false) {
+ throw new IllegalArgumentException("field name must match the field being merged");
+ }
+
+ return new PointValues() {
+
+ @Override
+ public void intersect(IntersectVisitor mergedVisitor) throws IOException {
+ for (int i=0;i<mergeState.pointsReaders.length;i++) {
+ PointsReader pointsReader = mergeState.pointsReaders[i];
+ if (pointsReader == null) {
+ // This segment has no points
+ continue;
+ }
+ FieldInfo readerFieldInfo = mergeState.fieldInfos[i].fieldInfo(fieldName);
+ if (readerFieldInfo == null) {
+ // This segment never saw this field
+ continue;
+ }
+
+ if (readerFieldInfo.getPointDimensionCount() == 0) {
+ // This segment saw this field, but the field did not index points in it:
+ continue;
+ }
+
+ PointValues values = pointsReader.getValues(fieldName);
+ if (values == null) {
+ continue;
+ }
+ MergeState.DocMap docMap = mergeState.docMaps[i];
+ values.intersect(new IntersectVisitor() {
+ @Override
+ public void visit(int docID) {
+ // Should never be called because our compare method never returns Relation.CELL_INSIDE_QUERY
+ throw new IllegalStateException();
+ }
+
+ @Override
+ public void visit(int docID, byte[] packedValue) throws IOException {
+ int newDocID = docMap.get(docID);
+ if (newDocID != -1) {
+ // Not deleted:
+ mergedVisitor.visit(newDocID, packedValue);
+ }
+ }
+
+ @Override
+ public Relation compare(byte[] minPackedValue, byte[] maxPackedValue) {
+ // Forces this segment's PointsReader to always visit all docs + values:
+ return Relation.CELL_CROSSES_QUERY;
+ }
+ });
+ }
+ }
+
+ @Override
+ public byte[] getMinPackedValue() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte[] getMaxPackedValue() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public int getNumDimensions() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public int getBytesPerDimension() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public long size() {
+ return finalMaxPointCount;
+ }
+
+ @Override
+ public int getDocCount() {
+ return finalDocCount;
+ }
+ };
+ }
+
+ @Override
+ public void checkIntegrity() throws IOException {
+ throw new UnsupportedOperationException();
+ }
+ });
}
/** Default merge implementation to merge incoming points readers by visiting all their points and
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsReader.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsReader.java
index 8c91a99..0eecdbb 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsReader.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsReader.java
@@ -30,6 +30,7 @@ import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.codecs.PointsReader;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.IndexFileNames;
+import org.apache.lucene.index.PointValues;
import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.store.ChecksumIndexInput;
import org.apache.lucene.store.IndexInput;
@@ -117,7 +118,8 @@ public class Lucene60PointsReader extends PointsReader implements Closeable {
/** Returns the underlying {@link BKDReader}.
*
* @lucene.internal */
- public BKDReader getBKDReader(String fieldName) {
+ @Override
+ public PointValues getValues(String fieldName) {
FieldInfo fieldInfo = readState.fieldInfos.fieldInfo(fieldName);
if (fieldInfo == null) {
throw new IllegalArgumentException("field=\"" + fieldName + "\" is unrecognized");
@@ -130,19 +132,6 @@ public class Lucene60PointsReader extends PointsReader implements Closeable {
}
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- BKDReader bkdReader = getBKDReader(fieldName);
-
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this point field were deleted in this segment:
- return;
- }
-
- bkdReader.intersect(visitor);
- }
-
- @Override
public long ramBytesUsed() {
long sizeInBytes = 0;
for(BKDReader reader : readers.values()) {
@@ -173,72 +162,5 @@ public class Lucene60PointsReader extends PointsReader implements Closeable {
readers.clear();
}
- @Override
- public byte[] getMinPackedValue(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this point field were deleted in this segment:
- return null;
- }
-
- return bkdReader.getMinPackedValue();
- }
-
- @Override
- public byte[] getMaxPackedValue(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this point field were deleted in this segment:
- return null;
- }
-
- return bkdReader.getMaxPackedValue();
- }
-
- @Override
- public int getNumDimensions(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this point field were deleted in this segment:
- return 0;
- }
- return bkdReader.getNumDimensions();
- }
-
- @Override
- public int getBytesPerDimension(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this point field were deleted in this segment:
- return 0;
- }
- return bkdReader.getBytesPerDimension();
- }
-
- @Override
- public long size(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this point field were deleted in this segment:
- return 0;
- }
- return bkdReader.getPointCount();
- }
-
- @Override
- public int getDocCount(String fieldName) {
- BKDReader bkdReader = getBKDReader(fieldName);
- if (bkdReader == null) {
- // Schema ghost corner case! This field did index points in the past, but
- // now all docs having this point field were deleted in this segment:
- return 0;
- }
- return bkdReader.getDocCount();
- }
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsWriter.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsWriter.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsWriter.java
index 05a1bf9..0f3af85 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene60/Lucene60PointsWriter.java
@@ -25,13 +25,14 @@ import java.util.List;
import java.util.Map;
import org.apache.lucene.codecs.CodecUtil;
-import org.apache.lucene.codecs.MutablePointsReader;
+import org.apache.lucene.codecs.MutablePointValues;
import org.apache.lucene.codecs.PointsReader;
import org.apache.lucene.codecs.PointsWriter;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.FieldInfos;
import org.apache.lucene.index.IndexFileNames;
import org.apache.lucene.index.MergeState;
+import org.apache.lucene.index.PointValues;
import org.apache.lucene.index.PointValues.IntersectVisitor;
import org.apache.lucene.index.PointValues.Relation;
import org.apache.lucene.index.SegmentWriteState;
@@ -85,9 +86,10 @@ public class Lucene60PointsWriter extends PointsWriter implements Closeable {
}
@Override
- public void writeField(FieldInfo fieldInfo, PointsReader values) throws IOException {
+ public void writeField(FieldInfo fieldInfo, PointsReader reader) throws IOException {
- boolean singleValuePerDoc = values.size(fieldInfo.name) == values.getDocCount(fieldInfo.name);
+ PointValues values = reader.getValues(fieldInfo.name);
+ boolean singleValuePerDoc = values.size() == values.getDocCount();
try (BKDWriter writer = new BKDWriter(writeState.segmentInfo.maxDoc(),
writeState.directory,
@@ -96,18 +98,18 @@ public class Lucene60PointsWriter extends PointsWriter implements Closeable {
fieldInfo.getPointNumBytes(),
maxPointsInLeafNode,
maxMBSortInHeap,
- values.size(fieldInfo.name),
+ values.size(),
singleValuePerDoc)) {
- if (values instanceof MutablePointsReader) {
- final long fp = writer.writeField(dataOut, fieldInfo.name, (MutablePointsReader) values);
+ if (values instanceof MutablePointValues) {
+ final long fp = writer.writeField(dataOut, fieldInfo.name, (MutablePointValues) values);
if (fp != -1) {
indexFPs.put(fieldInfo.name, fp);
}
return;
}
- values.intersect(fieldInfo.name, new IntersectVisitor() {
+ values.intersect(new IntersectVisitor() {
@Override
public void visit(int docID) {
throw new IllegalStateException();
@@ -166,8 +168,11 @@ public class Lucene60PointsWriter extends PointsWriter implements Closeable {
FieldInfos readerFieldInfos = mergeState.fieldInfos[i];
FieldInfo readerFieldInfo = readerFieldInfos.fieldInfo(fieldInfo.name);
if (readerFieldInfo != null && readerFieldInfo.getPointDimensionCount() > 0) {
- totMaxSize += reader.size(fieldInfo.name);
- singleValuePerDoc &= reader.size(fieldInfo.name) == reader.getDocCount(fieldInfo.name);
+ PointValues values = reader.getValues(fieldInfo.name);
+ if (values != null) {
+ totMaxSize += values.size();
+ singleValuePerDoc &= values.size() == values.getDocCount();
+ }
}
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/CheckIndex.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/CheckIndex.java b/lucene/core/src/java/org/apache/lucene/index/CheckIndex.java
index 27d7c50..aec7537 100644
--- a/lucene/core/src/java/org/apache/lucene/index/CheckIndex.java
+++ b/lucene/core/src/java/org/apache/lucene/index/CheckIndex.java
@@ -1793,8 +1793,8 @@ public final class CheckIndex implements Closeable {
try {
if (fieldInfos.hasPointValues()) {
- PointsReader values = reader.getPointsReader();
- if (values == null) {
+ PointsReader pointsReader = reader.getPointsReader();
+ if (pointsReader == null) {
throw new RuntimeException("there are fields with points, but reader.getPointsReader() is null");
}
for (FieldInfo fieldInfo : fieldInfos) {
@@ -1812,9 +1812,13 @@ public final class CheckIndex implements Closeable {
long[] pointCountSeen = new long[1];
- byte[] globalMinPackedValue = values.getMinPackedValue(fieldInfo.name);
- long size = values.size(fieldInfo.name);
- int docCount = values.getDocCount(fieldInfo.name);
+ PointValues values = pointsReader.getValues(fieldInfo.name);
+ if (values == null) {
+ continue;
+ }
+ byte[] globalMinPackedValue = values.getMinPackedValue();
+ long size = values.size();
+ int docCount = values.getDocCount();
if (docCount > size) {
throw new RuntimeException("point values for field \"" + fieldInfo.name + "\" claims to have size=" + size + " points and inconsistent docCount=" + docCount);
@@ -1831,7 +1835,7 @@ public final class CheckIndex implements Closeable {
} else if (globalMinPackedValue.length != packedBytesCount) {
throw new RuntimeException("getMinPackedValue for field \"" + fieldInfo.name + "\" return length=" + globalMinPackedValue.length + " array, but should be " + packedBytesCount);
}
- byte[] globalMaxPackedValue = values.getMaxPackedValue(fieldInfo.name);
+ byte[] globalMaxPackedValue = values.getMaxPackedValue();
if (globalMaxPackedValue == null) {
if (size != 0) {
throw new RuntimeException("getMaxPackedValue is null points for field \"" + fieldInfo.name + "\" yet size=" + size);
@@ -1840,8 +1844,7 @@ public final class CheckIndex implements Closeable {
throw new RuntimeException("getMaxPackedValue for field \"" + fieldInfo.name + "\" return length=" + globalMaxPackedValue.length + " array, but should be " + packedBytesCount);
}
- values.intersect(fieldInfo.name,
- new PointValues.IntersectVisitor() {
+ values.intersect(new PointValues.IntersectVisitor() {
private int lastDocID = -1;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/CodecReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/CodecReader.java b/lucene/core/src/java/org/apache/lucene/index/CodecReader.java
index b9ad73a..460a9d6 100644
--- a/lucene/core/src/java/org/apache/lucene/index/CodecReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/CodecReader.java
@@ -189,6 +189,18 @@ public abstract class CodecReader extends LeafReader implements Accountable {
}
@Override
+ public final PointValues getPointValues(String field) throws IOException {
+ ensureOpen();
+ FieldInfo fi = getFieldInfos().fieldInfo(field);
+ if (fi == null || fi.getPointDimensionCount() == 0) {
+ // Field does not exist or does not index points
+ return null;
+ }
+
+ return getPointsReader().getValues(field);
+ }
+
+ @Override
protected void doClose() throws IOException {
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java b/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java
index 13b6e8d..48cbda7 100644
--- a/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/FilterCodecReader.java
@@ -88,11 +88,6 @@ public abstract class FilterCodecReader extends CodecReader {
}
@Override
- public PointValues getPointValues() {
- return in.getPointValues();
- }
-
- @Override
public int numDocs() {
return in.numDocs();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/FilterLeafReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/FilterLeafReader.java b/lucene/core/src/java/org/apache/lucene/index/FilterLeafReader.java
index 9e9fcb4..9ed62e7 100644
--- a/lucene/core/src/java/org/apache/lucene/index/FilterLeafReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/FilterLeafReader.java
@@ -382,8 +382,8 @@ public abstract class FilterLeafReader extends LeafReader {
}
@Override
- public PointValues getPointValues() {
- return in.getPointValues();
+ public PointValues getPointValues(String field) throws IOException {
+ return in.getPointValues(field);
}
@Override
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/LeafReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/LeafReader.java b/lucene/core/src/java/org/apache/lucene/index/LeafReader.java
index 1b67767..acdd0d8 100644
--- a/lucene/core/src/java/org/apache/lucene/index/LeafReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/LeafReader.java
@@ -295,8 +295,9 @@ public abstract class LeafReader extends IndexReader {
public abstract Bits getLiveDocs();
/** Returns the {@link PointValues} used for numeric or
- * spatial searches, or null if there are no point fields. */
- public abstract PointValues getPointValues();
+ * spatial searches for the given field, or null if there
+ * are no point fields. */
+ public abstract PointValues getPointValues(String field) throws IOException;
/**
* Checks consistency of this reader.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/MergeReaderWrapper.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/MergeReaderWrapper.java b/lucene/core/src/java/org/apache/lucene/index/MergeReaderWrapper.java
index 4c06b38..7eb90df 100644
--- a/lucene/core/src/java/org/apache/lucene/index/MergeReaderWrapper.java
+++ b/lucene/core/src/java/org/apache/lucene/index/MergeReaderWrapper.java
@@ -197,8 +197,8 @@ class MergeReaderWrapper extends LeafReader {
}
@Override
- public PointValues getPointValues() {
- return in.getPointValues();
+ public PointValues getPointValues(String fieldName) throws IOException {
+ return in.getPointValues(fieldName);
}
@Override
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/ParallelLeafReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/ParallelLeafReader.java b/lucene/core/src/java/org/apache/lucene/index/ParallelLeafReader.java
index f4afa11..60886ea 100644
--- a/lucene/core/src/java/org/apache/lucene/index/ParallelLeafReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/ParallelLeafReader.java
@@ -321,99 +321,10 @@ public class ParallelLeafReader extends LeafReader {
}
@Override
- public PointValues getPointValues() {
- return new PointValues() {
- @Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- LeafReader reader = fieldToReader.get(fieldName);
- if (reader == null) {
- return;
- }
- PointValues dimValues = reader.getPointValues();
- if (dimValues == null) {
- return;
- }
- dimValues.intersect(fieldName, visitor);
- }
-
- @Override
- public byte[] getMinPackedValue(String fieldName) throws IOException {
- LeafReader reader = fieldToReader.get(fieldName);
- if (reader == null) {
- return null;
- }
- PointValues dimValues = reader.getPointValues();
- if (dimValues == null) {
- return null;
- }
- return dimValues.getMinPackedValue(fieldName);
- }
-
- @Override
- public byte[] getMaxPackedValue(String fieldName) throws IOException {
- LeafReader reader = fieldToReader.get(fieldName);
- if (reader == null) {
- return null;
- }
- PointValues dimValues = reader.getPointValues();
- if (dimValues == null) {
- return null;
- }
- return dimValues.getMaxPackedValue(fieldName);
- }
-
- @Override
- public int getNumDimensions(String fieldName) throws IOException {
- LeafReader reader = fieldToReader.get(fieldName);
- if (reader == null) {
- return 0;
- }
- PointValues dimValues = reader.getPointValues();
- if (dimValues == null) {
- return 0;
- }
- return dimValues.getNumDimensions(fieldName);
- }
-
- @Override
- public int getBytesPerDimension(String fieldName) throws IOException {
- LeafReader reader = fieldToReader.get(fieldName);
- if (reader == null) {
- return 0;
- }
- PointValues dimValues = reader.getPointValues();
- if (dimValues == null) {
- return 0;
- }
- return dimValues.getBytesPerDimension(fieldName);
- }
-
- @Override
- public long size(String fieldName) {
- LeafReader reader = fieldToReader.get(fieldName);
- if (reader == null) {
- return 0;
- }
- PointValues dimValues = reader.getPointValues();
- if (dimValues == null) {
- return 0;
- }
- return dimValues.size(fieldName);
- }
-
- @Override
- public int getDocCount(String fieldName) {
- LeafReader reader = fieldToReader.get(fieldName);
- if (reader == null) {
- return 0;
- }
- PointValues dimValues = reader.getPointValues();
- if (dimValues == null) {
- return 0;
- }
- return dimValues.getDocCount(fieldName);
- }
- };
+ public PointValues getPointValues(String fieldName) throws IOException {
+ ensureOpen();
+ LeafReader reader = fieldToReader.get(fieldName);
+ return reader == null ? null : reader.getPointValues(fieldName);
}
@Override
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/PointValues.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/PointValues.java b/lucene/core/src/java/org/apache/lucene/index/PointValues.java
index 528f971..ffac5f7 100644
--- a/lucene/core/src/java/org/apache/lucene/index/PointValues.java
+++ b/lucene/core/src/java/org/apache/lucene/index/PointValues.java
@@ -89,16 +89,14 @@ public abstract class PointValues {
/** Return the cumulated number of points across all leaves of the given
* {@link IndexReader}. Leaves that do not have points for the given field
* are ignored.
- * @see PointValues#size(String) */
+ * @see PointValues#size() */
public static long size(IndexReader reader, String field) throws IOException {
long size = 0;
for (LeafReaderContext ctx : reader.leaves()) {
- FieldInfo info = ctx.reader().getFieldInfos().fieldInfo(field);
- if (info == null || info.getPointDimensionCount() == 0) {
- continue;
+ PointValues values = ctx.reader().getPointValues(field);
+ if (values != null) {
+ size += values.size();
}
- PointValues values = ctx.reader().getPointValues();
- size += values.size(field);
}
return size;
}
@@ -106,16 +104,14 @@ public abstract class PointValues {
/** Return the cumulated number of docs that have points across all leaves
* of the given {@link IndexReader}. Leaves that do not have points for the
* given field are ignored.
- * @see PointValues#getDocCount(String) */
+ * @see PointValues#getDocCount() */
public static int getDocCount(IndexReader reader, String field) throws IOException {
int count = 0;
for (LeafReaderContext ctx : reader.leaves()) {
- FieldInfo info = ctx.reader().getFieldInfos().fieldInfo(field);
- if (info == null || info.getPointDimensionCount() == 0) {
- continue;
+ PointValues values = ctx.reader().getPointValues(field);
+ if (values != null) {
+ count += values.getDocCount();
}
- PointValues values = ctx.reader().getPointValues();
- count += values.getDocCount(field);
}
return count;
}
@@ -123,24 +119,23 @@ public abstract class PointValues {
/** Return the minimum packed values across all leaves of the given
* {@link IndexReader}. Leaves that do not have points for the given field
* are ignored.
- * @see PointValues#getMinPackedValue(String) */
+ * @see PointValues#getMinPackedValue() */
public static byte[] getMinPackedValue(IndexReader reader, String field) throws IOException {
byte[] minValue = null;
for (LeafReaderContext ctx : reader.leaves()) {
- FieldInfo info = ctx.reader().getFieldInfos().fieldInfo(field);
- if (info == null || info.getPointDimensionCount() == 0) {
+ PointValues values = ctx.reader().getPointValues(field);
+ if (values == null) {
continue;
}
- PointValues values = ctx.reader().getPointValues();
- byte[] leafMinValue = values.getMinPackedValue(field);
+ byte[] leafMinValue = values.getMinPackedValue();
if (leafMinValue == null) {
continue;
}
if (minValue == null) {
minValue = leafMinValue.clone();
} else {
- final int numDimensions = values.getNumDimensions(field);
- final int numBytesPerDimension = values.getBytesPerDimension(field);
+ final int numDimensions = values.getNumDimensions();
+ final int numBytesPerDimension = values.getBytesPerDimension();
for (int i = 0; i < numDimensions; ++i) {
int offset = i * numBytesPerDimension;
if (StringHelper.compare(numBytesPerDimension, leafMinValue, offset, minValue, offset) < 0) {
@@ -155,24 +150,23 @@ public abstract class PointValues {
/** Return the maximum packed values across all leaves of the given
* {@link IndexReader}. Leaves that do not have points for the given field
* are ignored.
- * @see PointValues#getMaxPackedValue(String) */
+ * @see PointValues#getMaxPackedValue() */
public static byte[] getMaxPackedValue(IndexReader reader, String field) throws IOException {
byte[] maxValue = null;
for (LeafReaderContext ctx : reader.leaves()) {
- FieldInfo info = ctx.reader().getFieldInfos().fieldInfo(field);
- if (info == null || info.getPointDimensionCount() == 0) {
+ PointValues values = ctx.reader().getPointValues(field);
+ if (values == null) {
continue;
}
- PointValues values = ctx.reader().getPointValues();
- byte[] leafMaxValue = values.getMaxPackedValue(field);
+ byte[] leafMaxValue = values.getMaxPackedValue();
if (leafMaxValue == null) {
continue;
}
if (maxValue == null) {
maxValue = leafMaxValue.clone();
} else {
- final int numDimensions = values.getNumDimensions(field);
- final int numBytesPerDimension = values.getBytesPerDimension(field);
+ final int numDimensions = values.getNumDimensions();
+ final int numBytesPerDimension = values.getBytesPerDimension();
for (int i = 0; i < numDimensions; ++i) {
int offset = i * numBytesPerDimension;
if (StringHelper.compare(numBytesPerDimension, leafMaxValue, offset, maxValue, offset) > 0) {
@@ -224,23 +218,23 @@ public abstract class PointValues {
/** Finds all documents and points matching the provided visitor.
* This method does not enforce live documents, so it's up to the caller
* to test whether each document is deleted, if necessary. */
- public abstract void intersect(String fieldName, IntersectVisitor visitor) throws IOException;
+ public abstract void intersect(IntersectVisitor visitor) throws IOException;
/** Returns minimum value for each dimension, packed, or null if {@link #size} is <code>0</code> */
- public abstract byte[] getMinPackedValue(String fieldName) throws IOException;
+ public abstract byte[] getMinPackedValue() throws IOException;
/** Returns maximum value for each dimension, packed, or null if {@link #size} is <code>0</code> */
- public abstract byte[] getMaxPackedValue(String fieldName) throws IOException;
+ public abstract byte[] getMaxPackedValue() throws IOException;
/** Returns how many dimensions were indexed */
- public abstract int getNumDimensions(String fieldName) throws IOException;
+ public abstract int getNumDimensions() throws IOException;
/** Returns the number of bytes per dimension */
- public abstract int getBytesPerDimension(String fieldName) throws IOException;
+ public abstract int getBytesPerDimension() throws IOException;
- /** Returns the total number of indexed points across all documents in this field. */
- public abstract long size(String fieldName);
+ /** Returns the total number of indexed points across all documents. */
+ public abstract long size();
- /** Returns the total number of documents that have indexed at least one point for this field. */
- public abstract int getDocCount(String fieldName);
+ /** Returns the total number of documents that have indexed at least one point. */
+ public abstract int getDocCount();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/PointValuesWriter.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/PointValuesWriter.java b/lucene/core/src/java/org/apache/lucene/index/PointValuesWriter.java
index daf1f33..fc50d52 100644
--- a/lucene/core/src/java/org/apache/lucene/index/PointValuesWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/PointValuesWriter.java
@@ -18,7 +18,7 @@ package org.apache.lucene.index;
import java.io.IOException;
-import org.apache.lucene.codecs.MutablePointsReader;
+import org.apache.lucene.codecs.MutablePointValues;
import org.apache.lucene.codecs.PointsReader;
import org.apache.lucene.codecs.PointsWriter;
import org.apache.lucene.util.ArrayUtil;
@@ -70,7 +70,7 @@ class PointValuesWriter {
}
public void flush(SegmentWriteState state, PointsWriter writer) throws IOException {
- PointsReader reader = new MutablePointsReader() {
+ PointValues values = new MutablePointValues() {
final int[] ords = new int[numPoints];
{
@@ -80,10 +80,7 @@ class PointValuesWriter {
}
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- if (fieldName.equals(fieldInfo.name) == false) {
- throw new IllegalArgumentException("fieldName must be the same");
- }
+ public void intersect(IntersectVisitor visitor) throws IOException {
final BytesRef scratch = new BytesRef();
final byte[] packedValue = new byte[packedBytesLength];
for(int i=0;i<numPoints;i++) {
@@ -95,52 +92,32 @@ class PointValuesWriter {
}
@Override
- public void checkIntegrity() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long ramBytesUsed() {
- return 0L;
- }
-
- @Override
- public void close() {
- }
-
- @Override
- public byte[] getMinPackedValue(String fieldName) {
+ public byte[] getMinPackedValue() {
throw new UnsupportedOperationException();
}
@Override
- public byte[] getMaxPackedValue(String fieldName) {
+ public byte[] getMaxPackedValue() {
throw new UnsupportedOperationException();
}
@Override
- public int getNumDimensions(String fieldName) {
+ public int getNumDimensions() {
throw new UnsupportedOperationException();
}
@Override
- public int getBytesPerDimension(String fieldName) {
+ public int getBytesPerDimension() {
throw new UnsupportedOperationException();
}
@Override
- public long size(String fieldName) {
- if (fieldName.equals(fieldInfo.name) == false) {
- throw new IllegalArgumentException("fieldName must be the same");
- }
+ public long size() {
return numPoints;
}
@Override
- public int getDocCount(String fieldName) {
- if (fieldName.equals(fieldInfo.name) == false) {
- throw new IllegalArgumentException("fieldName must be the same");
- }
+ public int getDocCount() {
return numDocs;
}
@@ -170,6 +147,31 @@ class PointValuesWriter {
}
};
+ PointsReader reader = new PointsReader() {
+
+ @Override
+ public PointValues getValues(String fieldName) {
+ if (fieldName.equals(fieldInfo.name) == false) {
+ throw new IllegalArgumentException("fieldName must be the same");
+ }
+ return values;
+ }
+
+ @Override
+ public void checkIntegrity() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public long ramBytesUsed() {
+ return 0L;
+ }
+
+ @Override
+ public void close() {
+ }
+ };
+
writer.writeField(fieldInfo, reader);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/SegmentReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/SegmentReader.java b/lucene/core/src/java/org/apache/lucene/index/SegmentReader.java
index ebbca4f..b01f0b8 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SegmentReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SegmentReader.java
@@ -230,7 +230,7 @@ public final class SegmentReader extends CodecReader {
}
@Override
- public PointValues getPointValues() {
+ public PointsReader getPointsReader() {
ensureOpen();
return core.pointsReader;
}
@@ -254,12 +254,6 @@ public final class SegmentReader extends CodecReader {
}
@Override
- public PointsReader getPointsReader() {
- ensureOpen();
- return core.pointsReader;
- }
-
- @Override
public String toString() {
// SegmentInfo.toString takes dir and number of
// *pending* deletions; so we reverse compute that here:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/SlowCodecReaderWrapper.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/SlowCodecReaderWrapper.java b/lucene/core/src/java/org/apache/lucene/index/SlowCodecReaderWrapper.java
index 618d375..d5b5c33 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SlowCodecReaderWrapper.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SlowCodecReaderWrapper.java
@@ -93,13 +93,8 @@ public final class SlowCodecReaderWrapper {
}
@Override
- public PointValues getPointValues() {
- return reader.getPointValues();
- }
-
- @Override
public PointsReader getPointsReader() {
- return pointValuesToReader(reader.getPointValues());
+ return pointValuesToReader(reader);
}
@Override
@@ -140,14 +135,12 @@ public final class SlowCodecReaderWrapper {
}
}
- private static PointsReader pointValuesToReader(PointValues values) {
- if (values == null) {
- return null;
- }
+ private static PointsReader pointValuesToReader(LeafReader reader) {
return new PointsReader() {
+
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- values.intersect(fieldName, visitor);
+ public PointValues getValues(String field) throws IOException {
+ return reader.getPointValues(field);
}
@Override
@@ -164,35 +157,6 @@ public final class SlowCodecReaderWrapper {
return 0;
}
- @Override
- public byte[] getMinPackedValue(String fieldName) throws IOException {
- return values.getMinPackedValue(fieldName);
- }
-
- @Override
- public byte[] getMaxPackedValue(String fieldName) throws IOException {
- return values.getMaxPackedValue(fieldName);
- }
-
- @Override
- public int getNumDimensions(String fieldName) throws IOException {
- return values.getNumDimensions(fieldName);
- }
-
- @Override
- public int getBytesPerDimension(String fieldName) throws IOException {
- return values.getBytesPerDimension(fieldName);
- }
-
- @Override
- public long size(String fieldName) {
- return values.size(fieldName);
- }
-
- @Override
- public int getDocCount(String fieldName) {
- return values.getDocCount(fieldName);
- }
};
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/index/SortingLeafReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/SortingLeafReader.java b/lucene/core/src/java/org/apache/lucene/index/SortingLeafReader.java
index b207432..8139ed1 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SortingLeafReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SortingLeafReader.java
@@ -297,9 +297,8 @@ class SortingLeafReader extends FilterLeafReader {
}
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- in.intersect(fieldName,
- new IntersectVisitor() {
+ public void intersect(IntersectVisitor visitor) throws IOException {
+ in.intersect(new IntersectVisitor() {
@Override
public void visit(int docID) throws IOException {
visitor.visit(docMap.oldToNew(docID));
@@ -318,33 +317,33 @@ class SortingLeafReader extends FilterLeafReader {
}
@Override
- public byte[] getMinPackedValue(String fieldName) throws IOException {
- return in.getMinPackedValue(fieldName);
+ public byte[] getMinPackedValue() throws IOException {
+ return in.getMinPackedValue();
}
@Override
- public byte[] getMaxPackedValue(String fieldName) throws IOException {
- return in.getMaxPackedValue(fieldName);
+ public byte[] getMaxPackedValue() throws IOException {
+ return in.getMaxPackedValue();
}
@Override
- public int getNumDimensions(String fieldName) throws IOException {
- return in.getNumDimensions(fieldName);
+ public int getNumDimensions() throws IOException {
+ return in.getNumDimensions();
}
@Override
- public int getBytesPerDimension(String fieldName) throws IOException {
- return in.getBytesPerDimension(fieldName);
+ public int getBytesPerDimension() throws IOException {
+ return in.getBytesPerDimension();
}
@Override
- public long size(String fieldName) {
- return in.size(fieldName);
+ public long size() {
+ return in.size();
}
@Override
- public int getDocCount(String fieldName) {
- return in.getDocCount(fieldName);
+ public int getDocCount() {
+ return in.getDocCount();
}
}
@@ -1049,8 +1048,8 @@ class SortingLeafReader extends FilterLeafReader {
}
@Override
- public PointValues getPointValues() {
- final PointValues inPointValues = in.getPointValues();
+ public PointValues getPointValues(String fieldName) throws IOException {
+ final PointValues inPointValues = in.getPointValues(fieldName);
if (inPointValues == null) {
return null;
} else {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/search/PointInSetQuery.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/search/PointInSetQuery.java b/lucene/core/src/java/org/apache/lucene/search/PointInSetQuery.java
index d41b91c..f37d21d 100644
--- a/lucene/core/src/java/org/apache/lucene/search/PointInSetQuery.java
+++ b/lucene/core/src/java/org/apache/lucene/search/PointInSetQuery.java
@@ -23,7 +23,6 @@ import java.util.Collection;
import java.util.Iterator;
import java.util.NoSuchElementException;
import org.apache.lucene.document.IntPoint;
-import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.LeafReader;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.PointValues.IntersectVisitor;
@@ -116,21 +115,18 @@ public abstract class PointInSetQuery extends Query {
@Override
public Scorer scorer(LeafReaderContext context) throws IOException {
LeafReader reader = context.reader();
- PointValues values = reader.getPointValues();
+
+ PointValues values = reader.getPointValues(field);
if (values == null) {
- // No docs in this segment indexed any points
- return null;
- }
- FieldInfo fieldInfo = reader.getFieldInfos().fieldInfo(field);
- if (fieldInfo == null) {
- // No docs in this segment indexed this field at all
+ // No docs in this segment/field indexed any points
return null;
}
- if (fieldInfo.getPointDimensionCount() != numDims) {
- throw new IllegalArgumentException("field=\"" + field + "\" was indexed with numDims=" + fieldInfo.getPointDimensionCount() + " but this query has numDims=" + numDims);
+
+ if (values.getNumDimensions() != numDims) {
+ throw new IllegalArgumentException("field=\"" + field + "\" was indexed with numDims=" + values.getNumDimensions() + " but this query has numDims=" + numDims);
}
- if (fieldInfo.getPointNumBytes() != bytesPerDim) {
- throw new IllegalArgumentException("field=\"" + field + "\" was indexed with bytesPerDim=" + fieldInfo.getPointNumBytes() + " but this query has bytesPerDim=" + bytesPerDim);
+ if (values.getBytesPerDimension() != bytesPerDim) {
+ throw new IllegalArgumentException("field=\"" + field + "\" was indexed with bytesPerDim=" + values.getBytesPerDimension() + " but this query has bytesPerDim=" + bytesPerDim);
}
DocIdSetBuilder result = new DocIdSetBuilder(reader.maxDoc(), values, field);
@@ -138,7 +134,7 @@ public abstract class PointInSetQuery extends Query {
if (numDims == 1) {
// We optimize this common case, effectively doing a merge sort of the indexed values vs the queried set:
- values.intersect(field, new MergePointVisitor(sortedPackedPoints, result));
+ values.intersect(new MergePointVisitor(sortedPackedPoints, result));
} else {
// NOTE: this is naive implementation, where for each point we re-walk the KD tree to intersect. We could instead do a similar
@@ -148,7 +144,7 @@ public abstract class PointInSetQuery extends Query {
TermIterator iterator = sortedPackedPoints.iterator();
for (BytesRef point = iterator.next(); point != null; point = iterator.next()) {
visitor.setPoint(point);
- values.intersect(field, visitor);
+ values.intersect(visitor);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java b/lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java
index 1cd2d35..5fd0167 100644
--- a/lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java
+++ b/lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java
@@ -24,7 +24,6 @@ import org.apache.lucene.index.PointValues;
import org.apache.lucene.index.PointValues.IntersectVisitor;
import org.apache.lucene.index.PointValues.Relation;
import org.apache.lucene.document.IntPoint; // javadocs
-import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.LeafReader;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.util.DocIdSetBuilder;
@@ -108,7 +107,7 @@ public abstract class PointRangeQuery extends Query {
private DocIdSet buildMatchingDocIdSet(LeafReader reader, PointValues values) throws IOException {
DocIdSetBuilder result = new DocIdSetBuilder(reader.maxDoc(), values, field);
- values.intersect(field,
+ values.intersect(
new IntersectVisitor() {
DocIdSetBuilder.BulkAdder adder;
@@ -171,27 +170,24 @@ public abstract class PointRangeQuery extends Query {
@Override
public Scorer scorer(LeafReaderContext context) throws IOException {
LeafReader reader = context.reader();
- PointValues values = reader.getPointValues();
+
+ PointValues values = reader.getPointValues(field);
if (values == null) {
- // No docs in this segment indexed any points
- return null;
- }
- FieldInfo fieldInfo = reader.getFieldInfos().fieldInfo(field);
- if (fieldInfo == null) {
- // No docs in this segment indexed this field at all
+ // No docs in this segment/field indexed any points
return null;
}
- if (fieldInfo.getPointDimensionCount() != numDims) {
- throw new IllegalArgumentException("field=\"" + field + "\" was indexed with numDims=" + fieldInfo.getPointDimensionCount() + " but this query has numDims=" + numDims);
+
+ if (values.getNumDimensions() != numDims) {
+ throw new IllegalArgumentException("field=\"" + field + "\" was indexed with numDims=" + values.getNumDimensions() + " but this query has numDims=" + numDims);
}
- if (bytesPerDim != fieldInfo.getPointNumBytes()) {
- throw new IllegalArgumentException("field=\"" + field + "\" was indexed with bytesPerDim=" + fieldInfo.getPointNumBytes() + " but this query has bytesPerDim=" + bytesPerDim);
+ if (bytesPerDim != values.getBytesPerDimension()) {
+ throw new IllegalArgumentException("field=\"" + field + "\" was indexed with bytesPerDim=" + values.getBytesPerDimension() + " but this query has bytesPerDim=" + bytesPerDim);
}
boolean allDocsMatch;
- if (values.getDocCount(field) == reader.maxDoc()) {
- final byte[] fieldPackedLower = values.getMinPackedValue(field);
- final byte[] fieldPackedUpper = values.getMaxPackedValue(field);
+ if (values.getDocCount() == reader.maxDoc()) {
+ final byte[] fieldPackedLower = values.getMinPackedValue();
+ final byte[] fieldPackedUpper = values.getMaxPackedValue();
allDocsMatch = true;
for (int i = 0; i < numDims; ++i) {
int offset = i * bytesPerDim;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/util/DocIdSetBuilder.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/DocIdSetBuilder.java b/lucene/core/src/java/org/apache/lucene/util/DocIdSetBuilder.java
index e56e211..954614b 100644
--- a/lucene/core/src/java/org/apache/lucene/util/DocIdSetBuilder.java
+++ b/lucene/core/src/java/org/apache/lucene/util/DocIdSetBuilder.java
@@ -116,7 +116,7 @@ public final class DocIdSetBuilder {
/** Create a {@link DocIdSetBuilder} instance that is optimized for
* accumulating docs that match the given {@link PointValues}. */
public DocIdSetBuilder(int maxDoc, PointValues values, String field) throws IOException {
- this(maxDoc, values.getDocCount(field), values.size(field));
+ this(maxDoc, values.getDocCount(), values.size());
}
DocIdSetBuilder(int maxDoc, int docCount, long valueCount) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java b/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java
index 9ca0bb4..0fe549a 100644
--- a/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java
+++ b/lucene/core/src/java/org/apache/lucene/util/bkd/BKDReader.java
@@ -21,8 +21,7 @@ import java.util.Arrays;
import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.index.CorruptIndexException;
-import org.apache.lucene.index.PointValues.IntersectVisitor;
-import org.apache.lucene.index.PointValues.Relation;
+import org.apache.lucene.index.PointValues;
import org.apache.lucene.store.IndexInput;
import org.apache.lucene.util.Accountable;
import org.apache.lucene.util.BytesRef;
@@ -32,7 +31,7 @@ import org.apache.lucene.util.StringHelper;
*
* @lucene.experimental */
-public class BKDReader implements Accountable {
+public class BKDReader extends PointValues implements Accountable {
// Packed array of byte[] holding all split values in the full binary tree:
final private byte[] splitPackedValues;
final long[] leafBlockFPs;
@@ -496,26 +495,32 @@ public class BKDReader implements Accountable {
leafBlockFPs.length * Long.BYTES;
}
+ @Override
public byte[] getMinPackedValue() {
return minPackedValue.clone();
}
+ @Override
public byte[] getMaxPackedValue() {
return maxPackedValue.clone();
}
+ @Override
public int getNumDimensions() {
return numDims;
}
+ @Override
public int getBytesPerDimension() {
return bytesPerDim;
}
- public long getPointCount() {
+ @Override
+ public long size() {
return pointCount;
}
+ @Override
public int getDocCount() {
return docCount;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java b/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
index 88a84e9..6ee178b 100644
--- a/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/util/bkd/BKDWriter.java
@@ -25,7 +25,7 @@ import java.util.List;
import java.util.function.IntFunction;
import org.apache.lucene.codecs.CodecUtil;
-import org.apache.lucene.codecs.MutablePointsReader;
+import org.apache.lucene.codecs.MutablePointValues;
import org.apache.lucene.index.MergeState;
import org.apache.lucene.index.PointValues.IntersectVisitor;
import org.apache.lucene.index.PointValues.Relation;
@@ -417,12 +417,12 @@ public class BKDWriter implements Closeable {
}
}
- /** Write a field from a {@link MutablePointsReader}. This way of writing
+ /** Write a field from a {@link MutablePointValues}. This way of writing
* points is faster than regular writes with {@link BKDWriter#add} since
* there is opportunity for reordering points before writing them to
* disk. This method does not use transient disk in order to reorder points.
*/
- public long writeField(IndexOutput out, String fieldName, MutablePointsReader reader) throws IOException {
+ public long writeField(IndexOutput out, String fieldName, MutablePointValues reader) throws IOException {
if (numDims == 1) {
return writeField1Dim(out, fieldName, reader);
} else {
@@ -433,7 +433,7 @@ public class BKDWriter implements Closeable {
/* In the 2+D case, we recursively pick the split dimension, compute the
* median value and partition other values around it. */
- private long writeFieldNDims(IndexOutput out, String fieldName, MutablePointsReader reader) throws IOException {
+ private long writeFieldNDims(IndexOutput out, String fieldName, MutablePointValues values) throws IOException {
if (pointCount != 0) {
throw new IllegalStateException("cannot mix add and writeField");
}
@@ -446,7 +446,7 @@ public class BKDWriter implements Closeable {
// Mark that we already finished:
heapPointWriter = null;
- long countPerLeaf = pointCount = reader.size(fieldName);
+ long countPerLeaf = pointCount = values.size();
long innerNodeCount = 1;
while (countPerLeaf > maxPointsInLeafNode) {
@@ -465,7 +465,7 @@ public class BKDWriter implements Closeable {
Arrays.fill(minPackedValue, (byte) 0xff);
Arrays.fill(maxPackedValue, (byte) 0);
for (int i = 0; i < Math.toIntExact(pointCount); ++i) {
- reader.getValue(i, scratchBytesRef1);
+ values.getValue(i, scratchBytesRef1);
for(int dim=0;dim<numDims;dim++) {
int offset = dim*bytesPerDim;
if (StringHelper.compare(bytesPerDim, scratchBytesRef1.bytes, scratchBytesRef1.offset + offset, minPackedValue, offset) < 0) {
@@ -476,10 +476,10 @@ public class BKDWriter implements Closeable {
}
}
- docsSeen.set(reader.getDocID(i));
+ docsSeen.set(values.getDocID(i));
}
- build(1, numLeaves, reader, 0, Math.toIntExact(pointCount), out,
+ build(1, numLeaves, values, 0, Math.toIntExact(pointCount), out,
minPackedValue, maxPackedValue, splitPackedValues, leafBlockFPs,
new int[maxPointsInLeafNode]);
@@ -491,12 +491,12 @@ public class BKDWriter implements Closeable {
/* In the 1D case, we can simply sort points in ascending order and use the
* same writing logic as we use at merge time. */
- private long writeField1Dim(IndexOutput out, String fieldName, MutablePointsReader reader) throws IOException {
- MutablePointsReaderUtils.sort(maxDoc, packedBytesLength, reader, 0, Math.toIntExact(reader.size(fieldName)));
+ private long writeField1Dim(IndexOutput out, String fieldName, MutablePointValues reader) throws IOException {
+ MutablePointsReaderUtils.sort(maxDoc, packedBytesLength, reader, 0, Math.toIntExact(reader.size()));
final OneDimensionBKDWriter oneDimWriter = new OneDimensionBKDWriter(out);
- reader.intersect(fieldName, new IntersectVisitor() {
+ reader.intersect(new IntersectVisitor() {
@Override
public void visit(int docID, byte[] packedValue) throws IOException {
@@ -1238,7 +1238,7 @@ public class BKDWriter implements Closeable {
/* Recursively reorders the provided reader and writes the bkd-tree on the fly. */
private void build(int nodeID, int leafNodeOffset,
- MutablePointsReader reader, int from, int to,
+ MutablePointValues reader, int from, int to,
IndexOutput out,
byte[] minPackedValue, byte[] maxPackedValue,
byte[] splitPackedValues,
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointsReaderUtils.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointsReaderUtils.java b/lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointsReaderUtils.java
index b439208..132ad3c 100644
--- a/lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointsReaderUtils.java
+++ b/lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointsReaderUtils.java
@@ -16,7 +16,7 @@
*/
package org.apache.lucene.util.bkd;
-import org.apache.lucene.codecs.MutablePointsReader;
+import org.apache.lucene.codecs.MutablePointValues;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.IntroSelector;
import org.apache.lucene.util.IntroSorter;
@@ -30,9 +30,9 @@ final class MutablePointsReaderUtils {
MutablePointsReaderUtils() {}
- /** Sort the given {@link MutablePointsReader} based on its packed value then doc ID. */
+ /** Sort the given {@link MutablePointValues} based on its packed value then doc ID. */
static void sort(int maxDoc, int packedBytesLength,
- MutablePointsReader reader, int from, int to) {
+ MutablePointValues reader, int from, int to) {
final int bitsPerDocId = PackedInts.bitsRequired(maxDoc - 1);
new MSBRadixSorter(packedBytesLength + (bitsPerDocId + 7) / 8) {
@@ -89,7 +89,7 @@ final class MutablePointsReaderUtils {
/** Sort points on the given dimension. */
static void sortByDim(int sortedDim, int bytesPerDim, int[] commonPrefixLengths,
- MutablePointsReader reader, int from, int to,
+ MutablePointValues reader, int from, int to,
BytesRef scratch1, BytesRef scratch2) {
// No need for a fancy radix sort here, this is called on the leaves only so
@@ -128,7 +128,7 @@ final class MutablePointsReaderUtils {
* than or equal to it and all values on the right must be greater than or
* equal to it. */
static void partition(int maxDoc, int splitDim, int bytesPerDim, int commonPrefixLen,
- MutablePointsReader reader, int from, int to, int mid,
+ MutablePointValues reader, int from, int to, int mid,
BytesRef scratch1, BytesRef scratch2) {
final int offset = splitDim * bytesPerDim + commonPrefixLen;
final int cmpBytes = bytesPerDim - commonPrefixLen;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/test/org/apache/lucene/index/Test2BPoints.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/index/Test2BPoints.java b/lucene/core/src/test/org/apache/lucene/index/Test2BPoints.java
index aeef023..2e32773 100644
--- a/lucene/core/src/test/org/apache/lucene/index/Test2BPoints.java
+++ b/lucene/core/src/test/org/apache/lucene/index/Test2BPoints.java
@@ -78,7 +78,7 @@ public class Test2BPoints extends LuceneTestCase {
DirectoryReader r = DirectoryReader.open(w);
IndexSearcher s = new IndexSearcher(r);
assertEquals(numDocs, s.count(LongPoint.newRangeQuery("long", Long.MIN_VALUE, Long.MAX_VALUE)));
- assertTrue(r.leaves().get(0).reader().getPointValues().size("long") > Integer.MAX_VALUE);
+ assertTrue(r.leaves().get(0).reader().getPointValues("long").size() > Integer.MAX_VALUE);
r.close();
w.close();
System.out.println("TEST: now CheckIndex");
@@ -126,7 +126,7 @@ public class Test2BPoints extends LuceneTestCase {
DirectoryReader r = DirectoryReader.open(w);
IndexSearcher s = new IndexSearcher(r);
assertEquals(numDocs, s.count(LongPoint.newRangeQuery("long", new long[] {Long.MIN_VALUE, Long.MIN_VALUE}, new long[] {Long.MAX_VALUE, Long.MAX_VALUE})));
- assertTrue(r.leaves().get(0).reader().getPointValues().size("long") > Integer.MAX_VALUE);
+ assertTrue(r.leaves().get(0).reader().getPointValues("long").size() > Integer.MAX_VALUE);
r.close();
w.close();
System.out.println("TEST: now CheckIndex");
[08/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7489: Better
sparsity support for Lucene70DocValuesFormat.
Posted by kr...@apache.org.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/index/EmptyDocValuesProducer.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/EmptyDocValuesProducer.java b/lucene/core/src/java/org/apache/lucene/index/EmptyDocValuesProducer.java
index a4b9049..4e9f0e0 100644
--- a/lucene/core/src/java/org/apache/lucene/index/EmptyDocValuesProducer.java
+++ b/lucene/core/src/java/org/apache/lucene/index/EmptyDocValuesProducer.java
@@ -44,12 +44,12 @@ public abstract class EmptyDocValuesProducer extends DocValuesProducer {
}
@Override
- public SortedNumericDocValues getSortedNumeric(FieldInfo field) {
+ public SortedNumericDocValues getSortedNumeric(FieldInfo field) throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public SortedSetDocValues getSortedSet(FieldInfo field) {
+ public SortedSetDocValues getSortedSet(FieldInfo field) throws IOException {
throw new UnsupportedOperationException();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/index/LegacySortedSetDocValues.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/LegacySortedSetDocValues.java b/lucene/core/src/java/org/apache/lucene/index/LegacySortedSetDocValues.java
index dae1179..0c6c809 100644
--- a/lucene/core/src/java/org/apache/lucene/index/LegacySortedSetDocValues.java
+++ b/lucene/core/src/java/org/apache/lucene/index/LegacySortedSetDocValues.java
@@ -17,6 +17,8 @@
package org.apache.lucene.index;
+import java.io.IOException;
+
import org.apache.lucene.util.BytesRef;
/**
@@ -103,7 +105,7 @@ public abstract class LegacySortedSetDocValues {
* Returns a {@link TermsEnum} over the values.
* The enum supports {@link TermsEnum#ord()} and {@link TermsEnum#seekExact(long)}.
*/
- public TermsEnum termsEnum() {
+ public TermsEnum termsEnum() throws IOException {
throw new UnsupportedOperationException();
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/index/SingletonSortedSetDocValues.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/SingletonSortedSetDocValues.java b/lucene/core/src/java/org/apache/lucene/index/SingletonSortedSetDocValues.java
index 225b6a6..cc7360e 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SingletonSortedSetDocValues.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SingletonSortedSetDocValues.java
@@ -95,7 +95,7 @@ final class SingletonSortedSetDocValues extends SortedSetDocValues {
}
@Override
- public TermsEnum termsEnum() {
+ public TermsEnum termsEnum() throws IOException {
return in.termsEnum();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/index/SortedDocValues.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/SortedDocValues.java b/lucene/core/src/java/org/apache/lucene/index/SortedDocValues.java
index ee70a64..7ff084f 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SortedDocValues.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SortedDocValues.java
@@ -104,7 +104,7 @@ public abstract class SortedDocValues extends BinaryDocValues {
* Returns a {@link TermsEnum} over the values.
* The enum supports {@link TermsEnum#ord()} and {@link TermsEnum#seekExact(long)}.
*/
- public TermsEnum termsEnum() {
+ public TermsEnum termsEnum() throws IOException {
return new SortedDocValuesTermsEnum(this);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/index/SortedSetDocValues.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/index/SortedSetDocValues.java b/lucene/core/src/java/org/apache/lucene/index/SortedSetDocValues.java
index e53a0e7..439843b 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SortedSetDocValues.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SortedSetDocValues.java
@@ -98,7 +98,7 @@ public abstract class SortedSetDocValues extends DocIdSetIterator {
* Returns a {@link TermsEnum} over the values.
* The enum supports {@link TermsEnum#ord()} and {@link TermsEnum#seekExact(long)}.
*/
- public TermsEnum termsEnum() {
+ public TermsEnum termsEnum() throws IOException {
return new SortedSetDocValuesTermsEnum(this);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/search/DocValuesRewriteMethod.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/search/DocValuesRewriteMethod.java b/lucene/core/src/java/org/apache/lucene/search/DocValuesRewriteMethod.java
index 0bf7a8e..46afe0d 100644
--- a/lucene/core/src/java/org/apache/lucene/search/DocValuesRewriteMethod.java
+++ b/lucene/core/src/java/org/apache/lucene/search/DocValuesRewriteMethod.java
@@ -81,7 +81,7 @@ public final class DocValuesRewriteMethod extends MultiTermQuery.RewriteMethod {
TermsEnum termsEnum = query.getTermsEnum(new Terms() {
@Override
- public TermsEnum iterator() {
+ public TermsEnum iterator() throws IOException {
return fcsi.termsEnum();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/util/packed/DirectMonotonicReader.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/packed/DirectMonotonicReader.java b/lucene/core/src/java/org/apache/lucene/util/packed/DirectMonotonicReader.java
index bdefdf3..676efcd 100644
--- a/lucene/core/src/java/org/apache/lucene/util/packed/DirectMonotonicReader.java
+++ b/lucene/core/src/java/org/apache/lucene/util/packed/DirectMonotonicReader.java
@@ -46,7 +46,6 @@ public final class DirectMonotonicReader {
public static class Meta implements Accountable {
private static final long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(Meta.class);
- final long numValues;
final int blockShift;
final int numBlocks;
final long[] mins;
@@ -55,7 +54,6 @@ public final class DirectMonotonicReader {
final long[] offsets;
Meta(long numValues, int blockShift) {
- this.numValues = numValues;
this.blockShift = blockShift;
long numBlocks = numValues >>> blockShift;
if ((numBlocks << blockShift) < numValues) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/test/org/apache/lucene/codecs/lucene70/TestLucene70DocValuesFormat.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/codecs/lucene70/TestLucene70DocValuesFormat.java b/lucene/core/src/test/org/apache/lucene/codecs/lucene70/TestLucene70DocValuesFormat.java
index fae82e0..5ad701e 100644
--- a/lucene/core/src/test/org/apache/lucene/codecs/lucene70/TestLucene70DocValuesFormat.java
+++ b/lucene/core/src/test/org/apache/lucene/codecs/lucene70/TestLucene70DocValuesFormat.java
@@ -25,14 +25,13 @@ import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.TreeSet;
+import java.util.function.Supplier;
import org.apache.lucene.analysis.MockAnalyzer;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.PostingsFormat;
import org.apache.lucene.codecs.asserting.AssertingCodec;
-import org.apache.lucene.codecs.lucene70.Lucene70DocValuesProducer.SparseNumericDocValues;
-import org.apache.lucene.codecs.lucene70.Lucene70DocValuesProducer.SparseNumericDocValuesRandomAccessWrapper;
import org.apache.lucene.codecs.lucene70.Lucene70DocValuesFormat;
import org.apache.lucene.document.BinaryDocValuesField;
import org.apache.lucene.document.Document;
@@ -62,7 +61,6 @@ import org.apache.lucene.index.SortedSetDocValues;
import org.apache.lucene.index.Term;
import org.apache.lucene.index.Terms;
import org.apache.lucene.index.TermsEnum.SeekStatus;
-import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.index.TermsEnum;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMFile;
@@ -70,7 +68,6 @@ import org.apache.lucene.store.RAMInputStream;
import org.apache.lucene.store.RAMOutputStream;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.BytesRefBuilder;
-import org.apache.lucene.util.LongValues;
import org.apache.lucene.util.TestUtil;
/**
@@ -123,7 +120,7 @@ public class TestLucene70DocValuesFormat extends BaseCompressingDocValuesFormatT
public void testTermsEnumFixedWidth() throws Exception {
int numIterations = atLeast(1);
for (int i = 0; i < numIterations; i++) {
- doTestTermsEnumRandom(TestUtil.nextInt(random(), 1025, 5121), 10, 10);
+ doTestTermsEnumRandom(TestUtil.nextInt(random(), 1025, 5121), () -> TestUtil.randomSimpleString(random(), 10, 10));
}
}
@@ -131,7 +128,7 @@ public class TestLucene70DocValuesFormat extends BaseCompressingDocValuesFormatT
public void testTermsEnumVariableWidth() throws Exception {
int numIterations = atLeast(1);
for (int i = 0; i < numIterations; i++) {
- doTestTermsEnumRandom(TestUtil.nextInt(random(), 1025, 5121), 1, 500);
+ doTestTermsEnumRandom(TestUtil.nextInt(random(), 1025, 5121), () -> TestUtil.randomSimpleString(random(), 1, 500));
}
}
@@ -139,7 +136,21 @@ public class TestLucene70DocValuesFormat extends BaseCompressingDocValuesFormatT
public void testTermsEnumRandomMany() throws Exception {
int numIterations = atLeast(1);
for (int i = 0; i < numIterations; i++) {
- doTestTermsEnumRandom(TestUtil.nextInt(random(), 1025, 8121), 1, 500);
+ doTestTermsEnumRandom(TestUtil.nextInt(random(), 1025, 8121), () -> TestUtil.randomSimpleString(random(), 1, 500));
+ }
+ }
+
+ public void testTermsEnumLongSharedPrefixes() throws Exception {
+ int numIterations = atLeast(1);
+ for (int i = 0; i < numIterations; i++) {
+ doTestTermsEnumRandom(TestUtil.nextInt(random(), 1025, 5121), () -> {
+ char[] chars = new char[random().nextInt(500)];
+ Arrays.fill(chars, 'a');
+ if (chars.length > 0) {
+ chars[random().nextInt(chars.length)] = 'b';
+ }
+ return new String(chars);
+ });
}
}
@@ -269,7 +280,7 @@ public class TestLucene70DocValuesFormat extends BaseCompressingDocValuesFormatT
// TODO: try to refactor this and some termsenum tests into the base class.
// to do this we need to fix the test class to get a DVF not a Codec so we can setup
// the postings format correctly.
- private void doTestTermsEnumRandom(int numDocs, int minLength, int maxLength) throws Exception {
+ private void doTestTermsEnumRandom(int numDocs, Supplier<String> valuesProducer) throws Exception {
Directory dir = newFSDirectory(createTempDir());
IndexWriterConfig conf = newIndexWriterConfig(new MockAnalyzer(random()));
conf.setMergeScheduler(new SerialMergeScheduler());
@@ -294,12 +305,11 @@ public class TestLucene70DocValuesFormat extends BaseCompressingDocValuesFormatT
Document doc = new Document();
Field idField = new StringField("id", Integer.toString(i), Field.Store.NO);
doc.add(idField);
- final int length = TestUtil.nextInt(random(), minLength, maxLength);
int numValues = random().nextInt(17);
// create a random list of strings
List<String> values = new ArrayList<>();
for (int v = 0; v < numValues; v++) {
- values.add(TestUtil.randomSimpleString(random(), minLength, length));
+ values.add(valuesProducer.get());
}
// add in any order to the indexed field
@@ -429,92 +439,6 @@ public class TestLucene70DocValuesFormat extends BaseCompressingDocValuesFormatT
}
}
- public void testSparseLongValues() throws IOException {
- final int iters = atLeast(5);
- for (int iter = 0; iter < iters; ++iter) {
- final int numDocs = TestUtil.nextInt(random(), 0, 100);
- final int[] docIds = new int[numDocs];
- final long[] values = new long[numDocs];
- final int maxDoc;
- if (numDocs == 0) {
- maxDoc = 1 + random().nextInt(10);
- } else {
- docIds[0] = random().nextInt(10);
- for (int i = 1; i < docIds.length; ++i) {
- docIds[i] = docIds[i - 1] + 1 + random().nextInt(100);
- }
- maxDoc = docIds[numDocs - 1] + 1 + random().nextInt(10);
- }
- for (int i = 0; i < values.length; ++i) {
- values[i] = random().nextLong();
- }
- final long missingValue = random().nextLong();
- final LongValues docIdsValues = new LongValues() {
- @Override
- public long get(long index) {
- return docIds[Math.toIntExact(index)];
- }
- };
- final LongValues valuesValues = new LongValues() {
- @Override
- public long get(long index) {
- return values[Math.toIntExact(index)];
- }
- };
- final SparseNumericDocValues sparseValues = new SparseNumericDocValues(numDocs, docIdsValues, valuesValues);
-
- // sequential access
- assertEquals(-1, sparseValues.docID());
- for (int i = 0; i < docIds.length; ++i) {
- assertEquals(docIds[i], sparseValues.nextDoc());
- }
- assertEquals(DocIdSetIterator.NO_MORE_DOCS, sparseValues.nextDoc());
-
- // advance
- for (int i = 0; i < 2000; ++i) {
- final int target = TestUtil.nextInt(random(), 0, maxDoc);
- int index = Arrays.binarySearch(docIds, target);
- if (index < 0) {
- index = -1 - index;
- }
- sparseValues.reset();
- if (index > 0) {
- assertEquals(docIds[index - 1], sparseValues.advance(Math.toIntExact(docIds[index - 1])));
- }
- if (index == docIds.length) {
- assertEquals(DocIdSetIterator.NO_MORE_DOCS, sparseValues.advance(target));
- } else {
- assertEquals(docIds[index], sparseValues.advance(target));
- }
- }
-
- final SparseNumericDocValuesRandomAccessWrapper raWrapper = new SparseNumericDocValuesRandomAccessWrapper(sparseValues, missingValue);
-
- // random-access
- for (int i = 0; i < 2000; ++i) {
- final int docId = TestUtil.nextInt(random(), 0, maxDoc - 1);
- final int idx = Arrays.binarySearch(docIds, docId);
- final long value = raWrapper.get(docId);
- if (idx >= 0) {
- assertEquals(values[idx], value);
- } else {
- assertEquals(missingValue, value);
- }
- }
-
- // sequential access
- for (int docId = 0; docId < maxDoc; docId += random().nextInt(3)) {
- final int idx = Arrays.binarySearch(docIds, docId);
- final long value = raWrapper.get(docId);
- if (idx >= 0) {
- assertEquals(values[idx], value);
- } else {
- assertEquals(missingValue, value);
- }
- }
- }
- }
-
@Slow
public void testSortedSetAroundBlockSize() throws IOException {
final int frontier = 1 << Lucene70DocValuesFormat.DIRECT_MONOTONIC_BLOCK_SHIFT;
[39/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9657: Addressing
test failures
Posted by kr...@apache.org.
SOLR-9657: Addressing test failures
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6893595f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/6893595f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/6893595f
Branch: refs/heads/jira/solr-8593
Commit: 6893595f340b748a9c5e5c3884704e71275deab4
Parents: c2e031a
Author: Noble Paul <no...@gmail.com>
Authored: Thu Oct 20 01:26:02 2016 +0530
Committer: Noble Paul <no...@gmail.com>
Committed: Thu Oct 20 01:26:02 2016 +0530
----------------------------------------------------------------------
.../solr/update/processor/SimpleUpdateProcessorFactory.java | 5 +++--
.../solr/update/processor/TemplateUpdateProcessorFactory.java | 3 ++-
2 files changed, 5 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6893595f/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java b/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java
index aec9d87..b1edea0 100644
--- a/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java
+++ b/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java
@@ -29,13 +29,14 @@ import org.apache.solr.update.AddUpdateCommand;
* This is deliberately made to support only the add operation
*/
public abstract class SimpleUpdateProcessorFactory extends UpdateRequestProcessorFactory {
- protected final String myName;
+ protected final String myName; // if classname==XyzUpdateProcessorFactory myName=Xyz
protected NamedList initArgs = new NamedList();
private static ThreadLocal<SolrQueryRequest> REQ = new ThreadLocal<>();
protected SimpleUpdateProcessorFactory() {
String simpleName = this.getClass().getSimpleName();
- this.myName = simpleName.substring(0, simpleName.indexOf("UpdateProcessorFactory"));
+ int idx = simpleName.indexOf("UpdateProcessorFactory");
+ this.myName = idx == -1 ? simpleName : simpleName.substring(0, idx);
}
@Override
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6893595f/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java b/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java
index 41d109b..dbe61b4 100644
--- a/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java
+++ b/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java
@@ -29,7 +29,8 @@ import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.update.AddUpdateCommand;
-
+//Adds new fields to documents based on a template pattern specified via Template.field
+// request parameters (multi-valued) or 'field' value specified in initArgs
public class TemplateUpdateProcessorFactory extends SimpleUpdateProcessorFactory {
@Override
protected void process(AddUpdateCommand cmd, SolrQueryRequest req, SolrQueryResponse rsp) {
[32/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9506: reverting the
previous commit
Posted by kr...@apache.org.
SOLR-9506: reverting the previous commit
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ffa5c4ba
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ffa5c4ba
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ffa5c4ba
Branch: refs/heads/jira/solr-8593
Commit: ffa5c4ba2c2d6fa6bb943a70196aad0058333fa2
Parents: 9aa764a
Author: Noble Paul <no...@gmail.com>
Authored: Wed Oct 19 00:50:30 2016 +0530
Committer: Noble Paul <no...@gmail.com>
Committed: Wed Oct 19 00:50:30 2016 +0530
----------------------------------------------------------------------
solr/CHANGES.txt | 2 -
.../src/java/org/apache/solr/core/SolrCore.java | 38 ----------
.../apache/solr/search/SolrIndexSearcher.java | 60 ++-------------
.../apache/solr/update/IndexFingerprint.java | 79 ++++++++------------
.../cloud/LeaderFailureAfterFreshStartTest.java | 32 ++++++++
.../solr/cloud/PeerSyncReplicationTest.java | 2 +-
.../org/apache/solr/update/PeerSyncTest.java | 20 +----
.../solr/cloud/AbstractDistribZkTestBase.java | 31 --------
8 files changed, 72 insertions(+), 192 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ffa5c4ba/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 4d75f88..be958d9 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -206,8 +206,6 @@ Optimizations
* SOLR-9566: Don't put replicas into recovery when first creating a Collection
(Alan Woodward)
-* SOLR-9506: cache IndexFingerprint for each segment (Pushkar Raste, noble)
-
Other Changes
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ffa5c4ba/solr/core/src/java/org/apache/solr/core/SolrCore.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/SolrCore.java b/solr/core/src/java/org/apache/solr/core/SolrCore.java
index e47c8b6..2827f03 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrCore.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrCore.java
@@ -58,7 +58,6 @@ import org.apache.lucene.codecs.Codec;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexDeletionPolicy;
import org.apache.lucene.index.IndexWriter;
-import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.IOContext;
@@ -128,7 +127,6 @@ import org.apache.solr.search.stats.LocalStatsCache;
import org.apache.solr.search.stats.StatsCache;
import org.apache.solr.update.DefaultSolrCoreState;
import org.apache.solr.update.DirectUpdateHandler2;
-import org.apache.solr.update.IndexFingerprint;
import org.apache.solr.update.SolrCoreState;
import org.apache.solr.update.SolrCoreState.IndexWriterCloser;
import org.apache.solr.update.SolrIndexWriter;
@@ -151,8 +149,6 @@ import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.google.common.collect.MapMaker;
-
import static com.google.common.base.Preconditions.checkNotNull;
import static org.apache.solr.common.params.CommonParams.NAME;
import static org.apache.solr.common.params.CommonParams.PATH;
@@ -204,8 +200,6 @@ public final class SolrCore implements SolrInfoMBean, Closeable {
private final ReentrantLock snapshotDelLock; // A lock instance to guard against concurrent deletions.
public Date getStartTimeStamp() { return startTime; }
-
- private final Map<Object, IndexFingerprint> perSegmentFingerprintCache = new MapMaker().weakKeys().makeMap();
public long getStartNanoTime() {
return startNanoTime;
@@ -1595,38 +1589,6 @@ public final class SolrCore implements SolrInfoMBean, Closeable {
}
/**
- * Computes fingerprint of a segment and caches it only if all the version in segment are included in the fingerprint.
- * We can't use computeIfAbsent as caching is conditional (as described above)
- * There is chance that two threads may compute fingerprint on the same segment. It might be OK to do so rather than locking entire map.
- * @param searcher searcher that includes specified LeaderReaderContext
- * @param ctx LeafReaderContext of a segment to compute fingerprint of
- * @param maxVersion maximum version number to consider for fingerprint computation
- * @return IndexFingerprint of the segment
- * @throws IOException Can throw IOException
- */
- public IndexFingerprint getIndexFingerprint(SolrIndexSearcher searcher, LeafReaderContext ctx, long maxVersion)
- throws IOException {
- IndexFingerprint f = null;
- f = perSegmentFingerprintCache.get(ctx.reader().getCoreCacheKey()) ;
- // fingerprint is either not cached or
- // we want fingerprint only up to a version less than maxVersionEncountered in the segment
- if(f == null || (f.getMaxInHash() > maxVersion)) {
- log.debug("IndexFingerprint cache miss for searcher:{} reader:{} readerHash:{} maxVersion:{}", searcher, ctx.reader() , ctx.reader().hashCode(), maxVersion);
- f = IndexFingerprint.getFingerprint(searcher, ctx, maxVersion);
- // cache fingerprint for the segment only if all the versions in the segment are included in the fingerprint
- if(f.getMaxVersionEncountered() == f.getMaxInHash()) {
- log.info("Caching fingerprint for searcher:{} leafReaderContext:{} mavVersion:{}", searcher, ctx, maxVersion);
- perSegmentFingerprintCache.put(ctx.reader().getCoreCacheKey(), f);
- }
-
- } else {
- log.debug("IndexFingerprint cache hit for searcher:{} reader:{} readerHash:{} maxVersion:{}", searcher, ctx.reader(), ctx.reader().hashCode(), maxVersion);
- }
- log.debug("Cache Size: {}, Segments Size:{}", perSegmentFingerprintCache.size(), searcher.getTopReaderContext().leaves().size());
- return f;
- }
-
- /**
* Returns the current registered searcher with its reference count incremented, or null if none are registered.
*/
public RefCounted<SolrIndexSearcher> getRegisteredSearcher() {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ffa5c4ba/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java b/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
index 02069ee..933477b 100644
--- a/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
+++ b/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
@@ -42,50 +42,22 @@ import org.apache.lucene.document.Document;
import org.apache.lucene.document.DocumentStoredFieldVisitor;
import org.apache.lucene.document.LazyDocument;
import org.apache.lucene.index.*;
+import org.apache.lucene.index.NumericDocValues;
import org.apache.lucene.index.StoredFieldVisitor.Status;
-import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.*;
import org.apache.lucene.search.BooleanClause.Occur;
-import org.apache.lucene.search.BooleanQuery;
-import org.apache.lucene.search.CollectionStatistics;
-import org.apache.lucene.search.Collector;
-import org.apache.lucene.search.ConstantScoreQuery;
-import org.apache.lucene.search.DocIdSet;
-import org.apache.lucene.search.DocIdSetIterator;
-import org.apache.lucene.search.EarlyTerminatingSortingCollector;
-import org.apache.lucene.search.Explanation;
-import org.apache.lucene.search.FieldDoc;
-import org.apache.lucene.search.IndexSearcher;
-import org.apache.lucene.search.LeafCollector;
-import org.apache.lucene.search.MatchAllDocsQuery;
-import org.apache.lucene.search.MultiCollector;
-import org.apache.lucene.search.Query;
-import org.apache.lucene.search.ScoreDoc;
-import org.apache.lucene.search.Scorer;
-import org.apache.lucene.search.SimpleCollector;
-import org.apache.lucene.search.Sort;
-import org.apache.lucene.search.SortField;
-import org.apache.lucene.search.TermQuery;
-import org.apache.lucene.search.TermStatistics;
-import org.apache.lucene.search.TimeLimitingCollector;
-import org.apache.lucene.search.TopDocs;
-import org.apache.lucene.search.TopDocsCollector;
-import org.apache.lucene.search.TopFieldCollector;
-import org.apache.lucene.search.TopFieldDocs;
-import org.apache.lucene.search.TopScoreDocCollector;
-import org.apache.lucene.search.TotalHitCountCollector;
-import org.apache.lucene.search.Weight;
import org.apache.lucene.store.Directory;
import org.apache.lucene.util.Bits;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.FixedBitSet;
import org.apache.solr.common.SolrDocumentBase;
-import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.SolrException;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.SimpleOrderedMap;
-import org.apache.solr.core.DirectoryFactory;
import org.apache.solr.core.DirectoryFactory.DirContext;
+import org.apache.solr.core.DirectoryFactory;
import org.apache.solr.core.SolrConfig;
import org.apache.solr.core.SolrCore;
import org.apache.solr.core.SolrInfoMBean;
@@ -2441,11 +2413,12 @@ public class SolrIndexSearcher extends IndexSearcher implements Closeable, SolrI
* gets a cached version of the IndexFingerprint for this searcher
**/
public IndexFingerprint getIndexFingerprint(long maxVersion) throws IOException {
+ final SolrIndexSearcher searcher = this;
final AtomicReference<IOException> exception = new AtomicReference<>();
try {
return maxVersionFingerprintCache.computeIfAbsent(maxVersion, key -> {
try {
- return computeFromPerSegmentIndexFingerprint(maxVersion);
+ return IndexFingerprint.getFingerprint(searcher, key);
} catch (IOException e) {
exception.set(e);
return null;
@@ -2456,27 +2429,6 @@ public class SolrIndexSearcher extends IndexSearcher implements Closeable, SolrI
}
}
- private IndexFingerprint computeFromPerSegmentIndexFingerprint(long maxVersion) throws IOException {
- final SolrIndexSearcher searcher = this;
- final AtomicReference<IOException> exception = new AtomicReference<>();
- try {
- return searcher.getTopReaderContext().leaves().stream()
- .map(ctx -> {
- try {
- return searcher.getCore().getIndexFingerprint(searcher, ctx, maxVersion);
- } catch (IOException e) {
- exception.set(e);
- return null;
- }
- })
- .filter(java.util.Objects::nonNull)
- .reduce(new IndexFingerprint(maxVersion), IndexFingerprint::reduce);
- } finally {
- if (exception.get() != null) throw exception.get();
- }
- }
-
-
/////////////////////////////////////////////////////////////////////
// SolrInfoMBean stuff: Statistics and Module Info
/////////////////////////////////////////////////////////////////////
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ffa5c4ba/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java b/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java
index 31adf07..877ef03 100644
--- a/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java
+++ b/solr/core/src/java/org/apache/solr/update/IndexFingerprint.java
@@ -52,14 +52,6 @@ public class IndexFingerprint implements MapSerializable {
private long numDocs;
private long maxDoc;
- public IndexFingerprint() {
- // default constructor
- }
-
- public IndexFingerprint (long maxVersionSpecified) {
- this.maxVersionSpecified = maxVersionSpecified;
- }
-
public long getMaxVersionSpecified() {
return maxVersionSpecified;
}
@@ -90,63 +82,53 @@ public class IndexFingerprint implements MapSerializable {
/** Opens a new realtime searcher and returns it's (possibly cached) fingerprint */
public static IndexFingerprint getFingerprint(SolrCore core, long maxVersion) throws IOException {
- RTimer timer = new RTimer();
core.getUpdateHandler().getUpdateLog().openRealtimeSearcher();
RefCounted<SolrIndexSearcher> newestSearcher = core.getUpdateHandler().getUpdateLog().uhandler.core.getRealtimeSearcher();
try {
- IndexFingerprint f = newestSearcher.get().getIndexFingerprint(maxVersion);
- final double duration = timer.stop();
- log.debug("IndexFingerprint time : {} result:{}" ,duration, f);
- return f;
+ return newestSearcher.get().getIndexFingerprint(maxVersion);
} finally {
if (newestSearcher != null) {
newestSearcher.decref();
}
}
-
}
-
- public static IndexFingerprint getFingerprint(SolrIndexSearcher searcher, LeafReaderContext ctx, Long maxVersion)
- throws IOException {
+
+ /** Calculates an index fingerprint */
+ public static IndexFingerprint getFingerprint(SolrIndexSearcher searcher, long maxVersion) throws IOException {
+ RTimer timer = new RTimer();
+
SchemaField versionField = VersionInfo.getAndCheckVersionField(searcher.getSchema());
+
+ IndexFingerprint f = new IndexFingerprint();
+ f.maxVersionSpecified = maxVersion;
+ f.maxDoc = searcher.maxDoc();
+
+ // TODO: this could be parallelized, or even cached per-segment if performance becomes an issue
ValueSource vs = versionField.getType().getValueSource(versionField, null);
Map funcContext = ValueSource.newContext(searcher);
vs.createWeight(funcContext, searcher);
-
- IndexFingerprint f = new IndexFingerprint();
- f.maxVersionSpecified = maxVersion;
- f.maxDoc = ctx.reader().maxDoc();
- f.numDocs = ctx.reader().numDocs();
-
- int maxDoc = ctx.reader().maxDoc();
- Bits liveDocs = ctx.reader().getLiveDocs();
- FunctionValues fv = vs.getValues(funcContext, ctx);
- for (int doc = 0; doc < maxDoc; doc++) {
- if (liveDocs != null && !liveDocs.get(doc)) continue;
- long v = fv.longVal(doc);
- f.maxVersionEncountered = Math.max(v, f.maxVersionEncountered);
- if (v <= f.maxVersionSpecified) {
- f.maxInHash = Math.max(v, f.maxInHash);
- f.versionsHash += Hash.fmix64(v);
- f.numVersions++;
+ for (LeafReaderContext ctx : searcher.getTopReaderContext().leaves()) {
+ int maxDoc = ctx.reader().maxDoc();
+ f.numDocs += ctx.reader().numDocs();
+ Bits liveDocs = ctx.reader().getLiveDocs();
+ FunctionValues fv = vs.getValues(funcContext, ctx);
+ for (int doc = 0; doc < maxDoc; doc++) {
+ if (liveDocs != null && !liveDocs.get(doc)) continue;
+ long v = fv.longVal(doc);
+ f.maxVersionEncountered = Math.max(v, f.maxVersionEncountered);
+ if (v <= f.maxVersionSpecified) {
+ f.maxInHash = Math.max(v, f.maxInHash);
+ f.versionsHash += Hash.fmix64(v);
+ f.numVersions++;
+ }
}
}
-
+
+ final double duration = timer.stop();
+ log.info("IndexFingerprint millis:" + duration + " result:" + f);
+
return f;
}
-
-
- public static IndexFingerprint reduce(IndexFingerprint acc, IndexFingerprint f2) {
- // acc should have maxVersionSpecified already set in it using IndexFingerprint(long maxVersionSpecified) constructor
- acc.maxDoc = Math.max(acc.maxDoc, f2.maxDoc);
- acc.numDocs += f2.numDocs;
- acc.maxVersionEncountered = Math.max(acc.maxVersionEncountered, f2.maxVersionEncountered);
- acc.maxInHash = Math.max(acc.maxInHash, f2.maxInHash);
- acc.versionsHash += f2.versionsHash;
- acc.numVersions += f2.numVersions;
-
- return acc;
- }
/** returns 0 for equal, negative if f1 is less recent than f2, positive if more recent */
public static int compare(IndexFingerprint f1, IndexFingerprint f2) {
@@ -218,5 +200,4 @@ public class IndexFingerprint implements MapSerializable {
public String toString() {
return toMap(new LinkedHashMap<>()).toString();
}
-
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ffa5c4ba/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java b/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
index ef21386..348532c 100644
--- a/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/LeaderFailureAfterFreshStartTest.java
@@ -29,6 +29,7 @@ import java.util.stream.Collectors;
import org.apache.commons.lang.RandomStringUtils;
import org.apache.lucene.util.LuceneTestCase.Slow;
import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.request.UpdateRequest;
import org.apache.solr.cloud.ZkTestServer.LimitViolationAction;
import org.apache.solr.common.SolrInputDocument;
@@ -36,8 +37,10 @@ import org.apache.solr.common.cloud.ClusterState;
import org.apache.solr.common.cloud.DocCollection;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.common.cloud.Slice.State;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.core.Diagnostics;
import org.apache.solr.handler.ReplicationHandler;
import org.junit.Test;
import org.slf4j.Logger;
@@ -194,6 +197,35 @@ public class LeaderFailureAfterFreshStartTest extends AbstractFullDistribZkTestB
}
+ static void waitForNewLeader(CloudSolrClient cloudClient, String shardName, Replica oldLeader, int maxWaitInSecs)
+ throws Exception {
+ log.info("Will wait for a node to become leader for {} secs", maxWaitInSecs);
+ boolean waitForLeader = true;
+ int i = 0;
+ ZkStateReader zkStateReader = cloudClient.getZkStateReader();
+ zkStateReader.forceUpdateCollection(DEFAULT_COLLECTION);
+
+ while(waitForLeader) {
+ ClusterState clusterState = zkStateReader.getClusterState();
+ DocCollection coll = clusterState.getCollection("collection1");
+ Slice slice = coll.getSlice(shardName);
+ if(slice.getLeader() != oldLeader && slice.getState() == State.ACTIVE) {
+ log.info("New leader got elected in {} secs", i);
+ break;
+ }
+
+ if(i == maxWaitInSecs) {
+ Diagnostics.logThreadDumps("Could not find new leader in specified timeout");
+ zkStateReader.getZkClient().printLayoutToStdOut();
+ fail("Could not find new leader even after waiting for " + maxWaitInSecs + "secs");
+ }
+
+ i++;
+ Thread.sleep(1000);
+ }
+ }
+
+
private void waitTillNodesActive() throws Exception {
for (int i = 0; i < 60; i++) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ffa5c4ba/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java b/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
index e00ea3c..3ded7d2 100644
--- a/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/PeerSyncReplicationTest.java
@@ -149,7 +149,7 @@ public class PeerSyncReplicationTest extends AbstractFullDistribZkTestBase {
log.info("Now shutting down initial leader");
forceNodeFailures(singletonList(initialLeaderJetty));
log.info("Updating mappings from zk");
- waitForNewLeader(cloudClient, "shard1", (Replica) initialLeaderJetty.client.info, 15);
+ LeaderFailureAfterFreshStartTest.waitForNewLeader(cloudClient, "shard1", (Replica) initialLeaderJetty.client.info, 15);
updateMappingsFromZk(jettys, clients, true);
assertEquals("PeerSynced node did not become leader", nodePeerSynced, shardToLeaderJetty.get("shard1"));
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ffa5c4ba/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java b/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java
index 60fed8b..64edd21 100644
--- a/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java
+++ b/solr/core/src/test/org/apache/solr/update/PeerSyncTest.java
@@ -16,9 +16,6 @@
*/
package org.apache.solr.update;
-import java.io.IOException;
-import java.util.Arrays;
-
import org.apache.solr.BaseDistributedSearchTestCase;
import org.apache.solr.SolrTestCaseJ4.SuppressSSL;
import org.apache.solr.client.solrj.SolrClient;
@@ -29,6 +26,9 @@ import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.StrUtils;
import org.junit.Test;
+import java.io.IOException;
+import java.util.Arrays;
+
import static org.apache.solr.update.processor.DistributedUpdateProcessor.DistribPhase;
import static org.apache.solr.update.processor.DistributingUpdateProcessorFactory.DISTRIB_UPDATE_PARAM;
@@ -145,20 +145,6 @@ public class PeerSyncTest extends BaseDistributedSearchTestCase {
assertSync(client1, numVersions, true, shardsArr[0]);
client0.commit(); client1.commit(); queryAndCompare(params("q", "*:*", "sort","_version_ desc"), client0, client1);
- // Test PeerSync after replica misses delete
- v = 2500;
- add(client0, seenLeader, sdoc("id", "2500", "_version_", ++v));
- add(client1, seenLeader, sdoc("id", "2500", "_version_", v));
- client0.commit();
- client1.commit();
- del(client0, params(DISTRIB_UPDATE_PARAM, FROM_LEADER, "_version_", Long.toString(-++v)), "2500");
- add(client0, seenLeader, sdoc("id", "2501", "_version_", ++v));
- add(client1, seenLeader, sdoc("id", "2501", "_version_", v));
- // Sync should be able to delete the document
- assertSync(client1, numVersions, true, shardsArr[0]);
- client0.commit();
- client1.commit();
- queryAndCompare(params("q", "*:*", "sort", "_version_ desc"), client0, client1);
//
// Test that handling reorders work when applying docs retrieved from peer
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ffa5c4ba/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
----------------------------------------------------------------------
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
index d04d996..03db71c 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
@@ -24,14 +24,11 @@ import java.util.concurrent.atomic.AtomicInteger;
import org.apache.commons.io.FileUtils;
import org.apache.solr.BaseDistributedSearchTestCase;
import org.apache.solr.client.solrj.embedded.JettySolrRunner;
-import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.common.cloud.ClusterState;
-import org.apache.solr.common.cloud.DocCollection;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.Slice;
import org.apache.solr.common.cloud.SolrZkClient;
import org.apache.solr.common.cloud.ZkStateReader;
-import org.apache.solr.common.cloud.Slice.State;
import org.apache.solr.core.Diagnostics;
import org.apache.solr.core.MockDirectoryFactory;
import org.apache.zookeeper.KeeperException;
@@ -225,34 +222,6 @@ public abstract class AbstractDistribZkTestBase extends BaseDistributedSearchTes
log.info("Collection has disappeared - collection: " + collection);
}
-
- static void waitForNewLeader(CloudSolrClient cloudClient, String shardName, Replica oldLeader, int maxWaitInSecs)
- throws Exception {
- log.info("Will wait for a node to become leader for {} secs", maxWaitInSecs);
- boolean waitForLeader = true;
- int i = 0;
- ZkStateReader zkStateReader = cloudClient.getZkStateReader();
- zkStateReader.forceUpdateCollection(DEFAULT_COLLECTION);
-
- while(waitForLeader) {
- ClusterState clusterState = zkStateReader.getClusterState();
- DocCollection coll = clusterState.getCollection("collection1");
- Slice slice = coll.getSlice(shardName);
- if(slice.getLeader() != oldLeader && slice.getState() == State.ACTIVE) {
- log.info("New leader got elected in {} secs", i);
- break;
- }
-
- if(i == maxWaitInSecs) {
- Diagnostics.logThreadDumps("Could not find new leader in specified timeout");
- zkStateReader.getZkClient().printLayoutToStdOut();
- fail("Could not find new leader even after waiting for " + maxWaitInSecs + "secs");
- }
-
- i++;
- Thread.sleep(1000);
- }
- }
public static void verifyReplicaStatus(ZkStateReader reader, String collection, String shard, String coreNodeName, Replica.State expectedState) throws InterruptedException {
int maxIterations = 100;
[04/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7486: Explain()
should initialize maxScore to NEGATIVE_INFINITY too.
Posted by kr...@apache.org.
LUCENE-7486: Explain() should initialize maxScore to NEGATIVE_INFINITY too.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/9304ef9f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/9304ef9f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/9304ef9f
Branch: refs/heads/jira/solr-8593
Commit: 9304ef9f118d24f76b280299706310ca8a0d40e6
Parents: ea12122
Author: Adrien Grand <jp...@gmail.com>
Authored: Fri Oct 14 15:18:29 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Fri Oct 14 15:29:30 2016 +0200
----------------------------------------------------------------------
.../src/java/org/apache/lucene/search/DisjunctionMaxQuery.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9304ef9f/lucene/core/src/java/org/apache/lucene/search/DisjunctionMaxQuery.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/search/DisjunctionMaxQuery.java b/lucene/core/src/java/org/apache/lucene/search/DisjunctionMaxQuery.java
index b29126b..44cbd52 100644
--- a/lucene/core/src/java/org/apache/lucene/search/DisjunctionMaxQuery.java
+++ b/lucene/core/src/java/org/apache/lucene/search/DisjunctionMaxQuery.java
@@ -141,7 +141,7 @@ public final class DisjunctionMaxQuery extends Query implements Iterable<Query>
@Override
public Explanation explain(LeafReaderContext context, int doc) throws IOException {
boolean match = false;
- float max = 0.0f, sum = 0.0f;
+ float max = Float.NEGATIVE_INFINITY, sum = 0.0f;
List<Explanation> subs = new ArrayList<>();
for (Weight wt : weights) {
Explanation e = wt.explain(context, doc);
[15/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7494: Give points
a per-field API.
Posted by kr...@apache.org.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/test/org/apache/lucene/index/TestPointValues.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestPointValues.java b/lucene/core/src/test/org/apache/lucene/index/TestPointValues.java
index a75290a..7228f37 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestPointValues.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestPointValues.java
@@ -503,8 +503,8 @@ public class TestPointValues extends LuceneTestCase {
IndexReader r = DirectoryReader.open(w);
for(LeafReaderContext ctx : r.leaves()) {
- PointValues points = ctx.reader().getPointValues();
- points.intersect("int",
+ PointValues points = ctx.reader().getPointValues("int");
+ points.intersect(
new IntersectVisitor() {
int lastDocID = -1;
@@ -553,8 +553,7 @@ public class TestPointValues extends LuceneTestCase {
w.forceMerge(1);
DirectoryReader r = w.getReader();
- assertEquals(0, r.leaves().get(0).reader().getPointValues().size("int"));
- assertEquals(0, r.leaves().get(0).reader().getPointValues().getDocCount("int"));
+ assertNull(r.leaves().get(0).reader().getPointValues("int"));
w.close();
r.close();
dir.close();
@@ -611,10 +610,10 @@ public class TestPointValues extends LuceneTestCase {
int size = 0;
String fieldName = "int" + field;
for(LeafReaderContext ctx : r.leaves()) {
- PointValues points = ctx.reader().getPointValues();
- if (ctx.reader().getFieldInfos().fieldInfo(fieldName) != null) {
- docCount += points.getDocCount(fieldName);
- size += points.size(fieldName);
+ PointValues points = ctx.reader().getPointValues(fieldName);
+ if (points != null) {
+ docCount += points.getDocCount();
+ size += points.size();
}
}
assertEquals(fieldDocCounts[field], docCount);
@@ -738,7 +737,7 @@ public class TestPointValues extends LuceneTestCase {
final IndexReader reader1 = DirectoryReader.open(w);
w.forceMerge(1);
final IndexReader reader2 = DirectoryReader.open(w);
- final PointValues expected = getOnlyLeafReader(reader2).getPointValues();
+ final PointValues expected = getOnlyLeafReader(reader2).getPointValues("field");
if (expected == null) {
assertNull(PointValues.getMinPackedValue(reader1, "field"));
assertNull(PointValues.getMaxPackedValue(reader1, "field"));
@@ -746,13 +745,13 @@ public class TestPointValues extends LuceneTestCase {
assertEquals(0, PointValues.size(reader1, "field"));
} else {
assertArrayEquals(
- expected.getMinPackedValue("field"),
+ expected.getMinPackedValue(),
PointValues.getMinPackedValue(reader1, "field"));
assertArrayEquals(
- expected.getMaxPackedValue("field"),
+ expected.getMaxPackedValue(),
PointValues.getMaxPackedValue(reader1, "field"));
- assertEquals(expected.getDocCount("field"), PointValues.getDocCount(reader1, "field"));
- assertEquals(expected.size("field"), PointValues.size(reader1, "field"));
+ assertEquals(expected.getDocCount(), PointValues.getDocCount(reader1, "field"));
+ assertEquals(expected.size(), PointValues.size(reader1, "field"));
}
IOUtils.close(w, reader1, reader2, dir);
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/test/org/apache/lucene/util/TestDocIdSetBuilder.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/util/TestDocIdSetBuilder.java b/lucene/core/src/test/org/apache/lucene/util/TestDocIdSetBuilder.java
index 62e85fe..625b8c2 100644
--- a/lucene/core/src/test/org/apache/lucene/util/TestDocIdSetBuilder.java
+++ b/lucene/core/src/test/org/apache/lucene/util/TestDocIdSetBuilder.java
@@ -307,37 +307,37 @@ public class TestDocIdSetBuilder extends LuceneTestCase {
}
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
+ public void intersect(IntersectVisitor visitor) throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public byte[] getMinPackedValue(String fieldName) throws IOException {
+ public byte[] getMinPackedValue() throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public byte[] getMaxPackedValue(String fieldName) throws IOException {
+ public byte[] getMaxPackedValue() throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public int getNumDimensions(String fieldName) throws IOException {
+ public int getNumDimensions() throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public int getBytesPerDimension(String fieldName) throws IOException {
+ public int getBytesPerDimension() throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public long size(String fieldName) {
+ public long size() {
return numPoints;
}
@Override
- public int getDocCount(String fieldName) {
+ public int getDocCount() {
return docCount;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/core/src/test/org/apache/lucene/util/bkd/TestMutablePointsReaderUtils.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/test/org/apache/lucene/util/bkd/TestMutablePointsReaderUtils.java b/lucene/core/src/test/org/apache/lucene/util/bkd/TestMutablePointsReaderUtils.java
index 4616ce3..8d2ea3e 100644
--- a/lucene/core/src/test/org/apache/lucene/util/bkd/TestMutablePointsReaderUtils.java
+++ b/lucene/core/src/test/org/apache/lucene/util/bkd/TestMutablePointsReaderUtils.java
@@ -20,7 +20,7 @@ import java.io.IOException;
import java.util.Arrays;
import java.util.Comparator;
-import org.apache.lucene.codecs.MutablePointsReader;
+import org.apache.lucene.codecs.MutablePointValues;
import org.apache.lucene.util.ArrayUtil;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.LuceneTestCase;
@@ -184,7 +184,7 @@ public class TestMutablePointsReaderUtils extends LuceneTestCase {
}
}
- private static class DummyPointsReader extends MutablePointsReader {
+ private static class DummyPointsReader extends MutablePointValues {
private final Point[] points;
@@ -193,16 +193,6 @@ public class TestMutablePointsReaderUtils extends LuceneTestCase {
}
@Override
- public void close() throws IOException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long ramBytesUsed() {
- return 0;
- }
-
- @Override
public void getValue(int i, BytesRef packedValue) {
packedValue.bytes = points[i].packedValue.bytes;
packedValue.offset = points[i].packedValue.offset;
@@ -226,42 +216,37 @@ public class TestMutablePointsReaderUtils extends LuceneTestCase {
}
@Override
- public void checkIntegrity() throws IOException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
+ public void intersect(IntersectVisitor visitor) throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public byte[] getMinPackedValue(String fieldName) throws IOException {
+ public byte[] getMinPackedValue() throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public byte[] getMaxPackedValue(String fieldName) throws IOException {
+ public byte[] getMaxPackedValue() throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public int getNumDimensions(String fieldName) throws IOException {
+ public int getNumDimensions() throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public int getBytesPerDimension(String fieldName) throws IOException {
+ public int getBytesPerDimension() throws IOException {
throw new UnsupportedOperationException();
}
@Override
- public long size(String fieldName) {
+ public long size() {
throw new UnsupportedOperationException();
}
@Override
- public int getDocCount(String fieldName) {
+ public int getDocCount() {
throw new UnsupportedOperationException();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/highlighter/src/java/org/apache/lucene/search/highlight/TermVectorLeafReader.java
----------------------------------------------------------------------
diff --git a/lucene/highlighter/src/java/org/apache/lucene/search/highlight/TermVectorLeafReader.java b/lucene/highlighter/src/java/org/apache/lucene/search/highlight/TermVectorLeafReader.java
index 07506f2..608e3d4 100644
--- a/lucene/highlighter/src/java/org/apache/lucene/search/highlight/TermVectorLeafReader.java
+++ b/lucene/highlighter/src/java/org/apache/lucene/search/highlight/TermVectorLeafReader.java
@@ -144,7 +144,7 @@ public class TermVectorLeafReader extends LeafReader {
}
@Override
- public PointValues getPointValues() {
+ public PointValues getPointValues(String fieldName) {
return null;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/join/src/java/org/apache/lucene/search/join/PointInSetIncludingScoreQuery.java
----------------------------------------------------------------------
diff --git a/lucene/join/src/java/org/apache/lucene/search/join/PointInSetIncludingScoreQuery.java b/lucene/join/src/java/org/apache/lucene/search/join/PointInSetIncludingScoreQuery.java
index f99f318..70c28d5 100644
--- a/lucene/join/src/java/org/apache/lucene/search/join/PointInSetIncludingScoreQuery.java
+++ b/lucene/join/src/java/org/apache/lucene/search/join/PointInSetIncludingScoreQuery.java
@@ -140,10 +140,6 @@ abstract class PointInSetIncludingScoreQuery extends Query {
@Override
public Scorer scorer(LeafReaderContext context) throws IOException {
LeafReader reader = context.reader();
- PointValues values = reader.getPointValues();
- if (values == null) {
- return null;
- }
FieldInfo fieldInfo = reader.getFieldInfos().fieldInfo(field);
if (fieldInfo == null) {
return null;
@@ -154,10 +150,14 @@ abstract class PointInSetIncludingScoreQuery extends Query {
if (fieldInfo.getPointNumBytes() != bytesPerDim) {
throw new IllegalArgumentException("field=\"" + field + "\" was indexed with bytesPerDim=" + fieldInfo.getPointNumBytes() + " but this query has bytesPerDim=" + bytesPerDim);
}
+ PointValues values = reader.getPointValues(field);
+ if (values == null) {
+ return null;
+ }
FixedBitSet result = new FixedBitSet(reader.maxDoc());
float[] scores = new float[reader.maxDoc()];
- values.intersect(field, new MergePointVisitor(sortedPackedPoints, result, scores));
+ values.intersect(new MergePointVisitor(sortedPackedPoints, result, scores));
return new Scorer(this) {
DocIdSetIterator disi = new BitSetIterator(result, 10L);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java
----------------------------------------------------------------------
diff --git a/lucene/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java b/lucene/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java
index 8db9c9e..ccbbf24 100644
--- a/lucene/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java
+++ b/lucene/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java
@@ -22,6 +22,7 @@ import java.util.Collection;
import java.util.Collections;
import java.util.Iterator;
import java.util.Map;
+import java.util.Objects;
import java.util.SortedMap;
import java.util.TreeMap;
@@ -1065,22 +1066,12 @@ public class MemoryIndex {
*/
private final class MemoryIndexReader extends LeafReader {
- private final PointValues pointValues;
private Fields memoryFields = new MemoryFields(fields);
private MemoryIndexReader() {
super(); // avoid as much superclass baggage as possible
- boolean hasPointValues = false;
for (Info info : fields.values()) {
info.prepareDocValuesAndPointValues();
- if (info.pointValues != null) {
- hasPointValues = true;
- }
- }
- if (hasPointValues) {
- pointValues = new MemoryIndexPointValues();
- } else {
- pointValues = null;
}
}
@@ -1198,8 +1189,12 @@ public class MemoryIndex {
}
@Override
- public PointValues getPointValues() {
- return pointValues;
+ public PointValues getPointValues(String fieldName) {
+ Info info = fields.get(fieldName);
+ if (info.pointValues == null) {
+ return null;
+ }
+ return new MemoryIndexPointValues(info);
}
@Override
@@ -1504,16 +1499,15 @@ public class MemoryIndex {
private class MemoryIndexPointValues extends PointValues {
+ final Info info;
+
+ MemoryIndexPointValues(Info info) {
+ this.info = Objects.requireNonNull(info);
+ }
+
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- Info info = fields.get(fieldName);
- if (info == null) {
- return;
- }
+ public void intersect(IntersectVisitor visitor) throws IOException {
BytesRef[] values = info.pointValues;
- if (values == null) {
- return;
- }
visitor.grow(info.pointValuesCount);
for (int i = 0; i < info.pointValuesCount; i++) {
@@ -1522,11 +1516,7 @@ public class MemoryIndex {
}
@Override
- public byte[] getMinPackedValue(String fieldName) throws IOException {
- Info info = fields.get(fieldName);
- if (info == null) {
- return null;
- }
+ public byte[] getMinPackedValue() throws IOException {
BytesRef[] values = info.pointValues;
if (values != null) {
return info.minPackedValue;
@@ -1536,63 +1526,28 @@ public class MemoryIndex {
}
@Override
- public byte[] getMaxPackedValue(String fieldName) throws IOException {
- Info info = fields.get(fieldName);
- if (info == null) {
- return null;
- }
- BytesRef[] values = info.pointValues;
- if (values != null) {
- return info.maxPackedValue;
- } else {
- return null;
- }
+ public byte[] getMaxPackedValue() throws IOException {
+ return info.maxPackedValue;
}
@Override
- public int getNumDimensions(String fieldName) throws IOException {
- Info info = fields.get(fieldName);
- if (info == null){
- return 0;
- }
+ public int getNumDimensions() throws IOException {
return info.fieldInfo.getPointDimensionCount();
}
@Override
- public int getBytesPerDimension(String fieldName) throws IOException {
- Info info = fields.get(fieldName);
- if (info == null){
- return 0;
- }
+ public int getBytesPerDimension() throws IOException {
return info.fieldInfo.getPointNumBytes();
}
@Override
- public long size(String fieldName) {
- Info info = fields.get(fieldName);
- if (info == null) {
- return 0;
- }
- BytesRef[] values = info.pointValues;
- if (values != null) {
- return info.pointValuesCount;
- } else {
- return 0;
- }
+ public long size() {
+ return info.pointValuesCount;
}
@Override
- public int getDocCount(String fieldName) {
- Info info = fields.get(fieldName);
- if (info == null) {
- return 0;
- }
- BytesRef[] values = info.pointValues;
- if (values != null) {
- return 1;
- } else {
- return 0;
- }
+ public int getDocCount() {
+ return 1;
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndex.java
----------------------------------------------------------------------
diff --git a/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndex.java b/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndex.java
index da3dd4c..ab4fe21 100644
--- a/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndex.java
+++ b/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndex.java
@@ -479,9 +479,9 @@ public class TestMemoryIndex extends LuceneTestCase {
MemoryIndex mi = MemoryIndex.fromDocument(doc, analyzer);
LeafReader leafReader = mi.createSearcher().getIndexReader().leaves().get(0).reader();
- assertEquals(1, leafReader.getPointValues().size("field"));
- assertArrayEquals(packedPoint, leafReader.getPointValues().getMinPackedValue("field"));
- assertArrayEquals(packedPoint, leafReader.getPointValues().getMaxPackedValue("field"));
+ assertEquals(1, leafReader.getPointValues("field").size());
+ assertArrayEquals(packedPoint, leafReader.getPointValues("field").getMinPackedValue());
+ assertArrayEquals(packedPoint, leafReader.getPointValues("field").getMaxPackedValue());
BinaryDocValues dvs = leafReader.getBinaryDocValues("field");
assertEquals(0, dvs.nextDoc());
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPoint.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPoint.java b/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPoint.java
index 19f80ab..1d3146e 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPoint.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPoint.java
@@ -21,7 +21,6 @@ import java.util.ArrayList;
import java.util.List;
import org.apache.lucene.codecs.lucene60.Lucene60PointsFormat;
-import org.apache.lucene.codecs.lucene60.Lucene60PointsReader;
import org.apache.lucene.geo.GeoUtils;
import org.apache.lucene.geo.Polygon;
import org.apache.lucene.index.FieldInfo;
@@ -307,13 +306,13 @@ public class LatLonPoint extends Field {
List<Bits> liveDocs = new ArrayList<>();
int totalHits = 0;
for(LeafReaderContext leaf : searcher.getIndexReader().leaves()) {
- PointValues points = leaf.reader().getPointValues();
+ PointValues points = leaf.reader().getPointValues(field);
if (points != null) {
- if (points instanceof Lucene60PointsReader == false) {
+ if (points instanceof BKDReader == false) {
throw new IllegalArgumentException("can only run on Lucene60PointsReader points implementation, but got " + points);
}
- totalHits += points.getDocCount(field);
- BKDReader reader = ((Lucene60PointsReader) points).getBKDReader(field);
+ totalHits += points.getDocCount();
+ BKDReader reader = (BKDReader) points;
if (reader != null) {
readers.add(reader);
docBases.add(leaf.docBase);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointDistanceQuery.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointDistanceQuery.java b/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointDistanceQuery.java
index 1b20d95..7a00cef 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointDistanceQuery.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointDistanceQuery.java
@@ -105,7 +105,7 @@ final class LatLonPointDistanceQuery extends Query {
@Override
public Scorer scorer(LeafReaderContext context) throws IOException {
LeafReader reader = context.reader();
- PointValues values = reader.getPointValues();
+ PointValues values = reader.getPointValues(field);
if (values == null) {
// No docs in this segment had any points fields
return null;
@@ -120,7 +120,7 @@ final class LatLonPointDistanceQuery extends Query {
// matching docids
DocIdSetBuilder result = new DocIdSetBuilder(reader.maxDoc(), values, field);
- values.intersect(field,
+ values.intersect(
new IntersectVisitor() {
DocIdSetBuilder.BulkAdder adder;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointInPolygonQuery.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointInPolygonQuery.java b/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointInPolygonQuery.java
index 036fe2c..ec7c682 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointInPolygonQuery.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/LatLonPointInPolygonQuery.java
@@ -98,7 +98,7 @@ final class LatLonPointInPolygonQuery extends Query {
@Override
public Scorer scorer(LeafReaderContext context) throws IOException {
LeafReader reader = context.reader();
- PointValues values = reader.getPointValues();
+ PointValues values = reader.getPointValues(field);
if (values == null) {
// No docs in this segment had any points fields
return null;
@@ -113,7 +113,7 @@ final class LatLonPointInPolygonQuery extends Query {
// matching docids
DocIdSetBuilder result = new DocIdSetBuilder(reader.maxDoc(), values, field);
- values.intersect(field,
+ values.intersect(
new IntersectVisitor() {
DocIdSetBuilder.BulkAdder adder;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/sandbox/src/java/org/apache/lucene/document/RangeFieldQuery.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/RangeFieldQuery.java b/lucene/sandbox/src/java/org/apache/lucene/document/RangeFieldQuery.java
index 1fa894f..7ebdec4 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/document/RangeFieldQuery.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/RangeFieldQuery.java
@@ -110,7 +110,7 @@ abstract class RangeFieldQuery extends Query {
final RangeFieldComparator comparator = new RangeFieldComparator();
private DocIdSet buildMatchingDocIdSet(LeafReader reader, PointValues values) throws IOException {
DocIdSetBuilder result = new DocIdSetBuilder(reader.maxDoc(), values, field);
- values.intersect(field,
+ values.intersect(
new IntersectVisitor() {
DocIdSetBuilder.BulkAdder adder;
@Override
@@ -157,7 +157,7 @@ abstract class RangeFieldQuery extends Query {
@Override
public Scorer scorer(LeafReaderContext context) throws IOException {
LeafReader reader = context.reader();
- PointValues values = reader.getPointValues();
+ PointValues values = reader.getPointValues(field);
if (values == null) {
// no docs in this segment indexed any ranges
return null;
@@ -168,9 +168,9 @@ abstract class RangeFieldQuery extends Query {
}
checkFieldInfo(fieldInfo);
boolean allDocsMatch = true;
- if (values.getDocCount(field) == reader.maxDoc()) {
+ if (values.getDocCount() == reader.maxDoc()) {
// if query crosses, docs need to be further scrutinized
- byte[] range = getInternalRange(values.getMinPackedValue(field), values.getMaxPackedValue(field));
+ byte[] range = getInternalRange(values.getMinPackedValue(), values.getMaxPackedValue());
// if the internal node is not equal and not contained by the query, all docs do not match
if ((!Arrays.equals(ranges, range)
&& (comparator.contains(range) && queryType != QueryType.CONTAINS)) == false) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/PointInGeo3DShapeQuery.java
----------------------------------------------------------------------
diff --git a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/PointInGeo3DShapeQuery.java b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/PointInGeo3DShapeQuery.java
index 105e5d8..56cb45d 100644
--- a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/PointInGeo3DShapeQuery.java
+++ b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/PointInGeo3DShapeQuery.java
@@ -70,7 +70,7 @@ final class PointInGeo3DShapeQuery extends Query {
@Override
public Scorer scorer(LeafReaderContext context) throws IOException {
LeafReader reader = context.reader();
- PointValues values = reader.getPointValues();
+ PointValues values = reader.getPointValues(field);
if (values == null) {
return null;
}
@@ -99,7 +99,7 @@ final class PointInGeo3DShapeQuery extends Query {
DocIdSetBuilder result = new DocIdSetBuilder(reader.maxDoc(), values, field);
- values.intersect(field, new PointInShapeIntersectVisitor(result, shape, shapeBounds));
+ values.intersect(new PointInShapeIntersectVisitor(result, shape, shapeBounds));
return new ConstantScoreScorer(this, score(), result.build().iterator());
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/TestGeo3DPoint.java
----------------------------------------------------------------------
diff --git a/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/TestGeo3DPoint.java b/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/TestGeo3DPoint.java
index 97606c7..2af096a 100644
--- a/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/TestGeo3DPoint.java
+++ b/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/TestGeo3DPoint.java
@@ -1486,11 +1486,11 @@ public class TestGeo3DPoint extends LuceneTestCase {
docID - reader.leaves().get(subIndex).docBase, 3, Integer.BYTES, b);
// Do first phase, where we just figure out the "path" that leads to the target docID:
- leafReader.getPointValues().intersect(fieldName, visitor);
+ leafReader.getPointValues(fieldName).intersect(visitor);
// Do second phase, where we we see how the wrapped visitor responded along that path:
visitor.startSecondPhase();
- leafReader.getPointValues().intersect(fieldName, visitor);
+ leafReader.getPointValues(fieldName).intersect(visitor);
return b.toString();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/test-framework/src/java/org/apache/lucene/codecs/asserting/AssertingPointsFormat.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/codecs/asserting/AssertingPointsFormat.java b/lucene/test-framework/src/java/org/apache/lucene/codecs/asserting/AssertingPointsFormat.java
index c6f5485..ff2e1b6 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/codecs/asserting/AssertingPointsFormat.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/codecs/asserting/AssertingPointsFormat.java
@@ -22,16 +22,13 @@ import java.util.Collection;
import org.apache.lucene.codecs.PointsFormat;
import org.apache.lucene.codecs.PointsReader;
import org.apache.lucene.codecs.PointsWriter;
+import org.apache.lucene.index.AssertingLeafReader;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.MergeState;
-import org.apache.lucene.index.PointValues.IntersectVisitor;
-import org.apache.lucene.index.PointValues.Relation;
import org.apache.lucene.index.PointValues;
import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.index.SegmentWriteState;
import org.apache.lucene.util.Accountable;
-import org.apache.lucene.util.BytesRef;
-import org.apache.lucene.util.StringHelper;
import org.apache.lucene.util.TestUtil;
/**
@@ -66,88 +63,6 @@ public final class AssertingPointsFormat extends PointsFormat {
return new AssertingPointsReader(state.segmentInfo.maxDoc(), in.fieldsReader(state));
}
- /** Validates in the 1D case that all points are visited in order, and point values are in bounds of the last cell checked */
- static class AssertingIntersectVisitor implements IntersectVisitor {
- final IntersectVisitor in;
- final int numDims;
- final int bytesPerDim;
- final byte[] lastDocValue;
- final byte[] lastMinPackedValue;
- final byte[] lastMaxPackedValue;
- private Relation lastCompareResult;
- private int lastDocID = -1;
- private int docBudget;
-
- public AssertingIntersectVisitor(int numDims, int bytesPerDim, IntersectVisitor in) {
- this.in = in;
- this.numDims = numDims;
- this.bytesPerDim = bytesPerDim;
- lastMaxPackedValue = new byte[numDims*bytesPerDim];
- lastMinPackedValue = new byte[numDims*bytesPerDim];
- if (numDims == 1) {
- lastDocValue = new byte[bytesPerDim];
- } else {
- lastDocValue = null;
- }
- }
-
- @Override
- public void visit(int docID) throws IOException {
- assert --docBudget >= 0 : "called add() more times than the last call to grow() reserved";
-
- // This method, not filtering each hit, should only be invoked when the cell is inside the query shape:
- assert lastCompareResult == Relation.CELL_INSIDE_QUERY;
- in.visit(docID);
- }
-
- @Override
- public void visit(int docID, byte[] packedValue) throws IOException {
- assert --docBudget >= 0 : "called add() more times than the last call to grow() reserved";
-
- // This method, to filter each doc's value, should only be invoked when the cell crosses the query shape:
- assert lastCompareResult == PointValues.Relation.CELL_CROSSES_QUERY;
-
- // This doc's packed value should be contained in the last cell passed to compare:
- for(int dim=0;dim<numDims;dim++) {
- assert StringHelper.compare(bytesPerDim, lastMinPackedValue, dim*bytesPerDim, packedValue, dim*bytesPerDim) <= 0: "dim=" + dim + " of " + numDims + " value=" + new BytesRef(packedValue);
- assert StringHelper.compare(bytesPerDim, lastMaxPackedValue, dim*bytesPerDim, packedValue, dim*bytesPerDim) >= 0: "dim=" + dim + " of " + numDims + " value=" + new BytesRef(packedValue);
- }
-
- // TODO: we should assert that this "matches" whatever relation the last call to compare had returned
- assert packedValue.length == numDims * bytesPerDim;
- if (numDims == 1) {
- int cmp = StringHelper.compare(bytesPerDim, lastDocValue, 0, packedValue, 0);
- if (cmp < 0) {
- // ok
- } else if (cmp == 0) {
- assert lastDocID <= docID: "doc ids are out of order when point values are the same!";
- } else {
- // out of order!
- assert false: "point values are out of order";
- }
- System.arraycopy(packedValue, 0, lastDocValue, 0, bytesPerDim);
- lastDocID = docID;
- }
- in.visit(docID, packedValue);
- }
-
- @Override
- public void grow(int count) {
- in.grow(count);
- docBudget = count;
- }
-
- @Override
- public Relation compare(byte[] minPackedValue, byte[] maxPackedValue) {
- for(int dim=0;dim<numDims;dim++) {
- assert StringHelper.compare(bytesPerDim, minPackedValue, dim*bytesPerDim, maxPackedValue, dim*bytesPerDim) <= 0;
- }
- System.arraycopy(maxPackedValue, 0, lastMaxPackedValue, 0, numDims*bytesPerDim);
- System.arraycopy(minPackedValue, 0, lastMinPackedValue, 0, numDims*bytesPerDim);
- lastCompareResult = in.compare(minPackedValue, maxPackedValue);
- return lastCompareResult;
- }
- }
static class AssertingPointsReader extends PointsReader {
private final PointsReader in;
@@ -169,9 +84,12 @@ public final class AssertingPointsFormat extends PointsFormat {
}
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- in.intersect(fieldName,
- new AssertingIntersectVisitor(in.getNumDimensions(fieldName), in.getBytesPerDimension(fieldName), visitor));
+ public PointValues getValues(String field) throws IOException {
+ PointValues values = this.in.getValues(field);
+ if (values == null) {
+ return null;
+ }
+ return new AssertingLeafReader.AssertingPointValues(values, maxDoc);
}
@Override
@@ -202,49 +120,6 @@ public final class AssertingPointsFormat extends PointsFormat {
public String toString() {
return getClass().getSimpleName() + "(" + in.toString() + ")";
}
-
- @Override
- public byte[] getMinPackedValue(String fieldName) throws IOException {
- assertStats(fieldName);
- return in.getMinPackedValue(fieldName);
- }
-
- @Override
- public byte[] getMaxPackedValue(String fieldName) throws IOException {
- assertStats(fieldName);
- return in.getMaxPackedValue(fieldName);
- }
-
- @Override
- public int getNumDimensions(String fieldName) throws IOException {
- assertStats(fieldName);
- return in.getNumDimensions(fieldName);
- }
-
- @Override
- public int getBytesPerDimension(String fieldName) throws IOException {
- assertStats(fieldName);
- return in.getBytesPerDimension(fieldName);
- }
-
- @Override
- public long size(String fieldName) {
- assertStats(fieldName);
- return in.size(fieldName);
- }
-
- @Override
- public int getDocCount(String fieldName) {
- assertStats(fieldName);
- return in.getDocCount(fieldName);
- }
-
- private void assertStats(String fieldName) {
- assert in.size(fieldName) >= 0;
- assert in.getDocCount(fieldName) >= 0;
- assert in.getDocCount(fieldName) <= in.size(fieldName);
- assert in.getDocCount(fieldName) <= maxDoc;
- }
}
static class AssertingPointsWriter extends PointsWriter {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/test-framework/src/java/org/apache/lucene/codecs/cranky/CrankyPointsFormat.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/codecs/cranky/CrankyPointsFormat.java b/lucene/test-framework/src/java/org/apache/lucene/codecs/cranky/CrankyPointsFormat.java
index fd2260b..ec7d75a 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/codecs/cranky/CrankyPointsFormat.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/codecs/cranky/CrankyPointsFormat.java
@@ -24,6 +24,7 @@ import org.apache.lucene.codecs.PointsReader;
import org.apache.lucene.codecs.PointsWriter;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.MergeState;
+import org.apache.lucene.index.PointValues;
import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.index.SegmentWriteState;
@@ -114,46 +115,67 @@ class CrankyPointsFormat extends PointsFormat {
}
@Override
- public void intersect(String fieldName, IntersectVisitor visitor) throws IOException {
- if (random.nextInt(100) == 0) {
- throw new IOException("Fake IOException");
- }
- delegate.intersect(fieldName, visitor);
- if (random.nextInt(100) == 0) {
- throw new IOException("Fake IOException");
- }
- }
-
- @Override
- public byte[] getMinPackedValue(String fieldName) throws IOException {
- if (random.nextInt(100) == 0) {
- throw new IOException("Fake IOException");
+ public PointValues getValues(String fieldName) throws IOException {
+ final PointValues delegate = this.delegate.getValues(fieldName);
+ if (delegate == null) {
+ return null;
}
- return delegate.getMinPackedValue(fieldName);
- }
-
- @Override
- public byte[] getMaxPackedValue(String fieldName) throws IOException {
- if (random.nextInt(100) == 0) {
- throw new IOException("Fake IOException");
- }
- return delegate.getMaxPackedValue(fieldName);
- }
-
- @Override
- public int getNumDimensions(String fieldName) throws IOException {
- if (random.nextInt(100) == 0) {
- throw new IOException("Fake IOException");
- }
- return delegate.getNumDimensions(fieldName);
- }
-
- @Override
- public int getBytesPerDimension(String fieldName) throws IOException {
- if (random.nextInt(100) == 0) {
- throw new IOException("Fake IOException");
- }
- return delegate.getBytesPerDimension(fieldName);
+ return new PointValues() {
+
+ @Override
+ public void intersect(IntersectVisitor visitor) throws IOException {
+ if (random.nextInt(100) == 0) {
+ throw new IOException("Fake IOException");
+ }
+ delegate.intersect(visitor);
+ if (random.nextInt(100) == 0) {
+ throw new IOException("Fake IOException");
+ }
+ }
+
+ @Override
+ public byte[] getMinPackedValue() throws IOException {
+ if (random.nextInt(100) == 0) {
+ throw new IOException("Fake IOException");
+ }
+ return delegate.getMinPackedValue();
+ }
+
+ @Override
+ public byte[] getMaxPackedValue() throws IOException {
+ if (random.nextInt(100) == 0) {
+ throw new IOException("Fake IOException");
+ }
+ return delegate.getMaxPackedValue();
+ }
+
+ @Override
+ public int getNumDimensions() throws IOException {
+ if (random.nextInt(100) == 0) {
+ throw new IOException("Fake IOException");
+ }
+ return delegate.getNumDimensions();
+ }
+
+ @Override
+ public int getBytesPerDimension() throws IOException {
+ if (random.nextInt(100) == 0) {
+ throw new IOException("Fake IOException");
+ }
+ return delegate.getBytesPerDimension();
+ }
+
+ @Override
+ public long size() {
+ return delegate.size();
+ }
+
+ @Override
+ public int getDocCount() {
+ return delegate.getDocCount();
+ }
+
+ };
}
@Override
@@ -168,15 +190,5 @@ class CrankyPointsFormat extends PointsFormat {
public long ramBytesUsed() {
return delegate.ramBytesUsed();
}
-
- @Override
- public long size(String fieldName) {
- return delegate.size(fieldName);
- }
-
- @Override
- public int getDocCount(String fieldName) {
- return delegate.getDocCount(fieldName);
- }
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java b/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java
index b2958c8..6686ec4 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java
@@ -18,10 +18,14 @@ package org.apache.lucene.index;
import java.io.IOException;
import java.util.Iterator;
+import java.util.Objects;
+import org.apache.lucene.index.PointValues.IntersectVisitor;
+import org.apache.lucene.index.PointValues.Relation;
import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.util.Bits;
import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.StringHelper;
import org.apache.lucene.util.VirtualMethod;
import org.apache.lucene.util.automaton.CompiledAutomaton;
@@ -776,7 +780,145 @@ public class AssertingLeafReader extends FilterLeafReader {
return result;
}
}
-
+
+ /** Wraps a SortedSetDocValues but with additional asserts */
+ public static class AssertingPointValues extends PointValues {
+
+ private final PointValues in;
+
+ /** Sole constructor. */
+ public AssertingPointValues(PointValues in, int maxDoc) {
+ this.in = in;
+ assertStats(maxDoc);
+ }
+
+ private void assertStats(int maxDoc) {
+ assert in.size() > 0;
+ assert in.getDocCount() > 0;
+ assert in.getDocCount() <= in.size();
+ assert in.getDocCount() <= maxDoc;
+ }
+
+ @Override
+ public void intersect(IntersectVisitor visitor) throws IOException {
+ in.intersect(new AssertingIntersectVisitor(in.getNumDimensions(), in.getBytesPerDimension(), visitor));
+ }
+
+ @Override
+ public byte[] getMinPackedValue() throws IOException {
+ return Objects.requireNonNull(in.getMinPackedValue());
+ }
+
+ @Override
+ public byte[] getMaxPackedValue() throws IOException {
+ return Objects.requireNonNull(in.getMaxPackedValue());
+ }
+
+ @Override
+ public int getNumDimensions() throws IOException {
+ return in.getNumDimensions();
+ }
+
+ @Override
+ public int getBytesPerDimension() throws IOException {
+ return in.getBytesPerDimension();
+ }
+
+ @Override
+ public long size() {
+ return in.size();
+ }
+
+ @Override
+ public int getDocCount() {
+ return in.getDocCount();
+ }
+
+ }
+
+ /** Validates in the 1D case that all points are visited in order, and point values are in bounds of the last cell checked */
+ static class AssertingIntersectVisitor implements IntersectVisitor {
+ final IntersectVisitor in;
+ final int numDims;
+ final int bytesPerDim;
+ final byte[] lastDocValue;
+ final byte[] lastMinPackedValue;
+ final byte[] lastMaxPackedValue;
+ private Relation lastCompareResult;
+ private int lastDocID = -1;
+ private int docBudget;
+
+ AssertingIntersectVisitor(int numDims, int bytesPerDim, IntersectVisitor in) {
+ this.in = in;
+ this.numDims = numDims;
+ this.bytesPerDim = bytesPerDim;
+ lastMaxPackedValue = new byte[numDims*bytesPerDim];
+ lastMinPackedValue = new byte[numDims*bytesPerDim];
+ if (numDims == 1) {
+ lastDocValue = new byte[bytesPerDim];
+ } else {
+ lastDocValue = null;
+ }
+ }
+
+ @Override
+ public void visit(int docID) throws IOException {
+ assert --docBudget >= 0 : "called add() more times than the last call to grow() reserved";
+
+ // This method, not filtering each hit, should only be invoked when the cell is inside the query shape:
+ assert lastCompareResult == Relation.CELL_INSIDE_QUERY;
+ in.visit(docID);
+ }
+
+ @Override
+ public void visit(int docID, byte[] packedValue) throws IOException {
+ assert --docBudget >= 0 : "called add() more times than the last call to grow() reserved";
+
+ // This method, to filter each doc's value, should only be invoked when the cell crosses the query shape:
+ assert lastCompareResult == PointValues.Relation.CELL_CROSSES_QUERY;
+
+ // This doc's packed value should be contained in the last cell passed to compare:
+ for(int dim=0;dim<numDims;dim++) {
+ assert StringHelper.compare(bytesPerDim, lastMinPackedValue, dim*bytesPerDim, packedValue, dim*bytesPerDim) <= 0: "dim=" + dim + " of " + numDims + " value=" + new BytesRef(packedValue);
+ assert StringHelper.compare(bytesPerDim, lastMaxPackedValue, dim*bytesPerDim, packedValue, dim*bytesPerDim) >= 0: "dim=" + dim + " of " + numDims + " value=" + new BytesRef(packedValue);
+ }
+
+ // TODO: we should assert that this "matches" whatever relation the last call to compare had returned
+ assert packedValue.length == numDims * bytesPerDim;
+ if (numDims == 1) {
+ int cmp = StringHelper.compare(bytesPerDim, lastDocValue, 0, packedValue, 0);
+ if (cmp < 0) {
+ // ok
+ } else if (cmp == 0) {
+ assert lastDocID <= docID: "doc ids are out of order when point values are the same!";
+ } else {
+ // out of order!
+ assert false: "point values are out of order";
+ }
+ System.arraycopy(packedValue, 0, lastDocValue, 0, bytesPerDim);
+ lastDocID = docID;
+ }
+ in.visit(docID, packedValue);
+ }
+
+ @Override
+ public void grow(int count) {
+ in.grow(count);
+ docBudget = count;
+ }
+
+ @Override
+ public Relation compare(byte[] minPackedValue, byte[] maxPackedValue) {
+ for(int dim=0;dim<numDims;dim++) {
+ assert StringHelper.compare(bytesPerDim, minPackedValue, dim*bytesPerDim, maxPackedValue, dim*bytesPerDim) <= 0;
+ }
+ System.arraycopy(maxPackedValue, 0, lastMaxPackedValue, 0, numDims*bytesPerDim);
+ System.arraycopy(minPackedValue, 0, lastMinPackedValue, 0, numDims*bytesPerDim);
+ lastCompareResult = in.compare(minPackedValue, maxPackedValue);
+ return lastCompareResult;
+ }
+ }
+
@Override
public NumericDocValues getNumericDocValues(String field) throws IOException {
NumericDocValues dv = super.getNumericDocValues(field);
@@ -860,7 +1002,16 @@ public class AssertingLeafReader extends FilterLeafReader {
return null;
}
}
-
+
+ @Override
+ public PointValues getPointValues(String field) throws IOException {
+ PointValues values = in.getPointValues(field);
+ if (values == null) {
+ return null;
+ }
+ return new AssertingPointValues(values, maxDoc());
+ }
+
/** Wraps a Bits but with additional asserts */
public static class AssertingBits implements Bits {
private final Thread creationThread = Thread.currentThread();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/test-framework/src/java/org/apache/lucene/index/BasePointsFormatTestCase.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/index/BasePointsFormatTestCase.java b/lucene/test-framework/src/java/org/apache/lucene/index/BasePointsFormatTestCase.java
index 540d3f9..6014fa5 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/index/BasePointsFormatTestCase.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/index/BasePointsFormatTestCase.java
@@ -80,11 +80,11 @@ public abstract class BasePointsFormatTestCase extends BaseIndexFileFormatTestCa
DirectoryReader r = DirectoryReader.open(dir);
LeafReader sub = getOnlyLeafReader(r);
- PointValues values = sub.getPointValues();
+ PointValues values = sub.getPointValues("dim");
// Simple test: make sure intersect can visit every doc:
BitSet seen = new BitSet();
- values.intersect("dim",
+ values.intersect(
new IntersectVisitor() {
@Override
public Relation compare(byte[] minPacked, byte[] maxPacked) {
@@ -122,11 +122,11 @@ public abstract class BasePointsFormatTestCase extends BaseIndexFileFormatTestCa
DirectoryReader r = DirectoryReader.open(dir);
LeafReader sub = getOnlyLeafReader(r);
- PointValues values = sub.getPointValues();
+ PointValues values = sub.getPointValues("dim");
// Simple test: make sure intersect can visit every doc:
BitSet seen = new BitSet();
- values.intersect("dim",
+ values.intersect(
new IntersectVisitor() {
@Override
public Relation compare(byte[] minPacked, byte[] maxPacked) {
@@ -168,7 +168,7 @@ public abstract class BasePointsFormatTestCase extends BaseIndexFileFormatTestCa
Bits liveDocs = MultiFields.getLiveDocs(r);
for(LeafReaderContext ctx : r.leaves()) {
- PointValues values = ctx.reader().getPointValues();
+ PointValues values = ctx.reader().getPointValues("dim");
NumericDocValues idValues = ctx.reader().getNumericDocValues("id");
if (idValues == null) {
@@ -184,7 +184,7 @@ public abstract class BasePointsFormatTestCase extends BaseIndexFileFormatTestCa
if (values != null) {
BitSet seen = new BitSet();
- values.intersect("dim",
+ values.intersect(
new IntersectVisitor() {
@Override
public Relation compare(byte[] minPacked, byte[] maxPacked) {
@@ -430,14 +430,14 @@ public abstract class BasePointsFormatTestCase extends BaseIndexFileFormatTestCa
final BitSet hits = new BitSet();
for(LeafReaderContext ctx : r.leaves()) {
- PointValues dimValues = ctx.reader().getPointValues();
+ PointValues dimValues = ctx.reader().getPointValues("field");
if (dimValues == null) {
continue;
}
final int docBase = ctx.docBase;
- dimValues.intersect("field", new IntersectVisitor() {
+ dimValues.intersect(new IntersectVisitor() {
@Override
public void visit(int docID) {
hits.set(docBase+docID);
@@ -735,13 +735,13 @@ public abstract class BasePointsFormatTestCase extends BaseIndexFileFormatTestCa
byte[] maxValues = new byte[numDims*numBytesPerDim];
for(LeafReaderContext ctx : r.leaves()) {
- PointValues dimValues = ctx.reader().getPointValues();
+ PointValues dimValues = ctx.reader().getPointValues("field");
if (dimValues == null) {
continue;
}
- byte[] leafMinValues = dimValues.getMinPackedValue("field");
- byte[] leafMaxValues = dimValues.getMaxPackedValue("field");
+ byte[] leafMinValues = dimValues.getMinPackedValue();
+ byte[] leafMaxValues = dimValues.getMaxPackedValue();
for(int dim=0;dim<numDims;dim++) {
if (StringHelper.compare(numBytesPerDim, leafMinValues, dim*numBytesPerDim, minValues, dim*numBytesPerDim) < 0) {
System.arraycopy(leafMinValues, dim*numBytesPerDim, minValues, dim*numBytesPerDim, numBytesPerDim);
@@ -792,14 +792,14 @@ public abstract class BasePointsFormatTestCase extends BaseIndexFileFormatTestCa
final BitSet hits = new BitSet();
for(LeafReaderContext ctx : r.leaves()) {
- PointValues dimValues = ctx.reader().getPointValues();
+ PointValues dimValues = ctx.reader().getPointValues("field");
if (dimValues == null) {
continue;
}
final int docBase = ctx.docBase;
- dimValues.intersect("field", new PointValues.IntersectVisitor() {
+ dimValues.intersect(new PointValues.IntersectVisitor() {
@Override
public void visit(int docID) {
if (liveDocs == null || liveDocs.get(docBase+docID)) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/test-framework/src/java/org/apache/lucene/index/RandomCodec.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/index/RandomCodec.java b/lucene/test-framework/src/java/org/apache/lucene/index/RandomCodec.java
index 127549f..161d1a9 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/index/RandomCodec.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/index/RandomCodec.java
@@ -105,9 +105,10 @@ public class RandomCodec extends AssertingCodec {
return new Lucene60PointsWriter(writeState, maxPointsInLeafNode, maxMBSortInHeap) {
@Override
- public void writeField(FieldInfo fieldInfo, PointsReader values) throws IOException {
+ public void writeField(FieldInfo fieldInfo, PointsReader reader) throws IOException {
- boolean singleValuePerDoc = values.size(fieldInfo.name) == values.getDocCount(fieldInfo.name);
+ PointValues values = reader.getValues(fieldInfo.name);
+ boolean singleValuePerDoc = values.size() == values.getDocCount();
try (BKDWriter writer = new RandomlySplittingBKDWriter(writeState.segmentInfo.maxDoc(),
writeState.directory,
@@ -116,10 +117,10 @@ public class RandomCodec extends AssertingCodec {
fieldInfo.getPointNumBytes(),
maxPointsInLeafNode,
maxMBSortInHeap,
- values.size(fieldInfo.name),
+ values.size(),
singleValuePerDoc,
bkdSplitRandomSeed ^ fieldInfo.name.hashCode())) {
- values.intersect(fieldInfo.name, new IntersectVisitor() {
+ values.intersect(new IntersectVisitor() {
@Override
public void visit(int docID) {
throw new IllegalStateException();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/test-framework/src/java/org/apache/lucene/search/QueryUtils.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/search/QueryUtils.java b/lucene/test-framework/src/java/org/apache/lucene/search/QueryUtils.java
index 4fd8fab..a3eaa80 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/search/QueryUtils.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/search/QueryUtils.java
@@ -262,7 +262,7 @@ public class QueryUtils {
}
@Override
- public PointValues getPointValues() {
+ public PointValues getPointValues(String fieldName) {
return null;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java b/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java
index e62c563..e9dc5ab 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java
@@ -2601,12 +2601,12 @@ public abstract class LuceneTestCase extends Assert {
final Map<Integer,Set<BytesRef>> docValues = new HashMap<>();
for(LeafReaderContext ctx : reader.leaves()) {
- PointValues points = ctx.reader().getPointValues();
+ PointValues points = ctx.reader().getPointValues(fieldName);
if (points == null) {
continue;
}
- points.intersect(fieldName,
+ points.intersect(
new PointValues.IntersectVisitor() {
@Override
public void visit(int docID) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java b/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
index 7c7098c..5031faf 100644
--- a/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
+++ b/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
@@ -237,7 +237,7 @@ public final class SlowCompositeReaderWrapper extends LeafReader {
}
@Override
- public PointValues getPointValues() {
+ public PointValues getPointValues(String field) {
ensureOpen();
return null;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/solr/core/src/java/org/apache/solr/uninverting/FieldCacheImpl.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/uninverting/FieldCacheImpl.java b/solr/core/src/java/org/apache/solr/uninverting/FieldCacheImpl.java
index f06f7e3..b63e5e9 100644
--- a/solr/core/src/java/org/apache/solr/uninverting/FieldCacheImpl.java
+++ b/solr/core/src/java/org/apache/solr/uninverting/FieldCacheImpl.java
@@ -277,12 +277,12 @@ class FieldCacheImpl implements FieldCache {
final void uninvertPoints(LeafReader reader, String field) throws IOException {
final int maxDoc = reader.maxDoc();
- PointValues values = reader.getPointValues();
+ PointValues values = reader.getPointValues(field);
assert values != null;
- assert values.size(field) > 0;
+ assert values.size() > 0;
final boolean setDocsWithField;
- final int docCount = values.getDocCount(field);
+ final int docCount = values.getDocCount();
assert docCount <= maxDoc;
if (docCount == maxDoc) {
// Fast case: all docs have this field:
@@ -293,7 +293,7 @@ class FieldCacheImpl implements FieldCache {
}
BytesRef scratch = new BytesRef();
- values.intersect(field, new IntersectVisitor() {
+ values.intersect(new IntersectVisitor() {
@Override
public void visit(int docID) throws IOException {
throw new AssertionError();
@@ -512,11 +512,11 @@ class FieldCacheImpl implements FieldCache {
private BitsEntry createValuePoints(LeafReader reader, String field) throws IOException {
final int maxDoc = reader.maxDoc();
- PointValues values = reader.getPointValues();
+ PointValues values = reader.getPointValues(field);
assert values != null;
- assert values.size(field) > 0;
+ assert values.size() > 0;
- final int docCount = values.getDocCount(field);
+ final int docCount = values.getDocCount();
assert docCount <= maxDoc;
if (docCount == maxDoc) {
// Fast case: all docs have this field:
@@ -615,14 +615,14 @@ class FieldCacheImpl implements FieldCache {
if (info.getPointDimensionCount() != 1) {
throw new IllegalStateException("Type mismatch: " + field + " was indexed with dimensions=" + info.getPointDimensionCount());
}
- PointValues values = reader.getPointValues();
+ PointValues values = reader.getPointValues(field);
// no actual points for this field (e.g. all points deleted)
- if (values == null || values.size(field) == 0) {
+ if (values == null || values.size() == 0) {
return DocValues.emptyNumeric();
}
// not single-valued
- if (values.size(field) != values.getDocCount(field)) {
- throw new IllegalStateException("Type mismatch: " + field + " was indexed with multiple values, numValues=" + values.size(field) + ",numDocs=" + values.getDocCount(field));
+ if (values.size() != values.getDocCount()) {
+ throw new IllegalStateException("Type mismatch: " + field + " was indexed with multiple values, numValues=" + values.size() + ",numDocs=" + values.getDocCount());
}
} else {
// postings case
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c22725f0/solr/core/src/test/org/apache/solr/search/TestDocSet.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/search/TestDocSet.java b/solr/core/src/test/org/apache/solr/search/TestDocSet.java
index 5b44514..2849f09 100644
--- a/solr/core/src/test/org/apache/solr/search/TestDocSet.java
+++ b/solr/core/src/test/org/apache/solr/search/TestDocSet.java
@@ -448,7 +448,7 @@ public class TestDocSet extends LuceneTestCase {
}
@Override
- public PointValues getPointValues() {
+ public PointValues getPointValues(String field) {
return null;
}
[43/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7507: Upgrade
morfologik-stemming to version 2.1.1 (fixes security manager issue with
Polish dictionary lookup).
Posted by kr...@apache.org.
LUCENE-7507: Upgrade morfologik-stemming to version 2.1.1 (fixes security manager issue with Polish dictionary lookup).
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/be8bd77b
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/be8bd77b
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/be8bd77b
Branch: refs/heads/jira/solr-8593
Commit: be8bd77b1745e9c7aab9f81c54e057da7dfd5e63
Parents: fe77dff
Author: Dawid Weiss <dw...@apache.org>
Authored: Wed Oct 19 23:31:03 2016 +0200
Committer: Dawid Weiss <dw...@apache.org>
Committed: Wed Oct 19 23:40:24 2016 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +++
lucene/ivy-versions.properties | 2 +-
lucene/licenses/morfologik-fsa-2.1.0.jar.sha1 | 1 -
lucene/licenses/morfologik-fsa-2.1.1.jar.sha1 | 1 +
lucene/licenses/morfologik-polish-2.1.0.jar.sha1 | 1 -
lucene/licenses/morfologik-polish-2.1.1.jar.sha1 | 1 +
lucene/licenses/morfologik-stemming-2.1.0.jar.sha1 | 1 -
lucene/licenses/morfologik-stemming-2.1.1.jar.sha1 | 1 +
solr/licenses/morfologik-fsa-2.1.0.jar.sha1 | 1 -
solr/licenses/morfologik-fsa-2.1.1.jar.sha1 | 1 +
solr/licenses/morfologik-polish-2.1.0.jar.sha1 | 1 -
solr/licenses/morfologik-polish-2.1.1.jar.sha1 | 1 +
solr/licenses/morfologik-stemming-2.1.0.jar.sha1 | 1 -
solr/licenses/morfologik-stemming-2.1.1.jar.sha1 | 1 +
14 files changed, 10 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 7105330..17e0b49 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -68,6 +68,9 @@ New Features
Bug Fixes
+* LUCENE-7507: Upgrade morfologik-stemming to version 2.1.1 (fixes security
+ manager issue with Polish dictionary lookup). (Dawid Weiss)
+
* LUCENE-7472: MultiFieldQueryParser.getFieldQuery() drops queries that are
neither BooleanQuery nor TermQuery. (Steve Rowe)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/lucene/ivy-versions.properties
----------------------------------------------------------------------
diff --git a/lucene/ivy-versions.properties b/lucene/ivy-versions.properties
index bc46ee6..7f8ac12 100644
--- a/lucene/ivy-versions.properties
+++ b/lucene/ivy-versions.properties
@@ -224,7 +224,7 @@ org.bouncycastle.version = 1.45
/org.carrot2/carrot2-mini = 3.12.0
-org.carrot2.morfologik.version = 2.1.0
+org.carrot2.morfologik.version = 2.1.1
/org.carrot2/morfologik-fsa = ${org.carrot2.morfologik.version}
/org.carrot2/morfologik-polish = ${org.carrot2.morfologik.version}
/org.carrot2/morfologik-stemming = ${org.carrot2.morfologik.version}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/lucene/licenses/morfologik-fsa-2.1.0.jar.sha1
----------------------------------------------------------------------
diff --git a/lucene/licenses/morfologik-fsa-2.1.0.jar.sha1 b/lucene/licenses/morfologik-fsa-2.1.0.jar.sha1
deleted file mode 100644
index 42c0fb3..0000000
--- a/lucene/licenses/morfologik-fsa-2.1.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-88e5993f73c102f378c711f6e47221b7a9e22d25
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/lucene/licenses/morfologik-fsa-2.1.1.jar.sha1
----------------------------------------------------------------------
diff --git a/lucene/licenses/morfologik-fsa-2.1.1.jar.sha1 b/lucene/licenses/morfologik-fsa-2.1.1.jar.sha1
new file mode 100644
index 0000000..4ceed4b
--- /dev/null
+++ b/lucene/licenses/morfologik-fsa-2.1.1.jar.sha1
@@ -0,0 +1 @@
+87866deba6aa5d19956fbe3406d8ddb5f19f5352
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/lucene/licenses/morfologik-polish-2.1.0.jar.sha1
----------------------------------------------------------------------
diff --git a/lucene/licenses/morfologik-polish-2.1.0.jar.sha1 b/lucene/licenses/morfologik-polish-2.1.0.jar.sha1
deleted file mode 100644
index 7f8b4c2..0000000
--- a/lucene/licenses/morfologik-polish-2.1.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-d5a9c7721bce2ef17444abbe25ac2e65bfaa181f
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/lucene/licenses/morfologik-polish-2.1.1.jar.sha1
----------------------------------------------------------------------
diff --git a/lucene/licenses/morfologik-polish-2.1.1.jar.sha1 b/lucene/licenses/morfologik-polish-2.1.1.jar.sha1
new file mode 100644
index 0000000..e625def
--- /dev/null
+++ b/lucene/licenses/morfologik-polish-2.1.1.jar.sha1
@@ -0,0 +1 @@
+41483a4bd785a065e03aad9be4449c21d89e2d50
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/lucene/licenses/morfologik-stemming-2.1.0.jar.sha1
----------------------------------------------------------------------
diff --git a/lucene/licenses/morfologik-stemming-2.1.0.jar.sha1 b/lucene/licenses/morfologik-stemming-2.1.0.jar.sha1
deleted file mode 100644
index 78f1961..0000000
--- a/lucene/licenses/morfologik-stemming-2.1.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-94167b64752138a246cc33cbf1a3b0bfe5274b7c
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/lucene/licenses/morfologik-stemming-2.1.1.jar.sha1
----------------------------------------------------------------------
diff --git a/lucene/licenses/morfologik-stemming-2.1.1.jar.sha1 b/lucene/licenses/morfologik-stemming-2.1.1.jar.sha1
new file mode 100644
index 0000000..6437873
--- /dev/null
+++ b/lucene/licenses/morfologik-stemming-2.1.1.jar.sha1
@@ -0,0 +1 @@
+5c169bab2e7dd04f5cb03d179a73a4339cc1d0a2
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/solr/licenses/morfologik-fsa-2.1.0.jar.sha1
----------------------------------------------------------------------
diff --git a/solr/licenses/morfologik-fsa-2.1.0.jar.sha1 b/solr/licenses/morfologik-fsa-2.1.0.jar.sha1
deleted file mode 100644
index 42c0fb3..0000000
--- a/solr/licenses/morfologik-fsa-2.1.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-88e5993f73c102f378c711f6e47221b7a9e22d25
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/solr/licenses/morfologik-fsa-2.1.1.jar.sha1
----------------------------------------------------------------------
diff --git a/solr/licenses/morfologik-fsa-2.1.1.jar.sha1 b/solr/licenses/morfologik-fsa-2.1.1.jar.sha1
new file mode 100644
index 0000000..4ceed4b
--- /dev/null
+++ b/solr/licenses/morfologik-fsa-2.1.1.jar.sha1
@@ -0,0 +1 @@
+87866deba6aa5d19956fbe3406d8ddb5f19f5352
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/solr/licenses/morfologik-polish-2.1.0.jar.sha1
----------------------------------------------------------------------
diff --git a/solr/licenses/morfologik-polish-2.1.0.jar.sha1 b/solr/licenses/morfologik-polish-2.1.0.jar.sha1
deleted file mode 100644
index 7f8b4c2..0000000
--- a/solr/licenses/morfologik-polish-2.1.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-d5a9c7721bce2ef17444abbe25ac2e65bfaa181f
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/solr/licenses/morfologik-polish-2.1.1.jar.sha1
----------------------------------------------------------------------
diff --git a/solr/licenses/morfologik-polish-2.1.1.jar.sha1 b/solr/licenses/morfologik-polish-2.1.1.jar.sha1
new file mode 100644
index 0000000..e625def
--- /dev/null
+++ b/solr/licenses/morfologik-polish-2.1.1.jar.sha1
@@ -0,0 +1 @@
+41483a4bd785a065e03aad9be4449c21d89e2d50
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/solr/licenses/morfologik-stemming-2.1.0.jar.sha1
----------------------------------------------------------------------
diff --git a/solr/licenses/morfologik-stemming-2.1.0.jar.sha1 b/solr/licenses/morfologik-stemming-2.1.0.jar.sha1
deleted file mode 100644
index 78f1961..0000000
--- a/solr/licenses/morfologik-stemming-2.1.0.jar.sha1
+++ /dev/null
@@ -1 +0,0 @@
-94167b64752138a246cc33cbf1a3b0bfe5274b7c
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/be8bd77b/solr/licenses/morfologik-stemming-2.1.1.jar.sha1
----------------------------------------------------------------------
diff --git a/solr/licenses/morfologik-stemming-2.1.1.jar.sha1 b/solr/licenses/morfologik-stemming-2.1.1.jar.sha1
new file mode 100644
index 0000000..6437873
--- /dev/null
+++ b/solr/licenses/morfologik-stemming-2.1.1.jar.sha1
@@ -0,0 +1 @@
+5c169bab2e7dd04f5cb03d179a73a4339cc1d0a2
[14/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9628: Set lastDocId
in Trie fields
Posted by kr...@apache.org.
SOLR-9628: Set lastDocId in Trie fields
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/19c7f8ea
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/19c7f8ea
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/19c7f8ea
Branch: refs/heads/jira/solr-8593
Commit: 19c7f8eacfccbe751279e6852fe6ec856403e658
Parents: c219525
Author: Anshum Gupta <an...@apache.org>
Authored: Sun Oct 16 19:07:40 2016 -0700
Committer: Anshum Gupta <an...@apache.org>
Committed: Mon Oct 17 18:59:41 2016 -0700
----------------------------------------------------------------------
solr/core/src/java/org/apache/solr/schema/TrieDoubleField.java | 1 +
solr/core/src/java/org/apache/solr/schema/TrieIntField.java | 1 +
solr/core/src/java/org/apache/solr/schema/TrieLongField.java | 1 +
3 files changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/19c7f8ea/solr/core/src/java/org/apache/solr/schema/TrieDoubleField.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/schema/TrieDoubleField.java b/solr/core/src/java/org/apache/solr/schema/TrieDoubleField.java
index 3891602..7faa38c 100644
--- a/solr/core/src/java/org/apache/solr/schema/TrieDoubleField.java
+++ b/solr/core/src/java/org/apache/solr/schema/TrieDoubleField.java
@@ -82,6 +82,7 @@ public class TrieDoubleField extends TrieField implements DoubleValueFieldType {
throw new IllegalArgumentException("docs out of order: lastDocID=" + lastDocID + " docID=" + docID);
}
if (docID > view.docID()) {
+ lastDocID = docID;
return docID == view.advance(docID);
} else {
return docID == view.docID();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/19c7f8ea/solr/core/src/java/org/apache/solr/schema/TrieIntField.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/schema/TrieIntField.java b/solr/core/src/java/org/apache/solr/schema/TrieIntField.java
index dedb667..d89dd0d 100644
--- a/solr/core/src/java/org/apache/solr/schema/TrieIntField.java
+++ b/solr/core/src/java/org/apache/solr/schema/TrieIntField.java
@@ -80,6 +80,7 @@ public class TrieIntField extends TrieField implements IntValueFieldType {
throw new IllegalArgumentException("docs out of order: lastDocID=" + lastDocID + " docID=" + docID);
}
if (docID > view.docID()) {
+ lastDocID = docID;
return docID == view.advance(docID);
} else {
return docID == view.docID();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/19c7f8ea/solr/core/src/java/org/apache/solr/schema/TrieLongField.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/schema/TrieLongField.java b/solr/core/src/java/org/apache/solr/schema/TrieLongField.java
index 229900c..c3a5440 100644
--- a/solr/core/src/java/org/apache/solr/schema/TrieLongField.java
+++ b/solr/core/src/java/org/apache/solr/schema/TrieLongField.java
@@ -80,6 +80,7 @@ public class TrieLongField extends TrieField implements LongValueFieldType {
throw new IllegalArgumentException("docs out of order: lastDocID=" + lastDocID + " docID=" + docID);
}
if (docID > view.docID()) {
+ lastDocID = docID;
return docID == view.advance(docID);
} else {
return docID == view.docID();
[44/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9417: Update
CHANGES.txt
Posted by kr...@apache.org.
SOLR-9417: Update CHANGES.txt
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/39db5489
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/39db5489
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/39db5489
Branch: refs/heads/jira/solr-8593
Commit: 39db5489b9816f7762aad92163b5eb3005331755
Parents: be8bd77
Author: Joel Bernstein <jb...@apache.org>
Authored: Wed Oct 19 18:28:35 2016 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Wed Oct 19 18:28:35 2016 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39db5489/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index efea20f..aca7601 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -140,6 +140,7 @@ New Features
* SOLR-9657: New TemplateUpdateProcessorFactory added (noble)
+* SOLR-9417: Allow daemons to terminate when they finish iterating a topic (Joel Bernstein)
Bug Fixes
----------------------
[38/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9417: Allow daemons
to terminate when they finish iterating a topic
Posted by kr...@apache.org.
SOLR-9417: Allow daemons to terminate when they finish iterating a topic
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/f43742ac
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/f43742ac
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/f43742ac
Branch: refs/heads/jira/solr-8593
Commit: f43742acc5148ea89a9a625818a7229d56e0558e
Parents: d03cc92
Author: Joel Bernstein <jb...@apache.org>
Authored: Wed Oct 19 13:16:01 2016 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Wed Oct 19 13:17:06 2016 -0400
----------------------------------------------------------------------
.../org/apache/solr/handler/StreamHandler.java | 4 +-
.../client/solrj/io/stream/DaemonStream.java | 44 ++++-
.../solrj/io/stream/StreamExpressionTest.java | 193 +++++++++++++++++--
3 files changed, 213 insertions(+), 28 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f43742ac/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index b9f30bc..3e841bd 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -19,6 +19,7 @@ package org.apache.solr.handler;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
import java.util.Collection;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
@@ -97,7 +98,7 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
private StreamFactory streamFactory = new StreamFactory();
private static final Logger logger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private String coreName;
- private Map<String, DaemonStream> daemons = new HashMap<>();
+ private Map<String, DaemonStream> daemons = Collections.synchronizedMap(new HashMap());
@Override
public PermissionNameProvider.Name getPermissionName(AuthorizationContext request) {
@@ -245,6 +246,7 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
if(daemons.containsKey(daemonStream.getId())) {
daemons.remove(daemonStream.getId()).close();
}
+ daemonStream.setDaemons(daemons);
daemonStream.open(); //This will start the deamonStream
daemons.put(daemonStream.getId(), daemonStream);
rsp.add("result-set", new DaemonResponseStream("Deamon:"+daemonStream.getId()+" started on "+coreName));
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f43742ac/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/DaemonStream.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/DaemonStream.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/DaemonStream.java
index 77648df..8214f9a 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/DaemonStream.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/DaemonStream.java
@@ -52,6 +52,8 @@ public class DaemonStream extends TupleStream implements Expressible {
private Exception exception;
private long runInterval;
private String id;
+ private Map<String, DaemonStream> daemons;
+ private boolean terminate;
private boolean closed = false;
private static final Logger logger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
@@ -64,10 +66,13 @@ public class DaemonStream extends TupleStream implements Expressible {
StreamExpressionNamedParameter idExpression = factory.getNamedOperand(expression, "id");
StreamExpressionNamedParameter runExpression = factory.getNamedOperand(expression, "runInterval");
StreamExpressionNamedParameter queueExpression = factory.getNamedOperand(expression, "queueSize");
+ StreamExpressionNamedParameter terminateExpression = factory.getNamedOperand(expression, "terminate");
+
String id = null;
long runInterval = 0L;
int queueSize = 0;
+ boolean terminate = false;
if(idExpression == null) {
throw new IOException("Invalid expression id parameter expected");
@@ -82,24 +87,26 @@ public class DaemonStream extends TupleStream implements Expressible {
}
if(queueExpression != null) {
- queueSize= Integer.parseInt(((StreamExpressionValue)queueExpression.getParameter()).getValue());
+ queueSize= Integer.parseInt(((StreamExpressionValue) queueExpression.getParameter()).getValue());
}
- // validate expression contains only what we want.
- if(expression.getParameters().size() != streamExpressions.size() + 2 &&
- expression.getParameters().size() != streamExpressions.size() + 3) {
- throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - unknown operands found", expression));
+ if(terminateExpression != null) {
+ terminate = Boolean.parseBoolean(((StreamExpressionValue) terminateExpression.getParameter()).getValue());
}
if(1 != streamExpressions.size()){
throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - expecting a single stream but found %d",expression, streamExpressions.size()));
}
- init(tupleStream, id, runInterval, queueSize);
+ init(tupleStream, id, runInterval, queueSize, terminate);
+ }
+
+ public DaemonStream(TupleStream tupleStream, String id, long runInterval, int queueSize, boolean terminate) {
+ init(tupleStream, id, runInterval, queueSize, terminate);
}
public DaemonStream(TupleStream tupleStream, String id, long runInterval, int queueSize) {
- init(tupleStream, id, runInterval, queueSize);
+ this(tupleStream, id, runInterval, queueSize, false);
}
@Override
@@ -126,6 +133,7 @@ public class DaemonStream extends TupleStream implements Expressible {
expression.addParameter(new StreamExpressionNamedParameter("id", id));
expression.addParameter(new StreamExpressionNamedParameter("runInterval", Long.toString(runInterval)));
expression.addParameter(new StreamExpressionNamedParameter("queueSize", Integer.toString(queueSize)));
+ expression.addParameter(new StreamExpressionNamedParameter("terminate", Boolean.toString(terminate)));
return expression;
}
@@ -148,10 +156,16 @@ public class DaemonStream extends TupleStream implements Expressible {
}
public void init(TupleStream tupleStream, String id, long runInterval, int queueSize) {
+ init(tupleStream, id, runInterval, queueSize, false);
+ }
+
+ public void init(TupleStream tupleStream, String id, long runInterval, int queueSize, boolean terminate) {
this.tupleStream = tupleStream;
this.id = id;
this.runInterval = runInterval;
this.queueSize = queueSize;
+ this.terminate = terminate;
+
if(queueSize > 0) {
queue = new ArrayBlockingQueue(queueSize);
eatTuples = false;
@@ -228,6 +242,10 @@ public class DaemonStream extends TupleStream implements Expressible {
return tuple;
}
+ public void setDaemons(Map<String, DaemonStream> daemons) {
+ this.daemons = daemons;
+ }
+
private synchronized void incrementIterations() {
++iterations;
}
@@ -279,6 +297,18 @@ public class DaemonStream extends TupleStream implements Expressible {
errors = 0; // Reset errors on successful run.
if (tuple.fields.containsKey("sleepMillis")) {
this.sleepMillis = tuple.getLong("sleepMillis");
+
+ if(terminate && sleepMillis > 0) {
+ //TopicStream provides sleepMillis > 0 if the last run had no Tuples.
+ //This means the topic queue is empty. Time to terminate.
+ //Remove ourselves from the daemons map.
+ if(daemons != null) {
+ daemons.remove(id);
+ }
+ //Break out of the thread loop and end the run.
+ break OUTER;
+ }
+
this.runInterval = -1;
}
break INNER;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f43742ac/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 842f6a6..7b5777d 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -533,24 +533,24 @@ public class StreamExpressionTest extends SolrCloudTestCase {
// Basic test desc
expression = StreamExpressionParser.parse("top("
- + "n=2,"
- + "unique("
- + "search(" + COLLECTION + ", q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f desc\"),"
- + "over=\"a_f\"),"
- + "sort=\"a_f desc\")");
+ + "n=2,"
+ + "unique("
+ + "search(" + COLLECTION + ", q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f desc\"),"
+ + "over=\"a_f\"),"
+ + "sort=\"a_f desc\")");
stream = new RankStream(expression, factory);
tuples = getTuples(stream);
assert(tuples.size() == 2);
- assertOrder(tuples, 4,3);
+ assertOrder(tuples, 4, 3);
// full factory
stream = factory.constructStream("top("
- + "n=4,"
- + "unique("
- + "search(" + COLLECTION + ", q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_i asc\"),"
- + "over=\"a_f\"),"
- + "sort=\"a_f asc\")");
+ + "n=4,"
+ + "unique("
+ + "search(" + COLLECTION + ", q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc, a_i asc\"),"
+ + "over=\"a_f\"),"
+ + "sort=\"a_f asc\")");
tuples = getTuples(stream);
assert(tuples.size() == 4);
@@ -827,7 +827,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
.withFunctionName("parallel", ParallelStream.class)
.withFunctionName("fetch", FetchStream.class);
- stream = factory.constructStream("parallel("+COLLECTION+", workers=2, sort=\"a_f asc\", fetch("+COLLECTION+", search(" + COLLECTION + ", q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc\", partitionKeys=\"id\"), on=\"id=a_i\", batchSize=\"2\", fl=\"subject\"))");
+ stream = factory.constructStream("parallel(" + COLLECTION + ", workers=2, sort=\"a_f asc\", fetch(" + COLLECTION + ", search(" + COLLECTION + ", q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc\", partitionKeys=\"id\"), on=\"id=a_i\", batchSize=\"2\", fl=\"subject\"))");
tuples = getTuples(stream);
assert(tuples.size() == 10);
@@ -853,7 +853,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
assertTrue("blah blah blah 9".equals(t.getString("subject")));
- stream = factory.constructStream("parallel("+COLLECTION+", workers=2, sort=\"a_f asc\", fetch("+COLLECTION+", search(" + COLLECTION + ", q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc\", partitionKeys=\"id\"), on=\"id=a_i\", batchSize=\"3\", fl=\"subject\"))");
+ stream = factory.constructStream("parallel(" + COLLECTION + ", workers=2, sort=\"a_f asc\", fetch(" + COLLECTION + ", search(" + COLLECTION + ", q=*:*, fl=\"id,a_s,a_i,a_f\", sort=\"a_f asc\", partitionKeys=\"id\"), on=\"id=a_i\", batchSize=\"3\", fl=\"subject\"))");
tuples = getTuples(stream);
assert(tuples.size() == 10);
@@ -1003,6 +1003,45 @@ public class StreamExpressionTest extends SolrCloudTestCase {
}
+
+ @Test
+ public void testTerminatingDaemonStream() throws Exception {
+
+ new UpdateRequest()
+ .add(id, "0", "a_s", "hello", "a_i", "0", "a_f", "1")
+ .add(id, "2", "a_s", "hello", "a_i", "2", "a_f", "2")
+ .add(id, "3", "a_s", "hello", "a_i", "3", "a_f", "3")
+ .add(id, "4", "a_s", "hello", "a_i", "4", "a_f", "4")
+ .add(id, "1", "a_s", "hello", "a_i", "1", "a_f", "5")
+ .add(id, "5", "a_s", "hello", "a_i", "10", "a_f", "6")
+ .add(id, "6", "a_s", "hello", "a_i", "11", "a_f", "7")
+ .add(id, "7", "a_s", "hello", "a_i", "12", "a_f", "8")
+ .add(id, "8", "a_s", "hello", "a_i", "13", "a_f", "9")
+ .add(id, "9", "a_s", "hello", "a_i", "14", "a_f", "10")
+ .commit(cluster.getSolrClient(), COLLECTION);
+
+ StreamFactory factory = new StreamFactory()
+ .withCollectionZkHost(COLLECTION, cluster.getZkServer().getZkAddress())
+ .withFunctionName("topic", TopicStream.class)
+ .withFunctionName("daemon", DaemonStream.class);
+
+ StreamExpression expression;
+ DaemonStream daemonStream;
+
+ SolrClientCache cache = new SolrClientCache();
+ StreamContext context = new StreamContext();
+ context.setSolrClientCache(cache);
+ expression = StreamExpressionParser.parse("daemon(topic("+COLLECTION+","+COLLECTION+", q=\"a_s:hello\", initialCheckpoint=0, id=\"topic1\", rows=2, fl=\"id\""
+ + "), id=test, runInterval=1000, terminate=true, queueSize=50)");
+ daemonStream = (DaemonStream)factory.constructStream(expression);
+ daemonStream.setStreamContext(context);
+
+ List<Tuple> tuples = getTuples(daemonStream);
+ assertTrue(tuples.size() == 10);
+ cache.close();
+ }
+
+
@Test
public void testRollupStream() throws Exception {
@@ -1367,7 +1406,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
assert(tuples.size() == 9);
- assertOrder(tuples, 0,1,2,3,4,7,6,8,9);
+ assertOrder(tuples, 0, 1, 2, 3, 4, 7, 6, 8, 9);
//Test descending
@@ -1376,7 +1415,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
tuples = getTuples(pstream);
assert(tuples.size() == 8);
- assertOrder(tuples, 9,8,6,4,3,2,1,0);
+ assertOrder(tuples, 9, 8, 6, 4, 3, 2, 1, 0);
}
@@ -1627,7 +1666,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
stream = new LeftOuterJoinStream(expression, factory);
tuples = getTuples(stream);
assert(tuples.size() == 10);
- assertOrder(tuples, 7,6,3,4,5,1,1,15,15,2);
+ assertOrder(tuples, 7, 6, 3, 4, 5, 1, 1, 15, 15, 2);
// Results in both searches, no join matches
expression = StreamExpressionParser.parse("leftOuterJoin("
@@ -1637,7 +1676,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
stream = new LeftOuterJoinStream(expression, factory);
tuples = getTuples(stream);
assert(tuples.size() == 8);
- assertOrder(tuples, 1,15,2,3,4,5,6,7);
+ assertOrder(tuples, 1, 15, 2, 3, 4, 5, 6, 7);
// Differing field names
expression = StreamExpressionParser.parse("leftOuterJoin("
@@ -1647,7 +1686,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
stream = new LeftOuterJoinStream(expression, factory);
tuples = getTuples(stream);
assert(tuples.size() == 10);
- assertOrder(tuples, 1,1,15,15,2,3,4,5,6,7);
+ assertOrder(tuples, 1, 1, 15, 15, 2, 3, 4, 5, 6, 7);
}
@@ -1764,7 +1803,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
stream = new OuterHashJoinStream(expression, factory);
tuples = getTuples(stream);
assert(tuples.size() == 10);
- assertOrder(tuples, 1,1,15,15,2,3,4,5,6,7);
+ assertOrder(tuples, 1, 1, 15, 15, 2, 3, 4, 5, 6, 7);
// Basic desc
expression = StreamExpressionParser.parse("outerHashJoin("
@@ -1794,7 +1833,7 @@ public class StreamExpressionTest extends SolrCloudTestCase {
stream = new OuterHashJoinStream(expression, factory);
tuples = getTuples(stream);
assert(tuples.size() == 10);
- assertOrder(tuples, 1,1,15,15,2,3,4,5,6,7);
+ assertOrder(tuples, 1, 1, 15, 15, 2, 3, 4, 5, 6, 7);
}
@Test
@@ -3202,6 +3241,120 @@ public class StreamExpressionTest extends SolrCloudTestCase {
CollectionAdminRequest.deleteCollection("parallelDestinationCollection1").process(cluster.getSolrClient());
}
+
+ @Test
+ public void testParallelTerminatingDaemonUpdateStream() throws Exception {
+
+ CollectionAdminRequest.createCollection("parallelDestinationCollection1", "conf", 2, 1).process(cluster.getSolrClient());
+ AbstractDistribZkTestBase.waitForRecoveriesToFinish("parallelDestinationCollection1", cluster.getSolrClient().getZkStateReader(),
+ false, true, TIMEOUT);
+
+ new UpdateRequest()
+ .add(id, "0", "a_s", "hello", "a_i", "0", "a_f", "0", "s_multi", "aaaa", "s_multi", "bbbb", "i_multi", "4", "i_multi", "7")
+ .add(id, "2", "a_s", "hello", "a_i", "2", "a_f", "0", "s_multi", "aaaa1", "s_multi", "bbbb1", "i_multi", "44", "i_multi", "77")
+ .add(id, "3", "a_s", "hello", "a_i", "3", "a_f", "3", "s_multi", "aaaa2", "s_multi", "bbbb2", "i_multi", "444", "i_multi", "777")
+ .add(id, "4", "a_s", "hello", "a_i", "4", "a_f", "4", "s_multi", "aaaa3", "s_multi", "bbbb3", "i_multi", "4444", "i_multi", "7777")
+ .add(id, "1", "a_s", "hello", "a_i", "1", "a_f", "1", "s_multi", "aaaa4", "s_multi", "bbbb4", "i_multi", "44444", "i_multi", "77777")
+ .commit(cluster.getSolrClient(), "collection1");
+
+ StreamExpression expression;
+ TupleStream stream;
+ Tuple t;
+
+ String zkHost = cluster.getZkServer().getZkAddress();
+ StreamFactory factory = new StreamFactory()
+ .withCollectionZkHost("collection1", cluster.getZkServer().getZkAddress())
+ .withCollectionZkHost("parallelDestinationCollection1", cluster.getZkServer().getZkAddress())
+ .withFunctionName("topic", TopicStream.class)
+ .withFunctionName("update", UpdateStream.class)
+ .withFunctionName("parallel", ParallelStream.class)
+ .withFunctionName("daemon", DaemonStream.class);
+
+ //Copy all docs to destinationCollection
+ String updateExpression = "daemon(update(parallelDestinationCollection1, batchSize=2, topic(collection1, collection1, q=\"a_s:hello\", fl=\"id,a_s,a_i,a_f,s_multi,i_multi\", partitionKeys=\"a_f\", initialCheckpoint=0, id=\"topic1\")), terminate=true, runInterval=\"1000\", id=\"test\")";
+ TupleStream parallelUpdateStream = factory.constructStream("parallel(collection1, " + updateExpression + ", workers=\"2\", zkHost=\""+zkHost+"\", sort=\"batchNumber asc\")");
+ List<Tuple> tuples = getTuples(parallelUpdateStream);
+ assert(tuples.size() == 2);
+
+
+ ModifiableSolrParams sParams = new ModifiableSolrParams(StreamingTest.mapParams(CommonParams.QT, "/stream", "action", "list"));
+
+ int workersComplete = 0;
+
+ //Daemons should terminate after the topic is completed
+ //Loop through all shards and wait for the daemons to be gone from the listing.
+ for(JettySolrRunner jetty : cluster.getJettySolrRunners()) {
+ INNER:
+ while(true) {
+ SolrStream solrStream = new SolrStream(jetty.getBaseUrl().toString() + "/collection1", sParams);
+ solrStream.open();
+ Tuple tupleResponse = solrStream.read();
+ if (tupleResponse.EOF) {
+ solrStream.close();
+ ++workersComplete;
+ break INNER;
+ } else {
+ solrStream.close();
+ Thread.sleep(1000);
+ }
+ }
+ }
+
+ assertEquals(cluster.getJettySolrRunners().size(), workersComplete);
+
+ cluster.getSolrClient().commit("parallelDestinationCollection1");
+
+ //Ensure that destinationCollection actually has the new docs.
+ expression = StreamExpressionParser.parse("search(parallelDestinationCollection1, q=*:*, fl=\"id,a_s,a_i,a_f,s_multi,i_multi\", sort=\"a_i asc\")");
+ stream = new CloudSolrStream(expression, factory);
+ tuples = getTuples(stream);
+ assertEquals(5, tuples.size());
+
+ Tuple tuple = tuples.get(0);
+ assert(tuple.getLong("id") == 0);
+ assert(tuple.get("a_s").equals("hello"));
+ assert(tuple.getLong("a_i") == 0);
+ assert(tuple.getDouble("a_f") == 0.0);
+ assertList(tuple.getStrings("s_multi"), "aaaa", "bbbb");
+ assertList(tuple.getLongs("i_multi"), Long.parseLong("4"), Long.parseLong("7"));
+
+ tuple = tuples.get(1);
+ assert(tuple.getLong("id") == 1);
+ assert(tuple.get("a_s").equals("hello"));
+ assert(tuple.getLong("a_i") == 1);
+ assert(tuple.getDouble("a_f") == 1.0);
+ assertList(tuple.getStrings("s_multi"), "aaaa4", "bbbb4");
+ assertList(tuple.getLongs("i_multi"), Long.parseLong("44444"), Long.parseLong("77777"));
+
+ tuple = tuples.get(2);
+ assert(tuple.getLong("id") == 2);
+ assert(tuple.get("a_s").equals("hello"));
+ assert(tuple.getLong("a_i") == 2);
+ assert(tuple.getDouble("a_f") == 0.0);
+ assertList(tuple.getStrings("s_multi"), "aaaa1", "bbbb1");
+ assertList(tuple.getLongs("i_multi"), Long.parseLong("44"), Long.parseLong("77"));
+
+ tuple = tuples.get(3);
+ assert(tuple.getLong("id") == 3);
+ assert(tuple.get("a_s").equals("hello"));
+ assert(tuple.getLong("a_i") == 3);
+ assert(tuple.getDouble("a_f") == 3.0);
+ assertList(tuple.getStrings("s_multi"), "aaaa2", "bbbb2");
+ assertList(tuple.getLongs("i_multi"), Long.parseLong("444"), Long.parseLong("777"));
+
+ tuple = tuples.get(4);
+ assert(tuple.getLong("id") == 4);
+ assert(tuple.get("a_s").equals("hello"));
+ assert(tuple.getLong("a_i") == 4);
+ assert(tuple.getDouble("a_f") == 4.0);
+ assertList(tuple.getStrings("s_multi"), "aaaa3", "bbbb3");
+ assertList(tuple.getLongs("i_multi"), Long.parseLong("4444"), Long.parseLong("7777"));
+
+ CollectionAdminRequest.deleteCollection("parallelDestinationCollection1").process(cluster.getSolrClient());
+ }
+
+
+
////////////////////////////////////////////
@Test
public void testCommitStream() throws Exception {
[48/50] [abbrv] lucene-solr:jira/solr-8593: Merge branch
'apache-https-master' into jira/solr-8593
Posted by kr...@apache.org.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2e7f5f92/solr/solrj/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java
----------------------------------------------------------------------
diff --cc solr/solrj/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java
index 2841922,41f3309..4ecade9
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/sql/JdbcTest.java
@@@ -397,10 -397,8 +398,9 @@@ public class JdbcTest extends SolrCloud
}
}
+ @Ignore("Fix error checking")
@Test
public void testErrorPropagation() throws Exception {
-
//Test error propagation
Properties props = new Properties();
props.put("aggregationMode", "facet");
[49/50] [abbrv] lucene-solr:jira/solr-8593: Merge branch
'apache-https-master' into jira/solr-8593
Posted by kr...@apache.org.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2e7f5f92/lucene/ivy-versions.properties
----------------------------------------------------------------------
diff --cc lucene/ivy-versions.properties
index a9aed1f,7f8ac12..e34639b
--- a/lucene/ivy-versions.properties
+++ b/lucene/ivy-versions.properties
@@@ -18,8 -18,10 +18,8 @@@ com.codahale.metrics.version = 3.0.
/com.codahale.metrics/metrics-healthchecks = ${com.codahale.metrics.version}
/com.cybozu.labs/langdetect = 1.1-20120112
- /com.drewnoakes/metadata-extractor = 2.6.2
+ /com.drewnoakes/metadata-extractor = 2.8.1
-/com.facebook.presto/presto-parser = 0.122
-
com.fasterxml.jackson.core.version = 2.5.4
/com.fasterxml.jackson.core/jackson-annotations = ${com.fasterxml.jackson.core.version}
/com.fasterxml.jackson.core/jackson-core = ${com.fasterxml.jackson.core.version}
@@@ -90,13 -99,7 +97,13 @@@ io.netty.netty-all.version = 4.0.36.Fin
/org.apache.ant/ant = 1.8.2
/org.apache.avro/avro = 1.7.5
+
- org.apache.calcite.version = 1.7.0
++org.apache.calcite.version = 1.10.0
+/org.apache.calcite.avatica/avatica = 1.7.1
+/org.apache.calcite/calcite-core = ${org.apache.calcite.version}
+/org.apache.calcite/calcite-linq4j = ${org.apache.calcite.version}
+
- /org.apache.commons/commons-compress = 1.8.1
+ /org.apache.commons/commons-compress = 1.11
/org.apache.commons/commons-exec = 1.3
/org.apache.commons/commons-math3 = 3.4.1
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2e7f5f92/solr/core/ivy.xml
----------------------------------------------------------------------
diff --cc solr/core/ivy.xml
index 9f3c190,67e4379..dcd7e78
--- a/solr/core/ivy.xml
+++ b/solr/core/ivy.xml
@@@ -129,16 -134,16 +134,20 @@@
<dependency org="antlr" name="antlr" rev="${/antlr/antlr}" conf="test.MiniKdc"/>
<dependency org="net.sf.ehcache" name="ehcache-core" rev="${/net.sf.ehcache/ehcache-core}" conf="test.MiniKdc"/>
+ <dependency org="org.apache.curator" name="curator-framework" rev="${/org.apache.curator/curator-framework}" conf="compile"/>
+ <dependency org="org.apache.curator" name="curator-client" rev="${/org.apache.curator/curator-client}" conf="compile"/>
+ <dependency org="org.apache.curator" name="curator-recipes" rev="${/org.apache.curator/curator-recipes}" conf="compile"/>
+
<!-- StatsComponents percentiles Dependencies-->
<dependency org="com.tdunning" name="t-digest" rev="${/com.tdunning/t-digest}" conf="compile->*"/>
- <!-- SQL Parser -->
- <dependency org="com.facebook.presto" name="presto-parser" rev="${/com.facebook.presto/presto-parser}"/>
- <dependency org="io.airlift" name="slice" rev="${/io.airlift/slice}"/>
+ <!-- SQL Parser -->
+ <dependency org="org.apache.calcite" name="calcite-core" rev="${/org.apache.calcite/calcite-core}"/>
+ <dependency org="org.apache.calcite" name="calcite-linq4j" rev="${/org.apache.calcite/calcite-linq4j}"/>
+ <dependency org="org.apache.calcite.avatica" name="avatica" rev="${/org.apache.calcite.avatica/avatica}"/>
+ <dependency org="net.hydromatic" name="eigenbase-properties" rev="1.1.5"/>
+ <dependency org="org.codehaus.janino" name="janino" rev="2.7.6"/>
+ <dependency org="org.codehaus.janino" name="commons-compiler" rev="2.7.6"/>
<exclude org="*" ext="*" matcher="regexp" type="${ivy.exclude.types}"/>
</dependencies>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2e7f5f92/solr/core/src/java/org/apache/solr/handler/SQLHandler.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/handler/SQLHandler.java
index 9e76726,3074d9b..c389fb7
--- a/solr/core/src/java/org/apache/solr/handler/SQLHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/SQLHandler.java
@@@ -55,8 -69,17 +55,10 @@@ import org.apache.solr.util.plugin.Solr
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.facebook.presto.sql.parser.SqlParser;
-
public class SQLHandler extends RequestHandlerBase implements SolrCoreAware , PermissionNameProvider {
+ private static final Logger logger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
private static String defaultZkhost = null;
private static String defaultWorkerCollection = null;
@@@ -77,17 -103,25 +82,21 @@@
}
public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throws Exception {
- SolrParams params = req.getParams();
- params = adjustParams(params);
- req.setParams(params);
+ ModifiableSolrParams params = new ModifiableSolrParams(req.getParams());
String sql = params.get("stmt");
- int numWorkers = params.getInt("numWorkers", 1);
- String workerCollection = params.get("workerCollection", defaultWorkerCollection);
- String workerZkhost = params.get("workerZkhost", defaultZkhost);
- String mode = params.get("aggregationMode", "map_reduce");
- StreamContext context = new StreamContext();
-
- // JDBC driver requires metadata from the SQLHandler. Default to false since this adds a new Metadata stream.
- boolean includeMetadata = params.getBool("includeMetadata", false);
+ // Set defaults for parameters
+ params.set("numWorkers", params.getInt("numWorkers", 1));
+ params.set("workerCollection", params.get("workerCollection", defaultWorkerCollection));
+ params.set("workerZkhost", params.get("workerZkhost", defaultZkhost));
+ params.set("aggregationMode", params.get("aggregationMode", "map_reduce"));
-
+ TupleStream tupleStream = null;
try {
+
+ if(!isCloud) {
+ throw new IllegalStateException(sqlNonCloudErrorMsg);
+ }
+
if(sql == null) {
throw new Exception("stmt parameter cannot be null");
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2e7f5f92/solr/core/src/test/org/apache/solr/handler/TestSQLHandler.java
----------------------------------------------------------------------
diff --cc solr/core/src/test/org/apache/solr/handler/TestSQLHandler.java
index 0bac4d6,4ce2798..d944148
--- a/solr/core/src/test/org/apache/solr/handler/TestSQLHandler.java
+++ b/solr/core/src/test/org/apache/solr/handler/TestSQLHandler.java
@@@ -18,10 -18,11 +18,8 @@@ package org.apache.solr.handler
import java.io.IOException;
import java.util.ArrayList;
- import java.util.HashMap;
-import java.util.Collections;
import java.util.List;
- import java.util.Map;
-import com.facebook.presto.sql.parser.SqlParser;
-import com.facebook.presto.sql.tree.Statement;
import org.apache.solr.client.solrj.io.Tuple;
import org.apache.solr.client.solrj.io.stream.ExceptionStream;
import org.apache.solr.client.solrj.io.stream.SolrStream;
@@@ -29,7 -30,11 +27,10 @@@ import org.apache.solr.client.solrj.io.
import org.apache.solr.cloud.AbstractFullDistribZkTestBase;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.params.CommonParams;
+ import org.apache.solr.common.params.ModifiableSolrParams;
+ import org.apache.solr.common.params.SolrParams;
+
import org.junit.After;
-import org.junit.AfterClass;
import org.junit.Before;
import org.junit.Test;
@@@ -69,19 -80,223 +70,20 @@@ public class TestSQLHandler extends Abs
@Test
public void doTest() throws Exception {
waitForRecoveriesToFinish(false);
- testPredicate();
testBasicSelect();
- testWhere();
- testMixedCaseFields();
++// testWhere();
+// testMixedCaseFields();
testBasicGrouping();
- testBasicGroupingFacets();
- testSelectDistinct();
- testSelectDistinctFacets();
+ testBasicGroupingFacets(); // TODO push down facets
+// testSelectDistinct(); // TODO fails due to sort asc by default missing
+// testSelectDistinctFacets(); // TODO push down facets and fails due to sort asc by default missing
testAggregatesWithoutGrouping();
- testSQLException();
- testTimeSeriesGrouping();
- testTimeSeriesGroupingFacet();
+// testSQLException(); // TODO fix exception checking
+// testTimeSeriesGrouping();
+// testTimeSeriesGroupingFacet(); // TODO push down facets
testParallelBasicGrouping();
- // testParallelSelectDistinct(); //TODO fails due to sort asc by default missing
- testParallelSelectDistinct();
- testParallelTimeSeriesGrouping();
- testCatalogStream();
- testSchemasStream();
- testTablesStream();
- }
-
- private void testPredicate() throws Exception {
-
- SqlParser parser = new SqlParser();
- String sql = "select a from b where c = 'd'";
- Statement statement = parser.createStatement(sql);
- SQLHandler.SQLVisitor sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:\"d\")", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c = 5";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:\"5\")", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c <> 'd'";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(-c:\"d\")", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c <> 5";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(-c:\"5\")", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c > 'd'";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:{\"d\" TO *])", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c > 5";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:{\"5\" TO *])", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c >= 'd'";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:[\"d\" TO *])", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c >= 5";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:[\"5\" TO *])", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c < 'd'";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:[* TO \"d\"})", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c < 5";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:[* TO \"5\"})", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c <= 'd'";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:[* TO \"d\"])", sqlVistor.query);
-
- parser = new SqlParser();
- sql = "select a from b where c <= 5";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, 0);
-
- assertEquals("(c:[* TO \"5\"])", sqlVistor.query);
-
- //Add parens
- parser = new SqlParser();
- sql = "select a from b where (c = 'd')";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
-
- assert(sqlVistor.query.equals("(c:\"d\")"));
-
-
- //Upper case
- parser = new SqlParser();
- sql = "select a from b where ('CcC' = 'D')";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
- assert(sqlVistor.query.equals("(CcC:\"D\")"));
-
- //Phrase
- parser = new SqlParser();
- sql = "select a from b where (c = 'd d')";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
-
- assert(sqlVistor.query.equals("(c:\"d d\")"));
-
- // AND
- parser = new SqlParser();
- sql = "select a from b where ((c = 'd') AND (l = 'z'))";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
-
- assert(sqlVistor.query.equals("((c:\"d\") AND (l:\"z\"))"));
-
- // OR
-
- parser = new SqlParser();
- sql = "select a from b where ((c = 'd') OR (l = 'z'))";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
-
- assert(sqlVistor.query.equals("((c:\"d\") OR (l:\"z\"))"));
-
- // AND NOT
-
- parser = new SqlParser();
- sql = "select a from b where ((c = 'd') AND NOT (l = 'z'))";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
-
- assert(sqlVistor.query.equals("((c:\"d\") AND -(l:\"z\"))"));
-
- // NESTED
- parser = new SqlParser();
- sql = "select a from b where ((c = 'd') OR ((l = 'z') AND (m = 'j')))";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
-
- assert(sqlVistor.query.equals("((c:\"d\") OR ((l:\"z\") AND (m:\"j\")))"));
-
- // NESTED NOT
- parser = new SqlParser();
- sql = "select a from b where ((c = 'd') OR ((l = 'z') AND NOT (m = 'j')))";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
-
- assert(sqlVistor.query.equals("((c:\"d\") OR ((l:\"z\") AND -(m:\"j\")))"));
-
- // RANGE - Will have to do until SQL BETWEEN is supported.
- // NESTED
- parser = new SqlParser();
- sql = "select a from b where ((c = '[0 TO 100]') OR ((l = '(z)') AND (m = 'j')))";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
-
- assert(sqlVistor.query.equals("((c:[0 TO 100]) OR ((l:(z)) AND (m:\"j\")))"));
-
- // Wildcard
- parser = new SqlParser();
- sql = "select a from b where ((c = '[0 TO 100]') OR ((l = '(z*)') AND (m = 'j')))";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
- assert(sqlVistor.query.equals("((c:[0 TO 100]) OR ((l:(z*)) AND (m:\"j\")))"));
-
- // Complex Lucene/Solr Query
- parser = new SqlParser();
- sql = "select a from b where (('c' = '[0 TO 100]') OR ((l = '(z*)') AND ('M' = '(j OR (k NOT s))')))";
- statement = parser.createStatement(sql);
- sqlVistor = new SQLHandler.SQLVisitor(new StringBuilder());
- sqlVistor.process(statement, new Integer(0));
- assert(sqlVistor.query.equals("((c:[0 TO 100]) OR ((l:(z*)) AND (M:(j OR (k NOT s)))))"));
++// testParallelSelectDistinct();
+// testParallelTimeSeriesGrouping();
}
private void testBasicSelect() throws Exception {
@@@ -101,12 -317,11 +103,11 @@@
indexDoc(sdoc("id", "7", "text", "XXXX XXXX", "str_s", "c", "field_i", "50"));
indexDoc(sdoc("id", "8", "text", "XXXX XXXX", "str_s", "c", "field_i", "60"));
commit();
+
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select 'id', field_i, str_s from collection1 where 'text'='XXXX' order by field_i desc");
++ "stmt", "select id, field_i, str_s from collection1 where text='XXXX' order by field_i desc");
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select id, field_i, str_s from collection1 where text='XXXX' order by field_i desc");
-
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
assert(tuples.size() == 8);
@@@ -153,14 -368,10 +154,10 @@@
assert(tuple.getLong("field_i") == 7);
assert(tuple.get("str_s").equals("a"));
-
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
-
//Test unlimited unsorted result. Should sort on _version_ desc
- params.put("stmt", "select id, field_i, str_s from collection1 where text='XXXX'");
- sParams = mapParams(CommonParams.QT, "/sql", "stmt", "select 'id', field_i, str_s from collection1 where 'text'='XXXX'");
++ sParams = mapParams(CommonParams.QT, "/sql", "stmt", "select id, field_i, str_s from collection1 where text='XXXX'");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 8);
@@@ -269,13 -475,11 +261,10 @@@
assert(tuple.getLong("myInt") == 7);
assert(tuple.get("myString").equals("a"));
-
+ sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id as myId, field_i as myInt, str_s as myString from collection1 where text='XXXX' AND id='(1 2 3)' order by field_i desc");
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select id as myId, field_i as myInt, str_s as myString from collection1 where text='XXXX' " +
- "AND id='(1 2 3)' order by field_i desc");
-
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 3);
@@@ -331,6 -534,142 +319,141 @@@
}
}
+ private void testWhere() throws Exception {
+ try {
+ CloudJettyRunner jetty = this.cloudJettys.get(0);
+
+ del("*:*");
+
+ commit();
+
+ indexDoc(sdoc("id", "1", "text", "XXXX XXXX", "str_s", "a", "field_i", "7"));
+ indexDoc(sdoc("id", "2", "text", "XXXX XXXX", "str_s", "b", "field_i", "8"));
+ indexDoc(sdoc("id", "3", "text", "XXXX XXXX", "str_s", "a", "field_i", "20"));
+ indexDoc(sdoc("id", "4", "text", "XXXX XXXX", "str_s", "b", "field_i", "11"));
+ indexDoc(sdoc("id", "5", "text", "XXXX XXXX", "str_s", "c", "field_i", "30"));
+ indexDoc(sdoc("id", "6", "text", "XXXX XXXX", "str_s", "c", "field_i", "40"));
+ indexDoc(sdoc("id", "7", "text", "XXXX XXXX", "str_s", "c", "field_i", "50"));
+ indexDoc(sdoc("id", "8", "text", "XXXX XXXX", "str_s", "c", "field_i", "60"));
+ commit();
+
+ // Equals
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id from collection1 where id = 1 order by id asc");
+
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
+ List<Tuple> tuples = getTuples(solrStream);
+
+ assertEquals(1, tuples.size());
+
+ Tuple tuple = tuples.get(0);
+ assertEquals(1L, tuple.get("id"));
+
+ // Not Equals <>
+ sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id from collection1 where id <> 1 order by id asc limit 10");
+
+ solrStream = new SolrStream(jetty.url, sParams);
+ tuples = getTuples(solrStream);
+
+ assertEquals(7, tuples.size());
+
+ tuple = tuples.get(0);
+ assertEquals(2L, tuple.get("id"));
+ tuple = tuples.get(1);
+ assertEquals(3L, tuple.get("id"));
+ tuple = tuples.get(2);
+ assertEquals(4L, tuple.get("id"));
+ tuple = tuples.get(3);
+ assertEquals(5L, tuple.get("id"));
+ tuple = tuples.get(4);
+ assertEquals(6L, tuple.get("id"));
+ tuple = tuples.get(5);
+ assertEquals(7L, tuple.get("id"));
+ tuple = tuples.get(6);
+ assertEquals(8L, tuple.get("id"));
+
+ // Not Equals !=
+ sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id from collection1 where id != 1 order by id asc limit 10");
+
+ solrStream = new SolrStream(jetty.url, sParams);
+ tuples = getTuples(solrStream);
+
+ assertEquals(7, tuples.size());
+
+ tuple = tuples.get(0);
+ assertEquals(2L, tuple.get("id"));
+ tuple = tuples.get(1);
+ assertEquals(3L, tuple.get("id"));
+ tuple = tuples.get(2);
+ assertEquals(4L, tuple.get("id"));
+ tuple = tuples.get(3);
+ assertEquals(5L, tuple.get("id"));
+ tuple = tuples.get(4);
+ assertEquals(6L, tuple.get("id"));
+ tuple = tuples.get(5);
+ assertEquals(7L, tuple.get("id"));
+ tuple = tuples.get(6);
+ assertEquals(8L, tuple.get("id"));
+
+ // Less than
+ sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id from collection1 where id < 2 order by id asc");
+
+ solrStream = new SolrStream(jetty.url, sParams);
+ tuples = getTuples(solrStream);
+
+ assertEquals(1, tuples.size());
+
+ tuple = tuples.get(0);
+ assertEquals(1L, tuple.get("id"));
+
+ // Less than equal
+ sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id from collection1 where id <= 2 order by id asc");
+
+ solrStream = new SolrStream(jetty.url, sParams);
+ tuples = getTuples(solrStream);
+
+ assertEquals(2, tuples.size());
+
+ tuple = tuples.get(0);
+ assertEquals(1L, tuple.get("id"));
+ tuple = tuples.get(1);
+ assertEquals(2L, tuple.get("id"));
+
+ // Greater than
+ sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id from collection1 where id > 7 order by id asc");
+
+ solrStream = new SolrStream(jetty.url, sParams);
+ tuples = getTuples(solrStream);
+
+ assertEquals(1, tuples.size());
+
+ tuple = tuples.get(0);
+ assertEquals(8L, tuple.get("id"));
+
+ // Greater than equal
+ sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id from collection1 where id >= 7 order by id asc");
+
+ solrStream = new SolrStream(jetty.url, sParams);
+ tuples = getTuples(solrStream);
+
+ assertEquals(2, tuples.size());
+
+ tuple = tuples.get(0);
+ assertEquals(7L, tuple.get("id"));
+ tuple = tuples.get(1);
+ assertEquals(8L, tuple.get("id"));
+
+ } finally {
+ delete();
+ }
+ }
+
-
private void testMixedCaseFields() throws Exception {
try {
@@@ -349,11 -688,10 +472,11 @@@
indexDoc(sdoc("id", "7", "Text_t", "XXXX XXXX", "Str_s", "c", "Field_i", "50"));
indexDoc(sdoc("id", "8", "Text_t", "XXXX XXXX", "Str_s", "c", "Field_i", "60"));
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select id, Field_i, Str_s from collection1 where Text_t='XXXX' order by Field_i desc");
+
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql",
+ "stmt", "select id, Field_i, Str_s from Collection1 where Text_t='XXXX' order by Field_i desc");
+
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
assert(tuples.size() == 8);
@@@ -400,12 -738,10 +523,10 @@@
assert(tuple.getLong("Field_i") == 7);
assert(tuple.get("Str_s").equals("a"));
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select Str_s, sum(Field_i) as `sum(Field_i)` from collection1 where id='(1 8)' group by Str_s " +
- "having (sum(Field_i) = 7 OR sum(Field_i) = 60) order by sum(Field_i) desc");
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select Str_s, sum(Field_i) from Collection1 where 'id'='(1 8)' group by Str_s having (sum(Field_i) = 7 OR 'sum(Field_i)' = 60) order by 'sum(Field_i)' desc");
++ "stmt", "select Str_s, sum(Field_i) as `sum(Field_i)` from Collection1 where 'id'='(1 8)' group by Str_s having (sum(Field_i) = 7 OR sum(Field_i) = 60) order by sum(Field_i) desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 2);
@@@ -418,12 -754,10 +539,10 @@@
assert(tuple.get("Str_s").equals("a"));
assert(tuple.getDouble("sum(Field_i)") == 7);
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select Str_s, sum(Field_i) as `sum(Field_i)` from collection1 where id='(1 8)' group by Str_s " +
- "having (sum(Field_i) = 7 OR sum(Field_i) = 60) order by sum(Field_i) desc");
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select Str_s, sum(Field_i) from Collection1 where 'id'='(1 8)' group by 'Str_s' having (sum(Field_i) = 7 OR 'sum(Field_i)' = 60) order by 'sum(Field_i)' desc");
++ "stmt", "select Str_s, sum(Field_i) as `sum(Field_i)` from Collection1 where id='(1 8)' group by Str_s having (sum(Field_i) = 7 OR sum(Field_i) = 60) order by sum(Field_i) desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 2);
@@@ -435,8 -769,8 +554,6 @@@
tuple = tuples.get(1);
assert(tuple.get("Str_s").equals("a"));
assert(tuple.getDouble("sum(Field_i)") == 7);
--
--
} finally {
delete();
}
@@@ -483,26 -815,20 +598,20 @@@
//An exception not detected by the parser thrown from the /select handler
assert(tuple.getException().contains("sort param field can't be found:"));
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s, count(*), sum(field_iff), min(field_i), max(field_i), " +
- "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
- "having ((sum(field_iff) = 19) AND (min(field_i) = 8))");
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s, count(*), sum(field_iff), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having ((sum(field_iff) = 19) AND (min(field_i) = 8))");
++ "stmt", "select str_s, count(*), sum(field_iff), min(field_i), max(field_i), cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s having ((sum(field_iff) = 19) AND (min(field_i) = 8))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuple = getTuple(new ExceptionStream(solrStream));
assert(tuple.EOF);
assert(tuple.EXCEPTION);
//An exception not detected by the parser thrown from the /export handler
assert(tuple.getException().contains("undefined field:"));
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s, count(*), blah(field_iff), min(field_i), max(field_i), " +
- "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
- "having ((sum(field_iff) = 19) AND (min(field_i) = 8))");
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s, count(*), blah(field_iff), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having ((sum(field_iff) = 19) AND (min(field_i) = 8))");
++ "stmt", "select str_s, count(*), blah(field_iff), min(field_i), max(field_i), cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s having ((sum(field_iff) = 19) AND (min(field_i) = 8))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuple = getTuple(new ExceptionStream(solrStream));
assert(tuple.EOF);
assert(tuple.EXCEPTION);
@@@ -542,12 -867,11 +650,11 @@@
indexr("id", "7", "text", "XXXX XXXX", "str_s", "c", "field_i", "50");
indexr("id", "8", "text", "XXXX XXXX", "str_s", "c", "field_i", "60");
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), cast(avg(1.0 * field_i) as float)" +
- " from collection1 where text='XXXX' group by str_s order by sum(field_i) asc limit 2");
-
+
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s, 'count(*)', sum('field_i'), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by 'str_s' order by 'sum(field_i)' asc limit 2");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s order by sum(field_i) asc limit 2");
+
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
//Only two results because of the limit.
@@@ -565,18 -889,17 +672,16 @@@
tuple = tuples.get(1);
assert(tuple.get("str_s").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
-
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 27); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s as myString, 'count(*)', sum('field_i') as sum, min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by 'str_s' order by sum asc limit 2");
++ "stmt", "select str_s as myString, 'count(*)', sum(field_i) as sum, min(field_i), max(field_i), cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s order by sum asc limit 2");
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s as myString, count(*), sum(field_i) as mySum, min(field_i), max(field_i), " +
- "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s order by mySum asc limit 2");
-
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
//Only two results because of the limit.
@@@ -592,21 -915,17 +697,20 @@@
tuple = tuples.get(1);
assert(tuple.get("myString").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
-
-
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("mySum") == 27);
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
+
+ // TODO fix test - Cannot apply 'NOT' to arguments of type 'NOT<JAVATYPE(CLASS JAVA.LANG.STRING)>'. Supported form(s): 'NOT<BOOLEAN>'
+ /*
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), "
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where (text='XXXX' AND NOT text='XXXX XXX') group by str_s order by str_s desc");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), "
+ + "cast(avg(1.0 * field_i) as float) from collection1 where (text='XXXX' AND NOT (text='XXXX XXX')) "
+ + "group by str_s order by str_s desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
//The sort by and order by match and no limit is applied. All the Tuples should be returned in
@@@ -632,21 -951,17 +736,20 @@@
tuple = tuples.get(2);
assert(tuple.get("str_s").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
-
-
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 27); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("avg(field_i)") == 13.5D); //avg(field_i)
+
+ // TODO fix test - Cannot apply 'NOT' to arguments of type 'NOT<JAVATYPE(CLASS JAVA.LANG.STRING)>'. Supported form(s): 'NOT<BOOLEAN>'
+ /*
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s as myString, count(*) as myCount, sum(field_i) as mySum, min(field_i) as myMin, "
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s as myString, count(*) as count, sum(field_i) as sum, min(field_i) as min, max(field_i) as max, avg(field_i) as avg from collection1 where (text='XXXX' AND NOT text='XXXX XXX') group by str_s order by str_s desc");
++ "stmt", "select str_s as myString, count(*) as myCount, sum(field_i) as mySum, min(field_i) as myMin, "
+ + "max(field_i) as myMax, cast(avg(1.0 * field_i) as float) as myAvg from collection1 "
+ + "where (text='XXXX' AND NOT (text='XXXX XXX')) group by str_s order by str_s desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
//The sort by and order by match and no limit is applied. All the Tuples should be returned in
@@@ -672,76 -987,69 +775,71 @@@
tuple = tuples.get(2);
assert(tuple.get("myString").equals("a"));
- assert(tuple.getDouble("count") == 2);
- assert(tuple.getDouble("sum") == 27);
- assert(tuple.getDouble("min") == 7);
- assert(tuple.getDouble("max") == 20);
- assert(tuple.getDouble("avg") == 13.5D);
-
-
+ assert(tuple.getDouble("myCount") == 2);
+ assert(tuple.getDouble("mySum") == 27);
+ assert(tuple.getDouble("myMin") == 7);
+ assert(tuple.getDouble("myMax") == 20);
+ assert(tuple.getDouble("myAvg") == 13.5D);
+ */
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), cast(avg(1.0 * field_i) as float) " +
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having sum(field_i) = 19");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), cast(avg(1.0 * field_i) as float) " +
+ "from collection1 where text='XXXX' group by str_s having sum(field_i) = 19");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 1);
tuple = tuples.get(0);
assert(tuple.get("str_s").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 19); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 8); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 11); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 9.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), cast(avg(1.0 * field_i) as float) " +
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having ((sum(field_i) = 19) AND (min(field_i) = 8))");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), cast(avg(1.0 * field_i) as float) " +
+ "from collection1 where text='XXXX' group by str_s having ((sum(field_i) = 19) AND (min(field_i) = 8))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 1);
tuple = tuples.get(0);
assert(tuple.get("str_s").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 19); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 8); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 11); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 9.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s, count(*), sum(field_i) as mySum, min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s as myString, count(*), sum(field_i) as sum, min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by myString having ((sum = 19) AND (min(field_i) = 8))");
++ "stmt", "select str_s, count(*), sum(field_i) as mySum, min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "having ((sum(field_i) = 19) AND (min(field_i) = 8))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 1);
tuple = tuples.get(0);
- assert(tuple.get("myString").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
-
+ assert(tuple.get("str_s").equals("b"));
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("mySum") == 19);
+ assert(tuple.getDouble("EXPR$3") == 8); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 11); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 9.5D); //avg(field_i)
-
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having ((sum(field_i) = 19) AND (min(field_i) = 100))");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "having ((sum(field_i) = 19) AND (min(field_i) = 100))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 0);
@@@ -767,12 -1079,10 +865,11 @@@
indexr("id", "7", "text", "XXXX XXXX", "str_s", "c", "field_i", "50");
indexr("id", "8", "text", "XXXX XXXX", "str_s", "c", "field_i", "60");
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select distinct str_s, field_i from collection1 order by str_s asc, field_i asc");
+
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select distinct 'str_s', 'field_i' from collection1 order by 'str_s' asc, 'field_i' asc");
++ "stmt", "select distinct str_s, field_i from collection1 order by str_s asc, field_i asc");
+
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
assert(tuples.size() == 6);
@@@ -900,14 -1204,11 +991,12 @@@
// Test without a sort. Sort should be asc by default.
-
+ /*
+ // TODO figure out what should be sort asc by default (version?)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select distinct str_s, field_i from collection1");
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
+ "stmt", "select distinct str_s, field_i from collection1");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 6);
@@@ -935,15 -1236,14 +1024,13 @@@
tuple = tuples.get(5);
assert(tuple.get("str_s").equals("c"));
assert(tuple.getLong("field_i") == 60);
-
+ */
// Test with a predicate.
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
-
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
++ "stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 2);
@@@ -955,14 -1255,14 +1042,11 @@@
tuple = tuples.get(1);
assert(tuple.get("str_s").equals("a"));
assert(tuple.getLong("field_i") == 20);
--
--
} finally {
delete();
}
}
--
private void testSelectDistinct() throws Exception {
try {
@@@ -981,11 -1281,11 +1065,11 @@@
indexr("id", "7", "text", "XXXX XXXX", "str_s", "c", "field_i", "50");
indexr("id", "8", "text", "XXXX XXXX", "str_s", "c", "field_i", "60");
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select distinct str_s, field_i from collection1 order by str_s asc, field_i asc");
-
+
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select distinct 'str_s', 'field_i' from collection1 order by 'str_s' asc, 'field_i' asc");
++ "stmt", "select distinct str_s, field_i from collection1 order by str_s asc, field_i asc");
+
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
assert(tuples.size() == 6);
@@@ -1110,11 -1407,11 +1191,10 @@@
// Test without a sort. Sort should be asc by default.
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select distinct str_s, field_i from collection1");
-
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select distinct str_s, field_i from collection1");
++ "stmt", "select distinct str_s, field_i from collection1");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 6);
@@@ -1144,11 -1441,11 +1224,10 @@@
assert(tuple.getLong("field_i") == 60);
// Test with a predicate.
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
-
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
++ "stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 2);
@@@ -1160,8 -1457,8 +1239,6 @@@
tuple = tuples.get(1);
assert(tuple.get("str_s").equals("a"));
assert(tuple.getLong("field_i") == 20);
--
--
} finally {
delete();
}
@@@ -1169,7 -1466,7 +1246,6 @@@
private void testParallelSelectDistinct() throws Exception {
try {
--
CloudJettyRunner jetty = this.cloudJettys.get(0);
del("*:*");
@@@ -1317,12 -1608,11 +1385,10 @@@
// Test without a sort. Sort should be asc by default.
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select distinct str_s, field_i from collection1");
-
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select distinct str_s, field_i from collection1");
++ "stmt", "select distinct str_s, field_i from collection1");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 6);
@@@ -1352,12 -1642,11 +1418,10 @@@
assert(tuple.getLong("field_i") == 60);
// Test with a predicate.
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
-
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
++ "stmt", "select distinct str_s, field_i from collection1 where str_s = 'a'");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 2);
@@@ -1375,11 -1664,11 +1439,8 @@@
}
}
--
--
private void testBasicGroupingFacets() throws Exception {
try {
--
CloudJettyRunner jetty = this.cloudJettys.get(0);
del("*:*");
@@@ -1395,14 -1684,10 +1456,13 @@@
indexr("id", "7", "text", "XXXX XXXX", "str_s", "c", "field_i", "50");
indexr("id", "8", "text", "XXXX XXXX", "str_s", "c", "field_i", "60");
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
++
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select 'str_s', 'count(*)', sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by 'str_s' order by 'sum(field_i)' asc limit 2");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "order by sum(field_i) asc limit 2");
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
//Only two results because of the limit.
@@@ -1420,22 -1705,16 +1480,20 @@@
tuple = tuples.get(1);
assert(tuple.get("str_s").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
-
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 27); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
+
+ // TODO fix test - Cannot apply 'NOT' to arguments of type 'NOT<JAVATYPE(CLASS JAVA.LANG.STRING)>'. Supported form(s): 'NOT<BOOLEAN>'
+ /*
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), "
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where (text='XXXX' AND NOT text='XXXX XXX') group by str_s order by str_s desc");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), "
+ + "cast(avg(1.0 * field_i) as float) from collection1 where (text='XXXX' AND NOT (text='XXXX XXX')) "
+ + "group by str_s order by str_s desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
//The sort by and order by match and no limit is applied. All the Tuples should be returned in
@@@ -1461,21 -1740,16 +1519,19 @@@
tuple = tuples.get(2);
assert(tuple.get("str_s").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 27); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
+ // TODO fix test - Cannot apply 'NOT' to arguments of type 'NOT<JAVATYPE(CLASS JAVA.LANG.STRING)>'. Supported form(s): 'NOT<BOOLEAN>'
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select str_s as myString, count(*), sum(field_i) as mySum, min(field_i), max(field_i), "
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select str_s as myString, count(*), sum(field_i) as sum, min(field_i), max(field_i), avg(field_i) from collection1 where (text='XXXX' AND NOT text='XXXX XXX') group by myString order by myString desc");
++ "stmt", "select str_s as myString, count(*), sum(field_i) as mySum, min(field_i), max(field_i), "
+ + "cast(avg(1.0 * field_i) as float) from collection1 where (text='XXXX' AND NOT (text='XXXX XXX')) "
+ + "group by str_s order by myString desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
//The sort by and order by match and no limit is applied. All the Tuples should be returned in
@@@ -1501,81 -1775,70 +1557,72 @@@
tuple = tuples.get(2);
assert(tuple.get("myString").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
-
-
-
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("mySum") == 27);
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
+ */
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having 'sum(field_i)' = 19");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s having sum(field_i) = 19");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 1);
tuple = tuples.get(0);
assert(tuple.get("str_s").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 19); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 8); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 11); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 9.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having (('sum(field_i)' = 19) AND (min(field_i) = 8))");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "having ((sum(field_i) = 19) AND (min(field_i) = 8))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 1);
tuple = tuples.get(0);
assert(tuple.get("str_s").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
-
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 19); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 8); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 11); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 9.5D); //avg(field_i)
-
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select str_s, count(*), sum(field_i) as mySum, min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select str_s myString, count(*), sum(field_i) as sum, min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by myString having ((sum = 19) AND (min(field_i) = 8))");
++ "stmt", "select str_s, count(*), sum(field_i) as mySum, min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "having ((sum(field_i) = 19) AND (min(field_i) = 8))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 1);
tuple = tuples.get(0);
- assert(tuple.get("myString").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
+ assert(tuple.get("str_s").equals("b"));
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("mySum") == 19);
+ assert(tuple.getDouble("EXPR$3") == 8); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 11); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 9.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having ((sum(field_i) = 19) AND (min(field_i) = 100))");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "having ((sum(field_i) = 19) AND (min(field_i) = 100))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 0);
@@@ -1601,14 -1871,10 +1648,13 @@@
indexr("id", "7", "text", "XXXX XXXX", "str_s", "c", "field_i", "50");
indexr("id", "8", "text", "XXXX XXXX", "str_s", "c", "field_i", "60");
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
++
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s order by sum(field_i) asc limit 2");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "order by sum(field_i) asc limit 2");
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
//Only two results because of the limit.
@@@ -1626,20 -1892,16 +1672,18 @@@
tuple = tuples.get(1);
assert(tuple.get("str_s").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 27); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
+
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select str_s, count(*), sum(field_i) as mySum, min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select str_s, count(*), sum(field_i) as sum, min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s order by sum asc limit 2");
++ "stmt", "select str_s, count(*), sum(field_i) as mySum, min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s order by mySum asc limit 2");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
//Only two results because of the limit.
@@@ -1655,20 -1917,17 +1699,18 @@@
tuple = tuples.get(1);
assert(tuple.get("str_s").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("mySum") == 27);
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s order by str_s desc");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s order by str_s desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
//The sort by and order by match and no limit is applied. All the Tuples should be returned in
@@@ -1694,20 -1953,17 +1736,18 @@@
tuple = tuples.get(2);
assert(tuple.get("str_s").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 27); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select str_s as myString, count(*), sum(field_i), min(field_i), max(field_i), " +
- sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select str_s as myString, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by myString order by myString desc");
++ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
++ "stmt", "select str_s as myString, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s order by myString desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
//The sort by and order by match and no limit is applied. All the Tuples should be returned in
@@@ -1733,64 -1989,63 +1773,57 @@@
tuple = tuples.get(2);
assert(tuple.get("myString").equals("a"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 27);
- assert(tuple.getDouble("min(field_i)") == 7);
- assert(tuple.getDouble("max(field_i)") == 20);
- assert(tuple.getDouble("avg(field_i)") == 13.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 27); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 7); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 20); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 13.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having sum(field_i) = 19");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s having sum(field_i) = 19");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 1);
tuple = tuples.get(0);
assert(tuple.get("str_s").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
-
- tuple = tuples.get(0);
- assert(tuple.get("str_s").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 19); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 8); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 11); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 9.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having ((sum(field_i) = 19) AND (min(field_i) = 8))");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "having ((sum(field_i) = 19) AND (min(field_i) = 8))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 1);
tuple = tuples.get(0);
assert(tuple.get("str_s").equals("b"));
- assert(tuple.getDouble("count(*)") == 2);
- assert(tuple.getDouble("sum(field_i)") == 19);
- assert(tuple.getDouble("min(field_i)") == 8);
- assert(tuple.getDouble("max(field_i)") == 11);
- assert(tuple.getDouble("avg(field_i)") == 9.5D);
+ assert(tuple.getDouble("EXPR$1") == 2); //count(*)
+ assert(tuple.getDouble("EXPR$2") == 19); //sum(field_i)
+ assert(tuple.getDouble("EXPR$3") == 8); //min(field_i)
+ assert(tuple.getDouble("EXPR$4") == 11); //max(field_i)
+ assert(tuple.getDouble("EXPR$5") == 9.5D); //avg(field_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", "2");
- params.put("stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from collection1 where text='XXXX' group by str_s having ((sum(field_i) = 19) AND (min(field_i) = 100))");
++ "stmt", "select str_s, count(*), sum(field_i), min(field_i), max(field_i), " +
+ "cast(avg(1.0 * field_i) as float) from collection1 where text='XXXX' group by str_s " +
+ "having ((sum(field_i) = 19) AND (min(field_i) = 100))");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
assert(tuples.size() == 0);
--
} finally {
delete();
}
@@@ -1798,7 -2053,7 +1831,6 @@@
private void testAggregatesWithoutGrouping() throws Exception {
--
CloudJettyRunner jetty = this.cloudJettys.get(0);
del("*:*");
@@@ -1818,13 -2073,11 +1850,11 @@@
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select count(*), sum(a_i), min(a_i), max(a_i), cast(avg(1.0 * a_i) as float), sum(a_f), " +
- SolrParams sParams = mapParams(CommonParams.QT, "/sql", "stmt", "select count(*), sum(a_i), min(a_i), max(a_i), avg(a_i), sum(a_f), min(a_f), max(a_f), avg(a_f) from collection1");
++ SolrParams sParams = mapParams(CommonParams.QT, "/sql", "stmt", "select count(*), sum(a_i), min(a_i), max(a_i), cast(avg(1.0 * a_i) as float), sum(a_f), " +
+ "min(a_f), max(a_f), avg(a_f) from collection1");
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
-
List<Tuple> tuples = getTuples(solrStream);
assert(tuples.size() == 1);
@@@ -1833,33 -2086,32 +1863,31 @@@
Tuple tuple = tuples.get(0);
- Double sumi = tuple.getDouble("sum(a_i)");
- Double sumf = tuple.getDouble("sum(a_f)");
- Double mini = tuple.getDouble("min(a_i)");
- Double minf = tuple.getDouble("min(a_f)");
- Double maxi = tuple.getDouble("max(a_i)");
- Double maxf = tuple.getDouble("max(a_f)");
- Double avgi = tuple.getDouble("avg(a_i)");
- Double avgf = tuple.getDouble("avg(a_f)");
- Double count = tuple.getDouble("count(*)");
-
- assertTrue(sumi.longValue() == 70);
- assertTrue(sumf.doubleValue() == 55.0D);
- assertTrue(mini.doubleValue() == 0.0D);
- assertTrue(minf.doubleValue() == 1.0D);
- assertTrue(maxi.doubleValue() == 14.0D);
- assertTrue(maxf.doubleValue() == 10.0D);
- assertTrue(avgi.doubleValue() == 7.0D);
- assertTrue(avgf.doubleValue() == 5.5D);
- assertTrue(count.doubleValue() == 10);
-
-
+ Double count = tuple.getDouble("EXPR$0"); //count(*)
+ Double sumi = tuple.getDouble("EXPR$1"); //sum(a_i)
+ Double mini = tuple.getDouble("EXPR$2"); //min(a_i)
+ Double maxi = tuple.getDouble("EXPR$3"); //max(a_i)
+ Double avgi = tuple.getDouble("EXPR$4"); //avg(a_i)
+ Double sumf = tuple.getDouble("EXPR$5"); //sum(a_f)
+ Double minf = tuple.getDouble("EXPR$6"); //min(a_f)
+ Double maxf = tuple.getDouble("EXPR$7"); //max(a_f)
+ Double avgf = tuple.getDouble("EXPR$8"); //avg(a_f)
+
+ assertTrue(count == 10);
+ assertTrue(sumi == 70);
+ assertTrue(mini == 0.0D);
+ assertTrue(maxi == 14.0D);
+ assertTrue(avgi == 7.0D);
+ assertTrue(sumf == 55.0D);
+ assertTrue(minf == 1.0D);
+ assertTrue(maxf == 10.0D);
+ assertTrue(avgf == 5.5D);
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select count(*) as myCount, sum(a_i) as mySum, min(a_i) as myMin, max(a_i) as myMax, " +
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select count(*) as count, sum(a_i) as sum, min(a_i) as min, max(a_i) as max, avg(a_i) as avg, sum(a_f), min(a_f), max(a_f), avg(a_f) from collection1");
++ "stmt", "select count(*) as myCount, sum(a_i) as mySum, min(a_i) as myMin, max(a_i) as myMax, " +
+ "cast(avg(1.0 * a_i) as float) as myAvg, sum(a_f), min(a_f), max(a_f), avg(a_f) from collection1");
- solrStream = new SolrStream(jetty.url, params);
-
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
@@@ -1869,33 -2121,35 +1897,32 @@@
tuple = tuples.get(0);
- sumi = tuple.getDouble("sum");
- sumf = tuple.getDouble("sum(a_f)");
- mini = tuple.getDouble("min");
- minf = tuple.getDouble("min(a_f)");
- maxi = tuple.getDouble("max");
- maxf = tuple.getDouble("max(a_f)");
- avgi = tuple.getDouble("avg");
- avgf = tuple.getDouble("avg(a_f)");
- count = tuple.getDouble("count");
-
- assertTrue(sumi.longValue() == 70);
- assertTrue(sumf.doubleValue() == 55.0D);
- assertTrue(mini.doubleValue() == 0.0D);
- assertTrue(minf.doubleValue() == 1.0D);
- assertTrue(maxi.doubleValue() == 14.0D);
- assertTrue(maxf.doubleValue() == 10.0D);
- assertTrue(avgi.doubleValue() == 7.0D);
- assertTrue(avgf.doubleValue() == 5.5D);
- assertTrue(count.doubleValue() == 10);
-
-
-
+ count = tuple.getDouble("myCount");
+ sumi = tuple.getDouble("mySum");
+ mini = tuple.getDouble("myMin");
+ maxi = tuple.getDouble("myMax");
+ avgi = tuple.getDouble("myAvg");
+ sumf = tuple.getDouble("EXPR$5"); //sum(a_f)
+ minf = tuple.getDouble("EXPR$6"); //min(a_f)
+ maxf = tuple.getDouble("EXPR$7"); //max(a_f)
+ avgf = tuple.getDouble("EXPR$8"); //avg(a_f)
+
+ assertTrue(count == 10);
+ assertTrue(mini == 0.0D);
+ assertTrue(maxi == 14.0D);
+ assertTrue(sumi == 70);
+ assertTrue(avgi == 7.0D);
+ assertTrue(sumf == 55.0D);
+ assertTrue(minf == 1.0D);
+ assertTrue(maxf == 10.0D);
+ assertTrue(avgf == 5.5D);
// Test where clause hits
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select count(*), sum(a_i), min(a_i), max(a_i), cast(avg(1.0 * a_i) as float), sum(a_f), " +
-
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select count(*), sum(a_i), min(a_i), max(a_i), avg(a_i), sum(a_f), min(a_f), max(a_f), avg(a_f) from collection1 where id = 2");
++ "stmt", "select count(*), sum(a_i), min(a_i), max(a_i), cast(avg(1.0 * a_i) as float), sum(a_f), " +
+ "min(a_f), max(a_f), avg(a_f) from collection1 where id = 2");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
@@@ -1925,12 -2179,11 +1952,11 @@@
// Test zero hits
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select count(*), sum(a_i), min(a_i), max(a_i), cast(avg(1.0 * a_i) as float), sum(a_f), " +
-
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select count(*), sum(a_i), min(a_i), max(a_i), avg(a_i), sum(a_f), min(a_f), max(a_f), avg(a_f) from collection1 where a_s = 'blah'");
++ "stmt", "select count(*), sum(a_i), min(a_i), max(a_i), cast(avg(1.0 * a_i) as float), sum(a_f), " +
+ "min(a_f), max(a_f), avg(a_f) from collection1 where a_s = 'blah'");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
@@@ -1962,9 -2215,12 +1988,8 @@@
commit();
}
-
-
-
private void testTimeSeriesGrouping() throws Exception {
try {
--
CloudJettyRunner jetty = this.cloudJettys.get(0);
del("*:*");
@@@ -1981,14 -2237,13 +2006,13 @@@
indexr("id", "8", "year_i", "2014", "month_i", "4", "day_i", "2", "item_i", "1");
commit();
+
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select year_i, sum(item_i) from collection1 group by year_i order by year_i desc");
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select year_i, sum(item_i) from collection1 group by year_i order by year_i desc");
++ "stmt", "select year_i, sum(item_i) from collection1 group by year_i order by year_i desc");
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 2);
Tuple tuple = null;
@@@ -1999,14 -2254,15 +2023,15 @@@
tuple = tuples.get(1);
assert(tuple.getLong("year_i") == 2014);
- assert(tuple.getDouble("sum(item_i)") == 7);
+ assert(tuple.getDouble("EXPR$1") == 7); //sum(item_i)
- params.put("stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i " +
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i order by year_i desc, month_i desc");
++ "stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i " +
+ "order by year_i desc, month_i desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 3);
tuple = null;
@@@ -2024,16 -2280,15 +2049,15 @@@
tuple = tuples.get(2);
assert(tuple.getLong("year_i") == 2014);
assert(tuple.getLong("month_i") == 4);
- assert(tuple.getDouble("sum(item_i)") == 7);
+ assert(tuple.getDouble("EXPR$2") == 7); //sum(item_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i " +
+ sParams = mapParams(CommonParams.QT, "/sql",
- "stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i order by year_i desc, month_i desc, day_i desc");
++ "stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i " +
+ "order by year_i desc, month_i desc, day_i desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 6);
tuple = null;
@@@ -2099,14 -2354,13 +2123,12 @@@
indexr("id", "8", "year_i", "2014", "month_i", "4", "day_i", "2", "item_i", "1");
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select year_i, sum(item_i) from collection1 group by year_i order by year_i desc");
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
+ "stmt", "select year_i, sum(item_i) from collection1 group by year_i order by year_i desc");
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 2);
Tuple tuple = null;
@@@ -2117,18 -2371,16 +2139,16 @@@
tuple = tuples.get(1);
assert(tuple.getLong("year_i") == 2014);
- assert(tuple.getDouble("sum(item_i)") == 7);
+ assert(tuple.getDouble("EXPR$1") == 7); //sum(item_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i " +
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i order by year_i desc, month_i desc");
++ "stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i " +
+ "order by year_i desc, month_i desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 3);
tuple = tuples.get(0);
@@@ -2144,18 -2396,16 +2164,16 @@@
tuple = tuples.get(2);
assert(tuple.getLong("year_i") == 2014);
assert(tuple.getLong("month_i") == 4);
- assert(tuple.getDouble("sum(item_i)") == 7);
+ assert(tuple.getDouble("EXPR$2") == 7); //sum(item_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("aggregationMode", "facet");
- params.put("stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i " +
+ sParams = mapParams(CommonParams.QT, "/sql", "aggregationMode", "facet",
- "stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i order by year_i desc, month_i desc, day_i desc");
++ "stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i " +
+ "order by year_i desc, month_i desc, day_i desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 6);
tuple = tuples.get(0);
@@@ -2200,7 -2450,7 +2218,6 @@@
private void testParallelTimeSeriesGrouping() throws Exception {
try {
--
CloudJettyRunner jetty = this.cloudJettys.get(0);
del("*:*");
@@@ -2217,14 -2467,13 +2234,12 @@@
indexr("id", "8", "year_i", "2014", "month_i", "4", "day_i", "2", "item_i", "1");
commit();
- Map params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", 2);
- params.put("stmt", "select year_i, sum(item_i) from collection1 group by year_i order by year_i desc");
+ SolrParams sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
+ "stmt", "select year_i, sum(item_i) from collection1 group by year_i order by year_i desc");
- SolrStream solrStream = new SolrStream(jetty.url, params);
+ SolrStream solrStream = new SolrStream(jetty.url, sParams);
List<Tuple> tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 2);
Tuple tuple = null;
@@@ -2236,17 -2485,15 +2251,15 @@@
tuple = tuples.get(1);
assert(tuple.getLong("year_i") == 2014);
- assert(tuple.getDouble("sum(item_i)") == 7);
+ assert(tuple.getDouble("EXPR$1") == 7); //sum(item_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", 2);
- params.put("stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i " +
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i order by year_i desc, month_i desc");
++ "stmt", "select year_i, month_i, sum(item_i) from collection1 group by year_i, month_i " +
+ "order by year_i desc, month_i desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 3);
tuple = null;
@@@ -2266,18 -2513,16 +2279,16 @@@
tuple = tuples.get(2);
assert(tuple.getLong("year_i") == 2014);
assert(tuple.getLong("month_i") == 4);
- assert(tuple.getDouble("sum(item_i)") == 7);
+ assert(tuple.getDouble("EXPR$2") == 7); //sum(item_i)
- params = new HashMap();
- params.put(CommonParams.QT, "/sql");
- params.put("numWorkers", 2);
- params.put("stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i " +
+ sParams = mapParams(CommonParams.QT, "/sql", "numWorkers", "2",
- "stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i order by year_i desc, month_i desc, day_i desc");
++ "stmt", "select year_i, month_i, day_i, sum(item_i) from collection1 group by year_i, month_i, day_i " +
+ "order by year_i desc, month_i desc, day_i desc");
- solrStream = new SolrStream(jetty.url, params);
+ solrStream = new SolrStream(jetty.url, sParams);
tuples = getTuples(solrStream);
- //Only two results because of the limit.
assert(tuples.size() == 6);
tuple = null;
[26/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7489: Wrap only
once in case GCD compression is used.
Posted by kr...@apache.org.
LUCENE-7489: Wrap only once in case GCD compression is used.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a17e9200
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a17e9200
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a17e9200
Branch: refs/heads/jira/solr-8593
Commit: a17e92006f087a0601d9329bf9b9c946ca72478b
Parents: 739981b
Author: Adrien Grand <jp...@gmail.com>
Authored: Tue Oct 18 16:07:52 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Tue Oct 18 16:08:29 2016 +0200
----------------------------------------------------------------------
.../lucene70/Lucene70DocValuesProducer.java | 63 +++++++++-----------
1 file changed, 27 insertions(+), 36 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a17e9200/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java
index 755da79..637c8ee 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java
@@ -424,47 +424,38 @@ final class Lucene70DocValuesProducer extends DocValuesProducer implements Close
};
} else {
final RandomAccessInput slice = data.randomAccessSlice(entry.valuesOffset, entry.valuesLength);
- LongValues values = DirectReader.getInstance(slice, entry.bitsPerValue);
- if (entry.gcd != 1) {
- values = applyGcd(values, entry.gcd);
- }
- if (entry.minValue != 0) {
- values = applyDelta(values, entry.minValue);
- }
+ final LongValues values = DirectReader.getInstance(slice, entry.bitsPerValue);
if (entry.table != null) {
- values = applyTable(values, entry.table);
+ final long[] table = entry.table;
+ return new LongValues() {
+ @Override
+ public long get(long index) {
+ return table[(int) values.get(index)];
+ }
+ };
+ } else if (entry.gcd != 1) {
+ final long gcd = entry.gcd;
+ final long minValue = entry.minValue;
+ return new LongValues() {
+ @Override
+ public long get(long index) {
+ return values.get(index) * gcd + minValue;
+ }
+ };
+ } else if (entry.minValue != 0) {
+ final long minValue = entry.minValue;
+ return new LongValues() {
+ @Override
+ public long get(long index) {
+ return values.get(index) + minValue;
+ }
+ };
+ } else {
+ return values;
}
- return values;
}
}
- private LongValues applyDelta(LongValues values, long delta) {
- return new LongValues() {
- @Override
- public long get(long index) {
- return delta + values.get(index);
- }
- };
- }
-
- private LongValues applyGcd(LongValues values, long gcd) {
- return new LongValues() {
- @Override
- public long get(long index) {
- return values.get(index) * gcd;
- }
- };
- }
-
- private LongValues applyTable(LongValues values, long[] table) {
- return new LongValues() {
- @Override
- public long get(long index) {
- return table[(int) values.get(index)];
- }
- };
- }
-
@Override
public BinaryDocValues getBinary(FieldInfo field) throws IOException {
BinaryEntry entry = binaries.get(field.name);
[19/50] [abbrv] lucene-solr:jira/solr-8593: remove unused imports
Posted by kr...@apache.org.
remove unused imports
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/f870c206
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/f870c206
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/f870c206
Branch: refs/heads/jira/solr-8593
Commit: f870c2069c49c1908c3fd0f26e7323711f5cf978
Parents: 4498cc7
Author: Mike McCandless <mi...@apache.org>
Authored: Tue Oct 18 06:48:50 2016 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Tue Oct 18 06:48:50 2016 -0400
----------------------------------------------------------------------
.../org/apache/lucene/analysis/core/TestDecimalDigitFilter.java | 2 --
1 file changed, 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f870c206/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java
----------------------------------------------------------------------
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java
index e5e18ef..e6af49d 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java
@@ -22,8 +22,6 @@ import org.apache.lucene.analysis.BaseTokenStreamTestCase;
import org.apache.lucene.analysis.MockTokenizer;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.search.DocIdSetIterator;
-import org.apache.lucene.util.ArrayUtil;
-import org.apache.lucene.util.Bits;
import org.apache.lucene.util.SparseFixedBitSet;
import org.apache.lucene.util.TestUtil;
[18/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-6914: Fix issue ID
in the change log.
Posted by kr...@apache.org.
LUCENE-6914: Fix issue ID in the change log.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/4498cc72
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/4498cc72
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/4498cc72
Branch: refs/heads/jira/solr-8593
Commit: 4498cc727f5a2f2db5ea8683b36a821b9b529ebb
Parents: 9260a13
Author: Adrien Grand <jp...@gmail.com>
Authored: Tue Oct 18 10:54:30 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Tue Oct 18 10:54:30 2016 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/4498cc72/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 810b529..f3501cc 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -86,7 +86,7 @@ Bug Fixes
dimensional points could cause unexpected merge exceptions (Hans
Lund, Mike McCandless)
-* LUCENE-7363: Fixed DecimalDigitFilter in case of supplementary code points.
+* LUCENE-6914: Fixed DecimalDigitFilter in case of supplementary code points.
(Hossman)
Improvements
[06/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9625: Add
HelloWorldSolrCloudTestCase class (Christine Poerschke, Alan Woodward,
Alexandre Rafalovitch)
Posted by kr...@apache.org.
SOLR-9625: Add HelloWorldSolrCloudTestCase class (Christine Poerschke, Alan Woodward, Alexandre Rafalovitch)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5261eb0a
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5261eb0a
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5261eb0a
Branch: refs/heads/jira/solr-8593
Commit: 5261eb0acd54848a5aafb542990d7391d77f94c1
Parents: 33db4de
Author: Christine Poerschke <cp...@apache.org>
Authored: Sat Oct 15 09:44:37 2016 -0400
Committer: Christine Poerschke <cp...@apache.org>
Committed: Sat Oct 15 09:44:37 2016 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +
.../solr/HelloWorldSolrCloudTestCase.java | 94 ++++++++++++++++++++
2 files changed, 96 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5261eb0a/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 2bee12e..4f8d682 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -275,6 +275,8 @@ Other Changes
* SOLR-9639: Test only fix. Prevent CDCR tests from removing collection during recovery that used to blow up jvm (Mikhail Khludnev)
+* SOLR-9625: Add HelloWorldSolrCloudTestCase class (Christine Poerschke, Alan Woodward, Alexandre Rafalovitch)
+
================== 6.2.1 ==================
Bug Fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5261eb0a/solr/core/src/test/org/apache/solr/HelloWorldSolrCloudTestCase.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/HelloWorldSolrCloudTestCase.java b/solr/core/src/test/org/apache/solr/HelloWorldSolrCloudTestCase.java
new file mode 100644
index 0000000..56a813c
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/HelloWorldSolrCloudTestCase.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr;
+
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.SolrInputDocument;
+
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * How to use this test class:
+ * #1 Run the test, e.g.
+ * in Eclipse 'Run As JUnit Test' or
+ * on the command line: cd solr/core ; ant test -Dtestcase=HelloWorldSolrCloudTestCase
+ * #2 Modify the test, e.g.
+ * in setupCluster add further documents and then re-run the test.
+ */
+public class HelloWorldSolrCloudTestCase extends SolrCloudTestCase {
+
+ private static final String COLLECTION = "hello_world" ;
+
+ private static final int numShards = 3;
+ private static final int numReplicas = 2;
+ private static final int maxShardsPerNode = 2;
+ private static final int nodeCount = (numShards*numReplicas + (maxShardsPerNode-1))/maxShardsPerNode;
+
+ private static final String id = "id";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+
+ // create and configure cluster
+ configureCluster(nodeCount)
+ .addConfig("conf", configset("cloud-dynamic"))
+ .configure();
+
+ // create an empty collection
+ CollectionAdminRequest.createCollection(COLLECTION, "conf", numShards, numReplicas)
+ .setMaxShardsPerNode(maxShardsPerNode)
+ .process(cluster.getSolrClient());
+
+ // add a document
+ final SolrInputDocument doc1 = sdoc(id, "1",
+ "title_s", "Here comes the sun",
+ "artist_s", "The Beatles",
+ "popularity_i", "123");
+ new UpdateRequest()
+ .add(doc1)
+ .commit(cluster.getSolrClient(), COLLECTION);
+
+ // add further document(s) here
+ // TODO
+ }
+
+ @Test
+ public void testHighestScoring() throws Exception {
+ final SolrQuery solrQuery = new SolrQuery("q", "*:*", "fl", "id,popularity_i", "sort", "popularity_i desc", "rows", "1");
+ final CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+ final QueryResponse rsp = cloudSolrClient.query(COLLECTION, solrQuery);
+ assertEquals(1, rsp.getResults().size());
+ assertEquals("1", rsp.getResults().get(0).getFieldValue(id));
+ }
+
+ @Test
+ public void testLowestScoring() throws Exception {
+ final SolrQuery solrQuery = new SolrQuery("q", "*:*", "fl", "id,popularity_i", "sort", "popularity_i asc", "rows", "1");
+ final CloudSolrClient cloudSolrClient = cluster.getSolrClient();
+ final QueryResponse rsp = cloudSolrClient.query(COLLECTION, solrQuery);
+ assertEquals(1, rsp.getResults().size());
+ assertEquals("1", rsp.getResults().get(0).getFieldValue(id));
+ }
+
+}
+
[28/50] [abbrv] lucene-solr:jira/solr-8593: Merge remote-tracking
branch 'origin/master'
Posted by kr...@apache.org.
Merge remote-tracking branch 'origin/master'
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/fe17b4e2
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/fe17b4e2
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/fe17b4e2
Branch: refs/heads/jira/solr-8593
Commit: fe17b4e2bb9c90bd3c39b01fe516fbe52d5ec914
Parents: bb907a2 a17e920
Author: Noble Paul <no...@gmail.com>
Authored: Tue Oct 18 19:45:06 2016 +0530
Committer: Noble Paul <no...@gmail.com>
Committed: Tue Oct 18 19:45:06 2016 +0530
----------------------------------------------------------------------
lucene/CHANGES.txt | 6 ++
.../simpletext/SimpleTextPointsReader.java | 17 +++++-
.../lucene70/Lucene70DocValuesProducer.java | 63 +++++++++-----------
.../org/apache/lucene/util/bkd/BKDReader.java | 32 +++++-----
.../org/apache/lucene/util/bkd/BKDWriter.java | 14 +++--
.../apache/lucene/facet/FacetsCollector.java | 42 ++++++++-----
.../apache/lucene/facet/TestDrillDownQuery.java | 11 ++++
7 files changed, 112 insertions(+), 73 deletions(-)
----------------------------------------------------------------------
[34/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9657: New
TemplateUpdateProcessorFactory added
Posted by kr...@apache.org.
SOLR-9657: New TemplateUpdateProcessorFactory added
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/c2e031ad
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/c2e031ad
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/c2e031ad
Branch: refs/heads/jira/solr-8593
Commit: c2e031add3d5db2c4e89a5a92afd7bb8cc1f481f
Parents: 53129ba
Author: Noble Paul <no...@gmail.com>
Authored: Wed Oct 19 11:04:10 2016 +0530
Committer: Noble Paul <no...@gmail.com>
Committed: Wed Oct 19 11:04:10 2016 +0530
----------------------------------------------------------------------
solr/CHANGES.txt | 3 +
.../handler/dataimport/TemplateTransformer.java | 5 +-
.../handler/dataimport/VariableResolver.java | 66 +++--------
.../src/java/org/apache/solr/core/SolrCore.java | 2 +-
.../processor/SimpleUpdateProcessorFactory.java | 45 +++++++-
.../TemplateUpdateProcessorFactory.java | 110 +++++++++++++++++++
.../processor/UpdateRequestProcessorChain.java | 14 ++-
.../processor/TemplateUpdateProcessorTest.java | 48 ++++++++
.../UpdateRequestProcessorFactoryTest.java | 15 +++
9 files changed, 247 insertions(+), 61 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index be958d9..880718c 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -135,6 +135,9 @@ New Features
* SOLR-9103: Restore ability for users to add custom Streaming Expressions (Cao Manh Dat)
+* SOLR-9657: New TemplateUpdateProcessorFactory added (noble)
+
+
Bug Fixes
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/TemplateTransformer.java
----------------------------------------------------------------------
diff --git a/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/TemplateTransformer.java b/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/TemplateTransformer.java
index 6bd6050..a5faa7e 100644
--- a/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/TemplateTransformer.java
+++ b/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/TemplateTransformer.java
@@ -56,12 +56,11 @@ public class TemplateTransformer extends Transformer {
public Object transformRow(Map<String, Object> row, Context context) {
- VariableResolver resolver = (VariableResolver) context
- .getVariableResolver();
+ VariableResolver resolver = context.getVariableResolver();
// Add current row to the copy of resolver map
-// for (Map.Entry<String, Object> entry : row.entrySet())
for (Map<String, String> map : context.getAllEntityFields()) {
+ map.entrySet();
String expr = map.get(TEMPLATE);
if (expr == null)
continue;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/VariableResolver.java
----------------------------------------------------------------------
diff --git a/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/VariableResolver.java b/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/VariableResolver.java
index 76930e2..f255657 100644
--- a/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/VariableResolver.java
+++ b/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/VariableResolver.java
@@ -16,16 +16,19 @@
*/
package org.apache.solr.handler.dataimport;
-import java.util.ArrayList;
-import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.WeakHashMap;
+import java.util.function.Function;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.apache.solr.update.processor.TemplateUpdateProcessorFactory;
+
+import static org.apache.solr.update.processor.TemplateUpdateProcessorFactory.Resolved;
+
/**
* <p>
* A set of nested maps that can resolve variables by namespaces. Variables are
@@ -48,20 +51,13 @@ import java.util.regex.Pattern;
public class VariableResolver {
private static final Pattern DOT_PATTERN = Pattern.compile("[.]");
- private static final Pattern PLACEHOLDER_PATTERN = Pattern
- .compile("[$][{](.*?)[}]");
private static final Pattern EVALUATOR_FORMAT_PATTERN = Pattern
.compile("^(\\w*?)\\((.*?)\\)$");
private Map<String,Object> rootNamespace;
private Map<String,Evaluator> evaluators;
private Map<String,Resolved> cache = new WeakHashMap<>();
-
- class Resolved {
- List<Integer> startIndexes = new ArrayList<>(2);
- List<Integer> endOffsets = new ArrayList<>(2);
- List<String> variables = new ArrayList<>(2);
- }
-
+ private Function<String,Object> fun = this::resolve;
+
public static final String FUNCTIONS_NAMESPACE = "dataimporter.functions.";
public static final String FUNCTIONS_NAMESPACE_SHORT = "dih.functions.";
@@ -145,48 +141,8 @@ public class VariableResolver {
* @return the string with the placeholders replaced with their values
*/
public String replaceTokens(String template) {
- if (template == null) {
- return null;
- }
- Resolved r = getResolved(template);
- if (r.startIndexes != null) {
- StringBuilder sb = new StringBuilder(template);
- for (int i = r.startIndexes.size() - 1; i >= 0; i--) {
- String replacement = resolve(r.variables.get(i)).toString();
- sb.replace(r.startIndexes.get(i), r.endOffsets.get(i), replacement);
- }
- return sb.toString();
- } else {
- return template;
- }
+ return TemplateUpdateProcessorFactory.replaceTokens(template, cache, fun);
}
-
- private Resolved getResolved(String template) {
- Resolved r = cache.get(template);
- if (r == null) {
- r = new Resolved();
- Matcher m = PLACEHOLDER_PATTERN.matcher(template);
- while (m.find()) {
- String variable = m.group(1);
- r.startIndexes.add(m.start(0));
- r.endOffsets.add(m.end(0));
- r.variables.add(variable);
- }
- cache.put(template, r);
- }
- return r;
- }
- /**
- * Get a list of variables embedded in the template string.
- */
- public List<String> getVariables(String template) {
- Resolved r = getResolved(template);
- if (r == null) {
- return Collections.emptyList();
- }
- return new ArrayList<>(r.variables);
- }
-
public void addNamespace(String name, Map<String,Object> newMap) {
if (newMap != null) {
if (name != null) {
@@ -204,7 +160,11 @@ public class VariableResolver {
}
}
}
-
+
+ public List<String> getVariables(String expr) {
+ return TemplateUpdateProcessorFactory.getVariables(expr, cache);
+ }
+
class CurrentLevel {
final Map<String,Object> map;
final int level;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/core/src/java/org/apache/solr/core/SolrCore.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/SolrCore.java b/solr/core/src/java/org/apache/solr/core/SolrCore.java
index 2827f03..7ba15af 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrCore.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrCore.java
@@ -184,7 +184,7 @@ public final class SolrCore implements SolrInfoMBean, Closeable {
private final long startNanoTime = System.nanoTime();
private final RequestHandlers reqHandlers;
private final PluginBag<SearchComponent> searchComponents = new PluginBag<>(SearchComponent.class, this);
- private final PluginBag<UpdateRequestProcessorFactory> updateProcessors = new PluginBag<>(UpdateRequestProcessorFactory.class, this);
+ private final PluginBag<UpdateRequestProcessorFactory> updateProcessors = new PluginBag<>(UpdateRequestProcessorFactory.class, this, true);
private final Map<String,UpdateRequestProcessorChain> updateProcessorChains;
private final Map<String, SolrInfoMBean> infoRegistry;
private final IndexDeletionPolicyWrapper solrDelPolicy;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java b/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java
index e9c5b2d..aec9d87 100644
--- a/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java
+++ b/solr/core/src/java/org/apache/solr/update/processor/SimpleUpdateProcessorFactory.java
@@ -18,6 +18,7 @@ package org.apache.solr.update.processor;
import java.io.IOException;
+import org.apache.solr.common.util.NamedList;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.update.AddUpdateCommand;
@@ -28,18 +29,60 @@ import org.apache.solr.update.AddUpdateCommand;
* This is deliberately made to support only the add operation
*/
public abstract class SimpleUpdateProcessorFactory extends UpdateRequestProcessorFactory {
+ protected final String myName;
+ protected NamedList initArgs = new NamedList();
+ private static ThreadLocal<SolrQueryRequest> REQ = new ThreadLocal<>();
+
+ protected SimpleUpdateProcessorFactory() {
+ String simpleName = this.getClass().getSimpleName();
+ this.myName = simpleName.substring(0, simpleName.indexOf("UpdateProcessorFactory"));
+ }
+
+ @Override
+ public void init(NamedList args) {
+ super.init(args);
+ this.initArgs = args;
+
+ }
@Override
public UpdateRequestProcessor getInstance(SolrQueryRequest req, SolrQueryResponse rsp, UpdateRequestProcessor next) {
return new UpdateRequestProcessor(next) {
@Override
public void processAdd(AddUpdateCommand cmd) throws IOException {
- process(cmd, req, rsp);
+ REQ.set(req);
+ try {
+ process(cmd, req, rsp);
+ } finally {
+ REQ.remove();
+ }
super.processAdd(cmd);
}
};
}
+ protected String getParam(String name) {
+ String[] v = getParams(name);
+ return v == null || v.length == 0 ? null : v[0];
+ }
+
+ /**returns value from init args or request parameter. the request parameter must have the
+ * URP shortname prefixed
+ */
+ protected String[] getParams(String name) {
+ Object v = REQ.get().getParams().getParams(_param(name));
+ if (v == null) v = initArgs.get(name);
+ if (v == null) return null;
+ if (v instanceof String[]) return (String[]) v;
+ return new String[]{v.toString()};
+
+ }
+
+ private String _param(String name) {
+ return myName + "." + name;
+ }
+
+
/**
* This object is reused across requests. So,this method should not store anything in the instance variable.
*/
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java b/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java
new file mode 100644
index 0000000..41d109b
--- /dev/null
+++ b/solr/core/src/java/org/apache/solr/update/processor/TemplateUpdateProcessorFactory.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.update.AddUpdateCommand;
+
+public class TemplateUpdateProcessorFactory extends SimpleUpdateProcessorFactory {
+ @Override
+ protected void process(AddUpdateCommand cmd, SolrQueryRequest req, SolrQueryResponse rsp) {
+ String[] vals = getParams("field");
+ SolrInputDocument doc = cmd.getSolrInputDocument();
+ if (vals != null && vals.length > 0) {
+ for (String val : vals) {
+ if (val == null || val.isEmpty()) continue;
+ int idx = val.indexOf(':');
+ if (idx == -1)
+ throw new RuntimeException("'field' must be of the format <field-name>:<the-template-string>");
+
+ String fName = val.substring(0, idx);
+ String template = val.substring(idx + 1);
+ doc.addField(fName, replaceTokens(template, null, s -> {
+ Object v = doc.getFieldValue(s);
+ return v == null ? "" : v;
+ }));
+ }
+ }
+
+ }
+
+
+ public static Resolved getResolved(String template, Map<String, Resolved> cache) {
+ Resolved r = cache == null ? null : cache.get(template);
+ if (r == null) {
+ r = new Resolved();
+ Matcher m = PLACEHOLDER_PATTERN.matcher(template);
+ while (m.find()) {
+ String variable = m.group(1);
+ r.startIndexes.add(m.start(0));
+ r.endOffsets.add(m.end(0));
+ r.variables.add(variable);
+ }
+ if (cache != null) cache.put(template, r);
+ }
+ return r;
+ }
+
+ /**
+ * Get a list of variables embedded in the template string.
+ */
+ public static List<String> getVariables(String template, Map<String, Resolved> cache) {
+ Resolved r = getResolved(template, cache);
+ if (r == null) {
+ return Collections.emptyList();
+ }
+ return new ArrayList<>(r.variables);
+ }
+
+ public static String replaceTokens(String template, Map<String, Resolved> cache, Function<String, Object> fun) {
+ if (template == null) {
+ return null;
+ }
+ Resolved r = getResolved(template, cache);
+ if (r.startIndexes != null) {
+ StringBuilder sb = new StringBuilder(template);
+ for (int i = r.startIndexes.size() - 1; i >= 0; i--) {
+ String replacement = fun.apply(r.variables.get(i)).toString();
+ sb.replace(r.startIndexes.get(i), r.endOffsets.get(i), replacement);
+ }
+ return sb.toString();
+ } else {
+ return template;
+ }
+ }
+
+
+ public static class Resolved {
+ public List<Integer> startIndexes = new ArrayList<>(2);
+ public List<Integer> endOffsets = new ArrayList<>(2);
+ public List<String> variables = new ArrayList<>(2);
+ }
+
+ public static final Pattern PLACEHOLDER_PATTERN = Pattern
+ .compile("[$][{](.*?)[}]");
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/core/src/java/org/apache/solr/update/processor/UpdateRequestProcessorChain.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/processor/UpdateRequestProcessorChain.java b/solr/core/src/java/org/apache/solr/update/processor/UpdateRequestProcessorChain.java
index e77dd82..0ed626c 100644
--- a/solr/core/src/java/org/apache/solr/update/processor/UpdateRequestProcessorChain.java
+++ b/solr/core/src/java/org/apache/solr/update/processor/UpdateRequestProcessorChain.java
@@ -101,7 +101,7 @@ public final class UpdateRequestProcessorChain implements PluginInfoInitialized
/**
* Initializes the chain using the factories specified by the <code>PluginInfo</code>.
- * if the chain includes the <code>RunUpdateProcessorFactory</code>, but
+ * if the chain includes the <code>RunUpdateProcessorFactory</code>, but
* does not include an implementation of the
* <code>DistributingUpdateProcessorFactory</code> interface, then an
* instance of <code>DistributedUpdateProcessorFactory</code> will be
@@ -269,8 +269,16 @@ public final class UpdateRequestProcessorChain implements PluginInfoInitialized
s = s.trim();
if (s.isEmpty()) continue;
UpdateRequestProcessorFactory p = core.getUpdateProcessors().get(s);
- if (p == null)
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No such processor " + s);
+ if (p == null) {
+ try {
+ p = core.createInstance(s + "UpdateProcessorFactory", UpdateRequestProcessorFactory.class,
+ "updateProcessor", null, core.getMemClassLoader());
+ core.getUpdateProcessors().put(s, p);
+ } catch (SolrException e) {
+ }
+ if (p == null)
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No such processor " + s);
+ }
result.add(p);
}
return result;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/core/src/test/org/apache/solr/update/processor/TemplateUpdateProcessorTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/update/processor/TemplateUpdateProcessorTest.java b/solr/core/src/test/org/apache/solr/update/processor/TemplateUpdateProcessorTest.java
new file mode 100644
index 0000000..7ee8a34
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/update/processor/TemplateUpdateProcessorTest.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.request.LocalSolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.update.AddUpdateCommand;
+
+public class TemplateUpdateProcessorTest extends SolrTestCaseJ4 {
+ public void testSimple() throws Exception {
+
+ AddUpdateCommand cmd = new AddUpdateCommand(new LocalSolrQueryRequest(null,
+ new ModifiableSolrParams()
+ .add("processor", "Template")
+ .add("Template.field", "id:${firstName}_${lastName}")
+ .add("Template.field", "another:${lastName}_${firstName}")
+ .add("Template.field", "missing:${lastName}_${unKnown}")
+
+ ));
+ cmd.solrDoc = new SolrInputDocument();
+ cmd.solrDoc.addField("firstName", "Tom");
+ cmd.solrDoc.addField("lastName", "Cruise");
+
+ new TemplateUpdateProcessorFactory().getInstance(cmd.getReq(), new SolrQueryResponse(), null).processAdd(cmd);
+ assertEquals("Tom_Cruise", cmd.solrDoc.getFieldValue("id"));
+ assertEquals("Cruise_Tom", cmd.solrDoc.getFieldValue("another"));
+ assertEquals("Cruise_", cmd.solrDoc.getFieldValue("missing"));
+
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c2e031ad/solr/core/src/test/org/apache/solr/update/processor/UpdateRequestProcessorFactoryTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/update/processor/UpdateRequestProcessorFactoryTest.java b/solr/core/src/test/org/apache/solr/update/processor/UpdateRequestProcessorFactoryTest.java
index e9dc93f..7ebefec 100644
--- a/solr/core/src/test/org/apache/solr/update/processor/UpdateRequestProcessorFactoryTest.java
+++ b/solr/core/src/test/org/apache/solr/update/processor/UpdateRequestProcessorFactoryTest.java
@@ -23,6 +23,7 @@ import java.util.Arrays;
import java.util.ArrayList;
import java.util.List;
+import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.core.SolrCore;
import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.util.AbstractSolrTestCase;
@@ -41,6 +42,20 @@ public class UpdateRequestProcessorFactoryTest extends AbstractSolrTestCase {
public static void beforeClass() throws Exception {
initCore("solrconfig-transformers.xml", "schema.xml");
}
+
+ public void testRequestTimeUrp(){
+ SolrCore core = h.getCore();
+ ModifiableSolrParams params = new ModifiableSolrParams()
+ .add("processor", "Template")
+ .add("Template.field", "id_t:${firstName}_${lastName}")
+ .add("Template.field", "another_t:${lastName}_${firstName}")
+ .add("Template.field", "missing_t:${lastName}_${unKnown}");
+ UpdateRequestProcessorChain chain = core.getUpdateProcessorChain(params);
+ List<UpdateRequestProcessorFactory> l = chain.getProcessors();
+ assertTrue(l.get(0) instanceof TemplateUpdateProcessorFactory);
+
+
+ }
public void testConfiguration() throws Exception
{
[12/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9642: Refactor the
snapshot cleanup mechanism to rely on Lucene
Posted by kr...@apache.org.
SOLR-9642: Refactor the snapshot cleanup mechanism to rely on Lucene
The current snapshot cleanup mechanism is based on reference counting
the index files shared between multiple segments. Since this mechanism
completely skips the Lucene APIs, it is not portable (e.g. it doesn't
work on 4.10.x version).
This patch provides an alternate implementation which relies exclusively
on Lucene IndexWriter (+ IndexDeletionPolicy) for cleanup.
mend
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/46aeb525
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/46aeb525
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/46aeb525
Branch: refs/heads/jira/solr-8593
Commit: 46aeb52588c6ecf298ee894a79fb162e4e3437fe
Parents: 65f5580
Author: Hrishikesh Gadre <hg...@cloudera.com>
Authored: Wed Aug 10 16:59:31 2016 -0700
Committer: yonik <yo...@apache.org>
Committed: Mon Oct 17 13:55:26 2016 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +
.../src/java/org/apache/solr/core/SolrCore.java | 83 ++++++++++++
.../core/snapshots/SolrSnapshotManager.java | 130 +++++++++----------
.../org/apache/solr/handler/IndexFetcher.java | 16 +--
.../org/apache/solr/handler/RestoreCore.java | 16 +--
.../solr/handler/admin/DeleteSnapshotOp.java | 37 ++----
.../org/apache/solr/update/SolrIndexWriter.java | 14 +-
.../core/snapshots/TestSolrCoreSnapshots.java | 127 ++----------------
8 files changed, 178 insertions(+), 247 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46aeb525/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 08abc76..0ea8191 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -279,6 +279,8 @@ Other Changes
* SOLR-9625: Add HelloWorldSolrCloudTestCase class (Christine Poerschke, Alan Woodward, Alexandre Rafalovitch)
+* SOLR-9642: Refactor the core level snapshot cleanup mechanism to rely on Lucene (Hrishikesh Gadre via yonik)
+
================== 6.2.1 ==================
Bug Fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46aeb525/solr/core/src/java/org/apache/solr/core/SolrCore.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/SolrCore.java b/solr/core/src/java/org/apache/solr/core/SolrCore.java
index dd422ad..2827f03 100644
--- a/solr/core/src/java/org/apache/solr/core/SolrCore.java
+++ b/solr/core/src/java/org/apache/solr/core/SolrCore.java
@@ -39,6 +39,7 @@ import java.util.LinkedHashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.Callable;
@@ -81,7 +82,9 @@ import org.apache.solr.common.util.ObjectReleaseTracker;
import org.apache.solr.common.util.SimpleOrderedMap;
import org.apache.solr.common.util.Utils;
import org.apache.solr.core.DirectoryFactory.DirContext;
+import org.apache.solr.core.snapshots.SolrSnapshotManager;
import org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager;
+import org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager.SnapshotMetaData;
import org.apache.solr.handler.IndexFetcher;
import org.apache.solr.handler.ReplicationHandler;
import org.apache.solr.handler.RequestHandlerBase;
@@ -194,6 +197,7 @@ public final class SolrCore implements SolrInfoMBean, Closeable {
private final List<Runnable> confListeners = new CopyOnWriteArrayList<>();
private final ReentrantLock ruleExpiryLock;
+ private final ReentrantLock snapshotDelLock; // A lock instance to guard against concurrent deletions.
public Date getStartTimeStamp() { return startTime; }
@@ -431,6 +435,83 @@ public final class SolrCore implements SolrInfoMBean, Closeable {
}
}
+ /**
+ * This method deletes the snapshot with the specified name. If the directory
+ * storing the snapshot is not the same as the *current* core index directory,
+ * then delete the files corresponding to this snapshot. Otherwise we leave the
+ * index files related to snapshot as is (assuming the underlying Solr IndexDeletionPolicy
+ * will clean them up appropriately).
+ *
+ * @param commitName The name of the snapshot to be deleted.
+ * @throws IOException in case of I/O error.
+ */
+ public void deleteNamedSnapshot(String commitName) throws IOException {
+ // Note this lock is required to prevent multiple snapshot deletions from
+ // opening multiple IndexWriter instances simultaneously.
+ this.snapshotDelLock.lock();
+ try {
+ Optional<SnapshotMetaData> metadata = snapshotMgr.release(commitName);
+ if (metadata.isPresent()) {
+ long gen = metadata.get().getGenerationNumber();
+ String indexDirPath = metadata.get().getIndexDirPath();
+
+ if (!indexDirPath.equals(getIndexDir())) {
+ Directory d = getDirectoryFactory().get(indexDirPath, DirContext.DEFAULT, "none");
+ try {
+ Collection<SnapshotMetaData> snapshots = snapshotMgr.listSnapshotsInIndexDir(indexDirPath);
+ log.info("Following snapshots exist in the index directory {} : {}", indexDirPath, snapshots);
+ if (snapshots.isEmpty()) {// No snapshots remain in this directory. Can be cleaned up!
+ log.info("Removing index directory {} since all named snapshots are deleted.", indexDirPath);
+ getDirectoryFactory().remove(d);
+ } else {
+ SolrSnapshotManager.deleteSnapshotIndexFiles(this, d, gen);
+ }
+ } finally {
+ getDirectoryFactory().release(d);
+ }
+ }
+ }
+ } finally {
+ snapshotDelLock.unlock();
+ }
+ }
+
+ /**
+ * This method deletes the index files not associated with any named snapshot only
+ * if the specified indexDirPath is not the *current* index directory.
+ *
+ * @param indexDirPath The path of the directory
+ * @throws IOException In case of I/O error.
+ */
+ public void deleteNonSnapshotIndexFiles(String indexDirPath) throws IOException {
+ // Skip if the specified indexDirPath is the *current* index directory.
+ if (getIndexDir().equals(indexDirPath)) {
+ return;
+ }
+
+ // Note this lock is required to prevent multiple snapshot deletions from
+ // opening multiple IndexWriter instances simultaneously.
+ this.snapshotDelLock.lock();
+ Directory dir = getDirectoryFactory().get(indexDirPath, DirContext.DEFAULT, "none");
+ try {
+ Collection<SnapshotMetaData> snapshots = snapshotMgr.listSnapshotsInIndexDir(indexDirPath);
+ log.info("Following snapshots exist in the index directory {} : {}", indexDirPath, snapshots);
+ // Delete the old index directory only if no snapshot exists in that directory.
+ if (snapshots.isEmpty()) {
+ log.info("Removing index directory {} since all named snapshots are deleted.", indexDirPath);
+ getDirectoryFactory().remove(dir);
+ } else {
+ SolrSnapshotManager.deleteNonSnapshotIndexFiles(this, dir, snapshots);
+ }
+ } finally {
+ snapshotDelLock.unlock();
+ if (dir != null) {
+ getDirectoryFactory().release(dir);
+ }
+ }
+ }
+
+
private void initListeners() {
final Class<SolrEventListener> clazz = SolrEventListener.class;
final String label = "Event Listener";
@@ -863,6 +944,8 @@ public final class SolrCore implements SolrInfoMBean, Closeable {
bufferUpdatesIfConstructing(coreDescriptor);
this.ruleExpiryLock = new ReentrantLock();
+ this.snapshotDelLock = new ReentrantLock();
+
registerConfListener();
assert ObjectReleaseTracker.track(this);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46aeb525/solr/core/src/java/org/apache/solr/core/snapshots/SolrSnapshotManager.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/snapshots/SolrSnapshotManager.java b/solr/core/src/java/org/apache/solr/core/snapshots/SolrSnapshotManager.java
index 95df3ff..4257baf 100644
--- a/solr/core/src/java/org/apache/solr/core/snapshots/SolrSnapshotManager.java
+++ b/solr/core/src/java/org/apache/solr/core/snapshots/SolrSnapshotManager.java
@@ -19,18 +19,18 @@ package org.apache.solr.core.snapshots;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
import java.util.Collection;
-import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
-import java.util.Map;
import java.util.Set;
-import java.util.function.Function;
-import java.util.stream.Collectors;
-
-import com.google.common.annotations.VisibleForTesting;
-import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexCommit;
+import org.apache.lucene.index.IndexDeletionPolicy;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.IndexWriterConfig.OpenMode;
+import org.apache.lucene.index.NoMergePolicy;
import org.apache.lucene.store.Directory;
+import org.apache.solr.core.SolrCore;
import org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager.SnapshotMetaData;
+import org.apache.solr.update.SolrIndexWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -43,92 +43,78 @@ public class SolrSnapshotManager {
/**
* This method deletes index files of the {@linkplain IndexCommit} for the specified generation number.
*
+ * @param core The Solr core
* @param dir The index directory storing the snapshot.
- * @param gen The generation number for the {@linkplain IndexCommit}
+ * @param gen The generation number of the {@linkplain IndexCommit} to be deleted.
* @throws IOException in case of I/O errors.
*/
- public static void deleteIndexFiles ( Directory dir, Collection<SnapshotMetaData> snapshots, long gen ) throws IOException {
- List<IndexCommit> commits = DirectoryReader.listCommits(dir);
- Map<String, Integer> refCounts = buildRefCounts(snapshots, commits);
- for (IndexCommit ic : commits) {
- if (ic.getGeneration() == gen) {
- deleteIndexFiles(dir,refCounts, ic);
- break;
+ public static void deleteSnapshotIndexFiles(SolrCore core, Directory dir, final long gen) throws IOException {
+ deleteSnapshotIndexFiles(core, dir, new IndexDeletionPolicy() {
+ @Override
+ public void onInit(List<? extends IndexCommit> commits) throws IOException {
+ for (IndexCommit ic : commits) {
+ if (gen == ic.getGeneration()) {
+ log.info("Deleting non-snapshotted index commit with generation {}", ic.getGeneration());
+ ic.delete();
+ }
+ }
}
- }
+
+ @Override
+ public void onCommit(List<? extends IndexCommit> commits)
+ throws IOException {}
+ });
}
/**
- * This method deletes all files not corresponding to a configured snapshot in the specified index directory.
+ * This method deletes index files not associated with the specified <code>snapshots</code>.
*
- * @param dir The index directory to search for.
+ * @param core The Solr core
+ * @param dir The index directory storing the snapshot.
+ * @param snapshots The snapshots to be preserved.
* @throws IOException in case of I/O errors.
*/
- public static void deleteNonSnapshotIndexFiles (Directory dir, Collection<SnapshotMetaData> snapshots) throws IOException {
- List<IndexCommit> commits = DirectoryReader.listCommits(dir);
- Map<String, Integer> refCounts = buildRefCounts(snapshots, commits);
- Set<Long> snapshotGenNumbers = snapshots.stream()
- .map(SnapshotMetaData::getGenerationNumber)
- .collect(Collectors.toSet());
- for (IndexCommit ic : commits) {
- if (!snapshotGenNumbers.contains(ic.getGeneration())) {
- deleteIndexFiles(dir,refCounts, ic);
- }
+ public static void deleteNonSnapshotIndexFiles(SolrCore core, Directory dir, Collection<SnapshotMetaData> snapshots) throws IOException {
+ final Set<Long> genNumbers = new HashSet<>();
+ for (SnapshotMetaData m : snapshots) {
+ genNumbers.add(m.getGenerationNumber());
}
- }
- /**
- * This method computes reference count for the index files by taking into consideration
- * (a) configured snapshots and (b) files sharing between two or more {@linkplain IndexCommit} instances.
- *
- * @param snapshots A collection of user configured snapshots
- * @param commits A list of {@linkplain IndexCommit} instances
- * @return A map containing reference count for each index file referred in one of the {@linkplain IndexCommit} instances.
- * @throws IOException in case of I/O error.
- */
- @VisibleForTesting
- static Map<String, Integer> buildRefCounts (Collection<SnapshotMetaData> snapshots, List<IndexCommit> commits) throws IOException {
- Map<String, Integer> result = new HashMap<>();
- Map<Long, IndexCommit> commitsByGen = commits.stream().collect(
- Collectors.toMap(IndexCommit::getGeneration, Function.identity()));
-
- for(SnapshotMetaData md : snapshots) {
- IndexCommit ic = commitsByGen.get(md.getGenerationNumber());
- if (ic != null) {
- Collection<String> fileNames = ic.getFileNames();
- for(String fileName : fileNames) {
- int refCount = result.getOrDefault(fileName, 0);
- result.put(fileName, refCount+1);
+ deleteSnapshotIndexFiles(core, dir, new IndexDeletionPolicy() {
+ @Override
+ public void onInit(List<? extends IndexCommit> commits) throws IOException {
+ for (IndexCommit ic : commits) {
+ if (!genNumbers.contains(ic.getGeneration())) {
+ log.info("Deleting non-snapshotted index commit with generation {}", ic.getGeneration());
+ ic.delete();
+ }
}
}
- }
- return result;
+ @Override
+ public void onCommit(List<? extends IndexCommit> commits)
+ throws IOException {}
+ });
}
/**
- * This method deletes the index files associated with specified <code>indexCommit</code> provided they
- * are not referred by some other {@linkplain IndexCommit}.
+ * This method deletes index files of the {@linkplain IndexCommit} for the specified generation number.
*
- * @param dir The index directory containing the {@linkplain IndexCommit} to be deleted.
- * @param refCounts A map containing reference counts for each file associated with every {@linkplain IndexCommit}
- * in the specified directory.
- * @param indexCommit The {@linkplain IndexCommit} whose files need to be deleted.
+ * @param core The Solr core
+ * @param dir The index directory storing the snapshot.
* @throws IOException in case of I/O errors.
*/
- private static void deleteIndexFiles ( Directory dir, Map<String, Integer> refCounts, IndexCommit indexCommit ) throws IOException {
- log.info("Deleting index files for index commit with generation {} in directory {}", indexCommit.getGeneration(), dir);
- for (String fileName : indexCommit.getFileNames()) {
- try {
- // Ensure that a file being deleted is not referred by some other commit.
- int ref = refCounts.getOrDefault(fileName, 0);
- log.debug("Reference count for file {} is {}", fileName, ref);
- if (ref == 0) {
- dir.deleteFile(fileName);
- }
- } catch (IOException e) {
- log.warn("Unable to delete file {} in directory {} due to exception {}", fileName, dir, e.getMessage());
- }
+ private static void deleteSnapshotIndexFiles(SolrCore core, Directory dir, IndexDeletionPolicy delPolicy) throws IOException {
+ IndexWriterConfig conf = core.getSolrConfig().indexConfig.toIndexWriterConfig(core);
+ conf.setOpenMode(OpenMode.APPEND);
+ conf.setMergePolicy(NoMergePolicy.INSTANCE);//Don't want to merge any commits here!
+ conf.setIndexDeletionPolicy(delPolicy);
+ conf.setCodec(core.getCodec());
+
+ try (SolrIndexWriter iw = new SolrIndexWriter("SolrSnapshotCleaner", dir, conf)) {
+ // Do nothing. The only purpose of opening index writer is to invoke the Lucene IndexDeletionPolicy#onInit
+ // method so that we can cleanup the files associated with specified index commit.
+ // Note the index writer creates a new commit during the close() operation (which is harmless).
}
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46aeb525/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java b/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
index b9d9f51..bdbd4e7 100644
--- a/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
+++ b/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
@@ -82,9 +82,6 @@ import org.apache.solr.core.DirectoryFactory;
import org.apache.solr.core.DirectoryFactory.DirContext;
import org.apache.solr.core.IndexDeletionPolicyWrapper;
import org.apache.solr.core.SolrCore;
-import org.apache.solr.core.snapshots.SolrSnapshotManager;
-import org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager;
-import org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager.SnapshotMetaData;
import org.apache.solr.handler.ReplicationHandler.*;
import org.apache.solr.request.LocalSolrQueryRequest;
import org.apache.solr.request.SolrQueryRequest;
@@ -478,17 +475,8 @@ public class IndexFetcher {
// may be closed
if (indexDir != null) {
solrCore.getDirectoryFactory().doneWithDirectory(indexDir);
-
- SolrSnapshotMetaDataManager snapshotsMgr = solrCore.getSnapshotMetaDataManager();
- Collection<SnapshotMetaData> snapshots = snapshotsMgr.listSnapshotsInIndexDir(indexDirPath);
-
- // Delete the old index directory only if no snapshot exists in that directory.
- if(snapshots.isEmpty()) {
- LOG.info("removing old index directory " + indexDir);
- solrCore.getDirectoryFactory().remove(indexDir);
- } else {
- SolrSnapshotManager.deleteNonSnapshotIndexFiles(indexDir, snapshots);
- }
+ // Cleanup all index files not associated with any *named* snapshot.
+ solrCore.deleteNonSnapshotIndexFiles(indexDirPath);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46aeb525/solr/core/src/java/org/apache/solr/handler/RestoreCore.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/RestoreCore.java b/solr/core/src/java/org/apache/solr/handler/RestoreCore.java
index 62cb93f..c00d7bd 100644
--- a/solr/core/src/java/org/apache/solr/handler/RestoreCore.java
+++ b/solr/core/src/java/org/apache/solr/handler/RestoreCore.java
@@ -19,7 +19,6 @@ package org.apache.solr.handler;
import java.lang.invoke.MethodHandles;
import java.net.URI;
import java.text.SimpleDateFormat;
-import java.util.Collection;
import java.util.Date;
import java.util.Locale;
import java.util.concurrent.Callable;
@@ -33,9 +32,6 @@ import org.apache.solr.common.SolrException;
import org.apache.solr.core.DirectoryFactory;
import org.apache.solr.core.SolrCore;
import org.apache.solr.core.backup.repository.BackupRepository;
-import org.apache.solr.core.snapshots.SolrSnapshotManager;
-import org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager;
-import org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager.SnapshotMetaData;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -135,16 +131,8 @@ public class RestoreCore implements Callable<Boolean> {
}
if (success) {
core.getDirectoryFactory().doneWithDirectory(indexDir);
-
- SolrSnapshotMetaDataManager snapshotsMgr = core.getSnapshotMetaDataManager();
- Collection<SnapshotMetaData> snapshots = snapshotsMgr.listSnapshotsInIndexDir(indexDirPath);
-
- // Delete the old index directory only if no snapshot exists in that directory.
- if (snapshots.isEmpty()) {
- core.getDirectoryFactory().remove(indexDir);
- } else {
- SolrSnapshotManager.deleteNonSnapshotIndexFiles(indexDir, snapshots);
- }
+ // Cleanup all index files not associated with any *named* snapshot.
+ core.deleteNonSnapshotIndexFiles(indexDirPath);
}
return true;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46aeb525/solr/core/src/java/org/apache/solr/handler/admin/DeleteSnapshotOp.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/admin/DeleteSnapshotOp.java b/solr/core/src/java/org/apache/solr/handler/admin/DeleteSnapshotOp.java
index 3dd9071..739837c 100644
--- a/solr/core/src/java/org/apache/solr/handler/admin/DeleteSnapshotOp.java
+++ b/solr/core/src/java/org/apache/solr/handler/admin/DeleteSnapshotOp.java
@@ -17,17 +17,11 @@
package org.apache.solr.handler.admin;
-import java.util.Optional;
-
-import org.apache.lucene.store.Directory;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.params.CoreAdminParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.core.CoreContainer;
-import org.apache.solr.core.DirectoryFactory;
import org.apache.solr.core.SolrCore;
-import org.apache.solr.core.snapshots.SolrSnapshotManager;
-import org.apache.solr.core.snapshots.SolrSnapshotMetaDataManager;
class DeleteSnapshotOp implements CoreAdminHandler.CoreAdminOp {
@@ -39,30 +33,15 @@ class DeleteSnapshotOp implements CoreAdminHandler.CoreAdminOp {
String commitName = params.required().get(CoreAdminParams.COMMIT_NAME);
String cname = params.required().get(CoreAdminParams.CORE);
- try (SolrCore core = cc.getCore(cname)) {
- if (core == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unable to locate core " + cname);
- }
-
- SolrSnapshotMetaDataManager mgr = core.getSnapshotMetaDataManager();
- Optional<SolrSnapshotMetaDataManager.SnapshotMetaData> metadata = mgr.release(commitName);
- if (metadata.isPresent()) {
- long gen = metadata.get().getGenerationNumber();
- String indexDirPath = metadata.get().getIndexDirPath();
+ SolrCore core = cc.getCore(cname);
+ if (core == null) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unable to locate core " + cname);
+ }
- // If the directory storing the snapshot is not the same as the *current* core
- // index directory, then delete the files corresponding to this snapshot.
- // Otherwise we leave the index files related to snapshot as is (assuming the
- // underlying Solr IndexDeletionPolicy will clean them up appropriately).
- if (!indexDirPath.equals(core.getIndexDir())) {
- Directory d = core.getDirectoryFactory().get(indexDirPath, DirectoryFactory.DirContext.DEFAULT, DirectoryFactory.LOCK_TYPE_NONE);
- try {
- SolrSnapshotManager.deleteIndexFiles(d, mgr.listSnapshotsInIndexDir(indexDirPath), gen);
- } finally {
- core.getDirectoryFactory().release(d);
- }
- }
- }
+ try {
+ core.deleteNamedSnapshot(commitName);
+ } finally {
+ core.close();
}
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46aeb525/solr/core/src/java/org/apache/solr/update/SolrIndexWriter.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/SolrIndexWriter.java b/solr/core/src/java/org/apache/solr/update/SolrIndexWriter.java
index fcfc559..d75214a 100644
--- a/solr/core/src/java/org/apache/solr/update/SolrIndexWriter.java
+++ b/solr/core/src/java/org/apache/solr/update/SolrIndexWriter.java
@@ -77,6 +77,15 @@ public class SolrIndexWriter extends IndexWriter {
}
}
+ public SolrIndexWriter(String name, Directory d, IndexWriterConfig conf) throws IOException {
+ super(d, conf);
+ this.name = name;
+ this.infoStream = conf.getInfoStream();
+ this.directory = d;
+ numOpens.incrementAndGet();
+ log.debug("Opened Writer " + name);
+ }
+
private SolrIndexWriter(SolrCore core, String name, String path, Directory directory, boolean create, IndexSchema schema, SolrIndexConfig config, IndexDeletionPolicy delPolicy, Codec codec) throws IOException {
super(directory,
config.toIndexWriterConfig(core).
@@ -182,7 +191,10 @@ public class SolrIndexWriter extends IndexWriter {
IOUtils.closeQuietly(infoStream);
}
numCloses.incrementAndGet();
- directoryFactory.release(directory);
+
+ if (directoryFactory != null) {
+ directoryFactory.release(directory);
+ }
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/46aeb525/solr/core/src/test/org/apache/solr/core/snapshots/TestSolrCoreSnapshots.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/core/snapshots/TestSolrCoreSnapshots.java b/solr/core/src/test/org/apache/solr/core/snapshots/TestSolrCoreSnapshots.java
index aacac52..da6dbac 100644
--- a/solr/core/src/test/org/apache/solr/core/snapshots/TestSolrCoreSnapshots.java
+++ b/solr/core/src/test/org/apache/solr/core/snapshots/TestSolrCoreSnapshots.java
@@ -17,8 +17,6 @@
package org.apache.solr.core.snapshots;
import java.lang.invoke.MethodHandles;
-import java.nio.file.Files;
-import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Collection;
@@ -124,7 +122,7 @@ public class TestSolrCoreSnapshots extends SolrCloudTestCase {
// and the other containing document deletions.
{
List<IndexCommit> commits = listCommits(metaData.getIndexDirPath());
- assertTrue(2 <= commits.size());
+ assertTrue(commits.size() >= 2);
}
// Backup the earlier created snapshot.
@@ -146,9 +144,11 @@ public class TestSolrCoreSnapshots extends SolrCloudTestCase {
}
// Verify that the old index directory (before restore) contains only those index commits referred by snapshots.
+ // The IndexWriter (used to cleanup index files) creates an additional commit during closing. Hence we expect 2 commits (instead
+ // of 1).
{
List<IndexCommit> commits = listCommits(metaData.getIndexDirPath());
- assertEquals(1, commits.size());
+ assertEquals(2, commits.size());
assertEquals(metaData.getGenerationNumber(), commits.get(0).getGeneration());
}
@@ -161,123 +161,16 @@ public class TestSolrCoreSnapshots extends SolrCloudTestCase {
// Delete second snapshot
deleteSnapshot(adminClient, coreName, duplicateCommit.getName());
- // Verify that corresponding index files have been deleted.
- assertTrue(listCommits(duplicateCommit.getIndexDirPath()).isEmpty());
+ // Verify that corresponding index files have been deleted. Ideally this directory should
+ // be removed immediately. But the current DirectoryFactory impl waits until the
+ // closing the core (or the directoryFactory) for actual removal. Since the IndexWriter
+ // (used to cleanup index files) creates an additional commit during closing, we expect a single
+ // commit (instead of 0).
+ assertEquals(1, listCommits(duplicateCommit.getIndexDirPath()).size());
}
}
@Test
- public void testHandlingSharedIndexFiles() throws Exception {
- CloudSolrClient solrClient = cluster.getSolrClient();
- String collectionName = "SolrCoreSnapshots_IndexFileSharing";
- CollectionAdminRequest.Create create = CollectionAdminRequest.createCollection(collectionName, "conf1", 1, 1);
- create.process(solrClient);
-
- int nDocs = BackupRestoreUtils.indexDocs(cluster.getSolrClient(), collectionName, docsSeed);
- DocCollection collectionState = solrClient.getZkStateReader().getClusterState().getCollection(collectionName);
- assertEquals(1, collectionState.getActiveSlices().size());
- Slice shard = collectionState.getActiveSlices().iterator().next();
- assertEquals(1, shard.getReplicas().size());
- Replica replica = shard.getReplicas().iterator().next();
-
- String replicaBaseUrl = replica.getStr(BASE_URL_PROP);
- String coreName = replica.getStr(ZkStateReader.CORE_NAME_PROP);
- String backupName = TestUtil.randomSimpleString(random(), 1, 5);
- String location = createTempDir().toFile().getAbsolutePath();
-
- try (
- SolrClient adminClient = getHttpSolrClient(cluster.getJettySolrRunners().get(0).getBaseUrl().toString());
- SolrClient masterClient = getHttpSolrClient(replica.getCoreUrl())) {
-
- int numTests = TestUtil.nextInt(random(), 2, 5);
- List<SnapshotMetaData> snapshots = new ArrayList<>(numTests);
-
- // Create multiple commits and create a snapshot per commit.
- // This should result in Lucene reusing some of the segments for later index commits.
- for (int attempt=0; attempt<numTests; attempt++) {
- if (nDocs > 0) {
- //Delete a few docs
- int numDeletes = TestUtil.nextInt(random(), 1, nDocs);
- for(int i=0; i<numDeletes; i++) {
- masterClient.deleteByQuery("id:" + i);
- }
- }
-
- // Add a few more
- int moreAdds = TestUtil.nextInt(random(), 1, 100);
- for (int i = 0; i < moreAdds; i++) {
- SolrInputDocument doc = new SolrInputDocument();
- doc.addField("id", i + nDocs);
- doc.addField("name", "name = " + (i + nDocs));
- masterClient.add(doc);
- }
- masterClient.commit();
-
- // Create a snapshot
- snapshots.add(createSnapshot(adminClient, coreName, "snapshot_" + attempt));
- }
-
- // Backup the earlier created snapshot.
- {
- Map<String,String> params = new HashMap<>();
- params.put("name", backupName);
- params.put("commitName", snapshots.get(0).getName());
- params.put("location", location);
- BackupRestoreUtils.runCoreAdminCommand(replicaBaseUrl, coreName, CoreAdminAction.BACKUPCORE.toString(), params);
- }
-
- // Restore the backup. The purpose of the restore operation is to change the *current* index directory.
- // This is required since we delegate the file deletion to underlying IndexDeletionPolicy in case of
- // *current* index directory. Hence for the purpose of this test, we want to ensure that the created
- // snapshots are NOT in the *current* index directory.
- {
- Map<String,String> params = new HashMap<>();
- params.put("name", "snapshot." + backupName);
- params.put("location", location);
- BackupRestoreUtils.runCoreAdminCommand(replicaBaseUrl, coreName, CoreAdminAction.RESTORECORE.toString(), params);
- }
-
- {
- SnapshotMetaData snapshotMetaData = snapshots.get(0);
-
- List<IndexCommit> commits = listCommits(snapshotMetaData.getIndexDirPath());
- // Check if number of index commits are > 0 to ensure index file sharing.
- assertTrue(commits.size() > 0);
- Map<String,Integer> refCounts = SolrSnapshotManager.buildRefCounts(snapshots, commits);
-
- Optional<IndexCommit> ic = commits.stream()
- .filter(entry -> entry.getGeneration() == snapshotMetaData.getGenerationNumber())
- .findFirst();
- assertTrue(ic.isPresent());
- Collection<String> nonSharedFiles = new ArrayList<>();
- Collection<String> sharedFiles = new ArrayList<>();
- for (String fileName : ic.get().getFileNames()) {
- if (refCounts.getOrDefault(fileName, 0) > 1) {
- sharedFiles.add(fileName);
- } else {
- nonSharedFiles.add(fileName);
- }
- }
-
- // Delete snapshot
- deleteSnapshot(adminClient, coreName, snapshotMetaData.getName());
-
- // Verify that the shared files are not deleted.
- for (String fileName : sharedFiles) {
- Path path = Paths.get(snapshotMetaData.getIndexDirPath(), fileName);
- assertTrue(path + " should exist.", Files.exists(path));
- }
-
- // Verify that the non-shared files are deleted.
- for (String fileName : nonSharedFiles) {
- Path path = Paths.get(snapshotMetaData.getIndexDirPath(), fileName);
- assertFalse(path + " should not exist.", Files.exists(path));
- }
- }
- }
- }
-
- @Test
public void testIndexOptimization() throws Exception {
CloudSolrClient solrClient = cluster.getSolrClient();
String collectionName = "SolrCoreSnapshots_IndexOptimization";
[47/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-99570: Various log
tidying at Solr startup
Posted by kr...@apache.org.
SOLR-99570: Various log tidying at Solr startup
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/97761966
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/97761966
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/97761966
Branch: refs/heads/jira/solr-8593
Commit: 97761966f30557c33b3bbb131ce64ea7905ae213
Parents: c4b4830
Author: Jan H�ydahl <ja...@apache.org>
Authored: Thu Oct 20 14:47:32 2016 +0200
Committer: Jan H�ydahl <ja...@apache.org>
Committed: Thu Oct 20 14:47:32 2016 +0200
----------------------------------------------------------------------
solr/CHANGES.txt | 15 +-
solr/bin/solr | 18 +-
solr/bin/solr.cmd | 28 +-
.../src/java/org/apache/solr/util/SolrCLI.java | 273 ++++++++++++++++++-
.../org/apache/solr/util/UtilsToolTest.java | 185 +++++++++++++
5 files changed, 481 insertions(+), 38 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/97761966/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index b3a2a30..b4dcf4c 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -86,8 +86,12 @@ Upgrade Notes
prefix, then you will now get an error as these options are incompatible with numeric faceting.
* Solr's logging verbosity at the INFO level has been greatly reduced, and
- you may need to update the log configs to use the DEBUG level to get the
- same logging messages as before.
+ you may need to update the log configs to use the DEBUG level to see all the
+ logging messages you used to see at INFO level before.
+
+* We are no longer backing up solr.log and solr_gc.log files in date-stamped copies forever. If you relied on
+ the solr_log_<date> or solr_gc_log_<date> being in the logs folder that will no longer be the case.
+ See SOLR-9570 for details.
* The create/deleteCollection methods on MiniSolrCloudCluster have been
deprecated. Clients should instead use the CollectionAdminRequest API. In
@@ -308,6 +312,13 @@ Other Changes
* SOLR-7850: Moved defaults within bin/solr.in.sh (and bin/solr.in.cmd on Windows) to bin/solr (and bin/solr.cmd)
such that the default state of these files is to set nothing. This makes Solr work better with Docker. (David Smiley)
+* SOLR-9570: Various log tidying now happens at Solr startup:
+ Old solr_log_<date> and solr_gc_log_<date> files are removed, avoiding disks to fill up,
+ solr.log.X files are rotated, preserving solr.log from last run in solr.log.1, solr.log.1 => solr.log.2 etc
+ solr-*-console.log files are moved into $SOLR_LOGS_DIR/archived/ instead of being overwritten
+ Last JVM garbage collection log solr_gc.log is moved into $SOLR_LOGS_DIR/archived/
+ (janhoy)
+
================== 6.2.1 ==================
Bug Fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/97761966/solr/bin/solr
----------------------------------------------------------------------
diff --git a/solr/bin/solr b/solr/bin/solr
index df6b4d0..6aa5709 100755
--- a/solr/bin/solr
+++ b/solr/bin/solr
@@ -1387,20 +1387,10 @@ if [ ! -e "$SOLR_HOME" ]; then
exit 1
fi
-# backup the log files before starting
-if [ -f "$SOLR_LOGS_DIR/solr.log" ]; then
- if $verbose ; then
- echo "Backing up $SOLR_LOGS_DIR/solr.log"
- fi
- mv "$SOLR_LOGS_DIR/solr.log" "$SOLR_LOGS_DIR/solr_log_$(date +"%Y%m%d_%H%M")"
-fi
-
-if [ -f "$SOLR_LOGS_DIR/solr_gc.log" ]; then
- if $verbose ; then
- echo "Backing up $SOLR_LOGS_DIR/solr_gc.log"
- fi
- mv "$SOLR_LOGS_DIR/solr_gc.log" "$SOLR_LOGS_DIR/solr_gc_log_$(date +"%Y%m%d_%H%M")"
-fi
+run_tool utils -s "$DEFAULT_SERVER_DIR" -l "$SOLR_LOGS_DIR" -remove_old_solr_logs 7 || echo "Failed removing old solr logs"
+run_tool utils -s "$DEFAULT_SERVER_DIR" -l "$SOLR_LOGS_DIR" -archive_gc_logs || echo "Failed archiving old GC logs"
+run_tool utils -s "$DEFAULT_SERVER_DIR" -l "$SOLR_LOGS_DIR" -archive_console_logs || echo "Failed archiving old console logs"
+run_tool utils -s "$DEFAULT_SERVER_DIR" -l "$SOLR_LOGS_DIR" -rotate_solr_logs 9 || echo "Failed rotating old solr logs"
java_ver_out=`echo "$("$JAVA" -version 2>&1)"`
JAVA_VERSION=`echo $java_ver_out | grep "java version" | awk '{ print substr($3, 2, length($3)-2); }'`
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/97761966/solr/bin/solr.cmd
----------------------------------------------------------------------
diff --git a/solr/bin/solr.cmd b/solr/bin/solr.cmd
index 10ea6d6..317a789 100644
--- a/solr/bin/solr.cmd
+++ b/solr/bin/solr.cmd
@@ -860,19 +860,11 @@ IF ERRORLEVEL 1 (
set IS_64bit=true
)
-REM backup log files (use current timestamp for backup name)
-For /f "tokens=2-4 delims=/ " %%a in ('date /t') do (set mydate=%%c-%%a-%%b)
-For /f "tokens=1-2 delims=/:" %%a in ("%TIME%") do (set mytime=%%a%%b)
-set now_ts=!mydate!_!mytime!
-IF EXIST "!SOLR_LOGS_DIR!\solr.log" (
- echo Backing up !SOLR_LOGS_DIR!\solr.log
- move /Y "!SOLR_LOGS_DIR!\solr.log" "!SOLR_LOGS_DIR!\solr_log_!now_ts!"
-)
-
-IF EXIST "!SOLR_LOGS_DIR!\solr_gc.log" (
- echo Backing up !SOLR_LOGS_DIR!\solr_gc.log
- move /Y "!SOLR_LOGS_DIR!\solr_gc.log" "!SOLR_LOGS_DIR!\solr_gc_log_!now_ts!"
-)
+REM Clean up and rotate logs
+call :run_utils "-remove_old_solr_logs 7" || echo "Failed removing old solr logs"
+call :run_utils "-archive_gc_logs" || echo "Failed archiving old GC logs"
+call :run_utils "-archive_console_logs" || echo "Failed archiving old console logs"
+call :run_utils "-rotate_solr_logs 9" || echo "Failed rotating old solr logs"
IF NOT "%ZK_HOST%"=="" set SOLR_MODE=solrcloud
@@ -1136,6 +1128,16 @@ goto done
org.apache.solr.util.SolrCLI version
goto done
+:run_utils
+set "TOOL_CMD=%~1"
+"%JAVA%" %SOLR_SSL_OPTS% %SOLR_ZK_CREDS_AND_ACLS% -Dsolr.install.dir="%SOLR_TIP%" -Dlog4j.configuration="file:%DEFAULT_SERVER_DIR%\scripts\cloud-scripts\log4j.properties" ^
+ -classpath "%DEFAULT_SERVER_DIR%\solr-webapp\webapp\WEB-INF\lib\*;%DEFAULT_SERVER_DIR%\lib\ext\*" ^
+ org.apache.solr.util.SolrCLI utils -s "%DEFAULT_SERVER_DIR%" -l "%SOLR_LOGS_DIR%" %TOOL_CMD%
+if errorlevel 1 (
+ exit /b 1
+)
+goto done
+
:parse_create_args
IF [%1]==[] goto run_create
IF "%1"=="-c" goto set_create_name
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/97761966/solr/core/src/java/org/apache/solr/util/SolrCLI.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/util/SolrCLI.java b/solr/core/src/java/org/apache/solr/util/SolrCLI.java
index 9724b81..39bf548 100644
--- a/solr/core/src/java/org/apache/solr/util/SolrCLI.java
+++ b/solr/core/src/java/org/apache/solr/util/SolrCLI.java
@@ -29,7 +29,10 @@ import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
import java.nio.file.attribute.FileOwnerAttributeView;
+import java.time.Instant;
+import java.time.Period;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
@@ -44,6 +47,8 @@ import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
@@ -108,11 +113,11 @@ import org.slf4j.LoggerFactory;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.solr.common.params.CommonParams.NAME;
+
/**
* Command-line utility for working with Solr.
*/
public class SolrCLI {
-
/**
* Defines the interface to a Solr tool that can be run from this command-line app.
*/
@@ -233,7 +238,6 @@ public class SolrCLI {
};
private static void exit(int exitStatus) {
- // TODO: determine if we're running in a test and don't exit
try {
System.exit(exitStatus);
} catch (java.lang.SecurityException secExc) {
@@ -259,6 +263,18 @@ public class SolrCLI {
exit(0);
}
+ Tool tool = findTool(args);
+ CommandLine cli = parseCmdLine(args, tool.getOptions());
+ System.exit(tool.runTool(cli));
+ }
+
+ public static Tool findTool(String[] args) throws Exception {
+ String toolType = args[0].trim().toLowerCase(Locale.ROOT);
+ return newTool(toolType);
+ }
+
+ public static CommandLine parseCmdLine(String[] args, Option[] toolOptions) throws Exception {
+
String builderClassName = System.getProperty("solr.authentication.httpclient.builder");
if (builderClassName!=null) {
try {
@@ -272,10 +288,6 @@ public class SolrCLI {
}
}
- // Determine the tool
- String toolType = args[0].trim().toLowerCase(Locale.ROOT);
- Tool tool = newTool(toolType);
-
// the parser doesn't like -D props
List<String> toolArgList = new ArrayList<String>();
List<String> dashDList = new ArrayList<String>();
@@ -291,7 +303,7 @@ public class SolrCLI {
// process command-line args to configure this application
CommandLine cli =
- processCommandLineArgs(joinCommonAndToolOptions(tool.getOptions()), toolArgs);
+ processCommandLineArgs(joinCommonAndToolOptions(toolOptions), toolArgs);
List argList = cli.getArgList();
argList.addAll(dashDList);
@@ -303,8 +315,7 @@ public class SolrCLI {
checkSslStoreSysProp(solrInstallDir, "trustStore");
}
- // run the tool
- exit(tool.runTool(cli));
+ return cli;
}
protected static void checkSslStoreSysProp(String solrInstallDir, String key) {
@@ -368,6 +379,8 @@ public class SolrCLI {
return new ZkLsTool();
else if ("assert".equals(toolType))
return new AssertTool();
+ else if ("utils".equals(toolType))
+ return new UtilsTool();
// If you add a built-in tool to this class, add it here to avoid
// classpath scanning
@@ -3339,4 +3352,246 @@ public class SolrCLI {
}
}
} // end AssertTool class
+
+ public static class UtilsTool extends ToolBase {
+ private Path serverPath;
+ private Path logsPath;
+ private boolean beQuiet;
+
+ public UtilsTool() { this(System.out); }
+ public UtilsTool(PrintStream stdout) { super(stdout); }
+
+ public String getName() {
+ return "prestart";
+ }
+
+ @SuppressWarnings("static-access")
+ public Option[] getOptions() {
+ return new Option[]{
+ OptionBuilder
+ .withArgName("path")
+ .hasArg()
+ .withDescription("Path to server dir. Required if logs path is relative")
+ .create("s"),
+ OptionBuilder
+ .withArgName("path")
+ .hasArg()
+ .withDescription("Path to logs dir. If relative, also provide server dir with -s")
+ .create("l"),
+ OptionBuilder
+ .withDescription("Be quiet, don't print to stdout, only return exit codes")
+ .create("q"),
+ OptionBuilder
+ .withArgName("daysToKeep")
+ .hasArg()
+ .withType(Integer.class)
+ .withDescription("Path to logs directory")
+ .create("remove_old_solr_logs"),
+ OptionBuilder
+ .withArgName("generations")
+ .hasArg()
+ .withType(Integer.class)
+ .withDescription("Rotate solr.log to solr.log.1 etc")
+ .create("rotate_solr_logs"),
+ OptionBuilder
+ .withDescription("Archive old garbage collection logs into archive/")
+ .create("archive_gc_logs"),
+ OptionBuilder
+ .withDescription("Archive old console logs into archive/")
+ .create("archive_console_logs")
+ };
+ }
+
+ @Override
+ public int runTool(CommandLine cli) throws Exception {
+ if (cli.getOptions().length == 0 || cli.getArgs().length > 0 || cli.hasOption("h")) {
+ new HelpFormatter().printHelp("bin/solr utils [OPTIONS]", getToolOptions(this));
+ return 1;
+ }
+ if (cli.hasOption("s")) {
+ serverPath = Paths.get(cli.getOptionValue("s"));
+ }
+ if (cli.hasOption("l")) {
+ logsPath = Paths.get(cli.getOptionValue("l"));
+ }
+ if (cli.hasOption("q")) {
+ beQuiet = cli.hasOption("q");
+ }
+ if (cli.hasOption("remove_old_solr_logs")) {
+ if (removeOldSolrLogs(Integer.parseInt(cli.getOptionValue("remove_old_solr_logs"))) > 0) return 1;
+ }
+ if (cli.hasOption("rotate_solr_logs")) {
+ if (rotateSolrLogs(Integer.parseInt(cli.getOptionValue("rotate_solr_logs"))) > 0) return 1;
+ }
+ if (cli.hasOption("archive_gc_logs")) {
+ if (archiveGcLogs() > 0) return 1;
+ }
+ if (cli.hasOption("archive_console_logs")) {
+ if (archiveConsoleLogs() > 0) return 1;
+ }
+ return 0;
+ }
+
+ /**
+ * Moves gc logs into archived/
+ * @return 0 on success
+ * @throws Exception on failure
+ */
+ public int archiveGcLogs() throws Exception {
+ prepareLogsPath();
+ Path archivePath = logsPath.resolve("archived");
+ if (!archivePath.toFile().exists()) {
+ Files.createDirectories(archivePath);
+ }
+ List<Path> archived = Files.find(archivePath, 1, (f, a)
+ -> a.isRegularFile() && String.valueOf(f.getFileName()).startsWith("solr_gc_"))
+ .collect(Collectors.toList());
+ for (Path p : archived) {
+ Files.delete(p);
+ }
+ List<Path> files = Files.find(logsPath, 1, (f, a)
+ -> a.isRegularFile() && String.valueOf(f.getFileName()).startsWith("solr_gc_"))
+ .collect(Collectors.toList());
+ if (files.size() > 0) {
+ out("Archiving " + files.size() + " old GC log files to " + archivePath);
+ for (Path p : files) {
+ Files.move(p, archivePath.resolve(p.getFileName()), StandardCopyOption.REPLACE_EXISTING);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Moves console log(s) into archiced/
+ * @return 0 on success
+ * @throws Exception on failure
+ */
+ public int archiveConsoleLogs() throws Exception {
+ prepareLogsPath();
+ Path archivePath = logsPath.resolve("archived");
+ if (!archivePath.toFile().exists()) {
+ Files.createDirectories(archivePath);
+ }
+ List<Path> archived = Files.find(archivePath, 1, (f, a)
+ -> a.isRegularFile() && String.valueOf(f.getFileName()).endsWith("-console.log"))
+ .collect(Collectors.toList());
+ for (Path p : archived) {
+ Files.delete(p);
+ }
+ List<Path> files = Files.find(logsPath, 1, (f, a)
+ -> a.isRegularFile() && String.valueOf(f.getFileName()).endsWith("-console.log"))
+ .collect(Collectors.toList());
+ if (files.size() > 0) {
+ out("Archiving " + files.size() + " console log files");
+ for (Path p : files) {
+ Files.move(p, archivePath.resolve(p.getFileName()), StandardCopyOption.REPLACE_EXISTING);
+ }
+ }
+ return 0;
+ }
+
+ /**
+ * Rotates solr.log before starting Solr. Mimics log4j2 behavior, i.e. with generations=9:
+ * <pre>
+ * solr.log.9 (and higher) are deleted
+ * solr.log.8 -> solr.log.9
+ * solr.log.7 -> solr.log.8
+ * ...
+ * solr.log -> solr.log.1
+ * </pre>
+ * @param generations number of generations to keep. Should agree with setting in log4j.properties
+ * @return 0 if success
+ * @throws Exception if problems
+ */
+ public int rotateSolrLogs(int generations) throws Exception {
+ prepareLogsPath();
+ if (logsPath.toFile().exists() && logsPath.resolve("solr.log").toFile().exists()) {
+ out("Rotating solr logs, keeping a max of "+generations+" generations");
+ try (Stream<Path> files = Files.find(logsPath, 1,
+ (f, a) -> a.isRegularFile() && String.valueOf(f.getFileName()).startsWith("solr.log."))
+ .sorted((b,a) -> new Integer(a.getFileName().toString().substring(9))
+ .compareTo(new Integer(b.getFileName().toString().substring(9))))) {
+ files.forEach(p -> {
+ try {
+ int number = Integer.parseInt(p.getFileName().toString().substring(9));
+ if (number >= generations) {
+ Files.delete(p);
+ } else {
+ Path renamed = p.getParent().resolve("solr.log." + (number + 1));
+ Files.move(p, renamed);
+ }
+ } catch (IOException e) {
+ out("Problem during rotation of log files: " + e.getMessage());
+ }
+ });
+ } catch (NumberFormatException nfe) {
+ throw new Exception("Do not know how to rotate solr.log.<ext> with non-numeric extension. Rotate aborted.", nfe);
+ }
+ Files.move(logsPath.resolve("solr.log"), logsPath.resolve("solr.log.1"));
+ }
+
+ return 0;
+ }
+
+ /**
+ * Deletes time-stamped old solr logs, if older than n days
+ * @param daysToKeep number of days logs to keep before deleting
+ * @return 0 on success
+ * @throws Exception on failure
+ */
+ public int removeOldSolrLogs(int daysToKeep) throws Exception {
+ prepareLogsPath();
+ if (logsPath.toFile().exists()) {
+ try (Stream<Path> stream = Files.find(logsPath, 2, (f, a) -> a.isRegularFile()
+ && Instant.now().minus(Period.ofDays(daysToKeep)).isAfter(a.lastModifiedTime().toInstant())
+ && String.valueOf(f.getFileName()).startsWith("solr_log_"))) {
+ List<Path> files = stream.collect(Collectors.toList());
+ if (files.size() > 0) {
+ out("Deleting "+files.size() + " solr_log_* files older than " + daysToKeep + " days.");
+ for (Path p : files) {
+ Files.delete(p);
+ }
+ }
+ }
+ }
+ return 0;
+ }
+
+ // Private methods to follow
+
+ private void out(String message) {
+ if (!beQuiet) {
+ stdout.print(message + "\n");
+ }
+ }
+
+ private void prepareLogsPath() throws Exception {
+ if (logsPath == null) {
+ throw new Exception("Command requires the -l <log-directory> option");
+ }
+ if (!logsPath.isAbsolute()) {
+ if (serverPath != null && serverPath.isAbsolute() && serverPath.toFile().exists()) {
+ logsPath = serverPath.resolve(logsPath);
+ } else {
+ throw new Exception("Logs directory must be an absolute path, or -s must be supplied");
+ }
+ }
+ }
+
+ @Override
+ protected void runImpl(CommandLine cli) throws Exception {
+ }
+
+ public void setLogPath(Path logsPath) {
+ this.logsPath = logsPath;
+ }
+
+ public void setServerPath(Path serverPath) {
+ this.serverPath = serverPath;
+ }
+
+ public void setQuiet(boolean shouldPrintStdout) {
+ this.beQuiet = shouldPrintStdout;
+ }
+ } // end UtilsTool class
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/97761966/solr/core/src/test/org/apache/solr/util/UtilsToolTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/util/UtilsToolTest.java b/solr/core/src/test/org/apache/solr/util/UtilsToolTest.java
new file mode 100644
index 0000000..fa39620
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/util/UtilsToolTest.java
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.util;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.FileTime;
+import java.time.Instant;
+import java.time.Period;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.commons.cli.CommandLine;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.apache.solr.util.SolrCLI.findTool;
+import static org.apache.solr.util.SolrCLI.parseCmdLine;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Unit test for SolrCLI's UtilsTool
+ */
+public class UtilsToolTest {
+
+ private Path dir;
+ private SolrCLI.UtilsTool tool;
+ private List<String> files = Arrays.asList(
+ "solr.log",
+ "solr.log.1",
+ "solr.log.2",
+ "solr.log.3",
+ "solr.log.9",
+ "solr.log.10",
+ "solr.log.11",
+ "solr_log_20160102",
+ "solr_log_20160304",
+ "solr-8983-console.log",
+ "solr_gc_log_20160102",
+ "solr_gc_log_2");
+
+ @Before
+ public void setUp() throws IOException {
+ dir = Files.createTempDirectory("Utils Tool Test");
+ files.stream().forEach(f -> {
+ try {
+ dir.resolve(f).toFile().createNewFile();
+ } catch (IOException e) {
+ assertTrue(false);
+ }
+ });
+ }
+
+ @After
+ public void tearDown() throws IOException {
+ org.apache.commons.io.FileUtils.deleteDirectory(dir.toFile());
+ }
+
+ @Test
+ public void testEmptyAndQuiet() throws Exception {
+ String[] args = {"utils", "-remove_old_solr_logs", "7",
+ "-rotate_solr_logs", "9",
+ "-archive_gc_logs",
+ "-archive_console_logs",
+ "-q",
+ "-l", dir.toString()};
+ assertEquals(0, runTool(args));
+ }
+
+ @Test
+ public void testNonexisting() throws Exception {
+ String nonexisting = dir.resolve("non-existing").toString();
+ String[] args = {"utils", "-remove_old_solr_logs", "7",
+ "-rotate_solr_logs", "9",
+ "-archive_gc_logs",
+ "-archive_console_logs",
+ "-l", nonexisting};
+ assertEquals(0, runTool(args));
+ }
+
+ @Test
+ public void testRemoveOldSolrLogs() throws Exception {
+ String[] args = {"utils", "-remove_old_solr_logs", "1", "-l", dir.toString()};
+ assertEquals(files.size(), fileCount());
+ assertEquals(0, runTool(args));
+ assertEquals(files.size(), fileCount()); // No logs older than 1 day
+ Files.setLastModifiedTime(dir.resolve("solr_log_20160102"), FileTime.from(Instant.now().minus(Period.ofDays(2))));
+ assertEquals(0, runTool(args));
+ assertEquals(files.size()-1, fileCount()); // One logs older than 1 day
+ Files.setLastModifiedTime(dir.resolve("solr_log_20160304"), FileTime.from(Instant.now().minus(Period.ofDays(3))));
+ assertEquals(0, runTool(args));
+ assertEquals(files.size()-2, fileCount()); // Two logs older than 1 day
+ }
+
+ @Test
+ public void testRelativePath() throws Exception {
+ String[] args = {"utils", "-remove_old_solr_logs", "0", "-l", dir.getFileName().toString(), "-s", dir.getParent().toString()};
+ assertEquals(files.size(), fileCount());
+ assertEquals(0, runTool(args));
+ assertEquals(files.size()-2, fileCount());
+ }
+
+ @Test
+ public void testRelativePathError() throws Exception {
+ String[] args = {"utils", "-remove_old_solr_logs", "0", "-l", dir.getFileName().toString()};
+ try {
+ runTool(args);
+ } catch (Exception e) {
+ return;
+ }
+ assertTrue(false);
+ }
+
+ @Test
+ public void testRemoveOldGcLogs() throws Exception {
+ String[] args = {"utils", "-archive_gc_logs", "-l", dir.toString()};
+ assertEquals(files.size(), fileCount());
+ assertEquals(0, runTool(args));
+ assertEquals(files.size()-2, fileCount());
+ assertFalse(listFiles().contains("solr_gc_log_2"));
+ assertTrue(Files.exists(dir.resolve("archived").resolve("solr_gc_log_2")));
+ assertEquals(0, runTool(args));
+ assertFalse(Files.exists(dir.resolve("archived").resolve("solr_gc_log_2")));
+ }
+
+ @Test
+ public void testArchiveConsoleLogs() throws Exception {
+ String[] args = {"utils", "-archive_console_logs", "-l", dir.toString()};
+ assertEquals(files.size(), fileCount());
+ assertEquals(0, runTool(args));
+ assertEquals(files.size()-1, fileCount());
+ assertFalse(listFiles().contains("solr-8983-console.log"));
+ assertTrue(Files.exists(dir.resolve("archived").resolve("solr-8983-console.log")));
+ assertEquals(0, runTool(args));
+ assertFalse(Files.exists(dir.resolve("archived").resolve("solr-8983-console.log")));
+ }
+
+ @Test
+ public void testRotateSolrLogs() throws Exception {
+ String[] args = {"utils", "-rotate_solr_logs", "9", "-l", dir.toString()};
+ assertEquals(files.size(), fileCount());
+ assertTrue(listFiles().contains("solr.log"));
+ assertEquals(0, runTool(args));
+ assertEquals(files.size()-3, fileCount());
+ assertTrue(listFiles().contains("solr.log.4"));
+ assertFalse(listFiles().contains("solr.log"));
+ assertFalse(listFiles().contains("solr.log.9"));
+ assertFalse(listFiles().contains("solr.log.10"));
+ assertFalse(listFiles().contains("solr.log.11"));
+ }
+
+ private List<String> listFiles() throws IOException {
+ return Files.find(dir, 1, (p, a) -> a.isRegularFile()).map(p -> p.getFileName().toString()).collect(Collectors.toList());
+ }
+
+ private long fileCount() throws IOException {
+ return listFiles().size();
+ }
+
+ private int runTool(String[] args) throws Exception {
+ SolrCLI.Tool tool = findTool(args);
+ CommandLine cli = parseCmdLine(args, tool.getOptions());
+ return tool.runTool(cli);
+ }
+}
\ No newline at end of file
[36/50] [abbrv] lucene-solr:jira/solr-8593: fix typo in comment
Posted by kr...@apache.org.
fix typo in comment
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/731c5f93
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/731c5f93
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/731c5f93
Branch: refs/heads/jira/solr-8593
Commit: 731c5f93166ad28c21f330eef94c9d8f451d11c7
Parents: 45ca4bd
Author: Mike McCandless <mi...@apache.org>
Authored: Wed Oct 19 09:47:15 2016 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Wed Oct 19 09:47:47 2016 -0400
----------------------------------------------------------------------
lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/731c5f93/lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java b/lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java
index cc5647e..57f0a32 100644
--- a/lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java
+++ b/lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java
@@ -157,7 +157,7 @@ public class DrillSideways {
DrillSidewaysQuery dsq = new DrillSidewaysQuery(baseQuery, drillDownCollector, drillSidewaysCollectors, drillDownQueries, scoreSubDocsAtOnce());
if (hitCollector.needsScores() == false) {
- // this is a borrible hack in order to make sure IndexSearcher will not
+ // this is a horrible hack in order to make sure IndexSearcher will not
// attempt to cache the DrillSidewaysQuery
hitCollector = new FilterCollector(hitCollector) {
@Override
[35/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7505:
AnalyzingInfixSuggester returned invalid results when allTermsRequired is
false and context filters are specified
Posted by kr...@apache.org.
LUCENE-7505: AnalyzingInfixSuggester returned invalid results when allTermsRequired is false and context filters are specified
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/45ca4bd3
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/45ca4bd3
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/45ca4bd3
Branch: refs/heads/jira/solr-8593
Commit: 45ca4bd3ed53b264cb1188bac5c76efa47d6e23e
Parents: c2e031a
Author: Mike McCandless <mi...@apache.org>
Authored: Wed Oct 19 09:44:08 2016 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Wed Oct 19 09:44:34 2016 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 4 ++
.../analyzing/AnalyzingInfixSuggester.java | 12 +++-
.../analyzing/AnalyzingInfixSuggesterTest.java | 76 ++++++++++++++++++++
3 files changed, 89 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45ca4bd3/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 6d83c53..21ded1a 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -92,6 +92,10 @@ Bug Fixes
* LUCENE-7493: FacetCollector.search threw an unexpected exception if
you asked for zero hits but wanted facets (Mahesh via Mike McCandless)
+* LUCENE-7505: AnalyzingInfixSuggester returned invalid results when
+ allTermsRequired is false and context filters are specified (Mike
+ McCandless)
+
Improvements
* LUCENE-7439: FuzzyQuery now matches all terms within the specified
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45ca4bd3/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java
----------------------------------------------------------------------
diff --git a/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java b/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java
index d05c39f..aa60237 100644
--- a/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java
+++ b/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java
@@ -560,12 +560,18 @@ public class AnalyzingInfixSuggester extends Lookup implements Closeable {
}
if (allMustNot) {
- //all are MUST_NOT: add the contextQuery to the main query instead (not as sub-query)
+ // All are MUST_NOT: add the contextQuery to the main query instead (not as sub-query)
for (BooleanClause clause : contextQuery.clauses()) {
query.add(clause);
}
+ } else if (allTermsRequired == false) {
+ // We must carefully upgrade the query clauses to MUST:
+ BooleanQuery.Builder newQuery = new BooleanQuery.Builder();
+ newQuery.add(query.build(), BooleanClause.Occur.MUST);
+ newQuery.add(contextQuery, BooleanClause.Occur.MUST);
+ query = newQuery;
} else {
- //Add contextQuery as sub-query
+ // Add contextQuery as sub-query
query.add(contextQuery, BooleanClause.Occur.MUST);
}
}
@@ -577,7 +583,7 @@ public class AnalyzingInfixSuggester extends Lookup implements Closeable {
Query finalQuery = finishQuery(query, allTermsRequired);
- //System.out.println("finalQuery=" + query);
+ //System.out.println("finalQuery=" + finalQuery);
// Sort by weight, descending:
TopFieldCollector c = TopFieldCollector.create(SORT, num, true, false, false);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45ca4bd3/lucene/suggest/src/test/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggesterTest.java
----------------------------------------------------------------------
diff --git a/lucene/suggest/src/test/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggesterTest.java b/lucene/suggest/src/test/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggesterTest.java
index 69d3ed6..d98d052 100644
--- a/lucene/suggest/src/test/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggesterTest.java
+++ b/lucene/suggest/src/test/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggesterTest.java
@@ -1258,4 +1258,80 @@ public class AnalyzingInfixSuggesterTest extends LuceneTestCase {
a.close();
}
}
+
+ public void testContextNotAllTermsRequired() throws Exception {
+
+ Input keys[] = new Input[] {
+ new Input("lend me your ear", 8, new BytesRef("foobar"), asSet("foo", "bar")),
+ new Input("a penny saved is a penny earned", 10, new BytesRef("foobaz"), asSet("foo", "baz"))
+ };
+ Path tempDir = createTempDir("analyzingInfixContext");
+
+ Analyzer a = new MockAnalyzer(random(), MockTokenizer.WHITESPACE, false);
+ AnalyzingInfixSuggester suggester = new AnalyzingInfixSuggester(newFSDirectory(tempDir), a, a, 3, false);
+ suggester.build(new InputArrayIterator(keys));
+
+ // No context provided, all results returned
+ List<LookupResult> results = suggester.lookup(TestUtil.stringToCharSequence("ear", random()), 10, false, true);
+ assertEquals(2, results.size());
+ LookupResult result = results.get(0);
+ assertEquals("a penny saved is a penny earned", result.key);
+ assertEquals("a penny saved is a penny <b>ear</b>ned", result.highlightKey);
+ assertEquals(10, result.value);
+ assertEquals(new BytesRef("foobaz"), result.payload);
+ assertNotNull(result.contexts);
+ assertEquals(2, result.contexts.size());
+ assertTrue(result.contexts.contains(new BytesRef("foo")));
+ assertTrue(result.contexts.contains(new BytesRef("baz")));
+
+ result = results.get(1);
+ assertEquals("lend me your ear", result.key);
+ assertEquals("lend me your <b>ear</b>", result.highlightKey);
+ assertEquals(8, result.value);
+ assertEquals(new BytesRef("foobar"), result.payload);
+ assertNotNull(result.contexts);
+ assertEquals(2, result.contexts.size());
+ assertTrue(result.contexts.contains(new BytesRef("foo")));
+ assertTrue(result.contexts.contains(new BytesRef("bar")));
+
+ // Both have "foo" context:
+ results = suggester.lookup(TestUtil.stringToCharSequence("ear", random()), asSet("foo"), 10, false, true);
+ assertEquals(2, results.size());
+
+ result = results.get(0);
+ assertEquals("a penny saved is a penny earned", result.key);
+ assertEquals("a penny saved is a penny <b>ear</b>ned", result.highlightKey);
+ assertEquals(10, result.value);
+ assertEquals(new BytesRef("foobaz"), result.payload);
+ assertNotNull(result.contexts);
+ assertEquals(2, result.contexts.size());
+ assertTrue(result.contexts.contains(new BytesRef("foo")));
+ assertTrue(result.contexts.contains(new BytesRef("baz")));
+
+ result = results.get(1);
+ assertEquals("lend me your ear", result.key);
+ assertEquals("lend me your <b>ear</b>", result.highlightKey);
+ assertEquals(8, result.value);
+ assertEquals(new BytesRef("foobar"), result.payload);
+ assertNotNull(result.contexts);
+ assertEquals(2, result.contexts.size());
+ assertTrue(result.contexts.contains(new BytesRef("foo")));
+ assertTrue(result.contexts.contains(new BytesRef("bar")));
+
+ // Only one has "foo" context and len
+ results = suggester.lookup(TestUtil.stringToCharSequence("len", random()), asSet("foo"), 10, false, true);
+ assertEquals(1, results.size());
+
+ result = results.get(0);
+ assertEquals("lend me your ear", result.key);
+ assertEquals("<b>len</b>d me your ear", result.highlightKey);
+ assertEquals(8, result.value);
+ assertEquals(new BytesRef("foobar"), result.payload);
+ assertNotNull(result.contexts);
+ assertEquals(2, result.contexts.size());
+ assertTrue(result.contexts.contains(new BytesRef("foo")));
+ assertTrue(result.contexts.contains(new BytesRef("bar")));
+
+ suggester.close();
+ }
}
[29/50] [abbrv] lucene-solr:jira/solr-8593: * SOLR-9506: cache
IndexFingerprint for each segment
Posted by kr...@apache.org.
* SOLR-9506: cache IndexFingerprint for each segment
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/9aa764a5
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/9aa764a5
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/9aa764a5
Branch: refs/heads/jira/solr-8593
Commit: 9aa764a54f50eca5a8ef805bdb29e4ad90fcce5e
Parents: fe17b4e
Author: Noble Paul <no...@gmail.com>
Authored: Tue Oct 18 21:09:42 2016 +0530
Committer: Noble Paul <no...@gmail.com>
Committed: Tue Oct 18 21:09:42 2016 +0530
----------------------------------------------------------------------
solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9aa764a5/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java b/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
index 59797b9..02069ee 100644
--- a/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
+++ b/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
@@ -277,7 +277,7 @@ public class SolrIndexSearcher extends IndexSearcher implements Closeable, SolrI
this.path = path;
this.directoryFactory = directoryFactory;
- this.reader = (DirectoryReader) super.getTopReaderContext().reader();
+ this.reader = (DirectoryReader) super.readerContext.reader();
this.rawReader = r;
this.leafReader = SlowCompositeReaderWrapper.wrap(this.reader);
this.core = core;
[45/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-8370: Display
configured Similarity in Schema-Browser
Posted by kr...@apache.org.
SOLR-8370: Display configured Similarity in Schema-Browser
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/14b6d93d
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/14b6d93d
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/14b6d93d
Branch: refs/heads/jira/solr-8593
Commit: 14b6d93db4b0a608809782d1ef01fa97840b80e0
Parents: 39db548
Author: Jan H�ydahl <ja...@apache.org>
Authored: Thu Oct 20 13:25:40 2016 +0200
Committer: Jan H�ydahl <ja...@apache.org>
Committed: Thu Oct 20 13:25:40 2016 +0200
----------------------------------------------------------------------
solr/CHANGES.txt | 3 +
.../solr/handler/admin/LukeRequestHandler.java | 1 +
.../similarities/SchemaSimilarityFactory.java | 39 ++++++++-----
solr/webapp/web/css/angular/schema.css | 23 ++++++++
.../webapp/web/js/angular/controllers/schema.js | 60 +++++++++++++-------
solr/webapp/web/partials/schema.html | 15 +++--
6 files changed, 101 insertions(+), 40 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/14b6d93d/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index aca7601..b3a2a30 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -142,6 +142,9 @@ New Features
* SOLR-9417: Allow daemons to terminate when they finish iterating a topic (Joel Bernstein)
+* SOLR-8370: Display configured Similarity in Schema-Browser, both global/default and per-field/field-type
+ (janhoy, Alexandre Rafalovitch)
+
Bug Fixes
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/14b6d93d/solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java b/solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
index a5fc36c..d0dd152 100644
--- a/solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
@@ -474,6 +474,7 @@ public class LukeRequestHandler extends RequestHandlerBase
finfo.add("uniqueKeyField",
null == uniqueField ? null : uniqueField.getName());
finfo.add("defaultSearchField", schema.getDefaultSearchFieldName());
+ finfo.add("similarity", getSimilarityInfo(schema.getSimilarity()));
finfo.add("types", types);
return finfo;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/14b6d93d/solr/core/src/java/org/apache/solr/search/similarities/SchemaSimilarityFactory.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/search/similarities/SchemaSimilarityFactory.java b/solr/core/src/java/org/apache/solr/search/similarities/SchemaSimilarityFactory.java
index e648481..a71de18 100644
--- a/solr/core/src/java/org/apache/solr/search/similarities/SchemaSimilarityFactory.java
+++ b/solr/core/src/java/org/apache/solr/search/similarities/SchemaSimilarityFactory.java
@@ -129,21 +129,32 @@ public class SchemaSimilarityFactory extends SimilarityFactory implements SolrCo
"' but that <fieldType> does not define a <similarity>");
}
}
- assert null != defaultSim;
- final Similarity defaultSimilarity = defaultSim;
- similarity = new PerFieldSimilarityWrapper() {
- @Override
- public Similarity get(String name) {
- FieldType fieldType = core.getLatestSchema().getFieldTypeNoEx(name);
- if (fieldType == null) {
- return defaultSimilarity;
- } else {
- Similarity similarity = fieldType.getSimilarity();
- return similarity == null ? defaultSimilarity : similarity;
- }
- }
- };
+ similarity = new SchemaSimilarity(defaultSim);
}
return similarity;
}
+
+ private class SchemaSimilarity extends PerFieldSimilarityWrapper {
+ private Similarity defaultSimilarity;
+
+ public SchemaSimilarity(Similarity defaultSimilarity) {
+ this.defaultSimilarity = defaultSimilarity;
+ }
+
+ @Override
+ public Similarity get(String name) {
+ FieldType fieldType = core.getLatestSchema().getFieldTypeNoEx(name);
+ if (fieldType == null) {
+ return defaultSimilarity;
+ } else {
+ Similarity similarity = fieldType.getSimilarity();
+ return similarity == null ? defaultSimilarity : similarity;
+ }
+ }
+
+ @Override
+ public String toString() {
+ return "SchemaSimilarity. Default: " + ((get("") == null) ? "null" : get("").toString());
+ }
+ }
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/14b6d93d/solr/webapp/web/css/angular/schema.css
----------------------------------------------------------------------
diff --git a/solr/webapp/web/css/angular/schema.css b/solr/webapp/web/css/angular/schema.css
index 626cdc2..98a857f 100644
--- a/solr/webapp/web/css/angular/schema.css
+++ b/solr/webapp/web/css/angular/schema.css
@@ -701,4 +701,27 @@ limitations under the License.
#content #schema .copyfield .updatable a {
float:left;
width:80%;
+}
+
+#content #schema dd.similarity.ng-binding::after {
+ content: attr(data-tip) ;
+
+ font-size: 12px;
+ position: relative;
+ white-space: nowrap;
+ bottom: 9999px;
+ left: 0;
+ background: lightyellow;
+ color: black;
+ padding: 4px 7px;
+ line-height: 24px;
+ height: 24px;
+ border: 1px solid darkgray;
+ opacity: 0;
+ transition:opacity 0.4s ease-out;
+}
+
+#content #schema dd.similarity.ng-binding:hover::after {
+ opacity: 90;
+ bottom: -20px;
}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/14b6d93d/solr/webapp/web/js/angular/controllers/schema.js
----------------------------------------------------------------------
diff --git a/solr/webapp/web/js/angular/controllers/schema.js b/solr/webapp/web/js/angular/controllers/schema.js
index ee23bd7..94dd93e 100644
--- a/solr/webapp/web/js/angular/controllers/schema.js
+++ b/solr/webapp/web/js/angular/controllers/schema.js
@@ -70,6 +70,10 @@ solrAdminApp.controller('SchemaController',
$scope.core = $routeParams.core;
$scope.defaultSearchField = data.default_search_field;
$scope.uniqueKeyField = data.unique_key_field;
+ $scope.similarity = data.similarity;
+ if ($scope.similarity && $scope.similarity.className) {
+ $scope.similarity.className = shortenPackages($scope.similarity.className);
+ }
$scope.isDefaultSearchField = ($scope.selectedType == "Field" && $scope.name == $scope.defaultSearchField);
$scope.isUniqueKeyField = ($scope.selectedType == "Field" && $scope.name == $scope.uniqueKeyField);
@@ -334,6 +338,7 @@ var mergeIndexAndSchemaData = function(index, schema) {
var data = {
default_search_field: null,
unique_key_field: null,
+ similarity: null,
key: {},
fields: {},
dynamic_fields: {},
@@ -354,6 +359,7 @@ var mergeIndexAndSchemaData = function(index, schema) {
data.default_search_field = schema.defaultSearchField;
data.unique_key_field = schema.uniqueKeyField;
+ data.similarity = schema.similarity;
data.dynamic_fields = schema.dynamicFields;
data.types = schema.types;
@@ -422,11 +428,11 @@ var mergeIndexAndSchemaData = function(index, schema) {
return data;
};
-var getFieldProperties = function(data, core, is, field) {
+var getFieldProperties = function(data, core, is, name) {
var display = {};
- display.partialState = is.field && !!data.fields[field].partial;
+ display.partialState = is.field && !!data.fields[name].partial;
display.columns = [];
display.rows = [];
@@ -446,23 +452,33 @@ var getFieldProperties = function(data, core, is, field) {
}
// Identify the rows for our field property table
- if (is.field && data.fields[field]) {
- if (data.fields[field].flags) {
- addRow('Properties', data.fields[field].flags);
- }
- if (data.fields[field].schema) {
- addRow('Schema', data.fields[field].schema);
- }
- if (data.fields[field].index) {
- addRow('Index', data.fields[field].index);
- }
- display.docs = data.fields[field].docs;
- display.docsUrl = "#/" + core + "/query?q=" + field + ":[* TO *]";
- display.distinct = data.fields[field].distinct;
- display.positionIncrementGap = data.fields[field].positionIncrementGap;
- display.similarity = data.fields[field].similarity;
- } else if (is.dynamicField && data.dynamic_fields[field] && data.dynamic_fields[field].flags) {
- addRow('Properties', data.dynamic_fields[field].flags);
+ if (is.field && data.fields[name]) {
+ if (data.fields[name].flags) {
+ addRow('Properties', data.fields[name].flags);
+ }
+ if (data.fields[name].schema) {
+ addRow('Schema', data.fields[name].schema);
+ }
+ if (data.fields[name].index) {
+ addRow('Index', data.fields[name].index);
+ }
+ display.docs = data.fields[name].docs;
+ display.docsUrl = "#/" + core + "/query?q=" + name + ":[* TO *]";
+ display.distinct = data.fields[name].distinct;
+ display.positionIncrementGap = data.fields[name].positionIncrementGap;
+ if (data.types[data.fields[name].type]) {
+ display.similarity = data.types[data.fields[name].type].similarity;
+ } else {
+ display.similarity = null;
+ }
+ } else if (is.dynamicField && data.dynamic_fields[name] && data.dynamic_fields[name].flags) {
+ addRow('Properties', data.dynamic_fields[name].flags);
+ display.similarity = data.types[data.dynamic_fields[name].type].similarity;
+ } else if (is.type && data.types[name]) {
+ display.similarity = data.types[name].similarity;
+ }
+ if (display.similarity && display.similarity.className) {
+ display.similarity.className = shortenPackages(display.similarity.className);
}
// identify columns in field property table:
@@ -591,7 +607,11 @@ var sortedObjectArray = function(list) {
objarr.push({"name": list[i]});
}
return objarr;
-}
+};
+
+var shortenPackages = function(className) {
+ return className.replace("org.apache.solr", "o.a.s").replace("org.apache.lucene", "o.a.l");
+};
/*
var get_width = function get_width()
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/14b6d93d/solr/webapp/web/partials/schema.html
----------------------------------------------------------------------
diff --git a/solr/webapp/web/partials/schema.html b/solr/webapp/web/partials/schema.html
index ca626fd..1c0347f 100644
--- a/solr/webapp/web/partials/schema.html
+++ b/solr/webapp/web/partials/schema.html
@@ -212,7 +212,7 @@ limitations under the License.
</h2>
</div>
- <div class="partial" ng-show="partialState">
+ <div class="partial" ng-show="display.partialState">
<p>Because your Index is empty, we do not have enough Information about this Field</p>
@@ -220,11 +220,11 @@ limitations under the License.
<dl class="options clearfix">
- <dt class="field-type">Field-Type:</dt>
- <dd class="field-type">{{analysis.data.className}}</dd>
+ <dt class="field-type" ng-show="analysis.data.className">Field-Type:</dt>
+ <dd class="field-type" ng-show="analysis.data.className">{{analysis.data.className}}</dd>
- <dt class="similarity" ng-show="display.similarity">Similarity:</dt>
- <dd class="similarity" ng-show="display.similarity">{{ display.similarity.details }} ({{ similarity.className }}) </dd>
+ <dt class="similarity" ng-show="display.similarity.className">Similarity:</dt>
+ <dd class="similarity" ng-show="display.similarity.className" data-tip="{{ display.similarity.className }}">{{ display.similarity.details }}</dd>
<dt class="position-increment-gap" ng-show="display.positionIncrementGap"><abbr title="Position Increment Gap">PI Gap</abbr>:</dt>
<dd class="position-increment-gap" ng-show="display.positionIncrementGap">{{ display.positionIncrementGap }}</dd>
@@ -287,7 +287,7 @@ limitations under the License.
</div>
- <div class="terminfo-holder loaded clearfix" ng-class="{disabled: noTermData}" ng-show="is.field">
+ <div class="terminfo-holder loaded clearfix" ng-class="{disabled: noTermData}" ng-show="is.field && !display.partialState">
<div class="trigger">
@@ -438,6 +438,9 @@ limitations under the License.
<dt class="default-search-field" ng-class="{active: isDefaultSearchField}" ng-show="defaultSearchField">Default Search Field</dt>
<dd class="default-search-field" ng-class="{active: isDefaultSearchField}"><a ng-href="#/{{core}}/schema?field={{defaultSearchField}}">{{defaultSearchField}}</a></dd>
+ <dt class="similarity" ng-class="{active: similarity.className}">Global Similarity:</dt>
+ <dd class="similarity" ng-class="{active: similarity.className}" data-tip="{{ similarity.className }}">{{ similarity.details }}</dd>
+
</dl>
</div>
[42/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-7850: Move defaults
in bin/solr.in.sh into bin/solr (incl. Windows)
Posted by kr...@apache.org.
SOLR-7850: Move defaults in bin/solr.in.sh into bin/solr (incl. Windows)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/fe77dff0
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/fe77dff0
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/fe77dff0
Branch: refs/heads/jira/solr-8593
Commit: fe77dff09406b0c848a269a6bfee490ea6c67015
Parents: 8ae3304
Author: David Smiley <ds...@apache.org>
Authored: Wed Oct 19 16:45:13 2016 -0400
Committer: David Smiley <ds...@apache.org>
Committed: Wed Oct 19 16:45:13 2016 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fe77dff0/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 56f3b80..efea20f 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -301,7 +301,7 @@ Other Changes
* SOLR-9634: Deprecate collection methods on MiniSolrCloudCluster (Alan Woodward)
-* SOLR-7580: Moved defaults within bin/solr.in.sh (and bin/solr.in.cmd on Windows) to bin/solr (and bin/solr.cmd)
+* SOLR-7850: Moved defaults within bin/solr.in.sh (and bin/solr.in.cmd on Windows) to bin/solr (and bin/solr.cmd)
such that the default state of these files is to set nothing. This makes Solr work better with Docker. (David Smiley)
================== 6.2.1 ==================
[05/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9325: solr.log is
now written to $SOLR_LOGS_DIR without changing log4j.properties
Posted by kr...@apache.org.
SOLR-9325: solr.log is now written to $SOLR_LOGS_DIR without changing log4j.properties
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/33db4de4
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/33db4de4
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/33db4de4
Branch: refs/heads/jira/solr-8593
Commit: 33db4de4d7d5e325f8bfd886d3957735b33310a8
Parents: 9304ef9
Author: Jan H�ydahl <ja...@apache.org>
Authored: Fri Oct 14 23:19:09 2016 +0200
Committer: Jan H�ydahl <ja...@apache.org>
Committed: Fri Oct 14 23:19:09 2016 +0200
----------------------------------------------------------------------
solr/CHANGES.txt | 2 ++
solr/bin/solr | 18 ++++++++++--
solr/bin/solr.cmd | 31 ++++++++++++++++----
solr/bin/solr.in.cmd | 3 ++
solr/bin/solr.in.sh | 5 ++--
.../apache/solr/servlet/SolrDispatchFilter.java | 1 +
.../solr/servlet/StartupLoggingUtils.java | 9 ++++++
solr/server/resources/log4j.properties | 2 +-
8 files changed, 59 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/33db4de4/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 3280778..2bee12e 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -171,6 +171,8 @@ Bug Fixes
* SOLR-9604: Pooled SSL connections were not being re-used (Alan Woodward,
Mikhail Khludnev, hossman)
+* SOLR-9325: solr.log is now written to $SOLR_LOGS_DIR without changing log4j.properties (janhoy)
+
Optimizations
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/33db4de4/solr/bin/solr
----------------------------------------------------------------------
diff --git a/solr/bin/solr b/solr/bin/solr
index f753d7d..90fffa1 100755
--- a/solr/bin/solr
+++ b/solr/bin/solr
@@ -1568,7 +1568,7 @@ function launch_solr() {
fi
SOLR_START_OPTS=('-server' "${JAVA_MEM_OPTS[@]}" "${GC_TUNE[@]}" "${GC_LOG_OPTS[@]}" \
- "${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" $SOLR_LOG_LEVEL_OPT \
+ "${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" $SOLR_LOG_LEVEL_OPT -Dsolr.log.dir="$SOLR_LOGS_DIR" \
"-Djetty.port=$SOLR_PORT" "-DSTOP.PORT=$stop_port" "-DSTOP.KEY=$STOP_KEY" \
"${SOLR_HOST_ARG[@]}" "-Duser.timezone=$SOLR_TIMEZONE" \
"-Djetty.home=$SOLR_SERVER_DIR" "-Dsolr.solr.home=$SOLR_HOME" "-Dsolr.install.dir=$SOLR_TIP" \
@@ -1578,7 +1578,21 @@ function launch_solr() {
IN_CLOUD_MODE=" in SolrCloud mode"
fi
- mkdir -p "$SOLR_LOGS_DIR"
+ mkdir -p "$SOLR_LOGS_DIR" 2>/dev/null
+ if [ $? -ne 0 ]; then
+ echo -e "\nERROR: Logs directory $SOLR_LOGS_DIR could not be created. Exiting"
+ exit 1
+ fi
+ if [ ! -w "$SOLR_LOGS_DIR" ]; then
+ echo -e "\nERROR: Logs directory $SOLR_LOGS_DIR is not writable. Exiting"
+ exit 1
+ fi
+ case "$SOLR_LOGS_DIR" in
+ contexts|etc|lib|modules|resources|scripts|solr|solr-webapp)
+ echo -e "\nERROR: Logs directory $SOLR_LOGS_DIR is invalid. Reserved for the system. Exiting"
+ exit 1
+ ;;
+ esac
if [ "$run_in_foreground" == "true" ]; then
exec "$JAVA" "${SOLR_START_OPTS[@]}" $SOLR_ADDL_ARGS -jar start.jar "${SOLR_JETTY_CONFIG[@]}"
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/33db4de4/solr/bin/solr.cmd
----------------------------------------------------------------------
diff --git a/solr/bin/solr.cmd b/solr/bin/solr.cmd
index 0bfb773..5376143 100644
--- a/solr/bin/solr.cmd
+++ b/solr/bin/solr.cmd
@@ -729,7 +729,13 @@ IF "%STOP_KEY%"=="" set STOP_KEY=solrrocks
@REM This is quite hacky, but examples rely on a different log4j.properties
@REM so that we can write logs for examples to %SOLR_HOME%\..\logs
-set "SOLR_LOGS_DIR=%SOLR_SERVER_DIR%\logs"
+IF [%SOLR_LOGS_DIR%] == [] (
+ set "SOLR_LOGS_DIR=%SOLR_SERVER_DIR%\logs"
+) ELSE (
+ set SOLR_LOGS_DIR=%SOLR_LOGS_DIR:"=%
+)
+set SOLR_LOGS_DIR_QUOTED="%SOLR_LOGS_DIR%"
+
set "EXAMPLE_DIR=%SOLR_TIP%\example"
set TMP=!SOLR_HOME:%EXAMPLE_DIR%=!
IF NOT "%TMP%"=="%SOLR_HOME%" (
@@ -965,13 +971,26 @@ IF NOT "%SOLR_SSL_OPTS%"=="" (
set "START_OPTS=%START_OPTS% %SOLR_SSL_OPTS% !SSL_PORT_PROP!"
)
IF NOT "%SOLR_LOG_LEVEL%"=="" set "START_OPTS=%START_OPTS% -Dsolr.log.level=%SOLR_LOG_LEVEL%"
-
+set "START_OPTS=%START_OPTS% -Dsolr.log.dir=%SOLR_LOGS_DIR_QUOTED%"
IF NOT DEFINED LOG4J_CONFIG set "LOG4J_CONFIG=file:%SOLR_SERVER_DIR%\resources\log4j.properties"
cd /d "%SOLR_SERVER_DIR%"
-IF NOT EXIST "!SOLR_LOGS_DIR!" (
- mkdir "!SOLR_LOGS_DIR!"
+IF NOT EXIST "%SOLR_LOGS_DIR%" (
+ mkdir "%SOLR_LOGS_DIR%"
+)
+copy /Y NUL "%SOLR_LOGS_DIR%\.writable" > NUL 2>&1 && set WRITEOK=1
+IF DEFINED WRITEOK (
+ del "%SOLR_LOGS_DIR%\.writable"
+) else (
+ echo "ERROR: Logs directory %SOLR_LOGS_DIR% is not writable or could not be created. Exiting"
+ GOTO :eof
+)
+echo " contexts etc lib modules resources scripts solr solr-webapp " > "%TEMP%\solr-pattern.txt"
+findstr /i /C:" %SOLR_LOGS_DIR% " "%TEMP%\solr-pattern.txt" 1>nul
+if %ERRORLEVEL% == 0 (
+ echo "ERROR: Logs directory %SOLR_LOGS_DIR% is invalid. Reserved for the system. Exiting"
+ GOTO :eof
)
IF NOT EXIST "%SOLR_SERVER_DIR%\tmp" (
@@ -988,13 +1007,13 @@ IF "%FG%"=="1" (
REM run solr in the foreground
title "Solr-%SOLR_PORT%"
echo %SOLR_PORT%>"%SOLR_TIP%"\bin\solr-%SOLR_PORT%.port
- "%JAVA%" %SERVEROPT% %SOLR_JAVA_MEM% %START_OPTS% %GCLOG_OPT%:"!SOLR_LOGS_DIR!"/solr_gc.log ^
+ "%JAVA%" %SERVEROPT% %SOLR_JAVA_MEM% %START_OPTS% %GCLOG_OPT%:"!SOLR_LOGS_DIR!/solr_gc.log" ^
-Dlog4j.configuration="%LOG4J_CONFIG%" -DSTOP.PORT=!STOP_PORT! -DSTOP.KEY=%STOP_KEY% ^
-Djetty.port=%SOLR_PORT% -Dsolr.solr.home="%SOLR_HOME%" -Dsolr.install.dir="%SOLR_TIP%" ^
-Djetty.home="%SOLR_SERVER_DIR%" -Djava.io.tmpdir="%SOLR_SERVER_DIR%\tmp" -jar start.jar "%SOLR_JETTY_CONFIG%"
) ELSE (
START /B "Solr-%SOLR_PORT%" /D "%SOLR_SERVER_DIR%" "%JAVA%" %SERVEROPT% %SOLR_JAVA_MEM% %START_OPTS% ^
- %GCLOG_OPT%:"!SOLR_LOGS_DIR!"/solr_gc.log -Dlog4j.configuration="%LOG4J_CONFIG%" -DSTOP.PORT=!STOP_PORT! ^
+ %GCLOG_OPT%:"!SOLR_LOGS_DIR!/solr_gc.log" -Dlog4j.configuration="%LOG4J_CONFIG%" -DSTOP.PORT=!STOP_PORT! ^
-Dsolr.log.muteconsole ^
-DSTOP.KEY=%STOP_KEY% -Djetty.port=%SOLR_PORT% -Dsolr.solr.home="%SOLR_HOME%" -Dsolr.install.dir="%SOLR_TIP%" ^
-Djetty.home="%SOLR_SERVER_DIR%" -Djava.io.tmpdir="%SOLR_SERVER_DIR%\tmp" -jar start.jar ^
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/33db4de4/solr/bin/solr.in.cmd
----------------------------------------------------------------------
diff --git a/solr/bin/solr.in.cmd b/solr/bin/solr.in.cmd
index 07bffcc..5a2d00f 100644
--- a/solr/bin/solr.in.cmd
+++ b/solr/bin/solr.in.cmd
@@ -31,6 +31,9 @@ REM Changes the logging level. Valid values: ALL, TRACE, DEBUG, INFO, WARN, ERRO
REM This is an alternative to changing the rootLogger in log4j.properties
REM set SOLR_LOG_LEVEL=INFO
+REM Location where Solr should write logs to. Absolute or relative to solr start dir
+REM set SOLR_LOGS_DIR=logs
+
REM These GC settings have shown to work well for a number of common Solr workloads
set GC_TUNE=-XX:NewRatio=3 ^
-XX:SurvivorRatio=4 ^
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/33db4de4/solr/bin/solr.in.sh
----------------------------------------------------------------------
diff --git a/solr/bin/solr.in.sh b/solr/bin/solr.in.sh
index 999e673..dd41f46 100644
--- a/solr/bin/solr.in.sh
+++ b/solr/bin/solr.in.sh
@@ -96,9 +96,8 @@ SOLR_OPTS="$SOLR_OPTS -Xss256k"
# This is an alternative to changing the rootLogger in log4j.properties
#SOLR_LOG_LEVEL=INFO
-# Location where Solr should write logs to; should agree with the file appender
-# settings in server/resources/log4j.properties
-#SOLR_LOGS_DIR=
+# Location where Solr should write logs to. Absolute or relative to solr start dir
+#SOLR_LOGS_DIR=logs
# Sets the port Solr binds to, default is 8983
#SOLR_PORT=8983
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/33db4de4/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java b/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
index 241749a..b9d3729 100644
--- a/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
+++ b/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
@@ -123,6 +123,7 @@ public class SolrDispatchFilter extends BaseSolrFilter {
{
log.trace("SolrDispatchFilter.init(): {}", this.getClass().getClassLoader());
+ StartupLoggingUtils.checkLogDir();
logWelcomeBanner();
String muteConsole = System.getProperty(SOLR_LOG_MUTECONSOLE);
if (muteConsole != null && !Arrays.asList("false","0","off","no").contains(muteConsole.toLowerCase(Locale.ROOT))) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/33db4de4/solr/core/src/java/org/apache/solr/servlet/StartupLoggingUtils.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/servlet/StartupLoggingUtils.java b/solr/core/src/java/org/apache/solr/servlet/StartupLoggingUtils.java
index feb88e9..3da0841 100644
--- a/solr/core/src/java/org/apache/solr/servlet/StartupLoggingUtils.java
+++ b/solr/core/src/java/org/apache/solr/servlet/StartupLoggingUtils.java
@@ -41,6 +41,15 @@ final class StartupLoggingUtils {
private final static StaticLoggerBinder binder = StaticLoggerBinder.getSingleton();
/**
+ * Checks whether mandatory log dir is given
+ */
+ static void checkLogDir() {
+ if (System.getProperty("solr.log.dir") == null) {
+ log.error("Missing Java Option solr.log.dir. Logging may be missing or incomplete.");
+ }
+ }
+
+ /**
* Disables all log4j ConsoleAppender's by modifying log4j configuration dynamically.
* Must only be used during early startup
* @return true if ok or else false if something happened, e.g. log4j classes were not in classpath
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/33db4de4/solr/server/resources/log4j.properties
----------------------------------------------------------------------
diff --git a/solr/server/resources/log4j.properties b/solr/server/resources/log4j.properties
index 9a5e3fd..9f9c4a0 100644
--- a/solr/server/resources/log4j.properties
+++ b/solr/server/resources/log4j.properties
@@ -1,6 +1,6 @@
# Default Solr log4j config
# rootLogger log level may be programmatically overridden by -Dsolr.log.level
-solr.log=logs
+solr.log=${solr.log.dir}
log4j.rootLogger=INFO, file, CONSOLE
# Console appender will be programmatically disabled when Solr is started with option -Dsolr.log.muteconsole
[02/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9639: CDCR Tests
only fix. Wait until recovery is over before remove the tmp_colletion.
Posted by kr...@apache.org.
SOLR-9639: CDCR Tests only fix. Wait until recovery is over before
remove the tmp_colletion.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/47446733
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/47446733
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/47446733
Branch: refs/heads/jira/solr-8593
Commit: 47446733884e030feaecac355c01c58f9e5e3169
Parents: f4d3ca8
Author: Mikhail Khludnev <mk...@apache.org>
Authored: Fri Oct 14 13:53:02 2016 +0300
Committer: Mikhail Khludnev <mk...@apache.org>
Committed: Fri Oct 14 14:43:07 2016 +0300
----------------------------------------------------------------------
solr/CHANGES.txt | 2 ++
.../src/test/org/apache/solr/cloud/BaseCdcrDistributedZkTest.java | 1 +
2 files changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/47446733/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 7124d40..3280778 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -271,6 +271,8 @@ Other Changes
* SOLR-9610: New AssertTool in SolrCLI for easier cross platform assertions from command line (janhoy)
+* SOLR-9639: Test only fix. Prevent CDCR tests from removing collection during recovery that used to blow up jvm (Mikhail Khludnev)
+
================== 6.2.1 ==================
Bug Fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/47446733/solr/core/src/test/org/apache/solr/cloud/BaseCdcrDistributedZkTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/BaseCdcrDistributedZkTest.java b/solr/core/src/test/org/apache/solr/cloud/BaseCdcrDistributedZkTest.java
index 3d758e0..35cc444 100644
--- a/solr/core/src/test/org/apache/solr/cloud/BaseCdcrDistributedZkTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/BaseCdcrDistributedZkTest.java
@@ -593,6 +593,7 @@ public class BaseCdcrDistributedZkTest extends AbstractDistribZkTestBase {
}
}
+ this.waitForRecoveriesToFinish(temporaryCollection,zkStateReader, true);
// delete the temporary collection - we will create our own collections later
this.deleteCollection(temporaryCollection);
this.waitForCollectionToDisappear(temporaryCollection);
[03/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7495: Fix doc
values iterators' assertions in advance().
Posted by kr...@apache.org.
LUCENE-7495: Fix doc values iterators' assertions in advance().
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ea121223
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ea121223
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ea121223
Branch: refs/heads/jira/solr-8593
Commit: ea1212232d2b95e53bfa6adbbbb3ce7700d3cff4
Parents: 4744673
Author: Adrien Grand <jp...@gmail.com>
Authored: Fri Oct 14 14:37:46 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Fri Oct 14 14:38:32 2016 +0200
----------------------------------------------------------------------
.../org/apache/lucene/search/join/BlockJoinSelector.java | 4 +++-
.../java/org/apache/lucene/index/AssertingLeafReader.java | 10 +++++-----
2 files changed, 8 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ea121223/lucene/join/src/java/org/apache/lucene/search/join/BlockJoinSelector.java
----------------------------------------------------------------------
diff --git a/lucene/join/src/java/org/apache/lucene/search/join/BlockJoinSelector.java b/lucene/join/src/java/org/apache/lucene/search/join/BlockJoinSelector.java
index 26b416d..a2e0c55 100644
--- a/lucene/join/src/java/org/apache/lucene/search/join/BlockJoinSelector.java
+++ b/lucene/join/src/java/org/apache/lucene/search/join/BlockJoinSelector.java
@@ -162,7 +162,9 @@ public class BlockJoinSelector {
return nextDoc();
}
int prevParentDocID = parents.prevSetBit(target-1);
- values.advance(prevParentDocID+1);
+ if (values.docID() <= prevParentDocID) {
+ values.advance(prevParentDocID+1);
+ }
return nextDoc();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ea121223/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java
----------------------------------------------------------------------
diff --git a/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java b/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java
index d450600..b2958c8 100644
--- a/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/index/AssertingLeafReader.java
@@ -423,7 +423,7 @@ public class AssertingLeafReader extends FilterLeafReader {
public int advance(int target) throws IOException {
assertThread("Numeric doc values", creationThread);
assert target >= 0;
- assert target >= in.docID();
+ assert target > in.docID();
int docID = in.advance(target);
assert docID >= target;
assert docID == NO_MORE_DOCS || docID < maxDoc;
@@ -488,7 +488,7 @@ public class AssertingLeafReader extends FilterLeafReader {
public int advance(int target) throws IOException {
assertThread("Binary doc values", creationThread);
assert target >= 0;
- assert target >= in.docID();
+ assert target > in.docID();
int docID = in.advance(target);
assert docID >= target;
assert docID == NO_MORE_DOCS || docID < maxDoc;
@@ -554,7 +554,7 @@ public class AssertingLeafReader extends FilterLeafReader {
public int advance(int target) throws IOException {
assertThread("Sorted doc values", creationThread);
assert target >= 0;
- assert target >= in.docID();
+ assert target > in.docID();
int docID = in.advance(target);
assert docID >= target;
assert docID == NO_MORE_DOCS || docID < maxDoc;
@@ -648,7 +648,7 @@ public class AssertingLeafReader extends FilterLeafReader {
public int advance(int target) throws IOException {
assertThread("Sorted numeric doc values", creationThread);
assert target >= 0;
- assert target >= in.docID();
+ assert target > in.docID();
int docID = in.advance(target);
assert docID == in.docID();
assert docID >= target;
@@ -720,7 +720,7 @@ public class AssertingLeafReader extends FilterLeafReader {
public int advance(int target) throws IOException {
assertThread("Sorted set doc values", creationThread);
assert target >= 0;
- assert target >= in.docID();
+ assert target > in.docID();
int docID = in.advance(target);
assert docID == in.docID();
assert docID >= target;
[10/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7489: Better
sparsity support for Lucene70DocValuesFormat.
Posted by kr...@apache.org.
LUCENE-7489: Better sparsity support for Lucene70DocValuesFormat.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/927fd51d
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/927fd51d
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/927fd51d
Branch: refs/heads/jira/solr-8593
Commit: 927fd51d64a6e72843018786daea855847416487
Parents: a4a314d
Author: Adrien Grand <jp...@gmail.com>
Authored: Wed Oct 12 12:55:16 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Mon Oct 17 09:28:20 2016 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +
.../lucene54/Lucene54DocValuesProducer.java | 14 +-
.../apache/lucene/codecs/DocValuesConsumer.java | 45 +-
.../lucene70/Lucene70DocValuesConsumer.java | 1001 +++-----
.../lucene70/Lucene70DocValuesFormat.java | 123 +-
.../lucene70/Lucene70DocValuesProducer.java | 2229 ++++++------------
.../lucene/index/EmptyDocValuesProducer.java | 4 +-
.../lucene/index/LegacySortedSetDocValues.java | 4 +-
.../index/SingletonSortedSetDocValues.java | 2 +-
.../apache/lucene/index/SortedDocValues.java | 2 +-
.../apache/lucene/index/SortedSetDocValues.java | 2 +-
.../lucene/search/DocValuesRewriteMethod.java | 2 +-
.../util/packed/DirectMonotonicReader.java | 2 -
.../lucene70/TestLucene70DocValuesFormat.java | 116 +-
14 files changed, 1240 insertions(+), 2309 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 1981c52..54329ab 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -29,6 +29,9 @@ Bug Fixes
Improvements
+* LUCENE-7489: Better storage of sparse doc-values fields with the default
+ codec. (Adrien Grand)
+
Optimizations
* LUCENE-7416: BooleanQuery optimizes queries that have queries that occur both
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java b/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java
index 25f4b5e..8a44c31 100644
--- a/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java
+++ b/lucene/backward-codecs/src/java/org/apache/lucene/codecs/lucene54/Lucene54DocValuesProducer.java
@@ -928,7 +928,7 @@ final class Lucene54DocValuesProducer extends DocValuesProducer implements Close
}
@Override
- public TermsEnum termsEnum() {
+ public TermsEnum termsEnum() throws IOException {
if (binary instanceof CompressedBinaryDocValues) {
return ((CompressedBinaryDocValues)binary).getTermsEnum();
} else {
@@ -1233,7 +1233,7 @@ final class Lucene54DocValuesProducer extends DocValuesProducer implements Close
}
@Override
- public TermsEnum termsEnum() {
+ public TermsEnum termsEnum() throws IOException {
if (binary instanceof CompressedBinaryDocValues) {
return ((CompressedBinaryDocValues)binary).getTermsEnum();
} else {
@@ -1292,7 +1292,7 @@ final class Lucene54DocValuesProducer extends DocValuesProducer implements Close
}
@Override
- public TermsEnum termsEnum() {
+ public TermsEnum termsEnum() throws IOException {
if (binary instanceof CompressedBinaryDocValues) {
return ((CompressedBinaryDocValues) binary).getTermsEnum();
} else {
@@ -1490,12 +1490,8 @@ final class Lucene54DocValuesProducer extends DocValuesProducer implements Close
}
}
- TermsEnum getTermsEnum() {
- try {
- return getTermsEnum(data.clone());
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
+ TermsEnum getTermsEnum() throws IOException {
+ return getTermsEnum(data.clone());
}
private CompressedBinaryTermsEnum getTermsEnum(IndexInput input) throws IOException {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java b/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java
index a07cc3d..e04d5b9 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/DocValuesConsumer.java
@@ -361,7 +361,7 @@ public abstract class DocValuesConsumer implements Closeable {
addSortedNumericField(mergeFieldInfo,
new EmptyDocValuesProducer() {
@Override
- public SortedNumericDocValues getSortedNumeric(FieldInfo fieldInfo) {
+ public SortedNumericDocValues getSortedNumeric(FieldInfo fieldInfo) throws IOException {
if (fieldInfo != mergeFieldInfo) {
throw new IllegalArgumentException("wrong FieldInfo");
}
@@ -375,11 +375,7 @@ public abstract class DocValuesConsumer implements Closeable {
if (docValuesProducer != null) {
FieldInfo readerFieldInfo = mergeState.fieldInfos[i].fieldInfo(mergeFieldInfo.name);
if (readerFieldInfo != null && readerFieldInfo.getDocValuesType() == DocValuesType.SORTED_NUMERIC) {
- try {
- values = docValuesProducer.getSortedNumeric(readerFieldInfo);
- } catch (IOException ioe) {
- throw new RuntimeException(ioe);
- }
+ values = docValuesProducer.getSortedNumeric(readerFieldInfo);
}
}
if (values == null) {
@@ -391,12 +387,7 @@ public abstract class DocValuesConsumer implements Closeable {
final long finalCost = cost;
- final DocIDMerger<SortedNumericDocValuesSub> docIDMerger;
- try {
- docIDMerger = new DocIDMerger<>(subs, mergeState.segmentInfo.getIndexSort() != null);
- } catch (IOException ioe) {
- throw new RuntimeException(ioe);
- }
+ final DocIDMerger<SortedNumericDocValuesSub> docIDMerger = new DocIDMerger<>(subs, mergeState.segmentInfo.getIndexSort() != null);
return new SortedNumericDocValues() {
@@ -521,7 +512,7 @@ public abstract class DocValuesConsumer implements Closeable {
addSortedField(fieldInfo,
new EmptyDocValuesProducer() {
@Override
- public SortedDocValues getSorted(FieldInfo fieldInfoIn) {
+ public SortedDocValues getSorted(FieldInfo fieldInfoIn) throws IOException {
if (fieldInfoIn != fieldInfo) {
throw new IllegalArgumentException("wrong FieldInfo");
}
@@ -536,11 +527,7 @@ public abstract class DocValuesConsumer implements Closeable {
if (docValuesProducer != null) {
FieldInfo readerFieldInfo = mergeState.fieldInfos[i].fieldInfo(fieldInfo.name);
if (readerFieldInfo != null && readerFieldInfo.getDocValuesType() == DocValuesType.SORTED) {
- try {
- values = docValuesProducer.getSorted(readerFieldInfo);
- } catch (IOException ioe) {
- throw new RuntimeException(ioe);
- }
+ values = docValuesProducer.getSorted(readerFieldInfo);
}
}
if (values == null) {
@@ -553,12 +540,7 @@ public abstract class DocValuesConsumer implements Closeable {
final long finalCost = cost;
- final DocIDMerger<SortedDocValuesSub> docIDMerger;
- try {
- docIDMerger = new DocIDMerger<>(subs, mergeState.segmentInfo.getIndexSort() != null);
- } catch (IOException ioe) {
- throw new RuntimeException(ioe);
- }
+ final DocIDMerger<SortedDocValuesSub> docIDMerger = new DocIDMerger<>(subs, mergeState.segmentInfo.getIndexSort() != null);
return new SortedDocValues() {
private int docID = -1;
@@ -693,7 +675,7 @@ public abstract class DocValuesConsumer implements Closeable {
addSortedSetField(mergeFieldInfo,
new EmptyDocValuesProducer() {
@Override
- public SortedSetDocValues getSortedSet(FieldInfo fieldInfo) {
+ public SortedSetDocValues getSortedSet(FieldInfo fieldInfo) throws IOException {
if (fieldInfo != mergeFieldInfo) {
throw new IllegalArgumentException("wrong FieldInfo");
}
@@ -709,11 +691,7 @@ public abstract class DocValuesConsumer implements Closeable {
if (docValuesProducer != null) {
FieldInfo readerFieldInfo = mergeState.fieldInfos[i].fieldInfo(mergeFieldInfo.name);
if (readerFieldInfo != null && readerFieldInfo.getDocValuesType() == DocValuesType.SORTED_SET) {
- try {
- values = docValuesProducer.getSortedSet(readerFieldInfo);
- } catch (IOException ioe) {
- throw new RuntimeException(ioe);
- }
+ values = docValuesProducer.getSortedSet(readerFieldInfo);
}
}
if (values == null) {
@@ -723,12 +701,7 @@ public abstract class DocValuesConsumer implements Closeable {
subs.add(new SortedSetDocValuesSub(mergeState.docMaps[i], values, map.getGlobalOrds(i)));
}
- final DocIDMerger<SortedSetDocValuesSub> docIDMerger;
- try {
- docIDMerger = new DocIDMerger<>(subs, mergeState.segmentInfo.getIndexSort() != null);
- } catch (IOException ioe) {
- throw new RuntimeException(ioe);
- }
+ final DocIDMerger<SortedSetDocValuesSub> docIDMerger = new DocIDMerger<>(subs, mergeState.segmentInfo.getIndexSort() != null);
final long finalCost = cost;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesConsumer.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesConsumer.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesConsumer.java
index 564db7c..e1b66e1 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesConsumer.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesConsumer.java
@@ -17,56 +17,47 @@
package org.apache.lucene.codecs.lucene70;
+import static org.apache.lucene.codecs.lucene70.Lucene70DocValuesFormat.DIRECT_MONOTONIC_BLOCK_SHIFT;
+
import java.io.Closeable; // javadocs
import java.io.IOException;
import java.util.Arrays;
-import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
-import java.util.Iterator;
import java.util.Map;
import java.util.Set;
-import java.util.SortedSet;
-import java.util.TreeSet;
-import java.util.stream.StreamSupport;
import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.codecs.DocValuesConsumer;
import org.apache.lucene.codecs.DocValuesProducer;
-import org.apache.lucene.codecs.LegacyDocValuesIterables;
+import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.EmptyDocValuesProducer;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.IndexFileNames;
import org.apache.lucene.index.SegmentWriteState;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.index.SortedSetDocValues;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.SortedSetSelector;
import org.apache.lucene.store.IndexOutput;
import org.apache.lucene.store.RAMOutputStream;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.BytesRefBuilder;
import org.apache.lucene.util.IOUtils;
-import org.apache.lucene.util.LongsRef;
import org.apache.lucene.util.MathUtil;
-import org.apache.lucene.util.PagedBytes.PagedBytesDataInput;
-import org.apache.lucene.util.PagedBytes;
import org.apache.lucene.util.StringHelper;
import org.apache.lucene.util.packed.DirectMonotonicWriter;
import org.apache.lucene.util.packed.DirectWriter;
-import org.apache.lucene.util.packed.MonotonicBlockPackedWriter;
-import org.apache.lucene.util.packed.PackedInts;
-
-import static org.apache.lucene.codecs.lucene70.Lucene70DocValuesFormat.*;
/** writer for {@link Lucene70DocValuesFormat} */
final class Lucene70DocValuesConsumer extends DocValuesConsumer implements Closeable {
- enum NumberType {
- /** Dense ordinals */
- ORDINAL,
- /** Random long values */
- VALUE;
- }
-
IndexOutput data, meta;
final int maxDoc;
-
+
/** expert: Creates a new writer */
public Lucene70DocValuesConsumer(SegmentWriteState state, String dataCodec, String dataExtension, String metaCodec, String metaExtension) throws IOException {
boolean success = false;
@@ -85,37 +76,53 @@ final class Lucene70DocValuesConsumer extends DocValuesConsumer implements Close
}
}
}
-
+
+ @Override
+ public void close() throws IOException {
+ boolean success = false;
+ try {
+ if (meta != null) {
+ meta.writeInt(-1); // write EOF marker
+ CodecUtil.writeFooter(meta); // write checksum
+ }
+ if (data != null) {
+ CodecUtil.writeFooter(data); // write checksum
+ }
+ success = true;
+ } finally {
+ if (success) {
+ IOUtils.close(data, meta);
+ } else {
+ IOUtils.closeWhileHandlingException(data, meta);
+ }
+ meta = data = null;
+ }
+ }
+
@Override
public void addNumericField(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {
- addNumericField(field, LegacyDocValuesIterables.numericIterable(field, valuesProducer, maxDoc), NumberType.VALUE);
+ meta.writeInt(field.number);
+ meta.writeByte(Lucene70DocValuesFormat.NUMERIC);
+
+ writeValues(field, new EmptyDocValuesProducer() {
+ @Override
+ public SortedNumericDocValues getSortedNumeric(FieldInfo field) throws IOException {
+ return DocValues.singleton(valuesProducer.getNumeric(field));
+ }
+ });
}
- void addNumericField(FieldInfo field, Iterable<Number> values, NumberType numberType) throws IOException {
- long count = 0;
- long minValue = Long.MAX_VALUE;
- long maxValue = Long.MIN_VALUE;
+ private long[] writeValues(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {
+ SortedNumericDocValues values = valuesProducer.getSortedNumeric(field);
+ int numDocsWithValue = 0;
+ long numValues = 0;
+ long min = Long.MAX_VALUE;
+ long max = Long.MIN_VALUE;
long gcd = 0;
- long missingCount = 0;
- long zeroCount = 0;
- // TODO: more efficient?
- HashSet<Long> uniqueValues = null;
- long missingOrdCount = 0;
- if (numberType == NumberType.VALUE) {
- uniqueValues = new HashSet<>();
-
- for (Number nv : values) {
- final long v;
- if (nv == null) {
- v = 0;
- missingCount++;
- zeroCount++;
- } else {
- v = nv.longValue();
- if (v == 0) {
- zeroCount++;
- }
- }
+ Set<Long> uniqueValues = new HashSet<>();
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ for (int i = 0, count = values.docValueCount(); i < count; ++i) {
+ long v = values.nextValue();
if (gcd != 1) {
if (v < Long.MIN_VALUE / 2 || v > Long.MAX_VALUE / 2) {
@@ -123,675 +130,393 @@ final class Lucene70DocValuesConsumer extends DocValuesConsumer implements Close
// wrong results. Since these extreme values are unlikely, we just discard
// GCD computation for them
gcd = 1;
- } else if (count != 0) { // minValue needs to be set first
- gcd = MathUtil.gcd(gcd, v - minValue);
+ } else if (numValues != 0) { // minValue needs to be set first
+ gcd = MathUtil.gcd(gcd, v - min);
}
}
- minValue = Math.min(minValue, v);
- maxValue = Math.max(maxValue, v);
+ min = Math.min(min, v);
+ max = Math.max(max, v);
- if (uniqueValues != null) {
- if (uniqueValues.add(v)) {
- if (uniqueValues.size() > 256) {
- uniqueValues = null;
- }
- }
+ if (uniqueValues != null
+ && uniqueValues.add(v)
+ && uniqueValues.size() > 256) {
+ uniqueValues = null;
}
- ++count;
- }
- } else {
- for (Number nv : values) {
- long v = nv.longValue();
- if (v == -1L) {
- missingOrdCount++;
- }
- minValue = Math.min(minValue, v);
- maxValue = Math.max(maxValue, v);
- ++count;
+ numValues++;
}
- }
-
- final long delta = maxValue - minValue;
- final int deltaBitsRequired = DirectWriter.unsignedBitsRequired(delta);
- final int tableBitsRequired = uniqueValues == null
- ? Integer.MAX_VALUE
- : DirectWriter.bitsRequired(uniqueValues.size() - 1);
-
- final boolean sparse; // 10% of docs or less have a value
- switch (numberType) {
- case VALUE:
- sparse = (double) missingCount / count >= 0.90;
- break;
- case ORDINAL:
- sparse = (double) missingOrdCount / count >= 0.90;
- break;
- default:
- throw new AssertionError();
+
+ numDocsWithValue++;
}
- final int format;
- if (uniqueValues != null
- && count <= Integer.MAX_VALUE
- && (uniqueValues.size() == 1
- || (uniqueValues.size() == 2 && missingCount > 0 && zeroCount == missingCount))) {
- // either one unique value C or two unique values: "missing" and C
- format = CONST_COMPRESSED;
- } else if (sparse && count >= 1024) {
- // require at least 1024 docs to avoid flipping back and forth when doing NRT search
- format = SPARSE_COMPRESSED;
- } else if (uniqueValues != null && tableBitsRequired < deltaBitsRequired) {
- format = TABLE_COMPRESSED;
- } else if (gcd != 0 && gcd != 1) {
- final long gcdDelta = (maxValue - minValue) / gcd;
- final long gcdBitsRequired = DirectWriter.unsignedBitsRequired(gcdDelta);
- format = gcdBitsRequired < deltaBitsRequired ? GCD_COMPRESSED : DELTA_COMPRESSED;
+ if (numDocsWithValue == 0) {
+ meta.writeLong(-2);
+ meta.writeLong(0L);
+ } else if (numDocsWithValue == maxDoc) {
+ meta.writeLong(-1);
+ meta.writeLong(0L);
} else {
- format = DELTA_COMPRESSED;
+ long offset = data.getFilePointer();
+ meta.writeLong(offset);
+ values = valuesProducer.getSortedNumeric(field);
+ IndexedDISI.writeBitSet(values, data);
+ meta.writeLong(data.getFilePointer() - offset);
}
- meta.writeVInt(field.number);
- meta.writeByte(Lucene70DocValuesFormat.NUMERIC);
- meta.writeVInt(format);
- if (format == SPARSE_COMPRESSED) {
- meta.writeLong(data.getFilePointer());
- final long numDocsWithValue;
- switch (numberType) {
- case VALUE:
- numDocsWithValue = count - missingCount;
- break;
- case ORDINAL:
- numDocsWithValue = count - missingOrdCount;
- break;
- default:
- throw new AssertionError();
- }
- final long maxDoc = writeSparseMissingBitset(values, numberType, numDocsWithValue);
- assert maxDoc == count;
- } else if (missingCount == 0) {
- meta.writeLong(ALL_LIVE);
- } else if (missingCount == count) {
- meta.writeLong(ALL_MISSING);
+
+ meta.writeLong(numValues);
+ final int numBitsPerValue;
+ Map<Long, Integer> encode = null;
+ if (min >= max) {
+ numBitsPerValue = 0;
+ meta.writeInt(-1);
} else {
- meta.writeLong(data.getFilePointer());
- writeMissingBitset(values);
- }
- meta.writeLong(data.getFilePointer());
- meta.writeVLong(count);
-
- switch (format) {
- case CONST_COMPRESSED:
- // write the constant (nonzero value in the n=2 case, singleton value otherwise)
- meta.writeLong(minValue < 0 ? Collections.min(uniqueValues) : Collections.max(uniqueValues));
- break;
- case GCD_COMPRESSED:
- meta.writeLong(minValue);
- meta.writeLong(gcd);
- final long maxDelta = (maxValue - minValue) / gcd;
- final int bits = DirectWriter.unsignedBitsRequired(maxDelta);
- meta.writeVInt(bits);
- final DirectWriter quotientWriter = DirectWriter.getInstance(data, count, bits);
- for (Number nv : values) {
- long value = nv == null ? 0 : nv.longValue();
- quotientWriter.add((value - minValue) / gcd);
- }
- quotientWriter.finish();
- break;
- case DELTA_COMPRESSED:
- final long minDelta = delta < 0 ? 0 : minValue;
- meta.writeLong(minDelta);
- meta.writeVInt(deltaBitsRequired);
- final DirectWriter writer = DirectWriter.getInstance(data, count, deltaBitsRequired);
- for (Number nv : values) {
- long v = nv == null ? 0 : nv.longValue();
- writer.add(v - minDelta);
+ if (uniqueValues != null
+ && uniqueValues.size() > 1
+ && DirectWriter.unsignedBitsRequired(uniqueValues.size() - 1) < DirectWriter.unsignedBitsRequired((max - min) / gcd)) {
+ numBitsPerValue = DirectWriter.unsignedBitsRequired(uniqueValues.size() - 1);
+ final Long[] sortedUniqueValues = uniqueValues.toArray(new Long[0]);
+ Arrays.sort(sortedUniqueValues);
+ meta.writeInt(sortedUniqueValues.length);
+ for (Long v : sortedUniqueValues) {
+ meta.writeLong(v);
}
- writer.finish();
- break;
- case TABLE_COMPRESSED:
- final Long[] decode = uniqueValues.toArray(new Long[uniqueValues.size()]);
- Arrays.sort(decode);
- final HashMap<Long,Integer> encode = new HashMap<>();
- meta.writeVInt(decode.length);
- for (int i = 0; i < decode.length; i++) {
- meta.writeLong(decode[i]);
- encode.put(decode[i], i);
+ encode = new HashMap<>();
+ for (int i = 0; i < sortedUniqueValues.length; ++i) {
+ encode.put(sortedUniqueValues[i], i);
}
- meta.writeVInt(tableBitsRequired);
- final DirectWriter ordsWriter = DirectWriter.getInstance(data, count, tableBitsRequired);
- for (Number nv : values) {
- ordsWriter.add(encode.get(nv == null ? 0 : nv.longValue()));
- }
- ordsWriter.finish();
- break;
- case SPARSE_COMPRESSED:
- final Iterable<Number> filteredMissingValues;
- switch (numberType) {
- case VALUE:
- meta.writeByte((byte) 0);
- filteredMissingValues = new Iterable<Number>() {
- @Override
- public Iterator<Number> iterator() {
- return StreamSupport
- .stream(values.spliterator(), false)
- .filter(value -> value != null)
- .iterator();
- }
- };
- break;
- case ORDINAL:
- meta.writeByte((byte) 1);
- filteredMissingValues = new Iterable<Number>() {
- @Override
- public Iterator<Number> iterator() {
- return StreamSupport
- .stream(values.spliterator(), false)
- .filter(value -> value.longValue() != -1L)
- .iterator();
- }
- };
- break;
- default:
- throw new AssertionError();
+ min = 0;
+ gcd = 1;
+ } else {
+ uniqueValues = null;
+ numBitsPerValue = DirectWriter.unsignedBitsRequired((max - min) / gcd);
+ if (gcd == 1 && min > 0
+ && DirectWriter.unsignedBitsRequired(max) == DirectWriter.unsignedBitsRequired(max - min)) {
+ min = 0;
}
- // Write non-missing values as a numeric field
- addNumericField(field, filteredMissingValues, numberType);
- break;
- default:
- throw new AssertionError();
- }
- meta.writeLong(data.getFilePointer());
- }
-
- // TODO: in some cases representing missing with minValue-1 wouldn't take up additional space and so on,
- // but this is very simple, and algorithms only check this for values of 0 anyway (doesnt slow down normal decode)
- void writeMissingBitset(Iterable<?> values) throws IOException {
- long bits = 0;
- int count = 0;
- for (Object v : values) {
- if (count == 64) {
- data.writeLong(bits);
- count = 0;
- bits = 0;
- }
- if (v != null) {
- bits |= 1L << count;
+ meta.writeInt(-1);
}
- count++;
- }
- if (count > 0) {
- data.writeLong(bits);
}
- }
-
- long writeSparseMissingBitset(Iterable<Number> values, NumberType numberType, long numDocsWithValue) throws IOException {
- meta.writeVLong(numDocsWithValue);
- // Write doc IDs that have a value
- meta.writeVInt(DIRECT_MONOTONIC_BLOCK_SHIFT);
- final DirectMonotonicWriter docIdsWriter = DirectMonotonicWriter.getInstance(meta, data, numDocsWithValue, DIRECT_MONOTONIC_BLOCK_SHIFT);
- long docID = 0;
- for (Number nv : values) {
- switch (numberType) {
- case VALUE:
- if (nv != null) {
- docIdsWriter.add(docID);
- }
- break;
- case ORDINAL:
- if (nv.longValue() != -1L) {
- docIdsWriter.add(docID);
+ meta.writeByte((byte) numBitsPerValue);
+ meta.writeLong(min);
+ meta.writeLong(gcd);
+ long startOffset = data.getFilePointer();
+ meta.writeLong(startOffset);
+ if (numBitsPerValue != 0) {
+ values = valuesProducer.getSortedNumeric(field);
+ DirectWriter writer = DirectWriter.getInstance(data, numValues, numBitsPerValue);
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ for (int i = 0, count = values.docValueCount(); i < count; ++i) {
+ long v = values.nextValue();
+ if (encode == null) {
+ writer.add((v - min) / gcd);
+ } else {
+ writer.add(encode.get(v));
}
- break;
- default:
- throw new AssertionError();
+ }
}
- docID++;
+ writer.finish();
}
- docIdsWriter.finish();
- return docID;
+ meta.writeLong(data.getFilePointer() - startOffset);
+
+ return new long[] {numDocsWithValue, numValues};
}
@Override
public void addBinaryField(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {
- addBinaryField(field, LegacyDocValuesIterables.binaryIterable(field, valuesProducer, maxDoc));
- }
-
- private void addBinaryField(FieldInfo field, Iterable<BytesRef> values) throws IOException {
- // write the byte[] data
- meta.writeVInt(field.number);
+ meta.writeInt(field.number);
meta.writeByte(Lucene70DocValuesFormat.BINARY);
+
+ BinaryDocValues values = valuesProducer.getBinary(field);
+ long start = data.getFilePointer();
+ meta.writeLong(start);
+ int numDocsWithField = 0;
int minLength = Integer.MAX_VALUE;
- int maxLength = Integer.MIN_VALUE;
- final long startFP = data.getFilePointer();
- long count = 0;
- long missingCount = 0;
- for(BytesRef v : values) {
- final int length;
- if (v == null) {
- length = 0;
- missingCount++;
- } else {
- length = v.length;
- }
- minLength = Math.min(minLength, length);
- maxLength = Math.max(maxLength, length);
- if (v != null) {
- data.writeBytes(v.bytes, v.offset, v.length);
- }
- count++;
+ int maxLength = 0;
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ numDocsWithField++;
+ BytesRef v = values.binaryValue();
+ int length = v.length;
+ data.writeBytes(v.bytes, v.offset, v.length);
+ minLength = Math.min(length, minLength);
+ maxLength = Math.max(length, maxLength);
}
- meta.writeVInt(minLength == maxLength ? BINARY_FIXED_UNCOMPRESSED : BINARY_VARIABLE_UNCOMPRESSED);
- if (missingCount == 0) {
- meta.writeLong(ALL_LIVE);
- } else if (missingCount == count) {
- meta.writeLong(ALL_MISSING);
+ assert numDocsWithField <= maxDoc;
+ meta.writeLong(data.getFilePointer() - start);
+
+ if (numDocsWithField == 0) {
+ meta.writeLong(-2);
+ meta.writeLong(0L);
+ } else if (numDocsWithField == maxDoc) {
+ meta.writeLong(-1);
+ meta.writeLong(0L);
} else {
- meta.writeLong(data.getFilePointer());
- writeMissingBitset(values);
+ long offset = data.getFilePointer();
+ meta.writeLong(offset);
+ values = valuesProducer.getBinary(field);
+ IndexedDISI.writeBitSet(values, data);
+ meta.writeLong(data.getFilePointer() - offset);
}
- meta.writeVInt(minLength);
- meta.writeVInt(maxLength);
- meta.writeVLong(count);
- meta.writeLong(startFP);
-
- // if minLength == maxLength, it's a fixed-length byte[], we are done (the addresses are implicit)
- // otherwise, we need to record the length fields...
- if (minLength != maxLength) {
- meta.writeLong(data.getFilePointer());
+
+ meta.writeInt(numDocsWithField);
+ meta.writeInt(minLength);
+ meta.writeInt(maxLength);
+ if (maxLength > minLength) {
+ start = data.getFilePointer();
+ meta.writeLong(start);
meta.writeVInt(DIRECT_MONOTONIC_BLOCK_SHIFT);
- final DirectMonotonicWriter writer = DirectMonotonicWriter.getInstance(meta, data, count + 1, DIRECT_MONOTONIC_BLOCK_SHIFT);
+ final DirectMonotonicWriter writer = DirectMonotonicWriter.getInstance(meta, data, numDocsWithField + 1, DIRECT_MONOTONIC_BLOCK_SHIFT);
long addr = 0;
writer.add(addr);
- for (BytesRef v : values) {
- if (v != null) {
- addr += v.length;
- }
+ values = valuesProducer.getBinary(field);
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ addr += values.binaryValue().length;
writer.add(addr);
}
writer.finish();
- meta.writeLong(data.getFilePointer());
+ meta.writeLong(data.getFilePointer() - start);
}
}
-
- /** expert: writes a value dictionary for a sorted/sortedset field */
- private void addTermsDict(FieldInfo field, final Iterable<BytesRef> values) throws IOException {
- // first check if it's a "fixed-length" terms dict, and compressibility if so
- int minLength = Integer.MAX_VALUE;
- int maxLength = Integer.MIN_VALUE;
- long numValues = 0;
- BytesRefBuilder previousValue = new BytesRefBuilder();
- long prefixSum = 0; // only valid for fixed-width data, as we have a choice there
- for (BytesRef v : values) {
- minLength = Math.min(minLength, v.length);
- maxLength = Math.max(maxLength, v.length);
- if (minLength == maxLength) {
- int termPosition = (int) (numValues & INTERVAL_MASK);
- if (termPosition == 0) {
- // first term in block, save it away to compare against the last term later
- previousValue.copyBytes(v);
- } else if (termPosition == INTERVAL_COUNT - 1) {
- // last term in block, accumulate shared prefix against first term
- prefixSum += StringHelper.bytesDifference(previousValue.get(), v);
- }
- }
- numValues++;
+
+ @Override
+ public void addSortedField(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {
+ meta.writeInt(field.number);
+ meta.writeByte(Lucene70DocValuesFormat.SORTED);
+ doAddSortedField(field, valuesProducer);
+ }
+
+ private void doAddSortedField(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {
+ SortedDocValues values = valuesProducer.getSorted(field);
+ int numDocsWithField = 0;
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ numDocsWithField++;
}
- // for fixed width data, look at the avg(shared prefix) before deciding how to encode:
- // prefix compression "costs" worst case 2 bytes per term because we must store suffix lengths.
- // so if we share at least 3 bytes on average, always compress.
- if (minLength == maxLength && prefixSum <= 3*(numValues >> INTERVAL_SHIFT)) {
- // no index needed: not very compressible, direct addressing by mult
- addBinaryField(field, values);
- } else if (numValues < REVERSE_INTERVAL_COUNT) {
- // low cardinality: waste a few KB of ram, but can't really use fancy index etc
- addBinaryField(field, values);
+
+ if (numDocsWithField == 0) {
+ meta.writeLong(-2);
+ meta.writeLong(0L);
+ } else if (numDocsWithField == maxDoc) {
+ meta.writeLong(-1);
+ meta.writeLong(0L);
} else {
- assert numValues > 0; // we don't have to handle the empty case
- // header
- meta.writeVInt(field.number);
- meta.writeByte(Lucene70DocValuesFormat.BINARY);
- meta.writeVInt(BINARY_PREFIX_COMPRESSED);
- meta.writeLong(-1L);
- // now write the bytes: sharing prefixes within a block
- final long startFP = data.getFilePointer();
- // currently, we have to store the delta from expected for every 1/nth term
- // we could avoid this, but it's not much and less overall RAM than the previous approach!
- RAMOutputStream addressBuffer = new RAMOutputStream();
- MonotonicBlockPackedWriter termAddresses = new MonotonicBlockPackedWriter(addressBuffer, MONOTONIC_BLOCK_SIZE);
- // buffers up 16 terms
- RAMOutputStream bytesBuffer = new RAMOutputStream();
- // buffers up block header
- RAMOutputStream headerBuffer = new RAMOutputStream();
- BytesRefBuilder lastTerm = new BytesRefBuilder();
- lastTerm.grow(maxLength);
- long count = 0;
- int suffixDeltas[] = new int[INTERVAL_COUNT];
- for (BytesRef v : values) {
- int termPosition = (int) (count & INTERVAL_MASK);
- if (termPosition == 0) {
- termAddresses.add(data.getFilePointer() - startFP);
- // abs-encode first term
- headerBuffer.writeVInt(v.length);
- headerBuffer.writeBytes(v.bytes, v.offset, v.length);
- lastTerm.copyBytes(v);
- } else {
- // prefix-code: we only share at most 255 characters, to encode the length as a single
- // byte and have random access. Larger terms just get less compression.
- int sharedPrefix = Math.min(255, StringHelper.bytesDifference(lastTerm.get(), v));
- bytesBuffer.writeByte((byte) sharedPrefix);
- bytesBuffer.writeBytes(v.bytes, v.offset + sharedPrefix, v.length - sharedPrefix);
- // we can encode one smaller, because terms are unique.
- suffixDeltas[termPosition] = v.length - sharedPrefix - 1;
- }
-
- count++;
- // flush block
- if ((count & INTERVAL_MASK) == 0) {
- flushTermsDictBlock(headerBuffer, bytesBuffer, suffixDeltas);
- }
- }
- // flush trailing crap
- int leftover = (int) (count & INTERVAL_MASK);
- if (leftover > 0) {
- Arrays.fill(suffixDeltas, leftover, suffixDeltas.length, 0);
- flushTermsDictBlock(headerBuffer, bytesBuffer, suffixDeltas);
- }
- final long indexStartFP = data.getFilePointer();
- // write addresses of indexed terms
- termAddresses.finish();
- addressBuffer.writeTo(data);
- addressBuffer = null;
- termAddresses = null;
- meta.writeVInt(minLength);
- meta.writeVInt(maxLength);
- meta.writeVLong(count);
- meta.writeLong(startFP);
- meta.writeLong(indexStartFP);
- meta.writeVInt(PackedInts.VERSION_CURRENT);
- meta.writeVInt(MONOTONIC_BLOCK_SIZE);
- addReverseTermIndex(field, values, maxLength);
- }
- }
- // writes term dictionary "block"
- // first term is absolute encoded as vint length + bytes.
- // lengths of subsequent N terms are encoded as either N bytes or N shorts.
- // in the double-byte case, the first byte is indicated with -1.
- // subsequent terms are encoded as byte suffixLength + bytes.
- private void flushTermsDictBlock(RAMOutputStream headerBuffer, RAMOutputStream bytesBuffer, int suffixDeltas[]) throws IOException {
- boolean twoByte = false;
- for (int i = 1; i < suffixDeltas.length; i++) {
- if (suffixDeltas[i] > 254) {
- twoByte = true;
- }
+ long offset = data.getFilePointer();
+ meta.writeLong(offset);
+ values = valuesProducer.getSorted(field);
+ IndexedDISI.writeBitSet(values, data);
+ meta.writeLong(data.getFilePointer() - offset);
}
- if (twoByte) {
- headerBuffer.writeByte((byte)255);
- for (int i = 1; i < suffixDeltas.length; i++) {
- headerBuffer.writeShort((short) suffixDeltas[i]);
- }
+
+ meta.writeInt(numDocsWithField);
+ if (values.getValueCount() <= 1) {
+ meta.writeByte((byte) 0);
+ meta.writeLong(0L);
+ meta.writeLong(0L);
} else {
- for (int i = 1; i < suffixDeltas.length; i++) {
- headerBuffer.writeByte((byte) suffixDeltas[i]);
+ int numberOfBitsPerOrd = DirectWriter.unsignedBitsRequired(values.getValueCount() - 1);
+ meta.writeByte((byte) numberOfBitsPerOrd);
+ long start = data.getFilePointer();
+ meta.writeLong(start);
+ DirectWriter writer = DirectWriter.getInstance(data, numDocsWithField, numberOfBitsPerOrd);
+ values = valuesProducer.getSorted(field);
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ writer.add(values.ordValue());
}
+ writer.finish();
+ meta.writeLong(data.getFilePointer() - start);
}
- headerBuffer.writeTo(data);
- headerBuffer.reset();
- bytesBuffer.writeTo(data);
- bytesBuffer.reset();
+
+ addTermsDict(DocValues.singleton(valuesProducer.getSorted(field)));
}
-
- // writes reverse term index: used for binary searching a term into a range of 64 blocks
- // for every 64 blocks (1024 terms) we store a term, trimming any suffix unnecessary for comparison
- // terms are written as a contiguous byte[], but never spanning 2^15 byte boundaries.
- private void addReverseTermIndex(FieldInfo field, final Iterable<BytesRef> values, int maxLength) throws IOException {
- long count = 0;
- BytesRefBuilder priorTerm = new BytesRefBuilder();
- priorTerm.grow(maxLength);
- BytesRef indexTerm = new BytesRef();
- long startFP = data.getFilePointer();
- PagedBytes pagedBytes = new PagedBytes(15);
- MonotonicBlockPackedWriter addresses = new MonotonicBlockPackedWriter(data, MONOTONIC_BLOCK_SIZE);
-
- for (BytesRef b : values) {
- int termPosition = (int) (count & REVERSE_INTERVAL_MASK);
- if (termPosition == 0) {
- int len = StringHelper.sortKeyLength(priorTerm.get(), b);
- indexTerm.bytes = b.bytes;
- indexTerm.offset = b.offset;
- indexTerm.length = len;
- addresses.add(pagedBytes.copyUsingLengthPrefix(indexTerm));
- } else if (termPosition == REVERSE_INTERVAL_MASK) {
- priorTerm.copyBytes(b);
+
+ private void addTermsDict(SortedSetDocValues values) throws IOException {
+ final long size = values.getValueCount();
+ meta.writeVLong(size);
+ meta.writeInt(Lucene70DocValuesFormat.TERMS_DICT_BLOCK_SHIFT);
+
+ RAMOutputStream addressBuffer = new RAMOutputStream();
+ meta.writeInt(DIRECT_MONOTONIC_BLOCK_SHIFT);
+ long numBlocks = (size + Lucene70DocValuesFormat.TERMS_DICT_BLOCK_MASK) >>> Lucene70DocValuesFormat.TERMS_DICT_BLOCK_SHIFT;
+ DirectMonotonicWriter writer = DirectMonotonicWriter.getInstance(meta, addressBuffer, numBlocks, DIRECT_MONOTONIC_BLOCK_SHIFT);
+
+ BytesRefBuilder previous = new BytesRefBuilder();
+ long ord = 0;
+ long start = data.getFilePointer();
+ int maxLength = 0;
+ TermsEnum iterator = values.termsEnum();
+ for (BytesRef term = iterator.next(); term != null; term = iterator.next()) {
+ if ((ord & Lucene70DocValuesFormat.TERMS_DICT_BLOCK_MASK) == 0) {
+ writer.add(data.getFilePointer() - start);
+ data.writeVInt(term.length);
+ data.writeBytes(term.bytes, term.offset, term.length);
+ } else {
+ final int prefixLength = StringHelper.bytesDifference(previous.get(), term);
+ final int suffixLength = term.length - prefixLength;
+ assert suffixLength > 0; // terms are unique
+
+ data.writeByte((byte) (Math.min(prefixLength, 15) | (Math.min(15, suffixLength - 1) << 4)));
+ if (prefixLength >= 15) {
+ data.writeVInt(prefixLength - 15);
+ }
+ if (suffixLength >= 16) {
+ data.writeVInt(suffixLength - 16);
+ }
+ data.writeBytes(term.bytes, term.offset + prefixLength, term.length - prefixLength);
}
- count++;
+ maxLength = Math.max(maxLength, term.length);
+ previous.copyBytes(term);
+ ++ord;
}
- addresses.finish();
- long numBytes = pagedBytes.getPointer();
- pagedBytes.freeze(true);
- PagedBytesDataInput in = pagedBytes.getDataInput();
- meta.writeLong(startFP);
- data.writeVLong(numBytes);
- data.copyBytes(in, numBytes);
- }
-
- @Override
- public void addSortedField(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {
- meta.writeVInt(field.number);
- meta.writeByte(Lucene70DocValuesFormat.SORTED);
- addTermsDict(field, LegacyDocValuesIterables.valuesIterable(valuesProducer.getSorted(field)));
- addNumericField(field, LegacyDocValuesIterables.sortedOrdIterable(valuesProducer, field, maxDoc), NumberType.ORDINAL);
+ writer.finish();
+ meta.writeInt(maxLength);
+ meta.writeLong(start);
+ meta.writeLong(data.getFilePointer() - start);
+ start = data.getFilePointer();
+ addressBuffer.writeTo(data);
+ meta.writeLong(start);
+ meta.writeLong(data.getFilePointer() - start);
+
+ // Now write the reverse terms index
+ writeTermsIndex(values);
}
- private void addSortedField(FieldInfo field, Iterable<BytesRef> values, Iterable<Number> ords) throws IOException {
- meta.writeVInt(field.number);
- meta.writeByte(Lucene70DocValuesFormat.SORTED);
- addTermsDict(field, values);
- addNumericField(field, ords, NumberType.ORDINAL);
+ private void writeTermsIndex(SortedSetDocValues values) throws IOException {
+ final long size = values.getValueCount();
+ meta.writeInt(Lucene70DocValuesFormat.TERMS_DICT_REVERSE_INDEX_SHIFT);
+ long start = data.getFilePointer();
+
+ long numBlocks = 1L + ((size + Lucene70DocValuesFormat.TERMS_DICT_REVERSE_INDEX_MASK) >>> Lucene70DocValuesFormat.TERMS_DICT_REVERSE_INDEX_SHIFT);
+ RAMOutputStream addressBuffer = new RAMOutputStream();
+ DirectMonotonicWriter writer = DirectMonotonicWriter.getInstance(meta, addressBuffer, numBlocks, DIRECT_MONOTONIC_BLOCK_SHIFT);
+
+ TermsEnum iterator = values.termsEnum();
+ BytesRefBuilder previous = new BytesRefBuilder();
+ long offset = 0;
+ long ord = 0;
+ for (BytesRef term = iterator.next(); term != null; term = iterator.next()) {
+ if ((ord & Lucene70DocValuesFormat.TERMS_DICT_REVERSE_INDEX_MASK) == 0) {
+ writer.add(offset);
+ int sortKeyLength = StringHelper.sortKeyLength(previous.get(), term);
+ offset += sortKeyLength;
+ data.writeBytes(term.bytes, term.offset, sortKeyLength);
+ } else if ((ord & Lucene70DocValuesFormat.TERMS_DICT_REVERSE_INDEX_MASK) == Lucene70DocValuesFormat.TERMS_DICT_REVERSE_INDEX_MASK) {
+ previous.copyBytes(term);
+ }
+ ++ord;
+ }
+ writer.add(offset);
+ writer.finish();
+ meta.writeLong(start);
+ meta.writeLong(data.getFilePointer() - start);
+ start = data.getFilePointer();
+ addressBuffer.writeTo(data);
+ meta.writeLong(start);
+ meta.writeLong(data.getFilePointer() - start);
}
@Override
- public void addSortedNumericField(FieldInfo field, final DocValuesProducer valuesProducer) throws IOException {
-
- final Iterable<Number> docToValueCount = LegacyDocValuesIterables.sortedNumericToDocCount(valuesProducer, field, maxDoc);
- final Iterable<Number> values = LegacyDocValuesIterables.sortedNumericToValues(valuesProducer, field);
-
- meta.writeVInt(field.number);
+ public void addSortedNumericField(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {
+ meta.writeInt(field.number);
meta.writeByte(Lucene70DocValuesFormat.SORTED_NUMERIC);
- if (isSingleValued(docToValueCount)) {
- meta.writeVInt(SORTED_SINGLE_VALUED);
- // The field is single-valued, we can encode it as NUMERIC
- addNumericField(field, singletonView(docToValueCount, values, null), NumberType.VALUE);
- } else {
- final SortedSet<LongsRef> uniqueValueSets = uniqueValueSets(docToValueCount, values);
- if (uniqueValueSets != null) {
- meta.writeVInt(SORTED_SET_TABLE);
- // write the set_id -> values mapping
- writeDictionary(uniqueValueSets);
+ long[] stats = writeValues(field, valuesProducer);
+ int numDocsWithField = Math.toIntExact(stats[0]);
+ long numValues = stats[1];
+ assert numValues >= numDocsWithField;
- // write the doc -> set_id as a numeric field
- addNumericField(field, docToSetId(uniqueValueSets, docToValueCount, values), NumberType.ORDINAL);
- } else {
- meta.writeVInt(SORTED_WITH_ADDRESSES);
- // write the stream of values as a numeric field
- addNumericField(field, values, NumberType.VALUE);
- // write the doc -> ord count as a absolute index to the stream
- addOrdIndex(field, docToValueCount);
+ meta.writeInt(numDocsWithField);
+ if (numValues > numDocsWithField) {
+ long start = data.getFilePointer();
+ meta.writeLong(start);
+ meta.writeVInt(DIRECT_MONOTONIC_BLOCK_SHIFT);
+
+ final DirectMonotonicWriter addressesWriter = DirectMonotonicWriter.getInstance(meta, data, numDocsWithField + 1L, DIRECT_MONOTONIC_BLOCK_SHIFT);
+ long addr = 0;
+ addressesWriter.add(addr);
+ SortedNumericDocValues values = valuesProducer.getSortedNumeric(field);
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ addr += values.docValueCount();
+ addressesWriter.add(addr);
}
+ addressesWriter.finish();
+ meta.writeLong(data.getFilePointer() - start);
}
}
@Override
public void addSortedSetField(FieldInfo field, DocValuesProducer valuesProducer) throws IOException {
-
- Iterable<BytesRef> values = LegacyDocValuesIterables.valuesIterable(valuesProducer.getSortedSet(field));
- Iterable<Number> docToOrdCount = LegacyDocValuesIterables.sortedSetOrdCountIterable(valuesProducer, field, maxDoc);
- Iterable<Number> ords = LegacyDocValuesIterables.sortedSetOrdsIterable(valuesProducer, field);
-
- meta.writeVInt(field.number);
+ meta.writeInt(field.number);
meta.writeByte(Lucene70DocValuesFormat.SORTED_SET);
- if (isSingleValued(docToOrdCount)) {
- meta.writeVInt(SORTED_SINGLE_VALUED);
- // The field is single-valued, we can encode it as SORTED
- addSortedField(field, values, singletonView(docToOrdCount, ords, -1L));
- } else {
- final SortedSet<LongsRef> uniqueValueSets = uniqueValueSets(docToOrdCount, ords);
- if (uniqueValueSets != null) {
- meta.writeVInt(SORTED_SET_TABLE);
-
- // write the set_id -> ords mapping
- writeDictionary(uniqueValueSets);
-
- // write the ord -> byte[] as a binary field
- addTermsDict(field, values);
-
- // write the doc -> set_id as a numeric field
- addNumericField(field, docToSetId(uniqueValueSets, docToOrdCount, ords), NumberType.ORDINAL);
- } else {
- meta.writeVInt(SORTED_WITH_ADDRESSES);
-
- // write the ord -> byte[] as a binary field
- addTermsDict(field, values);
-
- // write the stream of ords as a numeric field
- // NOTE: we could return an iterator that delta-encodes these within a doc
- addNumericField(field, ords, NumberType.ORDINAL);
-
- // write the doc -> ord count as a absolute index to the stream
- addOrdIndex(field, docToOrdCount);
+ SortedSetDocValues values = valuesProducer.getSortedSet(field);
+ int numDocsWithField = 0;
+ long numOrds = 0;
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ numDocsWithField++;
+ for (long ord = values.nextOrd(); ord != SortedSetDocValues.NO_MORE_ORDS; ord = values.nextOrd()) {
+ numOrds++;
}
}
- }
- private SortedSet<LongsRef> uniqueValueSets(Iterable<Number> docToValueCount, Iterable<Number> values) {
- Set<LongsRef> uniqueValueSet = new HashSet<>();
- LongsRef docValues = new LongsRef(256);
-
- Iterator<Number> valueCountIterator = docToValueCount.iterator();
- Iterator<Number> valueIterator = values.iterator();
- int totalDictSize = 0;
- while (valueCountIterator.hasNext()) {
- docValues.length = valueCountIterator.next().intValue();
- if (docValues.length > 256) {
- return null;
- }
- for (int i = 0; i < docValues.length; ++i) {
- docValues.longs[i] = valueIterator.next().longValue();
- }
- if (uniqueValueSet.contains(docValues)) {
- continue;
- }
- totalDictSize += docValues.length;
- if (totalDictSize > 256) {
- return null;
- }
- uniqueValueSet.add(new LongsRef(Arrays.copyOf(docValues.longs, docValues.length), 0, docValues.length));
+ if (numDocsWithField == numOrds) {
+ meta.writeByte((byte) 0);
+ doAddSortedField(field, new EmptyDocValuesProducer() {
+ @Override
+ public SortedDocValues getSorted(FieldInfo field) throws IOException {
+ return SortedSetSelector.wrap(valuesProducer.getSortedSet(field), SortedSetSelector.Type.MIN);
+ }
+ });
+ return;
}
- assert valueIterator.hasNext() == false;
- return new TreeSet<>(uniqueValueSet);
- }
+ meta.writeByte((byte) 1);
- private void writeDictionary(SortedSet<LongsRef> uniqueValueSets) throws IOException {
- int lengthSum = 0;
- for (LongsRef longs : uniqueValueSets) {
- lengthSum += longs.length;
+ assert numDocsWithField != 0;
+ if (numDocsWithField == maxDoc) {
+ meta.writeLong(-1);
+ meta.writeLong(0L);
+ } else {
+ long offset = data.getFilePointer();
+ meta.writeLong(offset);
+ values = valuesProducer.getSortedSet(field);
+ IndexedDISI.writeBitSet(values, data);
+ meta.writeLong(data.getFilePointer() - offset);
}
- meta.writeInt(lengthSum);
- for (LongsRef valueSet : uniqueValueSets) {
- for (int i = 0; i < valueSet.length; ++i) {
- meta.writeLong(valueSet.longs[valueSet.offset + i]);
+ int numberOfBitsPerOrd = DirectWriter.unsignedBitsRequired(values.getValueCount() - 1);
+ meta.writeByte((byte) numberOfBitsPerOrd);
+ long start = data.getFilePointer();
+ meta.writeLong(start);
+ DirectWriter writer = DirectWriter.getInstance(data, numOrds, numberOfBitsPerOrd);
+ values = valuesProducer.getSortedSet(field);
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ for (long ord = values.nextOrd(); ord != SortedSetDocValues.NO_MORE_ORDS; ord = values.nextOrd()) {
+ writer.add(ord);
}
}
+ writer.finish();
+ meta.writeLong(data.getFilePointer() - start);
- meta.writeInt(uniqueValueSets.size());
- for (LongsRef valueSet : uniqueValueSets) {
- meta.writeInt(valueSet.length);
- }
- }
-
- private Iterable<Number> docToSetId(SortedSet<LongsRef> uniqueValueSets, Iterable<Number> docToValueCount, Iterable<Number> values) {
- final Map<LongsRef, Integer> setIds = new HashMap<>();
- int i = 0;
- for (LongsRef set : uniqueValueSets) {
- setIds.put(set, i++);
- }
- assert i == uniqueValueSets.size();
-
- return new Iterable<Number>() {
-
- @Override
- public Iterator<Number> iterator() {
- final Iterator<Number> valueCountIterator = docToValueCount.iterator();
- final Iterator<Number> valueIterator = values.iterator();
- final LongsRef docValues = new LongsRef(256);
- return new Iterator<Number>() {
-
- @Override
- public boolean hasNext() {
- return valueCountIterator.hasNext();
- }
-
- @Override
- public Number next() {
- docValues.length = valueCountIterator.next().intValue();
- for (int i = 0; i < docValues.length; ++i) {
- docValues.longs[i] = valueIterator.next().longValue();
- }
- final Integer id = setIds.get(docValues);
- assert id != null;
- return id;
- }
-
- };
-
- }
- };
- }
-
- // writes addressing information as MONOTONIC_COMPRESSED integer
- private void addOrdIndex(FieldInfo field, Iterable<Number> values) throws IOException {
- meta.writeVInt(field.number);
- meta.writeByte(Lucene70DocValuesFormat.NUMERIC);
- meta.writeVInt(MONOTONIC_COMPRESSED);
- meta.writeLong(-1L);
- meta.writeLong(data.getFilePointer());
- meta.writeVLong(maxDoc);
+ meta.writeInt(numDocsWithField);
+ start = data.getFilePointer();
+ meta.writeLong(start);
meta.writeVInt(DIRECT_MONOTONIC_BLOCK_SHIFT);
- final DirectMonotonicWriter writer = DirectMonotonicWriter.getInstance(meta, data, maxDoc + 1, DIRECT_MONOTONIC_BLOCK_SHIFT);
+ final DirectMonotonicWriter addressesWriter = DirectMonotonicWriter.getInstance(meta, data, numDocsWithField + 1, DIRECT_MONOTONIC_BLOCK_SHIFT);
long addr = 0;
- writer.add(addr);
- for (Number v : values) {
- addr += v.longValue();
- writer.add(addr);
- }
- writer.finish();
- meta.writeLong(data.getFilePointer());
- }
-
- @Override
- public void close() throws IOException {
- boolean success = false;
- try {
- if (meta != null) {
- meta.writeVInt(-1); // write EOF marker
- CodecUtil.writeFooter(meta); // write checksum
- }
- if (data != null) {
- CodecUtil.writeFooter(data); // write checksum
- }
- success = true;
- } finally {
- if (success) {
- IOUtils.close(data, meta);
- } else {
- IOUtils.closeWhileHandlingException(data, meta);
+ addressesWriter.add(addr);
+ values = valuesProducer.getSortedSet(field);
+ for (int doc = values.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = values.nextDoc()) {
+ values.nextOrd();
+ addr++;
+ while (values.nextOrd() != SortedSetDocValues.NO_MORE_ORDS) {
+ addr++;
}
- meta = data = null;
+ addressesWriter.add(addr);
}
+ addressesWriter.finish();
+ meta.writeLong(data.getFilePointer() - start);
+
+ addTermsDict(values);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesFormat.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesFormat.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesFormat.java
index 2112341..ee477d6 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesFormat.java
@@ -23,39 +23,64 @@ import org.apache.lucene.codecs.DocValuesConsumer;
import org.apache.lucene.codecs.DocValuesFormat;
import org.apache.lucene.codecs.DocValuesProducer;
import org.apache.lucene.index.DocValuesType;
+import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.index.SegmentWriteState;
+import org.apache.lucene.store.DataOutput;
import org.apache.lucene.util.SmallFloat;
import org.apache.lucene.util.packed.DirectWriter;
/**
* Lucene 7.0 DocValues format.
* <p>
- * Encodes the five per-document value types (Numeric,Binary,Sorted,SortedSet,SortedNumeric) with these strategies:
+ * Documents that have a value for the field are encoded in a way that it is always possible to
+ * know the ordinal of the current document in the set of documents that have a value. For instance,
+ * say the set of documents that have a value for the field is <tt>{1, 5, 6, 11}</tt>. When the
+ * iterator is on <tt>6</tt>, it knows that this is the 3rd item of the set. This way, values can
+ * be stored densely and accessed based on their index at search time. If all documents in a segment
+ * have a value for the field, the index is the same as the doc ID, so this case is encoded implicitly
+ * and is very fast at query time. On the other hand if some documents are missing a value for the
+ * field then the set of documents that have a value is encoded into blocks. All doc IDs that share
+ * the same upper 16 bits are encoded into the same block with the following strategies:
+ * <ul>
+ * <li>SPARSE: This strategy is used when a block contains at most 4095 documents. The lower 16
+ * bits of doc IDs are stored as {@link DataOutput#writeShort(short) shorts} while the upper
+ * 16 bits are given by the block ID.
+ * <li>DENSE: This strategy is used when a block contains between 4096 and 65535 documents. The
+ * lower bits of doc IDs are stored in a bit set. Advancing is performed using
+ * {@link Long#numberOfTrailingZeros(long) ntz} operations while the index is computed by
+ * accumulating the {@link Long#bitCount(long) bit counts} of the visited longs.
+ * <li>ALL: This strategy is used when a block contains exactly 65536 documents, meaning that
+ * the block is full. In that case doc IDs do not need to be stored explicitly. This is
+ * typically faster than both SPARSE and DENSE which is a reason why it is preferable to have
+ * all documents that have a value for a field using contiguous doc IDs, for instance by
+ * using {@link IndexWriterConfig#setIndexSort(org.apache.lucene.search.Sort) index sorting}.
+ * </ul>
+ * <p>
+ * Then the five per-document value types (Numeric,Binary,Sorted,SortedSet,SortedNumeric) are
+ * encoded using the following strategies:
* <p>
* {@link DocValuesType#NUMERIC NUMERIC}:
* <ul>
* <li>Delta-compressed: per-document integers written as deltas from the minimum value,
* compressed with bitpacking. For more information, see {@link DirectWriter}.
* <li>Table-compressed: when the number of unique values is very small (< 256), and
- * when there are unused "gaps" in the range of values used (such as {@link SmallFloat}),
- * a lookup table is written instead. Each per-document entry is instead the ordinal
- * to this table, and those ordinals are compressed with bitpacking ({@link DirectWriter}).
+ * when there are unused "gaps" in the range of values used (such as {@link SmallFloat}),
+ * a lookup table is written instead. Each per-document entry is instead the ordinal
+ * to this table, and those ordinals are compressed with bitpacking ({@link DirectWriter}).
* <li>GCD-compressed: when all numbers share a common divisor, such as dates, the greatest
* common denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics.
* <li>Monotonic-compressed: when all numbers are monotonically increasing offsets, they are written
* as blocks of bitpacked integers, encoding the deviation from the expected delta.
- * <li>Const-compressed: when there is only one possible non-missing value, only the missing
- * bitset is encoded.
- * <li>Sparse-compressed: only documents with a value are stored, and lookups are performed
- * using binary search.
+ * <li>Const-compressed: when there is only one possible value, no per-document data is needed and
+ * this value is encoded alone.
* </ul>
* <p>
* {@link DocValuesType#BINARY BINARY}:
* <ul>
* <li>Fixed-width Binary: one large concatenated byte[] is written, along with the fixed length.
- * Each document's value can be addressed directly with multiplication ({@code docID * length}).
- * <li>Variable-width Binary: one large concatenated byte[] is written, along with end addresses
+ * Each document's value can be addressed directly with multiplication ({@code docID * length}).
+ * <li>Variable-width Binary: one large concatenated byte[] is written, along with end addresses
* for each document. The addresses are written as Monotonic-compressed numerics.
* <li>Prefix-compressed Binary: values are written in chunks of 16, with the first value written
* completely and other values sharing prefixes. chunk addresses are written as Monotonic-compressed
@@ -64,27 +89,21 @@ import org.apache.lucene.util.packed.DirectWriter;
* <p>
* {@link DocValuesType#SORTED SORTED}:
* <ul>
- * <li>Sorted: a mapping of ordinals to deduplicated terms is written as Binary,
+ * <li>Sorted: a mapping of ordinals to deduplicated terms is written as Prefix-compressed Binary,
* along with the per-document ordinals written using one of the numeric strategies above.
* </ul>
* <p>
* {@link DocValuesType#SORTED_SET SORTED_SET}:
* <ul>
* <li>Single: if all documents have 0 or 1 value, then data are written like SORTED.
- * <li>SortedSet table: when there are few unique sets of values (< 256) then each set is assigned
- * an id, a lookup table is written and the mapping from document to set id is written using the
- * numeric strategies above.
- * <li>SortedSet: a mapping of ordinals to deduplicated terms is written as Binary,
- * an ordinal list and per-document index into this list are written using the numeric strategies
+ * <li>SortedSet: a mapping of ordinals to deduplicated terms is written as Binary,
+ * an ordinal list and per-document index into this list are written using the numeric strategies
* above.
* </ul>
* <p>
* {@link DocValuesType#SORTED_NUMERIC SORTED_NUMERIC}:
* <ul>
* <li>Single: if all documents have 0 or 1 value, then data are written like NUMERIC.
- * <li>SortedSet table: when there are few unique sets of values (< 256) then each set is assigned
- * an id, a lookup table is written and the mapping from document to set id is written using the
- * numeric strategies above.
* <li>SortedNumeric: a value list and per-document index into this list are written using the numeric
* strategies above.
* </ul>
@@ -112,72 +131,30 @@ public final class Lucene70DocValuesFormat extends DocValuesFormat {
public DocValuesProducer fieldsProducer(SegmentReadState state) throws IOException {
return new Lucene70DocValuesProducer(state, DATA_CODEC, DATA_EXTENSION, META_CODEC, META_EXTENSION);
}
-
- static final String DATA_CODEC = "Lucene54DocValuesData";
+
+ static final String DATA_CODEC = "Lucene70DocValuesData";
static final String DATA_EXTENSION = "dvd";
- static final String META_CODEC = "Lucene54DocValuesMetadata";
+ static final String META_CODEC = "Lucene70DocValuesMetadata";
static final String META_EXTENSION = "dvm";
static final int VERSION_START = 0;
static final int VERSION_CURRENT = VERSION_START;
-
+
// indicates docvalues type
static final byte NUMERIC = 0;
static final byte BINARY = 1;
static final byte SORTED = 2;
static final byte SORTED_SET = 3;
static final byte SORTED_NUMERIC = 4;
-
- // address terms in blocks of 16 terms
- static final int INTERVAL_SHIFT = 4;
- static final int INTERVAL_COUNT = 1 << INTERVAL_SHIFT;
- static final int INTERVAL_MASK = INTERVAL_COUNT - 1;
-
- // build reverse index from every 1024th term
- static final int REVERSE_INTERVAL_SHIFT = 10;
- static final int REVERSE_INTERVAL_COUNT = 1 << REVERSE_INTERVAL_SHIFT;
- static final int REVERSE_INTERVAL_MASK = REVERSE_INTERVAL_COUNT - 1;
-
- // for conversion from reverse index to block
- static final int BLOCK_INTERVAL_SHIFT = REVERSE_INTERVAL_SHIFT - INTERVAL_SHIFT;
- static final int BLOCK_INTERVAL_COUNT = 1 << BLOCK_INTERVAL_SHIFT;
- static final int BLOCK_INTERVAL_MASK = BLOCK_INTERVAL_COUNT - 1;
- /** Compressed using packed blocks of ints. */
- static final int DELTA_COMPRESSED = 0;
- /** Compressed by computing the GCD. */
- static final int GCD_COMPRESSED = 1;
- /** Compressed by giving IDs to unique values. */
- static final int TABLE_COMPRESSED = 2;
- /** Compressed with monotonically increasing values */
- static final int MONOTONIC_COMPRESSED = 3;
- /** Compressed with constant value (uses only missing bitset) */
- static final int CONST_COMPRESSED = 4;
- /** Compressed with sparse arrays. */
- static final int SPARSE_COMPRESSED = 5;
-
- /** Uncompressed binary, written directly (fixed length). */
- static final int BINARY_FIXED_UNCOMPRESSED = 0;
- /** Uncompressed binary, written directly (variable length). */
- static final int BINARY_VARIABLE_UNCOMPRESSED = 1;
- /** Compressed binary with shared prefixes */
- static final int BINARY_PREFIX_COMPRESSED = 2;
-
- /** Standard storage for sorted set values with 1 level of indirection:
- * {@code docId -> address -> ord}. */
- static final int SORTED_WITH_ADDRESSES = 0;
- /** Single-valued sorted set values, encoded as sorted values, so no level
- * of indirection: {@code docId -> ord}. */
- static final int SORTED_SINGLE_VALUED = 1;
- /** Compressed giving IDs to unique sets of values:
- * {@code docId -> setId -> ords} */
- static final int SORTED_SET_TABLE = 2;
-
- /** placeholder for missing offset that means there are no missing values */
- static final int ALL_LIVE = -1;
- /** placeholder for missing offset that means all values are missing */
- static final int ALL_MISSING = -2;
-
// addressing uses 16k blocks
static final int MONOTONIC_BLOCK_SIZE = 16384;
static final int DIRECT_MONOTONIC_BLOCK_SHIFT = 16;
+
+ static final int TERMS_DICT_BLOCK_SHIFT = 4;
+ static final int TERMS_DICT_BLOCK_SIZE = 1 << TERMS_DICT_BLOCK_SHIFT;
+ static final int TERMS_DICT_BLOCK_MASK = TERMS_DICT_BLOCK_SIZE - 1;
+
+ static final int TERMS_DICT_REVERSE_INDEX_SHIFT = 10;
+ static final int TERMS_DICT_REVERSE_INDEX_SIZE = 1 << TERMS_DICT_REVERSE_INDEX_SHIFT;
+ static final int TERMS_DICT_REVERSE_INDEX_MASK = TERMS_DICT_REVERSE_INDEX_SIZE - 1;
}
[20/50] [abbrv] lucene-solr:jira/solr-8593: SOLR-9632: Add
deleteAllCollections method to MiniSolrCloudCluster
Posted by kr...@apache.org.
SOLR-9632: Add deleteAllCollections method to MiniSolrCloudCluster
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/20ea5355
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/20ea5355
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/20ea5355
Branch: refs/heads/jira/solr-8593
Commit: 20ea5355c91e61617d59310ea51ca5cb5383c490
Parents: f870c20
Author: Alan Woodward <ro...@apache.org>
Authored: Thu Oct 6 08:15:53 2016 +0100
Committer: Alan Woodward <ro...@apache.org>
Committed: Tue Oct 18 12:33:52 2016 +0100
----------------------------------------------------------------------
solr/CHANGES.txt | 2 ++
.../src/test/org/apache/solr/cloud/rule/RulesTest.java | 13 ++++++-------
.../org/apache/solr/cloud/MiniSolrCloudCluster.java | 10 ++++++++++
3 files changed, 18 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/20ea5355/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index a3c14b0..281140a 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -283,6 +283,8 @@ Other Changes
* SOLR-9627: Add QParser.getSortSpec, deprecate misleadingly named QParser.getSort (Judith Silverman, Christine Poerschke)
+* SOLR-9632: Add MiniSolrCloudCluster#deleteAllCollections() method (Alan Woodward)
+
================== 6.2.1 ==================
Bug Fixes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/20ea5355/solr/core/src/test/org/apache/solr/cloud/rule/RulesTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/rule/RulesTest.java b/solr/core/src/test/org/apache/solr/cloud/rule/RulesTest.java
index eec3d93..6955418 100644
--- a/solr/core/src/test/org/apache/solr/cloud/rule/RulesTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/rule/RulesTest.java
@@ -29,6 +29,7 @@ import org.apache.solr.client.solrj.request.GenericSolrRequest;
import org.apache.solr.cloud.SolrCloudTestCase;
import org.apache.solr.common.cloud.DocCollection;
import org.apache.solr.common.params.ModifiableSolrParams;
+import org.junit.After;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -54,6 +55,11 @@ public class RulesTest extends SolrCloudTestCase {
@org.junit.Rule
public ExpectedException expectedException = ExpectedException.none();
+ @After
+ public void removeCollections() throws Exception {
+ cluster.deleteAllCollections();
+ }
+
@Test
public void doIntegrationTest() throws Exception {
final long minGB = (random().nextBoolean() ? 1 : 0);
@@ -80,8 +86,6 @@ public class RulesTest extends SolrCloudTestCase {
CollectionAdminRequest.createShard(rulesColl, "shard2").process(cluster.getSolrClient());
CollectionAdminRequest.addReplicaToShard(rulesColl, "shard2").process(cluster.getSolrClient());
- CollectionAdminRequest.deleteCollection(rulesColl).process(cluster.getSolrClient());
-
}
@Test
@@ -105,7 +109,6 @@ public class RulesTest extends SolrCloudTestCase {
assertEquals(1, list.size());
assertEquals ( "ImplicitSnitch", ((Map)list.get(0)).get("class"));
- CollectionAdminRequest.deleteCollection(rulesColl).process(cluster.getSolrClient());
}
@Test
@@ -134,7 +137,6 @@ public class RulesTest extends SolrCloudTestCase {
assertEquals(1, list.size());
assertEquals("ImplicitSnitch", list.get(0).get("class"));
- CollectionAdminRequest.deleteCollection(rulesColl).process(cluster.getSolrClient());
}
@@ -157,8 +159,6 @@ public class RulesTest extends SolrCloudTestCase {
.setSnitch("class:ImplicitSnitch")
.process(cluster.getSolrClient());
- CollectionAdminRequest.deleteCollection(rulesColl).process(cluster.getSolrClient());
-
}
@@ -201,6 +201,5 @@ public class RulesTest extends SolrCloudTestCase {
assertEquals(1, list.size());
assertEquals("ImplicitSnitch", ((Map) list.get(0)).get("class"));
- CollectionAdminRequest.deleteCollection(rulesColl).process(cluster.getSolrClient());
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/20ea5355/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
----------------------------------------------------------------------
diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
index 74cae53..7e809c7 100644
--- a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
+++ b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
@@ -46,6 +46,7 @@ import org.apache.solr.client.solrj.embedded.JettySolrRunner;
import org.apache.solr.client.solrj.embedded.SSLConfig;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.impl.CloudSolrClient.Builder;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
import org.apache.solr.client.solrj.request.QueryRequest;
import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.SolrZkClient;
@@ -400,6 +401,15 @@ public class MiniSolrCloudCluster {
manager.uploadConfigDir(configDir.toPath(), configName);
}
}
+
+ public void deleteAllCollections() throws Exception {
+ try (ZkStateReader reader = new ZkStateReader(solrClient.getZkStateReader().getZkClient())) {
+ reader.createClusterStateWatchersAndUpdate();
+ for (String collection : reader.getClusterState().getCollectionStates().keySet()) {
+ CollectionAdminRequest.deleteCollection(collection).process(solrClient);
+ }
+ }
+ }
public NamedList<Object> createCollection(String name, int numShards, int replicationFactor,
String configName, Map<String, String> collectionProperties) throws SolrServerException, IOException {
[25/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7493:
FacetCollector.search now accepts limit=0,
for getting facets but not search hits
Posted by kr...@apache.org.
LUCENE-7493: FacetCollector.search now accepts limit=0, for getting facets but not search hits
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/739981b6
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/739981b6
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/739981b6
Branch: refs/heads/jira/solr-8593
Commit: 739981b6c8e6ccd60279216b320d8a25d06c70e9
Parents: e836071
Author: Mike McCandless <mi...@apache.org>
Authored: Tue Oct 18 09:23:24 2016 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Tue Oct 18 09:23:24 2016 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 ++
.../apache/lucene/facet/FacetsCollector.java | 42 ++++++++++++--------
.../apache/lucene/facet/TestDrillDownQuery.java | 11 +++++
3 files changed, 40 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/739981b6/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 745d8fd..6d83c53 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -89,6 +89,9 @@ Bug Fixes
* LUCENE-6914: Fixed DecimalDigitFilter in case of supplementary code points.
(Hossman)
+* LUCENE-7493: FacetCollector.search threw an unexpected exception if
+ you asked for zero hits but wanted facets (Mahesh via Mike McCandless)
+
Improvements
* LUCENE-7439: FuzzyQuery now matches all terms within the specified
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/739981b6/lucene/facet/src/java/org/apache/lucene/facet/FacetsCollector.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/FacetsCollector.java b/lucene/facet/src/java/org/apache/lucene/facet/FacetsCollector.java
index d3f2eb8..b942f7e 100644
--- a/lucene/facet/src/java/org/apache/lucene/facet/FacetsCollector.java
+++ b/lucene/facet/src/java/org/apache/lucene/facet/FacetsCollector.java
@@ -36,6 +36,7 @@ import org.apache.lucene.search.TopDocsCollector;
import org.apache.lucene.search.TopFieldCollector;
import org.apache.lucene.search.TopFieldDocs;
import org.apache.lucene.search.TopScoreDocCollector;
+import org.apache.lucene.search.TotalHitCountCollector;
import org.apache.lucene.util.ArrayUtil;
import org.apache.lucene.util.BitDocIdSet;
import org.apache.lucene.util.FixedBitSet;
@@ -251,23 +252,32 @@ public class FacetsCollector extends SimpleCollector implements Collector {
+ after.doc + " limit=" + limit);
}
- TopDocsCollector<?> hitsCollector;
- if (sort != null) {
- if (after != null && !(after instanceof FieldDoc)) {
- // TODO: if we fix type safety of TopFieldDocs we can
- // remove this
- throw new IllegalArgumentException("after must be a FieldDoc; got " + after);
- }
- boolean fillFields = true;
- hitsCollector = TopFieldCollector.create(sort, n,
- (FieldDoc) after,
- fillFields,
- doDocScores,
- doMaxScore);
+ TopDocs topDocs = null;
+ if (n==0) {
+ TotalHitCountCollector totalHitCountCollector = new TotalHitCountCollector();
+ searcher.search(q, MultiCollector.wrap(totalHitCountCollector, fc));
+ topDocs = new TopDocs(totalHitCountCollector.getTotalHits(), new ScoreDoc[0], Float.NaN);
} else {
- hitsCollector = TopScoreDocCollector.create(n, after);
+ TopDocsCollector<?> hitsCollector;
+ if (sort != null) {
+ if (after != null && !(after instanceof FieldDoc)) {
+ // TODO: if we fix type safety of TopFieldDocs we can
+ // remove this
+ throw new IllegalArgumentException("after must be a FieldDoc; got " + after);
+ }
+ boolean fillFields = true;
+ hitsCollector = TopFieldCollector.create(sort, n,
+ (FieldDoc) after,
+ fillFields,
+ doDocScores,
+ doMaxScore);
+ } else {
+ hitsCollector = TopScoreDocCollector.create(n, after);
+ }
+ searcher.search(q, MultiCollector.wrap(hitsCollector, fc));
+
+ topDocs = hitsCollector.topDocs();
}
- searcher.search(q, MultiCollector.wrap(hitsCollector, fc));
- return hitsCollector.topDocs();
+ return topDocs;
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/739981b6/lucene/facet/src/test/org/apache/lucene/facet/TestDrillDownQuery.java
----------------------------------------------------------------------
diff --git a/lucene/facet/src/test/org/apache/lucene/facet/TestDrillDownQuery.java b/lucene/facet/src/test/org/apache/lucene/facet/TestDrillDownQuery.java
index f76e839..bf8d0f4 100644
--- a/lucene/facet/src/test/org/apache/lucene/facet/TestDrillDownQuery.java
+++ b/lucene/facet/src/test/org/apache/lucene/facet/TestDrillDownQuery.java
@@ -182,6 +182,17 @@ public class TestDrillDownQuery extends FacetTestCase {
assertEquals(10, docs.totalHits);
}
+ public void testZeroLimit() throws IOException {
+ IndexSearcher searcher = newSearcher(reader);
+ DrillDownQuery q = new DrillDownQuery(config);
+ q.add("b", "1");
+ int limit = 0;
+ FacetsCollector facetCollector = new FacetsCollector();
+ FacetsCollector.search(searcher, q, limit, facetCollector);
+ Facets facets = getTaxonomyFacetCounts(taxo, config, facetCollector, config.getDimConfig("b").indexFieldName);
+ assertNotNull(facets.getTopChildren(10, "b"));
+ }
+
public void testScoring() throws IOException {
// verify that drill-down queries do not modify scores
IndexSearcher searcher = newSearcher(reader);
[17/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7363: Fixed
DecimalDigitFilter in case of supplementary code points.
Posted by kr...@apache.org.
LUCENE-7363: Fixed DecimalDigitFilter in case of supplementary code points.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/9260a137
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/9260a137
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/9260a137
Branch: refs/heads/jira/solr-8593
Commit: 9260a1378f942f072911b7af984c82f58e377a00
Parents: c22725f
Author: Adrien Grand <jp...@gmail.com>
Authored: Tue Oct 18 10:38:51 2016 +0200
Committer: Adrien Grand <jp...@gmail.com>
Committed: Tue Oct 18 10:38:51 2016 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +
.../analysis/core/DecimalDigitFilter.java | 2 +-
.../analysis/core/TestDecimalDigitFilter.java | 151 +++++++++++++++++--
3 files changed, 140 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9260a137/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 6defb27..810b529 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -86,6 +86,9 @@ Bug Fixes
dimensional points could cause unexpected merge exceptions (Hans
Lund, Mike McCandless)
+* LUCENE-7363: Fixed DecimalDigitFilter in case of supplementary code points.
+ (Hossman)
+
Improvements
* LUCENE-7439: FuzzyQuery now matches all terms within the specified
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9260a137/lucene/analysis/common/src/java/org/apache/lucene/analysis/core/DecimalDigitFilter.java
----------------------------------------------------------------------
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/core/DecimalDigitFilter.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/core/DecimalDigitFilter.java
index b81d42f..de459cf 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/core/DecimalDigitFilter.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/core/DecimalDigitFilter.java
@@ -52,7 +52,7 @@ public final class DecimalDigitFilter extends TokenFilter {
buffer[i] = (char) ('0' + Character.getNumericValue(ch));
// if the original was supplementary, shrink the string
if (ch > 0xFFFF) {
- length = StemmerUtil.delete(buffer, ++i, length);
+ length = StemmerUtil.delete(buffer, i+1, length);
termAtt.setLength(length);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9260a137/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java
----------------------------------------------------------------------
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java
index ae25193..e5e18ef 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestDecimalDigitFilter.java
@@ -21,14 +21,42 @@ import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.BaseTokenStreamTestCase;
import org.apache.lucene.analysis.MockTokenizer;
import org.apache.lucene.analysis.Tokenizer;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.SparseFixedBitSet;
import org.apache.lucene.util.TestUtil;
+import java.util.Random;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+
/**
* Tests for {@link DecimalDigitFilter}
*/
public class TestDecimalDigitFilter extends BaseTokenStreamTestCase {
private Analyzer tokenized;
private Analyzer keyword;
+
+ private static SparseFixedBitSet DECIMAL_DIGIT_CODEPOINTS;
+
+ @BeforeClass
+ public static void init_DECIMAL_DIGIT_CODEPOINTS() {
+ DECIMAL_DIGIT_CODEPOINTS = new SparseFixedBitSet(Character.MAX_CODE_POINT);
+ for (int codepoint = Character.MIN_CODE_POINT; codepoint < Character.MAX_CODE_POINT; codepoint++) {
+ if (Character.isDigit(codepoint)) {
+ DECIMAL_DIGIT_CODEPOINTS.set(codepoint);
+ }
+ }
+ assert 0 < DECIMAL_DIGIT_CODEPOINTS.cardinality();
+ }
+
+ @AfterClass
+ public static void destroy_DECIMAL_DIGIT_CODEPOINTS() {
+ DECIMAL_DIGIT_CODEPOINTS = null;
+ }
+
@Override
public void setUp() throws Exception {
@@ -64,30 +92,83 @@ public class TestDecimalDigitFilter extends BaseTokenStreamTestCase {
}
/**
- * test all digits in different locations of strings.
+ * test that double struck digits are normalized
*/
- public void testRandom() throws Exception {
- for (int codepoint = Character.MIN_CODE_POINT; codepoint < Character.MAX_CODE_POINT; codepoint++) {
- if (Character.isDigit(codepoint)) {
- // add some a-z before/after the string
- String prefix = TestUtil.randomSimpleString(random());
- String suffix = TestUtil.randomSimpleString(random());
+ public void testDoubleStruck() throws Exception {
+ // MATHEMATICAL DOUBLE-STRUCK DIGIT ... 1, 9, 8, 4
+ final String input = "\U0001d7d9 \U0001d7e1 \U0001d7e0 \U0001d7dc";
+ final String expected = "1 9 8 4";
+ checkOneTerm(keyword, input, expected);
+ checkOneTerm(keyword, input.replaceAll("\\s",""), expected.replaceAll("\\s",""));
+ }
+
+ /**
+ * test sequences of digits mixed with other random simple string data
+ */
+ public void testRandomSequences() throws Exception {
+
+ // test numIters random strings containing a sequence of numDigits codepoints
+ final int numIters = atLeast(5);
+ for (int iter = 0; iter < numIters; iter++) {
+ final int numDigits = atLeast(20);
+ final StringBuilder expected = new StringBuilder();
+ final StringBuilder actual = new StringBuilder();
+ for (int digitCounter = 0; digitCounter < numDigits; digitCounter++) {
- StringBuilder expected = new StringBuilder();
+ // increased odds of 0 length random string prefix
+ final String prefix = random().nextBoolean() ? "" : TestUtil.randomSimpleString(random());
expected.append(prefix);
+ actual.append(prefix);
+
+ int codepoint = getRandomDecimalDigit(random());
+
int value = Character.getNumericValue(codepoint);
assert value >= 0 && value <= 9;
expected.append(Integer.toString(value));
- expected.append(suffix);
-
- StringBuilder actual = new StringBuilder();
- actual.append(prefix);
actual.appendCodePoint(codepoint);
- actual.append(suffix);
-
- checkOneTerm(keyword, actual.toString(), expected.toString());
}
+ // occasional suffix, increased odds of 0 length random string
+ final String suffix = random().nextBoolean() ? "" : TestUtil.randomSimpleString(random());
+ expected.append(suffix);
+ actual.append(suffix);
+
+ checkOneTerm(keyword, actual.toString(), expected.toString());
}
+
+ }
+
+ /**
+ * test each individual digit in different locations of strings.
+ */
+ public void testRandom() throws Exception {
+ int numCodePointsChecked = 0; // sanity check
+ for (int codepoint = DECIMAL_DIGIT_CODEPOINTS.nextSetBit(0);
+ codepoint != DocIdSetIterator.NO_MORE_DOCS;
+ codepoint = DECIMAL_DIGIT_CODEPOINTS.nextSetBit(codepoint+1)) {
+
+ assert Character.isDigit(codepoint);
+
+ // add some a-z before/after the string
+ String prefix = TestUtil.randomSimpleString(random());
+ String suffix = TestUtil.randomSimpleString(random());
+
+ StringBuilder expected = new StringBuilder();
+ expected.append(prefix);
+ int value = Character.getNumericValue(codepoint);
+ assert value >= 0 && value <= 9;
+ expected.append(Integer.toString(value));
+ expected.append(suffix);
+
+ StringBuilder actual = new StringBuilder();
+ actual.append(prefix);
+ actual.appendCodePoint(codepoint);
+ actual.append(suffix);
+
+ checkOneTerm(keyword, actual.toString(), expected.toString());
+
+ numCodePointsChecked++;
+ }
+ assert DECIMAL_DIGIT_CODEPOINTS.cardinality() == numCodePointsChecked;
}
/**
@@ -103,4 +184,44 @@ public class TestDecimalDigitFilter extends BaseTokenStreamTestCase {
public void testRandomStrings() throws Exception {
checkRandomData(random(), tokenized, 1000*RANDOM_MULTIPLIER);
}
+
+ /** returns a psuedo-random codepoint which is a Decimal Digit */
+ public static int getRandomDecimalDigit(Random r) {
+ final int aprox = TestUtil.nextInt(r, 0, DECIMAL_DIGIT_CODEPOINTS.length()-1);
+
+ if (DECIMAL_DIGIT_CODEPOINTS.get(aprox)) { // lucky guess
+ assert Character.isDigit(aprox);
+ return aprox;
+ }
+
+ // seek up and down for closest set bit
+ final int lower = DECIMAL_DIGIT_CODEPOINTS.prevSetBit(aprox);
+ final int higher = DECIMAL_DIGIT_CODEPOINTS.nextSetBit(aprox);
+
+ // sanity check edge cases
+ if (lower < 0) {
+ assert higher != DocIdSetIterator.NO_MORE_DOCS;
+ assert Character.isDigit(higher);
+ return higher;
+ }
+ if (higher == DocIdSetIterator.NO_MORE_DOCS) {
+ assert 0 <= lower;
+ assert Character.isDigit(lower);
+ return lower;
+ }
+
+ // which is closer?
+ final int cmp = Integer.compare(aprox - lower, higher - aprox);
+
+ if (0 == cmp) {
+ // dead even, flip a coin
+ final int result = random().nextBoolean() ? lower : higher;
+ assert Character.isDigit(result);
+ return result;
+ }
+
+ final int result = (cmp < 0) ? lower : higher;
+ assert Character.isDigit(result);
+ return result;
+ }
}
[31/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7497: add test case
Posted by kr...@apache.org.
LUCENE-7497: add test case
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/b78f2219
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/b78f2219
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/b78f2219
Branch: refs/heads/jira/solr-8593
Commit: b78f2219f45ca64c6a4b7261a87fae89477ec26f
Parents: 3be6701
Author: Mike McCandless <mi...@apache.org>
Authored: Tue Oct 18 14:02:02 2016 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Tue Oct 18 14:02:40 2016 -0400
----------------------------------------------------------------------
.../search/join/ToParentBlockJoinQuery.java | 2 +-
.../lucene/search/join/TestBlockJoin.java | 66 ++++++++++++++++++++
2 files changed, 67 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b78f2219/lucene/join/src/java/org/apache/lucene/search/join/ToParentBlockJoinQuery.java
----------------------------------------------------------------------
diff --git a/lucene/join/src/java/org/apache/lucene/search/join/ToParentBlockJoinQuery.java b/lucene/join/src/java/org/apache/lucene/search/join/ToParentBlockJoinQuery.java
index 3abdeeb..432ebcc 100644
--- a/lucene/join/src/java/org/apache/lucene/search/join/ToParentBlockJoinQuery.java
+++ b/lucene/join/src/java/org/apache/lucene/search/join/ToParentBlockJoinQuery.java
@@ -203,7 +203,7 @@ public class ToParentBlockJoinQuery extends Query {
public abstract int[] swapChildDocs(int[] other);
}
- static class BlockJoinScorer extends ChildrenMatchesScorer{
+ static class BlockJoinScorer extends ChildrenMatchesScorer {
private final Scorer childScorer;
private final BitSet parentBits;
private final ScoreMode scoreMode;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/b78f2219/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java
----------------------------------------------------------------------
diff --git a/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java b/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java
index af9ff5c..cf21fa4 100644
--- a/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java
+++ b/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java
@@ -187,7 +187,73 @@ public class TestBlockJoin extends LuceneTestCase {
dir.close();
}
+ // You must use ToParentBlockJoinSearcher if you want to do BQ SHOULD queries:
+ public void testBQShouldJoinedChild() throws Exception {
+ final Directory dir = newDirectory();
+ final RandomIndexWriter w = new RandomIndexWriter(random(), dir);
+
+ final List<Document> docs = new ArrayList<>();
+
+ docs.add(makeJob("java", 2007));
+ docs.add(makeJob("python", 2010));
+ docs.add(makeResume("Lisa", "United Kingdom"));
+ w.addDocuments(docs);
+
+ docs.clear();
+ docs.add(makeJob("ruby", 2005));
+ docs.add(makeJob("java", 2006));
+ docs.add(makeResume("Frank", "United States"));
+ w.addDocuments(docs);
+
+ IndexReader r = w.getReader();
+ w.close();
+ IndexSearcher s = new ToParentBlockJoinIndexSearcher(r);
+ //IndexSearcher s = newSearcher(r, false);
+ //IndexSearcher s = new IndexSearcher(r);
+
+ // Create a filter that defines "parent" documents in the index - in this case resumes
+ BitSetProducer parentsFilter = new QueryBitSetProducer(new TermQuery(new Term("docType", "resume")));
+ CheckJoinIndex.check(r, parentsFilter);
+
+ // Define child document criteria (finds an example of relevant work experience)
+ BooleanQuery.Builder childQuery = new BooleanQuery.Builder();
+ childQuery.add(new BooleanClause(new TermQuery(new Term("skill", "java")), Occur.MUST));
+ childQuery.add(new BooleanClause(IntPoint.newRangeQuery("year", 2006, 2011), Occur.MUST));
+
+ // Define parent document criteria (find a resident in the UK)
+ Query parentQuery = new TermQuery(new Term("country", "United Kingdom"));
+
+ // Wrap the child document query to 'join' any matches
+ // up to corresponding parent:
+ ToParentBlockJoinQuery childJoinQuery = new ToParentBlockJoinQuery(childQuery.build(), parentsFilter, ScoreMode.Avg);
+
+ // Combine the parent and nested child queries into a single query for a candidate
+ BooleanQuery.Builder fullQuery = new BooleanQuery.Builder();
+ fullQuery.add(new BooleanClause(parentQuery, Occur.SHOULD));
+ fullQuery.add(new BooleanClause(childJoinQuery, Occur.SHOULD));
+
+ ToParentBlockJoinCollector c = new ToParentBlockJoinCollector(Sort.RELEVANCE, 1, true, true);
+ s.search(fullQuery.build(), c);
+ TopGroups<Integer> results = c.getTopGroups(childJoinQuery, null, 0, 10, 0, true);
+ assertEquals(1, results.totalGroupedHitCount);
+ assertEquals(1, results.groups.length);
+ final GroupDocs<Integer> group = results.groups[0];
+ assertEquals(1, group.totalHits);
+ assertFalse(Float.isNaN(group.score));
+
+ Document childDoc = s.doc(group.scoreDocs[0].doc);
+ //System.out.println(" doc=" + group.scoreDocs[0].doc);
+ assertEquals("java", childDoc.get("skill"));
+ assertNotNull(group.groupValue);
+ Document parentDoc = s.doc(group.groupValue);
+ assertEquals("Lisa", parentDoc.get("name"));
+
+
+ r.close();
+ dir.close();
+ }
+
public void testSimple() throws Exception {
final Directory dir = newDirectory();
[09/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7489: Better
sparsity support for Lucene70DocValuesFormat.
Posted by kr...@apache.org.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/927fd51d/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java
index e806ea5..755da79 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene70/Lucene70DocValuesProducer.java
@@ -18,84 +18,53 @@ package org.apache.lucene.codecs.lucene70;
import java.io.Closeable;
import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
import java.util.HashMap;
-import java.util.List;
import java.util.Map;
-import java.util.concurrent.atomic.AtomicLong;
import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.codecs.DocValuesProducer;
-import org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.NumberType;
-import org.apache.lucene.index.*;
-import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.index.FieldInfos;
+import org.apache.lucene.index.IndexFileNames;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PostingsEnum;
+import org.apache.lucene.index.SegmentReadState;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.index.SortedSetDocValues;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.index.TermsEnum.SeekStatus;
import org.apache.lucene.store.ChecksumIndexInput;
import org.apache.lucene.store.IndexInput;
import org.apache.lucene.store.RandomAccessInput;
-import org.apache.lucene.util.Accountable;
-import org.apache.lucene.util.Accountables;
-import org.apache.lucene.util.Bits;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.IOUtils;
import org.apache.lucene.util.LongValues;
-import org.apache.lucene.util.PagedBytes;
import org.apache.lucene.util.RamUsageEstimator;
import org.apache.lucene.util.packed.DirectMonotonicReader;
import org.apache.lucene.util.packed.DirectReader;
-import org.apache.lucene.util.packed.MonotonicBlockPackedReader;
-
-import static org.apache.lucene.codecs.lucene70.Lucene70DocValuesFormat.*;
/** reader for {@link Lucene70DocValuesFormat} */
final class Lucene70DocValuesProducer extends DocValuesProducer implements Closeable {
private final Map<String,NumericEntry> numerics = new HashMap<>();
private final Map<String,BinaryEntry> binaries = new HashMap<>();
+ private final Map<String,SortedEntry> sorted = new HashMap<>();
private final Map<String,SortedSetEntry> sortedSets = new HashMap<>();
- private final Map<String,SortedSetEntry> sortedNumerics = new HashMap<>();
- private final Map<String,NumericEntry> ords = new HashMap<>();
- private final Map<String,NumericEntry> ordIndexes = new HashMap<>();
- private final int numFields;
- private final AtomicLong ramBytesUsed;
+ private final Map<String,SortedNumericEntry> sortedNumerics = new HashMap<>();
+ private long ramBytesUsed;
private final IndexInput data;
private final int maxDoc;
- // memory-resident structures
- private final Map<String,MonotonicBlockPackedReader> addressInstances = new HashMap<>();
- private final Map<String,ReverseTermsIndex> reverseIndexInstances = new HashMap<>();
- private final Map<String,DirectMonotonicReader.Meta> directAddressesMeta = new HashMap<>();
-
- private final boolean merging;
-
- // clone for merge: when merging we don't do any instances.put()s
- Lucene70DocValuesProducer(Lucene70DocValuesProducer original) throws IOException {
- assert Thread.holdsLock(original);
- numerics.putAll(original.numerics);
- binaries.putAll(original.binaries);
- sortedSets.putAll(original.sortedSets);
- sortedNumerics.putAll(original.sortedNumerics);
- ords.putAll(original.ords);
- ordIndexes.putAll(original.ordIndexes);
- numFields = original.numFields;
- ramBytesUsed = new AtomicLong(original.ramBytesUsed.get());
- data = original.data.clone();
- maxDoc = original.maxDoc;
-
- addressInstances.putAll(original.addressInstances);
- reverseIndexInstances.putAll(original.reverseIndexInstances);
- merging = true;
- }
-
/** expert: instantiates a new reader */
Lucene70DocValuesProducer(SegmentReadState state, String dataCodec, String dataExtension, String metaCodec, String metaExtension) throws IOException {
String metaName = IndexFileNames.segmentFileName(state.segmentInfo.name, state.segmentSuffix, metaExtension);
this.maxDoc = state.segmentInfo.maxDoc();
- merging = false;
- ramBytesUsed = new AtomicLong(RamUsageEstimator.shallowSizeOfInstance(getClass()));
+ ramBytesUsed = RamUsageEstimator.shallowSizeOfInstance(getClass());
int version = -1;
- int numFields = -1;
// read in the entries from the metadata file.
try (ChecksumIndexInput in = state.directory.openChecksumInput(metaName, state.context)) {
@@ -106,7 +75,7 @@ final class Lucene70DocValuesProducer extends DocValuesProducer implements Close
Lucene70DocValuesFormat.VERSION_CURRENT,
state.segmentInfo.getId(),
state.segmentSuffix);
- numFields = readFields(in, state.fieldInfos);
+ readFields(in, state.fieldInfos);
} catch (Throwable exception) {
priorE = exception;
} finally {
@@ -114,7 +83,6 @@ final class Lucene70DocValuesProducer extends DocValuesProducer implements Close
}
}
- this.numFields = numFields;
String dataName = IndexFileNames.segmentFileName(state.segmentInfo.name, state.segmentSuffix, dataExtension);
this.data = state.directory.openInput(dataName, state.context);
boolean success = false;
@@ -142,1277 +110,841 @@ final class Lucene70DocValuesProducer extends DocValuesProducer implements Close
}
}
- private void readSortedField(FieldInfo info, IndexInput meta) throws IOException {
- // sorted = binary + numeric
- if (meta.readVInt() != info.number) {
- throw new CorruptIndexException("sorted entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.BINARY) {
- throw new CorruptIndexException("sorted entry for field: " + info.name + " is corrupt", meta);
- }
- BinaryEntry b = readBinaryEntry(info, meta);
- binaries.put(info.name, b);
-
- if (meta.readVInt() != info.number) {
- throw new CorruptIndexException("sorted entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.NUMERIC) {
- throw new CorruptIndexException("sorted entry for field: " + info.name + " is corrupt", meta);
- }
- NumericEntry n = readNumericEntry(info, meta);
- ords.put(info.name, n);
- }
-
- private void readSortedSetFieldWithAddresses(FieldInfo info, IndexInput meta) throws IOException {
- // sortedset = binary + numeric (addresses) + ordIndex
- if (meta.readVInt() != info.number) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.BINARY) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- BinaryEntry b = readBinaryEntry(info, meta);
- binaries.put(info.name, b);
-
- if (meta.readVInt() != info.number) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.NUMERIC) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- NumericEntry n1 = readNumericEntry(info, meta);
- ords.put(info.name, n1);
-
- if (meta.readVInt() != info.number) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.NUMERIC) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- NumericEntry n2 = readNumericEntry(info, meta);
- ordIndexes.put(info.name, n2);
- }
-
- private void readSortedSetFieldWithTable(FieldInfo info, IndexInput meta) throws IOException {
- // sortedset table = binary + ordset table + ordset index
- if (meta.readVInt() != info.number) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.BINARY) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
-
- BinaryEntry b = readBinaryEntry(info, meta);
- binaries.put(info.name, b);
-
- if (meta.readVInt() != info.number) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.NUMERIC) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- NumericEntry n = readNumericEntry(info, meta);
- ords.put(info.name, n);
- }
-
- private int readFields(IndexInput meta, FieldInfos infos) throws IOException {
- int numFields = 0;
- int fieldNumber = meta.readVInt();
- while (fieldNumber != -1) {
- numFields++;
+ private void readFields(ChecksumIndexInput meta, FieldInfos infos) throws IOException {
+ for (int fieldNumber = meta.readInt(); fieldNumber != -1; fieldNumber = meta.readInt()) {
FieldInfo info = infos.fieldInfo(fieldNumber);
if (info == null) {
- // trickier to validate more: because we use multiple entries for "composite" types like sortedset, etc.
throw new CorruptIndexException("Invalid field number: " + fieldNumber, meta);
}
byte type = meta.readByte();
if (type == Lucene70DocValuesFormat.NUMERIC) {
- numerics.put(info.name, readNumericEntry(info, meta));
+ numerics.put(info.name, readNumeric(meta));
} else if (type == Lucene70DocValuesFormat.BINARY) {
- BinaryEntry b = readBinaryEntry(info, meta);
- binaries.put(info.name, b);
+ binaries.put(info.name, readBinary(meta));
} else if (type == Lucene70DocValuesFormat.SORTED) {
- readSortedField(info, meta);
+ sorted.put(info.name, readSorted(meta));
} else if (type == Lucene70DocValuesFormat.SORTED_SET) {
- SortedSetEntry ss = readSortedSetEntry(meta);
- sortedSets.put(info.name, ss);
- if (ss.format == SORTED_WITH_ADDRESSES) {
- readSortedSetFieldWithAddresses(info, meta);
- } else if (ss.format == SORTED_SET_TABLE) {
- readSortedSetFieldWithTable(info, meta);
- } else if (ss.format == SORTED_SINGLE_VALUED) {
- if (meta.readVInt() != fieldNumber) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.SORTED) {
- throw new CorruptIndexException("sortedset entry for field: " + info.name + " is corrupt", meta);
- }
- readSortedField(info, meta);
- } else {
- throw new AssertionError();
- }
+ sortedSets.put(info.name, readSortedSet(meta));
} else if (type == Lucene70DocValuesFormat.SORTED_NUMERIC) {
- SortedSetEntry ss = readSortedSetEntry(meta);
- sortedNumerics.put(info.name, ss);
- if (ss.format == SORTED_WITH_ADDRESSES) {
- if (meta.readVInt() != fieldNumber) {
- throw new CorruptIndexException("sortednumeric entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.NUMERIC) {
- throw new CorruptIndexException("sortednumeric entry for field: " + info.name + " is corrupt", meta);
- }
- numerics.put(info.name, readNumericEntry(info, meta));
- if (meta.readVInt() != fieldNumber) {
- throw new CorruptIndexException("sortednumeric entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.NUMERIC) {
- throw new CorruptIndexException("sortednumeric entry for field: " + info.name + " is corrupt", meta);
- }
- NumericEntry ordIndex = readNumericEntry(info, meta);
- ordIndexes.put(info.name, ordIndex);
- } else if (ss.format == SORTED_SET_TABLE) {
- if (meta.readVInt() != info.number) {
- throw new CorruptIndexException("sortednumeric entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.NUMERIC) {
- throw new CorruptIndexException("sortednumeric entry for field: " + info.name + " is corrupt", meta);
- }
- NumericEntry n = readNumericEntry(info, meta);
- ords.put(info.name, n);
- } else if (ss.format == SORTED_SINGLE_VALUED) {
- if (meta.readVInt() != fieldNumber) {
- throw new CorruptIndexException("sortednumeric entry for field: " + info.name + " is corrupt", meta);
- }
- if (meta.readByte() != Lucene70DocValuesFormat.NUMERIC) {
- throw new CorruptIndexException("sortednumeric entry for field: " + info.name + " is corrupt", meta);
- }
- numerics.put(info.name, readNumericEntry(info, meta));
- } else {
- throw new AssertionError();
- }
+ sortedNumerics.put(info.name, readSortedNumeric(meta));
} else {
throw new CorruptIndexException("invalid type: " + type, meta);
}
- fieldNumber = meta.readVInt();
}
- return numFields;
}
- private NumericEntry readNumericEntry(FieldInfo info, IndexInput meta) throws IOException {
+ private NumericEntry readNumeric(ChecksumIndexInput meta) throws IOException {
NumericEntry entry = new NumericEntry();
- entry.format = meta.readVInt();
- entry.missingOffset = meta.readLong();
- if (entry.format == SPARSE_COMPRESSED) {
- // sparse bits need a bit more metadata
- entry.numDocsWithValue = meta.readVLong();
- final int blockShift = meta.readVInt();
- entry.monotonicMeta = DirectMonotonicReader.loadMeta(meta, entry.numDocsWithValue, blockShift);
- ramBytesUsed.addAndGet(entry.monotonicMeta.ramBytesUsed());
- directAddressesMeta.put(info.name, entry.monotonicMeta);
+ readNumeric(meta, entry);
+ return entry;
+ }
+
+ private void readNumeric(ChecksumIndexInput meta, NumericEntry entry) throws IOException {
+ entry.docsWithFieldOffset = meta.readLong();
+ entry.docsWithFieldLength = meta.readLong();
+ entry.numValues = meta.readLong();
+ int tableSize = meta.readInt();
+ if (tableSize < -1 || tableSize > 256) {
+ throw new CorruptIndexException("invalid table size: " + tableSize, meta);
}
- entry.offset = meta.readLong();
- entry.count = meta.readVLong();
- switch(entry.format) {
- case CONST_COMPRESSED:
- entry.minValue = meta.readLong();
- if (entry.count > Integer.MAX_VALUE) {
- // currently just a limitation e.g. of bits interface and so on.
- throw new CorruptIndexException("illegal CONST_COMPRESSED count: " + entry.count, meta);
- }
- break;
- case GCD_COMPRESSED:
- entry.minValue = meta.readLong();
- entry.gcd = meta.readLong();
- entry.bitsPerValue = meta.readVInt();
- break;
- case TABLE_COMPRESSED:
- final int uniqueValues = meta.readVInt();
- if (uniqueValues > 256) {
- throw new CorruptIndexException("TABLE_COMPRESSED cannot have more than 256 distinct values, got=" + uniqueValues, meta);
- }
- entry.table = new long[uniqueValues];
- for (int i = 0; i < uniqueValues; ++i) {
- entry.table[i] = meta.readLong();
- }
- ramBytesUsed.addAndGet(RamUsageEstimator.sizeOf(entry.table));
- entry.bitsPerValue = meta.readVInt();
- break;
- case DELTA_COMPRESSED:
- entry.minValue = meta.readLong();
- entry.bitsPerValue = meta.readVInt();
- break;
- case MONOTONIC_COMPRESSED:
- final int blockShift = meta.readVInt();
- entry.monotonicMeta = DirectMonotonicReader.loadMeta(meta, maxDoc + 1, blockShift);
- ramBytesUsed.addAndGet(entry.monotonicMeta.ramBytesUsed());
- directAddressesMeta.put(info.name, entry.monotonicMeta);
- break;
- case SPARSE_COMPRESSED:
- final byte numberType = meta.readByte();
- switch (numberType) {
- case 0:
- entry.numberType = NumberType.VALUE;
- break;
- case 1:
- entry.numberType = NumberType.ORDINAL;
- break;
- default:
- throw new CorruptIndexException("Number type can only be 0 or 1, got=" + numberType, meta);
- }
+ if (tableSize >= 0) {
+ entry.table = new long[tableSize];
+ ramBytesUsed += RamUsageEstimator.sizeOf(entry.table);
+ for (int i = 0; i < tableSize; ++i) {
+ entry.table[i] = meta.readLong();
+ }
+ }
+ entry.bitsPerValue = meta.readByte();
+ entry.minValue = meta.readLong();
+ entry.gcd = meta.readLong();
+ entry.valuesOffset = meta.readLong();
+ entry.valuesLength = meta.readLong();
+ }
- // now read the numeric entry for non-missing values
- final int fieldNumber = meta.readVInt();
- if (fieldNumber != info.number) {
- throw new CorruptIndexException("Field numbers mistmatch: " + fieldNumber + " != " + info.number, meta);
- }
- final int dvFormat = meta.readByte();
- if (dvFormat != NUMERIC) {
- throw new CorruptIndexException("Formats mistmatch: " + dvFormat + " != " + NUMERIC, meta);
- }
- entry.nonMissingValues = readNumericEntry(info, meta);
- break;
- default:
- throw new CorruptIndexException("Unknown format: " + entry.format + ", input=", meta);
+ private BinaryEntry readBinary(ChecksumIndexInput meta) throws IOException {
+ BinaryEntry entry = new BinaryEntry();
+ entry.dataOffset = meta.readLong();
+ entry.dataLength = meta.readLong();
+ entry.docsWithFieldOffset = meta.readLong();
+ entry.docsWithFieldLength = meta.readLong();
+ entry.numDocsWithField = meta.readInt();
+ entry.minLength = meta.readInt();
+ entry.maxLength = meta.readInt();
+ if (entry.minLength < entry.maxLength) {
+ entry.addressesOffset = meta.readLong();
+ final int blockShift = meta.readVInt();
+ entry.addressesMeta = DirectMonotonicReader.loadMeta(meta, entry.numDocsWithField + 1L, blockShift);
+ ramBytesUsed += entry.addressesMeta.ramBytesUsed();
+ entry.addressesLength = meta.readLong();
}
- entry.endOffset = meta.readLong();
return entry;
}
- private BinaryEntry readBinaryEntry(FieldInfo info, IndexInput meta) throws IOException {
- BinaryEntry entry = new BinaryEntry();
- entry.format = meta.readVInt();
- entry.missingOffset = meta.readLong();
- entry.minLength = meta.readVInt();
- entry.maxLength = meta.readVInt();
- entry.count = meta.readVLong();
- entry.offset = meta.readLong();
- switch(entry.format) {
- case BINARY_FIXED_UNCOMPRESSED:
- break;
- case BINARY_PREFIX_COMPRESSED:
- entry.addressesOffset = meta.readLong();
- entry.packedIntsVersion = meta.readVInt();
- entry.blockSize = meta.readVInt();
- entry.reverseIndexOffset = meta.readLong();
- break;
- case BINARY_VARIABLE_UNCOMPRESSED:
- entry.addressesOffset = meta.readLong();
- final int blockShift = meta.readVInt();
- entry.addressesMeta = DirectMonotonicReader.loadMeta(meta, entry.count + 1, blockShift);
- ramBytesUsed.addAndGet(entry.addressesMeta.ramBytesUsed());
- directAddressesMeta.put(info.name, entry.addressesMeta);
- entry.addressesEndOffset = meta.readLong();
+ private SortedEntry readSorted(ChecksumIndexInput meta) throws IOException {
+ SortedEntry entry = new SortedEntry();
+ entry.docsWithFieldOffset = meta.readLong();
+ entry.docsWithFieldLength = meta.readLong();
+ entry.numDocsWithField = meta.readInt();
+ entry.bitsPerValue = meta.readByte();
+ entry.ordsOffset = meta.readLong();
+ entry.ordsLength = meta.readLong();
+ readTermDict(meta, entry);
+ return entry;
+ }
+
+ private SortedSetEntry readSortedSet(ChecksumIndexInput meta) throws IOException {
+ SortedSetEntry entry = new SortedSetEntry();
+ byte multiValued = meta.readByte();
+ switch (multiValued) {
+ case 0: // singlevalued
+ entry.singleValueEntry = readSorted(meta);
+ return entry;
+ case 1: // multivalued
break;
default:
- throw new CorruptIndexException("Unknown format: " + entry.format, meta);
+ throw new CorruptIndexException("Invalid multiValued flag: " + multiValued, meta);
}
+ entry.docsWithFieldOffset = meta.readLong();
+ entry.docsWithFieldLength = meta.readLong();
+ entry.bitsPerValue = meta.readByte();
+ entry.ordsOffset = meta.readLong();
+ entry.ordsLength = meta.readLong();
+ entry.numDocsWithField = meta.readInt();
+ entry.addressesOffset = meta.readLong();
+ final int blockShift = meta.readVInt();
+ entry.addressesMeta = DirectMonotonicReader.loadMeta(meta, entry.numDocsWithField + 1, blockShift);
+ ramBytesUsed += entry.addressesMeta.ramBytesUsed();
+ entry.addressesLength = meta.readLong();
+ readTermDict(meta, entry);
return entry;
}
- SortedSetEntry readSortedSetEntry(IndexInput meta) throws IOException {
- SortedSetEntry entry = new SortedSetEntry();
- entry.format = meta.readVInt();
- if (entry.format == SORTED_SET_TABLE) {
- final int totalTableLength = meta.readInt();
- if (totalTableLength > 256) {
- throw new CorruptIndexException("SORTED_SET_TABLE cannot have more than 256 values in its dictionary, got=" + totalTableLength, meta);
- }
- entry.table = new long[totalTableLength];
- for (int i = 0; i < totalTableLength; ++i) {
- entry.table[i] = meta.readLong();
- }
- ramBytesUsed.addAndGet(RamUsageEstimator.sizeOf(entry.table));
- final int tableSize = meta.readInt();
- if (tableSize > totalTableLength + 1) { // +1 because of the empty set
- throw new CorruptIndexException("SORTED_SET_TABLE cannot have more set ids than ords in its dictionary, got " + totalTableLength + " ords and " + tableSize + " sets", meta);
- }
- entry.tableOffsets = new int[tableSize + 1];
- for (int i = 1; i < entry.tableOffsets.length; ++i) {
- entry.tableOffsets[i] = entry.tableOffsets[i - 1] + meta.readInt();
- }
- ramBytesUsed.addAndGet(RamUsageEstimator.sizeOf(entry.tableOffsets));
- } else if (entry.format != SORTED_SINGLE_VALUED && entry.format != SORTED_WITH_ADDRESSES) {
- throw new CorruptIndexException("Unknown format: " + entry.format, meta);
+ private static void readTermDict(ChecksumIndexInput meta, TermsDictEntry entry) throws IOException {
+ entry.termsDictSize = meta.readVLong();
+ entry.termsDictBlockShift = meta.readInt();
+ final int blockShift = meta.readInt();
+ final long addressesSize = (entry.termsDictSize + (1L << entry.termsDictBlockShift) - 1) >>> entry.termsDictBlockShift;
+ entry.termsAddressesMeta = DirectMonotonicReader.loadMeta(meta, addressesSize, blockShift);
+ entry.maxTermLength = meta.readInt();
+ entry.termsDataOffset = meta.readLong();
+ entry.termsDataLength = meta.readLong();
+ entry.termsAddressesOffset = meta.readLong();
+ entry.termsAddressesLength = meta.readLong();
+ entry.termsDictIndexShift = meta.readInt();
+ final long indexSize = (entry.termsDictSize + (1L << entry.termsDictIndexShift) - 1) >>> entry.termsDictIndexShift;
+ entry.termsIndexAddressesMeta = DirectMonotonicReader.loadMeta(meta, 1 + indexSize, blockShift);
+ entry.termsIndexOffset = meta.readLong();
+ entry.termsIndexLength = meta.readLong();
+ entry.termsIndexAddressesOffset = meta.readLong();
+ entry.termsIndexAddressesLength = meta.readLong();
+ }
+
+ private SortedNumericEntry readSortedNumeric(ChecksumIndexInput meta) throws IOException {
+ SortedNumericEntry entry = new SortedNumericEntry();
+ readNumeric(meta, entry);
+ entry.numDocsWithField = meta.readInt();
+ if (entry.numDocsWithField != entry.numValues) {
+ entry.addressesOffset = meta.readLong();
+ final int blockShift = meta.readVInt();
+ entry.addressesMeta = DirectMonotonicReader.loadMeta(meta, entry.numDocsWithField + 1, blockShift);
+ ramBytesUsed += entry.addressesMeta.ramBytesUsed();
+ entry.addressesLength = meta.readLong();
}
return entry;
}
@Override
- public NumericDocValues getNumeric(FieldInfo field) throws IOException {
- NumericEntry entry = numerics.get(field.name);
- DocIdSetIterator docsWithField;
-
- if (entry.format == SPARSE_COMPRESSED) {
- return getSparseNumericDocValues(entry);
- } else {
- if (entry.missingOffset == ALL_MISSING) {
- return DocValues.emptyNumeric();
- } else if (entry.missingOffset == ALL_LIVE) {
- LongValues values = getNumeric(entry);
- return new NumericDocValues() {
- private int docID = -1;
-
- @Override
- public int docID() {
- return docID;
- }
-
- @Override
- public int nextDoc() {
- docID++;
- if (docID == maxDoc) {
- docID = NO_MORE_DOCS;
- }
- return docID;
- }
-
- @Override
- public int advance(int target) {
- if (target >= maxDoc) {
- docID = NO_MORE_DOCS;
- } else {
- docID = target;
- }
- return docID;
- }
-
- @Override
- public long cost() {
- return entry.count;
- }
-
- @Override
- public long longValue() {
- return values.get(docID);
- }
- };
- } else {
- docsWithField = getDocsWithField(entry.missingOffset, maxDoc);
- LongValues values = getNumeric(entry);
- return new NumericDocValues() {
-
- @Override
- public long longValue() {
- return values.get(docsWithField.docID());
- }
-
- @Override
- public int docID() {
- return docsWithField.docID();
- }
+ public void close() throws IOException {
+ data.close();
+ }
- @Override
- public int nextDoc() throws IOException {
- return docsWithField.nextDoc();
- }
+ private static class NumericEntry {
+ long[] table;
+ byte bitsPerValue;
+ long docsWithFieldOffset;
+ long docsWithFieldLength;
+ long numValues;
+ long minValue;
+ long gcd;
+ long valuesOffset;
+ long valuesLength;
+ }
- @Override
- public int advance(int target) throws IOException {
- return docsWithField.advance(target);
- }
+ private static class BinaryEntry {
+ long dataOffset;
+ long dataLength;
+ long docsWithFieldOffset;
+ long docsWithFieldLength;
+ int numDocsWithField;
+ int minLength;
+ int maxLength;
+ long addressesOffset;
+ long addressesLength;
+ DirectMonotonicReader.Meta addressesMeta;
+ }
- @Override
- public long cost() {
- return docsWithField.cost();
- }
+ private static class TermsDictEntry {
+ long termsDictSize;
+ int termsDictBlockShift;
+ DirectMonotonicReader.Meta termsAddressesMeta;
+ int maxTermLength;
+ long termsDataOffset;
+ long termsDataLength;
+ long termsAddressesOffset;
+ long termsAddressesLength;
+ int termsDictIndexShift;
+ DirectMonotonicReader.Meta termsIndexAddressesMeta;
+ long termsIndexOffset;
+ long termsIndexLength;
+ long termsIndexAddressesOffset;
+ long termsIndexAddressesLength;
+ }
- };
- }
- }
+ private static class SortedEntry extends TermsDictEntry {
+ long docsWithFieldOffset;
+ long docsWithFieldLength;
+ int numDocsWithField;
+ byte bitsPerValue;
+ long ordsOffset;
+ long ordsLength;
}
- @Override
- public long ramBytesUsed() {
- return ramBytesUsed.get();
+ private static class SortedSetEntry extends TermsDictEntry {
+ SortedEntry singleValueEntry;
+ long docsWithFieldOffset;
+ long docsWithFieldLength;
+ int numDocsWithField;
+ byte bitsPerValue;
+ long ordsOffset;
+ long ordsLength;
+ DirectMonotonicReader.Meta addressesMeta;
+ long addressesOffset;
+ long addressesLength;
}
- @Override
- public synchronized Collection<Accountable> getChildResources() {
- List<Accountable> resources = new ArrayList<>();
- resources.addAll(Accountables.namedAccountables("addresses field", addressInstances));
- resources.addAll(Accountables.namedAccountables("reverse index field", reverseIndexInstances));
- resources.addAll(Accountables.namedAccountables("direct addresses meta field", directAddressesMeta));
- return Collections.unmodifiableList(resources);
+ private static class SortedNumericEntry extends NumericEntry {
+ int numDocsWithField;
+ DirectMonotonicReader.Meta addressesMeta;
+ long addressesOffset;
+ long addressesLength;
}
@Override
- public void checkIntegrity() throws IOException {
- CodecUtil.checksumEntireFile(data);
+ public long ramBytesUsed() {
+ return ramBytesUsed;
}
@Override
- public String toString() {
- return getClass().getSimpleName() + "(fields=" + numFields + ")";
+ public NumericDocValues getNumeric(FieldInfo field) throws IOException {
+ NumericEntry entry = numerics.get(field.name);
+ return getNumeric(entry);
}
- LongValues getNumeric(NumericEntry entry) throws IOException {
- switch (entry.format) {
- case CONST_COMPRESSED: {
- final long constant = entry.minValue;
- final Bits live = getLiveBits(entry.missingOffset, (int)entry.count);
- return new LongValues() {
- @Override
- public long get(long index) {
- return live.get((int)index) ? constant : 0;
- }
- };
- }
- case DELTA_COMPRESSED: {
- RandomAccessInput slice = this.data.randomAccessSlice(entry.offset, entry.endOffset - entry.offset);
- final long delta = entry.minValue;
- final LongValues values = DirectReader.getInstance(slice, entry.bitsPerValue, 0);
- return new LongValues() {
- @Override
- public long get(long id) {
- return delta + values.get(id);
- }
- };
- }
- case GCD_COMPRESSED: {
- RandomAccessInput slice = this.data.randomAccessSlice(entry.offset, entry.endOffset - entry.offset);
- final long min = entry.minValue;
- final long mult = entry.gcd;
- final LongValues quotientReader = DirectReader.getInstance(slice, entry.bitsPerValue, 0);
- return new LongValues() {
- @Override
- public long get(long id) {
- return min + mult * quotientReader.get(id);
- }
- };
- }
- case TABLE_COMPRESSED: {
- RandomAccessInput slice = this.data.randomAccessSlice(entry.offset, entry.endOffset - entry.offset);
- final long table[] = entry.table;
- final LongValues ords = DirectReader.getInstance(slice, entry.bitsPerValue, 0);
- return new LongValues() {
- @Override
- public long get(long id) {
- return table[(int) ords.get(id)];
- }
- };
- }
- case SPARSE_COMPRESSED:
- final SparseNumericDocValues values = getSparseNumericDocValues(entry);
- final long missingValue;
- switch (entry.numberType) {
- case ORDINAL:
- missingValue = -1L;
- break;
- case VALUE:
- missingValue = 0L;
- break;
- default:
- throw new AssertionError();
- }
- return new SparseNumericDocValuesRandomAccessWrapper(values, missingValue);
- default:
- throw new AssertionError();
- }
- }
+ private NumericDocValues getNumeric(NumericEntry entry) throws IOException {
+ if (entry.docsWithFieldOffset == -2) {
+ // empty
+ return DocValues.emptyNumeric();
+ } else if (entry.docsWithFieldOffset == -1) {
+ // dense
+ final LongValues normValues = getNumericValues(entry);
+ return new NumericDocValues() {
- static final class SparseNumericDocValues extends NumericDocValues {
+ int doc = -1;
- final int docIDsLength;
- final LongValues docIds, values;
+ @Override
+ public long longValue() throws IOException {
+ return normValues.get(doc);
+ }
- int index, doc;
+ @Override
+ public int docID() {
+ return doc;
+ }
- SparseNumericDocValues(int docIDsLength, LongValues docIDs, LongValues values) {
- this.docIDsLength = docIDsLength;
- this.docIds = docIDs;
- this.values = values;
- reset();
- }
+ @Override
+ public int nextDoc() throws IOException {
+ return advance(doc + 1);
+ }
- void reset() {
- index = -1;
- doc = -1;
- }
+ @Override
+ public int advance(int target) throws IOException {
+ if (target >= maxDoc) {
+ return doc = NO_MORE_DOCS;
+ }
+ return doc = target;
+ }
- @Override
- public int docID() {
- return doc;
- }
+ @Override
+ public long cost() {
+ return maxDoc;
+ }
- @Override
- public int nextDoc() throws IOException {
- if (index >= docIDsLength - 1) {
- index = docIDsLength;
- return doc = NO_MORE_DOCS;
- }
- return doc = (int) docIds.get(++index);
- }
+ };
+ } else {
+ // sparse
+ final LongValues values = getNumericValues(entry);
+ final IndexedDISI disi = new IndexedDISI(data, entry.docsWithFieldOffset, entry.docsWithFieldLength, entry.numValues);
+ return new NumericDocValues() {
- @Override
- public int advance(int target) throws IOException {
- long loIndex = index;
- long step = 1;
- long hiIndex;
- int hiDoc;
-
- // gallop forward by exponentially growing the interval
- // in order to find an interval so that the target doc
- // is in ]lo, hi]. Compared to a regular binary search,
- // this optimizes the case that the caller performs many
- // advance calls by small deltas
- do {
- hiIndex = index + step;
- if (hiIndex >= docIDsLength) {
- hiIndex = docIDsLength;
- hiDoc = NO_MORE_DOCS;
- break;
- }
- hiDoc = (int) docIds.get(hiIndex);
- if (hiDoc >= target) {
- break;
+ @Override
+ public int advance(int target) throws IOException {
+ return disi.advance(target);
}
- step <<= 1;
- } while (true);
-
- // now binary search
- while (loIndex + 1 < hiIndex) {
- final long midIndex = (loIndex + 1 + hiIndex) >>> 1;
- final int midDoc = (int) docIds.get(midIndex);
- if (midDoc >= target) {
- hiIndex = midIndex;
- hiDoc = midDoc;
- } else {
- loIndex = midIndex;
+
+ @Override
+ public int nextDoc() throws IOException {
+ return disi.nextDoc();
}
- }
- index = (int) hiIndex;
- return doc = hiDoc;
- }
+ @Override
+ public int docID() {
+ return disi.docID();
+ }
- @Override
- public long longValue() {
- assert index >= 0;
- assert index < docIDsLength;
- return values.get(index);
- }
+ @Override
+ public long cost() {
+ return disi.cost();
+ }
- @Override
- public long cost() {
- return docIDsLength;
+ @Override
+ public long longValue() throws IOException {
+ return values.get(disi.index());
+ }
+ };
}
}
- static class SparseNumericDocValuesRandomAccessWrapper extends LongValues {
-
- final SparseNumericDocValues values;
- final long missingValue;
-
- SparseNumericDocValuesRandomAccessWrapper(SparseNumericDocValues values, long missingValue) {
- this.values = values;
- this.missingValue = missingValue;
- }
-
- @Override
- public long get(long longIndex) {
- final int index = Math.toIntExact(longIndex);
- int doc = values.docID();
- if (doc >= index) {
- values.reset();
+ private LongValues getNumericValues(NumericEntry entry) throws IOException {
+ if (entry.bitsPerValue == 0) {
+ return new LongValues() {
+ @Override
+ public long get(long index) {
+ return entry.minValue;
+ }
+ };
+ } else {
+ final RandomAccessInput slice = data.randomAccessSlice(entry.valuesOffset, entry.valuesLength);
+ LongValues values = DirectReader.getInstance(slice, entry.bitsPerValue);
+ if (entry.gcd != 1) {
+ values = applyGcd(values, entry.gcd);
}
- assert values.docID() < index;
- try {
- doc = values.advance(index);
- } catch (IOException e) {
- throw new RuntimeException(e);
+ if (entry.minValue != 0) {
+ values = applyDelta(values, entry.minValue);
}
- if (doc == index) {
- return values.longValue();
- } else {
- return missingValue;
+ if (entry.table != null) {
+ values = applyTable(values, entry.table);
}
- }
-
- }
-
- LegacyBinaryDocValues getLegacyBinary(FieldInfo field) throws IOException {
- BinaryEntry bytes = binaries.get(field.name);
- switch(bytes.format) {
- case BINARY_FIXED_UNCOMPRESSED:
- return getFixedBinary(field, bytes);
- case BINARY_VARIABLE_UNCOMPRESSED:
- return getVariableBinary(field, bytes);
- case BINARY_PREFIX_COMPRESSED:
- return getCompressedBinary(field, bytes);
- default:
- throw new AssertionError();
+ return values;
}
}
- @Override
- public BinaryDocValues getBinary(FieldInfo field) throws IOException {
- BinaryEntry be = binaries.get(field.name);
- DocIdSetIterator docsWithField = getDocsWithField(be.missingOffset, maxDoc);
- LegacyBinaryDocValues values = getLegacyBinary(field);
- return new BinaryDocValues() {
-
- @Override
- public int nextDoc() throws IOException {
- return docsWithField.nextDoc();
- }
-
+ private LongValues applyDelta(LongValues values, long delta) {
+ return new LongValues() {
@Override
- public int docID() {
- return docsWithField.docID();
- }
-
- @Override
- public long cost() {
- return docsWithField.cost();
- }
-
- @Override
- public int advance(int target) throws IOException {
- return docsWithField.advance(target);
- }
-
- @Override
- public BytesRef binaryValue() {
- return values.get(docsWithField.docID());
+ public long get(long index) {
+ return delta + values.get(index);
}
};
}
- private LegacyBinaryDocValues getFixedBinary(FieldInfo field, final BinaryEntry bytes) throws IOException {
- final IndexInput data = this.data.slice("fixed-binary", bytes.offset, bytes.count * bytes.maxLength);
-
- final BytesRef term = new BytesRef(bytes.maxLength);
- final byte[] buffer = term.bytes;
- final int length = term.length = bytes.maxLength;
-
- return new LongBinaryDocValues() {
+ private LongValues applyGcd(LongValues values, long gcd) {
+ return new LongValues() {
@Override
- public BytesRef get(long id) {
- try {
- data.seek(id * length);
- data.readBytes(buffer, 0, buffer.length);
- return term;
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
+ public long get(long index) {
+ return values.get(index) * gcd;
}
};
}
- private LegacyBinaryDocValues getVariableBinary(FieldInfo field, final BinaryEntry bytes) throws IOException {
- final RandomAccessInput addressesData = this.data.randomAccessSlice(bytes.addressesOffset, bytes.addressesEndOffset - bytes.addressesOffset);
- final LongValues addresses = DirectMonotonicReader.getInstance(bytes.addressesMeta, addressesData);
-
- final IndexInput data = this.data.slice("var-binary", bytes.offset, bytes.addressesOffset - bytes.offset);
- final BytesRef term = new BytesRef(Math.max(0, bytes.maxLength));
- final byte buffer[] = term.bytes;
-
- return new LongBinaryDocValues() {
+ private LongValues applyTable(LongValues values, long[] table) {
+ return new LongValues() {
@Override
- public BytesRef get(long id) {
- long startAddress = addresses.get(id);
- long endAddress = addresses.get(id+1);
- int length = (int) (endAddress - startAddress);
- try {
- data.seek(startAddress);
- data.readBytes(buffer, 0, length);
- term.length = length;
- return term;
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
+ public long get(long index) {
+ return table[(int) values.get(index)];
}
};
}
- /** returns an address instance for prefix-compressed binary values. */
- private synchronized MonotonicBlockPackedReader getIntervalInstance(FieldInfo field, BinaryEntry bytes) throws IOException {
- MonotonicBlockPackedReader addresses = addressInstances.get(field.name);
- if (addresses == null) {
- data.seek(bytes.addressesOffset);
- final long size = (bytes.count + INTERVAL_MASK) >>> INTERVAL_SHIFT;
- addresses = MonotonicBlockPackedReader.of(data, bytes.packedIntsVersion, bytes.blockSize, size, false);
- if (!merging) {
- addressInstances.put(field.name, addresses);
- ramBytesUsed.addAndGet(addresses.ramBytesUsed() + Integer.BYTES);
- }
- }
- return addresses;
- }
-
- /** returns a reverse lookup instance for prefix-compressed binary values. */
- private synchronized ReverseTermsIndex getReverseIndexInstance(FieldInfo field, BinaryEntry bytes) throws IOException {
- ReverseTermsIndex index = reverseIndexInstances.get(field.name);
- if (index == null) {
- index = new ReverseTermsIndex();
- data.seek(bytes.reverseIndexOffset);
- long size = (bytes.count + REVERSE_INTERVAL_MASK) >>> REVERSE_INTERVAL_SHIFT;
- index.termAddresses = MonotonicBlockPackedReader.of(data, bytes.packedIntsVersion, bytes.blockSize, size, false);
- long dataSize = data.readVLong();
- PagedBytes pagedBytes = new PagedBytes(15);
- pagedBytes.copy(data, dataSize);
- index.terms = pagedBytes.freeze(true);
- if (!merging) {
- reverseIndexInstances.put(field.name, index);
- ramBytesUsed.addAndGet(index.ramBytesUsed());
- }
- }
- return index;
- }
-
- private LegacyBinaryDocValues getCompressedBinary(FieldInfo field, final BinaryEntry bytes) throws IOException {
- final MonotonicBlockPackedReader addresses = getIntervalInstance(field, bytes);
- final ReverseTermsIndex index = getReverseIndexInstance(field, bytes);
- assert addresses.size() > 0; // we don't have to handle empty case
- IndexInput slice = data.slice("terms", bytes.offset, bytes.addressesOffset - bytes.offset);
- return new CompressedBinaryDocValues(bytes, addresses, index, slice);
- }
-
@Override
- public SortedDocValues getSorted(FieldInfo field) throws IOException {
- final int valueCount = (int) binaries.get(field.name).count;
- final LegacyBinaryDocValues binary = getLegacyBinary(field);
- NumericEntry entry = ords.get(field.name);
- final LongValues ordinals = getNumeric(entry);
- if (entry.format == SPARSE_COMPRESSED) {
- final SparseNumericDocValues sparseValues = ((SparseNumericDocValuesRandomAccessWrapper) ordinals).values;
- return new SortedDocValues() {
+ public BinaryDocValues getBinary(FieldInfo field) throws IOException {
+ BinaryEntry entry = binaries.get(field.name);
+ if (entry.docsWithFieldOffset == -2) {
+ return DocValues.emptyBinary();
+ }
+ IndexInput bytesSlice = data.slice("fixed-binary", entry.dataOffset, entry.dataLength);
+ BytesRefs bytesRefs;
+ if (entry.minLength == entry.maxLength) {
+ bytesRefs = new BytesRefs() {
+ BytesRef bytes = new BytesRef(new byte[entry.maxLength], 0, entry.maxLength);
@Override
- public int ordValue() {
- return (int) sparseValues.longValue();
+ public BytesRef get(int index) throws IOException {
+ bytesSlice.seek((long) index * bytes.length);
+ bytesSlice.readBytes(bytes.bytes, 0, bytes.length);
+ return bytes;
}
-
+ };
+ } else {
+ final RandomAccessInput addressesData = this.data.randomAccessSlice(entry.addressesOffset, entry.addressesLength);
+ final LongValues addresses = DirectMonotonicReader.getInstance(entry.addressesMeta, addressesData);
+ bytesRefs = new BytesRefs() {
+ BytesRef bytes = new BytesRef(entry.maxLength);
@Override
- public BytesRef lookupOrd(int ord) {
- return binary.get(ord);
+ BytesRef get(int index) throws IOException {
+ long startOffset = addresses.get(index);
+ bytes.length = (int) (addresses.get(index + 1L) - startOffset);
+ bytesSlice.seek(startOffset);
+ bytesSlice.readBytes(bytes.bytes, 0, bytes.length);
+ return bytes;
}
+ };
+ }
+
+ if (entry.docsWithFieldOffset == -1) {
+ // dense
+ return new BinaryDocValues() {
+
+ int doc = -1;
@Override
- public int getValueCount() {
- return valueCount;
+ public int nextDoc() throws IOException {
+ return advance(doc + 1);
}
@Override
public int docID() {
- return sparseValues.docID();
+ return doc;
}
@Override
- public int nextDoc() throws IOException {
- return sparseValues.nextDoc();
+ public long cost() {
+ return maxDoc;
}
@Override
public int advance(int target) throws IOException {
- return sparseValues.advance(target);
+ if (target >= maxDoc) {
+ return doc = NO_MORE_DOCS;
+ }
+ return doc = target;
}
@Override
- public long cost() {
- return sparseValues.cost();
+ public BytesRef binaryValue() throws IOException {
+ return bytesRefs.get(doc);
}
-
};
- }
- return new SortedDocValues() {
- private int docID = -1;
- private int ord;
+ } else {
+ // sparse
+ final IndexedDISI disi = new IndexedDISI(data, entry.docsWithFieldOffset, entry.docsWithFieldLength, entry.numDocsWithField);
+ return new BinaryDocValues() {
- @Override
- public int docID() {
- return docID;
- }
-
- @Override
- public int nextDoc() throws IOException {
- assert docID != NO_MORE_DOCS;
- while (true) {
- docID++;
- if (docID == maxDoc) {
- docID = NO_MORE_DOCS;
- break;
- }
- ord = (int) ordinals.get(docID);
- if (ord != -1) {
- break;
- }
+ @Override
+ public int nextDoc() throws IOException {
+ return disi.nextDoc();
}
- return docID;
- }
- @Override
- public int advance(int target) throws IOException {
- if (target >= maxDoc) {
- docID = NO_MORE_DOCS;
- return docID;
- } else {
- docID = target-1;
- return nextDoc();
+ @Override
+ public int docID() {
+ return disi.docID();
}
- }
-
- @Override
- public int ordValue() {
- return ord;
- }
-
- @Override
- public long cost() {
- // TODO
- return 0;
- }
-
- @Override
- public BytesRef lookupOrd(int ord) {
- return binary.get(ord);
- }
- @Override
- public int getValueCount() {
- return valueCount;
- }
+ @Override
+ public long cost() {
+ return disi.cost();
+ }
- @Override
- public int lookupTerm(BytesRef key) throws IOException {
- if (binary instanceof CompressedBinaryDocValues) {
- return (int) ((CompressedBinaryDocValues)binary).lookupTerm(key);
- } else {
- return super.lookupTerm(key);
+ @Override
+ public int advance(int target) throws IOException {
+ return disi.advance(target);
}
- }
- @Override
- public TermsEnum termsEnum() {
- if (binary instanceof CompressedBinaryDocValues) {
- return ((CompressedBinaryDocValues)binary).getTermsEnum();
- } else {
- return super.termsEnum();
+ @Override
+ public BytesRef binaryValue() throws IOException {
+ return bytesRefs.get(disi.index());
}
- }
- };
+ };
+ }
}
- /** returns an address instance for sortedset ordinal lists */
- private LongValues getOrdIndexInstance(FieldInfo field, NumericEntry entry) throws IOException {
- RandomAccessInput data = this.data.randomAccessSlice(entry.offset, entry.endOffset - entry.offset);
- return DirectMonotonicReader.getInstance(entry.monotonicMeta, data);
+ private static abstract class BytesRefs {
+ abstract BytesRef get(int index) throws IOException;
}
@Override
- public SortedNumericDocValues getSortedNumeric(FieldInfo field) throws IOException {
- SortedSetEntry ss = sortedNumerics.get(field.name);
- if (ss.format == SORTED_SINGLE_VALUED) {
- NumericEntry numericEntry = numerics.get(field.name);
- final LongValues values = getNumeric(numericEntry);
- if (numericEntry.format == SPARSE_COMPRESSED) {
- SparseNumericDocValues sparseValues = ((SparseNumericDocValuesRandomAccessWrapper) values).values;
- return new SortedNumericDocValues() {
-
- @Override
- public long nextValue() throws IOException {
- return sparseValues.longValue();
- }
-
- @Override
- public int docValueCount() {
- return 1;
- }
-
- @Override
- public int docID() {
- return sparseValues.docID();
- }
-
- @Override
- public int nextDoc() throws IOException {
- return sparseValues.nextDoc();
- }
-
- @Override
- public int advance(int target) throws IOException {
- return sparseValues.advance(target);
- }
-
- @Override
- public long cost() {
- return sparseValues.cost();
- }
+ public SortedDocValues getSorted(FieldInfo field) throws IOException {
+ SortedEntry entry = sorted.get(field.name);
+ return getSorted(entry);
+ }
- };
- }
- final DocIdSetIterator docsWithField = getDocsWithField(numericEntry.missingOffset, maxDoc);
-
- return new SortedNumericDocValues() {
+ private SortedDocValues getSorted(SortedEntry entry) throws IOException {
+ if (entry.docsWithFieldOffset == -2) {
+ return DocValues.emptySorted();
+ }
+ final LongValues ords;
+ if (entry.bitsPerValue == 0) {
+ ords = new LongValues() {
@Override
- public int docID() {
- return docsWithField.docID();
+ public long get(long index) {
+ return 0L;
}
+ };
+ } else {
+ final RandomAccessInput slice = data.randomAccessSlice(entry.ordsOffset, entry.ordsLength);
+ ords = DirectReader.getInstance(slice, entry.bitsPerValue);
+ }
+
+ if (entry.docsWithFieldOffset == -1) {
+ // dense
+ return new BaseSortedDocValues(entry, data) {
+
+ int doc = -1;
@Override
public int nextDoc() throws IOException {
- return docsWithField.nextDoc();
+ return advance(doc + 1);
}
@Override
- public int advance(int target) throws IOException {
- return docsWithField.advance(target);
+ public int docID() {
+ return doc;
}
@Override
public long cost() {
- return docsWithField.cost();
+ return maxDoc;
}
@Override
- public int docValueCount() {
- return 1;
+ public int advance(int target) throws IOException {
+ if (target >= maxDoc) {
+ return doc = NO_MORE_DOCS;
+ }
+ return doc = target;
}
@Override
- public long nextValue() {
- return values.get(docsWithField.docID());
+ public int ordValue() {
+ return (int) ords.get(doc);
}
};
- } else if (ss.format == SORTED_WITH_ADDRESSES) {
- NumericEntry numericEntry = numerics.get(field.name);
- final LongValues values = getNumeric(numericEntry);
- final LongValues ordIndex = getOrdIndexInstance(field, ordIndexes.get(field.name));
-
- return new SortedNumericDocValues() {
- long startOffset;
- long endOffset;
- int docID = -1;
- long upto;
+ } else {
+ // sparse
+ final IndexedDISI disi = new IndexedDISI(data, entry.docsWithFieldOffset, entry.docsWithFieldLength, entry.numDocsWithField);
+ return new BaseSortedDocValues(entry, data) {
@Override
- public int docID() {
- return docID;
+ public int nextDoc() throws IOException {
+ return disi.nextDoc();
}
@Override
- public int nextDoc() {
- while (true) {
- docID++;
- if (docID == maxDoc) {
- docID = NO_MORE_DOCS;
- return docID;
- }
- startOffset = ordIndex.get(docID);
- endOffset = ordIndex.get(docID+1L);
- if (endOffset > startOffset) {
- break;
- }
- }
- upto = startOffset;
- return docID;
+ public int docID() {
+ return disi.docID();
}
@Override
- public int advance(int target) {
- if (target >= maxDoc) {
- docID = NO_MORE_DOCS;
- return docID;
- } else {
- docID = target-1;
- return nextDoc();
- }
- }
-
- @Override
public long cost() {
- // TODO
- return 0;
+ return disi.cost();
}
-
+
@Override
- public int docValueCount() {
- return (int) (endOffset - startOffset);
+ public int advance(int target) throws IOException {
+ return disi.advance(target);
}
-
+
@Override
- public long nextValue() {
- return values.get(upto++);
+ public int ordValue() {
+ return (int) ords.get(disi.index());
}
};
- } else if (ss.format == SORTED_SET_TABLE) {
- NumericEntry entry = ords.get(field.name);
- final LongValues ordinals = getNumeric(entry);
+ }
+ }
- final long[] table = ss.table;
- final int[] offsets = ss.tableOffsets;
- return new SortedNumericDocValues() {
- int startOffset;
- int endOffset;
- int docID = -1;
- int upto;
+ private static abstract class BaseSortedDocValues extends SortedDocValues {
- @Override
- public int docID() {
- return docID;
- }
+ final SortedEntry entry;
+ final IndexInput data;
+ final TermsEnum termsEnum;
- @Override
- public int nextDoc() {
- while (true) {
- docID++;
- if (docID == maxDoc) {
- docID = NO_MORE_DOCS;
- return docID;
- }
- int ord = (int) ordinals.get(docID);
- startOffset = offsets[ord];
- endOffset = offsets[ord+1];
- if (endOffset > startOffset) {
- break;
- }
- }
- upto = startOffset;
- return docID;
- }
+ BaseSortedDocValues(SortedEntry entry, IndexInput data) throws IOException {
+ this.entry = entry;
+ this.data = data;
+ this.termsEnum = termsEnum();
+ }
- @Override
- public int advance(int target) {
- if (target >= maxDoc) {
- docID = NO_MORE_DOCS;
- return docID;
- } else {
- docID = target-1;
- return nextDoc();
- }
- }
-
- @Override
- public long cost() {
- // TODO
- return 0;
- }
+ @Override
+ public int getValueCount() {
+ return Math.toIntExact(entry.termsDictSize);
+ }
- @Override
- public int docValueCount() {
- return endOffset - startOffset;
- }
-
- @Override
- public long nextValue() {
- return table[upto++];
- }
- };
- } else {
- throw new AssertionError();
+ @Override
+ public BytesRef lookupOrd(int ord) throws IOException {
+ termsEnum.seekExact(ord);
+ return termsEnum.term();
}
- }
- @Override
- public SortedSetDocValues getSortedSet(FieldInfo field) throws IOException {
- SortedSetEntry ss = sortedSets.get(field.name);
- switch (ss.format) {
- case SORTED_SINGLE_VALUED:
- return DocValues.singleton(getSorted(field));
- case SORTED_WITH_ADDRESSES:
- return getSortedSetWithAddresses(field);
- case SORTED_SET_TABLE:
- return getSortedSetTable(field, ss);
- default:
- throw new AssertionError();
+ @Override
+ public int lookupTerm(BytesRef key) throws IOException {
+ SeekStatus status = termsEnum.seekCeil(key);
+ switch (status) {
+ case FOUND:
+ return Math.toIntExact(termsEnum.ord());
+ default:
+ return Math.toIntExact(-1L - termsEnum.ord());
+ }
+ }
+
+ @Override
+ public TermsEnum termsEnum() throws IOException {
+ return new TermsDict(entry, data);
}
}
- private SortedSetDocValues getSortedSetWithAddresses(FieldInfo field) throws IOException {
- final long valueCount = binaries.get(field.name).count;
- // we keep the byte[]s and list of ords on disk, these could be large
- final LongBinaryDocValues binary = (LongBinaryDocValues) getLegacyBinary(field);
- final LongValues ordinals = getNumeric(ords.get(field.name));
- // but the addresses to the ord stream are in RAM
- final LongValues ordIndex = getOrdIndexInstance(field, ordIndexes.get(field.name));
+ private static abstract class BaseSortedSetDocValues extends SortedSetDocValues {
- return new LegacySortedSetDocValuesWrapper(new LegacySortedSetDocValues() {
- long startOffset;
- long offset;
- long endOffset;
+ final SortedSetEntry entry;
+ final IndexInput data;
+ final TermsEnum termsEnum;
- @Override
- public long nextOrd() {
- if (offset == endOffset) {
- return NO_MORE_ORDS;
- } else {
- long ord = ordinals.get(offset);
- offset++;
- return ord;
- }
- }
+ BaseSortedSetDocValues(SortedSetEntry entry, IndexInput data) throws IOException {
+ this.entry = entry;
+ this.data = data;
+ this.termsEnum = termsEnum();
+ }
- @Override
- public void setDocument(int docID) {
- startOffset = offset = ordIndex.get(docID);
- endOffset = ordIndex.get(docID+1L);
- }
+ @Override
+ public long getValueCount() {
+ return entry.termsDictSize;
+ }
- @Override
- public BytesRef lookupOrd(long ord) {
- return binary.get(ord);
- }
+ @Override
+ public BytesRef lookupOrd(long ord) throws IOException {
+ termsEnum.seekExact(ord);
+ return termsEnum.term();
+ }
- @Override
- public long getValueCount() {
- return valueCount;
+ @Override
+ public long lookupTerm(BytesRef key) throws IOException {
+ SeekStatus status = termsEnum.seekCeil(key);
+ switch (status) {
+ case FOUND:
+ return termsEnum.ord();
+ default:
+ return -1L - termsEnum.ord();
}
+ }
- @Override
- public long lookupTerm(BytesRef key) {
- if (binary instanceof CompressedBinaryDocValues) {
- return ((CompressedBinaryDocValues)binary).lookupTerm(key);
- } else {
- return super.lookupTerm(key);
+ @Override
+ public TermsEnum termsEnum() throws IOException {
+ return new TermsDict(entry, data);
+ }
+ }
+
+ private static class TermsDict extends TermsEnum {
+
+ final TermsDictEntry entry;
+ final LongValues blockAddresses;
+ final IndexInput bytes;
+ final long blockMask;
+ final LongValues indexAddresses;
+ final IndexInput indexBytes;
+ final BytesRef term;
+ long ord = -1;
+
+ TermsDict(TermsDictEntry entry, IndexInput data) throws IOException {
+ this.entry = entry;
+ RandomAccessInput addressesSlice = data.randomAccessSlice(entry.termsAddressesOffset, entry.termsAddressesLength);
+ blockAddresses = DirectMonotonicReader.getInstance(entry.termsAddressesMeta, addressesSlice);
+ bytes = data.slice("terms", entry.termsDataOffset, entry.termsDataLength);
+ blockMask = (1L << entry.termsDictBlockShift) - 1;
+ RandomAccessInput indexAddressesSlice = data.randomAccessSlice(entry.termsIndexAddressesOffset, entry.termsIndexAddressesLength);
+ indexAddresses = DirectMonotonicReader.getInstance(entry.termsIndexAddressesMeta, indexAddressesSlice);
+ indexBytes = data.slice("terms-index", entry.termsIndexOffset, entry.termsIndexLength);
+ term = new BytesRef(entry.maxTermLength);
+ }
+
+ @Override
+ public BytesRef next() throws IOException {
+ if (++ord >= entry.termsDictSize) {
+ return null;
+ }
+ if ((ord & blockMask) == 0L) {
+ term.length = bytes.readVInt();
+ bytes.readBytes(term.bytes, 0, term.length);
+ } else {
+ final int token = Byte.toUnsignedInt(bytes.readByte());
+ int prefixLength = token & 0x0F;
+ int suffixLength = 1 + (token >>> 4);
+ if (prefixLength == 15) {
+ prefixLength += bytes.readVInt();
+ }
+ if (suffixLength == 16) {
+ suffixLength += bytes.readVInt();
}
+ term.length = prefixLength + suffixLength;
+ bytes.readBytes(term.bytes, prefixLength, suffixLength);
}
+ return term;
+ }
- @Override
- public TermsEnum termsEnum() {
- if (binary instanceof CompressedBinaryDocValues) {
- return ((CompressedBinaryDocValues)binary).getTermsEnum();
+ @Override
+ public void seekExact(long ord) throws IOException {
+ if (ord < 0 || ord >= entry.termsDictSize) {
+ throw new IndexOutOfBoundsException();
+ }
+ final long blockIndex = ord >>> entry.termsDictBlockShift;
+ final long blockAddress = blockAddresses.get(blockIndex);
+ bytes.seek(blockAddress);
+ this.ord = (blockIndex << entry.termsDictBlockShift) - 1;
+ do {
+ next();
+ } while (this.ord < ord);
+ }
+
+ private BytesRef getTermFromIndex(long index) throws IOException {
+ assert index >= 0 && index <= (entry.termsDictSize - 1) >>> entry.termsDictIndexShift;
+ final long start = indexAddresses.get(index);
+ term.length = (int) (indexAddresses.get(index + 1) - start);
+ indexBytes.seek(start);
+ indexBytes.readBytes(term.bytes, 0, term.length);
+ return term;
+ }
+
+ private long seekTermsIndex(BytesRef text) throws IOException {
+ long lo = 0L;
+ long hi = (entry.termsDictSize - 1) >>> entry.termsDictIndexShift;
+ while (lo <= hi) {
+ final long mid = (lo + hi) >>> 1;
+ getTermFromIndex(mid);
+ final int cmp = term.compareTo(text);
+ if (cmp <= 0) {
+ lo = mid + 1;
} else {
- return super.termsEnum();
+ hi = mid - 1;
}
}
- }, maxDoc);
- }
- private SortedSetDocValues getSortedSetTable(FieldInfo field, SortedSetEntry ss) throws IOException {
- final long valueCount = binaries.get(field.name).count;
- final LongBinaryDocValues binary = (LongBinaryDocValues) getLegacyBinary(field);
- final NumericEntry ordinalsEntry = ords.get(field.name);
- final LongValues ordinals = getNumeric(ordinalsEntry);
+ assert hi < 0 || getTermFromIndex(hi).compareTo(text) <= 0;
+ assert hi == ((entry.termsDictSize - 1) >>> entry.termsDictIndexShift) || getTermFromIndex(hi + 1).compareTo(text) > 0;
- final long[] table = ss.table;
- final int[] offsets = ss.tableOffsets;
-
- return new LegacySortedSetDocValuesWrapper(new LegacySortedSetDocValues() {
+ return hi;
+ }
- int offset, startOffset, endOffset;
+ private BytesRef getFirstTermFromBlock(long block) throws IOException {
+ assert block >= 0 && block <= (entry.termsDictSize - 1) >>> entry.termsDictBlockShift;
+ final long blockAddress = blockAddresses.get(block);
+ bytes.seek(blockAddress);
+ term.length = bytes.readVInt();
+ bytes.readBytes(term.bytes, 0, term.length);
+ return term;
+ }
- @Override
- public void setDocument(int docID) {
- final int ord = (int) ordinals.get(docID);
- offset = startOffset = offsets[ord];
- endOffset = offsets[ord + 1];
+ private long seekBlock(BytesRef text) throws IOException {
+ long index = seekTermsIndex(text);
+ if (index == -1L) {
+ return -1L;
}
- @Override
- public long nextOrd() {
- if (offset == endOffset) {
- return NO_MORE_ORDS;
+ long ordLo = index << entry.termsDictIndexShift;
+ long ordHi = Math.min(entry.termsDictSize, ordLo + (1L << entry.termsDictIndexShift)) - 1L;
+
+ long blockLo = ordLo >>> entry.termsDictBlockShift;
+ long blockHi = ordHi >>> entry.termsDictBlockShift;
+
+ while (blockLo <= blockHi) {
+ final long blockMid = (blockLo + blockHi) >>> 1;
+ getFirstTermFromBlock(blockMid);
+ final int cmp = term.compareTo(text);
+ if (cmp <= 0) {
+ blockLo = blockMid + 1;
} else {
- return table[offset++];
+ blockHi = blockMid - 1;
}
}
- @Override
- public BytesRef lookupOrd(long ord) {
- return binary.get(ord);
- }
+ assert blockHi < 0 || getFirstTermFromBlock(blockHi).compareTo(text) <= 0;
+ assert blockHi == ((entry.termsDictSize - 1) >>> entry.termsDictBlockShift) || getFirstTermFromBlock(blockHi + 1).compareTo(text) > 0;
- @Override
- public long getValueCount() {
- return valueCount;
- }
+ return blockHi;
+ }
- @Override
- public long lookupTerm(BytesRef key) {
- if (binary instanceof CompressedBinaryDocValues) {
- return ((CompressedBinaryDocValues) binary).lookupTerm(key);
- } else {
- return super.lookupTerm(key);
+ @Override
+ public SeekStatus seekCeil(BytesRef text) throws IOException {
+ final long block = seekBlock(text);
+ if (block == -1) {
+ // before the first term
+ seekExact(0L);
+ return SeekStatus.NOT_FOUND;
+ }
+ final long blockAddress = blockAddresses.get(block);
+ this.ord = block << entry.termsDictBlockShift;
+ bytes.seek(blockAddress);
+ term.length = bytes.readVInt();
+ bytes.readBytes(term.bytes, 0, term.length);
+ while (true) {
+ int cmp = term.compareTo(text);
+ if (cmp == 0) {
+ return SeekStatus.FOUND;
+ } else if (cmp > 0) {
+ return SeekStatus.NOT_FOUND;
+ }
+ if (next() == null) {
+ return SeekStatus.END;
}
}
+ }
- @Override
- public TermsEnum termsEnum() {
- if (binary instanceof CompressedBinaryDocValues) {
- return ((CompressedBinaryDocValues) binary).getTermsEnum();
- } else {
- return super.termsEnum();
- }
- }
- }, maxDoc);
- }
+ @Override
+ public BytesRef term() throws IOException {
+ return term;
+ }
- private Bits getLiveBits(final long offset, final int count) throws IOException {
- if (offset == ALL_MISSING) {
- return new Bits.MatchNoBits(count);
- } else if (offset == ALL_LIVE) {
- return new Bits.MatchAllBits(count);
- } else {
- int length = (int) ((count + 63L) >>> 6);
- final RandomAccessInput in = data.randomAccessSlice(offset, length << 3);
- return new Bits() {
+ @Override
+ public long ord() throws IOException {
+ return ord;
+ }
- int wordIndex = -1;
- long word = 0;
+ @Override
+ public long totalTermFreq() throws IOException {
+ return -1L;
+ }
- @Override
- public boolean get(int index) {
- try {
- int i = index >>> 6;
- if (wordIndex != i) {
- wordIndex = i;
- word = in.readLong(i << 3);
- }
- return (word & (1L << index)) != 0;
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
+ @Override
+ public PostingsEnum postings(PostingsEnum reuse, int flags) throws IOException {
+ throw new UnsupportedOperationException();
+ }
- @Override
- public int length() {
- return count;
- }
- };
+ @Override
+ public int docFreq() throws IOException {
+ throw new UnsupportedOperationException();
}
}
- private DocIdSetIterator getDocsWithField(final long offset, final int count) throws IOException {
- if (offset == ALL_MISSING) {
- return DocIdSetIterator.empty();
- } else if (offset == ALL_LIVE) {
- return DocIdSetIterator.all(count);
- } else {
- int length = (int) ((count + 63L) >>> 6);
- final RandomAccessInput in = data.randomAccessSlice(offset, length << 3);
- return new DocIdSetIterator() {
+ @Override
+ public SortedNumericDocValues getSortedNumeric(FieldInfo field) throws IOException {
+ SortedNumericEntry entry = sortedNumerics.get(field.name);
+ if (entry.numValues == entry.numDocsWithField) {
+ return DocValues.singleton(getNumeric(entry));
+ }
- int doc = -1;
- int wordIndex = -1;
- long word = 0;
+ final RandomAccessInput addressesInput = data.randomAccessSlice(entry.addressesOffset, entry.addressesLength);
+ final LongValues addresses = DirectMonotonicReader.getInstance(entry.addressesMeta, addressesInput);
- private int nextSetBit(int index) throws IOException {
- int i = index >>> 6;
+ final LongValues values = getNumericValues(entry);
- if (wordIndex != i) {
- wordIndex = i;
- word = in.readLong(i << 3);
- }
- long w = word >>> index;
- if (w != 0) {
- return index + Long.numberOfTrailingZeros(w);
- }
-
- while (++i < length) {
- wordIndex = i;
- word = in.readLong(i << 3);
- if (word != 0) {
- return (i << 6) + Long.numberOfTrailingZeros(word);
- }
- }
+ if (entry.docsWithFieldOffset == -1) {
+ // dense
+ return new SortedNumericDocValues() {
- return DocIdSetIterator.NO_MORE_DOCS;
- }
+ int doc = -1;
+ long start, end;
+ int count;
@Override
public int nextDoc() throws IOException {
- return advance(docID() + 1);
+ return advance(doc + 1);
}
@Override
@@ -1422,395 +954,196 @@ final class Lucene70DocValuesProducer extends DocValuesProducer implements Close
@Override
public long cost() {
- // TODO: what is the exact cardinality
- return count;
+ return maxDoc;
}
@Override
public int advance(int target) throws IOException {
- if (target >= count) {
+ if (target >= maxDoc) {
return doc = NO_MORE_DOCS;
}
- return doc = nextSetBit(target);
+ start = addresses.get(target);
+ end = addresses.get(target + 1L);
+ count = (int) (end - start);
+ return doc = target;
}
- };
- }
- }
-
- private SparseNumericDocValues getSparseNumericDocValues(NumericEntry entry) throws IOException {
- final RandomAccessInput docIdsData = this.data.randomAccessSlice(entry.missingOffset, entry.offset - entry.missingOffset);
- final LongValues docIDs = DirectMonotonicReader.getInstance(entry.monotonicMeta, docIdsData);
- final LongValues values = getNumeric(entry.nonMissingValues); // cannot be sparse
- return new SparseNumericDocValues(Math.toIntExact(entry.numDocsWithValue), docIDs, values);
- }
-
- @Override
- public synchronized DocValuesProducer getMergeInstance() throws IOException {
- return new Lucene70DocValuesProducer(this);
- }
-
- @Override
- public void close() throws IOException {
- data.close();
- }
-
- /** metadata entry for a numeric docvalues field */
- static class NumericEntry {
- private NumericEntry() {}
- /** offset to the bitset representing docsWithField, or -1 if no documents have missing values */
- long missingOffset;
- /** offset to the actual numeric values */
- public long offset;
- /** end offset to the actual numeric values */
- public long endOffset;
- /** bits per value used to pack the numeric values */
- public int bitsPerValue;
-
- int format;
- /** count of values written */
- public long count;
-
- /** monotonic meta */
- public DirectMonotonicReader.Meta monotonicMeta;
-
- long minValue;
- long gcd;
- long table[];
- /** for sparse compression */
- long numDocsWithValue;
- NumericEntry nonMissingValues;
- NumberType numberType;
+ @Override
+ public long nextValue() throws IOException {
+ return values.get(start++);
+ }
- }
+ @Override
+ public int docValueCount() {
+ return count;
+ }
+ };
+ } else {
+ // sparse
+ final IndexedDISI disi = new IndexedDISI(data, entry.docsWithFieldOffset, entry.docsWithFieldLength, entry.numDocsWithField);
+ return new SortedNumericDocValues() {
- /** metadata entry for a binary docvalues field */
- static class BinaryEntry {
- private BinaryEntry() {}
- /** offset to the bitset representing docsWithField, or -1 if no documents have missing values */
- long missingOffset;
- /** offset to the actual binary values */
- long offset;
-
- int format;
- /** count of values written */
- public long count;
- int minLength;
- int maxLength;
- /** offset to the addressing data that maps a value to its slice of the byte[] */
- public long addressesOffset, addressesEndOffset;
- /** meta data for addresses */
- public DirectMonotonicReader.Meta addressesMeta;
- /** offset to the reverse index */
- public long reverseIndexOffset;
- /** packed ints version used to encode addressing information */
- public int packedIntsVersion;
- /** packed ints blocksize */
- public int blockSize;
- }
+ boolean set;
+ long start, end;
+ int count;
- /** metadata entry for a sorted-set docvalues field */
- static class SortedSetEntry {
- private SortedSetEntry() {}
- int format;
+ @Override
+ public int nextDoc() throws IOException {
+ set = false;
+ return disi.nextDoc();
+ }
- long[] table;
- int[] tableOffsets;
- }
+ @Override
+ public int docID() {
+ return disi.docID();
+ }
- // internally we compose complex dv (sorted/sortedset) from other ones
- static abstract class LongBinaryDocValues extends LegacyBinaryDocValues {
- @Override
- public final BytesRef get(int docID) {
- return get((long)docID);
- }
+ @Override
+ public long cost() {
+ return disi.cost();
+ }
- abstract BytesRef get(long id);
- }
+ @Override
+ public int advance(int target) throws IOException {
+ set = false;
+ return disi.advance(target);
+ }
- // used for reverse lookup to a small range of blocks
- static class ReverseTermsIndex implements Accountable {
- public MonotonicBlockPackedReader termAddresses;
- public PagedBytes.Reader terms;
+ @Override
+ public long nextValue() throws IOException {
+ set();
+ return values.get(start++);
+ }
- @Override
- public long ramBytesUsed() {
- return termAddresses.ramBytesUsed() + terms.ramBytesUsed();
- }
+ @Override
+ public int docValueCount() {
+ set();
+ return count;
+ }
- @Override
- public Collection<Accountable> getChildResources() {
- List<Accountable> resources = new ArrayList<>();
- resources.add(Accountables.namedAccountable("term bytes", terms));
- resources.add(Accountables.namedAccountable("term addresses", termAddresses));
- return Collections.unmodifiableList(resources);
- }
+ private void set() {
+ if (set == false) {
+ final int index = disi.index();
+ start = addresses.get(index);
+ end = addresses.get(index + 1L);
+ count = (int) (end - start);
+ set = true;
+ }
+ }
- @Override
- public String toString() {
- return getClass().getSimpleName() + "(size=" + termAddresses.size() + ")";
+ };
}
}
- //in the compressed case, we add a few additional operations for
- //more efficient reverse lookup and enumeration
- static final class CompressedBinaryDocValues extends LongBinaryDocValues {
- final long numValues;
- final long numIndexValues;
- final int maxTermLength;
- final MonotonicBlockPackedReader addresses;
- final IndexInput data;
- final CompressedBinaryTermsEnum termsEnum;
- final PagedBytes.Reader reverseTerms;
- final MonotonicBlockPackedReader reverseAddresses;
- final long numReverseIndexValues;
-
- public CompressedBinaryDocValues(BinaryEntry bytes, MonotonicBlockPackedReader addresses, ReverseTermsIndex index, IndexInput data) throws IOException {
- this.maxTermLength = bytes.maxLength;
- this.numValues = bytes.count;
- this.addresses = addresses;
- this.numIndexValues = addresses.size();
- this.data = data;
- this.reverseTerms = index.terms;
- this.reverseAddresses = index.termAddresses;
- this.numReverseIndexValues = reverseAddresses.size();
- this.termsEnum = getTermsEnum(data);
- }
-
- @Override
- public BytesRef get(long id) {
- try {
- termsEnum.seekExact(id);
- return termsEnum.term();
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
+ @Override
+ public SortedSetDocValues getSortedSet(FieldInfo field) throws IOException {
+ SortedSetEntry entry = sortedSets.get(field.name);
+ if (entry.singleValueEntry != null) {
+ return DocValues.singleton(getSorted(entry.singleValueEntry));
}
- long lookupTerm(BytesRef key) {
- try {
- switch (termsEnum.seekCeil(key)) {
- case FOUND: return termsEnum.ord();
- case NOT_FOUND: return -termsEnum.ord()-1;
- default: return -numValues-1;
- }
- } catch (IOException bogus) {
- throw new RuntimeException(bogus);
- }
- }
+ final RandomAccessInput slice = data.randomAccessSlice(entry.ordsOffset, entry.ordsLength);
+ final LongValues ords = DirectReader.getInstance(slice, entry.bitsPerValue);
- TermsEnum getTermsEnum() {
- try {
- return getTermsEnum(data.clone());
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
+ final RandomAccessInput addressesInput = data.randomAccessSlice(entry.addressesOffset, entry.addressesLength);
+ final LongValues addresses = DirectMonotonicReader.getInstance(entry.addressesMeta, addressesInput);
- private CompressedBinaryTermsEnum getTermsEnum(IndexInput input) throws IOException {
- return new CompressedBinaryTermsEnum(input);
- }
+ if (entry.docsWithFieldOffset == -1) {
+ // dense
+ return new BaseSortedSetDocValues(entry, data) {
- class CompressedBinaryTermsEnum extends TermsEnum {
- private long currentOrd = -1;
- // offset to the start of the current block
- private long currentBlockStart;
- private final IndexInput input;
- // delta from currentBlockStart to start of each term
- private final int offsets[] = new int[INTERVAL_COUNT];
- private final byte buffer[] = new byte[2*INTERVAL_COUNT-1];
-
- private final BytesRef term = new BytesRef(maxTermLength);
- private final BytesRef firstTerm = new BytesRef(maxTermLength);
- private final BytesRef scratch = new BytesRef();
-
- CompressedBinaryTermsEnum(IndexInput input) throws IOException {
- this.input = input;
- input.seek(0);
- }
+ int doc = -1;
+ long start;
+ long end;
- private void readHeader() throws IOException {
- firstTerm.length = input.readVInt();
- input.readBytes(firstTerm.bytes, 0, firstTerm.length);
- input.readBytes(buffer, 0, INTERVAL_COUNT-1);
- if (buffer[0] == -1) {
- readShortAddresses();
- } else {
- readByteAddresses();
+ @Override
+ public int nextDoc() throws IOException {
+ return advance(doc + 1);
}
- currentBlockStart = input.getFilePointer();
- }
- // read single byte addresses: each is delta - 2
- // (shared prefix byte and length > 0 are both implicit)
- private void readByteAddresses() throws IOException {
- int addr = 0;
- for (int i = 1; i < offsets.length; i++) {
- addr += 2 + (buffer[i-1] & 0xFF);
- offsets[i] = addr;
- }
- }
-
- // read double byte addresses: each is delta - 2
- // (shared prefix byte and length > 0 are both implicit)
- private void readShortAddresses() throws IOException {
- input.readBytes(buffer, INTERVAL_COUNT-1, INTERVAL_COUNT);
- int addr = 0;
- for (int i = 1; i < offsets.length; i++) {
- int x = i<<1;
- addr += 2 + ((buffer[x-1] << 8) | (buffer[x] & 0xFF));
- offsets[i] = addr;
+ @Override
+ public int docID() {
+ return doc;
}
- }
-
- // set term to the first term
- private void readFirstTerm() throws IOException {
- term.length = firstTerm.length;
- System.arraycopy(firstTerm.bytes, firstTerm.offset, term.bytes, 0, term.length);
- }
-
- // read term at offset, delta encoded from first term
- private void readTerm(int offset) throws IOException {
- int start = input.readByte() & 0xFF;
- System.arraycopy(firstTerm.bytes, firstTerm.offset, term.bytes, 0, start);
- int suffix = offsets[offset] - offsets[offset-1] - 1;
- input.readBytes(term.bytes, start, suffix);
- term.length = start + suffix;
- }
- @Override
- public BytesRef next() throws IOException {
- currentOrd++;
- if (currentOrd >= numValues) {
- return null;
- } else {
- int offset = (int) (currentOrd & INTERVAL_MASK);
- if (offset == 0) {
- // switch to next block
- readHeader();
- readFirstTerm();
- } else {
- readTerm(offset);
- }
- return term;
+ @Override
+ public long cost() {
+ return maxDoc;
}
- }
- // binary search reverse index to find smaller
- // range of blocks to search
- long binarySearchIndex(BytesRef text) throws IOException {
- long low = 0;
- long high = numReverseIndexValues - 1;
- while (low <= high) {
- long mid = (low + high) >>> 1;
- reverseTerms.fill(scratch, reverseAddresses.get(mid));
- int cmp = scratch.compareTo(text);
-
- if (cmp < 0) {
- low = mid + 1;
- } else if (cmp > 0) {
- high = mid - 1;
- } else {
- return mid;
+ @Override
+ public int advance(int target) throws IOException {
+ if (target >= maxDoc) {
+ return doc = NO_MORE_DOCS;
}
+ start = addresses.get(target);
+ end = addresses.get(target + 1L);
+ return doc = target;
}
- return high;
- }
- // binary search against first term in block range
- // to find term's block
- long binarySearchBlock(BytesRef text, long low, long high) throws IOException {
- while (low <= high) {
- long mid = (low + high) >>> 1;
- input.seek(addresses.get(mid));
- term.length = input.readVInt();
- input.readBytes(term.bytes, 0, term.length);
- int cmp = term.compareTo(text);
-
- if (cmp < 0) {
- low = mid + 1;
- } else if (cmp > 0) {
- high = mid - 1;
- } else {
- return mid;
+ @Override
+ public long nextOrd() throws IOException {
+ if (start == end) {
+ return NO_MORE_ORDS;
}
+ return ords.get(start++);
}
- return high;
- }
- @Override
- public SeekStatus seekCeil(BytesRef text) throws IOException {
- // locate block: narrow to block range with index, then search blocks
- final long block;
- long indexPos = binarySearchIndex(text);
- if (indexPos < 0) {
- block = 0;
- } else {
- long low = indexPos << BLOCK_INTERVAL_SHIFT;
- long high = Math.min(numIndexValues - 1, low + BLOCK_INTERVAL_MASK);
- block = Math.max(low, binarySearchBlock(text, low, high));
- }
+ };
+ } else {
+ // sparse
+ final IndexedDISI disi = new IndexedDISI(data, entry.docsWithFieldOffset, entry.docsWithFieldLength, entry.numDocsWithField);
+ return new BaseSortedSetDocValues(entry, data) {
- // position before block, then scan to term.
- input.seek(addresses.get(block));
- currentOrd = (block << INTERVAL_SHIFT) - 1;
+ boolean set;
+ long start;
+ long end = 0;
- while (next() != null) {
- int cmp = term.compareTo(text);
- if (cmp == 0) {
- return SeekStatus.FOUND;
- } else if (cmp > 0) {
- return SeekStatus.NOT_FOUND;
- }
+ @Override
+ public int nextDoc() throws IOException {
+ set = false;
+ return disi.nextDoc();
}
- return SeekStatus.END;
- }
- @Override
- public void seekExact(long ord) throws IOException {
- long block = ord >>> INTERVAL_SHIFT;
- if (block != currentOrd >>> INTERVAL_SHIFT) {
- // switch to different block
- input.seek(addresses.get(block));
- readHeader();
+ @Override
+ public int docID() {
+ return disi.docID();
}
- currentOrd = ord;
-
- int offset = (int) (ord & INTERVAL_MASK);
- if (offset == 0) {
- readFirstTerm();
- } else {
- input.seek(currentBlockStart + offsets[offset-1]);
- readTerm(offset);
+ @Override
+ public long cost() {
+ return disi.cost();
}
- }
-
- @Override
- public BytesRef term() throws IOException {
- return term;
- }
-
- @Over
<TRUNCATED>
[46/50] [abbrv] lucene-solr:jira/solr-8593: LUCENE-7496: Better
toString for SweetSpotSimilarity
Posted by kr...@apache.org.
LUCENE-7496: Better toString for SweetSpotSimilarity
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/c4b4830a
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/c4b4830a
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/c4b4830a
Branch: refs/heads/jira/solr-8593
Commit: c4b4830ac1c984e54e23c374ec7b83e598c7fc4b
Parents: 14b6d93
Author: Jan H�ydahl <ja...@apache.org>
Authored: Thu Oct 20 14:07:11 2016 +0200
Committer: Jan H�ydahl <ja...@apache.org>
Committed: Thu Oct 20 14:07:11 2016 +0200
----------------------------------------------------------------------
lucene/CHANGES.txt | 2 ++
.../org/apache/lucene/misc/SweetSpotSimilarity.java | 16 ++++++++++++++++
2 files changed, 18 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c4b4830a/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 17e0b49..2bd4c28 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -109,6 +109,8 @@ Improvements
* LUCENE-7439: FuzzyQuery now matches all terms within the specified
edit distance, even if they are short terms (Mike McCandless)
+* LUCENE-7496: Better toString for SweetSpotSimilarity (janhoy)
+
Optimizations
* LUCENE-7501: BKDReader should not store the split dimension explicitly in the
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c4b4830a/lucene/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java
----------------------------------------------------------------------
diff --git a/lucene/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java b/lucene/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java
index ce26080..7eeeae0 100644
--- a/lucene/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java
+++ b/lucene/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java
@@ -223,4 +223,20 @@ public class SweetSpotSimilarity extends ClassicSimilarity {
}
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ sb.append("SweetSpotSimilarity")
+ .append("(")
+ .append("ln_min="+ln_min+", ")
+ .append("ln_max=").append(ln_max).append(", ")
+ .append("ln_steep=").append(ln_steep).append(", ")
+ .append("tf_base=").append(tf_base).append(", ")
+ .append("tf_min=").append(tf_min).append(", ")
+ .append("tf_hyper_min=").append(tf_hyper_min).append(", ")
+ .append("tf_hyper_max=").append(tf_hyper_max).append(", ")
+ .append("tf_hyper_base=").append(tf_hyper_base).append(", ")
+ .append("tf_hyper_xoffset=").append(tf_hyper_xoffset)
+ .append(")");
+ return sb.toString();
+ }
}