You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by gu...@apache.org on 2019/03/04 17:31:59 UTC
[lucene-solr] branch solr-13131 updated (8bff5b4 -> 5bda03c)
This is an automated email from the ASF dual-hosted git repository.
gus pushed a change to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git.
from 8bff5b4 Merge pull request #583 from moshebla/SOLR-13151
add 55b4d2d LUCENE-8697: GraphTokenStreamFiniteStrings correctly handles side paths with gaps
add 6f61bde LUCENE-8685: Refactor LatLonShape tests
add ec801b4 LUCENE-8635: add option to move FSTs off-heap, and do so for the FST terms index in the default codec for non-primary-key fields if MMapDirectory is being used
add af3ff11 SOLR-11763: Upgrade Guava to 25.1-jre (Markus Jelsma, Kevin Risden)
add 3ef6e01 LUCENE-8699: Change HeapPointWriter to use a single byte array instead to a list of byte arrays. In addition a new interface PointValue is added to abstract out the different formats between offline and on-heap writers.
add a89b065 SOLR-13255 : ClasscastException when URPs try to read a String field which returns a ByteArrayUTF8CHarSequence . This is a regression in release 7.7
add 86fe24c SOLR-13222: Improve logging in StreamingSolrClients (Peter Cseh via Kevin Risden)
add 9251d45 Fix typo in 'Index Size Trigger' example JSON.
add bcd90f1 SOLR-12708: CREATE collection request doesn't fail or cleanup when the request fails (#568)
add 6e81def Fix TestPullReplica.testKillLeader
add 4625270 LUCENE-8686: Fix test bug in TestTaxonomyFacetSumValueSource#testRandom
add f5a4159 LUCENE-8704: Use the HeapPointWriter to spill offline as BKDWriter does
add 3754959 OSQ-8696: Come up with a simple test case that exercises the random failure
add 5ab5ba7 Minor cleanup to ExportWriter
add 906a088 Fix TestTlogReplica
add 9b8a4a9 LUCENE-8699: Add lucene internal tag to PointValue interface and fix some typos
add 6b4e906 SOLR-13261: Make SortableTextField work with export/streaming
add 0de3905 SOLR-12055: Enable async logging by default SOLR-12753: Async logging ring buffer and OOM error
add 5209762 SOLR-11883: reporting syntax errors as Bad Request instead of 500.
add fe5a96a SOLR-13268: Clean up any test failures resulting from SOLR-12055 (async logging). Kevin's upgrades
add 9e7ec87 SOLR-9763: Remove the workaround implemented for HADOOP-12767 (Kevin Risden)
add c599ad9 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
add 9bf2c8c LUCENE-8696: Update test to be what's actually failing
add edc73ee LUCENE-8702: Simplify some Intervals factory methods
add 303d119 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
add 8c34da8 LUCENE-8696: Fix precommit objections
new d87e0c2 Merge branch 'master' of https://gitbox.apache.org/repos/asf/lucene-solr into solr-13131
add 149469d LUCENE-8696: Refactor, in preparation for creating a new SegmentEndpoint implementation to fix the problem.
add 6a886b2 SOLR-13060: Improve HdfsAutoAddReplicasIntegrationTest and HdfsCollectionsAPIDistributedZkTest
add 9a6f942 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
add ff799ac LUCENE-8696: Rework how endpoint circles are represented to allow for consistency on WGS84.
add 9753e00 SOLR-13074: MoveReplicaHDFSTest leaks threads, falls into an endless loop, logging like crazy (Kevin Risden)
add 8f78c05 SOLR-9762: Remove the workaround implemented for HADOOP-13346 (Kevin Risden)
add 58beaeb SOLR-13227: don't throw exceptions for counting facet.range.other.
add 1ad45a7 fix typo in javadocs
add 78a57f7 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
add 00c0229 SOLR-12297: Remove debugging System.out line
add 15f3c3b SOLR-7321: Remove reflection in FSHDFSUtils.java (Mike Drob, Kevin Risden)
add 5b2f064 LUCENE-8709: Handle case of creating a HeapPointWriter with size equal 0
add dcb022b SOLR-10436: Add hashRollup Streaming Expression
add f9b3004 SOLR-10436: Update CHANGES.txt
add 02f1773 Lucene and Solr 7.7.1 release, DOAP changes
add 5e28e94 fix AfterClass in OverseerTest to not fail with NPE if assumeWorkingMockito() call short circuited the BeforeClass
add 3ae52e1 SOLR-13282: JDBCStreamTest should use assertEquals(double, double, delta) instead of '==' to compare 2 double values
add 15dd4a1 UCENE-8703: Build point writers in the BKD tree only when they are needed
add d381b73 LUCENE-8709: Set pointWriter to null when size of the HeapPointWriter is 0
add 7ad0ac0 LUCENE-8714: Don't use NoMergePolicy in norms tests
add fed8059 SOLR-12923: increase all await() times in TestSimTriggerIntegration
new fc0b9fa Merge branch 'master' of https://gitbox.apache.org/repos/asf/lucene-solr into solr-13131
add 4b482d6 SOLR-13152: make maxCardinality part of createCRACmd constructor params and validate CRA params during construction of alias
new d6704e5 Merge branch 'SOLR-13152' of https://github.com/moshebla/lucene-solr into moshebla-SOLR-13152
new 6034261 SOLR-13152 Check for missing route value
new 6a93ee4 SOLR-13152 Add check for values that resolve to the __CRA__ collection name infix which is reserved
new 64d1601 SOLR-13151 fix mustMatch to match on value not the final collection name so that users can choose to fail on non-english text if desired.
new 46490b3 SOLR-13152 v2 api support
new d7cb6d6 SOLR-13152 precommit fix
new da21eae Merge branch 'moshebla-SOLR-13152' into solr-13131
new 5bda03c SOLR-13152 documentation
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
dev-tools/doap/lucene.rdf | 7 +
dev-tools/doap/solr.rdf | 7 +
lucene/CHANGES.txt | 21 +
.../codecs/simpletext/SimpleTextBKDWriter.java | 154 ++----
.../codecs/blocktree/BlockTreeTermsReader.java | 9 +-
.../lucene/codecs/blocktree/FieldReader.java | 11 +-
.../java/org/apache/lucene/search/TotalHits.java | 2 +-
.../apache/lucene/store/ByteBufferIndexInput.java | 2 +-
.../apache/lucene/util/bkd/BKDRadixSelector.java | 336 ++++++++----
.../java/org/apache/lucene/util/bkd/BKDWriter.java | 136 ++---
.../apache/lucene/util/bkd/HeapPointReader.java | 73 ++-
.../apache/lucene/util/bkd/HeapPointWriter.java | 127 ++---
.../apache/lucene/util/bkd/OfflinePointReader.java | 66 ++-
.../apache/lucene/util/bkd/OfflinePointWriter.java | 22 +-
.../org/apache/lucene/util/bkd/PointReader.java | 11 +-
.../util/bkd/{PointReader.java => PointValue.java} | 26 +-
.../org/apache/lucene/util/bkd/PointWriter.java | 9 +-
.../src/java/org/apache/lucene/util/fst/FST.java | 41 +-
.../java/org/apache/lucene/util/fst/FSTStore.java | 25 +-
.../apache/lucene/util/fst/OffHeapFSTStore.java | 69 +++
.../org/apache/lucene/util/fst/OnHeapFSTStore.java | 97 ++++
.../lucene/util/fst/ReverseRandomAccessReader.java | 63 +++
.../util/graph/GraphTokenStreamFiniteStrings.java | 17 +-
.../test/org/apache/lucene/util/bkd/TestBKD.java | 12 +-
.../lucene/util/bkd/TestBKDRadixSelector.java | 76 +--
.../graph/TestGraphTokenStreamFiniteStrings.java | 48 +-
.../taxonomy/TestTaxonomyFacetSumValueSource.java | 6 +-
lucene/ivy-versions.properties | 16 +-
.../apache/lucene/search/intervals/Intervals.java | 12 +
.../apache/lucene/document/TestLatLonShape.java | 532 ------------------
...tLonShape.java => TestLatLonShapeEncoding.java} | 381 +------------
.../search/intervals/TestSimplifications.java | 49 ++
.../lucene/spatial3d/geom/GeoStandardPath.java | 595 +++++++++++++--------
.../apache/lucene/spatial3d/geom/GeoPathTest.java | 40 +-
.../lucene/index/BaseNormsFormatTestCase.java | 2 +-
solr/CHANGES.txt | 49 ++
...geIdentifierUpdateProcessorFactoryTestCase.java | 26 +-
solr/core/ivy.xml | 1 +
.../cloud/api/collections/CategoryRoutedAlias.java | 52 +-
.../cloud/api/collections/CreateCollectionCmd.java | 2 +-
.../OverseerCollectionMessageHandler.java | 56 +-
.../solr/cloud/api/collections/RestoreCmd.java | 41 +-
.../cloud/api/collections/TimeRoutedAlias.java | 4 +-
.../solr/handler/admin/CoreAdminHandler.java | 3 +-
.../solr/handler/component/HighlightComponent.java | 4 +-
.../handler/component/RangeFacetProcessor.java | 48 +-
.../solr/handler/component/StatsValuesFactory.java | 3 +-
.../apache/solr/handler/export/ExportWriter.java | 138 +++--
.../org/apache/solr/search/FunctionQParser.java | 60 ++-
.../security/DelegationTokenKerberosFilter.java | 15 +-
.../org/apache/solr/security/HadoopAuthFilter.java | 17 +-
.../org/apache/solr/security/HadoopAuthPlugin.java | 24 +-
.../org/apache/solr/security/KerberosPlugin.java | 32 +-
.../org/apache/solr/update/DocumentBuilder.java | 6 +-
.../apache/solr/update/StreamingSolrClients.java | 2 +-
.../src/java/org/apache/solr/util/FSHDFSUtils.java | 37 +-
.../org/apache/solr/util/StartupLoggingUtils.java | 18 +
solr/core/src/test-files/log4j2.xml | 45 +-
.../collection1/conf/schema-sortingresponse.xml | 25 +-
.../configsets/cloud-managed/conf/managed-schema | 12 +
.../cloud/AsyncCallRequestStatusResponseTest.java | 26 +-
.../solr/cloud/CreateCollectionCleanupTest.java | 37 +-
.../apache/solr/cloud/DocValuesNotIndexedTest.java | 44 +-
.../solr/cloud/MoveReplicaHDFSFailoverTest.java | 3 +-
.../org/apache/solr/cloud/MoveReplicaHDFSTest.java | 60 +--
.../org/apache/solr/cloud/MoveReplicaTest.java | 79 +--
.../test/org/apache/solr/cloud/OverseerTest.java | 9 +-
.../org/apache/solr/cloud/TestPullReplica.java | 2 +-
.../org/apache/solr/cloud/TestTlogReplica.java | 27 +-
.../AbstractCloudBackupRestoreTestCase.java | 73 ++-
.../CollectionsAPIDistributedZkTest.java | 91 +---
.../HdfsCollectionsAPIDistributedZkTest.java | 138 +----
.../collections/TestHdfsCloudBackupRestore.java | 6 +-
.../collections/TestLocalFSCloudBackupRestore.java | 6 +-
.../AutoAddReplicasIntegrationTest.java | 16 +-
.../HdfsAutoAddReplicasIntegrationTest.java | 38 +-
.../autoscaling/sim/TestSimTriggerIntegration.java | 173 +++---
.../solr/cloud/hdfs/HDFSCollectionsAPITest.java | 5 +-
.../apache/solr/cloud/rule/ImplicitSnitchTest.java | 3 +-
.../apache/solr/handler/RequestLoggingTest.java | 29 +-
.../TestDistributedStatsComponentCardinality.java | 3 +-
.../solr/handler/export/TestExportWriter.java | 32 +-
.../org/apache/solr/logging/TestLogWatcher.java | 82 ++-
.../test/org/apache/solr/schema/UUIDFieldTest.java | 4 +-
.../search/SignificantTermsQParserPluginTest.java | 4 +-
.../solr/search/function/TestFunctionQuery.java | 47 ++
.../CategoryRoutedAliasUpdateProcessorTest.java | 73 ++-
.../apache/solr/util/TestObjectReleaseTracker.java | 3 +-
.../org/apache/solr/util/TestSystemIdResolver.java | 3 +-
solr/licenses/avatica-core-1.10.0.jar.sha1 | 1 -
solr/licenses/avatica-core-1.13.0.jar.sha1 | 1 +
solr/licenses/calcite-core-1.13.0.jar.sha1 | 1 -
solr/licenses/calcite-core-1.18.0.jar.sha1 | 1 +
solr/licenses/calcite-linq4j-1.13.0.jar.sha1 | 1 -
solr/licenses/calcite-linq4j-1.18.0.jar.sha1 | 1 +
solr/licenses/commons-compiler-2.7.6.jar.sha1 | 1 -
solr/licenses/commons-compiler-3.0.9.jar.sha1 | 1 +
solr/licenses/curator-client-2.13.0.jar.sha1 | 1 +
solr/licenses/curator-client-2.8.0.jar.sha1 | 1 -
solr/licenses/curator-framework-2.13.0.jar.sha1 | 1 +
solr/licenses/curator-framework-2.8.0.jar.sha1 | 1 -
solr/licenses/curator-recipes-2.13.0.jar.sha1 | 1 +
solr/licenses/curator-recipes-2.8.0.jar.sha1 | 1 -
solr/licenses/disruptor-3.4.0.jar.sha1 | 1 -
solr/licenses/disruptor-3.4.2.jar.sha1 | 1 +
solr/licenses/guava-14.0.1.jar.sha1 | 1 -
solr/licenses/guava-25.1-jre.jar.sha1 | 1 +
solr/licenses/janino-2.7.6.jar.sha1 | 1 -
solr/licenses/janino-3.0.9.jar.sha1 | 1 +
solr/licenses/json-path-2.4.0.jar.sha1 | 1 +
solr/licenses/json-path-LICENSE-ASL.txt | 202 +++++++
.../json-path-NOTICE.txt} | 0
solr/licenses/log4j-1.2-api-2.11.0.jar.sha1 | 1 -
solr/licenses/log4j-1.2-api-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-api-2.11.0.jar.sha1 | 1 -
solr/licenses/log4j-api-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-core-2.11.0.jar.sha1 | 1 -
solr/licenses/log4j-core-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-slf4j-impl-2.11.0.jar.sha1 | 1 -
solr/licenses/log4j-slf4j-impl-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-web-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-web-LICENSE-ASL.txt | 202 +++++++
solr/licenses/log4j-web-NOTICE.txt | 17 +
solr/server/ivy.xml | 1 +
solr/server/resources/log4j2-console.xml | 38 +-
solr/server/resources/log4j2.xml | 68 ++-
solr/solr-ref-guide/src/aliases.adoc | 267 +++++++++
solr/solr-ref-guide/src/collections-api.adoc | 60 ++-
solr/solr-ref-guide/src/distributed-requests.adoc | 1 +
solr/solr-ref-guide/src/how-solrcloud-works.adoc | 4 +-
solr/solr-ref-guide/src/solr-upgrade-notes.adoc | 10 +
.../src/solrcloud-autoscaling-triggers.adoc | 2 +-
solr/solr-ref-guide/src/time-routed-aliases.adoc | 119 -----
.../solr/client/solrj/impl/Http2SolrClient.java | 1 -
.../java/org/apache/solr/client/solrj/io/Lang.java | 1 +
.../client/solrj/io/stream/HashRollupStream.java | 256 +++++++++
.../client/solrj/io/stream/metrics/MaxMetric.java | 16 +-
.../client/solrj/io/stream/metrics/MeanMetric.java | 10 +-
.../client/solrj/io/stream/metrics/MinMetric.java | 16 +-
.../client/solrj/io/stream/metrics/SumMetric.java | 8 +-
.../solrj/request/CollectionAdminRequest.java | 17 +-
.../org/apache/solr/common/SolrInputField.java | 61 ++-
.../common/util/ByteArrayUtf8CharSequence.java | 5 +
.../org/apache/solr/common/util/JavaBinCodec.java | 2 +-
.../resources/apispec/collections.Commands.json | 8 +
.../org/apache/solr/client/solrj/io/TestLang.java | 2 +-
.../client/solrj/io/stream/JDBCStreamTest.java | 22 +-
.../solrj/io/stream/StreamDecoratorTest.java | 274 ++++++++++
.../src/java/org/apache/solr/SolrTestCaseJ4.java | 2 +
.../solr/cloud/AbstractFullDistribZkTestBase.java | 2 +
150 files changed, 3911 insertions(+), 2749 deletions(-)
copy lucene/core/src/java/org/apache/lucene/util/bkd/{PointReader.java => PointValue.java} (60%)
copy solr/core/src/test/org/apache/solr/search/SignificantTermsQParserPluginTest.java => lucene/core/src/java/org/apache/lucene/util/fst/FSTStore.java (56%)
create mode 100644 lucene/core/src/java/org/apache/lucene/util/fst/OffHeapFSTStore.java
create mode 100644 lucene/core/src/java/org/apache/lucene/util/fst/OnHeapFSTStore.java
create mode 100644 lucene/core/src/java/org/apache/lucene/util/fst/ReverseRandomAccessReader.java
copy lucene/sandbox/src/test/org/apache/lucene/document/{TestLatLonShape.java => TestLatLonShapeEncoding.java} (58%)
create mode 100644 lucene/sandbox/src/test/org/apache/lucene/search/intervals/TestSimplifications.java
delete mode 100644 solr/licenses/avatica-core-1.10.0.jar.sha1
create mode 100644 solr/licenses/avatica-core-1.13.0.jar.sha1
delete mode 100644 solr/licenses/calcite-core-1.13.0.jar.sha1
create mode 100644 solr/licenses/calcite-core-1.18.0.jar.sha1
delete mode 100644 solr/licenses/calcite-linq4j-1.13.0.jar.sha1
create mode 100644 solr/licenses/calcite-linq4j-1.18.0.jar.sha1
delete mode 100644 solr/licenses/commons-compiler-2.7.6.jar.sha1
create mode 100644 solr/licenses/commons-compiler-3.0.9.jar.sha1
create mode 100644 solr/licenses/curator-client-2.13.0.jar.sha1
delete mode 100644 solr/licenses/curator-client-2.8.0.jar.sha1
create mode 100644 solr/licenses/curator-framework-2.13.0.jar.sha1
delete mode 100644 solr/licenses/curator-framework-2.8.0.jar.sha1
create mode 100644 solr/licenses/curator-recipes-2.13.0.jar.sha1
delete mode 100644 solr/licenses/curator-recipes-2.8.0.jar.sha1
delete mode 100644 solr/licenses/disruptor-3.4.0.jar.sha1
create mode 100644 solr/licenses/disruptor-3.4.2.jar.sha1
delete mode 100644 solr/licenses/guava-14.0.1.jar.sha1
create mode 100644 solr/licenses/guava-25.1-jre.jar.sha1
delete mode 100644 solr/licenses/janino-2.7.6.jar.sha1
create mode 100644 solr/licenses/janino-3.0.9.jar.sha1
create mode 100644 solr/licenses/json-path-2.4.0.jar.sha1
create mode 100644 solr/licenses/json-path-LICENSE-ASL.txt
copy solr/{solrj/src/test-files/solrj/solr/multicore/core1/core.properties => licenses/json-path-NOTICE.txt} (100%)
delete mode 100644 solr/licenses/log4j-1.2-api-2.11.0.jar.sha1
create mode 100644 solr/licenses/log4j-1.2-api-2.11.2.jar.sha1
delete mode 100644 solr/licenses/log4j-api-2.11.0.jar.sha1
create mode 100644 solr/licenses/log4j-api-2.11.2.jar.sha1
delete mode 100644 solr/licenses/log4j-core-2.11.0.jar.sha1
create mode 100644 solr/licenses/log4j-core-2.11.2.jar.sha1
delete mode 100644 solr/licenses/log4j-slf4j-impl-2.11.0.jar.sha1
create mode 100644 solr/licenses/log4j-slf4j-impl-2.11.2.jar.sha1
create mode 100644 solr/licenses/log4j-web-2.11.2.jar.sha1
create mode 100644 solr/licenses/log4j-web-LICENSE-ASL.txt
create mode 100644 solr/licenses/log4j-web-NOTICE.txt
create mode 100644 solr/solr-ref-guide/src/aliases.adoc
delete mode 100644 solr/solr-ref-guide/src/time-routed-aliases.adoc
create mode 100644 solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/HashRollupStream.java
[lucene-solr] 02/10: Merge branch 'master' of
https://gitbox.apache.org/repos/asf/lucene-solr into solr-13131
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit fc0b9fafbfef57b9c5793accc2f4c8781ded2149
Merge: d87e0c2 fed8059
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sat Mar 2 12:08:48 2019 -0500
Merge branch 'master' of https://gitbox.apache.org/repos/asf/lucene-solr into solr-13131
dev-tools/doap/lucene.rdf | 7 +
dev-tools/doap/solr.rdf | 7 +
lucene/CHANGES.txt | 3 +
.../codecs/simpletext/SimpleTextBKDWriter.java | 93 ++--
.../java/org/apache/lucene/search/TotalHits.java | 2 +-
.../java/org/apache/lucene/util/bkd/BKDWriter.java | 78 ++-
.../apache/lucene/util/bkd/HeapPointReader.java | 7 +-
.../apache/lucene/util/bkd/HeapPointWriter.java | 13 +-
.../test/org/apache/lucene/util/bkd/TestBKD.java | 12 +-
.../lucene/spatial3d/geom/GeoStandardPath.java | 595 +++++++++++++--------
.../apache/lucene/spatial3d/geom/GeoPathTest.java | 19 +-
.../lucene/index/BaseNormsFormatTestCase.java | 2 +-
solr/CHANGES.txt | 12 +
.../handler/component/RangeFacetProcessor.java | 48 +-
.../org/apache/solr/security/HadoopAuthPlugin.java | 24 +-
.../org/apache/solr/security/KerberosPlugin.java | 32 +-
.../src/java/org/apache/solr/util/FSHDFSUtils.java | 37 +-
.../solr/cloud/MoveReplicaHDFSFailoverTest.java | 3 +-
.../org/apache/solr/cloud/MoveReplicaHDFSTest.java | 60 +--
.../org/apache/solr/cloud/MoveReplicaTest.java | 79 +--
.../test/org/apache/solr/cloud/OverseerTest.java | 9 +-
.../CollectionsAPIDistributedZkTest.java | 91 +---
.../HdfsCollectionsAPIDistributedZkTest.java | 138 +----
.../AutoAddReplicasIntegrationTest.java | 16 +-
.../HdfsAutoAddReplicasIntegrationTest.java | 38 +-
.../autoscaling/sim/TestSimTriggerIntegration.java | 173 +++---
.../solr/cloud/hdfs/HDFSCollectionsAPITest.java | 5 +-
.../solr/client/solrj/impl/Http2SolrClient.java | 1 -
.../java/org/apache/solr/client/solrj/io/Lang.java | 1 +
.../client/solrj/io/stream/HashRollupStream.java | 256 +++++++++
.../client/solrj/io/stream/metrics/MaxMetric.java | 16 +-
.../client/solrj/io/stream/metrics/MeanMetric.java | 10 +-
.../client/solrj/io/stream/metrics/MinMetric.java | 16 +-
.../client/solrj/io/stream/metrics/SumMetric.java | 8 +-
.../org/apache/solr/client/solrj/io/TestLang.java | 2 +-
.../client/solrj/io/stream/JDBCStreamTest.java | 22 +-
.../solrj/io/stream/StreamDecoratorTest.java | 274 ++++++++++
37 files changed, 1322 insertions(+), 887 deletions(-)
[lucene-solr] 05/10: SOLR-13152 Add check for values that resolve
to the __CRA__ collection name infix which is reserved
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit 6a93ee4bf37e7c9c68747224f44a8b97af955eba
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sun Mar 3 12:23:29 2019 -0500
SOLR-13152 Add check for values that resolve to the __CRA__ collection
name infix which is reserved
---
.../apache/solr/cloud/api/collections/CategoryRoutedAlias.java | 10 +++++++++-
.../processor/CategoryRoutedAliasUpdateProcessorTest.java | 4 +++-
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java b/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
index 0b0114d..5089eeb 100644
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
@@ -128,7 +128,7 @@ public class CategoryRoutedAlias implements RoutedAlias {
if (fieldValue == null) {
throw new SolrException(BAD_REQUEST,"Route value is null");
}
-
+
String dataValue = String.valueOf(fieldValue);
String candidateCollectionName = buildCollectionNameFromValue(dataValue);
@@ -138,6 +138,14 @@ public class CategoryRoutedAlias implements RoutedAlias {
return;
}
+ // this check will become very important for future work
+ int infix = candidateCollectionName.indexOf(COLLECTION_INFIX);
+ int valueStart = infix + COLLECTION_INFIX.length();
+ if (candidateCollectionName.substring(valueStart).contains(COLLECTION_INFIX)) {
+ throw new SolrException(BAD_REQUEST, "No portion of the route value may resolve to the 7 character sequence " +
+ "__CRA__");
+ }
+
if (mustMatch != null && !mustMatch.matcher(candidateCollectionName).matches()) {
throw new SolrException(BAD_REQUEST, "Collection name " + candidateCollectionName
+ " does not match " + ROUTER_MUST_MATCH + ": " + mustMatch);
diff --git a/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java b/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
index e84912e..813a326 100644
--- a/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
+++ b/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
@@ -203,6 +203,8 @@ public class CategoryRoutedAliasUpdateProcessorTest extends RoutedAliasUpdatePro
// make sure we fail if we have no value to route on.
testFailedDocument(newDoc(null), "Route value is null");
+ testFailedDocument(newDoc("foo__CRA__bar"), "7 character sequence __CRA__");
+ testFailedDocument(newDoc("fóóCRAóóbar"), "7 character sequence __CRA__");
}
@@ -464,7 +466,7 @@ public class CategoryRoutedAliasUpdateProcessorTest extends RoutedAliasUpdatePro
// if we have a TolerantUpdateProcessor then we see it there)
final Object errors = resp.getResponseHeader().get("errors"); // Tolerant URP
assertNotNull(errors);
- assertTrue(errors.toString().contains(errorMsg));
+ assertTrue("Expected to find " + errorMsg + " in errors: " + errors.toString(),errors.toString().contains(errorMsg));
} catch (SolrException e) {
assertTrue(e.getMessage().contains(errorMsg));
}
[lucene-solr] 07/10: SOLR-13152 v2 api support
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit 46490b390a41ca134457f54002a2b406d5cee638
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sun Mar 3 15:27:54 2019 -0500
SOLR-13152 v2 api support
---
solr/solrj/src/resources/apispec/collections.Commands.json | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/solr/solrj/src/resources/apispec/collections.Commands.json b/solr/solrj/src/resources/apispec/collections.Commands.json
index dc8e251..f24fe72 100644
--- a/solr/solrj/src/resources/apispec/collections.Commands.json
+++ b/solr/solrj/src/resources/apispec/collections.Commands.json
@@ -175,6 +175,14 @@
"autoDeleteAge": {
"type": "string",
"description": "A date math expressions yielding a time in the past. Collections covering a period of time entirely before this age will be automatically deleted."
+ },
+ "maxCardinality": {
+ "type": "integer",
+ "description": "The maximum number of categories allowed for this alias."
+ },
+ "mustMatch": {
+ "type": "string",
+ "description": "A regular expression that the value of the field specified by `router.field` must match before a corresponding collection will be created."
}
}
},
[lucene-solr] 08/10: SOLR-13152 precommit fix
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit d7cb6d68534b431e32fb6a4ecc24fdee613a077e
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sun Mar 3 20:30:46 2019 -0500
SOLR-13152 precommit fix
---
.../java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java b/solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
index 4951003..1ae3229 100644
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
@@ -38,9 +38,8 @@ import java.util.Set;
import java.util.TimeZone;
import java.util.concurrent.TimeUnit;
-import com.google.common.base.Objects;
-import org.apache.solr.cloud.ZkController;
import com.google.common.base.MoreObjects;
+import org.apache.solr.cloud.ZkController;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.cloud.Aliases;
import org.apache.solr.common.params.CommonParams;
[lucene-solr] 04/10: SOLR-13152 Check for missing route value
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit 6034261fdfeffa35f95ef4135ab5725c1fd1ad0d
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sat Mar 2 18:06:56 2019 -0500
SOLR-13152 Check for missing route value
---
.../cloud/api/collections/CategoryRoutedAlias.java | 25 ++++++++++++++++------
.../CategoryRoutedAliasUpdateProcessorTest.java | 18 ++++++++++++----
2 files changed, 32 insertions(+), 11 deletions(-)
diff --git a/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java b/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
index 308ae6e..0b0114d 100644
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
@@ -29,6 +29,7 @@ import java.util.regex.PatternSyntaxException;
import org.apache.solr.cloud.ZkController;
import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
import org.apache.solr.common.cloud.Aliases;
import org.apache.solr.common.params.CommonParams;
import org.apache.solr.core.CoreContainer;
@@ -39,6 +40,8 @@ import org.apache.solr.update.AddUpdateCommand;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
+
public class CategoryRoutedAlias implements RoutedAlias {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final String COLLECTION_INFIX = "__CRA__";
@@ -119,7 +122,15 @@ public class CategoryRoutedAlias implements RoutedAlias {
updateParsedCollectionAliases(cmd.getReq().getCore().getCoreContainer().getZkController());
}
- String dataValue = String.valueOf(cmd.getSolrInputDocument().getFieldValue(getRouteField()));
+ Object fieldValue = cmd.getSolrInputDocument().getFieldValue(getRouteField());
+ // possible future enhancement: allow specification of an "unknown" category name to where we can send
+ // docs that are uncategorized.
+ if (fieldValue == null) {
+ throw new SolrException(BAD_REQUEST,"Route value is null");
+ }
+
+ String dataValue = String.valueOf(fieldValue);
+
String candidateCollectionName = buildCollectionNameFromValue(dataValue);
List<String> cols = getCollectionList(this.parsedAliases);
@@ -128,13 +139,13 @@ public class CategoryRoutedAlias implements RoutedAlias {
}
if (mustMatch != null && !mustMatch.matcher(candidateCollectionName).matches()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Collection name " + candidateCollectionName
+ throw new SolrException(BAD_REQUEST, "Collection name " + candidateCollectionName
+ " does not match " + ROUTER_MUST_MATCH + ": " + mustMatch);
}
if (cols.stream()
.filter(x -> !x.contains(UNINITIALIZED)).count() >= maxCardinality) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Max cardinality " + maxCardinality
+ throw new SolrException(BAD_REQUEST, "Max cardinality " + maxCardinality
+ " reached for Category Routed Alias: " + getAliasName());
}
}
@@ -190,7 +201,7 @@ public class CategoryRoutedAlias implements RoutedAlias {
// we should see some sort of update to our aliases
if (!updateParsedCollectionAliases(coreContainer.getZkController())) { // thus we didn't make progress...
// this is not expected, even in known failure cases, but we check just in case
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
+ throw new SolrException(ErrorCode.SERVER_ERROR,
"We need to create a new category routed collection but for unknown reasons were unable to do so.");
}
}
@@ -198,7 +209,7 @@ public class CategoryRoutedAlias implements RoutedAlias {
} catch (SolrException e) {
throw e;
} catch (Exception e) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e);
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
}
}
@@ -206,7 +217,7 @@ public class CategoryRoutedAlias implements RoutedAlias {
try {
return Integer.valueOf(maxCardinality);
} catch (NumberFormatException e) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ROUTER_MAX_CARDINALITY + " must be a valid Integer"
+ throw new SolrException(BAD_REQUEST, ROUTER_MAX_CARDINALITY + " must be a valid Integer"
+ ", instead got: " + maxCardinality);
}
}
@@ -215,7 +226,7 @@ public class CategoryRoutedAlias implements RoutedAlias {
try {
return Pattern.compile(mustMatch);
} catch (PatternSyntaxException e) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ROUTER_MUST_MATCH + " must be a valid regular"
+ throw new SolrException(BAD_REQUEST, ROUTER_MUST_MATCH + " must be a valid regular"
+ " expression, instead got: " + mustMatch);
}
}
diff --git a/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java b/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
index 54cc7d8..e84912e 100644
--- a/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
+++ b/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
@@ -200,6 +200,10 @@ public class CategoryRoutedAliasUpdateProcessorTest extends RoutedAliasUpdatePro
newDoc(SHIPS[4]));
assertInvariants(colVogon, colHoG, colStunt, colArk, colBistro);
+
+ // make sure we fail if we have no value to route on.
+ testFailedDocument(newDoc(null), "Route value is null");
+
}
private String noSpaces(String ship) {
@@ -424,9 +428,14 @@ public class CategoryRoutedAliasUpdateProcessorTest extends RoutedAliasUpdatePro
}
private SolrInputDocument newDoc(String routedValue) {
- return sdoc("id", Integer.toString(++lastDocId),
- categoryField, routedValue,
- intField, "0"); // always 0
+ if (routedValue != null) {
+ return sdoc("id", Integer.toString(++lastDocId),
+ categoryField, routedValue,
+ intField, "0"); // always 0
+ } else {
+ return sdoc("id", Integer.toString(++lastDocId),
+ intField, "0"); // always 0
+ }
}
@Override
@@ -454,7 +463,8 @@ public class CategoryRoutedAliasUpdateProcessorTest extends RoutedAliasUpdatePro
final UpdateResponse resp = solrClient.add(getAlias(), sdoc);
// if we have a TolerantUpdateProcessor then we see it there)
final Object errors = resp.getResponseHeader().get("errors"); // Tolerant URP
- assertTrue(errors != null && errors.toString().contains(errorMsg));
+ assertNotNull(errors);
+ assertTrue(errors.toString().contains(errorMsg));
} catch (SolrException e) {
assertTrue(e.getMessage().contains(errorMsg));
}
[lucene-solr] 03/10: Merge branch 'SOLR-13152' of
https://github.com/moshebla/lucene-solr into moshebla-SOLR-13152
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit d6704e50b4fc9928a001758fe11382ed7a24d87a
Merge: fc0b9fa 4b482d6
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sat Mar 2 12:09:57 2019 -0500
Merge branch 'SOLR-13152' of https://github.com/moshebla/lucene-solr into moshebla-SOLR-13152
.../cloud/api/collections/CategoryRoutedAlias.java | 19 ++++++----
.../CategoryRoutedAliasUpdateProcessorTest.java | 43 +++++-----------------
.../solrj/request/CollectionAdminRequest.java | 17 +++------
3 files changed, 28 insertions(+), 51 deletions(-)
[lucene-solr] 10/10: SOLR-13152 documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit 5bda03c2b108f73fe196f68470cc52130ef8d48f
Author: Gus Heck <gu...@apache.org>
AuthorDate: Mon Mar 4 01:58:51 2019 -0500
SOLR-13152 documentation
---
solr/solr-ref-guide/src/aliases.adoc | 267 ++++++++++++++++++++++
solr/solr-ref-guide/src/collections-api.adoc | 60 +++--
solr/solr-ref-guide/src/distributed-requests.adoc | 1 +
solr/solr-ref-guide/src/how-solrcloud-works.adoc | 4 +-
solr/solr-ref-guide/src/time-routed-aliases.adoc | 119 ----------
5 files changed, 307 insertions(+), 144 deletions(-)
diff --git a/solr/solr-ref-guide/src/aliases.adoc b/solr/solr-ref-guide/src/aliases.adoc
new file mode 100644
index 0000000..52ce75d
--- /dev/null
+++ b/solr/solr-ref-guide/src/aliases.adoc
@@ -0,0 +1,267 @@
+= Aliases
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+== Standard Aliases
+
+Since version 6, SolrCloud has had the ability to query one or more collections via an alternative name. These
+alternative names for collections are known as aliases, and are useful when you want to:
+
+1. Atomically switch to using a newly (re)indexed collection with zero down time (by re-defining the alias)
+1. Insulate the client programming versus changes in collection names
+1. Issue a single query against several collections with identical schemas
+
+It's also possible to send update commands to aliases, but this is rarely useful if the
+ alias refers to more than one collection (as in case 3 above).
+Since there is no logic by which to distribute documents among the collections, all updates will simply be
+ directed to the first collection in the list.
+
+Standard aliases are created and updated using the <<collections-api.adoc#createalias,CREATEALIAS>> command.
+The current list of collections that are members of an alias can be verified via the
+ <<collections-api.adoc#clusterstatus,CLUSTERSTATUS>> command.
+The full definition of all aliases including metadata about that alias (in the case of routed aliases, see below)
+ can be verified via the <<collections-api.adoc#listaliases,LISTALIASES>> command.
+Alternatively this information is available by checking `/aliases.json` in zookeeper via a zookeeper
+ client or in the <<cloud-screens.adoc#tree-view,tree page>> of the cloud menu in the admin UI.
+Aliases may be deleted via the <<collections-api.adoc#deletealias,DELETEALIAS>> command.
+The underlying collections are *unaffected* by this command.
+
+TIP: Any alias (standard or routed) that references multiple collections may complicate relevancy.
+By default, SolrCloud scores documents on a per shard basis.
+With multiple collections in an alias this is always a problem, so if you have a use case for which BM25 or
+ TF/IDF relevancy is important you will want to turn on one of the
+ <<distributed-requests.adoc#distributedidf,ExactStatsCache>> implementations.
+However, for analytical use cases where results are sorted on numeric, date or alphanumeric field values rather
+ than relevancy calculations this is not a problem.
+
+== Routed Aliases
+
+To address the update limitations associated with standard aliases and provide additional useful features, the concept of
+ RoutedAliases has been developed.
+There are presently two types of Routed Alias time routed and category routed. These are described in detail below,
+ but share some common behavior.
+
+When processing an update for a routed alias, Solr initializes its
+ <<update-request-processors.adoc#update-request-processors,UpdateRequestProcessor>> chain as usual, but
+ when `DistributedUpdateProcessor` (DUP) initializes, it detects that the update targets a routed alias and injects
+ `RoutedAliasUpdateProcessor` (RAUP) in front of itself.
+RAUP, in coordination with the Overseer, is the main part of a routed alias, and must immediately precede DUP. It is not
+ possible to configure custom chains with other types of UpdateRequestProcessors between RAUP and DUP.
+
+Ideally, as a user of a routed alias, you needn't concern yourself with the particulars of the collection naming pattern
+ since both queries and updates may be done via the alias.
+When adding data, you should usually direct documents to the alias (e.g., reference the alias name instead of any collection).
+The Solr server and CloudSolrClient will direct an update request to the first collection that an alias points to.
+Once the server receives the data it will perform the necessary routing.
+
+WARNING: It is possible to update the collections
+ directly, but there is no safeguard against putting data in the incorrect collection if the alias is circumvented
+ in this manner.
+
+CAUTION: It's probably a bad idea to use "data driven" mode with routed aliases, as duplicate schema mutations might happen
+concurrently leading to errors.
+
+
+== Time Routed Aliases
+
+Starting in Solr 7.4, Time Routed Aliases (TRAs) are a SolrCloud feature that manages an alias and a time sequential
+ series of collections.
+
+It automatically creates new collections and (optionally) deletes old ones as it routes documents to the correct
+ collection based on its timestamp.
+This approach allows for indefinite indexing of data without degradation of performance otherwise experienced due to the
+ continuous growth of a single index.
+
+If you need to store a lot of timestamped data in Solr, such as logs or IoT sensor data, then this feature probably
+ makes more sense than creating one sharded hash-routed collection.
+
+=== How It Works
+
+First you create a time routed aliases using the <<collections-api.adoc#createalias,CREATEALIAS>> command with some
+ router settings.
+Most of the settings are editable at a later time using the <<collections-api.adoc#aliasprop,ALIASPROP>> command.
+
+The first collection will be created automatically, along with an alias pointing to it.
+Each underlying Solr "core" in a collection that is a member of a TRA has a special core property referencing the alias.
+The name of each collection is comprised of the TRA name and the start timestamp (UTC), with trailing zeros and symbols
+ truncated.
+
+The collections list for a TRA is always reverse sorted, and thus the connection path of the request will route to the
+ lead collection. Using CloudSolrClient is preferable as it can reduce the number of underlying physical HTTP requests by one.
+If you know that a particular set of documents to be delivered is going to a particular older collection then you could
+ direct it there from the client side as an optimization but it's not necessary. CloudSolrClient does not (yet) do this.
+
+
+TRUP first reads TRA configuration from the alias properties when it is initialized. As it sees each document, it checks for
+ changes to TRA properties, updates its cached configuration if needed and then determines which collection the
+ document belongs to:
+
+* If TRUP needs to send it to a time segment represented by a collection other than the one that
+ the client chose to communicate with, then it will do so using mechanisms shared with DUP.
+ Once the document is forwarded to the correct collection (i.e., the correct TRA time segment), it skips directly to
+ DUP on the target collection and continues normally, potentially being routed again to the correct shard & replica
+ within the target collection.
+
+* If it belongs in the current collection (which is usually the case if processing events as they occur), the document
+ passes through to DUP. DUP does it's normal collection-level processing that may involve routing the document
+ to another shard & replica.
+
+* If the time stamp on the document is more recent than the most recent TRA segment, then a new collection needs to be
+ added at the front of the TRA.
+ TRUP will create this collection, add it to the alias and then forward the document to the collection it just created.
+ This can happen recursively if more than one collection needs to be created.
++
+Each time a new collection is added, the oldest collections in the TRA are examined for possible deletion, if that has
+ been configured.
+All this happens synchronously, potentially adding seconds to the update request and indexing latency.
+If `router.preemptiveCreateMath` is configured and if the document arrives within this window then it will occur
+asynchronously.
+
+Any other type of update like a commit or delete is routed by TRUP to all collections.
+Generally speaking, this is not a performance concern. When Solr receives a delete or commit wherein nothing is deleted
+or nothing needs to be committed, then it's pretty cheap.
+
+
+=== Limitations & Assumptions
+
+* Only *time* routed aliases are supported. If you instead have some other sequential number, you could fake it
+ as a time (e.g., convert to a timestamp assuming some epoch and increment).
++
+The smallest possible interval is one second.
+No other routing scheme is supported, although this feature was developed with considerations that it could be
+ extended/improved to other schemes.
+
+* The underlying collections form a contiguous sequence without gaps. This will not be suitable when there are
+ large gaps in the underlying data, as Solr will insist that there be a collection for each increment. This
+ is due in part on Solr calculating the end time of each interval collection based on the timestamp of
+ the next collection, since it is otherwise not stored in any way.
+
+* Avoid sending updates to the oldest collection if you have also configured that old collections should be
+ automatically deleted. It could lead to exceptions bubbling back to the indexing client.
+
+== Category Routed Aliases
+
+Starting in Solr 8.1, Category Routed Aliases (CRAs) are a feature to manage aliases and a set of dependent collections
+based on the value of a single field.
+
+CRAs automatically create new collections but because the partitioning is on categorical information rather than continuous
+numerically based values there's no logic for automatic deletion. This approach allows for simplified indexing of data
+that must be segregated into collections for cluster management or security reasons.
+
+=== How It Works
+
+First you create a time routed aliases using the <<collections-api.adoc#createalias,CREATEALIAS>> command with some
+ router settings.
+ Most of the settings are editable at a later time using the <<collections-api.adoc#aliasprop,ALIASPROP>> command.
+
+The alias will be created with a special place-holder collection which will always be named
+ `myAlias__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA__TEMP`. The first document indexed into the CRA
+ will create a second collection named `myAlias__CRA__foo` (for a routed field value of `foo`). The second document
+ indexed will cause the temporary place holder collection to be deleted. Thereafter collections will be created whenever
+ a new value for the field is encountered.
+
+CAUTION: To guard against runaway collection creation options for limiting the total number of categories, and for
+rejecting values that don't match a regular expression are provided (see <<collections-api.adoc#createalias,CREATEALIAS>> for
+details). Note that by providing very large or very permissive values for these options you are accepting the risk that
+garbled data could potentially create thousands of collections and bring your cluster to a grinding halt.
+
+Please note that the values (and thus the collection names) are case sensitive. As elsewhere in Solr manipulation and
+cleaning of the data is expected to be done by external processes before data is sent to Solr with one exception.
+Throughout Solr there are limitations on the allowable characters in collection names. Any characters other than ASCII
+alphanumeric characters (`A-Za-z0-9`), hyphen (`-`) or underscore (`_`) are replaced with an underscore when calculating
+the collection name for a category. For a CRA named `myAlias` the following table shows how collection names would be
+calculated:
+
+|===
+|Value |CRA Collection Name
+
+|foo
+|+myAlias__CRA__foo+
+
+|Foo
+|+myAlias__CRA__Foo+
+
+|foo bar
+|+myAlias__CRA__foo_bar+
+
+|+FOÓB&R+
+|+myAlias__CRA__FO_B_R+
+
+|+中文的东西+
+|+myAlias__CRA_______+
+
+|+foo__CRA__bar+
+|*Causes 400 Bad Request*
+
+|+<null>+
+|*Causes 400 Bad Request*
+
+|===
+
+Since collection creation can take upwards of 1-3 seconds, systems inserting data in a CRA should be
+ constructed to handle such pauses whenever a new collection is created.
+Unlike time routed aliases, there is no way to predict the next value so such pauses are unavoidable.
+
+There is no automated means of removing a category. If a category needs to be removed from a CRA
+the following procedure is recommended:
+
+1. Ensure that no documents with the value corresponding to the category to be removed will be sent
+ either by stopping indexing or by fixing the incoming data stream
+1. Modify the alias definition in zookeeper, removing the collection corresponding to the category.
+1. Delete the collection corresponding to the category. Note that if the collection is not removed
+ from the alias first, this step will fail.
+
+=== Limitations & Assumptions
+
+* CRAs are presently unsuitable for non-english data values due to the limits on collection names.
+ This can be worked around by duplicating the route value to a *_url safe_* base 64 encoded field
+ and routing on that value instead.
+
+* The check for the __CRA__ infix is independent of the regular expression validation and occurs after
+ the name of the collection to be created has been calculated. It may not be avoided and is necessary
+ to support future features.
+
+== Improvement Possibilities
+
+Routed aliases are a relatively new feature of SolrCloud that can be expected to be improved.
+Some _potential_ areas for improvement that _are not implemented yet_ are:
+
+* *TRAs*: Searches with time filters should only go to applicable collections.
+
+* *TRAs*: Ways to automatically optimize (or reduce the resources of) older collections that aren't expected to receive more
+ updates, and might have less search demand.
+
+* *CRAs*: Intrinsic support for non-english text via base64 encoding
+
+* *CRAs*: Supply an initial list of values for cases where these are known before hand to reduce pauses during indexing
+
+* CloudSolrClient could route documents to the correct collection based on the route value instead always picking the
+ latest/first.
+
+* Presently only updates are routed and queries are distributed to all collections in the alias, but future
+ features might enable routing of the query to the single appropriate collection based on a special parameter or perhaps
+ a filter on the routed field.
+
+* Collections might be constrained by their size instead of or in addition to time or category value.
+ This might be implemented as another type of routed alias, or possibly as an option on the existing routed aliases
+
+* Compatibility with CDCR.
+
+* Option for deletion of aliases that also deletes the underlying collections in one step. Routed Aliases may quickly
+ create more collections than expected during initial testing. Removing them after such events is overly tedious.
+
+As always, patches and pull requests are welcome!
\ No newline at end of file
diff --git a/solr/solr-ref-guide/src/collections-api.adoc b/solr/solr-ref-guide/src/collections-api.adoc
index c78db6d..192adff 100644
--- a/solr/solr-ref-guide/src/collections-api.adoc
+++ b/solr/solr-ref-guide/src/collections-api.adoc
@@ -538,7 +538,7 @@ Aliases come in 2 flavors: standard and routed.
*Standard aliases* are simple: CREATEALIAS registers the alias name with the names of one or more collections provided
by the command.
If an existing alias exists, it is replaced/updated.
-A standard alias can serve to have the appearance of renaming a collection, and can be used to atomically swap
+A standard alias can serve as a means to rename a collection, and can be used to atomically swap
which backing/underlying collection is "live" for various purposes.
When Solr searches an alias pointing to multiple collections, Solr will search all shards of all the collections as an
aggregated whole.
@@ -547,23 +547,18 @@ While it is possible to send updates to an alias spanning multiple collections,
`/admin/collections?action=CREATEALIAS&name=_name_&collections=_collectionlist_`
-*Routed aliases* are aliases with additional capabilities to act as a kind of super-collection -- routing
- updates to the correct collection.
-Since the only routing strategy at present is time oriented, these are also called *Time Routed Aliases* (TRAs).
-A TRA manages an alias and a time sequential series of collections that it will both create and optionally delete on-demand.
-See <<time-routed-aliases.adoc#time-routed-aliases,Time Routed Aliases>> for some important high-level information
+*Routed aliases* are aliases with additional capabilities to act as a kind of super-collection that route
+ updates to the correct collection. Routing is data driven and may be based on a temporal field or on categories
+ specified in a field (normally string based).
+See <<aliases.adoc#routed-aliases,Routed Aliases>> for some important high-level information
before getting started.
-NOTE: Presently this is only supported for temporal fields stored as a
-<<field-types-included-with-solr.adoc#field-types-included-with-solr,DatePointField or TrieDateField>> type. Other
-well ordered field types may be added in future versions.
-
[source,text]
----
localhost:8983/solr/admin/collections?action=CREATEALIAS&name=timedata&router.start=NOW/DAY&router.field=evt_dt&router.name=time&router.interval=%2B1DAY&router.maxFutureMs=3600000&create-collection.collection.configName=myConfig&create-collection.numShards=2
----
-If run on Jan 15, 2018, the above will create an alias named timedata, that contains collections with names prefixed
+If run on Jan 15, 2018, the above will create an time routed alias named timedata, that contains collections with names prefixed
with `timedata` and an initial collection named `timedata_2018_01_15` will be created immediately. Updates sent to this
alias with a (required) value in `evt_dt` that is before or after 2018-01-15 will be rejected, until the last 60
minutes of 2018-01-15. After 2018-01-15T23:00:00 documents for either 2018-01-15 or 2018-01-16 will be accepted.
@@ -605,6 +600,21 @@ prohibited. If routing parameters are present this parameter is prohibited.
Most routed alias parameters become _alias properties_ that can subsequently be inspected and <<aliasprop,modified>>.
+`router.name`::
+The type of routing to use. Presently only `time` and `category` are valid. This parameter is required.
+
+`router.field`::
+The field to inspect to determine which underlying collection an incoming document should be routed to.
+This field is required on all incoming documents.
+
+`create-collection.*`::
+The * wildcard can be replaced with any parameter from the <<create,CREATE>> command except `name`. All other fields
+are identical in requirements and naming except that we insist that the configset be explicitly specified.
+The configset must be created beforehand, either uploaded or copied and modified.
+It's probably a bad idea to use "data driven" mode as schema mutations might happen concurrently leading to errors.
+
+==== Time Routed Alias Parameters
+
`router.start`::
The start date/time of data for this time routed alias in Solr's standard date/time format (i.e., ISO-8601 or "NOW"
optionally with <<working-with-dates.adoc#date-math,date math>>).
@@ -625,13 +635,6 @@ myAlias_2018-01-15_01 collection (assuming an interval of +1HOUR).
+
The default timezone is UTC.
-`router.field`::
-The date field to inspect to determine which underlying collection an incoming document should be routed to.
-This field is required on all incoming documents.
-
-`router.name`::
-The type of routing to use. Presently only `time` is valid. This parameter is required.
-
`router.interval`::
A date math expression that will be appended to a timestamp to determine the next collection in the series.
Any date math expression that can be evaluated if appended to a timestamp of the form 2018-01-15T16:17:18 will
@@ -677,11 +680,18 @@ Example: `/DAY-90DAYS`.
+
The default is not to delete.
-`create-collection.*`::
-The * wildcard can be replaced with any parameter from the <<create,CREATE>> command except `name`. All other fields
-are identical in requirements and naming except that we insist that the configset be explicitly specified.
-The configset must be created beforehand, either uploaded or copied and modified.
-It's probably a bad idea to use "data driven" mode as schema mutations might happen concurrently leading to errors.
+==== Category Routed Alias Parameters
+
+`router.maxCardinality`::
+The maximum number of categories allowed for this alias.
+This setting safeguards against the inadvertent creation of an infinite number of collections in the event of bad data.
+
+`router.mustMatch`::
+A regular expression that the value of the field specified by `router.field` must match before a corresponding
+collection will be created. Note that changing this setting after data has been added will not alter the data already
+indexed. Any valid Java regular expression pattern may be specified. This expression is pre-compiled at the start of
+each request so batching of updates is strongly recommended. Overly complex patterns will produce cpu
+or garbage collecting overhead during indexing as determined by the JVM's implementation of regular expressions.
=== CREATEALIAS Response
@@ -836,6 +846,10 @@ The `ALIASPROP` action modifies the properties (metadata) on an alias. If a key
`/admin/collections?action=ALIASPROP&name=_name_&property.someKey=somevalue`
+WARNING: This command allows you to revise any property. No alias specific validation is performed.
+ Routed aliases may cease to function, function incorrectly or cause errors if property values
+ are set carelessly.
+
=== ALIASPROP Parameters
`name`::
diff --git a/solr/solr-ref-guide/src/distributed-requests.adoc b/solr/solr-ref-guide/src/distributed-requests.adoc
index be129ed..3422540 100644
--- a/solr/solr-ref-guide/src/distributed-requests.adoc
+++ b/solr/solr-ref-guide/src/distributed-requests.adoc
@@ -127,6 +127,7 @@ If you need to disable this feature for backwards compatibility, you can set the
+
NOTE: In SolrCloud mode, if at least one node is included in the whitelist, then the `live_nodes` will no longer be used as source for the list. This means that if you need to do a cross-cluster request using the `shards` parameter in SolrCloud mode (in addition to regular within-cluster requests), you'll need to add all nodes (local cluster + remote nodes) to the whitelist.
+[[distributedidf]]
== Configuring statsCache (Distributed IDF)
Document and term statistics are needed in order to calculate relevancy. Solr provides four implementations out of the box when it comes to document stats calculation:
diff --git a/solr/solr-ref-guide/src/how-solrcloud-works.adoc b/solr/solr-ref-guide/src/how-solrcloud-works.adoc
index b0e42e4..7f4db3b 100644
--- a/solr/solr-ref-guide/src/how-solrcloud-works.adoc
+++ b/solr/solr-ref-guide/src/how-solrcloud-works.adoc
@@ -1,5 +1,5 @@
= How SolrCloud Works
-:page-children: shards-and-indexing-data-in-solrcloud, distributed-requests, time-routed-aliases
+:page-children: shards-and-indexing-data-in-solrcloud, distributed-requests, aliases
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -21,7 +21,7 @@ The following sections cover provide general information about how various SolrC
* <<shards-and-indexing-data-in-solrcloud.adoc#shards-and-indexing-data-in-solrcloud,Shards and Indexing Data in SolrCloud>>
* <<distributed-requests.adoc#distributed-requests,Distributed Requests>>
-* <<time-routed-aliases.adoc#time-routed-aliases,Time Routed Aliases>>
+* <<aliases.adoc#aliases,Standard and Routed Aliases>>
If you are already familiar with SolrCloud concepts and basic functionality, you can skip to the section covering <<solrcloud-configuration-and-parameters.adoc#solrcloud-configuration-and-parameters,SolrCloud Configuration and Parameters>>.
diff --git a/solr/solr-ref-guide/src/time-routed-aliases.adoc b/solr/solr-ref-guide/src/time-routed-aliases.adoc
deleted file mode 100644
index 3ef0e19..0000000
--- a/solr/solr-ref-guide/src/time-routed-aliases.adoc
+++ /dev/null
@@ -1,119 +0,0 @@
-= Time Routed Aliases
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-Time Routed Aliases (TRAs) is a SolrCloud feature that manages an alias and a time sequential series of collections.
-
-It automatically creates new collections and (optionally) deletes old ones as it routes documents to the correct
- collection based on its timestamp.
-This approach allows for indefinite indexing of data without degradation of performance otherwise experienced due to the
- continuous growth of a single index.
-
-If you need to store a lot of timestamped data in Solr, such as logs or IoT sensor data, then this feature probably
- makes more sense than creating one sharded hash-routed collection.
-
-== How It Works
-
-First you create a time routed aliases using the <<collections-api.adoc#createalias,CREATEALIAS>> command with some
- router settings.
-Most of the settings are editable at a later time using the <<collections-api.adoc#aliasprop,ALIASPROP>> command.
-
-The first collection will be created automatically, along with an alias pointing to it.
-Each underlying Solr "core" in a collection that is a member of a TRA has a special core property referencing the alias.
-The name of each collection is comprised of the TRA name and the start timestamp (UTC), with trailing zeros and symbols
- truncated.
-Ideally, as a user of this feature, you needn't concern yourself with the particulars of the collection naming pattern
- since both queries and updates may be done via the alias.
-
-When adding data, you should usually direct documents to the alias (e.g., reference the alias name instead of any collection).
-The Solr server and CloudSolrClient will direct an update request to the first collection that an alias points to.
-
-The collections list for a TRA is always reverse sorted, and thus the connection path of the request will route to the
- lead collection. Using CloudSolrClient is preferable as it can reduce the number of underlying physical HTTP requests by one.
-If you know that a particular set of documents to be delivered is going to a particular older collection then you could
- direct it there from the client side as an optimization but it's not necessary. CloudSolrClient does not (yet) do this.
-
-When processing an update for a TRA, Solr initializes its
- <<update-request-processors.adoc#update-request-processors,UpdateRequestProcessor>> chain as usual, but
- when `DistributedUpdateProcessor` (DUP) initializes, it detects that the update targets a TRA and injects
- `TimeRoutedUpdateProcessor` (TRUP) in front of itself.
-TRUP, in coordination with the Overseer, is the main part of a TRA, and must immediately precede DUP. It is not
- possible to configure custom chains with other types of UpdateRequestProcessors between TRUP and DUP.
-
-TRUP first reads TRA configuration from the alias properties when it is initialized. As it sees each document, it checks for
- changes to TRA properties, updates its cached configuration if needed and then determines which collection the
- document belongs to:
-
-* If TRUP needs to send it to a time segment represented by a collection other than the one that
- the client chose to communicate with, then it will do so using mechanisms shared with DUP.
- Once the document is forwarded to the correct collection (i.e., the correct TRA time segment), it skips directly to
- DUP on the target collection and continues normally, potentially being routed again to the correct shard & replica
- within the target collection.
-
-* If it belongs in the current collection (which is usually the case if processing events as they occur), the document
- passes through to DUP. DUP does it's normal collection-level processing that may involve routing the document
- to another shard & replica.
-
-* If the time stamp on the document is more recent than the most recent TRA segment, then a new collection needs to be
- added at the front of the TRA.
- TRUP will create this collection, add it to the alias and then forward the document to the collection it just created.
- This can happen recursively if more than one collection needs to be created.
-+
-Each time a new collection is added, the oldest collections in the TRA are examined for possible deletion, if that has
- been configured.
-All this happens synchronously, potentially adding seconds to the update request and indexing latency.
-If `router.preemptiveCreateMath` is configured and if the document arrives within this window then it will occur
-asynchronously.
-
-Any other type of update like a commit or delete is routed by TRUP to all collections.
-Generally speaking, this is not a performance concern. When Solr receives a delete or commit wherein nothing is deleted
-or nothing needs to be committed, then it's pretty cheap.
-
-== Improvement Possibilities
-
-This is a new feature of SolrCloud that can be expected to be improved.
-Some _potential_ areas for improvement that _are not implemented yet_ are:
-
-* Searches with time filters should only go to applicable collections.
-
-* Collections ought to be constrained by their size instead of or in addition to time.
- Based on the underlying design, this would only apply to the lead collection.
-
-* Ways to automatically optimize (or reduce the resources of) older collections that aren't expected to receive more
- updates, and might have less search demand.
-
-* CloudSolrClient could route documents to the correct collection based on a timestamp instead always picking the
- latest.
-
-* Compatibility with CDCR.
-
-== Limitations & Assumptions
-
-* Only *time* routed aliases are supported. If you instead have some other sequential number, you could fake it
- as a time (e.g., convert to a timestamp assuming some epoch and increment).
-+
-The smallest possible interval is one second.
-No other routing scheme is supported, although this feature was developed with considerations that it could be
- extended/improved to other schemes.
-
-* The underlying collections form a contiguous sequence without gaps. This will not be suitable when there are
- large gaps in the underlying data, as Solr will insist that there be a collection for each increment. This
- is due in part on Solr calculating the end time of each interval collection based on the timestamp of
- the next collection, since it is otherwise not stored in any way.
-
-* Avoid sending updates to the oldest collection if you have also configured that old collections should be
- automatically deleted. It could lead to exceptions bubbling back to the indexing client.
[lucene-solr] 01/10: Merge branch 'master' of
https://gitbox.apache.org/repos/asf/lucene-solr into solr-13131
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit d87e0c2039d14739559eb0ff5b9ae12c9668c58c
Merge: 8bff5b4 8c34da8
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sat Mar 2 12:05:39 2019 -0500
Merge branch 'master' of https://gitbox.apache.org/repos/asf/lucene-solr into solr-13131
lucene/CHANGES.txt | 18 +
.../codecs/simpletext/SimpleTextBKDWriter.java | 73 +--
.../codecs/blocktree/BlockTreeTermsReader.java | 9 +-
.../lucene/codecs/blocktree/FieldReader.java | 11 +-
.../apache/lucene/store/ByteBufferIndexInput.java | 2 +-
.../apache/lucene/util/bkd/BKDRadixSelector.java | 336 +++++++++----
.../java/org/apache/lucene/util/bkd/BKDWriter.java | 62 +--
.../apache/lucene/util/bkd/HeapPointReader.java | 68 ++-
.../apache/lucene/util/bkd/HeapPointWriter.java | 124 ++---
.../apache/lucene/util/bkd/OfflinePointReader.java | 66 ++-
.../apache/lucene/util/bkd/OfflinePointWriter.java | 22 +-
.../org/apache/lucene/util/bkd/PointReader.java | 11 +-
.../util/bkd/{PointReader.java => PointValue.java} | 26 +-
.../org/apache/lucene/util/bkd/PointWriter.java | 9 +-
.../src/java/org/apache/lucene/util/fst/FST.java | 41 +-
.../java/org/apache/lucene/util/fst/FSTStore.java | 25 +-
.../apache/lucene/util/fst/OffHeapFSTStore.java | 69 +++
.../org/apache/lucene/util/fst/OnHeapFSTStore.java | 97 ++++
.../lucene/util/fst/ReverseRandomAccessReader.java | 63 +++
.../util/graph/GraphTokenStreamFiniteStrings.java | 17 +-
.../lucene/util/bkd/TestBKDRadixSelector.java | 76 +--
.../graph/TestGraphTokenStreamFiniteStrings.java | 48 +-
.../taxonomy/TestTaxonomyFacetSumValueSource.java | 6 +-
lucene/ivy-versions.properties | 16 +-
.../apache/lucene/search/intervals/Intervals.java | 12 +
.../apache/lucene/document/TestLatLonShape.java | 532 ---------------------
...tLonShape.java => TestLatLonShapeEncoding.java} | 381 +--------------
.../search/intervals/TestSimplifications.java | 49 ++
.../apache/lucene/spatial3d/geom/GeoPathTest.java | 39 +-
solr/CHANGES.txt | 37 ++
...geIdentifierUpdateProcessorFactoryTestCase.java | 26 +-
solr/core/ivy.xml | 1 +
.../cloud/api/collections/CreateCollectionCmd.java | 2 +-
.../OverseerCollectionMessageHandler.java | 56 ++-
.../solr/cloud/api/collections/RestoreCmd.java | 41 +-
.../cloud/api/collections/TimeRoutedAlias.java | 3 +-
.../solr/handler/admin/CoreAdminHandler.java | 3 +-
.../solr/handler/component/HighlightComponent.java | 4 +-
.../solr/handler/component/StatsValuesFactory.java | 3 +-
.../apache/solr/handler/export/ExportWriter.java | 138 +++---
.../org/apache/solr/search/FunctionQParser.java | 60 ++-
.../security/DelegationTokenKerberosFilter.java | 15 +-
.../org/apache/solr/security/HadoopAuthFilter.java | 17 +-
.../org/apache/solr/update/DocumentBuilder.java | 6 +-
.../apache/solr/update/StreamingSolrClients.java | 2 +-
.../org/apache/solr/util/StartupLoggingUtils.java | 18 +
solr/core/src/test-files/log4j2.xml | 45 +-
.../collection1/conf/schema-sortingresponse.xml | 25 +-
.../configsets/cloud-managed/conf/managed-schema | 12 +
.../cloud/AsyncCallRequestStatusResponseTest.java | 26 +-
.../solr/cloud/CreateCollectionCleanupTest.java | 37 +-
.../apache/solr/cloud/DocValuesNotIndexedTest.java | 44 +-
.../org/apache/solr/cloud/TestPullReplica.java | 2 +-
.../org/apache/solr/cloud/TestTlogReplica.java | 27 +-
.../AbstractCloudBackupRestoreTestCase.java | 73 ++-
.../collections/TestHdfsCloudBackupRestore.java | 6 +-
.../collections/TestLocalFSCloudBackupRestore.java | 6 +-
.../apache/solr/cloud/rule/ImplicitSnitchTest.java | 3 +-
.../apache/solr/handler/RequestLoggingTest.java | 29 +-
.../TestDistributedStatsComponentCardinality.java | 3 +-
.../solr/handler/export/TestExportWriter.java | 32 +-
.../org/apache/solr/logging/TestLogWatcher.java | 82 ++--
.../test/org/apache/solr/schema/UUIDFieldTest.java | 4 +-
.../search/SignificantTermsQParserPluginTest.java | 4 +-
.../solr/search/function/TestFunctionQuery.java | 47 ++
.../apache/solr/util/TestObjectReleaseTracker.java | 3 +-
.../org/apache/solr/util/TestSystemIdResolver.java | 3 +-
solr/licenses/avatica-core-1.10.0.jar.sha1 | 1 -
solr/licenses/avatica-core-1.13.0.jar.sha1 | 1 +
solr/licenses/calcite-core-1.13.0.jar.sha1 | 1 -
solr/licenses/calcite-core-1.18.0.jar.sha1 | 1 +
solr/licenses/calcite-linq4j-1.13.0.jar.sha1 | 1 -
solr/licenses/calcite-linq4j-1.18.0.jar.sha1 | 1 +
solr/licenses/commons-compiler-2.7.6.jar.sha1 | 1 -
solr/licenses/commons-compiler-3.0.9.jar.sha1 | 1 +
solr/licenses/curator-client-2.13.0.jar.sha1 | 1 +
solr/licenses/curator-client-2.8.0.jar.sha1 | 1 -
solr/licenses/curator-framework-2.13.0.jar.sha1 | 1 +
solr/licenses/curator-framework-2.8.0.jar.sha1 | 1 -
solr/licenses/curator-recipes-2.13.0.jar.sha1 | 1 +
solr/licenses/curator-recipes-2.8.0.jar.sha1 | 1 -
solr/licenses/disruptor-3.4.0.jar.sha1 | 1 -
solr/licenses/disruptor-3.4.2.jar.sha1 | 1 +
solr/licenses/guava-14.0.1.jar.sha1 | 1 -
solr/licenses/guava-25.1-jre.jar.sha1 | 1 +
solr/licenses/janino-2.7.6.jar.sha1 | 1 -
solr/licenses/janino-3.0.9.jar.sha1 | 1 +
solr/licenses/json-path-2.4.0.jar.sha1 | 1 +
solr/licenses/json-path-LICENSE-ASL.txt | 202 ++++++++
solr/licenses/json-path-NOTICE.txt | 0
solr/licenses/log4j-1.2-api-2.11.0.jar.sha1 | 1 -
solr/licenses/log4j-1.2-api-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-api-2.11.0.jar.sha1 | 1 -
solr/licenses/log4j-api-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-core-2.11.0.jar.sha1 | 1 -
solr/licenses/log4j-core-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-slf4j-impl-2.11.0.jar.sha1 | 1 -
solr/licenses/log4j-slf4j-impl-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-web-2.11.2.jar.sha1 | 1 +
solr/licenses/log4j-web-LICENSE-ASL.txt | 202 ++++++++
solr/licenses/log4j-web-NOTICE.txt | 17 +
solr/server/ivy.xml | 1 +
solr/server/resources/log4j2-console.xml | 38 +-
solr/server/resources/log4j2.xml | 68 ++-
solr/solr-ref-guide/src/solr-upgrade-notes.adoc | 10 +
.../src/solrcloud-autoscaling-triggers.adoc | 2 +-
.../org/apache/solr/common/SolrInputField.java | 61 ++-
.../common/util/ByteArrayUtf8CharSequence.java | 5 +
.../org/apache/solr/common/util/JavaBinCodec.java | 2 +-
.../src/java/org/apache/solr/SolrTestCaseJ4.java | 2 +
.../solr/cloud/AbstractFullDistribZkTestBase.java | 2 +
111 files changed, 2222 insertions(+), 1673 deletions(-)
diff --cc solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
index dcd4a37,2da81da..4951003
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
@@@ -34,12 -31,12 +34,13 @@@ import java.util.HashSet
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.TimeZone;
import java.util.concurrent.TimeUnit;
-import java.util.function.Predicate;
-import java.util.function.Supplier;
+import com.google.common.base.Objects;
+import org.apache.solr.cloud.ZkController;
+ import com.google.common.base.MoreObjects;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.cloud.Aliases;
import org.apache.solr.common.params.CommonParams;
[lucene-solr] 09/10: Merge branch 'moshebla-SOLR-13152' into
solr-13131
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit da21eae589e118c10ab8a33a75f8587911c52e9d
Merge: fc0b9fa d7cb6d6
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sun Mar 3 21:06:24 2019 -0500
Merge branch 'moshebla-SOLR-13152' into solr-13131
.../cloud/api/collections/CategoryRoutedAlias.java | 52 ++++++++++-----
.../cloud/api/collections/TimeRoutedAlias.java | 3 +-
.../CategoryRoutedAliasUpdateProcessorTest.java | 73 +++++++++-------------
.../solrj/request/CollectionAdminRequest.java | 17 ++---
.../resources/apispec/collections.Commands.json | 8 +++
5 files changed, 84 insertions(+), 69 deletions(-)
[lucene-solr] 06/10: SOLR-13151 fix mustMatch to match on value not
the final collection name so that users can choose to fail on non-english
text if desired.
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gus pushed a commit to branch solr-13131
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
commit 64d1601d4b4fae0c21b9a3c71c7be2ad38d2110f
Author: Gus Heck <gu...@apache.org>
AuthorDate: Sun Mar 3 15:26:40 2019 -0500
SOLR-13151 fix mustMatch to match on value not the final collection
name so that users can choose to fail on non-english text if desired.
---
.../apache/solr/cloud/api/collections/CategoryRoutedAlias.java | 4 ++--
.../processor/CategoryRoutedAliasUpdateProcessorTest.java | 10 +++++-----
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java b/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
index 5089eeb..7d73149 100644
--- a/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
+++ b/solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
@@ -146,8 +146,8 @@ public class CategoryRoutedAlias implements RoutedAlias {
"__CRA__");
}
- if (mustMatch != null && !mustMatch.matcher(candidateCollectionName).matches()) {
- throw new SolrException(BAD_REQUEST, "Collection name " + candidateCollectionName
+ if (mustMatch != null && !mustMatch.matcher(dataValue).matches()) {
+ throw new SolrException(BAD_REQUEST, "Route value " + dataValue
+ " does not match " + ROUTER_MUST_MATCH + ": " + mustMatch);
}
diff --git a/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java b/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
index 813a326..77ecb22 100644
--- a/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
+++ b/solr/core/src/test/org/apache/solr/update/processor/CategoryRoutedAliasUpdateProcessorTest.java
@@ -223,16 +223,16 @@ public class CategoryRoutedAliasUpdateProcessorTest extends RoutedAliasUpdatePro
public void testMustMatch() throws Exception {
String configName = getSaferTestName();
createConfigSet(configName);
- final String mustMatchRegex = ".+_solr";
+ final String mustMatchRegex = "HHS\\s.+_solr";
final int maxCardinality = Integer.MAX_VALUE; // max cardinality for current test
// Start with one collection manually created (and use higher numShards & replicas than we'll use for others)
// This tests we may pre-create the collection and it's acceptable.
- final String colVogon = getAlias() + "__CRA__" + noSpaces(SHIPS[0]) + "_solr";
+ final String colVogon = getAlias() + "__CRA__" + noSpaces("HHS "+ SHIPS[0]) + "_solr";
// we expect changes ensuring a legal collection name.
- final String colHoG = getAlias() + "__CRA__" + noSpaces(SHIPS[1]) + "_solr";
+ final String colHoG = getAlias() + "__CRA__" + noSpaces("HHS "+ SHIPS[1]) + "_solr";
List<String> retrievedConfigSetNames = new ConfigSetAdminRequest.List().process(solrClient).getConfigSets();
List<String> expectedConfigSetNames = Arrays.asList("_default", configName);
@@ -249,13 +249,13 @@ public class CategoryRoutedAliasUpdateProcessorTest extends RoutedAliasUpdatePro
.process(solrClient);
// now we index a document
- addDocsAndCommit(true, newDoc(SHIPS[0] + "_solr"));
+ addDocsAndCommit(true, newDoc("HHS " + SHIPS[0] + "_solr"));
//assertDocRoutedToCol(lastDocId, col23rd);
String uninitialized = getAlias() + "__CRA__" + CategoryRoutedAlias.UNINITIALIZED;
assertInvariants(colVogon, uninitialized);
- addDocsAndCommit(true, newDoc(SHIPS[1] + "_solr"));
+ addDocsAndCommit(true, newDoc("HHS "+ SHIPS[1] + "_solr"));
assertInvariants(colVogon, colHoG);