You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ab...@apache.org on 2017/10/13 10:57:40 UTC
[1/8] lucene-solr:jira/solr-11285: Ref Guide: fix anchor refs that
failed the build
Repository: lucene-solr
Updated Branches:
refs/heads/jira/solr-11285 ab4748a19 -> ec0214bb6
Ref Guide: fix anchor refs that failed the build
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a28474ec
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a28474ec
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a28474ec
Branch: refs/heads/jira/solr-11285
Commit: a28474ec6f3d4a0f6e527967be8edc7cca98ccdd
Parents: d8c1321
Author: Cassandra Targett <ct...@apache.org>
Authored: Thu Oct 12 06:34:54 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Thu Oct 12 06:34:54 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a28474ec/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
index 6d0dcb9..c14965a 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
@@ -85,13 +85,13 @@ Now that we have an idea about how cluster management operations use policy and
The `autoAddReplicas` parameter passed to the create collection API in the quickstart section automatically creates a trigger that watches for a node going away. When the trigger fires, it computes and executes a plan to move all replicas hosted by the lost node to new nodes in the cluster. The target nodes are chosen based on the policy and preferences.
-You can learn more about Triggers in the section <<solrcloud-autoscaling-triggers.adoc,AutoScaling Triggers>>.
+You can learn more about Triggers in the section <<solrcloud-autoscaling-triggers.adoc#solrcloud-autoscaling-triggers,AutoScaling Triggers>>.
== Listeners
An AutoScaling *Listener* can be attached to a trigger. Solr calls the listener each time the trigger fires as well as before and after the actions performed by the trigger. Listeners are useful as a call back mechanism to perform tasks such as logging or informing external systems about events. For example, a listener is automatically added by Solr to each Trigger to log details of the trigger fire and actions to the `.system` collection.
-You can learn more about Listeners in the section <<solrcloud-autoscaling-policy-preferences.adoc,AutoScaling Listeners>>.
+You can learn more about Listeners in the section <<solrcloud-autoscaling-policy-preferences.adoc#solrcloud-autoscaling-policy-preferences,AutoScaling Listeners>>.
== Autoscaling APIs
[6/8] lucene-solr:jira/solr-11285: SOLR-11467: Added extra logging to
debug jenkins CDCR failures
Posted by ab...@apache.org.
SOLR-11467: Added extra logging to debug jenkins CDCR failures
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6fecf849
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/6fecf849
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/6fecf849
Branch: refs/heads/jira/solr-11285
Commit: 6fecf849b63be6e75972e9cfd97cbf2ee7962447
Parents: 4bc7fd2
Author: Varun Thacker <va...@apache.org>
Authored: Thu Oct 12 20:45:41 2017 -0700
Committer: Varun Thacker <va...@apache.org>
Committed: Thu Oct 12 20:45:41 2017 -0700
----------------------------------------------------------------------
.../org/apache/solr/handler/CdcrReplicator.java | 19 ++++++++++++++++
.../apache/solr/handler/CdcrRequestHandler.java | 11 ++++++++++
.../update/processor/CdcrUpdateProcessor.java | 3 +++
.../apache/solr/cloud/CdcrBootstrapTest.java | 23 ++++++++++++++++++--
4 files changed, 54 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6fecf849/solr/core/src/java/org/apache/solr/handler/CdcrReplicator.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/CdcrReplicator.java b/solr/core/src/java/org/apache/solr/handler/CdcrReplicator.java
index a151a43..c40a2e5 100644
--- a/solr/core/src/java/org/apache/solr/handler/CdcrReplicator.java
+++ b/solr/core/src/java/org/apache/solr/handler/CdcrReplicator.java
@@ -21,6 +21,7 @@ import java.lang.invoke.MethodHandles;
import java.nio.charset.Charset;
import java.util.List;
+import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.request.UpdateRequest;
@@ -62,6 +63,9 @@ public class CdcrReplicator implements Runnable {
}
try {
+ // check what's there in target before sending further
+ printDocsFromTargetForAdditionalDebugging();
+
// create update request
UpdateRequest req = new UpdateRequest();
// Add the param to indicate the {@link CdcrUpdateProcessor} to keep the provided version number
@@ -77,6 +81,10 @@ public class CdcrReplicator implements Runnable {
Object o = subReader.next();
if (o == null) break; // we have reached the end of the update logs, we should close the batch
+ //additional logging
+ List entry = (List) o;
+ log.info("cdcr: current tlog entry in replicator: " + entry);
+
if (isDelete(o)) {
/*
@@ -140,6 +148,17 @@ public class CdcrReplicator implements Runnable {
state.resetConsecutiveErrors();
}
+
+ private void printDocsFromTargetForAdditionalDebugging() throws IOException, SolrServerException, CdcrReplicatorException {
+ try {
+ log.info("cdcr: docs in target: "+ state + " in replicator: "
+ + state.getClient().query(new SolrQuery("*:*")).getResults());
+ } catch (Exception e) {
+ log.warn("cdcr: exception while querying to target: " + e);
+ }
+ }
+
+
private boolean isDelete(Object o) {
List entry = (List) o;
int operationAndFlags = (Integer) entry.get(0);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6fecf849/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java b/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java
index 430237e..7f57eee 100644
--- a/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java
@@ -416,6 +416,8 @@ public class CdcrRequestHandler extends RequestHandlerBase implements SolrCoreAw
for (final Future<Long> future : parallelExecutor.invokeAll(callables)) {
long version = future.get();
+ //additional logging
+ log.info("cdcr: shardcheckpoint: versions on target: " + version);
if (version < checkpoint) { // we must take the lowest checkpoint from all the shards
checkpoint = version;
}
@@ -629,16 +631,23 @@ public class CdcrRequestHandler extends RequestHandlerBase implements SolrCoreAw
}
CountDownLatch latch = new CountDownLatch(1); // latch to make sure BOOTSTRAP_STATUS gives correct response
+ //additional logging
+ log.info("cdcr: bootstrap executing for collection: " + collectionName + " and shard: " + shard);
+
Runnable runnable = () -> {
Lock recoveryLock = req.getCore().getSolrCoreState().getRecoveryLock();
boolean locked = recoveryLock.tryLock();
SolrCoreState coreState = core.getSolrCoreState();
try {
if (!locked) {
+ //additional logging
+ log.info("cdcr: couldn't acquire lock for bootstrap, issue CANCEL");
handleCancelBootstrap(req, rsp);
} else if (leaderStateManager.amILeader()) {
coreState.setCdcrBootstrapRunning(true);
latch.countDown(); // free the latch as current bootstrap is executing
+ //additional logging
+ log.info("cdcr: acquire lock for bootstrap, latch removed");
//running.set(true);
String masterUrl = req.getParams().get(ReplicationHandler.MASTER_URL);
BootstrapCallable bootstrapCallable = new BootstrapCallable(masterUrl, core);
@@ -665,6 +674,8 @@ public class CdcrRequestHandler extends RequestHandlerBase implements SolrCoreAw
latch.countDown(); // free the latch as current bootstrap is executing
}
}
+ //additional logging
+ log.info("cdcr: bootstrap status after action: " + coreState.getCdcrBootstrapRunning());
};
try {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6fecf849/solr/core/src/java/org/apache/solr/update/processor/CdcrUpdateProcessor.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/processor/CdcrUpdateProcessor.java b/solr/core/src/java/org/apache/solr/update/processor/CdcrUpdateProcessor.java
index ee45467..9cb7a8a 100644
--- a/solr/core/src/java/org/apache/solr/update/processor/CdcrUpdateProcessor.java
+++ b/solr/core/src/java/org/apache/solr/update/processor/CdcrUpdateProcessor.java
@@ -67,6 +67,9 @@ public class CdcrUpdateProcessor extends DistributedUpdateProcessor {
cmd.setFlags(cmd.getFlags() & ~UpdateCommand.PEER_SYNC);
}
+ //additional logging
+ log.info("cdcr: version received at target: " + cmd.getVersion());
+
return result;
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6fecf849/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java b/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java
index 6959bd8..529b3d0 100644
--- a/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/CdcrBootstrapTest.java
@@ -269,13 +269,16 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
req.add(doc);
}
req.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
- System.out.println("Adding " + docs + " docs with commit=true, numDocs=" + numDocs);
+ log.info("Adding " + docs + " docs with commit=true, numDocs=" + numDocs);
req.process(sourceSolrClient);
}
QueryResponse response = sourceSolrClient.query(new SolrQuery("*:*"));
assertEquals("", numDocs, response.getResults().getNumFound());
+ //additional logging
+ log.info("cdcr: numDocs after initial indexing in source: " + numDocs);
+
// setup the target cluster
target.uploadConfigSet(configset("cdcr-target"), "cdcr-target");
CollectionAdminRequest.createCollection("cdcr-target", "cdcr-target", 1, 1)
@@ -284,8 +287,16 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
targetSolrClient.setDefaultCollection("cdcr-target");
Thread.sleep(1000);
+ //additional logging
+ log.info("cdcr: numDocs in target-1 : " + targetSolrClient.query(new SolrQuery("*:*")).getResults().getNumFound());
+
cdcrStart(targetSolrClient);
cdcrStart(sourceSolrClient);
+
+ //additional logging
+ log.info("cdcr: cdcr is enabled on source and target both, bootstrap should be triggered");
+ log.info("cdcr: current numDocs in target-2: " + targetSolrClient.query(new SolrQuery("*:*")).getResults().getNumFound());
+
int c = 0;
for (int k = 0; k < docs; k++) {
UpdateRequest req = new UpdateRequest();
@@ -296,13 +307,20 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
req.add(doc);
}
req.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
- System.out.println("Adding " + docs + " docs with commit=true, numDocs=" + numDocs);
+ log.info("Adding " + docs + " docs with commit=true, numDocs=" + numDocs);
req.process(sourceSolrClient);
}
+ //additional logging
+ log.info("cdcr: current numDocs in target-3: " + targetSolrClient.query(new SolrQuery("*:*")).getResults().getNumFound());
+
response = sourceSolrClient.query(new SolrQuery("*:*"));
assertEquals("", numDocs, response.getResults().getNumFound());
+ //additional logging
+ log.info("cdcr: numDocs after second indexing in source: " + numDocs);
+ log.info("cdcr: current numDocs in target-4: " + targetSolrClient.query(new SolrQuery("*:*")).getResults().getNumFound());
+
response = getCdcrQueue(sourceSolrClient);
System.out.println("Cdcr queue response: " + response.getResponse());
long foundDocs = waitForTargetToSync(numDocs, targetSolrClient);
@@ -323,6 +341,7 @@ public class CdcrBootstrapTest extends SolrTestCaseJ4 {
try {
targetSolrClient.commit();
response = targetSolrClient.query(new SolrQuery("*:*"));
+ log.info("cdcr: numDocs in waitForTargetToSync: " + response.getResults().getNumFound());
if (response.getResults().getNumFound() == numDocs) {
break;
}
[8/8] lucene-solr:jira/solr-11285: Merge branch 'master' into
jira/solr-11285
Posted by ab...@apache.org.
Merge branch 'master' into jira/solr-11285
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ec0214bb
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ec0214bb
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ec0214bb
Branch: refs/heads/jira/solr-11285
Commit: ec0214bb660b1b9a6e725965d0446dea9e337879
Parents: ab4748a 8a7d04a
Author: Andrzej Bialecki <ab...@apache.org>
Authored: Fri Oct 13 12:57:28 2017 +0200
Committer: Andrzej Bialecki <ab...@apache.org>
Committed: Fri Oct 13 12:57:28 2017 +0200
----------------------------------------------------------------------
solr/CHANGES.txt | 5 +-
.../src/java/org/apache/solr/cloud/Assign.java | 2 +-
.../java/org/apache/solr/cloud/Overseer.java | 10 +-
.../solr/cloud/OverseerNodePrioritizer.java | 1 -
.../org/apache/solr/cloud/RecoveryStrategy.java | 1 -
.../org/apache/solr/cloud/ZkController.java | 1 -
.../cloud/autoscaling/AutoScalingHandler.java | 1 -
.../cloud/autoscaling/ExecutePlanAction.java | 2 +-
.../autoscaling/ZkDistributedQueueFactory.java | 16 +++
.../solr/cloud/overseer/ZkStateWriter.java | 96 +++++--------
.../org/apache/solr/handler/CdcrReplicator.java | 19 +++
.../apache/solr/handler/CdcrRequestHandler.java | 11 ++
.../update/processor/CdcrUpdateProcessor.java | 3 +
.../apache/solr/cloud/CdcrBootstrapTest.java | 23 ++-
.../cloud/LeaderElectionContextKeyTest.java | 3 +
.../org/apache/solr/cloud/OverseerTest.java | 59 ++++++++
.../cloud/TestRandomRequestDistribution.java | 1 -
.../autoscaling/ExecutePlanActionTest.java | 46 +++---
.../solr/cloud/overseer/ZkStateReaderTest.java | 9 +-
.../solr/cloud/overseer/ZkStateWriterTest.java | 139 +++++++++++--------
solr/solr-ref-guide/src/meta-docs/jekyll.adoc | 11 +-
.../DelegatingClusterStateProvider.java | 2 -
.../client/solrj/cloud/autoscaling/Policy.java | 1 -
.../solr/client/solrj/cloud/package-info.java | 23 +++
.../client/solrj/impl/ClusterStateProvider.java | 2 -
.../solrj/impl/CloudSolrClientCacheTest.java | 2 -
26 files changed, 314 insertions(+), 175 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/Assign.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/Assign.java
index 5f861f0,f22312f..36663e4
--- a/solr/core/src/java/org/apache/solr/cloud/Assign.java
+++ b/solr/core/src/java/org/apache/solr/cloud/Assign.java
@@@ -359,7 -356,6 +359,7 @@@ public class Assign
List l = (List) coll.get(DocCollection.RULE);
List<ReplicaPosition> replicaPositions = null;
if (l != null) {
- // TODO nocommit: make it so that this method doesn't require access to CC
++ // TODO: make it so that this method doesn't require access to CC
replicaPositions = getNodesViaRules(clusterState, shard, nrtReplicas, cc, coll, createNodeList, l);
}
String policyName = coll.getStr(POLICY);
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/Overseer.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/Overseer.java
index d1be7b2,74a236d..f0968d8
--- a/solr/core/src/java/org/apache/solr/cloud/Overseer.java
+++ b/solr/core/src/java/org/apache/solr/cloud/Overseer.java
@@@ -19,14 -19,17 +19,13 @@@ package org.apache.solr.cloud
import java.io.Closeable;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
-import java.util.ArrayList;
import java.util.Collections;
-import java.util.LinkedList;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.atomic.AtomicInteger;
import com.codahale.metrics.Timer;
- import org.apache.solr.client.solrj.cloud.DistributedQueue;
-import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.cloud.autoscaling.SolrCloudManager;
+import org.apache.solr.client.solrj.impl.ClusterStateProvider;
import org.apache.solr.cloud.autoscaling.OverseerTriggerThread;
import org.apache.solr.cloud.overseer.ClusterStateMutator;
import org.apache.solr.cloud.overseer.CollectionMutator;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/OverseerNodePrioritizer.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/OverseerNodePrioritizer.java
index f0e3676,7db1315..34ee041
--- a/solr/core/src/java/org/apache/solr/cloud/OverseerNodePrioritizer.java
+++ b/solr/core/src/java/org/apache/solr/cloud/OverseerNodePrioritizer.java
@@@ -31,7 -31,7 +31,6 @@@ import org.apache.solr.common.util.Util
import org.apache.solr.handler.component.ShardHandler;
import org.apache.solr.handler.component.ShardHandlerFactory;
import org.apache.solr.handler.component.ShardRequest;
--import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
index 6aa9d3d,73e6142..53e3743
--- a/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
+++ b/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
@@@ -64,7 -64,7 +64,6 @@@ import org.apache.solr.update.processor
import org.apache.solr.util.RefCounted;
import org.apache.solr.util.SolrPluginUtils;
import org.apache.solr.util.plugin.NamedListInitializedPlugin;
--import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/ZkController.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/ZkController.java
index cc2f215,e55c966..add4908c
--- a/solr/core/src/java/org/apache/solr/cloud/ZkController.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
@@@ -51,16 -51,10 +51,15 @@@ import java.util.concurrent.atomic.Atom
import com.google.common.base.Strings;
import org.apache.commons.lang.StringUtils;
- import org.apache.solr.client.solrj.cloud.DistributedQueue;
+import org.apache.solr.client.solrj.cloud.autoscaling.SolrCloudManager;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.HttpSolrClient.Builder;
+import org.apache.solr.client.solrj.impl.SolrClientCloudManager;
+import org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider;
import org.apache.solr.client.solrj.request.CoreAdminRequest.WaitForState;
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventType;
+import org.apache.solr.cloud.autoscaling.ZkDistributedQueueFactory;
import org.apache.solr.cloud.overseer.OverseerAction;
import org.apache.solr.cloud.overseer.SliceMutator;
import org.apache.solr.common.SolrException;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScalingHandler.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScalingHandler.java
index e4d6b8a,6ad84b8..f29d498
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScalingHandler.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScalingHandler.java
@@@ -46,10 -45,9 +46,9 @@@ import org.apache.solr.client.solrj.clo
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventProcessorStage;
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventType;
import org.apache.solr.client.solrj.impl.CloudSolrClient;
-import org.apache.solr.client.solrj.impl.SolrClientDataProvider;
+import org.apache.solr.client.solrj.impl.SolrClientCloudManager;
import org.apache.solr.common.MapWriter;
import org.apache.solr.common.SolrException;
- import org.apache.solr.common.cloud.SolrZkClient;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.params.AutoScalingParams;
import org.apache.solr.common.params.CollectionAdminParams;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/autoscaling/ExecutePlanAction.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/autoscaling/ExecutePlanAction.java
index ccc18a9,8221ff8..8b3b637
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ExecutePlanAction.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ExecutePlanAction.java
@@@ -70,16 -76,10 +70,16 @@@ public class ExecutePlanAction extends
if (operation instanceof CollectionAdminRequest.AsyncCollectionAdminRequest) {
CollectionAdminRequest.AsyncCollectionAdminRequest req = (CollectionAdminRequest.AsyncCollectionAdminRequest) operation;
String asyncId = event.getSource() + '/' + event.getId() + '/' + counter;
- String znode = saveAsyncId(event, context, asyncId);
+ String znode = saveAsyncId(dataProvider.getDistribStateManager(), event, asyncId);
log.debug("Saved requestId: {} in znode: {}", asyncId, znode);
- // nocommit - make sure the async logic below is correct when using dataProvider API !!!
- asyncId = req.processAsync(asyncId, cloudSolrClient);
- CollectionAdminRequest.RequestStatusResponse statusResponse = waitForTaskToFinish(cloudSolrClient, asyncId,
++ // TODO: find a better way of using async calls using dataProvider API !!!
+ req.setAsyncId(asyncId);
+ SolrResponse asyncResponse = dataProvider.request(req);
+ if (asyncResponse.getResponse().get("error") != null) {
+ throw new IOException("" + asyncResponse.getResponse().get("error"));
+ }
+ asyncId = (String)asyncResponse.getResponse().get("requestid");
+ CollectionAdminRequest.RequestStatusResponse statusResponse = waitForTaskToFinish(dataProvider, asyncId,
DEFAULT_TASK_TIMEOUT_SECONDS, TimeUnit.SECONDS);
if (statusResponse != null) {
RequestStatusState state = statusResponse.getRequestStatus();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/autoscaling/ZkDistributedQueueFactory.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/autoscaling/ZkDistributedQueueFactory.java
index a6cb945,0000000..0b1c5f8
mode 100644,000000..100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ZkDistributedQueueFactory.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ZkDistributedQueueFactory.java
@@@ -1,28 -1,0 +1,44 @@@
++/*
++ * Licensed to the Apache Software Foundation (ASF) under one or more
++ * contributor license agreements. See the NOTICE file distributed with
++ * this work for additional information regarding copyright ownership.
++ * The ASF licenses this file to You under the Apache License, Version 2.0
++ * (the "License"); you may not use this file except in compliance with
++ * the License. You may obtain a copy of the License at
++ *
++ * http://www.apache.org/licenses/LICENSE-2.0
++ *
++ * Unless required by applicable law or agreed to in writing, software
++ * distributed under the License is distributed on an "AS IS" BASIS,
++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++ * See the License for the specific language governing permissions and
++ * limitations under the License.
++ */
+package org.apache.solr.cloud.autoscaling;
+
+import java.io.IOException;
+
+import org.apache.solr.client.solrj.cloud.DistributedQueue;
+import org.apache.solr.client.solrj.cloud.autoscaling.SolrCloudManager;
+import org.apache.solr.cloud.ZkDistributedQueue;
+import org.apache.solr.common.cloud.SolrZkClient;
+
+/**
+ *
+ */
+public class ZkDistributedQueueFactory implements SolrCloudManager.DistributedQueueFactory {
+ private final SolrZkClient zkClient;
+
+ public ZkDistributedQueueFactory(SolrZkClient zkClient) {
+ this.zkClient = zkClient;
+ }
+ @Override
+ public DistributedQueue makeQueue(String path) throws IOException {
+ return new ZkDistributedQueue(zkClient, path);
+ }
+
+ @Override
+ public void removeQueue(String path) throws IOException {
+
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
----------------------------------------------------------------------
diff --cc solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
index 0876f3e,9c810e8..068a808
--- a/solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
+++ b/solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
@@@ -61,9 -61,10 +62,10 @@@ public class ZkStateWriter
public static ZkWriteCommand NO_OP = ZkWriteCommand.noop();
protected final ZkStateReader reader;
- protected final Overseer.Stats stats;
+ protected final Stats stats;
protected Map<String, DocCollection> updates = new HashMap<>();
+ private int numUpdates = 0;
protected ClusterState clusterState = null;
protected boolean isClusterStateModified = false;
protected long lastUpdatedTime = 0;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
----------------------------------------------------------------------
diff --cc solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
index 1fb6201,97e8a17..be8da4d
--- a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
@@@ -37,13 -37,9 +37,15 @@@ import com.codahale.metrics.Snapshot
import com.codahale.metrics.Timer;
import org.apache.lucene.util.LuceneTestCase.Slow;
import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.client.solrj.cloud.DistributedQueue;
+import org.apache.solr.client.solrj.cloud.autoscaling.SolrCloudManager;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.SolrClientCloudManager;
+import org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider;
+import org.apache.solr.cloud.autoscaling.ZkDistributedQueueFactory;
+ import org.apache.solr.cloud.overseer.NodeMutator;
import org.apache.solr.cloud.overseer.OverseerAction;
+ import org.apache.solr.cloud.overseer.ZkWriteCommand;
import org.apache.solr.common.cloud.ClusterState;
import org.apache.solr.common.cloud.DocCollection;
import org.apache.solr.common.cloud.Replica;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/test/org/apache/solr/cloud/autoscaling/ExecutePlanActionTest.java
----------------------------------------------------------------------
diff --cc solr/core/src/test/org/apache/solr/cloud/autoscaling/ExecutePlanActionTest.java
index 34c9bdb,e92d8a0..74a1a82
--- a/solr/core/src/test/org/apache/solr/cloud/autoscaling/ExecutePlanActionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/autoscaling/ExecutePlanActionTest.java
@@@ -113,30 -118,28 +115,28 @@@ public class ExecutePlanActionTest exte
AtomicBoolean znodeCreated = new AtomicBoolean(false);
CollectionAdminRequest.AsyncCollectionAdminRequest moveReplica = new CollectionAdminRequest.MoveReplica(collectionName, replicas.get(0).getName(), survivor.getNodeName());
- CollectionAdminRequest.AsyncCollectionAdminRequest mockRequest = new CollectionAdminRequest.AsyncCollectionAdminRequest(CollectionParams.CollectionAction.LIST) {
+ CollectionAdminRequest.AsyncCollectionAdminRequest mockRequest = new CollectionAdminRequest.AsyncCollectionAdminRequest(CollectionParams.CollectionAction.OVERSEERSTATUS) {
- // nocommit: this doesn't work because async requests are now processed differently in ExecutePlanAction
- //
- // @Override
- // public String processAsync(String asyncId, SolrClient client) throws IOException, SolrServerException {
- // String parentPath = ZkStateReader.SOLR_AUTOSCALING_TRIGGER_STATE_PATH + "/xyz/execute_plan";
- // try {
- // if (zkClient().exists(parentPath, true)) {
- // java.util.List<String> children = zkClient().getChildren(parentPath, null, true);
- // if (!children.isEmpty()) {
- // String child = children.get(0);
- // byte[] data = zkClient().getData(parentPath + "/" + child, null, null, true);
- // Map m = (Map) Utils.fromJSON(data);
- // if (m.containsKey("requestid")) {
- // znodeCreated.set(m.get("requestid").equals(asyncId));
- // }
- // }
- // }
- // } catch (Exception e) {
- // throw new RuntimeException(e);
- // }
- // super.processAsync(asyncId, client);
- // return asyncId;
- // }
+ @Override
- public String processAsync(String asyncId, SolrClient client) throws IOException, SolrServerException {
++ public void setAsyncId(String asyncId) {
++ super.setAsyncId(asyncId);
+ String parentPath = ZkStateReader.SOLR_AUTOSCALING_TRIGGER_STATE_PATH + "/xyz/execute_plan";
+ try {
+ if (zkClient().exists(parentPath, true)) {
+ java.util.List<String> children = zkClient().getChildren(parentPath, null, true);
+ if (!children.isEmpty()) {
+ String child = children.get(0);
+ byte[] data = zkClient().getData(parentPath + "/" + child, null, null, true);
+ Map m = (Map) Utils.fromJSON(data);
+ if (m.containsKey("requestid")) {
+ znodeCreated.set(m.get("requestid").equals(asyncId));
+ }
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
- super.processAsync(asyncId, client);
- return asyncId;
++
+ }
};
List<CollectionAdminRequest.AsyncCollectionAdminRequest> operations = Lists.asList(moveReplica, new CollectionAdminRequest.AsyncCollectionAdminRequest[]{mockRequest});
NodeLostTrigger.NodeLostEvent nodeLostEvent = new NodeLostTrigger.NodeLostEvent(TriggerEventType.NODELOST,
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateReaderTest.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
----------------------------------------------------------------------
diff --cc solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
index d8e6a9a,436d72e..7265ce7
--- a/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
@@@ -23,8 -25,8 +25,9 @@@ import java.util.concurrent.atomic.Atom
import org.apache.lucene.util.IOUtils;
import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.cloud.AbstractZkTestCase;
+ import org.apache.solr.cloud.Overseer;
import org.apache.solr.cloud.OverseerTest;
+import org.apache.solr.cloud.Stats;
import org.apache.solr.cloud.ZkController;
import org.apache.solr.cloud.ZkTestServer;
import org.apache.solr.common.cloud.ClusterState;
@@@ -60,64 -87,52 +88,52 @@@ public class ZkStateWriterTest extends
try (ZkStateReader reader = new ZkStateReader(zkClient)) {
reader.createClusterStateWatchersAndUpdate();
- ZkStateWriter writer = new ZkStateWriter(reader, new Stats());
-
- assertFalse("Deletes can always be batched", writer.maybeFlushBefore(new ZkWriteCommand("xyz", null)));
- assertFalse("Deletes can always be batched", writer.maybeFlushAfter(new ZkWriteCommand("xyz", null)));
-
zkClient.makePath(ZkStateReader.COLLECTIONS_ZKNODE + "/c1", true);
zkClient.makePath(ZkStateReader.COLLECTIONS_ZKNODE + "/c2", true);
+ zkClient.makePath(ZkStateReader.COLLECTIONS_ZKNODE + "/c3", true);
- // create new collection with stateFormat = 2
ZkWriteCommand c1 = new ZkWriteCommand("c1",
new DocCollection("c1", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c1"));
- assertFalse("First requests can always be batched", writer.maybeFlushBefore(c1));
-
- ClusterState clusterState = writer.enqueueUpdate(reader.getClusterState(), c1, null);
-
ZkWriteCommand c2 = new ZkWriteCommand("c2",
new DocCollection("c2", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c2"));
- assertFalse("Different (new) collection create can be batched together with another create", writer.maybeFlushBefore(c2));
-
- // simulate three state changes on same collection, all should be batched together before
- assertFalse(writer.maybeFlushBefore(c1));
- assertFalse(writer.maybeFlushBefore(c1));
- assertFalse(writer.maybeFlushBefore(c1));
- // and after too
- assertFalse(writer.maybeFlushAfter(c1));
- assertFalse(writer.maybeFlushAfter(c1));
- assertFalse(writer.maybeFlushAfter(c1));
-
- // simulate three state changes on two different collections with stateFormat=2, all should be batched
- assertFalse(writer.maybeFlushBefore(c1));
- // flushAfter has to be called as it updates the internal batching related info
- assertFalse(writer.maybeFlushAfter(c1));
- assertFalse(writer.maybeFlushBefore(c2));
- assertFalse(writer.maybeFlushAfter(c2));
- assertFalse(writer.maybeFlushBefore(c1));
- assertFalse(writer.maybeFlushAfter(c1));
-
- // create a collection in stateFormat = 1 i.e. inside the main cluster state
ZkWriteCommand c3 = new ZkWriteCommand("c3",
- new DocCollection("c3", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.CLUSTER_STATE));
- clusterState = writer.enqueueUpdate(clusterState, c3, null);
+ new DocCollection("c3", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c3"));
- ZkStateWriter writer = new ZkStateWriter(reader, new Overseer.Stats());
++ ZkStateWriter writer = new ZkStateWriter(reader, new Stats());
- // simulate three state changes in c3, all should be batched
- for (int i = 0; i < 3; i++) {
- assertFalse(writer.maybeFlushBefore(c3));
- assertFalse(writer.maybeFlushAfter(c3));
+ // First write is flushed immediately
+ ClusterState clusterState = writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c1), null);
+ clusterState = writer.enqueueUpdate(clusterState, Collections.singletonList(c1), FAIL_ON_WRITE);
+ clusterState = writer.enqueueUpdate(clusterState, Collections.singletonList(c2), FAIL_ON_WRITE);
+
+ Thread.sleep(Overseer.STATE_UPDATE_DELAY + 100);
+ AtomicBoolean didWrite = new AtomicBoolean(false);
+ clusterState = writer.enqueueUpdate(clusterState, Collections.singletonList(c3), new ZkStateWriter.ZkWriteCallback() {
+ @Override
+ public void onEnqueue() throws Exception {
+
+ }
+
+ @Override
+ public void onWrite() throws Exception {
+ didWrite.set(true);
+ }
+ });
+ assertTrue("Exceed the update delay, should be flushed", didWrite.get());
+
+ for (int i = 0; i <= Overseer.STATE_UPDATE_BATCH_SIZE; i++) {
+ clusterState = writer.enqueueUpdate(clusterState, Collections.singletonList(c3), new ZkStateWriter.ZkWriteCallback() {
+ @Override
+ public void onEnqueue() throws Exception {
+
+ }
+
+ @Override
+ public void onWrite() throws Exception {
+ didWrite.set(true);
+ }
+ });
}
-
- // simulate state change in c3 (stateFormat=1) interleaved with state changes from c1,c2 (stateFormat=2)
- // none should be batched together
- assertFalse(writer.maybeFlushBefore(c3));
- assertFalse(writer.maybeFlushAfter(c3));
- assertTrue("different stateFormat, should be flushed", writer.maybeFlushBefore(c1));
- assertFalse(writer.maybeFlushAfter(c1));
- assertTrue("different stateFormat, should be flushed", writer.maybeFlushBefore(c3));
- assertFalse(writer.maybeFlushAfter(c3));
- assertTrue("different stateFormat, should be flushed", writer.maybeFlushBefore(c2));
- assertFalse(writer.maybeFlushAfter(c2));
+ assertTrue("Exceed the update batch size, should be flushed", didWrite.get());
}
} finally {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/DelegatingClusterStateProvider.java
----------------------------------------------------------------------
diff --cc solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/DelegatingClusterStateProvider.java
index 10f18e3,0000000..0fecfed
mode 100644,000000..100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/DelegatingClusterStateProvider.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/DelegatingClusterStateProvider.java
@@@ -1,115 -1,0 +1,113 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.cloud.autoscaling;
+
+import java.io.IOException;
- import java.util.Collection;
+import java.util.Collections;
- import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.solr.client.solrj.impl.ClusterStateProvider;
+import org.apache.solr.common.cloud.ClusterState;
+
+/**
+ *
+ */
+public class DelegatingClusterStateProvider implements ClusterStateProvider {
+ protected ClusterStateProvider delegate;
+
+ public DelegatingClusterStateProvider(ClusterStateProvider delegate) {
+ this.delegate = delegate;
+ }
+
+ @Override
+ public ClusterState.CollectionRef getState(String collection) {
+ if (delegate != null) {
+ return delegate.getState(collection);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public Set<String> getLiveNodes() {
+ if (delegate != null) {
+ return delegate.getLiveNodes();
+ } else {
+ return Collections.emptySet();
+ }
+ }
+
+ @Override
+ public String getAlias(String alias) {
+ if (delegate != null) {
+ return delegate.getAlias(alias);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public String getCollectionName(String name) {
+ if (delegate != null) {
+ return delegate.getCollectionName(name);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public ClusterState getClusterState() throws IOException {
+ if (delegate != null) {
+ return delegate.getClusterState();
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public Map<String, Object> getClusterProperties() {
+ if (delegate != null) {
+ return delegate.getClusterProperties();
+ } else {
+ return Collections.emptyMap();
+ }
+ }
+
+ @Override
+ public String getPolicyNameByCollection(String coll) {
+ if (delegate != null) {
+ return delegate.getPolicyNameByCollection(coll);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public void connect() {
+ if (delegate != null) {
+ delegate.connect();
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (delegate != null) {
+ delegate.close();
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
----------------------------------------------------------------------
diff --cc solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
index 67d65fb,78961d6..fc22cd7
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
@@@ -36,10 -36,8 +36,9 @@@ import java.util.function.Consumer
import java.util.function.Supplier;
import java.util.stream.Collectors;
- import org.apache.solr.client.solrj.SolrClient;
import org.apache.solr.client.solrj.SolrRequest;
import org.apache.solr.client.solrj.cloud.autoscaling.Clause.Violation;
+import org.apache.solr.client.solrj.impl.ClusterStateProvider;
import org.apache.solr.common.IteratorWriter;
import org.apache.solr.common.MapWriter;
import org.apache.solr.common.cloud.Replica;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/package-info.java
----------------------------------------------------------------------
diff --cc solr/solrj/src/java/org/apache/solr/client/solrj/cloud/package-info.java
index 0000000,0000000..0ce8a66
new file mode 100644
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/package-info.java
@@@ -1,0 -1,0 +1,23 @@@
++/*
++ * Licensed to the Apache Software Foundation (ASF) under one or more
++ * contributor license agreements. See the NOTICE file distributed with
++ * this work for additional information regarding copyright ownership.
++ * The ASF licenses this file to You under the Apache License, Version 2.0
++ * (the "License"); you may not use this file except in compliance with
++ * the License. You may obtain a copy of the License at
++ *
++ * http://www.apache.org/licenses/LICENSE-2.0
++ *
++ * Unless required by applicable law or agreed to in writing, software
++ * distributed under the License is distributed on an "AS IS" BASIS,
++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++ * See the License for the specific language governing permissions and
++ * limitations under the License.
++ */
++
++/**
++ * Common classes for SolrCloud.
++ */
++
++package org.apache.solr.client.solrj.cloud;
++
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ClusterStateProvider.java
----------------------------------------------------------------------
diff --cc solr/solrj/src/java/org/apache/solr/client/solrj/impl/ClusterStateProvider.java
index 1517cf1,b913cd4..7cb5d51
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ClusterStateProvider.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ClusterStateProvider.java
@@@ -48,17 -46,9 +48,15 @@@ public interface ClusterStateProvider e
String getCollectionName(String name);
/**
- * Obtain a cluster property, or null if it doesn't exist.
+ * Obtain the current cluster state.
- * @return
- * @throws IOException
+ */
+ ClusterState getClusterState() throws IOException;
+
+ /**
+ * Obtain cluster properties.
+ * @return configured cluster properties, or an empty map, never null.
*/
- Object getClusterProperty(String propertyName);
+ Map<String, Object> getClusterProperties();
/**
* Obtain a cluster property, or the default value if it doesn't exist.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ec0214bb/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientCacheTest.java
----------------------------------------------------------------------
diff --cc solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientCacheTest.java
index 77421ed,0a83138..a3dd436
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientCacheTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/impl/CloudSolrClientCacheTest.java
@@@ -17,8 -17,8 +17,6 @@@
package org.apache.solr.client.solrj.impl;
--
--import java.io.IOException;
import java.net.ConnectException;
import java.net.SocketException;
import java.util.Collections;
[7/8] lucene-solr:jira/solr-11285: SOLR-11447: ZkStateWriter should
process commands in atomic
Posted by ab...@apache.org.
SOLR-11447: ZkStateWriter should process commands in atomic
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/8a7d04a6
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/8a7d04a6
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/8a7d04a6
Branch: refs/heads/jira/solr-11285
Commit: 8a7d04a6582e6f1509660300d5421da034593420
Parents: 6fecf84
Author: Cao Manh Dat <da...@apache.org>
Authored: Fri Oct 13 15:23:44 2017 +0700
Committer: Cao Manh Dat <da...@apache.org>
Committed: Fri Oct 13 15:23:44 2017 +0700
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +
.../java/org/apache/solr/cloud/Overseer.java | 9 +-
.../solr/cloud/overseer/ZkStateWriter.java | 96 +++++--------
.../org/apache/solr/cloud/OverseerTest.java | 59 ++++++++
.../solr/cloud/overseer/ZkStateReaderTest.java | 9 +-
.../solr/cloud/overseer/ZkStateWriterTest.java | 138 +++++++++++--------
6 files changed, 181 insertions(+), 132 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8a7d04a6/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 8c0ef45..9101440 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -50,6 +50,8 @@ Bug Fixes
* SOLR-11445: Overseer should not hang when process bad message. (Cao Manh Dat, shalin)
+* SOLR-11447: ZkStateWriter should process commands in atomic. (Cao Manh Dat, shalin)
+
================== 7.1.0 ==================
Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8a7d04a6/solr/core/src/java/org/apache/solr/cloud/Overseer.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/Overseer.java b/solr/core/src/java/org/apache/solr/cloud/Overseer.java
index 3589fd8..74a236d 100644
--- a/solr/core/src/java/org/apache/solr/cloud/Overseer.java
+++ b/solr/core/src/java/org/apache/solr/cloud/Overseer.java
@@ -65,8 +65,9 @@ import static org.apache.solr.common.params.CommonParams.ID;
public class Overseer implements Closeable {
public static final String QUEUE_OPERATION = "operation";
- public static final int STATE_UPDATE_DELAY = 2000; // delay between cloud state updates
- public static final int STATE_UPDATE_BATCH_SIZE = 10000;
+ // System properties are used in tests to make them run fast
+ public static final int STATE_UPDATE_DELAY = Integer.getInteger("solr.OverseerStateUpdateDelay", 2000); // delay between cloud state updates
+ public static final int STATE_UPDATE_BATCH_SIZE = Integer.getInteger("solr.OverseerStateUpdateBatchSize", 10000);
public static final int STATE_UPDATE_MAX_QUEUE = 20000;
public static final int NUM_RESPONSES_TO_STORE = 10000;
@@ -287,9 +288,7 @@ public class Overseer implements Closeable {
timerContext.stop();
}
if (zkWriteCommands != null) {
- for (ZkWriteCommand zkWriteCommand : zkWriteCommands) {
- clusterState = zkStateWriter.enqueueUpdate(clusterState, zkWriteCommand, callback);
- }
+ clusterState = zkStateWriter.enqueueUpdate(clusterState, zkWriteCommands, callback);
if (!enableBatching) {
clusterState = zkStateWriter.writePendingUpdates();
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8a7d04a6/solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java b/solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
index 911a9e3..9c810e8 100644
--- a/solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
+++ b/solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
@@ -18,6 +18,7 @@ package org.apache.solr.cloud.overseer;
import java.lang.invoke.MethodHandles;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
@@ -41,11 +42,11 @@ import static java.util.Collections.singletonMap;
* each of which get their own individual state.json in ZK.
*
* Updates to the cluster state are specified using the
- * {@link #enqueueUpdate(ClusterState, ZkWriteCommand, ZkWriteCallback)} method. The class buffers updates
+ * {@link #enqueueUpdate(ClusterState, List, ZkWriteCallback)} method. The class buffers updates
* to reduce the number of writes to ZK. The buffered updates are flushed during <code>enqueueUpdate</code>
* automatically if necessary. The {@link #writePendingUpdates()} can be used to force flush any pending updates.
*
- * If either {@link #enqueueUpdate(ClusterState, ZkWriteCommand, ZkWriteCallback)} or {@link #writePendingUpdates()}
+ * If either {@link #enqueueUpdate(ClusterState, List, ZkWriteCallback)} or {@link #writePendingUpdates()}
* throws a {@link org.apache.zookeeper.KeeperException.BadVersionException} then the internal buffered state of the
* class is suspect and the current instance of the class should be discarded and a new instance should be created
* and used for any future updates.
@@ -63,14 +64,11 @@ public class ZkStateWriter {
protected final Overseer.Stats stats;
protected Map<String, DocCollection> updates = new HashMap<>();
+ private int numUpdates = 0;
protected ClusterState clusterState = null;
protected boolean isClusterStateModified = false;
protected long lastUpdatedTime = 0;
- // state information which helps us batch writes
- protected int lastStateFormat = -1; // sentinel value
- protected String lastCollectionName = null;
-
/**
* Set to true if we ever get a BadVersionException so that we can disallow future operations
* with this instance
@@ -96,7 +94,7 @@ public class ZkStateWriter {
* be used to force an immediate flush of pending cluster state changes.
*
* @param prevState the cluster state information on which the given <code>cmd</code> is applied
- * @param cmd the {@link ZkWriteCommand} which specifies the change to be applied to cluster state
+ * @param cmds the list of {@link ZkWriteCommand} which specifies the change to be applied to cluster state in atomic
* @param callback a {@link org.apache.solr.cloud.overseer.ZkStateWriter.ZkWriteCallback} object to be used
* for any callbacks
* @return modified cluster state created after applying <code>cmd</code> to <code>prevState</code>. If
@@ -107,48 +105,31 @@ public class ZkStateWriter {
* in a {@link org.apache.zookeeper.KeeperException.BadVersionException} this instance becomes unusable and
* must be discarded
*/
- public ClusterState enqueueUpdate(ClusterState prevState, ZkWriteCommand cmd, ZkWriteCallback callback) throws IllegalStateException, Exception {
+ public ClusterState enqueueUpdate(ClusterState prevState, List<ZkWriteCommand> cmds, ZkWriteCallback callback) throws IllegalStateException, Exception {
if (invalidState) {
throw new IllegalStateException("ZkStateWriter has seen a tragic error, this instance can no longer be used");
}
- if (cmd == NO_OP) return prevState;
-
- if (maybeFlushBefore(cmd)) {
- // we must update the prev state to the new one
- prevState = clusterState = writePendingUpdates();
- if (callback != null) {
- callback.onWrite();
- }
- }
+ if (cmds.isEmpty()) return prevState;
+ if (isNoOps(cmds)) return prevState;
if (callback != null) {
callback.onEnqueue();
}
- /*
- We need to know if the collection has moved from stateFormat=1 to stateFormat=2 (as a result of MIGRATECLUSTERSTATE)
- */
- DocCollection previousCollection = prevState.getCollectionOrNull(cmd.name);
- boolean wasPreviouslyStateFormat1 = previousCollection != null && previousCollection.getStateFormat() == 1;
- boolean isCurrentlyStateFormat1 = cmd.collection != null && cmd.collection.getStateFormat() == 1;
-
- if (cmd.collection == null) {
- if (wasPreviouslyStateFormat1) {
+ for (ZkWriteCommand cmd : cmds) {
+ if (cmd == NO_OP) continue;
+ if (!isClusterStateModified && clusterStateGetModifiedWith(cmd, prevState)) {
isClusterStateModified = true;
}
- clusterState = prevState.copyWith(cmd.name, null);
- updates.put(cmd.name, null);
- } else {
- if (!isCurrentlyStateFormat1) {
+ prevState = prevState.copyWith(cmd.name, cmd.collection);
+ if (cmd.collection == null || cmd.collection.getStateFormat() != 1) {
updates.put(cmd.name, cmd.collection);
+ numUpdates++;
}
- if (isCurrentlyStateFormat1 || wasPreviouslyStateFormat1) {
- isClusterStateModified = true;
- }
- clusterState = prevState.copyWith(cmd.name, cmd.collection);
}
+ clusterState = prevState;
- if (maybeFlushAfter(cmd)) {
+ if (maybeFlushAfter()) {
ClusterState state = writePendingUpdates();
if (callback != null) {
callback.onWrite();
@@ -159,35 +140,33 @@ public class ZkStateWriter {
return clusterState;
}
- /**
- * Logic to decide a flush before processing a ZkWriteCommand
- *
- * @param cmd the ZkWriteCommand instance
- * @return true if a flush is required, false otherwise
- */
- protected boolean maybeFlushBefore(ZkWriteCommand cmd) {
- if (cmd.collection == null || lastStateFormat <= 0) {
- return false;
+ private boolean isNoOps(List<ZkWriteCommand> cmds) {
+ for (ZkWriteCommand cmd : cmds) {
+ if (cmd != NO_OP) return false;
}
- return cmd.collection.getStateFormat() != lastStateFormat;
+ return true;
}
/**
- * Logic to decide a flush after processing a ZkWriteCommand
+ * Check whether {@value ZkStateReader#CLUSTER_STATE} (for stateFormat = 1) get changed given command
+ */
+ private boolean clusterStateGetModifiedWith(ZkWriteCommand command, ClusterState state) {
+ DocCollection previousCollection = state.getCollectionOrNull(command.name);
+ boolean wasPreviouslyStateFormat1 = previousCollection != null && previousCollection.getStateFormat() == 1;
+ boolean isCurrentlyStateFormat1 = command.collection != null && command.collection.getStateFormat() == 1;
+ return wasPreviouslyStateFormat1 || isCurrentlyStateFormat1;
+ }
+ /**
+ * Logic to decide a flush after processing a list of ZkWriteCommand
*
- * @param cmd the ZkWriteCommand instance
* @return true if a flush to ZK is required, false otherwise
*/
- protected boolean maybeFlushAfter(ZkWriteCommand cmd) {
- if (cmd.collection == null)
- return false;
- lastCollectionName = cmd.name;
- lastStateFormat = cmd.collection.getStateFormat();
- return System.nanoTime() - lastUpdatedTime > MAX_FLUSH_INTERVAL || updates.size() > Overseer.STATE_UPDATE_BATCH_SIZE;
+ private boolean maybeFlushAfter() {
+ return System.nanoTime() - lastUpdatedTime > MAX_FLUSH_INTERVAL || numUpdates > Overseer.STATE_UPDATE_BATCH_SIZE;
}
public boolean hasPendingUpdates() {
- return !updates.isEmpty() || isClusterStateModified;
+ return numUpdates != 0 || isClusterStateModified;
}
/**
@@ -235,6 +214,7 @@ public class ZkStateWriter {
}
updates.clear();
+ numUpdates = 0;
}
if (isClusterStateModified) {
@@ -266,14 +246,6 @@ public class ZkStateWriter {
}
/**
- * @return time returned by System.nanoTime at which the main cluster state was last written to ZK or 0 if
- * never
- */
- public long getLastUpdatedTime() {
- return lastUpdatedTime;
- }
-
- /**
* @return the most up-to-date cluster state until the last enqueueUpdate operation
*/
public ClusterState getClusterState() {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8a7d04a6/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
index 62f97cb..97e8a17 100644
--- a/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
@@ -37,7 +37,9 @@ import com.codahale.metrics.Snapshot;
import com.codahale.metrics.Timer;
import org.apache.lucene.util.LuceneTestCase.Slow;
import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.cloud.overseer.NodeMutator;
import org.apache.solr.cloud.overseer.OverseerAction;
+import org.apache.solr.cloud.overseer.ZkWriteCommand;
import org.apache.solr.common.cloud.ClusterState;
import org.apache.solr.common.cloud.DocCollection;
import org.apache.solr.common.cloud.Replica;
@@ -389,6 +391,63 @@ public class OverseerTest extends SolrTestCaseJ4 {
}
}
+ @Test
+ public void testDownNodeFailover() throws Exception {
+ String zkDir = createTempDir("zkData").toFile().getAbsolutePath();
+
+ ZkTestServer server = new ZkTestServer(zkDir);
+
+ MockZKController zkController = null;
+ SolrZkClient zkClient = null;
+ SolrZkClient overseerClient = null;
+
+ try {
+ server.run();
+ AbstractZkTestCase.tryCleanSolrZkNode(server.getZkHost());
+ AbstractZkTestCase.makeSolrZkNode(server.getZkHost());
+
+ zkClient = new SolrZkClient(server.getZkAddress(), TIMEOUT);
+ ZkController.createClusterZkNodes(zkClient);
+
+ overseerClient = electNewOverseer(server.getZkAddress());
+
+ ZkStateReader reader = new ZkStateReader(zkClient);
+ reader.createClusterStateWatchersAndUpdate();
+
+ zkController = new MockZKController(server.getZkAddress(), "127.0.0.1");
+
+ for (int i = 0; i < 5; i++) {
+ zkController.createCollection("collection" + i, 1);
+ assertNotNull("shard got no id?", zkController.publishState("collection"+i, "core1",
+ "core_node1", "shard1" , Replica.State.ACTIVE, 1));
+ }
+ ZkNodeProps m = new ZkNodeProps(Overseer.QUEUE_OPERATION, OverseerAction.DOWNNODE.toLower(),
+ ZkStateReader.NODE_NAME_PROP, "127.0.0.1");
+ List<ZkWriteCommand> commands = new NodeMutator().downNode(reader.getClusterState(), m);
+
+ ZkDistributedQueue q = Overseer.getStateUpdateQueue(zkClient);
+ // More than Overseer.STATE_UPDATE_DELAY
+ Thread.sleep(2200);
+ q.offer(Utils.toJSON(m));
+
+ verifyReplicaStatus(reader, commands.get(0).name, "shard1", "core_node1", Replica.State.DOWN);
+ overseerClient.close();
+ Thread.sleep(1000); // wait for overseer to get killed
+
+ overseerClient = electNewOverseer(server.getZkAddress());
+ for (int i = 0; i < 5; i++) {
+ verifyReplicaStatus(reader, "collection"+i, "shard1", "core_node1", Replica.State.DOWN);
+ }
+ } finally {
+ close(zkClient);
+ if (zkController != null) {
+ zkController.close();
+ }
+ close(overseerClient);
+ server.shutdown();
+ }
+ }
+
//wait until collections are available
private void waitForCollections(ZkStateReader stateReader, String... collections) throws InterruptedException, KeeperException {
int maxIterations = 100;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8a7d04a6/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateReaderTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateReaderTest.java b/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateReaderTest.java
index cb0bac5..167279d 100644
--- a/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateReaderTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateReaderTest.java
@@ -16,6 +16,7 @@
*/
package org.apache.solr.cloud.overseer;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;
@@ -88,7 +89,7 @@ public class ZkStateReaderTest extends SolrTestCaseJ4 {
// create new collection with stateFormat = 1
DocCollection stateV1 = new DocCollection("c1", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.CLUSTER_STATE);
ZkWriteCommand c1 = new ZkWriteCommand("c1", stateV1);
- writer.enqueueUpdate(reader.getClusterState(), c1, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c1), null);
writer.writePendingUpdates();
Map map = (Map) Utils.fromJSON(zkClient.getData("/clusterstate.json", null, null, true));
@@ -111,7 +112,7 @@ public class ZkStateReaderTest extends SolrTestCaseJ4 {
// Now update the collection to stateFormat = 2
DocCollection stateV2 = new DocCollection("c1", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c1/state.json");
ZkWriteCommand c2 = new ZkWriteCommand("c1", stateV2);
- writer.enqueueUpdate(reader.getClusterState(), c2, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c2), null);
writer.writePendingUpdates();
Map map = (Map) Utils.fromJSON(zkClient.getData("/clusterstate.json", null, null, true));
@@ -160,7 +161,7 @@ public class ZkStateReaderTest extends SolrTestCaseJ4 {
// create new collection with stateFormat = 2
ZkWriteCommand c1 = new ZkWriteCommand("c1",
new DocCollection("c1", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c1/state.json"));
- writer.enqueueUpdate(reader.getClusterState(), c1, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c1), null);
writer.writePendingUpdates();
reader.forceUpdateCollection("c1");
@@ -211,7 +212,7 @@ public class ZkStateReaderTest extends SolrTestCaseJ4 {
// create new collection with stateFormat = 2
DocCollection state = new DocCollection("c1", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.CLUSTER_STATE + "/c1/state.json");
ZkWriteCommand wc = new ZkWriteCommand("c1", state);
- writer.enqueueUpdate(reader.getClusterState(), wc, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(wc), null);
writer.writePendingUpdates();
assertTrue(zkClient.exists(ZkStateReader.COLLECTIONS_ZKNODE + "/c1/state.json", true));
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8a7d04a6/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java b/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
index f96b5e2..436d72e 100644
--- a/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/overseer/ZkStateWriterTest.java
@@ -17,8 +17,10 @@
package org.apache.solr.cloud.overseer;
import java.lang.invoke.MethodHandles;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
+import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.lucene.util.IOUtils;
import org.apache.solr.SolrTestCaseJ4;
@@ -35,12 +37,37 @@ import org.apache.solr.common.cloud.SolrZkClient;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.util.Utils;
import org.apache.zookeeper.KeeperException;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class ZkStateWriterTest extends SolrTestCaseJ4 {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final ZkStateWriter.ZkWriteCallback FAIL_ON_WRITE = new ZkStateWriter.ZkWriteCallback() {
+ @Override
+ public void onEnqueue() throws Exception {
+
+ }
+
+ @Override
+ public void onWrite() throws Exception {
+ fail("Got unexpected flush");
+ }
+ };
+
+ @BeforeClass
+ public static void setup() {
+ System.setProperty("solr.OverseerStateUpdateDelay", "1000");
+ System.setProperty("solr.OverseerStateUpdateBatchSize", "10");
+ }
+
+ @AfterClass
+ public static void cleanup() {
+ System.clearProperty("solr.OverseerStateUpdateDelay");
+ System.clearProperty("solr.OverseerStateUpdateBatchSize");
+ }
public void testZkStateWriterBatching() throws Exception {
String zkDir = createTempDir("testZkStateWriterBatching").toFile().getAbsolutePath();
@@ -60,64 +87,52 @@ public class ZkStateWriterTest extends SolrTestCaseJ4 {
try (ZkStateReader reader = new ZkStateReader(zkClient)) {
reader.createClusterStateWatchersAndUpdate();
- ZkStateWriter writer = new ZkStateWriter(reader, new Overseer.Stats());
-
- assertFalse("Deletes can always be batched", writer.maybeFlushBefore(new ZkWriteCommand("xyz", null)));
- assertFalse("Deletes can always be batched", writer.maybeFlushAfter(new ZkWriteCommand("xyz", null)));
-
zkClient.makePath(ZkStateReader.COLLECTIONS_ZKNODE + "/c1", true);
zkClient.makePath(ZkStateReader.COLLECTIONS_ZKNODE + "/c2", true);
+ zkClient.makePath(ZkStateReader.COLLECTIONS_ZKNODE + "/c3", true);
- // create new collection with stateFormat = 2
ZkWriteCommand c1 = new ZkWriteCommand("c1",
new DocCollection("c1", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c1"));
- assertFalse("First requests can always be batched", writer.maybeFlushBefore(c1));
-
- ClusterState clusterState = writer.enqueueUpdate(reader.getClusterState(), c1, null);
-
ZkWriteCommand c2 = new ZkWriteCommand("c2",
new DocCollection("c2", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c2"));
- assertFalse("Different (new) collection create can be batched together with another create", writer.maybeFlushBefore(c2));
-
- // simulate three state changes on same collection, all should be batched together before
- assertFalse(writer.maybeFlushBefore(c1));
- assertFalse(writer.maybeFlushBefore(c1));
- assertFalse(writer.maybeFlushBefore(c1));
- // and after too
- assertFalse(writer.maybeFlushAfter(c1));
- assertFalse(writer.maybeFlushAfter(c1));
- assertFalse(writer.maybeFlushAfter(c1));
-
- // simulate three state changes on two different collections with stateFormat=2, all should be batched
- assertFalse(writer.maybeFlushBefore(c1));
- // flushAfter has to be called as it updates the internal batching related info
- assertFalse(writer.maybeFlushAfter(c1));
- assertFalse(writer.maybeFlushBefore(c2));
- assertFalse(writer.maybeFlushAfter(c2));
- assertFalse(writer.maybeFlushBefore(c1));
- assertFalse(writer.maybeFlushAfter(c1));
-
- // create a collection in stateFormat = 1 i.e. inside the main cluster state
ZkWriteCommand c3 = new ZkWriteCommand("c3",
- new DocCollection("c3", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.CLUSTER_STATE));
- clusterState = writer.enqueueUpdate(clusterState, c3, null);
+ new DocCollection("c3", new HashMap<>(), new HashMap<>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c3"));
+ ZkStateWriter writer = new ZkStateWriter(reader, new Overseer.Stats());
- // simulate three state changes in c3, all should be batched
- for (int i = 0; i < 3; i++) {
- assertFalse(writer.maybeFlushBefore(c3));
- assertFalse(writer.maybeFlushAfter(c3));
+ // First write is flushed immediately
+ ClusterState clusterState = writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c1), null);
+ clusterState = writer.enqueueUpdate(clusterState, Collections.singletonList(c1), FAIL_ON_WRITE);
+ clusterState = writer.enqueueUpdate(clusterState, Collections.singletonList(c2), FAIL_ON_WRITE);
+
+ Thread.sleep(Overseer.STATE_UPDATE_DELAY + 100);
+ AtomicBoolean didWrite = new AtomicBoolean(false);
+ clusterState = writer.enqueueUpdate(clusterState, Collections.singletonList(c3), new ZkStateWriter.ZkWriteCallback() {
+ @Override
+ public void onEnqueue() throws Exception {
+
+ }
+
+ @Override
+ public void onWrite() throws Exception {
+ didWrite.set(true);
+ }
+ });
+ assertTrue("Exceed the update delay, should be flushed", didWrite.get());
+
+ for (int i = 0; i <= Overseer.STATE_UPDATE_BATCH_SIZE; i++) {
+ clusterState = writer.enqueueUpdate(clusterState, Collections.singletonList(c3), new ZkStateWriter.ZkWriteCallback() {
+ @Override
+ public void onEnqueue() throws Exception {
+
+ }
+
+ @Override
+ public void onWrite() throws Exception {
+ didWrite.set(true);
+ }
+ });
}
-
- // simulate state change in c3 (stateFormat=1) interleaved with state changes from c1,c2 (stateFormat=2)
- // none should be batched together
- assertFalse(writer.maybeFlushBefore(c3));
- assertFalse(writer.maybeFlushAfter(c3));
- assertTrue("different stateFormat, should be flushed", writer.maybeFlushBefore(c1));
- assertFalse(writer.maybeFlushAfter(c1));
- assertTrue("different stateFormat, should be flushed", writer.maybeFlushBefore(c3));
- assertFalse(writer.maybeFlushAfter(c3));
- assertTrue("different stateFormat, should be flushed", writer.maybeFlushBefore(c2));
- assertFalse(writer.maybeFlushAfter(c2));
+ assertTrue("Exceed the update batch size, should be flushed", didWrite.get());
}
} finally {
@@ -152,7 +167,7 @@ public class ZkStateWriterTest extends SolrTestCaseJ4 {
ZkWriteCommand c1 = new ZkWriteCommand("c1",
new DocCollection("c1", new HashMap<String, Slice>(), new HashMap<String, Object>(), DocRouter.DEFAULT, 0, ZkStateReader.CLUSTER_STATE));
- ClusterState clusterState = writer.enqueueUpdate(reader.getClusterState(), c1, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c1), null);
writer.writePendingUpdates();
Map map = (Map) Utils.fromJSON(zkClient.getData("/clusterstate.json", null, null, true));
@@ -194,7 +209,7 @@ public class ZkStateWriterTest extends SolrTestCaseJ4 {
ZkWriteCommand c1 = new ZkWriteCommand("c1",
new DocCollection("c1", new HashMap<String, Slice>(), new HashMap<String, Object>(), DocRouter.DEFAULT, 0, ZkStateReader.COLLECTIONS_ZKNODE + "/c1/state.json"));
- ClusterState clusterState = writer.enqueueUpdate(reader.getClusterState(), c1, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c1), null);
writer.writePendingUpdates();
Map map = (Map) Utils.fromJSON(zkClient.getData("/clusterstate.json", null, null, true));
@@ -238,7 +253,7 @@ public class ZkStateWriterTest extends SolrTestCaseJ4 {
// create collection 1 with stateFormat = 1
ZkWriteCommand c1 = new ZkWriteCommand("c1",
new DocCollection("c1", new HashMap<String, Slice>(), new HashMap<String, Object>(), DocRouter.DEFAULT, 0, ZkStateReader.CLUSTER_STATE));
- writer.enqueueUpdate(reader.getClusterState(), c1, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c1), null);
writer.writePendingUpdates();
reader.forceUpdateCollection("c1");
@@ -252,22 +267,23 @@ public class ZkStateWriterTest extends SolrTestCaseJ4 {
zkClient.setData("/clusterstate.json", data, true);
// enqueue another c1 so that ZkStateWriter has pending updates
- writer.enqueueUpdate(clusterState, c1, null);
+ writer.enqueueUpdate(clusterState, Collections.singletonList(c1), null);
assertTrue(writer.hasPendingUpdates());
- // create collection 2 with stateFormat = 1
+ // Will trigger flush
+ Thread.sleep(Overseer.STATE_UPDATE_DELAY + 100);
ZkWriteCommand c2 = new ZkWriteCommand("c2",
new DocCollection("c2", new HashMap<String, Slice>(), new HashMap<String, Object>(), DocRouter.DEFAULT, 0, ZkStateReader.getCollectionPath("c2")));
try {
- writer.enqueueUpdate(clusterState, c2, null); // we are sending in the old cluster state object
+ writer.enqueueUpdate(clusterState, Collections.singletonList(c2), null); // we are sending in the old cluster state object
fail("Enqueue should not have succeeded");
} catch (KeeperException.BadVersionException bve) {
// expected
}
try {
- writer.enqueueUpdate(reader.getClusterState(), c2, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c2), null);
fail("enqueueUpdate after BadVersionException should not have succeeded");
} catch (IllegalStateException e) {
// expected
@@ -315,7 +331,7 @@ public class ZkStateWriterTest extends SolrTestCaseJ4 {
// create collection 2 with stateFormat = 2
ZkWriteCommand c2 = new ZkWriteCommand("c2",
new DocCollection("c2", new HashMap<String, Slice>(), new HashMap<String, Object>(), DocRouter.DEFAULT, 0, ZkStateReader.getCollectionPath("c2")));
- state = writer.enqueueUpdate(reader.getClusterState(), c2, null);
+ state = writer.enqueueUpdate(state, Collections.singletonList(c2), null);
assertFalse(writer.hasPendingUpdates()); // first write is flushed immediately
int sharedClusterStateVersion = state.getZkClusterStateVersion();
@@ -333,27 +349,27 @@ public class ZkStateWriterTest extends SolrTestCaseJ4 {
assertEquals(sharedClusterStateVersion, (int) state.getZkClusterStateVersion());
assertEquals(stateFormat2Version + 1, state.getCollection("c2").getZNodeVersion());
- // enqueue an update to stateFormat2 collection such that update is pending
- state = writer.enqueueUpdate(state, c2, null);
+ writer.enqueueUpdate(state, Collections.singletonList(c2), null);
assertTrue(writer.hasPendingUpdates());
// get the most up-to-date state
reader.forceUpdateCollection("c2");
state = reader.getClusterState();
- // enqueue a stateFormat=1 collection which should cause a flush
+ // Will trigger flush
+ Thread.sleep(Overseer.STATE_UPDATE_DELAY+100);
ZkWriteCommand c1 = new ZkWriteCommand("c1",
new DocCollection("c1", new HashMap<String, Slice>(), new HashMap<String, Object>(), DocRouter.DEFAULT, 0, ZkStateReader.CLUSTER_STATE));
try {
- writer.enqueueUpdate(state, c1, null);
+ writer.enqueueUpdate(state, Collections.singletonList(c1), null);
fail("Enqueue should not have succeeded");
} catch (KeeperException.BadVersionException bve) {
// expected
}
try {
- writer.enqueueUpdate(reader.getClusterState(), c2, null);
+ writer.enqueueUpdate(reader.getClusterState(), Collections.singletonList(c2), null);
fail("enqueueUpdate after BadVersionException should not have succeeded");
} catch (IllegalStateException e) {
// expected
[2/8] lucene-solr:jira/solr-11285: SOLR-11451:
ComputePlanActionTest.testNodeLost() failure
Posted by ab...@apache.org.
SOLR-11451: ComputePlanActionTest.testNodeLost() failure
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/4f52d2db
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/4f52d2db
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/4f52d2db
Branch: refs/heads/jira/solr-11285
Commit: 4f52d2db109557c5e6ddf2b6f0bdf7090707f41b
Parents: a28474e
Author: Noble Paul <no...@apache.org>
Authored: Fri Oct 13 00:13:42 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Fri Oct 13 00:13:42 2017 +1030
----------------------------------------------------------------------
.../cloud/autoscaling/ComputePlanAction.java | 4 +--
.../solr/cloud/autoscaling/TriggerEvent.java | 8 ++----
.../autoscaling/ComputePlanActionTest.java | 26 ++++++++++++++------
3 files changed, 23 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/4f52d2db/solr/core/src/java/org/apache/solr/cloud/autoscaling/ComputePlanAction.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ComputePlanAction.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ComputePlanAction.java
index dd710a4..094556b 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/ComputePlanAction.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/ComputePlanAction.java
@@ -101,12 +101,12 @@ public class ComputePlanAction extends TriggerActionBase {
case NODEADDED:
suggester = session.getSuggester(CollectionParams.CollectionAction.MOVEREPLICA)
.hint(Policy.Suggester.Hint.TARGET_NODE, event.getProperty(TriggerEvent.NODE_NAMES));
- log.debug("Created suggester with targetNode: {}", event.getProperty(TriggerEvent.NODE_NAMES));
+ log.debug("NODEADDED Created suggester with targetNode: {}", event.getProperty(TriggerEvent.NODE_NAMES));
break;
case NODELOST:
suggester = session.getSuggester(CollectionParams.CollectionAction.MOVEREPLICA)
.hint(Policy.Suggester.Hint.SRC_NODE, event.getProperty(TriggerEvent.NODE_NAMES));
- log.debug("Created suggester with srcNode: {}", event.getProperty(TriggerEvent.NODE_NAMES));
+ log.debug("NODELOST Created suggester with srcNode: {}", event.getProperty(TriggerEvent.NODE_NAMES));
break;
default:
throw new UnsupportedOperationException("No support for events other than nodeAdded and nodeLost, received: " + event.getEventType());
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/4f52d2db/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEvent.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEvent.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEvent.java
index 33eeebe..3d436af 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEvent.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEvent.java
@@ -22,6 +22,7 @@ import java.util.Map;
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventType;
import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.util.Utils;
import org.apache.solr.util.IdUtils;
/**
@@ -145,11 +146,6 @@ public class TriggerEvent implements MapWriter {
@Override
public String toString() {
- return this.getClass().getSimpleName() + "{" +
- "id='" + id + '\'' +
- ", source='" + source + '\'' +
- ", eventTime=" + eventTime +
- ", properties=" + properties +
- '}';
+ return Utils.toJSONString(this);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/4f52d2db/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java b/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java
index ae69d00..c5aa775 100644
--- a/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/autoscaling/ComputePlanActionTest.java
@@ -19,6 +19,7 @@ package org.apache.solr.cloud.autoscaling;
import java.io.IOException;
import java.lang.invoke.MethodHandles;
+import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
@@ -55,7 +56,7 @@ import static org.apache.solr.common.params.CollectionParams.CollectionAction.MO
/**
* Test for {@link ComputePlanAction}
*/
-@LogLevel("org.apache.solr.cloud.autoscaling=DEBUG;org.apache.solr.cloud.Overseer=DEBUG;org.apache.solr.cloud.overseer=DEBUG;")
+@LogLevel("org.apache.solr.cloud.autoscaling=DEBUG;org.apache.solr.cloud.Overseer=DEBUG;org.apache.solr.cloud.overseer=DEBUG;org.apache.solr.client.solrj.impl.SolrClientDataProvider=DEBUG;")
public class ComputePlanActionTest extends SolrCloudTestCase {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
@@ -143,13 +144,14 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
// let's start a node so that we have at least two
JettySolrRunner runner = cluster.startJettySolrRunner();
String node = runner.getNodeName();
+ AssertingTriggerAction.expectedNode = node;
CloudSolrClient solrClient = cluster.getSolrClient();
String setTriggerCommand = "{" +
"'set-trigger' : {" +
"'name' : 'node_lost_trigger'," +
"'event' : 'nodeLost'," +
- "'waitFor' : '1s'," +
+ "'waitFor' : '7s'," +
"'enabled' : true," +
"'actions' : [{'name':'compute_plan', 'class' : 'solr.ComputePlanAction'}," +
"{'name':'test','class':'" + ComputePlanActionTest.AssertingTriggerAction.class.getName() + "'}]" +
@@ -186,14 +188,15 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
break;
}
}
+ log.info("$$$$$$$$$$$$$$Stopped_node : {}", node);
cluster.waitForAllNodes(30);
- assertTrue("Trigger was not fired even after 5 seconds", triggerFiredLatch.await(5, TimeUnit.SECONDS));
+ assertTrue("Trigger was not fired even after 10 seconds", triggerFiredLatch.await(10, TimeUnit.SECONDS));
assertTrue(fired.get());
Map context = actionContextPropsRef.get();
assertNotNull(context);
List<SolrRequest> operations = (List<SolrRequest>) context.get("operations");
- assertNotNull("The operations computed by ComputePlanAction should not be null , "+ getDataProviderState(), operations);
+ assertNotNull("The operations computed by ComputePlanAction should not be null , " + getDataProviderState() + eventRef.get(), operations);
assertEquals("ComputePlanAction should have computed exactly 1 operation", 1, operations.size());
SolrRequest solrRequest = operations.get(0);
SolrParams params = solrRequest.getParams();
@@ -214,13 +217,16 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
String result = "SolrClientDataProvider.DEBUG";
if(SolrClientDataProvider.INST != null) {
result+= Utils.toJSONString(SolrClientDataProvider.INST);
- if(SolrClientDataProvider.INST.config != null) result+= Utils.toJSONString(SolrClientDataProvider.INST.config);
+ if (SolrClientDataProvider.INST.config != null)
+ result += "autoscalingconf: " + Utils.toJSONString(SolrClientDataProvider.INST.config);
}
return result;
}
public void testNodeWithMultipleReplicasLost() throws Exception {
+ AssertingTriggerAction.expectedNode = null;
+
// start 3 more nodes
cluster.startJettySolrRunner();
cluster.startJettySolrRunner();
@@ -280,7 +286,7 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
Map context = actionContextPropsRef.get();
assertNotNull(context);
List<SolrRequest> operations = (List<SolrRequest>) context.get("operations");
- assertNotNull("The operations computed by ComputePlanAction should not be null "+getDataProviderState(), operations);
+ assertNotNull("The operations computed by ComputePlanAction should not be null " + getDataProviderState() + actionContextPropsRef.get(), operations);
operations.forEach(solrRequest -> log.info(solrRequest.getParams().toString()));
assertEquals("ComputePlanAction should have computed exactly 2 operation", 2, operations.size());
@@ -294,6 +300,7 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
@Test
public void testNodeAdded() throws Exception {
+ AssertingTriggerAction.expectedNode = null;
CloudSolrClient solrClient = cluster.getSolrClient();
String setTriggerCommand = "{" +
"'set-trigger' : {" +
@@ -346,7 +353,7 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
Map context = actionContextPropsRef.get();
assertNotNull(context);
List<SolrRequest> operations = (List<SolrRequest>) context.get("operations");
- assertNotNull("The operations computed by ComputePlanAction should not be null" + getDataProviderState(), operations);
+ assertNotNull("The operations computed by ComputePlanAction should not be null" + getDataProviderState() + context, operations);
assertEquals("ComputePlanAction should have computed exactly 1 operation", 1, operations.size());
SolrRequest request = operations.get(0);
SolrParams params = request.getParams();
@@ -356,6 +363,7 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
}
public static class AssertingTriggerAction implements TriggerAction {
+ static String expectedNode;
@Override
public String getName() {
@@ -364,6 +372,10 @@ public class ComputePlanActionTest extends SolrCloudTestCase {
@Override
public void process(TriggerEvent event, ActionContext context) {
+ if (expectedNode != null) {
+ Collection nodes = (Collection) event.getProperty(TriggerEvent.NODE_NAMES);
+ if (nodes == null || !nodes.contains(expectedNode)) return;//this is not the event we are looking for
+ }
if (fired.compareAndSet(false, true)) {
eventRef.set(event);
actionContextPropsRef.set(context.getProperties());
[4/8] lucene-solr:jira/solr-11285: SOLR-11469: disable
LeaderElectionContextKeyTest since we know it's logically flawed
Posted by ab...@apache.org.
SOLR-11469: disable LeaderElectionContextKeyTest since we know it's logically flawed
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/cd1a6358
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/cd1a6358
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/cd1a6358
Branch: refs/heads/jira/solr-11285
Commit: cd1a635898e2d37b723ae648a270242f9fc80323
Parents: 05ce968
Author: Chris Hostetter <ho...@apache.org>
Authored: Thu Oct 12 13:41:10 2017 -0700
Committer: Chris Hostetter <ho...@apache.org>
Committed: Thu Oct 12 13:41:10 2017 -0700
----------------------------------------------------------------------
.../test/org/apache/solr/cloud/LeaderElectionContextKeyTest.java | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cd1a6358/solr/core/src/test/org/apache/solr/cloud/LeaderElectionContextKeyTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/LeaderElectionContextKeyTest.java b/solr/core/src/test/org/apache/solr/cloud/LeaderElectionContextKeyTest.java
index 728ea1b..bfb1c13 100644
--- a/solr/core/src/test/org/apache/solr/cloud/LeaderElectionContextKeyTest.java
+++ b/solr/core/src/test/org/apache/solr/cloud/LeaderElectionContextKeyTest.java
@@ -34,6 +34,9 @@ import org.hamcrest.CoreMatchers;
import org.junit.BeforeClass;
import org.junit.Test;
+import org.apache.lucene.util.LuceneTestCase.AwaitsFix;
+
+@AwaitsFix(bugUrl="https://issues.apache.org/jira/browse/SOLR-11469")
public class LeaderElectionContextKeyTest extends SolrCloudTestCase {
private static final String TEST_COLLECTION_1 = "testCollection1";
[5/8] lucene-solr:jira/solr-11285: SOLR-11445: Upgrade CHANGES.txt
Posted by ab...@apache.org.
SOLR-11445: Upgrade CHANGES.txt
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/4bc7fd2b
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/4bc7fd2b
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/4bc7fd2b
Branch: refs/heads/jira/solr-11285
Commit: 4bc7fd2bd54c6a06f88c9aa2de9d3b94f5a77efb
Parents: cd1a635
Author: Cao Manh Dat <da...@apache.org>
Authored: Fri Oct 13 08:17:09 2017 +0700
Committer: Cao Manh Dat <da...@apache.org>
Committed: Fri Oct 13 08:17:09 2017 +0700
----------------------------------------------------------------------
solr/CHANGES.txt | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/4bc7fd2b/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index aa252b8..8c0ef45 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -45,9 +45,10 @@ Apache UIMA 2.3.1
Apache ZooKeeper 3.4.10
Jetty 9.3.20.v20170531
+Bug Fixes
+----------------------
-(No Changes)
-
+* SOLR-11445: Overseer should not hang when process bad message. (Cao Manh Dat, shalin)
================== 7.1.0 ==================
[3/8] lucene-solr:jira/solr-11285: Ref Guide: fix "how to edit
Jekyll" docs so section-toc optional parameter displays in output
Posted by ab...@apache.org.
Ref Guide: fix "how to edit Jekyll" docs so section-toc optional parameter displays in output
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/05ce9684
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/05ce9684
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/05ce9684
Branch: refs/heads/jira/solr-11285
Commit: 05ce9684825434c5c2da1e4fa6d6c8eeb8386460
Parents: 4f52d2d
Author: Cassandra Targett <ct...@apache.org>
Authored: Thu Oct 12 09:50:58 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Thu Oct 12 09:51:59 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/meta-docs/jekyll.adoc | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/05ce9684/solr/solr-ref-guide/src/meta-docs/jekyll.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/meta-docs/jekyll.adoc b/solr/solr-ref-guide/src/meta-docs/jekyll.adoc
index f2a8b72..264a5cd 100644
--- a/solr/solr-ref-guide/src/meta-docs/jekyll.adoc
+++ b/solr/solr-ref-guide/src/meta-docs/jekyll.adoc
@@ -56,10 +56,17 @@ The Solr Ref Guide uses the front matter to define some custom attributes on a p
There are also some optional custom attributes that can be defined in pages to affect the Table of Contents presentation in jekyll:
* `page-toclevels` - changes how "deep" the TOC will be in terms of nested section/sub-section titles (default = 2)
-* `page-tocclass` - changes the CSS class applied to the TOC, default = "normal", resulting in the class name `toc-normal`. The other option is "right", to put the TOC on the right side of the page.
+* `page-tocclass` - changes the CSS class applied to the TOC. The default is `normal`, resulting in the class name `toc-normal` being used in the HTML. The other option is `right`, to put the TOC on the right side of the page.
* `page-toc` - if this is false, then no TOCs will be generated for the page at all.
-NOTE: The special macro `{section-toc}` can be used anywhere in a page to create an "In this Section" TOC covering only the sub-headings in the same section. `:page-toc: false` will also prevent this macro from working, so if you want no "top level" TOC, but you do want section TOCs, use `:page-toclevels: 0`
+[NOTE]
+====
+The special macro `\{section-toc}` can be used anywhere in a page to create an "In this Section" TOC covering only the sub-headings in the same section.
+
+Setting `:page-toc: false` will prevent this macro from working, so if you want no "top level" TOC, but you do want section TOCs, use `:page-toclevels: 0`.
+====
+
+The format for adding any parameter to the front matter is to use colons on both sides of the parameter, followed by the value for the parameter (such as `:page-toc: false`).
=== Layouts