You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/02/17 15:11:07 UTC
[GitHub] [lucene-solr] murblanc opened a new pull request #2390: SOLR-15157: refactor Collection API to separate from Overseer and message handling abstractions
murblanc opened a new pull request #2390:
URL: https://github.com/apache/lucene-solr/pull/2390
This refactoring is inserting a layer of abstraction (`CollectionCommandContext`) between Collection API commands and the `OverseerCollectionMessageHandler`, to enable future changes where Collection API messages are executed outside of Overseer.
There are (almost) no other changes in this PR. Two exceptions: 1. in `CreateCollectionCmd`, appropriate conditions were added around recently added calls related to Per Replica States collections and 2. a minor fix in test `OverseerStatusTest` regarding distributed updates and Overseer stats.
I'd rather this PR not linger around for too long if possible. Given `OverseerCollectionMessageHandler` has lost most of its contents, merging in new changes is a highly manual process.
This refactoring is intended to eventually enable constructions such as shown in the PoC (based on a slightly older snapshot of master). See line [348 in CollectionsHandler](https://github.com/murblanc/lucene-solr/commit/10cdad0e2618ea82b2b97f4a78cc0f3ff8df8082#diff-582348d44491dcb0ce1dfb169fb544e9e95620b2d0448eb1a1744f4e8dd5a349R348) there, when a request is received to create a collection, rather than enqueue a ZK message to the Collection API queue for Overseer, the execution is done locally in method [distributedCollectionCreation](https://github.com/murblanc/lucene-solr/commit/10cdad0e2618ea82b2b97f4a78cc0f3ff8df8082#diff-582348d44491dcb0ce1dfb169fb544e9e95620b2d0448eb1a1744f4e8dd5a349R280). This method needs to call the Collection API command, and does not execute in the context of the Overseer.
The motivation for doing the refactoring separately (as discussed in [Slack](https://the-asf.slack.com/archives/CEKUCUNE9/p1612908679186300)) is to keep the hardest to review part (code moving around) clean of any other changes (and a side benefit is that the actual Collection API distribution will take quite some time, and having part of it be such a refactoring waiting for a few weeks on a branch is a recipe for not being able to merge it safely).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on pull request #2390: SOLR-15157: refactor Collection API to separate from Overseer and message handling abstractions
Posted by GitBox <gi...@apache.org>.
murblanc commented on pull request #2390:
URL: https://github.com/apache/lucene-solr/pull/2390#issuecomment-780625973
@madrob @chatman @gerlowskija @sigram @noblepaul - if you want to have a look. You've all recently touched parts of the code and/or are familiar with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] murblanc merged pull request #2390: SOLR-15157: refactor Collection API to separate from Overseer and message handling abstractions
Posted by GitBox <gi...@apache.org>.
murblanc merged pull request #2390:
URL: https://github.com/apache/lucene-solr/pull/2390
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] muse-dev[bot] commented on a change in pull request #2390: SOLR-15157: refactor Collection API to separate from Overseer and message handling abstractions
Posted by GitBox <gi...@apache.org>.
muse-dev[bot] commented on a change in pull request #2390:
URL: https://github.com/apache/lucene-solr/pull/2390#discussion_r577720307
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CollectionHandlingUtils.java
##########
@@ -0,0 +1,592 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cloud.api.collections;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.solr.client.solrj.SolrResponse;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.cloud.AlreadyExistsException;
+import org.apache.solr.client.solrj.cloud.BadVersionException;
+import org.apache.solr.client.solrj.cloud.DistribStateManager;
+import org.apache.solr.client.solrj.cloud.SolrCloudManager;
+import org.apache.solr.client.solrj.impl.BaseHttpSolrClient;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.AbstractUpdateRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.UpdateResponse;
+import org.apache.solr.cloud.DistributedClusterStateUpdater;
+import org.apache.solr.cloud.Overseer;
+import org.apache.solr.cloud.ZkController;
+import org.apache.solr.cloud.overseer.ClusterStateMutator;
+import org.apache.solr.cloud.overseer.OverseerAction;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.cloud.*;
+import org.apache.solr.common.params.CollectionAdminParams;
+import org.apache.solr.common.params.CoreAdminParams;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.common.util.TimeSource;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.backup.BackupId;
+import org.apache.solr.core.backup.repository.BackupRepository;
+import org.apache.solr.handler.component.ShardHandler;
+import org.apache.solr.handler.component.ShardHandlerFactory;
+import org.apache.solr.handler.component.ShardRequest;
+import org.apache.solr.handler.component.ShardResponse;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.solr.common.params.CollectionParams.CollectionAction.DELETE;
+import static org.apache.solr.common.params.CommonAdminParams.ASYNC;
+import static org.apache.solr.common.params.CommonParams.NAME;
+import static org.apache.solr.common.util.Utils.makeMap;
+
+/**
+ * This class contains helper methods used by commands of the Collection API. Previously these methods were in
+ * {@link OverseerCollectionMessageHandler} and were refactored out to (eventually) allow Collection API commands to be
+ * executed outside the context of the Overseer.
+ */
+public class CollectionHandlingUtils {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String NUM_SLICES = "numShards";
+
+ public static final boolean CREATE_NODE_SET_SHUFFLE_DEFAULT = true;
+ public static final String CREATE_NODE_SET_SHUFFLE = CollectionAdminParams.CREATE_NODE_SET_SHUFFLE_PARAM;
+ public static final String CREATE_NODE_SET_EMPTY = "EMPTY";
+ public static final String CREATE_NODE_SET = CollectionAdminParams.CREATE_NODE_SET_PARAM;
+
+ public static final String ROUTER = "router";
+
+ public static final String SHARDS_PROP = "shards";
+
+ public static final String REQUESTID = "requestid";
+
+ public static final String ONLY_IF_DOWN = "onlyIfDown";
+
+ public static final String SHARD_UNIQUE = "shardUnique";
+
+ public static final String ONLY_ACTIVE_NODES = "onlyactivenodes";
+
+ static final String SKIP_CREATE_REPLICA_IN_CLUSTER_STATE = "skipCreateReplicaInClusterState";
+
+ public static final Map<String, Object> COLLECTION_PROPS_AND_DEFAULTS = Collections.unmodifiableMap(makeMap(
+ ROUTER, DocRouter.DEFAULT_NAME,
+ ZkStateReader.REPLICATION_FACTOR, "1",
+ ZkStateReader.NRT_REPLICAS, "1",
+ ZkStateReader.TLOG_REPLICAS, "0",
+ DocCollection.PER_REPLICA_STATE, null,
+ ZkStateReader.PULL_REPLICAS, "0"));
+
+ protected static final Random RANDOM;
+ static {
+ // We try to make things reproducible in the context of our tests by initializing the random instance
+ // based on the current seed
+ String seed = System.getProperty("tests.seed");
+ if (seed == null) {
+ RANDOM = new Random();
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.Random) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on pull request #2390: SOLR-15157: refactor Collection API to separate from Overseer and message handling abstractions
Posted by GitBox <gi...@apache.org>.
murblanc commented on pull request #2390:
URL: https://github.com/apache/lucene-solr/pull/2390#issuecomment-780629746
I have done my best to preserve the order of code originating in `OverseerCollectionMessageHandler` as it moved to `CollectionHandlingUtils` and `CollApiCmds` to make an eyeball diff/comparison _slightly_ easier...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org