You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "epugh (via GitHub)" <gi...@apache.org> on 2025/11/30 11:40:05 UTC
[PR] bats test to simulate remnant core [solr]
epugh opened a new pull request, #3904:
URL: https://github.com/apache/solr/pull/3904
https://issues.apache.org/jira/browse/SOLR-XXXXX
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Solr:
* https://issues.apache.org/jira/projects/SOLR
For something minor (i.e. that wouldn't be worth putting in release notes), you can skip JIRA.
To create a Jira issue, you will need to create an account there first.
The title of the PR should reference the Jira issue number in the form:
* SOLR-####: <short description of problem or changes>
SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
Please provide a short description of the changes you're making with this pull request.
# Solution
Please provide a short description of the approach taken to implement your solution.
# Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
# Checklist
Please review the following and check all that apply:
- [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
- [ ] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
- [ ] I have added a [changelog entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #3904:
URL: https://github.com/apache/solr/pull/3904#issuecomment-3707093451
@dsmiley tests are cleaned up, and I investigated if the core admin level api would work (it doesn't!). I'd love to get this in soon if it looks okay to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2659740286
##########
solr/solr-ref-guide/modules/configuration-guide/pages/solr-properties.adoc:
##########
@@ -44,7 +44,7 @@ NOTE: Properties marked with "!" indicate inverted meaning between pre Solr 10 a
|solr.cloud.prep.recovery.read.timeout.additional.ms|prepRecoveryReadTimeoutExtraWait|8000|Specifies additional milliseconds to wait during recovery read operations in SolrCloud mode.
-|solr.cloud.startup.delete.unknown.cores.enabled|solr.deleteUnknownCores|false|Controls whether unknown cores are deleted at startup in SolrCloud mode.
+|solr.cloud.delete.unknown.cores.enabled|solr.deleteUnknownCores|false|Determines if unknown cores should be removed at startup and during collection and replica creation in SolrCloud mode.
Review Comment:
I gave it another stab...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2658945873
##########
solr/packaging/test/test_create_collection_with_remnants.bats:
##########
@@ -0,0 +1,92 @@
+#!/usr/bin/env bats
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Tests what happens when remnant core directories and and files still are on disk.
+# Flip solr.delete.unknown.cores=false to see the out of box behavior and the failures.
Review Comment:
The BATS test comment on line 19 references "solr.delete.unknown.cores" but should reference "solr.cloud.delete.unknown.cores.enabled" to match the actual property name being used in this PR.
```suggestion
# Flip solr.cloud.delete.unknown.cores.enabled=false to see the out of box behavior and the failures.
```
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
+ log.warn(
+ "There appears to be an existing directory for core {}, now deleting it",
Review Comment:
The warning message on line 1518-1520 could be more informative for production debugging. Currently it only logs the core name, but doesn't indicate that this is an automatic deletion triggered by the "solr.cloud.delete.unknown.cores.enabled" property, or provide the full path being deleted. Consider including the property name in the warning message and the full path to help administrators understand what's happening and why. For example: "Automatically deleting existing directory at [path] for core [name] because solr.cloud.delete.unknown.cores.enabled is true".
```suggestion
"Automatically deleting existing directory at [{}] for core [{}] because solr.cloud.delete.unknown.cores.enabled is true",
cd.getInstanceDir().toAbsolutePath(),
```
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.CoreAdminRequest;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ Replica primaryReplica = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ JettySolrRunner primaryJetty = cluster.getReplicaJetty(primaryReplica);
+ String originalCoreName = primaryReplica.getCoreName();
+ Path remnantInstanceDir;
+ try (SolrCore core = primaryJetty.getCoreContainer().getCore(originalCoreName)) {
+ CoreDescriptor cd = core.getCoreDescriptor();
+ remnantInstanceDir = cd.getInstanceDir();
+ }
+
+ CollectionAdminRequest.deleteCollection(collectionName).process(cluster.getSolrClient());
+ waitForState("Expected collection deletion", collectionName, (n, c) -> c == null);
+
+ // Simulate a core remnant still exists by creating the directory and core.properties
+ Files.createDirectories(remnantInstanceDir);
+ Files.writeString(remnantInstanceDir.resolve("core.properties"), "", StandardCharsets.UTF_8);
+ }
+
+ /**
+ * Shared setup for testing replica addition with remnants. Creates a collection, then simulates a
+ * remnant directory on the single node that will impact the next addReplica command.
+ */
+ private void setupReplicaRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ int nextReplicaIndex = 3; // Yep, from 1 to 3 due to how we count in ZK and setup.
+ String expectedNewReplicaName = collectionName + "_shard1_replica_n" + nextReplicaIndex;
+
+ // Simulate a core remnant on the single node adjacent to the existing replica instance path
+ Replica existing = getReplicaOnNode(collectionName, "shard1", primaryNode);
Review Comment:
The error message assertion on line 116 is fragile. It checks for the exact string "Underlying core creation failed", which could change in future Solr versions or may not be the exact message in all failure scenarios. Consider checking for a more general pattern or multiple possible error indicators (e.g., checking if the message contains "core" AND ("creation" OR "failed")) to make the test more robust to message changes.
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
+ log.warn(
+ "There appears to be an existing directory for core {}, now deleting it",
+ cd.getName());
+ SolrCore.deleteUnloadedCore(cd, true, true);
Review Comment:
The deletion operation on line 1521 (SolrCore.deleteUnloadedCore) could fail due to I/O errors, permission issues, or if files are locked. However, this failure is not explicitly caught and handled. If the deletion fails, the code continues to line 1526 (coresLocator.create), which will likely also fail since the directory still exists. Consider wrapping the deletion in a try-catch block to log specific deletion failures and provide a clearer error message to the user about why core creation failed, rather than letting it bubble up as a generic core creation failure.
```suggestion
try {
SolrCore.deleteUnloadedCore(cd, true, true);
} catch (Exception e) {
String msg =
"Failed to delete existing instance directory for core '"
+ cd.getName()
+ "' at "
+ cd.getInstanceDir()
+ " before creating the core.";
log.error(msg, e);
throw new SolrException(ErrorCode.SERVER_ERROR, msg, e);
}
```
##########
solr/packaging/test/test_create_collection_with_remnants.bats:
##########
@@ -0,0 +1,92 @@
+#!/usr/bin/env bats
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Tests what happens when remnant core directories and and files still are on disk.
Review Comment:
Grammatical error: "directories and and files" should be "directories and files" (remove duplicate "and").
```suggestion
# Tests what happens when remnant core directories and files still are on disk.
```
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
+ log.warn(
+ "There appears to be an existing directory for core {}, now deleting it",
+ cd.getName());
+ SolrCore.deleteUnloadedCore(cd, true, true);
+ }
Review Comment:
The deletion logic on lines 1517-1522 deletes the remnant directory immediately before calling coresLocator.create() on line 1526. However, there's a potential race condition: if another process or thread creates files in the instance directory between the deletion (line 1521) and the coresLocator.create() call (line 1526), this could lead to unexpected behavior. Consider checking if the directory exists again right before calling coresLocator.create() or handling potential FileAlreadyExistsException from coresLocator.create().
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
+ log.warn(
+ "There appears to be an existing directory for core {}, now deleting it",
+ cd.getName());
+ SolrCore.deleteUnloadedCore(cd, true, true);
+ }
Review Comment:
The new deletion logic deletes directories without validating whether the existing directory actually contains remnant core files or is a valid/active core directory. The code only checks if the directory exists (Files.exists(cd.getInstanceDir())), but doesn't verify that it's truly a remnant (e.g., checking for the absence of a valid core descriptor in ZooKeeper or validating the directory is not in use). This could potentially delete legitimate core directories if there's a configuration issue or timing problem. Consider adding validation to confirm the directory is actually a remnant before deletion, such as checking if the core is registered in ZooKeeper or if it's currently loaded in the CoreContainer.
##########
solr/packaging/test/test_create_collection_with_remnants.bats:
##########
@@ -0,0 +1,92 @@
+#!/usr/bin/env bats
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Tests what happens when remnant core directories and and files still are on disk.
+# Flip solr.delete.unknown.cores=false to see the out of box behavior and the failures.
+
+load bats_helper
+
+setup_file() {
+ common_clean_setup
+ solr start -Dsolr.delete.unknown.cores=true
Review Comment:
The BATS test uses an incorrect property name "solr.delete.unknown.cores" on line 25, but should use the new property name "solr.cloud.delete.unknown.cores.enabled" to be consistent with the property rename in this PR. While the PR description mentions this test will be deleted before merging, it should still use the correct property name while it exists in the codebase.
```suggestion
# Flip solr.cloud.delete.unknown.cores.enabled=false to see the out of box behavior and the failures.
load bats_helper
setup_file() {
common_clean_setup
solr start -Dsolr.cloud.delete.unknown.cores.enabled=true
```
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
Review Comment:
The property retrieval pattern on lines 1514-1516 is duplicated at line 1681-1683 (with the same property name). Consider extracting this into a private helper method or a constant field to avoid duplication and ensure consistency. For example, create a method like `private boolean shouldDeleteUnknownCores()` that can be reused in both locations.
##########
solr/solrj/src/resources/DeprecatedSystemPropertyMappings.properties:
##########
@@ -18,7 +19,8 @@ solr.auth.superuser=solr.authorization.superuser
solr.cloud.shardsplit.checkdiskspace.enabled=solr.shard.split.check.disk.space.enabled
solr.cloud.prep.recovery.read.timeout.additional.ms=prep.recovery.read.timeout.extra.wait
solr.cloud.prs.enabled=solr.prs.default
-solr.cloud.startup.delete.unknown.cores.enabled=solr.delete.unknown.cores
+solr.cloud.delete.unknown.cores.enabled=solr.delete.unknown.cores
+#solr.cloud.delete.unknown.cores.enabled=solr.cloud.startup.delete.unknown.cores.enabled
Review Comment:
The deprecated property mapping appears to be inverted. Based on the file header comment (line 4), when renaming a property, the format should be "new.property.name=old.property.name". Currently, line 22 maps the new name to an even older legacy name (solr.delete.unknown.cores), while the actual old property name (solr.cloud.startup.delete.unknown.cores.enabled) is commented out on line 23. The commented line 23 should be active, and line 22 should be removed or adjusted. Otherwise, users with the old property name "solr.cloud.startup.delete.unknown.cores.enabled" won't have it properly mapped to the new name.
```suggestion
solr.cloud.delete.unknown.cores.enabled=solr.cloud.startup.delete.unknown.cores.enabled
```
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.CoreAdminRequest;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ Replica primaryReplica = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ JettySolrRunner primaryJetty = cluster.getReplicaJetty(primaryReplica);
+ String originalCoreName = primaryReplica.getCoreName();
+ Path remnantInstanceDir;
+ try (SolrCore core = primaryJetty.getCoreContainer().getCore(originalCoreName)) {
+ CoreDescriptor cd = core.getCoreDescriptor();
+ remnantInstanceDir = cd.getInstanceDir();
+ }
+
+ CollectionAdminRequest.deleteCollection(collectionName).process(cluster.getSolrClient());
+ waitForState("Expected collection deletion", collectionName, (n, c) -> c == null);
+
+ // Simulate a core remnant still exists by creating the directory and core.properties
+ Files.createDirectories(remnantInstanceDir);
+ Files.writeString(remnantInstanceDir.resolve("core.properties"), "", StandardCharsets.UTF_8);
+ }
+
+ /**
+ * Shared setup for testing replica addition with remnants. Creates a collection, then simulates a
+ * remnant directory on the single node that will impact the next addReplica command.
+ */
+ private void setupReplicaRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ int nextReplicaIndex = 3; // Yep, from 1 to 3 due to how we count in ZK and setup.
+ String expectedNewReplicaName = collectionName + "_shard1_replica_n" + nextReplicaIndex;
+
+ // Simulate a core remnant on the single node adjacent to the existing replica instance path
+ Replica existing = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ try (SolrCore core =
+ cluster.getReplicaJetty(existing).getCoreContainer().getCore(existing.getCoreName())) {
+ Path siblingDir = core.getInstancePath().getParent().resolve(expectedNewReplicaName);
+ Files.createDirectories(siblingDir);
+ Files.writeString(
+ siblingDir.resolve("core.properties"),
+ "name="
+ + expectedNewReplicaName
+ + "_remnant\n"
+ + "collection="
+ + collectionName
+ + "_remnant\n"
+ + "shard=shard1\n"
+ + "coreNodeName=core_node_remnant\n",
+ StandardCharsets.UTF_8);
+ }
+ }
+
+ /**
+ * Shared setup for testing DeleteCore admin API with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant core directory.
+ */
+ private String setupCoreRemnantForUnloadCoreOperation(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ Replica primaryReplica = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ JettySolrRunner primaryJetty = cluster.getReplicaJetty(primaryReplica);
+ String originalCoreName = primaryReplica.getCoreName();
+ Path remnantInstanceDir;
+ try (SolrCore core = primaryJetty.getCoreContainer().getCore(originalCoreName)) {
+ CoreDescriptor cd = core.getCoreDescriptor();
+ remnantInstanceDir = cd.getInstanceDir();
+ }
+
+ CollectionAdminRequest.deleteCollection(collectionName).process(cluster.getSolrClient());
+ waitForState("Expected collection deletion", collectionName, (n, c) -> c == null);
+
+ // Simulate a core remnant still exists by creating the directory and core.properties
+ Files.createDirectories(remnantInstanceDir);
+ Files.writeString(
+ remnantInstanceDir.resolve("core.properties"),
+ "name=" + originalCoreName + "\n",
+ StandardCharsets.UTF_8);
+
+ return originalCoreName;
+ }
+
+ @Test
+ public void testCreateCollectionWithRemnantsFailsWithoutSetting() throws Exception {
+ assertNull(
+ "Property should not be set by default", System.getProperty(DELETE_UNKNOWN_CORES_PROP));
+
+ String collectionName = "coreRemnantCreateNoSetting";
+ setupCollectionRemnant(collectionName);
+
+ // Try to create the collection again - this demonstrates the behavior without the setting
+ // In typical environments, this might fail, but behavior depends on configuration
+ CollectionAdminRequest.Create recreateRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ recreateRequest.setCreateNodeSet(jettys.getFirst().getNodeName());
+
+ // The request to create a collection SHOULD fail based on the remnant file, if it does not it
+ // means we've changed Solr's behavior when creating a core and
+ // remnants exist, and therefore we should rethink the utility of this setting.
+ try {
+ recreateRequest.process(cluster.getSolrClient());
+ fail("This request to recreate the collection should have failed due to remnant files.");
+ } catch (Exception e) {
+ assertTrue(
+ "Verify the exception was due to core creation failed.",
+ e.getMessage().contains("Underlying core creation failed"));
+ }
+ }
+
+ @Test
+ public void testCreateCollectionWithRemnantsWithSetting() throws Exception {
+ System.setProperty(DELETE_UNKNOWN_CORES_PROP, "true");
+
+ String collectionName = "coreRemnantCreateWithSetting";
+ setupCollectionRemnant(collectionName);
+
+ // With the setting enabled, collection creation should succeed despite remnants
+ CollectionAdminRequest.Create recreateRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ recreateRequest.setCreateNodeSet(jettys.getFirst().getNodeName());
+ recreateRequest.process(cluster.getSolrClient());
Review Comment:
The error message assertion on line 215 is fragile. It checks for the exact string "ADDREPLICA failed to create replica", which could change in future Solr versions or may not be the exact message in all failure scenarios. Consider checking for a more general pattern or multiple possible error indicators to make the test more robust to message changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2660197461
##########
solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc:
##########
@@ -59,3 +59,14 @@ For instance, a client application may want to keep a log of which update reques
NOTE: In previous version of Solr, the `min_rf` parameter had to be specified to ask Solr for the achieved replication factor.
Now it is always included in the response.
+
+== Solr Core Lifecycle Inconsistency
Review Comment:
Until we have a number of tips & tricks, it'd be a pretty lame FAQ. And the number of such that might be applicable could be vast for Solr. And I wonder if anyone would bother to actually read the FAQ.
Another suitable page might be the collection creation or replica creation APIs.
But my preference remains for logging a warning with a reference to the setting. It's not either-or.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2575556326
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1451,6 +1455,14 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ if (Files.exists(cd.getInstanceDir())) {
Review Comment:
The error you are trying to avoid, coming out of coresLocate.create, seems like a relatively fine protection in case a valid core is there. Curious; does calling a core admin level API to delete the core work? I don't mean replica deletion.
##########
solr/packaging/test/test_create_collection2.bats:
##########
@@ -0,0 +1,62 @@
+#!/usr/bin/env bats
Review Comment:
I *really* don't think we should add a bats test for matters that JUnit could easily handle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2576529974
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1451,6 +1455,14 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ if (Files.exists(cd.getInstanceDir())) {
Review Comment:
i will take a looksee.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2659314424
##########
changelog/unreleased/SOLR-18008-simulate_solr_core_remnants.yml:
##########
@@ -0,0 +1,5 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+title: Enhance existing setting for removing unknown cores at startup to also apply when creating collections and adding replicas and remnant core files already exist.
+type: added # added, changed, fixed, deprecated, removed, dependency_update, security, other
Review Comment:
IMO we should use "added" conservatively. Even your title shows you aren't so much adding something as you are modifying an existing thing.
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
Review Comment:
We can assume one node, so we don't care what its name is later.
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.CoreAdminRequest;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
Review Comment:
Use `{@link #DELETE_UNKNOWN_CORES_PROP}` -- IntelliJ will help write such things with auto-completion. Or just don't bother being specific.
Moreover, I wouldn't be overly specific in javadoc, especially on tests. Documentation needs to be maintained; it isn't purely free/good (more isn't always better). This documentation should say less. You are including advise. This is a waste of time to say such things here; leave that to the ref guide.
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
Review Comment:
needless
##########
solr/solr-ref-guide/modules/configuration-guide/pages/solr-properties.adoc:
##########
@@ -44,7 +44,7 @@ NOTE: Properties marked with "!" indicate inverted meaning between pre Solr 10 a
|solr.cloud.prep.recovery.read.timeout.additional.ms|prepRecoveryReadTimeoutExtraWait|8000|Specifies additional milliseconds to wait during recovery read operations in SolrCloud mode.
-|solr.cloud.startup.delete.unknown.cores.enabled|solr.deleteUnknownCores|false|Controls whether unknown cores are deleted at startup in SolrCloud mode.
+|solr.cloud.delete.unknown.cores.enabled|solr.deleteUnknownCores|false|Determines if unknown cores should be removed at startup and during collection and replica creation in SolrCloud mode.
Review Comment:
I view these things as separate:
* Deleting unexpected cores (according to ZK state.json as truth) on startup.
* Deleting unexpected preexisting core data on replica creation.
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ Replica primaryReplica = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ JettySolrRunner primaryJetty = cluster.getReplicaJetty(primaryReplica);
+ String originalCoreName = primaryReplica.getCoreName();
+ Path remnantInstanceDir;
+ try (SolrCore core = primaryJetty.getCoreContainer().getCore(originalCoreName)) {
+ CoreDescriptor cd = core.getCoreDescriptor();
+ remnantInstanceDir = cd.getInstanceDir();
+ }
+
+ CollectionAdminRequest.deleteCollection(collectionName).process(cluster.getSolrClient());
+ waitForState("Expected collection deletion", collectionName, (n, c) -> c == null);
+
+ // Simulate a core remnant still exists by creating the directory and core.properties
+ Files.createDirectories(remnantInstanceDir);
+ Files.writeString(remnantInstanceDir.resolve("core.properties"), "", StandardCharsets.UTF_8);
+ }
+
+ @Test
+ public void testCreateCollectionWithRemnantsFailsWithoutSetting() throws Exception {
+ assertNull(
+ "Property should not be set by default", System.getProperty(DELETE_UNKNOWN_CORES_PROP));
+
+ String collectionName = "coreRemnantCreateNoSetting";
+ setupCollectionRemnant(collectionName);
+
+ // Try to create the collection again - this demonstrates the behavior without the setting
+ // In typical environments, this might fail, but behavior depends on configuration
+ CollectionAdminRequest.Create recreateRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ recreateRequest.setCreateNodeSet(jettys.getFirst().getNodeName());
+
+ // The request to create a collection SHOULD fail based on the remnant file, if it does not it
+ // means we've changed Solr's behavior when creating a core and
+ // remnants exist, and therefore we should rethink the utility of this setting.
+ try {
Review Comment:
See `assertThrows` for this well known pattern
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
Review Comment:
why specify this?
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,17 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
Review Comment:
I thought you were very well aware of EnvUtils
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ Replica primaryReplica = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ JettySolrRunner primaryJetty = cluster.getReplicaJetty(primaryReplica);
+ String originalCoreName = primaryReplica.getCoreName();
Review Comment:
Instead you could simply call `org.apache.solr.core.CoreContainer#getAllCoreNames` and grab the first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2659748313
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.CoreAdminRequest;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
Review Comment:
I think part of why I wrote this is because we don't actually have anything in the ref guide beyond a short mention in `solr-properties.adoc`, so it ended up here. I'll look for a place in Ref Guide to move this advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2576529028
##########
solr/packaging/test/test_create_collection2.bats:
##########
@@ -0,0 +1,62 @@
+#!/usr/bin/env bats
Review Comment:
I'm going to move it to junit. My challenge with the junit tests, and it might just be my knowledge, is that i sturggle to really feel like I understand how these junit tests really map to the real world. Whereas the bats tests map much closer. Maybe because I think from outside solr looking in, versus as a java coder, inside looking out.
For example, the logic of creating a fake directory to trigger the failure case. I get it in bats, and don't have a good sense of doing that in junit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2678283600
##########
changelog/unreleased/SOLR-18008-simulate_solr_core_remnants.yml:
##########
@@ -0,0 +1,9 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+title: Enhance existing setting for removing unknown cores at startup to also apply when creating collections and adding replicas and remnant core files already exist.
Review Comment:
False. Instead say the setting we added
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
Review Comment:
You have finally touched CoreContainer, Eric... achievement unlocked ;-)
##########
solr/core/src/java/org/apache/solr/core/CorePropertiesLocator.java:
##########
@@ -240,4 +242,23 @@ protected Properties buildCoreProperties(CoreDescriptor cd) {
p.putAll(cd.getPersistableUserProperties());
return p;
}
+
+ protected void checkForExistingCore(CoreDescriptor cd) {
+ if (Files.exists(cd.getInstanceDir())) {
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
Review Comment:
There's nothing SolrCloud about the condition leading us to this method, yet we've named the property constraining it apparently to SolrCloud. To reconcile that, lets modify the condition at instanceDir existence check to also that that the `cd` has a cloud descriptor. Or make the property more generic. I have no opinion. If you go the latter way, maybe name it `solr.admin.core.create.replaces.enabled`. Shrug. Also need to keep in sync with the warning log, ref guide page, changelog yml, test. Ugh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #3904:
URL: https://github.com/apache/solr/pull/3904#issuecomment-3730890011
> I still these this should be a separate setting than combined but I won't push the matter any further.
Thanks for the chat on the phone, and I split it up to a seperate setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2658951992
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
+ log.warn(
+ "There appears to be an existing directory for core {}, now deleting it",
+ cd.getName());
+ SolrCore.deleteUnloadedCore(cd, true, true);
Review Comment:
Ugh, I think I'm not going to do this one, as I don't know how far down the rabbit hole to go...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2673035195
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1512,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
Review Comment:
It took me two tries to understand your point, and then it clicked.... I made the change, and am running tests and then will push up...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2695066053
##########
solr/core/src/java/org/apache/solr/core/CorePropertiesLocator.java:
##########
@@ -240,4 +242,23 @@ protected Properties buildCoreProperties(CoreDescriptor cd) {
p.putAll(cd.getPersistableUserProperties());
return p;
}
+
+ protected void checkForExistingCore(CoreDescriptor cd) {
+ if (Files.exists(cd.getInstanceDir())) {
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
Review Comment:
yeah, I guess the only reason that it has `.cloud.` in the name is that we don't see this behavior in non cloud use cases, probably because non cloud users don't create/delete cores rapidly, at least most don't.. The whole "let me create/remove/move/delete cores" really feel slike a cloud feature (driven by alias use, adding replicates etc).
I will check and see about adding in the cloud descriptor check.. or go with the name you proposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2706264333
##########
solr/core/src/java/org/apache/solr/core/CorePropertiesLocator.java:
##########
@@ -240,4 +242,23 @@ protected Properties buildCoreProperties(CoreDescriptor cd) {
p.putAll(cd.getPersistableUserProperties());
return p;
}
+
+ protected void checkForExistingCore(CoreDescriptor cd) {
+ if (Files.exists(cd.getInstanceDir())) {
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
Review Comment:
Going with the the cloud descriptor check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2660202987
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
Review Comment:
the node count is irrelevant to this aspect being tested, so I wouldn't add complication for no benefit in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2658951296
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1451,6 +1455,14 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ if (Files.exists(cd.getInstanceDir())) {
Review Comment:
I added a unit test to investigate this ,and what I discovered is that the core admin level api won't even attempt to delete a remnant because it insists that you must have a core descriptor in ZK first, so it doesn't even get to there. I suppose this kind of make sense.
We could extend/augment the core admin level api with some sort of property that says "Do the delete even though you have no core descriptor" if we wanted to. However, I think the other way, of just handling it during create collection and add replica is simpler conceptually for a sys admin person! Sys admin type person doesn't want to get to the level of understanding what cores need to be deleted, that is the thrust of this entire feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2659738322
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
Review Comment:
I wondered about that as well, and then got the "well, maybe it is requried", I should have tried removing it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #3904:
URL: https://github.com/apache/solr/pull/3904#issuecomment-3708208815
Responded to all the feedback, I think this is ready to go in the next few days!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #3904:
URL: https://github.com/apache/solr/pull/3904#issuecomment-3702370284
Okay, two things.
1) If the JUnit test looks good, then I'll delete the bats test.
2) I'm thinking that maybe we want to slip into the current 10x release process a update to the system property name. In this PR i removed the `.startup.` from the name because it's not a startup ONLY thing. However in release 10.0 we moved the name from `solr.deleteUnknownCores` to `solr.cloud.startup.delete.unknown.cores.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2658952119
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.CoreAdminRequest;
+import org.apache.solr.client.solrj.response.json.JsonMapResponseParser;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ Replica primaryReplica = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ JettySolrRunner primaryJetty = cluster.getReplicaJetty(primaryReplica);
+ String originalCoreName = primaryReplica.getCoreName();
+ Path remnantInstanceDir;
+ try (SolrCore core = primaryJetty.getCoreContainer().getCore(originalCoreName)) {
+ CoreDescriptor cd = core.getCoreDescriptor();
+ remnantInstanceDir = cd.getInstanceDir();
+ }
+
+ CollectionAdminRequest.deleteCollection(collectionName).process(cluster.getSolrClient());
+ waitForState("Expected collection deletion", collectionName, (n, c) -> c == null);
+
+ // Simulate a core remnant still exists by creating the directory and core.properties
+ Files.createDirectories(remnantInstanceDir);
+ Files.writeString(remnantInstanceDir.resolve("core.properties"), "", StandardCharsets.UTF_8);
+ }
+
+ /**
+ * Shared setup for testing replica addition with remnants. Creates a collection, then simulates a
+ * remnant directory on the single node that will impact the next addReplica command.
+ */
+ private void setupReplicaRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ int nextReplicaIndex = 3; // Yep, from 1 to 3 due to how we count in ZK and setup.
+ String expectedNewReplicaName = collectionName + "_shard1_replica_n" + nextReplicaIndex;
+
+ // Simulate a core remnant on the single node adjacent to the existing replica instance path
+ Replica existing = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ try (SolrCore core =
+ cluster.getReplicaJetty(existing).getCoreContainer().getCore(existing.getCoreName())) {
+ Path siblingDir = core.getInstancePath().getParent().resolve(expectedNewReplicaName);
+ Files.createDirectories(siblingDir);
+ Files.writeString(
+ siblingDir.resolve("core.properties"),
+ "name="
+ + expectedNewReplicaName
+ + "_remnant\n"
+ + "collection="
+ + collectionName
+ + "_remnant\n"
+ + "shard=shard1\n"
+ + "coreNodeName=core_node_remnant\n",
+ StandardCharsets.UTF_8);
+ }
+ }
+
+ /**
+ * Shared setup for testing DeleteCore admin API with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant core directory.
+ */
+ private String setupCoreRemnantForUnloadCoreOperation(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
+ createRequest.process(cluster.getSolrClient());
+
+ waitForState(
+ "Expected collection to be fully active",
+ collectionName,
+ (n, c) -> SolrCloudTestCase.replicasForCollectionAreFullyActive(n, c, 1, 1));
+
+ Replica primaryReplica = getReplicaOnNode(collectionName, "shard1", primaryNode);
+ JettySolrRunner primaryJetty = cluster.getReplicaJetty(primaryReplica);
+ String originalCoreName = primaryReplica.getCoreName();
+ Path remnantInstanceDir;
+ try (SolrCore core = primaryJetty.getCoreContainer().getCore(originalCoreName)) {
+ CoreDescriptor cd = core.getCoreDescriptor();
+ remnantInstanceDir = cd.getInstanceDir();
+ }
+
+ CollectionAdminRequest.deleteCollection(collectionName).process(cluster.getSolrClient());
+ waitForState("Expected collection deletion", collectionName, (n, c) -> c == null);
+
+ // Simulate a core remnant still exists by creating the directory and core.properties
+ Files.createDirectories(remnantInstanceDir);
+ Files.writeString(
+ remnantInstanceDir.resolve("core.properties"),
+ "name=" + originalCoreName + "\n",
+ StandardCharsets.UTF_8);
+
+ return originalCoreName;
+ }
+
+ @Test
+ public void testCreateCollectionWithRemnantsFailsWithoutSetting() throws Exception {
+ assertNull(
+ "Property should not be set by default", System.getProperty(DELETE_UNKNOWN_CORES_PROP));
+
+ String collectionName = "coreRemnantCreateNoSetting";
+ setupCollectionRemnant(collectionName);
+
+ // Try to create the collection again - this demonstrates the behavior without the setting
+ // In typical environments, this might fail, but behavior depends on configuration
+ CollectionAdminRequest.Create recreateRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ recreateRequest.setCreateNodeSet(jettys.getFirst().getNodeName());
+
+ // The request to create a collection SHOULD fail based on the remnant file, if it does not it
+ // means we've changed Solr's behavior when creating a core and
+ // remnants exist, and therefore we should rethink the utility of this setting.
+ try {
+ recreateRequest.process(cluster.getSolrClient());
+ fail("This request to recreate the collection should have failed due to remnant files.");
+ } catch (Exception e) {
+ assertTrue(
+ "Verify the exception was due to core creation failed.",
+ e.getMessage().contains("Underlying core creation failed"));
+ }
+ }
+
+ @Test
+ public void testCreateCollectionWithRemnantsWithSetting() throws Exception {
+ System.setProperty(DELETE_UNKNOWN_CORES_PROP, "true");
+
+ String collectionName = "coreRemnantCreateWithSetting";
+ setupCollectionRemnant(collectionName);
+
+ // With the setting enabled, collection creation should succeed despite remnants
+ CollectionAdminRequest.Create recreateRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ recreateRequest.setCreateNodeSet(jettys.getFirst().getNodeName());
+ recreateRequest.process(cluster.getSolrClient());
Review Comment:
we have this everywhere...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2659734364
##########
solr/core/src/test/org/apache/solr/cloud/DeleteCoreRemnantsOnCreateTest.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.cloud;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Optional;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.core.CoreDescriptor;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Solr occasionally gets into an inconsistent state with its cores lifecycle where remnant files
+ * are left on disk after various operations that delete a core. Examples include deleting a
+ * collection operation that doesn't properly finish, or maybe the Solr process unexpectedly gets
+ * killed. The system property "solr.cloud.delete.unknown.cores.enabled" is an expert setting that
+ * when enabled automatically deletes any remnant core data on disk when new cores are created that
+ * would otherwise fail due to the preexisting files. You should be cautious in enabling this
+ * feature, as it means that something isn't working well in your Solr setup.
+ */
+public class DeleteCoreRemnantsOnCreateTest extends SolrCloudTestCase {
+ private static final String DELETE_UNKNOWN_CORES_PROP = "solr.cloud.delete.unknown.cores.enabled";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ configureCluster(1).addConfig("conf", configset("cloud-minimal")).configure();
+ }
+
+ @Before
+ public void resetProperty() {
+ System.clearProperty(DELETE_UNKNOWN_CORES_PROP);
+ }
+
+ /**
+ * Shared setup for testing collection creation with remnants. Creates a collection, deletes it,
+ * and then leaves behind a remnant directory.
+ */
+ private void setupCollectionRemnant(String collectionName) throws Exception {
+ List<JettySolrRunner> jettys = cluster.getJettySolrRunners();
+ String primaryNode = jettys.getFirst().getNodeName();
+
+ CollectionAdminRequest.Create createRequest =
+ CollectionAdminRequest.createCollection(collectionName, "conf", 1, 1);
+ createRequest.setCreateNodeSet(primaryNode);
Review Comment:
I actually removed it, and then started thinking "oh, but wait, what if we want to try with multiple nodes" and put it back... YAGNI applies here... You aint gonna need it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2659817051
##########
solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc:
##########
@@ -59,3 +59,14 @@ For instance, a client application may want to keep a log of which update reques
NOTE: In previous version of Solr, the `min_rf` parameter had to be specified to ask Solr for the achieved replication factor.
Now it is always included in the response.
+
+== Solr Core Lifecycle Inconsistency
Review Comment:
I appreciate you want to document this, and that this page isn't a terrible/obviously-wrong choice, but IMO it further degrades a very important page that should be more focused on how key aspects of SolrCloud works.
Perhaps what we should do is log a warning with this suggestion that will change. Maybe also reference the JIRA issue where an interested user can get more background / advice. After all, it's an obscure matter. WDYT?
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1512,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
Review Comment:
Shouldn't this logic be adjacent to the logic that fails in `CoreProperitesLocator.exists`? (I assume that's the spot). Doing so would be better organized and probably lead to combining in a more clear way with coordinated logging and existence check. I forget if this was discussed already
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2660079766
##########
solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc:
##########
@@ -59,3 +59,14 @@ For instance, a client application may want to keep a log of which update reques
NOTE: In previous version of Solr, the `min_rf` parameter had to be specified to ask Solr for the achieved replication factor.
Now it is always included in the response.
+
+== Solr Core Lifecycle Inconsistency
Review Comment:
humm... log a warning if we bump into the core remnant problem and you haven't enabled the feature, and reference that JIRA? That could work...
What do you think about a FAQ style page or just a new page about "tips and tricks with SolrCloud", would that be generic enough that this and other bits of advice could go there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh merged PR #3904:
URL: https://github.com/apache/solr/pull/3904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2674839325
##########
solr/core/src/java/org/apache/solr/core/CorePropertiesLocator.java:
##########
@@ -65,6 +66,7 @@ public CorePropertiesLocator(Path coreDiscoveryRoot) {
@Override
public void create(CoreContainer cc, CoreDescriptor... coreDescriptors) {
for (CoreDescriptor cd : coreDescriptors) {
+ checkForExistingCore(cd);
Path propertiesFile = cd.getInstanceDir().resolve(PROPERTIES_FILENAME);
if (Files.exists(propertiesFile))
throw new SolrException(
Review Comment:
This is fine but I want to clarify/remind that my last review suggested the move of the logic into here could be an opportunity to integrate with the existing logic. It means triggering the logs/errors based on core.properites instead of the parent dir (seems fine). Any way, this is fine if you wish.
##########
solr/core/src/java/org/apache/solr/core/CorePropertiesLocator.java:
##########
@@ -240,4 +242,23 @@ protected Properties buildCoreProperties(CoreDescriptor cd) {
p.putAll(cd.getPersistableUserProperties());
return p;
}
+
+ protected void checkForExistingCore(CoreDescriptor cd) {
+ if (Files.exists(cd.getInstanceDir())) {
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
+ if (deleteUnknownCores) {
+ log.warn(
+ "Automatically deleting existing directory at [{}] for core [{}] because solr.cloud.delete.unknown.cores.enabled is true",
+ cd.getInstanceDir().toAbsolutePath(),
+ cd.getName());
+ SolrCore.deleteUnloadedCore(cd, true, true);
+ } else {
+ log.warn(
+ "Directory at [{}] for core[{}] already exists preventing create operation. Set solr.cloud.delete.unknown.cores.enabled=true to delete directory. (SOLR-18008)",
Review Comment:
I would change "preventing" to "may prevent". As I look at the logic, it appears an empty dir may work since the code following `checkForExistingCore` only looks for core.properties existing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2672959436
##########
solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc:
##########
@@ -59,3 +59,14 @@ For instance, a client application may want to keep a log of which update reques
NOTE: In previous version of Solr, the `min_rf` parameter had to be specified to ask Solr for the achieved replication factor.
Now it is always included in the response.
+
+== Solr Core Lifecycle Inconsistency
Review Comment:
Okay, I moved the text to the create collection, and then I added a warning from solr when the situation is hit. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2660199089
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1512,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
Review Comment:
If CorePropertiesLocator was the beast (not CoreContainer), that would explain why you did what you did, but this is the inverse situation, and you chose to touch CoreContainer when you could have avoided it. Let's move it then; ehh? Any way, I'm here for peer review so don't be afraid ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2659746481
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,17 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
Review Comment:
I am, I think I just wasn't thinkig of it because I was following what was laready in CoreContainer as a pattern. SHould we mkae an effort to get everythign migrated to EnvUtils? We're kind of in the middle I guess on that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2695066053
##########
solr/core/src/java/org/apache/solr/core/CorePropertiesLocator.java:
##########
@@ -240,4 +242,23 @@ protected Properties buildCoreProperties(CoreDescriptor cd) {
p.putAll(cd.getPersistableUserProperties());
return p;
}
+
+ protected void checkForExistingCore(CoreDescriptor cd) {
+ if (Files.exists(cd.getInstanceDir())) {
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
Review Comment:
yeah, I guess the only reason that it has `.cloud.` in the name is that we don't see this behavior in non cloud use cases, probably because non cloud users don't create/delete cores rapidly, at least most don't.. The whole "let me create/remove/move/delete cores" really feel slike a cloud feature driven by alias and all of that.
I will check and see about adding in the cloud descriptor check.. or go with the name you proposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #3904:
URL: https://github.com/apache/solr/pull/3904#issuecomment-3730892611
After chatting with @dsmiley on the phone, with the last few commits this is now ready for a merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh closed pull request #3904: SOLR-18008: Remnant core files prevent Collections for working
URL: https://github.com/apache/solr/pull/3904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2655637838
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1451,6 +1455,14 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ if (Files.exists(cd.getInstanceDir())) {
Review Comment:
i still need to check on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2660081792
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1512,16 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ EnvUtils.getPropertyAsBool("solr.cloud.delete.unknown.cores.enabled", false);
+ if (deleteUnknownCores && Files.exists(cd.getInstanceDir())) {
Review Comment:
Maybe... Honestly, in this PR, I was attempting to make minimal changes to CoreContainer as it's a beast already and NOT an area i have worked much in the past. I don't have a good overall understanding of it as this PR is my first time actually touching it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18008: Remnant core files prevent Collections for working [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3904:
URL: https://github.com/apache/solr/pull/3904#discussion_r2660201891
##########
solr/core/src/java/org/apache/solr/core/CoreContainer.java:
##########
@@ -1511,6 +1511,17 @@ public SolrCore create(
preExistingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
+ final boolean deleteUnknownCores =
+ Boolean.parseBoolean(
+ System.getProperty("solr.cloud.delete.unknown.cores.enabled", "false"));
Review Comment:
Ah, it's fair to minimally touch the line if you prefer then.
Personally I'm not too motivated to do a big EnvUtils conversion; I'll leave that to others if they wish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org