You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "mlbiscoc (via GitHub)" <gi...@apache.org> on 2025/04/01 01:18:57 UTC
[PR] SOLR-16903: Nightly test failures from Path migration [solr]
mlbiscoc opened a new pull request, #3300:
URL: https://github.com/apache/solr/pull/3300
https://issues.apache.org/jira/browse/SOLR-16903
# Description
Tests on nightly failing.
# Solution
- `TestReplicationHandler` needs to check fileName from string
- `TestStressThreadBackup.java` `toList()` is immutable. Collect with `.collect(Collectors.toList())`
- `BaseDistributedSearchTestCase.java` seems with a nightly test, the file/directory already exists. Use `StandardCopyOption.REPLACE_EXISTING` to overwrite as `FileUtils.copyDirectory` does
- CustomTLogDirTest.java When using `LuceneTestCase.createTempDir()`, certain seeds generate a [WindowsFS](https://github.com/apache/lucene/blob/main/lucene/test-framework/src/java/org/apache/lucene/tests/mockfile/WindowsFS.java#L34) and this causes try-with-resources failing to [close the file stream](https://github.com/apache/solr/blob/470ae762b883b259da6f127844855395e9eedf9f/solr/core/src/test/org/apache/solr/update/CustomTLogDirTest.java#L166). Unwrap it to just get a generic NIO Path so stream can close
# Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
# Checklist
Please review the following and check all that apply:
- [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
- [ ] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-16903: Nightly test failures from Path migration [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley merged PR #3300:
URL: https://github.com/apache/solr/pull/3300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-16903: Nightly test failures from Path migration [solr]
Posted by "mlbiscoc (via GitHub)" <gi...@apache.org>.
mlbiscoc commented on code in PR #3300:
URL: https://github.com/apache/solr/pull/3300#discussion_r2023738100
##########
solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java:
##########
@@ -406,45 +407,35 @@ public void testUpConfigLinkConfigClearZk() throws Exception {
assertEquals("Comparing downloaded files to what is in ZK", files.size(), zkFiles.size());
}
- Path sourceConfDir = ExternalPaths.TECHPRODUCTS_CONFIGSET;
- // filter out all directories starting with . (e.g. .svn)
- try (Stream<Path> stream = Files.walk(sourceConfDir)) {
- List<Path> files =
- stream
- .filter(Files::isRegularFile)
- .filter(path -> path.getFileName().startsWith("."))
Review Comment:
Thanks for fixing this test. I need to be more mindful when modifying a block this big. I agree startsWith is very confusing with path for migration and the same issue was ran into in another part of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-16903: Nightly test failures from Path migration [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3300:
URL: https://github.com/apache/solr/pull/3300#discussion_r2022994072
##########
solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java:
##########
@@ -376,26 +377,26 @@ public void testUpConfigLinkConfigClearZk() throws Exception {
assertEquals(confsetname, collectionProps.getStr("configName"));
// test down config
- Path configSetDir =
- tmpDir.resolve(
- "solrtest-confdropspot-" + this.getClass().getName() + "-" + System.nanoTime());
- assertFalse(Files.exists(configSetDir));
+ Path destDir =
Review Comment:
A clearer name, as this test deals with many configSetDirs, as input and output.
##########
solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java:
##########
@@ -406,45 +407,35 @@ public void testUpConfigLinkConfigClearZk() throws Exception {
assertEquals("Comparing downloaded files to what is in ZK", files.size(), zkFiles.size());
}
- Path sourceConfDir = ExternalPaths.TECHPRODUCTS_CONFIGSET;
- // filter out all directories starting with . (e.g. .svn)
- try (Stream<Path> stream = Files.walk(sourceConfDir)) {
- List<Path> files =
- stream
- .filter(Files::isRegularFile)
- .filter(path -> path.getFileName().startsWith("."))
Review Comment:
There were a comedy of errors in this test; the "worst" one was here, having multiple flaws. Firstly, `filter` chooses those matching the condition; someone forgot to invert this! Thus the test didn't loop over anything. Secondly, the interpretation of this method is for "." to be a path, since we are calling Path.startsWith *not* String.startsWith. IMO the JDK developers chose a terribly confusing name that is forbidden-api worthy using "startsWith" taking a String. So any way, this ends up sort of kind of being, filter to paths that are under the current directory as that's what "." resolves to (CWD)... but only if normalize().toAbsolute() were to be called; which it wasn't and we don't want any way since this wasn't the actual intention.
In the end I dropped the intended filter since I don't see that we would encounter a dot-file.
CC @epugh (original author) @mlbiscoc (latest author)
##########
solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java:
##########
@@ -406,45 +407,35 @@ public void testUpConfigLinkConfigClearZk() throws Exception {
assertEquals("Comparing downloaded files to what is in ZK", files.size(), zkFiles.size());
}
- Path sourceConfDir = ExternalPaths.TECHPRODUCTS_CONFIGSET;
- // filter out all directories starting with . (e.g. .svn)
- try (Stream<Path> stream = Files.walk(sourceConfDir)) {
- List<Path> files =
- stream
- .filter(Files::isRegularFile)
- .filter(path -> path.getFileName().startsWith("."))
- .toList();
- files.forEach(
- (sourceFile) -> {
- int indexOfRelativePath =
- sourceFile
- .toString()
- .lastIndexOf(
- "sample_techproducts_configs"
- + sourceFile.getFileSystem().getSeparator()
- + "conf");
- String relativePathofFile = sourceFile.toString().substring(indexOfRelativePath + 33);
Review Comment:
I should have caught this in a code review, as it screams of pre-Path thinking. It's highly suspicious to be doing string.indexOf when we transition to the Path API. My replacement uses path.relativize()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-16903: Nightly test failures from Path migration [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #3300:
URL: https://github.com/apache/solr/pull/3300#discussion_r2023028541
##########
solr/core/src/test/org/apache/solr/cli/ZkSubcommandsTest.java:
##########
@@ -406,45 +407,35 @@ public void testUpConfigLinkConfigClearZk() throws Exception {
assertEquals("Comparing downloaded files to what is in ZK", files.size(), zkFiles.size());
}
- Path sourceConfDir = ExternalPaths.TECHPRODUCTS_CONFIGSET;
- // filter out all directories starting with . (e.g. .svn)
- try (Stream<Path> stream = Files.walk(sourceConfDir)) {
- List<Path> files =
- stream
- .filter(Files::isRegularFile)
- .filter(path -> path.getFileName().startsWith("."))
- .toList();
- files.forEach(
- (sourceFile) -> {
- int indexOfRelativePath =
- sourceFile
- .toString()
- .lastIndexOf(
- "sample_techproducts_configs"
- + sourceFile.getFileSystem().getSeparator()
- + "conf");
- String relativePathofFile = sourceFile.toString().substring(indexOfRelativePath + 33);
- Path downloadedFile = confDir.resolve(relativePathofFile);
- if (ConfigSetService.UPLOAD_FILENAME_EXCLUDE_PATTERN
- .matcher(relativePathofFile)
- .matches()) {
- assertFalse(
- sourceFile + " exists in ZK, downloaded:" + downloadedFile,
- Files.exists(downloadedFile));
- } else {
- assertTrue(
- downloadedFile + " does not exist source:" + sourceFile,
- Files.exists(downloadedFile));
- try {
- assertEquals(
- relativePathofFile + " content changed",
- -1,
- Files.mismatch(sourceFile, downloadedFile));
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
- });
+ // compare the original source with the downloaded destination
+ try (Stream<Path> stream = Files.walk(ExternalPaths.TECHPRODUCTS_CONFIGSET)) {
+ stream
+ .filter(Files::isRegularFile)
+ .forEach(
+ (sourceFile) -> {
+ Path sourceFileRelative =
+ ExternalPaths.TECHPRODUCTS_CONFIGSET.relativize(sourceFile);
+ Path downloadedFile = destDir.resolve("conf").resolve(sourceFileRelative);
Review Comment:
this was a huge mistake I fixed. The source and destination were the same! The destination *should* have been at the temp dir but it was a generically worded "confDir" that just so happened to be `ExternalPaths.TECHPRODUCTS_CONFIGSET` (the source) (!). Thus the test was very flawed if it even ran, but it didn't as I explained all files were filtered out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org