You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/01/14 00:57:56 UTC
[GitHub] [solr] risdenk opened a new pull request #528: SOLR-15405: Make HDFS Directory handle little endian Lucene LUCENE-9047
risdenk opened a new pull request #528:
URL: https://github.com/apache/solr/pull/528
https://issues.apache.org/jira/browse/SOLR-15405
# Description
Uses Lucene `ByteArrayDataInput` as inspiration.
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/ByteArrayDataInput.java
# Solution
`CustomBufferedIndexInput` needs to ensure that
it handles LUCNE-9047 changes for little
endianess. This replaces the existing methods
with little endian implementations.
# Tests
Working through running all the HDFS tests.
# Checklist
Please review the following and check all that apply:
- [X] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [X] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [X] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #528: SOLR-15405: Make HDFS Directory handle little endian Lucene LUCENE-9047
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #528:
URL: https://github.com/apache/solr/pull/528#issuecomment-1012691706
@dsmiley copied almost exactly from:
Uses Lucene `ByteArrayDataInput` and `ByteBuffersDataInput` as inspiration.
* https://github.com/apache/lucene/blob/releases/lucene%2F9.0.0/lucene/core/src/java/org/apache/lucene/store/ByteArrayDataInput.java
* https://github.com/apache/lucene/blob/releases/lucene%2F9.0.0/lucene/core/src/java/org/apache/lucene/store/ByteBuffersDataInput.java#L212
minor changes to use the byte array that existed previously. The diff from https://github.com/apache/lucene/commit/b84e0c272b6e1beb29e2a22d16d1d8a2bd05a731#diff-81cd76910207cb8480920e2a4d746617f8c31c320be57542f753cab3cb6f8c3f matches almost exactly the previous code so it wasn't a bad replacement at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #528: SOLR-15405: Make HDFS handle little endian Lucene LUCENE-9047
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #528:
URL: https://github.com/apache/solr/pull/528#issuecomment-1017785616
The bit shift logic should be go away due to LUCENE-10366 and https://github.com/apache/lucene/pull/592 - I plan to do this as a follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk merged pull request #528: SOLR-15405: Make HDFS handle little endian LUCENE-9047
Posted by GitBox <gi...@apache.org>.
risdenk merged pull request #528:
URL: https://github.com/apache/solr/pull/528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #528: SOLR-15405: Make HDFS Directory handle little endian Lucene LUCENE-9047
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #528:
URL: https://github.com/apache/solr/pull/528#issuecomment-1013474272
I've run
```
./gradlew test --tests "*hdfs*" --tests "*Hdfs*" --tests "*HDFS*" --tests "*Hadoop*" -Ptests.jvms=4 -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.nightly=true -Ptests.awaitsfix=false -Ptests.badapples=false -Ptests.file.encoding=UTF-8
```
a bunch (10?) of times and haven't gotten many failures. The failures have been varied with the current set of failures below from a few different runs:
```
- org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove (:solr:core)
Test output: /Users/risdenk/repos/apache/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.MoveReplicaHDFSTest.txt
Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove" -Ptests.jvms=4 -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=5BCDC1F5C3A460AF -Ptests.nightly=true -Ptests.awaitsfix=true -Ptests.badapples=false -Ptests.file.encoding=UTF-8
- org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth.testForwarding (:solr:core)
Test output: /Users/risdenk/repos/apache/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth.txt
Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth.testForwarding" -Ptests.jvms=4 -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=5BCDC1F5C3A460AF -Ptests.nightly=true -Ptests.awaitsfix=true -Ptests.badapples=false -Ptests.file.encoding=UTF-8
- org.apache.solr.core.backup.repository.HdfsBackupRepositoryIntegrationTest.classMethod (:solr:core)
Test output: /Users/risdenk/repos/apache/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.core.backup.repository.HdfsBackupRepositoryIntegrationTest.txt
Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.core.backup.repository.HdfsBackupRepositoryIntegrationTest.classMethod" -Ptests.jvms=4 -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=FFAA80532AC84F42 -Ptests.nightly=true -Ptests.awaitsfix=false -Ptests.badapples=false -Ptests.file.encoding=UTF-8
- org.apache.solr.core.backup.repository.HdfsBackupRepositoryIntegrationTest.classMethod (:solr:core)
Test output: /Users/risdenk/repos/apache/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.core.backup.repository.HdfsBackupRepositoryIntegrationTest.txt
Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.core.backup.repository.HdfsBackupRepositoryIntegrationTest.classMethod" -Ptests.jvms=4 -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=5175C707E497BF0A -Ptests.nightly=true -Ptests.awaitsfix=false -Ptests.badapples=false -Ptests.file.encoding=UTF-8
```
I haven't looked into this yet, but the "negative position" errors are definitely gone and haven't seen failures for the originally reported issue. A few say `classMethod` so I'm guessing its thread leak related or something like that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #528: SOLR-15405: Make HDFS handle little endian LUCENE-9047
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #528:
URL: https://github.com/apache/solr/pull/528#issuecomment-1017800768
Rebased on main, updated the commit message, and added `solr/CHANGES.txt`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #528: SOLR-15405: Make HDFS Directory handle little endian Lucene LUCENE-9047
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #528:
URL: https://github.com/apache/solr/pull/528#issuecomment-1013481565
The byte shift logic looks to be going away in LUCENE-10376 / https://github.com/apache/lucene/pull/602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #528: SOLR-15405: Make HDFS Directory handle little endian Lucene LUCENE-9047
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #528:
URL: https://github.com/apache/solr/pull/528#issuecomment-1012653842
I'm still figuring out some of the gradle build process for testing so will need to do more checks before this is fully ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org