You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/01/21 05:12:13 UTC
[GitHub] [solr] risdenk opened a new pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
risdenk opened a new pull request #553:
URL: https://github.com/apache/solr/pull/553
https://issues.apache.org/jira/browse/SOLR-15942
# Description
Upgrade Hadoop to 3.3.1
The `org.apache.hadoop` file changes are related to updating those classes to match Apache Hadoop 3.3.1 from the older versions.
The `transitive=false` removals happened since Hadoop changed a few dependencies which including shading Guava. I'm not sure all these changes are necessary - but it seemed a lot cleaner. For `testImplementation` I didn't see an issue with removing `transitive=false` since those should only be used during tests and not in the final build.
# Tests
```
./gradlew test --tests "*hdfs*" --tests "*Hdfs*" --tests "*HDFS*" --tests "*Hadoop*" -Ptests.jvms=4 -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.nightly=true -Ptests.awaitsfix=false -Ptests.badapples=false -Ptests.file.encoding=UTF-8
```
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [x] I have created a Jira issue and added the issue ID to my pull request title.
- [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [x] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
madrob commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1023539913
> I hope you can go through the dependency tree to use the knowledge/intuition you have (or other x-Cloudera folks; whoever knows HDFS) to prune out stuff that you think is suspicious
This is difficult because in the tests we are not just using a client and a mock HDFS service, we end up spinning up the full (albeit mini) DFS stack. Which carries its own internal communication channels and libraries, etc.
> Maybe we should delay this PR until after the module separation to be absolutely sure we only explode the dependencies in a module and not here in core
Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1021461272
@madrob think I addressed your concerns. Removed copy of `HttpServer2` since not needed anymore, updated `package-info.java` and fixed comment.
@janhoy still working on `Do you perhaps have a list of actual added runtime deps/jars as well?`. I'm slowly playing with the Hadoop dependencies to see what can be done here.
Also been talking with @warperwolf about getting PR #324 in and how it relates to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1035162389
So been working through the reviews and cleaning up some changes:
* Removed HADOOP-15681 workaround based on comment from @madrob in the jira
* Enabled test that was disabled due to HADOOP-11077 since that was fixed
TODO
* jar license errors
* error from ecjLint as reported by @madrob
* Working through running tests
There was a comment from Mike about waiting until Hadoop 3.3.2, but it looks like 3.3.2 release has stalled for the past ~10 days after RC3 failed - https://lists.apache.org/thread/tt9to5y6lj31gslg579bm7hy0hwm5d5n
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1022521366
@dsmiley agree about trying to reduce dependencies, but at the same time `transitive=false` requires a whole bunch of knowledge about how the code is being used. I think it makes sense to remove `transitive=false` in as many cases as possible and not try to exclude in every little case of a jar not being used. I understand this is sensitive in "core", but once #324 is merged, I think it would make sense to remove `transitive=false`.
I'm currently looking into this a bit more especially in the contrib/modules where a lot of the `transitive=false` means we have constraints in `versions.props` that we don't need to keep track of any more. I think overall it simplifies our dependency tracking story and makes it a lot easier to reason about what is going on.
I also have a POC about moving to Hadoop shaded client to avoid a ton of the dependency nightmare that HDFS has been. This should also happen post moving HDFS support to contrib/module. The hadoop auth stuff can't easily use the shaded dependencies, but hadoop auth stuff in Solr is relatively contained compared to the HDFS dependencies.
@dsmiley do you agree that once HDFS is a contrib/module - then don't need to worry as much about the exclusions? I'm perfectly ok waiting until PR #324 is merged to finish this clean up. I think that PR #324 is pretty close to mergable based on talking with @warperwolf - he had some improvements about test configs there, but I don't think we should keep kicking the can down the road on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk edited a comment on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk edited a comment on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1035162389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r805171301
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11219 (Hadoop and Netty 4)
Review comment:
I bumped up the thread leak linger to minimum 1 second to allow this to shut down. I haven't seen this failure since.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r809420774
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11219 (Hadoop and Netty 4)
+ return true;
+ } else if (name.startsWith("Command processor")) { // HDFS-14997
Review comment:
I don't know I removed that line and tests seem to pass. Not sure why I was seeing that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r805171301
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11219 (Hadoop and Netty 4)
Review comment:
I bumped up the thread leak linger to minimum 1 second to allow this to shut down. I haven't seen this failure since.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1044604844
@madrob / @dsmiley I think I've resolved all the comments for this. Hadoop 3.3.2 still hasn't been released. I think it would be better to get this merged in for 9.0 and handle the 3.3.2 upgrade separately (potentially for 9.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1018189921
I need to review the `check` output and fix any license file issues. I figured it made sense to get this reviewed early - especially by @dsmiley with the dependency/transitive changes.
This is most lkely the list of "added" dependencies - either for tests or in general:
```
Execution failed for task ':solr:core:validateJarLicenses'.
> Certain license/ notice files are missing:
- License file missing ('com.jcraft:jsch:0.1.55'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jsch-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('com.sun.jersey:jersey-core:1.19'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jersey-core-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jersey-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('com.sun.jersey:jersey-json:1.19'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jersey-json-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jersey-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('com.sun.jersey:jersey-server:1.19'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jersey-server-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jersey-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('com.sun.xml.bind:jaxb-impl:2.2.3-1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jaxb-impl-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jaxb-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('com.thoughtworks.paranamer:paranamer:2.3'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/paranamer-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('commons-beanutils:commons-beanutils:1.9.4'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/commons-beanutils-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/commons-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('commons-daemon:commons-daemon:1.0.13'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/commons-daemon-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/commons-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('commons-net:commons-net:3.6'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/commons-net-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/commons-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('dnsjava:dnsjava:2.1.7'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/dnsjava-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('io.netty:netty-all:4.1.68.Final'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/netty-all-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/netty-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('io.netty:netty:3.10.6.Final'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/netty-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('javax.activation:activation:1.1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/activation-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('javax.servlet.jsp:jsp-api:2.1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jsp-api-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jsp-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('javax.ws.rs:jsr311-api:1.1.1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jsr311-api-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jsr311-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('javax.xml.bind:jaxb-api:2.2.2'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jaxb-api-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jaxb-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('javax.xml.stream:stax-api:1.0-2'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/stax-api-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/stax-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.apache.avro:avro:1.7.7'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/avro-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/hadoop-shaded-guava-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/hadoop-shaded-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/hadoop-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/hadoop-shaded-protobuf_3_7-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/hadoop-shaded-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/hadoop-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.apache.kerby:kerby-xdr:1.0.1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/kerby-xdr-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/kerby-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.apache.kerby:token-provider:1.0.1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/token-provider-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/token-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.codehaus.jackson:jackson-jaxrs:1.9.2'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jackson-jaxrs-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jackson-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.codehaus.jackson:jackson-mapper-asl:1.9.13'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jackson-mapper-asl-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jackson-mapper-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jackson-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.codehaus.jackson:jackson-xc:1.9.2'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jackson-xc-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/jackson-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.codehaus.jettison:jettison:1.1'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/jettison-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
- License file missing ('org.fusesource.leveldbjni:leveldbjni-all:1.8'), expected it at: /Users/risdenk/repos/apache/solr/solr/licenses/leveldbjni-all-LICENSE-[type].txt or /Users/risdenk/repos/apache/solr/solr/licenses/leveldbjni-LICENSE-[type].txt, where [type] can be any of [ASL, BSD, BSD_LIKE, CDDL, CPL, EPL, MIT, MPL, PD, SUN, COMPOUND].
```
PS - it would be nice if this printed out if these were used for tests or not, but I think its possible to figure that out from the version lock or somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
madrob commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1018607670
We shouldn't need https://github.com/apache/solr/blob/ebc9e00fa331f52db6373e6e0c9af6adbe217bf1/solr/core/src/test/org/apache/hadoop/http/HttpServer2.java anymore, right? Since that was an update to deal with Jetty 9.4
Also, probably need to update the comments in https://github.com/apache/solr/blob/main/solr/core/src/test/org/apache/hadoop/package-info.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
madrob commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1021553232
@risdenk I get several failures when trying to build this - jar license errors, and then also this one which is pretty weird from `ecjLint`:
```
import javax.xml.stream.XMLInputFactory;
^^^^^^^^^^^^^^^^
The package javax.xml.stream is accessible from more than one module: <unnamed>, java.xml
```
I think the new dependencies are adding an additional xml library that causes conflicts? Maybe we need excludes of some flavor.
I ran the tests, it looks like security manager is enabled by default, so thankfully that bit is happy with your changes.
Manually reviewed the changes you made to our forked classes, comparing them with current 3.3.1 released classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1026217867
Now that SOLR-14660 has been merged - working on cleaning this up. I put it back in draft for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk edited a comment on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk edited a comment on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1035162389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r802987077
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11219 (Hadoop and Netty 4)
Review comment:
FWIW globalEventExecutor is leaking in a few HDFS tests - without this change being merged in yet. I haven't looked into it just saw the jenkins emails go by.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
madrob commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r792068327
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11219 (Hadoop and Netty 4)
+ return true;
+ } else if (name.startsWith("Command processor")) { // HDFS-14997
Review comment:
Why does this leak, it's a daemon thread? https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java#L1309
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11219 (Hadoop and Netty 4)
Review comment:
This... shouldn't leak? Why does it leak? According to https://netty.io/4.0/api/io/netty/util/concurrent/GlobalEventExecutor.html it will clean itself up after a second of inactivity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
dsmiley commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1022557589
I basically agree. That said, I hope you can go through the dependency tree to use the knowledge/intuition you have (or other x-Cloudera folks; whoever knows HDFS) to prune out stuff that you think is suspicious (not actually a needed dependency). For the test side, I do not agree -- if tests pass without a test-only dependency, then lets not include that depencency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
madrob commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r789752564
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11245
Review comment:
This JIRA claims it is only in 3.4.0, not 3.3.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
dsmiley commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1022559062
Maybe we should delay this PR until after the module separation to be absolutely sure we only explode the dependencies in a module and not here in core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1018268865
I wish we did not need to add licenses for test-dependencies (https://issues.apache.org/jira/browse/SOLR-15465), it just creates noise that makes it harder to understand what runtime dependencies we are actually adding in the binary distro.
Do you perhaps have a list of actual added runtime deps/jars as well?
Looks fishy to have different versions of `io.netty:netty` and `io.netty:netty-all`. Also, our build already contains tons of netty deps, so could be you can exclude netty-all and hadoop would still be happy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
madrob commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1035180957
Hadoop 3.3.2 appears to be waiting on https://github.com/apache/hadoop/pull/3940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r789842243
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11245
Review comment:
yea so I agree with you - I don't quite understand. I need to dig a bit more into this. It definitely seems related to Netty 4 in Hadoop though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1037254994
So I pushed changes that address most of the review comments so far. There is a big change that helps alleviate future Hadoop dependency issues. This uses the `hadoop-client-api`, `hadoop-client-runtime`, and `hadoop-client-minicluster` in the HDFS module which are shaded to avoid issues with Guava, protobuf, and other classes moving forward.
I have one thing to look at with https://github.com/apache/solr/pull/553#discussion_r792068327
Hadoop 3.3.2 RC4 is now being voted on https://lists.apache.org/thread/rk2msqvtb5bhlpbkjfox6gj8rqmnfb9z
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
dsmiley commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1021837229
When doing any non-trivial change to dependencies (as was done here), I strongly recommend reading what Dawid wrote: `help/dependencies.txt`. In particular, I find this useful:
gradlew -p solr/core dependencies --configuration runtimeClasspath
And here I found it useful to save this and do a diff of this output to compare with this PR and without this PR. **Wow**; tons of added stuff like okhttp and kotlin and lots of stuff. Is all this really needed? Honestly this is where your expertise comes into play. My only gauge is if tests pass without the dependency. I wish this dependency tree might be committed to source control so that if people mess with the dependencies, the impact would be shown in a useful way in the review process.
If you want to remove some particular transitive:false additions (as you did to many places), then do so one at a time where you then carefully observe the changes to the versions.lock file and dependency tree and ensure the additions are good (e.g. compile; run tests perhaps) before moving onto the next. Maybe even do a commit in-between. I found the `gw why --hash=deadbeef` command useful too. I did all this when I did the big dependency cleanup a few months ago. You may find added libs that we don't actually need; they should be excluded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk edited a comment on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk edited a comment on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1035162389
So been working through the reviews and cleaning up some changes:
* Removed HADOOP-15681 workaround based on comment from @madrob in the jira
* Enabled test that was disabled due to HADOOP-11077 since that was fixed
TODO
- [ ] jar license errors
- [ ] error from ecjLint as reported by @madrob
- [ ] BadHdfsThreadsFilter.java changes need a second look based on @madrob review
- [ ] Working through running tests
There was a comment from Mike about waiting until Hadoop 3.3.2, but it looks like 3.3.2 release has stalled for the past ~10 days after RC3 failed - https://lists.apache.org/thread/tt9to5y6lj31gslg579bm7hy0hwm5d5n
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk edited a comment on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk edited a comment on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1035162389
So been working through the reviews and cleaning up some changes:
* Removed HADOOP-15681 workaround based on comment from @madrob in the jira
* Enabled test that was disabled due to HADOOP-11077 since that was fixed
TODO
- [x] jar license errors
- [x] error from ecjLint as reported by @madrob
- [x] BadHdfsThreadsFilter.java changes need a second look based on @madrob review
- [x] Working through running tests
There was a comment from Mike about waiting until Hadoop 3.3.2, but it looks like 3.3.2 release has stalled for the past ~10 days after RC3 failed - https://lists.apache.org/thread/tt9to5y6lj31gslg579bm7hy0hwm5d5n
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #553:
URL: https://github.com/apache/solr/pull/553#issuecomment-1037254994
So I pushed changes that address most of the review comments so far. There is a big change that helps alleviate future Hadoop dependency issues. This uses the `hadoop-client-api`, `hadoop-client-runtime`, and `hadoop-client-minicluster` in the HDFS module which are shaded to avoid issues with Guava, protobuf, and other classes moving forward.
I have one thing to look at with https://github.com/apache/solr/pull/553#discussion_r792068327
Hadoop 3.3.2 RC4 is now being voted on https://lists.apache.org/thread/rk2msqvtb5bhlpbkjfox6gj8rqmnfb9z
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk merged pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk merged pull request #553:
URL: https://github.com/apache/solr/pull/553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r791983469
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11245
Review comment:
Updated the comment to point to Netty 4 Hadoop jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #553: SOLR-15942: Upgrade Hadoop to 3.3.1
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #553:
URL: https://github.com/apache/solr/pull/553#discussion_r794911327
##########
File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java
##########
@@ -41,6 +41,10 @@ public boolean reject(Thread t) {
return true;
} else if (name.startsWith("nioEventLoopGroup")) {
return true; // netty threads waiting for a web server stop confirm that won't happen
+ } else if (name.startsWith("globalEventExecutor")) { // HADOOP-11219 (Hadoop and Netty 4)
Review comment:
I think the issue is `ThreadLeakLingering` was adjusted to be really short (less than 1 second) for a bunch of the Hadoop tests and sometimes we run into `globalEventExecutor` not having been shut down yet.
See 1c80803a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org