You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "jnturton (via GitHub)" <gi...@apache.org> on 2023/08/11 13:22:24 UTC
[PR] DRILL-8436: Upgrade Hadoop 3.2.4 -> 3.3.6 (drill)
jnturton opened a new pull request, #2821:
URL: https://github.com/apache/drill/pull/2821
# [DRILL-8436](https://issues.apache.org/jira/browse/DRILL-8436): Upgrade Hadoop 3.2.4 -> 3.3.6
## Description
Hadoop is upgraded to 3.3.6. Jetty is upgraded to 9.4.51.v20230217.
## Documentation
N/A
## Testing
Existing unit tests, manual testing of Drill HTTP services.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on code in PR #2821:
URL: https://github.com/apache/drill/pull/2821#discussion_r1296126455
##########
exec/jdbc-all/pom.xml:
##########
@@ -102,6 +102,18 @@
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-server</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
Review Comment:
hadoop 3.3.6 uses my fork of jersey-json -- see https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.3.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on code in PR #2821:
URL: https://github.com/apache/drill/pull/2821#discussion_r1305095341
##########
exec/jdbc-all/pom.xml:
##########
@@ -679,86 +688,85 @@
<filter>
<artifact>*:*</artifact>
<excludes>
- <exclude>**/logback.xml</exclude>
Review Comment:
Another list that's been sorted alphabetically in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on PR #2821:
URL: https://github.com/apache/drill/pull/2821#issuecomment-1688246753
I've got the JDBC driver working by bundling a core-site.xml file in it that handles the relocation of org.apache.hadoop to oadd.org.apache.hadoop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton merged PR #2821:
URL: https://github.com/apache/drill/pull/2821
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on PR #2821:
URL: https://github.com/apache/drill/pull/2821#issuecomment-1690970313
> I thought I was reviewing the other PR for the library updates. Could we rebase this on master once that has been merged?
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on code in PR #2821:
URL: https://github.com/apache/drill/pull/2821#discussion_r1305092866
##########
contrib/storage-phoenix/pom.xml:
##########
@@ -29,9 +29,9 @@
<name>Drill : Contrib : Storage : Phoenix</name>
<properties>
- <phoenix.version>5.1.2</phoenix.version>
+ <phoenix.version>5.1.3</phoenix.version>
<!-- Keep the 2.4.2 to reduce dependency conflict -->
Review Comment:
@cgivre I need to reword this comment slightly and prune some commented sections from java-exec/pom.xml below, but are you happy with this PR at this stage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on code in PR #2821:
URL: https://github.com/apache/drill/pull/2821#discussion_r1296649086
##########
exec/jdbc-all/pom.xml:
##########
@@ -102,6 +102,18 @@
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-server</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
Review Comment:
Ah! Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "cgivre (via GitHub)" <gi...@apache.org>.
cgivre commented on PR #2821:
URL: https://github.com/apache/drill/pull/2821#issuecomment-1690588883
I thought I was reviewing the other PR for the library updates. Could we rebase this on master once that has been merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on code in PR #2821:
URL: https://github.com/apache/drill/pull/2821#discussion_r1305093910
##########
exec/jdbc-all/pom.xml:
##########
@@ -388,51 +396,52 @@
<include>*:*</include>
</includes>
<excludes>
- <exclude>io.protostuff:*</exclude>
Review Comment:
@cgivre All the diff noise here comes about because I decided to sort this list alphabetically in the hopes that contributors will keep it sorted from here on, making checking what's in it that much easier. I can revert the sorting if preferred though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on PR #2821:
URL: https://github.com/apache/drill/pull/2821#issuecomment-1682344122
I've just set this PR to Draft because I rediscovered a problem in the Drill JDBC driver. I'll paste a chat message I sent to @vvysotskyi a few months back below, to reveal the nature of the problem. I'm sure it's ultimately fixable, but I don't know of an elegant fix yet.
> Hi Vova! I decided to try upgrading Drill's Hadoop libs to 3.3.5. Things are working but there is a problem in the Drill JDBC fat jar. There, the shade plugin relocates Hadoop to underneath oadd as usual but now there are class names present in the core-default.xml file in hadoop-common.jar which are not updated by the shade plugin. The result is that the JDBC driver is broken. While the shade plugin can update some kinds of text config files, it doesn't appear that it can update arbitrary XML config like core-default.xml. I thought of including our own manually updated copy of core-default.xml in exec/jdbc-all/src/resources and trying to make sure the shade plugin picks that one instead of the one in hadoop-common.jar. My only reservation is that introducing this copy creates a maintenance burden for the future so I thought to ask you if you have any ideas...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on PR #2821:
URL: https://github.com/apache/drill/pull/2821#issuecomment-1680623578
So, this seems to work but not in JDK 8 😒
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] DRILL-8436: Upgrade Hadoop 3.2.4 → 3.3.6 (drill)
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #2821:
URL: https://github.com/apache/drill/pull/2821#issuecomment-1680885511
> So, this seems to work but not in JDK 8 😒
In JDK, it can't find io/netty/handler/codec/http/HttpRequest. Maybe, we need to add an explicit dependency on the io.netty:netty-codec-http jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org