You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2021/12/17 09:05:30 UTC
[GitHub] [bigtop] elukey opened a new pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
elukey opened a new pull request #844:
URL: https://github.com/apache/bigtop/pull/844
This commit is based on the work done in BIGTOP-3615
for Hive 3.x. In HIVE-25795 there seem to be no
indication that upstream wants to fix branch-2.3 too,
so this change should be enough for the moment to secure
Hive users of Bigtop 1.5.
Caveats:
- The Hive branch-2.3 is now at version 2.3.10 (in development),
that is far from our 2.3.6. I don't think it is needed to bump
Hive to 2.3.9/10 at the moment, but it can be discussed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey edited a comment on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey edited a comment on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-996551970
Built on Debian 10 x86:
```
$ dpkg -c output/hive/hive_2.3.6-3_all.deb | grep -e log4j.*.jar
-rw-r--r-- root/root 207875 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-1.2-api-2.16.0.jar
-rw-r--r-- root/root 301861 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-api-2.16.0.jar
-rw-r--r-- root/root 1789526 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-core-2.16.0.jar
-rw-r--r-- root/root 22539 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-jul-2.5.jar
-rw-r--r-- root/root 24229 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-slf4j-impl-2.16.0.jar
-rw-r--r-- root/root 35324 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-web-2.16.0.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey edited a comment on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey edited a comment on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997348608
@iwasakims thanks a lot for the patience in reviewing, my experience with maven is not a lot :)
Your suggestions make sense, I am going to amend the patch and re-test, will report asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997203062
Forcing the dependency in various pom.xml files like the following seems to lead to positive results:
```
<dependency>
<groupId>com.lmax</groupId>
<artifactId>disruptor</artifactId>
<version>${disruptor.version}</version>
</dependency>
```
Will rebuild and test (now I have a working docker provisioner!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on a change in pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #844:
URL: https://github.com/apache/bigtop/pull/844#discussion_r771907105
##########
File path: bigtop-packages/src/common/hive/patch7-log4j2-2.16.0.diff
##########
@@ -0,0 +1,167 @@
+diff --git a/bin/hive-config.sh b/bin/hive-config.sh
+index d52b84eb5f..8381a25a05 100644
+--- a/bin/hive-config.sh
++++ b/bin/hive-config.sh
+@@ -68,3 +68,7 @@ export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH
+
+ # Default to use 256MB
+ export HADOOP_HEAPSIZE=${HADOOP_HEAPSIZE:-256}
++
++# Disable the JNDI. This feature has critical RCE vulnerability.
++# when 2.x <= log4j.version <= 2.14.1
++export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dlog4j2.formatMsgNoLookups=true"
+diff --git a/druid-handler/pom.xml b/druid-handler/pom.xml
+index 4af3d5f00b..349be6628d 100644
+--- a/druid-handler/pom.xml
++++ b/druid-handler/pom.xml
+@@ -223,6 +223,11 @@
+ <artifactId>calcite-druid</artifactId>
+ <version>${calcite.version}</version>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+ <!-- test inter-project -->
+ <dependency>
+ <groupId>junit</groupId>
+diff --git a/hbase-handler/pom.xml b/hbase-handler/pom.xml
+index 9b12f42651..0f009b6792 100644
+--- a/hbase-handler/pom.xml
++++ b/hbase-handler/pom.xml
+@@ -102,6 +102,11 @@
+ <artifactId>hbase-hadoop-compat</artifactId>
+ <version>${hbase.version}</version>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+
+ <!-- test inter-project -->
+ <dependency>
+diff --git a/jdbc/pom.xml b/jdbc/pom.xml
+index 45d9a733b4..d303de1645 100644
+--- a/jdbc/pom.xml
++++ b/jdbc/pom.xml
+@@ -109,6 +109,11 @@
+ <version>${hadoop.version}</version>
+ <optional>true</optional>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+ <!-- test inter-project -->
+ <dependency>
+ <groupId>junit</groupId>
+diff --git a/llap-server/pom.xml b/llap-server/pom.xml
+index d73c733d3e..3ee1765a02 100644
+--- a/llap-server/pom.xml
++++ b/llap-server/pom.xml
+@@ -230,6 +230,11 @@
+ <groupId>org.codehaus.jettison</groupId>
+ <artifactId>jettison</artifactId>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+
+ <!-- test inter-project -->
+ <dependency>
+diff --git a/pom.xml b/pom.xml
+index f91f7f43a6..d3ef636c2b 100644
+--- a/pom.xml
++++ b/pom.xml
+@@ -98,7 +98,7 @@
+ <maven.install.plugin.version>2.4</maven.install.plugin.version>
+ <maven.jar.plugin.version>2.4</maven.jar.plugin.version>
+ <maven.javadoc.plugin.version>2.4</maven.javadoc.plugin.version>
+- <maven.shade.plugin.version>2.4.3</maven.shade.plugin.version>
++ <maven.shade.plugin.version>3.2.0</maven.shade.plugin.version>
+ <maven.surefire.plugin.version>2.18.1</maven.surefire.plugin.version>
+ <maven.war.plugin.version>2.4</maven.war.plugin.version>
+ <maven.dependency.plugin.version>2.8</maven.dependency.plugin.version>
+@@ -143,7 +143,7 @@
+ <hamcrest.version>1.1</hamcrest.version>
+ <hbase.version>1.1.1</hbase.version>
+ <!-- required for logging test to avoid including hbase which pulls disruptor transitively -->
+- <disruptor.version>3.3.0</disruptor.version>
++ <disruptor.version>3.3.7</disruptor.version>
+ <hikaricp.version>2.5.1</hikaricp.version>
+ <!-- httpcomponents are not always in version sync -->
+ <httpcomponents.client.version>4.4</httpcomponents.client.version>
+@@ -173,7 +173,7 @@
+ <kryo.version>3.0.3</kryo.version>
+ <libfb303.version>0.9.3</libfb303.version>
+ <libthrift.version>0.9.3</libthrift.version>
+- <log4j2.version>2.6.2</log4j2.version>
++ <log4j2.version>2.16.0</log4j2.version>
+ <opencsv.version>2.3</opencsv.version>
+ <orc.version>1.3.4</orc.version>
+ <mockito-all.version>1.9.5</mockito-all.version>
+diff --git a/ql/pom.xml b/ql/pom.xml
+index 3677dc843e..2cdcac43aa 100644
+--- a/ql/pom.xml
++++ b/ql/pom.xml
+@@ -154,6 +154,11 @@
+ <artifactId>log4j-1.2-api</artifactId>
+ <version>${log4j2.version}</version>
+ </dependency>
++ <dependency>
++ <groupId>org.apache.logging.log4j</groupId>
++ <artifactId>log4j-core</artifactId>
++ <version>${log4j2.version}</version>
++ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+diff --git a/ql/src/java/org/apache/hadoop/hive/ql/log/NoDeleteRollingFileAppender.java b/ql/src/java/org/apache/hadoop/hive/ql/log/NoDeleteRollingFileAppender.java
+index be32f06e43..5d10233054 100644
+--- a/ql/src/java/org/apache/hadoop/hive/ql/log/NoDeleteRollingFileAppender.java
++++ b/ql/src/java/org/apache/hadoop/hive/ql/log/NoDeleteRollingFileAppender.java
+@@ -27,7 +27,7 @@
+ import org.apache.log4j.Layout;
+ import org.apache.log4j.helpers.CountingQuietWriter;
+ import org.apache.log4j.helpers.LogLog;
+-import org.apache.log4j.helpers.OptionConverter;
++import org.apache.logging.log4j.core.util.OptionConverter;
+ import org.apache.log4j.spi.LoggingEvent;
+
+ public class NoDeleteRollingFileAppender extends FileAppender {
+diff --git a/service/pom.xml b/service/pom.xml
+index aef22881fd..c247d38717 100644
+--- a/service/pom.xml
++++ b/service/pom.xml
+@@ -81,7 +81,12 @@
+ </exclusion>
+ </exclusions>
+ </dependency>
+- <!-- used by thrift generated code -->
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
++ <!-- used by thrift generated code -->
+ <dependency>
+ <groupId>commons-lang</groupId>
+ <artifactId>commons-lang</artifactId>
Review comment:
same as jdbc/pom.xml, hive-exec does not directly depend on disruptor. It is via hive-llap-server. I think we can omit the change of service/pom.xml.
```
[INFO] org.apache.hive:hive-service:jar:2.3.6
...
[INFO] +- org.apache.hive:hive-llap-server:jar:2.3.6:compile
...
[INFO] | +- org.apache.hbase:hbase-server:jar:1.5.0:compile
...
[INFO] | | \- com.lmax:disruptor:jar:3.3.0:compile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997359317
@iwasakims new version pushed, all the tests done with 2.17.0 (since a new cve came out sigh), seems working fine!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #844: BIGTOP-3619 - Upgrade hive 2.3.6 to log4j 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997566850
I merged this. Thanks, @elukey.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey edited a comment on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey edited a comment on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-996766671
@iwasakims thanks a lot for the test and the analysis, my docker provisioner is still not working properly :(
I'll try to revert the disruptor change, not sure if really needed or not (maybe be a leftover of a test).
Edit: now I get the problem, I didn't review carefully the stacktrace yesterday. So it seems that log4j needs a more up to date version of disruptor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997229434
Test done so far:
- started the docker-hadoop.sh script with hive/hadoop/yarn/hdfs/mapreduce on Bigtop 1.5 (release versions, no custom packages)
- From the /bigtop-home dir, I manually dpkg -i the new hive packages and restarted the hive server and metastore.
- Verified on logs, lsof, etc.. that the new log4 version was picked up.
- Tried the hive cli utility with SHOW DATABASES.
- Checked the logs for stacktraces etc.. but didn't find any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on a change in pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #844:
URL: https://github.com/apache/bigtop/pull/844#discussion_r771907420
##########
File path: bigtop-packages/src/common/hive/patch7-log4j2-2.16.0.diff
##########
@@ -0,0 +1,167 @@
+diff --git a/bin/hive-config.sh b/bin/hive-config.sh
+index d52b84eb5f..8381a25a05 100644
+--- a/bin/hive-config.sh
++++ b/bin/hive-config.sh
+@@ -68,3 +68,7 @@ export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH
+
+ # Default to use 256MB
+ export HADOOP_HEAPSIZE=${HADOOP_HEAPSIZE:-256}
++
++# Disable the JNDI. This feature has critical RCE vulnerability.
++# when 2.x <= log4j.version <= 2.14.1
++export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dlog4j2.formatMsgNoLookups=true"
+diff --git a/druid-handler/pom.xml b/druid-handler/pom.xml
+index 4af3d5f00b..349be6628d 100644
+--- a/druid-handler/pom.xml
++++ b/druid-handler/pom.xml
+@@ -223,6 +223,11 @@
+ <artifactId>calcite-druid</artifactId>
+ <version>${calcite.version}</version>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+ <!-- test inter-project -->
+ <dependency>
+ <groupId>junit</groupId>
+diff --git a/hbase-handler/pom.xml b/hbase-handler/pom.xml
+index 9b12f42651..0f009b6792 100644
+--- a/hbase-handler/pom.xml
++++ b/hbase-handler/pom.xml
+@@ -102,6 +102,11 @@
+ <artifactId>hbase-hadoop-compat</artifactId>
+ <version>${hbase.version}</version>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+
+ <!-- test inter-project -->
+ <dependency>
+diff --git a/jdbc/pom.xml b/jdbc/pom.xml
+index 45d9a733b4..d303de1645 100644
+--- a/jdbc/pom.xml
++++ b/jdbc/pom.xml
+@@ -109,6 +109,11 @@
+ <version>${hadoop.version}</version>
+ <optional>true</optional>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+ <!-- test inter-project -->
+ <dependency>
+ <groupId>junit</groupId>
+diff --git a/llap-server/pom.xml b/llap-server/pom.xml
+index d73c733d3e..3ee1765a02 100644
+--- a/llap-server/pom.xml
++++ b/llap-server/pom.xml
+@@ -230,6 +230,11 @@
+ <groupId>org.codehaus.jettison</groupId>
+ <artifactId>jettison</artifactId>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+
+ <!-- test inter-project -->
+ <dependency>
+diff --git a/pom.xml b/pom.xml
+index f91f7f43a6..d3ef636c2b 100644
+--- a/pom.xml
++++ b/pom.xml
+@@ -98,7 +98,7 @@
+ <maven.install.plugin.version>2.4</maven.install.plugin.version>
+ <maven.jar.plugin.version>2.4</maven.jar.plugin.version>
+ <maven.javadoc.plugin.version>2.4</maven.javadoc.plugin.version>
+- <maven.shade.plugin.version>2.4.3</maven.shade.plugin.version>
++ <maven.shade.plugin.version>3.2.0</maven.shade.plugin.version>
+ <maven.surefire.plugin.version>2.18.1</maven.surefire.plugin.version>
+ <maven.war.plugin.version>2.4</maven.war.plugin.version>
+ <maven.dependency.plugin.version>2.8</maven.dependency.plugin.version>
+@@ -143,7 +143,7 @@
+ <hamcrest.version>1.1</hamcrest.version>
+ <hbase.version>1.1.1</hbase.version>
+ <!-- required for logging test to avoid including hbase which pulls disruptor transitively -->
+- <disruptor.version>3.3.0</disruptor.version>
++ <disruptor.version>3.3.7</disruptor.version>
+ <hikaricp.version>2.5.1</hikaricp.version>
+ <!-- httpcomponents are not always in version sync -->
+ <httpcomponents.client.version>4.4</httpcomponents.client.version>
+@@ -173,7 +173,7 @@
+ <kryo.version>3.0.3</kryo.version>
+ <libfb303.version>0.9.3</libfb303.version>
+ <libthrift.version>0.9.3</libthrift.version>
+- <log4j2.version>2.6.2</log4j2.version>
++ <log4j2.version>2.16.0</log4j2.version>
+ <opencsv.version>2.3</opencsv.version>
+ <orc.version>1.3.4</orc.version>
+ <mockito-all.version>1.9.5</mockito-all.version>
+diff --git a/ql/pom.xml b/ql/pom.xml
+index 3677dc843e..2cdcac43aa 100644
+--- a/ql/pom.xml
++++ b/ql/pom.xml
+@@ -154,6 +154,11 @@
+ <artifactId>log4j-1.2-api</artifactId>
+ <version>${log4j2.version}</version>
+ </dependency>
++ <dependency>
++ <groupId>org.apache.logging.log4j</groupId>
++ <artifactId>log4j-core</artifactId>
++ <version>${log4j2.version}</version>
++ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
Review comment:
hive-exec depends on disruptor in test scope only. We do not need to add compile scope dependency here.
```
[INFO] org.apache.hive:hive-exec:jar:2.3.6
...
[INFO] +- com.lmax:disruptor:jar:3.3.7:test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997227730
With the new patch I can see disruptor 3.3.7 in the hive's .deb and the startup error seems gone. I need to test more the packages on the docker provisioner though, if anybody else want to re-test the last version of the patch if would be great :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-996551970
```
$ dpkg -c output/hive/hive_2.3.6-3_all.deb | grep -e log4j.*.jar
-rw-r--r-- root/root 207875 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-1.2-api-2.16.0.jar
-rw-r--r-- root/root 301861 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-api-2.16.0.jar
-rw-r--r-- root/root 1789526 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-core-2.16.0.jar
-rw-r--r-- root/root 22539 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-jul-2.5.jar
-rw-r--r-- root/root 24229 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-slf4j-impl-2.16.0.jar
-rw-r--r-- root/root 35324 2021-12-17 09:29 ./usr/lib/hive/lib/log4j-web-2.16.0.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997348608
@iwasakims thanks a lot for the patience in reviewing, my experience with maven is not a lot and it is clear :)
Your suggestions make sense, I am going to amend the patch and re-test, will report asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-996660069
Thanks for working on this @elukey.
I got following error on starting up HiveServer2 when I ran docker provisioner:
<pre>
Exception in thread "main" java.lang.NoSuchMethodError: com.lmax.disruptor.dsl.Disruptor.<init>(Lcom/lmax/disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
at org.apache.logging.log4j.core.async.AsyncLoggerDisruptor.start(AsyncLoggerDisruptor.java:108)
at org.apache.logging.log4j.core.async.AsyncLoggerContext.maybeStartHelper(AsyncLoggerContext.java:97)
at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:86)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:242)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:160)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:132)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:102)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:211)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:154)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:90)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:82)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:65)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:716)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
</pre>
disruptor-3.3.0 instead of 3.3.7 is containd in the package.
```
$ rpm -q -l -p output/hive/noarch/hive-2.3.6-3.el8.noarch.rpm | grep disruptor
/usr/lib/hive/lib/disruptor-3.3.0.jar
```
Dependency tree shows that Hive itself depends on disruptor only in test scope. Since hbase-server and druid-common (and more?) depends on disruptor in compile scope, fixing `disruptor.version` in pom.xml of Hive seems not to affect the version of disrupoter pulled as transitive dependency.
```
$ mvn -Dhbase.version=1.5.0 -Dzookeeper.version=3.4.13 -Dhadoop.version=2.10.1 -DskipTests -Dtez.version=0.9.2 -Dspark.version=2.4.5 dependency:tree
...
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ hive-common ---
[INFO] org.apache.hive:hive-common:jar:2.3.6
...
[INFO] +- com.lmax:disruptor:jar:3.3.7:test
...
[INFO] +- org.apache.hbase:hbase-server:jar:1.5.0:compile
...
[INFO] | \- com.lmax:disruptor:jar:3.3.0:compile
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims edited a comment on pull request #844: BIGTOP-3619 - Upgrade hive 2.3.6 to log4j 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims edited a comment on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997566330
+1. I tested this no CentOS 8 again.
I filed new JIRA issue and updated the title of PR since BIGTOP-3613 is used for general discussion. one-to-one relationship between JIRA and commit would make following changes easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-996766671
@iwasakims thanks a lot for the test and the analysis, my docker provisioner is still not working properly :(
I'll try to revert the disruptor change, not sure if really needed or not (maybe be a leftover of a test).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on a change in pull request #844: BIGTOP-3613 - Upgrade hive 2.3.6 to log4j 2.16.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on a change in pull request #844:
URL: https://github.com/apache/bigtop/pull/844#discussion_r771905476
##########
File path: bigtop-packages/src/common/hive/patch7-log4j2-2.16.0.diff
##########
@@ -0,0 +1,167 @@
+diff --git a/bin/hive-config.sh b/bin/hive-config.sh
+index d52b84eb5f..8381a25a05 100644
+--- a/bin/hive-config.sh
++++ b/bin/hive-config.sh
+@@ -68,3 +68,7 @@ export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH
+
+ # Default to use 256MB
+ export HADOOP_HEAPSIZE=${HADOOP_HEAPSIZE:-256}
++
++# Disable the JNDI. This feature has critical RCE vulnerability.
++# when 2.x <= log4j.version <= 2.14.1
++export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dlog4j2.formatMsgNoLookups=true"
+diff --git a/druid-handler/pom.xml b/druid-handler/pom.xml
+index 4af3d5f00b..349be6628d 100644
+--- a/druid-handler/pom.xml
++++ b/druid-handler/pom.xml
+@@ -223,6 +223,11 @@
+ <artifactId>calcite-druid</artifactId>
+ <version>${calcite.version}</version>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+ <!-- test inter-project -->
+ <dependency>
+ <groupId>junit</groupId>
+diff --git a/hbase-handler/pom.xml b/hbase-handler/pom.xml
+index 9b12f42651..0f009b6792 100644
+--- a/hbase-handler/pom.xml
++++ b/hbase-handler/pom.xml
+@@ -102,6 +102,11 @@
+ <artifactId>hbase-hadoop-compat</artifactId>
+ <version>${hbase.version}</version>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+
+ <!-- test inter-project -->
+ <dependency>
+diff --git a/jdbc/pom.xml b/jdbc/pom.xml
+index 45d9a733b4..d303de1645 100644
+--- a/jdbc/pom.xml
++++ b/jdbc/pom.xml
+@@ -109,6 +109,11 @@
+ <version>${hadoop.version}</version>
+ <optional>true</optional>
+ </dependency>
++ <dependency>
++ <groupId>com.lmax</groupId>
++ <artifactId>disruptor</artifactId>
++ <version>${disruptor.version}</version>
++ </dependency>
+ <!-- test inter-project -->
+ <dependency>
+ <groupId>junit</groupId>
Review comment:
hive-jdbc seems to depend on disruptor via hive-llap-server. If we modify llap-server/pom.xml, we do not need fix jdbc/pom.xml?
```
[INFO] org.apache.hive:hive-jdbc:jar:2.3.6
...
[INFO] +- org.apache.hive:hive-service:jar:2.3.6:compile
[INFO] | +- org.apache.hive:hive-llap-server:jar:2.3.6:compile
...
[INFO] | | +- org.apache.hbase:hbase-server:jar:1.5.0:compile
[INFO] | | | +- org.apache.hbase:hbase-procedure:jar:1.5.0:compile
...
[INFO] | | | \- com.lmax:disruptor:jar:3.3.0:compile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #844: BIGTOP-3619 - Upgrade hive 2.3.6 to log4j 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #844:
URL: https://github.com/apache/bigtop/pull/844#issuecomment-997566330
+1. I tested this no CentOS 8 again.
I filed new JIRA issue and updated the title of PR. one-to-one relationship between JIRA and commit would make following changes easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims merged pull request #844: BIGTOP-3619 - Upgrade hive 2.3.6 to log4j 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims merged pull request #844:
URL: https://github.com/apache/bigtop/pull/844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org