You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2021/12/22 08:49:17 UTC
[GitHub] [bigtop] elukey opened a new pull request #846: BIGTOP-3621 - Bump log4j dependencies to 2.17.0
elukey opened a new pull request #846:
URL: https://github.com/apache/bigtop/pull/846
Even if Oozie uses log4j 1.x, its Hive dependencies bring
in log4j 2.6.2 libraries, that end up in the final package
that Bigtop ships.
This change forces core and hcatalog POMs to use a fixed/safe
version of log4j 2.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1001007846
Addded explicit dependency on log4j2 brings extra log4j2 jar to /usr/lib/oozie/lib/.
```
$ rpm -qlp oozie-4.3.0-1.el8.noarch.rpm > /tmp/oozie-rpm-qlp-1.txt
$ rpm -qlp oozie-4.3.0-2.el8.noarch.rpm > /tmp/oozie-rpm-qlp-2.txt
$ diff /tmp/oozie-rpm-qlp-1.txt /tmp/oozie-rpm-qlp-2.txt
232a233
> /usr/lib/oozie/lib/log4j-1.2-api-2.17.0.jar
```
This seems to break OozieSharelibCLIZ.
```
[root@47d74e9acba3 /]# sudo -u oozie /usr/lib/oozie/bin/oozie-setup.sh sharelib create -fs hdfs://172.17.0.2:8020/
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting CATALINA_TMPDIR=/var/lib/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_LOG=/var/log/oozie
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting CATALINA_TMPDIR=/var/lib/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_LOG=/var/log/oozie
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/oozie/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/oozie/lib/log4j-slf4j-impl-2.17.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/oozie/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.log4j.helpers.OptionConverter.instantiateByKey(Ljava/util/Properties;Ljava/lang/String;Ljava/lang/Class;Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395)
at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955)
at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89)
at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58)
at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947)
at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473)
at org.apache.oozie.service.XLogService.init(XLogService.java:177)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.<init>(Services.java:111)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:155)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:67)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1000976850
part of relevant log entries in /var/log/oozie/oozie-error.log::
```
...
2021-12-25 05:07:03,969 WARN ActionStartXCommand:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-211225050521438-oozie-oozi-W] ACTION[0000001-211225050521438-oozie-oozi-W@mr-node] Failing Job due to failed action [mr-node]
2021-12-25 05:07:03,972 WARN LiteWorkflowInstance:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[root] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-211225050521438-oozie-oozi-W] ACTION[0000001-211225050521438-oozie-oozi-W@mr-node] Workflow Failed. Failing node [mr-node]
2021-12-25 05:07:05,233 WARN ParameterVerifier:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2021-12-25 05:07:05,241 WARN LiteWorkflowAppService:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://6debda8a5ccc.bigtop.apache.org:8020/user/root/oozie.1640408783176/examples/apps/hive2/lib] does not exist
2021-12-25 05:07:05,358 WARN ActionStartXCommand:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[root] GROUP[-] TOKEN[] APP[hive2-wf] JOB[0000002-211225050521438-oozie-oozi-W] ACTION[0000002-211225050521438-oozie-oozi-W@hive2-node] Error starting action [hive2-node]. ErrorType [FAILED], ErrorCode [EJ001], Message [Could not locate Oozie sharelib]
org.apache.oozie.action.ActionExecutorException: Could not locate Oozie sharelib
at org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:758)
at org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:853)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:844)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1089)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1369)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-12-25 05:07:05,359 WARN ActionStartXCommand:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[root] GROUP[-] TOKEN[] APP[hive2-wf] JOB[0000002-211225050521438-oozie-oozi-W] ACTION[0000002-211225050521438-oozie-oozi-W@hive2-node] Failing Job due to failed action [hive2-node]
2021-12-25 05:07:05,361 WARN LiteWorkflowInstance:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[root] GROUP[-] TOKEN[] APP[hive2-wf] JOB[0000002-211225050521438-oozie-oozi-W] ACTION[0000002-211225050521438-oozie-oozi-W@hive2-node] Workflow Failed. Failing node [hive2-node]
2021-12-25 05:07:06,814 WARN LiteWorkflowAppService:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://6debda8a5ccc.bigtop.apache.org:8020/user/root/oozie.1640408783176/examples/apps/no-op/lib] does not exist
2021-12-25 05:07:08,482 WARN ActionStartXCommand:523 - SERVER[6debda8a5ccc.bigtop.apache.org] USER[root] GROUP[-] TOKEN[] APP[java-main-wf] JOB[0000004-211225050521438-oozie-oozi-W] ACTION[0000004-211225050521438-oozie-oozi-W@java-node] Error starting action [java-node]. ErrorType [FAILED], ErrorCode [EJ001], Message [Could not locate Oozie sharelib]
org.apache.oozie.action.ActionExecutorException: Could not locate Oozie sharelib
at org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:758)
at org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:853)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:844)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1089)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1369)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1000945089
smoke-tests failed due to following exception. I'm looking into the cause.
```
org.apache.oozie.action.ActionExecutorException: Could not locate Oozie sharelib
at org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:758)
at org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:853)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:844)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1089)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1369)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1000944932
Thanks to [BIGTOP-3619](https://issues.apache.org/jira/browse/BIGTOP-3619)(#844), the jars in Oozie package seems to be bumped even without this patch by running oozie-pkg after hive-pkg. The modified pom of Hive is pulled from local repository (i.e. ~/.m2/repository) on building Oozie.
```
$ ./gradlew hive-pkg oozie-pkg
$ rpm -qlp output/oozie/noarch/oozie-4.3.0-1.el8.noarch.rpm | grep log4j
/etc/oozie/conf.dist/oozie-log4j.properties
/usr/lib/oozie/lib/apache-log4j-extras-1.2.17.jar
/usr/lib/oozie/lib/log4j-1.2.17.jar
/usr/lib/oozie/lib/log4j-api-2.17.0.jar
/usr/lib/oozie/lib/log4j-core-2.17.0.jar
/usr/lib/oozie/lib/log4j-slf4j-impl-2.17.0.jar
/usr/lib/oozie/lib/log4j-web-2.17.0.jar
/usr/lib/oozie/lib/slf4j-log4j12-1.6.6.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey edited a comment on pull request #846: BIGTOP-3621 - Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
elukey edited a comment on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-999410417
```
$ dpkg -c output/oozie/oozie_4.3.0-1_all.deb | grep log4j
-rw-r--r-- root/root 6129 2021-12-22 08:56 ./etc/oozie/conf.dist/oozie-log4j.properties
-rw-r--r-- root/root 448794 2021-12-22 08:17 ./usr/lib/oozie/lib/apache-log4j-extras-1.2.17.jar
-rw-r--r-- root/root 208235 2021-12-22 08:54 ./usr/lib/oozie/lib/log4j-1.2-api-2.17.0.jar
-rw-r--r-- root/root 489884 2021-12-22 08:14 ./usr/lib/oozie/lib/log4j-1.2.17.jar
-rw-r--r-- root/root 301776 2021-12-22 08:16 ./usr/lib/oozie/lib/log4j-api-2.17.0.jar
-rw-r--r-- root/root 1789339 2021-12-22 08:16 ./usr/lib/oozie/lib/log4j-core-2.17.0.jar
-rw-r--r-- root/root 24252 2021-12-22 08:48 ./usr/lib/oozie/lib/log4j-slf4j-impl-2.17.0.jar
-rw-r--r-- root/root 35920 2021-12-22 08:48 ./usr/lib/oozie/lib/log4j-web-2.17.0.jar
-rw-r--r-- root/root 9711 2021-12-22 08:14 ./usr/lib/oozie/lib/slf4j-log4j12-1.6.6.jar
```
I tried to run the Smoke tests but they are currently failing even with the "official" oozie version 4.3.0-1.
Edit: after some logs checking I realized that the smoke tests need hive to be deployed on the docker provisioner, tests in progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1001517014
> While `./gradlew hive-pkg oozie-pkg` works, this fix could be useful for incrementing the package release number and making sure that log4j is bumped even oozie alone is built.
I tested this procedure and it seems working fine. I am wondering if it is enough for the 1.5.0 branch, spending a ton of time in resolving dependencies for an old version of oozie (given an easier fix is available) may not be the best right now (given the amount of packages that we have to fix etc..).
We could bump oozie's version to `-2` and add the Debian `rules` fix, leaving dependencies as they are, and warning users in the Wiki or other places to build hive and oozie together to fix the log4j dependencies. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1000976925
Comparing /var/log/oozie/oozie-instrumentation.log shows that sharelib.system.libpath is not set as expected if the patch applied.
```
158c158
< sharelib.system.libpath: hdfs://78d2e928b7cc.bigtop.apache.org:8020/user/oozie/share/lib/lib_20211225015343
---
> sharelib.system.libpath: /user/oozie/share/lib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1001131586
Thanks a lot for the testing! I have upgraded the oozie package manually and ran the smoke tests (I didn't have all packages built to spin up a cluster from local repo) and I missed the shared lib CLI issue.
The above stacktrace makes sense, in the hive patch I had to fix something related:
```
-import org.apache.log4j.helpers.OptionConverter;
+import org.apache.logging.log4j.core.util.OptionConverter;
```
The main problem is that can't find where the OptionConverter is set (in code or properties files). Will keep looking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1002284901
> We could bump oozie's version to -2 and add the Debian rules fix, leaving dependencies as they are, and warning users in the Wiki or other places to build hive and oozie together to fix the log4j dependencies.
+1. I'm merging this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1000976996
I tested this on CentOS 8 by docker provisioner.
```
$ ./docker-hadoop.sh --create 1 --image bigtop/puppet:trunk-centos-8 --memory 16g --repo file:///bigtop-home/output --disable-gpg-check --stack hdfs,yarn,mapreduce,hive,oozie --smoke-tests oozie
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims merged pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims merged pull request #846:
URL: https://github.com/apache/bigtop/pull/846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1000944984
While `./gradlew hive-pkg oozie-pkg` works, this fix could be useful for incrementing the package release number and making sure that log4j is bumped even oozie alone is built.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey edited a comment on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
elukey edited a comment on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1001131586
Thanks a lot for the testing! I have upgraded the oozie package manually and ran the smoke tests (I didn't have all packages built to spin up a cluster from local repo) and I missed the shared lib CLI issue.
The above stacktrace makes sense, in the hive patch I had to fix something related:
```
-import org.apache.log4j.helpers.OptionConverter;
+import org.apache.logging.log4j.core.util.OptionConverter;
```
The main problem is that can't find where the OptionConverter is set (in code or properties files). Will keep looking.
EDIT: SLF4J is probably the culprit right? It uses a more up-to-date version of the log4j bindings and it break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1002178598
Updated the PR with the simpler set of commits, I think that the commit msg + some infos in BIGTOP-3621 is enough for this use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #846: BIGTOP-3621: Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-1000976789
The failure is reproducible. The tests passes without the patch.;
```
Job ID App Name Status User Group Started Ended
------------------------------------------------------------------------------------------------------------------------------------
0000007-211225041747664-oozie-oozi-W map-reduce-wfFAILED root - 2021-12-25 04:19 GMT 2021-12-25 04:19 GMT
------------------------------------------------------------------------------------------------------------------------------------
0000006-211225041747664-oozie-oozi-W hadoop-el-wf FAILED root - 2021-12-25 04:19 GMT 2021-12-25 04:19 GMT
------------------------------------------------------------------------------------------------------------------------------------
0000005-211225041747664-oozie-oozi-W streaming-wf FAILED root - 2021-12-25 04:19 GMT 2021-12-25 04:19 GMT
------------------------------------------------------------------------------------------------------------------------------------
0000004-211225041747664-oozie-oozi-W java-main-wf FAILED root - 2021-12-25 04:19 GMT 2021-12-25 04:19 GMT
------------------------------------------------------------------------------------------------------------------------------------
0000003-211225041747664-oozie-oozi-W no-op-wf SUCCEEDED root - 2021-12-25 04:19 GMT 2021-12-25 04:19 GMT
------------------------------------------------------------------------------------------------------------------------------------
0000002-211225041747664-oozie-oozi-W hive2-wf FAILED root - 2021-12-25 04:19 GMT 2021-12-25 04:19 GMT
------------------------------------------------------------------------------------------------------------------------------------
0000001-211225041747664-oozie-oozi-W map-reduce-wfFAILED root - 2021-12-25 04:19 GMT 2021-12-25 04:19 GMT
------------------------------------------------------------------------------------------------------------------------------------
0000000-211225041747664-oozie-oozi-W subwf-wf KILLED root - 2021-12-25 04:19 GMT 2021-12-25 04:19 GMT
------------------------------------------------------------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #846: BIGTOP-3621 - Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-999410417
```
$ dpkg -c output/oozie/oozie_4.3.0-1_all.deb | grep log4j
-rw-r--r-- root/root 6129 2021-12-22 08:56 ./etc/oozie/conf.dist/oozie-log4j.properties
-rw-r--r-- root/root 448794 2021-12-22 08:17 ./usr/lib/oozie/lib/apache-log4j-extras-1.2.17.jar
-rw-r--r-- root/root 208235 2021-12-22 08:54 ./usr/lib/oozie/lib/log4j-1.2-api-2.17.0.jar
-rw-r--r-- root/root 489884 2021-12-22 08:14 ./usr/lib/oozie/lib/log4j-1.2.17.jar
-rw-r--r-- root/root 301776 2021-12-22 08:16 ./usr/lib/oozie/lib/log4j-api-2.17.0.jar
-rw-r--r-- root/root 1789339 2021-12-22 08:16 ./usr/lib/oozie/lib/log4j-core-2.17.0.jar
-rw-r--r-- root/root 24252 2021-12-22 08:48 ./usr/lib/oozie/lib/log4j-slf4j-impl-2.17.0.jar
-rw-r--r-- root/root 35920 2021-12-22 08:48 ./usr/lib/oozie/lib/log4j-web-2.17.0.jar
-rw-r--r-- root/root 9711 2021-12-22 08:14 ./usr/lib/oozie/lib/slf4j-log4j12-1.6.6.jar
```
I tried to run the Smoke tests but they are currently failing even with the "official" oozie version 4.3.0-1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #846: BIGTOP-3621 - Bump Oozie's log4j dependencies to 2.17.0
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #846:
URL: https://github.com/apache/bigtop/pull/846#issuecomment-999449896
Smoke tests passed with the new package (those include hive2 actions so it should be a good proof that the new log4j libs change is fine).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org