You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2021/04/02 14:47:01 UTC
[GitHub] [bigtop] iwasakims opened a new pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
iwasakims opened a new pull request #763:
URL: https://github.com/apache/bigtop/pull/763
https://issues.apache.org/jira/browse/BIGTOP-3531
I needed lots of fix than I expected just make oozie-server run.
* fixed layout based on the expectation of scripts under /usr/lib/oozie/bin.
* made /usr/lib/oozie/lib symlink to embedded-oozie-server/webapp/WEB-INF/lib.
* set guava version to 27.0-jre.
* set JETTY_PID_FILE must be set to /var/run/oozie/oozie.pid.
* fixed /etc/init.d/oozie.
* excluded log4j2 from transitive dependencies of hive3 artifacts.
Let me address smoke-tests issue on [BIGTOP-3532](https://issues.apache.org/jira/browse/BIGTOP-3532) after this is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-816417724
Thanks, @sekikn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813222765
The cause turned out to be hive-exec.jar containing unshaded guava classes.
```
root@afa1d143f9b8:/# cat /tmp/oozie-cp.txt | sed -z -e 's/:/\n/g' | xargs -n 1 grep 'com/google/common/base/Preconditions.class'
Binary file /usr/lib/oozie/lib/avro-1.8.2.jar matches
Binary file /usr/lib/oozie/lib/guava-27.0-jre.jar matches
Binary file /usr/lib/oozie/lib/hbase-shaded-miscellaneous-1.0.1.jar matches
Binary file /usr/lib/oozie/lib/hive-exec-3.1.2.jar matches
Binary file /usr/lib/oozie/lib/jersey-guava-2.25.1.jar matches
grep: /usr/lib/oozie/libext/*.jar: No such file or directory
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/avro-1.8.2.jar | grep 'com/google/common/base/Preconditions.class'
5173 Mon Apr 05 04:43:28 GMT 2021 avro/shaded/com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/hive-exec-3.1.2.jar | grep 'com/google/common/base/Preconditions.class'
5246 Mon Apr 05 04:43:28 GMT 2021 com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/hbase-shaded-miscellaneous-1.0.1.jar | grep 'com/google/common/base/Preconditions.class'
20764 Mon Apr 05 04:43:28 GMT 2021 org/apache/hadoop/hbase/shaded/com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/jersey-guava-2.25.1.jar | grep 'com/google/common/base/Preconditions.class'
5281 Mon Apr 05 04:43:28 GMT 2021 jersey/repackaged/com/google/common/base/Preconditions.class
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-815860352
+1, I also did the same check on CentOS 7, Debian 10, and Ubuntu 20.04. Thanks @iwasakims!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813234914
Another issue specific to Ubuntu is that oozie-client put files /usr/lib/oozie/lib. oozie-server does not create symlink to /usr/lib/oozie/embedded-oozie-server/webapp/WEB-INF/lib as a result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims edited a comment on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims edited a comment on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813222765
EmbeddedOozieServer failed to start due to version mismatch of Guava.
```
sudo -u oozie bash -x -c 'export SHELLOPTS && env JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 /usr/lib/oozie/bin/oozied.sh run'
...
+ run_jetty
+ /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -Doozie.home.dir=/usr/lib/oozie -Doozie.config.dir=/etc/oozie/conf -Doozie.log.dir=/var/log/oozie -Doozie.data.dir=/var/lib/oozie -Doozie.instance.id=afa1d143f9b8.bigtop.apache.org -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Djava.library.path= -cp '/usr/lib/oozie/embedded-oozie-server/*:/usr/lib/oozie/embedded-oozie-server/dependency/*:/usr/lib/oozie/lib/*:/usr/lib/oozie/libtools/*:/usr/lib/oozie/embedded-oozie-server' org.apache.oozie.server.EmbeddedOozieServer
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
...
```
The cause turned out to be hive-exec.jar containing unshaded guava classes.
```
root@afa1d143f9b8:/# cat /tmp/oozie-cp.txt | sed -z -e 's/:/\n/g' | xargs grep 'com/google/common/base/Preconditions.class'
Binary file /usr/lib/oozie/lib/avro-1.8.2.jar matches
Binary file /usr/lib/oozie/lib/guava-27.0-jre.jar matches
Binary file /usr/lib/oozie/lib/hbase-shaded-miscellaneous-1.0.1.jar matches
Binary file /usr/lib/oozie/lib/hive-exec-3.1.2.jar matches
Binary file /usr/lib/oozie/lib/jersey-guava-2.25.1.jar matches
grep: /usr/lib/oozie/libext/*.jar: No such file or directory
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/avro-1.8.2.jar | grep 'com/google/common/base/Preconditions.class'
5173 Mon Apr 05 04:43:28 GMT 2021 avro/shaded/com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/hive-exec-3.1.2.jar | grep 'com/google/common/base/Preconditions.class'
5246 Mon Apr 05 04:43:28 GMT 2021 com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/hbase-shaded-miscellaneous-1.0.1.jar | grep 'com/google/common/base/Preconditions.class'
20764 Mon Apr 05 04:43:28 GMT 2021 org/apache/hadoop/hbase/shaded/com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/jersey-guava-2.25.1.jar | grep 'com/google/common/base/Preconditions.class'
5281 Mon Apr 05 04:43:28 GMT 2021 jersey/repackaged/com/google/common/base/Preconditions.class
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813196385
This worked on CentOS 8 but not on Ubuntu-18.04. I'm looking into the cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims edited a comment on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims edited a comment on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813234914
Another issue specific to Ubuntu is that oozie-client put files /usr/lib/oozie/lib. oozie (server package) does not create symlink to /usr/lib/oozie/embedded-oozie-server/webapp/WEB-INF/lib as a result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813460271
Bigtop applies [the patch](https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hive/patch3-HIVE-21569.diff) of [HIVE-21569](https://issues.apache.org/jira/browse/HIVE-21569) which bump guava version of Hive. We need to do `hive-pkg` before `oozie-pkg` to install patched Hive artifacts into local repo.
While both 27.0-jre and 28.1-jre have the `Precondition#checkArgument(boolean, String, Object)`, 27.0-jre which is used by hadoop-3.2.2 should be better.
* https://guava.dev/releases/27.0-jre/api/docs/com/google/common/base/Preconditions.html#checkArgument-boolean-java.lang.String-java.lang.Object-
* https://guava.dev/releases/28.1-jre/api/docs/com/google/common/base/Preconditions.html#checkArgument-boolean-java.lang.String-java.lang.Object-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813496013
I tested the update on CentOS 8 and Ubuntu 18.04.
```
$ ./docker-hadoop.sh \
--create 1 \
--image bigtop/puppet:trunk-centos-8 \
--memory 16g \
--repo file:///bigtop-home/output \
--disable-gpg-check \
--stack hdfs,yarn,mapreduce,oozie
...
Notice: Applied catalog in 182.76 seconds
$ ./docker-hadoop.sh --exec 1 systemctl --no-pager status oozie
● oozie.service - LSB: Oozie server daemon
Loaded: loaded (/etc/rc.d/init.d/oozie; generated)
Active: active (running) since Mon 2021-04-05 15:40:36 UTC; 11min ago
Docs: man:systemd-sysv-generator(8)
Tasks: 59 (limit: 201543)
Memory: 780.4M
CGroup: /docker/f780b7e2bfcb9de3e471e9c36edbe7aad1caf7022e237f7540a398ebb24fe0b3/system.slice/oozie.service
└─8841 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/…
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813225626
The situation is same for CentOS 8. If classes contained in guava-27.0-jre.jar are loaded before hive-exec-3.1.2.jar by chance, there is no error on starting up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn merged pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
sekikn merged pull request #763:
URL: https://github.com/apache/bigtop/pull/763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813904271
I verified that there is no class loading issue on `oozie` command even if only oozie-client pakcage is installed on CentOS 8 and Ubuntu 18.04.
```
$ ./docker-hadoop.sh --create 1 --image bigtop/puppet:trunk-centos-8 --memory 16g --repo file:///bigtop-home/output --disable-gpg-check --stack hdfs,yarn,mapreduce,oozie
$ ./docker-hadoop.sh --exec 1 systemctl --no-pager status oozie
● oozie.service - LSB: Oozie server daemon
Loaded: loaded (/etc/rc.d/init.d/oozie; generated)
Active: active (running) since Tue 2021-04-06 07:29:39 UTC; 39s ago
...
$ ./docker-hadoop.sh --exec 1 rpm -ev oozie
Preparing packages...
oozie-5.2.1-1.el8.noarch
warning: /etc/oozie/conf.dist/oozie-site.xml saved as /etc/oozie/conf.dist/oozie-site.xml.rpmsave
$ ./docker-hadoop.sh --exec 1 oozie
Invalid sub-command: missing sub-command
use 'help [sub-command]' for help details
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813863510
oozie-client RPM alone does not work since jars pointed by symbolic link are contained in oozie (sever) package. I'm fixing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims edited a comment on pull request #763: BIGTOP-3531. Fix packaging of Oozie 5.2.1.
Posted by GitBox <gi...@apache.org>.
iwasakims edited a comment on pull request #763:
URL: https://github.com/apache/bigtop/pull/763#issuecomment-813222765
The cause turned out to be hive-exec.jar containing unshaded guava classes.
```
root@afa1d143f9b8:/# cat /tmp/oozie-cp.txt | sed -z -e 's/:/\n/g' | xargs grep 'com/google/common/base/Preconditions.class'
Binary file /usr/lib/oozie/lib/avro-1.8.2.jar matches
Binary file /usr/lib/oozie/lib/guava-27.0-jre.jar matches
Binary file /usr/lib/oozie/lib/hbase-shaded-miscellaneous-1.0.1.jar matches
Binary file /usr/lib/oozie/lib/hive-exec-3.1.2.jar matches
Binary file /usr/lib/oozie/lib/jersey-guava-2.25.1.jar matches
grep: /usr/lib/oozie/libext/*.jar: No such file or directory
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/avro-1.8.2.jar | grep 'com/google/common/base/Preconditions.class'
5173 Mon Apr 05 04:43:28 GMT 2021 avro/shaded/com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/hive-exec-3.1.2.jar | grep 'com/google/common/base/Preconditions.class'
5246 Mon Apr 05 04:43:28 GMT 2021 com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/hbase-shaded-miscellaneous-1.0.1.jar | grep 'com/google/common/base/Preconditions.class'
20764 Mon Apr 05 04:43:28 GMT 2021 org/apache/hadoop/hbase/shaded/com/google/common/base/Preconditions.class
root@afa1d143f9b8:/# jar tvf /usr/lib/oozie/lib/jersey-guava-2.25.1.jar | grep 'com/google/common/base/Preconditions.class'
5281 Mon Apr 05 04:43:28 GMT 2021 jersey/repackaged/com/google/common/base/Preconditions.class
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org