You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2020/09/22 15:32:23 UTC
[GitHub] [bigtop] sekikn opened a new pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
sekikn opened a new pull request #676:
URL: https://github.com/apache/bigtop/pull/676
This PR contains the following changes.
* Change class loading order defined in catalina.properties so as to start Oozie webapp without jar signing error.
* Add environment variables to smoke-tests.sh, which are required to run Oozie smoke test. Also reorder them in alphabetical order for maintainability.
* Disable unit tests which depend on Pig functionality.
* Replace the hive test with hive2 since the former doesn't seem to work with the default hive setup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #676:
URL: https://github.com/apache/bigtop/pull/676#issuecomment-696799372
With this PR, I confirmed Oozie smoke test succeeded on Ubuntu 18.04 as follows.
Build and deploy Oozie and its dependencies:
```
$ ./gradlew oozie-clean oozie-pkg repo -Dbuildwithdeps=true
(snip)
BUILD SUCCESSFUL in 33m 29s
10 actionable tasks: 10 executed
$ cat /etc/puppet/hieradata/site.yaml
---
bigtop::hadoop_head_node: ubuntu1804.localdomain
hadoop::hadoop_storage_dirs:
- /data
hadoop_cluster_node::cluster_components:
- hdfs
- yarn
- mapreduce
- hive
- oozie
bigtop::bigtop_repo_uri: file:///home/vagrant/bigtop/output/apt
bigtop::jdk_preinstalled: true
bigtop::bigtop_repo_gpg_check: false
$ sudo puppet apply --hiera_config=/etc/puppet/hiera.yaml --modulepath=bigtop-deploy/puppet/modules:/usr/share/puppet/modules bigtop-deploy/puppet/manifests
(snip)
Notice: Applied catalog in 320.00 seconds
```
Upload Oozie sharelib manually and run Oozie smoke test as root:
```
# . /usr/lib/bigtop-utils/bigtop-detect-javahome
# export HADOOP_HOME=/usr/lib/hadoop
# export HADOOP_CONF_DIR=/etc/hadoop/conf
# export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
# export HIVE_HOME=/usr/lib/hive
# export HIVE_CONF_DIR=/etc/hive/conf
# export OOZIE_URL=http://$(hostname -f):11000/oozie/
# export OOZIE_TAR_HOME=/usr/share/doc/oozie
# sudo -u hdfs hdfs dfs -chown -R oozie:oozie /user/oozie
# oozie-setup sharelib create -fs hdfs://$(hostname -f):8020/
(snip)
the destination path for sharelib is: /user/oozie/share/lib/lib_20200922080033
# oozie admin -sharelibupdate
(snip)
[ShareLib update status]
sharelibDirOld = /user/oozie/share/lib
host = http://ubuntu1804.localdomain:11000/oozie
sharelibDirNew = hdfs://ubuntu1804.localdomain:8020/user/oozie/share/lib/lib_20200922080033
status = Successful
# ./gradlew bigtop-tests:smoke-tests:oozie:test -Psmoke.tests --info
(snip)
> Task :bigtop-tests:smoke-tests:oozie:test
Finished generating test XML results (0.003 secs) into: /home/vagrant/bigtop/bigtop-tests/smoke-tests/oozie/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.015 secs) into: /home/vagrant/bigtop/bigtop-tests/smoke-tests/oozie/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:oozie:test (Thread[Daemon worker Thread 3,5,main]) completed. Took 6 mins 58.169 secs.
BUILD SUCCESSFUL in 7m 7s
6 actionable tasks: 6 executed
Stopped 1 worker daemon(s).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn edited a comment on pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
Posted by GitBox <gi...@apache.org>.
sekikn edited a comment on pull request #676:
URL: https://github.com/apache/bigtop/pull/676#issuecomment-700493035
I updated the PR. With this PR, I confirmed the smoke test succeeded on Ubuntu 18.04, as follows:
```
~/repos/bigtop/provisioner/docker$ cat config_ubuntu-18.04.yaml
(snip)
docker:
memory_limit: "4g"
image: "bigtop/puppet:trunk-ubuntu-18.04"
repo: "file:///bigtop-home/output/apt"
distro: debian
components: [hdfs, yarn, mapreduce, hive, oozie]
enable_local_repo: true
smoke_test_components: [oozie]
~/repos/bigtop/provisioner/docker$ ./docker-hadoop.sh -d -C config_ubuntu-18.04.yaml -c 1 -s
(snip)
> Task :bigtop-tests:smoke-tests:oozie:test
Finished generating test XML results (0.011 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.016 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:oozie:test (Thread[Daemon worker,5,main]) completed. Took 6 mins 25.139 secs.
BUILD SUCCESSFUL in 11m 39s
```
But even with this, the smoke test failed on CentOS 8.
```
$ ./docker-hadoop.sh -d -C config_centos-8.yaml -c 1 -s
(snip)
===== START TO RUN SMOKE TESTS: oozie =====
(snip)
+ oozie-setup sharelib create -fs hdfs://325b0282b92b.bigtop.apache.org:8020/
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting CATALINA_TMPDIR=/var/lib/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_LOG=/var/log/oozie
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting CATALINA_TMPDIR=/var/lib/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_LOG=/var/log/oozie
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more
+ oozie admin -sharelibupdate
Error: Could not find or load main class org.apache.oozie.cli.OozieCLI
(snip)
> Task :bigtop-tests:smoke-tests:oozie:test
9 tests completed, 7 failed
Finished generating test XML results (0.025 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.052 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/reports/tests/test
> Task :bigtop-tests:smoke-tests:oozie:test FAILED
:bigtop-tests:smoke-tests:oozie:test (Thread[Daemon worker,5,main]) completed. Took 16.517 secs.
FAILURE: Build failed with an exception.
```
It looks like Oozie lacks dependent jars on CentOS 8, which are installed on Ubuntu 18.04 with the same deployment manifests.
CentOS 8:
```
$ docker exec -it 20200929_045903_r6957_bigtop_1 ls /usr/lib/oozie/lib
hadoop-annotations.jar hadoop-mapreduce-client-jobclient.jar
hadoop-auth.jar hadoop-mapreduce-client-shuffle.jar
hadoop-common.jar hadoop-yarn-api.jar
hadoop-hdfs-client.jar hadoop-yarn-client.jar
hadoop-mapreduce-client-app.jar hadoop-yarn-common.jar
hadoop-mapreduce-client-common.jar hadoop-yarn-server-common.jar
hadoop-mapreduce-client-core.jar
$ docker exec -it 20200929_045903_r6957_bigtop_1 ls -l /usr/lib/oozie/lib | wc -l
14
```
Ubuntu 18.04:
```
$ docker exec -it 20200929135113r4152_bigtop_1 ls /usr/lib/oozie/lib
HikariCP-2.5.1.jar
HikariCP-java7-2.4.12.jar
JavaEWAH-0.7.9.jar
ST4-4.0.4.jar
accessors-smart-1.2.jar
activation-1.1.jar
activemq-client-5.13.3.jar
aircompressor-0.8.jar
ant-1.9.1.jar
ant-launcher-1.9.1.jar
antlr-2.7.7.jar
antlr-runtime-3.4.jar
aopalliance-1.0.jar
apache-log4j-extras-1.2.17.jar
apacheds-i18n-2.0.0-M15.jar
apacheds-kerberos-codec-2.0.0-M15.jar
api-asn1-api-1.0.0-M20.jar
api-util-1.0.0-M20.jar
asm-3.1.jar
asm-5.0.4.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
avatica-1.8.0.jar
avatica-metrics-1.8.0.jar
avro-1.7.7.jar
bonecp-0.8.0.RELEASE.jar
c3p0-0.9.1.1.jar
calcite-core-1.10.0.jar
calcite-druid-1.10.0.jar
calcite-linq4j-1.10.0.jar
collections-generic-4.01.jar
commons-beanutils-1.9.4.jar
commons-cli-1.2.jar
commons-codec-1.4.jar
commons-collections-3.2.2.jar
commons-compiler-2.7.6.jar
commons-compress-1.9.jar
commons-configuration-1.6.jar
commons-daemon-1.0.13.jar
commons-dbcp-1.4.jar
commons-digester-1.8.jar
commons-el-1.0.jar
commons-httpclient-3.0.1.jar
commons-io-2.1.jar
commons-jexl-2.1.1.jar
commons-lang-2.4.jar
commons-lang3-3.3.2.jar
commons-logging-1.1.jar
commons-math-2.2.jar
commons-math3-3.1.1.jar
commons-net-3.1.jar
commons-pool-1.5.4.jar
curator-client-2.5.0.jar
curator-framework-2.5.0.jar
curator-recipes-2.5.0.jar
curator-x-discovery-2.5.0.jar
datanucleus-api-jdo-4.2.4.jar
datanucleus-core-4.1.17.jar
datanucleus-rdbms-4.1.19.jar
derby-10.10.1.1.jar
disruptor-3.3.0.jar
dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
ehcache-3.3.1.jar
ehcache-core-2.6.3.jar
eigenbase-properties-1.1.5.jar
fastutil-6.5.6.jar
findbugs-annotations-1.3.9-1.jar
fst-2.50.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-j2ee-management_1.1_spec-1.0.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jcache_1.0_spec-1.0-alpha-1.jar
geronimo-jms_1.1_spec-1.1.1.jar
geronimo-jpa_2.0_spec-1.1.jar
geronimo-jta_1.1_spec-1.1.1.jar
groovy-all-2.4.4.jar
gson-2.7.jar
guava-11.0.2.jar
guice-3.0.jar
guice-assistedinject-3.0.jar
guice-servlet-3.0.jar
hadoop-annotations.jar
hadoop-auth-2.10.0.jar
hadoop-auth.jar
hadoop-common.jar
hadoop-hdfs-client.jar
hadoop-mapreduce-client-app.jar
hadoop-mapreduce-client-common.jar
hadoop-mapreduce-client-core.jar
hadoop-mapreduce-client-jobclient.jar
hadoop-mapreduce-client-shuffle.jar
hadoop-yarn-api.jar
hadoop-yarn-client.jar
hadoop-yarn-common.jar
hadoop-yarn-server-common.jar
hawtbuf-1.11.jar
hbase-annotations-1.1.1.jar
hbase-client-1.1.1.jar
hbase-common-1.1.1-tests.jar
hbase-common-1.1.1.jar
hbase-hadoop-compat-1.1.1.jar
hbase-hadoop2-compat-1.1.1.jar
hbase-prefix-tree-1.1.1.jar
hbase-procedure-1.1.1.jar
hbase-protocol-1.1.1.jar
hbase-server-1.1.1.jar
hive-common-2.3.6.jar
hive-exec-2.3.6-core.jar
hive-exec-2.3.6.jar
hive-hcatalog-core-2.3.6.jar
hive-hcatalog-server-extensions-2.3.6.jar
hive-jdbc-2.3.6.jar
hive-llap-client-2.3.6.jar
hive-llap-common-2.3.6-tests.jar
hive-llap-common-2.3.6.jar
hive-llap-server-2.3.6.jar
hive-llap-tez-2.3.6.jar
hive-metastore-2.3.6.jar
hive-serde-2.3.6.jar
hive-service-2.3.6.jar
hive-service-rpc-2.3.6.jar
hive-shims-0.23-2.3.6.jar
hive-shims-2.3.6.jar
hive-shims-common-2.3.6.jar
hive-shims-scheduler-2.3.6.jar
hive-storage-api-2.4.0.jar
hive-vector-code-gen-2.3.6.jar
hive-webhcat-java-client-2.3.6.jar
hsqldb-1.8.0.10.jar
htrace-core-3.1.0-incubating.jar
htrace-core4-4.1.0-incubating.jar
httpclient-4.3.6.jar
httpcore-4.3.3.jar
ivy-2.4.0.jar
jackson-annotations-2.2.2.jar
jackson-core-2.2.2.jar
jackson-core-asl-1.9.13.jar
jackson-databind-2.2.2.jar
jackson-jaxrs-1.9.13.jar
jackson-mapper-asl-1.9.13.jar
jackson-xc-1.9.13.jar
jakarta.jms-api-2.0.2.jar
jamon-runtime-2.3.1.jar
janino-2.7.6.jar
java-util-1.9.0.jar
javax.inject-1.jar
javax.jdo-3.2.0-m3.jar
javax.servlet-3.0.0.v201112011016.jar
javolution-5.5.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcip-annotations-1.0-1.jar
jcodings-1.0.8.jar
jcommander-1.30.jar
jdo-api-3.0.1.jar
jdom-1.1.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jettison-1.1.jar
jetty-6.1.14.jar
jetty-all-7.6.0.v20120127.jar
jetty-sslengine-6.1.26.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
joda-time-2.1.jar
jol-core-0.2.jar
joni-2.1.2.jar
jpam-1.1.jar
jsch-0.1.50.jar
json-1.8.jar
json-io-2.5.1.jar
json-simple-1.1.jar
json-smart-2.3.jar
jsp-2.1-6.1.14.jar
jsp-api-2.0.jar
jsr305-1.3.9.jar
jta-1.1.jar
jung-algorithms-2.0.1.jar
jung-api-2.0.1.jar
jung-graph-impl-2.0.1.jar
jung-visualization-2.0.1.jar
kryo-2.22.jar
leveldbjni-all-1.8.jar
libfb303-0.9.3.jar
libthrift-0.9.3.jar
log4j-1.2.17.jar
log4j-api-2.6.2.jar
log4j-core-2.6.2.jar
log4j-slf4j-impl-2.6.2.jar
log4j-web-2.6.2.jar
mail-1.4.jar
metrics-core-2.2.0.jar
metrics-core-3.0.2.jar
metrics-core-3.1.2.jar
metrics-ganglia-3.1.2.jar
metrics-graphite-3.1.2.jar
metrics-json-3.0.2.jar
metrics-json-3.1.0.jar
metrics-jvm-3.0.2.jar
metrics-jvm-3.1.0.jar
mssql-jdbc-6.2.1.jre7.jar
netty-3.10.5.Final.jar
netty-3.7.0.Final.jar
netty-all-4.0.23.Final.jar
nimbus-jose-jwt-4.41.1.jar
okhttp-2.7.5.jar
okio-1.6.0.jar
oozie-client-4.3.0.jar
oozie-core-4.3.0.jar
oozie-hadoop-auth-hadoop-2-4.3.0.jar
oozie-hadoop-utils-hadoop-2-4.3.0.jar
oozie-sharelib-distcp-4.3.0.jar
oozie-sharelib-hcatalog-4.3.0.jar
oozie-sharelib-hive-4.3.0.jar
oozie-sharelib-hive2-4.3.0.jar
oozie-sharelib-oozie-4.3.0.jar
oozie-sharelib-spark-4.3.0.jar
oozie-sharelib-sqoop-4.3.0.jar
oozie-sharelib-streaming-4.3.0.jar
opencsv-2.3.jar
openjpa-jdbc-2.4.1.jar
openjpa-kernel-2.4.1.jar
openjpa-lib-2.4.1.jar
openjpa-persistence-2.4.1.jar
openjpa-persistence-jdbc-2.4.1.jar
orc-core-1.3.4.jar
org.eclipse.jgit-3.3.1.201403241930-r.jar
oro-2.0.8.jar
paranamer-2.3.jar
parquet-hadoop-bundle-1.8.1.jar
pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
postgresql-9.0-801.jdbc4.jar
protobuf-java-2.5.0.jar
quartz-2.1.7.jar
serp-1.15.1.jar
servlet-api-2.5-6.1.14.jar
servlet-api-2.5.jar
slf4j-api-1.6.6.jar
slf4j-log4j12-1.6.6.jar
slf4j-simple-1.6.6.jar
slice-0.29.jar
slider-core-0.90.2-incubating.jar
snappy-java-1.0.5.jar
stax-api-1.0-2.jar
stax-api-1.0.1.jar
stax2-api-3.1.4.jar
stringtemplate-3.2.1.jar
tephra-api-0.6.0.jar
tephra-core-0.6.0.jar
tephra-hbase-compat-1.0-0.6.0.jar
transaction-api-1.1.jar
twill-api-0.6.0-incubating.jar
twill-common-0.6.0-incubating.jar
twill-core-0.6.0-incubating.jar
twill-discovery-api-0.6.0-incubating.jar
twill-discovery-core-0.6.0-incubating.jar
twill-zookeeper-0.6.0-incubating.jar
velocity-1.5.jar
woodstox-core-5.0.3.jar
xbean-asm5-shaded-3.17.jar
xercesImpl-2.10.0.jar
xml-apis-1.4.01.jar
xmlenc-0.52.jar
zookeeper-3.4.6.jar
zookeeper-3.4.9.jar
$ docker exec -it 20200929135113r4152_bigtop_1 ls /usr/lib/oozie/lib | wc -l
268
```
I'll continue the investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #676:
URL: https://github.com/apache/bigtop/pull/676#issuecomment-700647371
I got the same problem on CentOS 7 too. There could be bug on building RPM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims merged pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
Posted by GitBox <gi...@apache.org>.
iwasakims merged pull request #676:
URL: https://github.com/apache/bigtop/pull/676
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #676:
URL: https://github.com/apache/bigtop/pull/676#issuecomment-700493035
I updated the PR. With this PR, I confirmed the smoke test succeeded on Ubuntu 18.04, as follows:
```
~/repos/bigtop/provisioner/docker$ cat config_ubuntu-18.04.yaml
(snip)
docker:
memory_limit: "4g"
image: "bigtop/puppet:trunk-ubuntu-18.04"
repo: "file:///bigtop-home/output/apt"
distro: debian
components: [hdfs, yarn, mapreduce, hive, oozie]
enable_local_repo: true
smoke_test_components: [oozie]
~/repos/bigtop/provisioner/docker$ ./docker-hadoop.sh -d -C config_ubuntu-18.04.yaml -c 1 -s
(snip)
> Task :bigtop-tests:smoke-tests:oozie:test
Finished generating test XML results (0.011 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.016 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:oozie:test (Thread[Daemon worker,5,main]) completed. Took 6 mins 25.139 secs.
BUILD SUCCESSFUL in 11m 39s
```
But even with this, the smoke test failed on CentOS 8.
```
$ ./docker-hadoop.sh -d -C config_centos-8.yaml -c 1 -s
(snip)
===== START TO RUN SMOKE TESTS: oozie =====
(snip)
+ oozie-setup sharelib create -fs hdfs://325b0282b92b.bigtop.apache.org:8020/
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting CATALINA_TMPDIR=/var/lib/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_LOG=/var/log/oozie
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting CATALINA_TMPDIR=/var/lib/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_LOG=/var/log/oozie
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more
+ oozie admin -sharelibupdate
Error: Could not find or load main class org.apache.oozie.cli.OozieCLI
(snip)
> Task :bigtop-tests:smoke-tests:oozie:test
9 tests completed, 7 failed
Finished generating test XML results (0.025 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.052 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/reports/tests/test
> Task :bigtop-tests:smoke-tests:oozie:test FAILED
:bigtop-tests:smoke-tests:oozie:test (Thread[Daemon worker,5,main]) completed. Took 16.517 secs.
FAILURE: Build failed with an exception.
```
It looks like Oozie lacks dependent jars on CentOS 8, which are installed on Ubuntu 18.04 with the same deployment manifests.
CentOS 8:
```
$ docker exec -it 20200929_045903_r6957_bigtop_1 ls /usr/lib/oozie/lib
hadoop-annotations.jar hadoop-mapreduce-client-jobclient.jar
hadoop-auth.jar hadoop-mapreduce-client-shuffle.jar
hadoop-common.jar hadoop-yarn-api.jar
hadoop-hdfs-client.jar hadoop-yarn-client.jar
hadoop-mapreduce-client-app.jar hadoop-yarn-common.jar
hadoop-mapreduce-client-common.jar hadoop-yarn-server-common.jar
hadoop-mapreduce-client-core.jar
$ docker exec -it 20200929_045903_r6957_bigtop_1 ls -l /usr/lib/oozie/lib | wc -l
14
```
Ubuntu 18.04:
```
$ docker exec -it 20200929135113r4152_bigtop_1 ls /usr/lib/oozie/lib
HikariCP-2.5.1.jar
HikariCP-java7-2.4.12.jar
JavaEWAH-0.7.9.jar
ST4-4.0.4.jar
accessors-smart-1.2.jar
activation-1.1.jar
activemq-client-5.13.3.jar
aircompressor-0.8.jar
ant-1.9.1.jar
ant-launcher-1.9.1.jar
antlr-2.7.7.jar
antlr-runtime-3.4.jar
aopalliance-1.0.jar
apache-log4j-extras-1.2.17.jar
apacheds-i18n-2.0.0-M15.jar
apacheds-kerberos-codec-2.0.0-M15.jar
api-asn1-api-1.0.0-M20.jar
api-util-1.0.0-M20.jar
asm-3.1.jar
asm-5.0.4.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
avatica-1.8.0.jar
avatica-metrics-1.8.0.jar
avro-1.7.7.jar
bonecp-0.8.0.RELEASE.jar
c3p0-0.9.1.1.jar
calcite-core-1.10.0.jar
calcite-druid-1.10.0.jar
calcite-linq4j-1.10.0.jar
collections-generic-4.01.jar
commons-beanutils-1.9.4.jar
commons-cli-1.2.jar
commons-codec-1.4.jar
commons-collections-3.2.2.jar
commons-compiler-2.7.6.jar
commons-compress-1.9.jar
commons-configuration-1.6.jar
commons-daemon-1.0.13.jar
commons-dbcp-1.4.jar
commons-digester-1.8.jar
commons-el-1.0.jar
commons-httpclient-3.0.1.jar
commons-io-2.1.jar
commons-jexl-2.1.1.jar
commons-lang-2.4.jar
commons-lang3-3.3.2.jar
commons-logging-1.1.jar
commons-math-2.2.jar
commons-math3-3.1.1.jar
commons-net-3.1.jar
commons-pool-1.5.4.jar
curator-client-2.5.0.jar
curator-framework-2.5.0.jar
curator-recipes-2.5.0.jar
curator-x-discovery-2.5.0.jar
datanucleus-api-jdo-4.2.4.jar
datanucleus-core-4.1.17.jar
datanucleus-rdbms-4.1.19.jar
derby-10.10.1.1.jar
disruptor-3.3.0.jar
dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
ehcache-3.3.1.jar
ehcache-core-2.6.3.jar
eigenbase-properties-1.1.5.jar
fastutil-6.5.6.jar
findbugs-annotations-1.3.9-1.jar
fst-2.50.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-j2ee-management_1.1_spec-1.0.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jcache_1.0_spec-1.0-alpha-1.jar
geronimo-jms_1.1_spec-1.1.1.jar
geronimo-jpa_2.0_spec-1.1.jar
geronimo-jta_1.1_spec-1.1.1.jar
groovy-all-2.4.4.jar
gson-2.7.jar
guava-11.0.2.jar
guice-3.0.jar
guice-assistedinject-3.0.jar
guice-servlet-3.0.jar
hadoop-annotations.jar
hadoop-auth-2.10.0.jar
hadoop-auth.jar
hadoop-common.jar
hadoop-hdfs-client.jar
hadoop-mapreduce-client-app.jar
hadoop-mapreduce-client-common.jar
hadoop-mapreduce-client-core.jar
hadoop-mapreduce-client-jobclient.jar
hadoop-mapreduce-client-shuffle.jar
hadoop-yarn-api.jar
hadoop-yarn-client.jar
hadoop-yarn-common.jar
hadoop-yarn-server-common.jar
hawtbuf-1.11.jar
hbase-annotations-1.1.1.jar
hbase-client-1.1.1.jar
hbase-common-1.1.1-tests.jar
hbase-common-1.1.1.jar
hbase-hadoop-compat-1.1.1.jar
hbase-hadoop2-compat-1.1.1.jar
hbase-prefix-tree-1.1.1.jar
hbase-procedure-1.1.1.jar
hbase-protocol-1.1.1.jar
hbase-server-1.1.1.jar
hive-common-2.3.6.jar
hive-exec-2.3.6-core.jar
hive-exec-2.3.6.jar
hive-hcatalog-core-2.3.6.jar
hive-hcatalog-server-extensions-2.3.6.jar
hive-jdbc-2.3.6.jar
hive-llap-client-2.3.6.jar
hive-llap-common-2.3.6-tests.jar
hive-llap-common-2.3.6.jar
hive-llap-server-2.3.6.jar
hive-llap-tez-2.3.6.jar
hive-metastore-2.3.6.jar
hive-serde-2.3.6.jar
hive-service-2.3.6.jar
hive-service-rpc-2.3.6.jar
hive-shims-0.23-2.3.6.jar
hive-shims-2.3.6.jar
hive-shims-common-2.3.6.jar
hive-shims-scheduler-2.3.6.jar
hive-storage-api-2.4.0.jar
hive-vector-code-gen-2.3.6.jar
hive-webhcat-java-client-2.3.6.jar
hsqldb-1.8.0.10.jar
htrace-core-3.1.0-incubating.jar
htrace-core4-4.1.0-incubating.jar
httpclient-4.3.6.jar
httpcore-4.3.3.jar
ivy-2.4.0.jar
jackson-annotations-2.2.2.jar
jackson-core-2.2.2.jar
jackson-core-asl-1.9.13.jar
jackson-databind-2.2.2.jar
jackson-jaxrs-1.9.13.jar
jackson-mapper-asl-1.9.13.jar
jackson-xc-1.9.13.jar
jakarta.jms-api-2.0.2.jar
jamon-runtime-2.3.1.jar
janino-2.7.6.jar
java-util-1.9.0.jar
javax.inject-1.jar
javax.jdo-3.2.0-m3.jar
javax.servlet-3.0.0.v201112011016.jar
javolution-5.5.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcip-annotations-1.0-1.jar
jcodings-1.0.8.jar
jcommander-1.30.jar
jdo-api-3.0.1.jar
jdom-1.1.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jettison-1.1.jar
jetty-6.1.14.jar
jetty-all-7.6.0.v20120127.jar
jetty-sslengine-6.1.26.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
joda-time-2.1.jar
jol-core-0.2.jar
joni-2.1.2.jar
jpam-1.1.jar
jsch-0.1.50.jar
json-1.8.jar
json-io-2.5.1.jar
json-simple-1.1.jar
json-smart-2.3.jar
jsp-2.1-6.1.14.jar
jsp-api-2.0.jar
jsr305-1.3.9.jar
jta-1.1.jar
jung-algorithms-2.0.1.jar
jung-api-2.0.1.jar
jung-graph-impl-2.0.1.jar
jung-visualization-2.0.1.jar
kryo-2.22.jar
leveldbjni-all-1.8.jar
libfb303-0.9.3.jar
libthrift-0.9.3.jar
log4j-1.2.17.jar
log4j-api-2.6.2.jar
log4j-core-2.6.2.jar
log4j-slf4j-impl-2.6.2.jar
log4j-web-2.6.2.jar
mail-1.4.jar
metrics-core-2.2.0.jar
metrics-core-3.0.2.jar
metrics-core-3.1.2.jar
metrics-ganglia-3.1.2.jar
metrics-graphite-3.1.2.jar
metrics-json-3.0.2.jar
metrics-json-3.1.0.jar
metrics-jvm-3.0.2.jar
metrics-jvm-3.1.0.jar
mssql-jdbc-6.2.1.jre7.jar
netty-3.10.5.Final.jar
netty-3.7.0.Final.jar
netty-all-4.0.23.Final.jar
nimbus-jose-jwt-4.41.1.jar
okhttp-2.7.5.jar
okio-1.6.0.jar
oozie-client-4.3.0.jar
oozie-core-4.3.0.jar
oozie-hadoop-auth-hadoop-2-4.3.0.jar
oozie-hadoop-utils-hadoop-2-4.3.0.jar
oozie-sharelib-distcp-4.3.0.jar
oozie-sharelib-hcatalog-4.3.0.jar
oozie-sharelib-hive-4.3.0.jar
oozie-sharelib-hive2-4.3.0.jar
oozie-sharelib-oozie-4.3.0.jar
oozie-sharelib-spark-4.3.0.jar
oozie-sharelib-sqoop-4.3.0.jar
oozie-sharelib-streaming-4.3.0.jar
opencsv-2.3.jar
openjpa-jdbc-2.4.1.jar
openjpa-kernel-2.4.1.jar
openjpa-lib-2.4.1.jar
openjpa-persistence-2.4.1.jar
openjpa-persistence-jdbc-2.4.1.jar
orc-core-1.3.4.jar
org.eclipse.jgit-3.3.1.201403241930-r.jar
oro-2.0.8.jar
paranamer-2.3.jar
parquet-hadoop-bundle-1.8.1.jar
pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
postgresql-9.0-801.jdbc4.jar
protobuf-java-2.5.0.jar
quartz-2.1.7.jar
serp-1.15.1.jar
servlet-api-2.5-6.1.14.jar
servlet-api-2.5.jar
slf4j-api-1.6.6.jar
slf4j-log4j12-1.6.6.jar
slf4j-simple-1.6.6.jar
slice-0.29.jar
slider-core-0.90.2-incubating.jar
snappy-java-1.0.5.jar
stax-api-1.0-2.jar
stax-api-1.0.1.jar
stax2-api-3.1.4.jar
stringtemplate-3.2.1.jar
tephra-api-0.6.0.jar
tephra-core-0.6.0.jar
tephra-hbase-compat-1.0-0.6.0.jar
transaction-api-1.1.jar
twill-api-0.6.0-incubating.jar
twill-common-0.6.0-incubating.jar
twill-core-0.6.0-incubating.jar
twill-discovery-api-0.6.0-incubating.jar
twill-discovery-core-0.6.0-incubating.jar
twill-zookeeper-0.6.0-incubating.jar
velocity-1.5.jar
woodstox-core-5.0.3.jar
xbean-asm5-shaded-3.17.jar
xercesImpl-2.10.0.jar
xml-apis-1.4.01.jar
xmlenc-0.52.jar
zookeeper-3.4.6.jar
zookeeper-3.4.9.jar
$ docker exec -it 20200929135113r4152_bigtop_1 ls /usr/lib/oozie/lib | wc -l
268
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] elukey commented on pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
Posted by GitBox <gi...@apache.org>.
elukey commented on pull request #676:
URL: https://github.com/apache/bigtop/pull/676#issuecomment-715899945
Would it be possible to run something like `dpkg -L` for CenOs to see what libs are in `oozie` and `oozie-client`? After that it should be easier to figure out where the problem is (maybe it is only one package's fault).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #676:
URL: https://github.com/apache/bigtop/pull/676#issuecomment-717644530
@elukey The cause is settings for docker container which removes doc files.
https://issues.apache.org/jira/browse/BIGTOP-3406?focusedCommentId=17201537&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17201537
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #676:
URL: https://github.com/apache/bigtop/pull/676#issuecomment-696799372
With this PR, I confirmed Oozie smoke test succeeded on Ubuntu 18.04 as follows.
Build and deploy Oozie and its dependencies:
```
$ ./gradlew oozie-clean oozie-pkg repo -Dbuildwithdeps=true
(snip)
BUILD SUCCESSFUL in 33m 29s
10 actionable tasks: 10 executed
$ cat /etc/puppet/hieradata/site.yaml
---
bigtop::hadoop_head_node: ubuntu1804.localdomain
hadoop::hadoop_storage_dirs:
- /data
hadoop_cluster_node::cluster_components:
- hdfs
- yarn
- mapreduce
- hive
- oozie
bigtop::bigtop_repo_uri: file:///home/vagrant/bigtop/output/apt
bigtop::jdk_preinstalled: true
bigtop::bigtop_repo_gpg_check: false
$ sudo puppet apply --hiera_config=/etc/puppet/hiera.yaml --modulepath=bigtop-deploy/puppet/modules:/usr/share/puppet/modules bigtop-deploy/puppet/manifests
(snip)
Notice: Applied catalog in 320.00 seconds
```
Upload Oozie sharelib manually and run Oozie smoke test as root:
```
# . /usr/lib/bigtop-utils/bigtop-detect-javahome
# export HADOOP_HOME=/usr/lib/hadoop
# export HADOOP_CONF_DIR=/etc/hadoop/conf
# export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
# export HIVE_HOME=/usr/lib/hive
# export HIVE_CONF_DIR=/etc/hive/conf
# export OOZIE_URL=http://$(hostname -f):11000/oozie/
# export OOZIE_TAR_HOME=/usr/share/doc/oozie
# sudo -u hdfs hdfs dfs -chown -R oozie:oozie /user/oozie
# oozie-setup sharelib create -fs hdfs://$(hostname -f):8020/
(snip)
the destination path for sharelib is: /user/oozie/share/lib/lib_20200922080033
# oozie admin -sharelibupdate
(snip)
[ShareLib update status]
sharelibDirOld = /user/oozie/share/lib
host = http://ubuntu1804.localdomain:11000/oozie
sharelibDirNew = hdfs://ubuntu1804.localdomain:8020/user/oozie/share/lib/lib_20200922080033
status = Successful
# ./gradlew bigtop-tests:smoke-tests:oozie:test -Psmoke.tests --info
(snip)
> Task :bigtop-tests:smoke-tests:oozie:test
Finished generating test XML results (0.003 secs) into: /home/vagrant/bigtop/bigtop-tests/smoke-tests/oozie/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.015 secs) into: /home/vagrant/bigtop/bigtop-tests/smoke-tests/oozie/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:oozie:test (Thread[Daemon worker Thread 3,5,main]) completed. Took 6 mins 58.169 secs.
BUILD SUCCESSFUL in 7m 7s
6 actionable tasks: 6 executed
Stopped 1 worker daemon(s).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org