You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2020/09/29 07:05:10 UTC
[GitHub] [bigtop] sekikn edited a comment on pull request #676: BIGTOP-3406. Fix Oozie smoke test to work.
sekikn edited a comment on pull request #676:
URL: https://github.com/apache/bigtop/pull/676#issuecomment-700493035
I updated the PR. With this PR, I confirmed the smoke test succeeded on Ubuntu 18.04, as follows:
```
~/repos/bigtop/provisioner/docker$ cat config_ubuntu-18.04.yaml
(snip)
docker:
memory_limit: "4g"
image: "bigtop/puppet:trunk-ubuntu-18.04"
repo: "file:///bigtop-home/output/apt"
distro: debian
components: [hdfs, yarn, mapreduce, hive, oozie]
enable_local_repo: true
smoke_test_components: [oozie]
~/repos/bigtop/provisioner/docker$ ./docker-hadoop.sh -d -C config_ubuntu-18.04.yaml -c 1 -s
(snip)
> Task :bigtop-tests:smoke-tests:oozie:test
Finished generating test XML results (0.011 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.016 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:oozie:test (Thread[Daemon worker,5,main]) completed. Took 6 mins 25.139 secs.
BUILD SUCCESSFUL in 11m 39s
```
But even with this, the smoke test failed on CentOS 8.
```
$ ./docker-hadoop.sh -d -C config_centos-8.yaml -c 1 -s
(snip)
===== START TO RUN SMOKE TESTS: oozie =====
(snip)
+ oozie-setup sharelib create -fs hdfs://325b0282b92b.bigtop.apache.org:8020/
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting CATALINA_TMPDIR=/var/lib/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_LOG=/var/log/oozie
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting CATALINA_TMPDIR=/var/lib/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting CATALINA_BASE=/var/lib/oozie/tomcat-deployment
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_LOG=/var/log/oozie
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more
+ oozie admin -sharelibupdate
Error: Could not find or load main class org.apache.oozie.cli.OozieCLI
(snip)
> Task :bigtop-tests:smoke-tests:oozie:test
9 tests completed, 7 failed
Finished generating test XML results (0.025 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.052 secs) into: /bigtop-home/bigtop-tests/smoke-tests/oozie/build/reports/tests/test
> Task :bigtop-tests:smoke-tests:oozie:test FAILED
:bigtop-tests:smoke-tests:oozie:test (Thread[Daemon worker,5,main]) completed. Took 16.517 secs.
FAILURE: Build failed with an exception.
```
It looks like Oozie lacks dependent jars on CentOS 8, which are installed on Ubuntu 18.04 with the same deployment manifests.
CentOS 8:
```
$ docker exec -it 20200929_045903_r6957_bigtop_1 ls /usr/lib/oozie/lib
hadoop-annotations.jar hadoop-mapreduce-client-jobclient.jar
hadoop-auth.jar hadoop-mapreduce-client-shuffle.jar
hadoop-common.jar hadoop-yarn-api.jar
hadoop-hdfs-client.jar hadoop-yarn-client.jar
hadoop-mapreduce-client-app.jar hadoop-yarn-common.jar
hadoop-mapreduce-client-common.jar hadoop-yarn-server-common.jar
hadoop-mapreduce-client-core.jar
$ docker exec -it 20200929_045903_r6957_bigtop_1 ls -l /usr/lib/oozie/lib | wc -l
14
```
Ubuntu 18.04:
```
$ docker exec -it 20200929135113r4152_bigtop_1 ls /usr/lib/oozie/lib
HikariCP-2.5.1.jar
HikariCP-java7-2.4.12.jar
JavaEWAH-0.7.9.jar
ST4-4.0.4.jar
accessors-smart-1.2.jar
activation-1.1.jar
activemq-client-5.13.3.jar
aircompressor-0.8.jar
ant-1.9.1.jar
ant-launcher-1.9.1.jar
antlr-2.7.7.jar
antlr-runtime-3.4.jar
aopalliance-1.0.jar
apache-log4j-extras-1.2.17.jar
apacheds-i18n-2.0.0-M15.jar
apacheds-kerberos-codec-2.0.0-M15.jar
api-asn1-api-1.0.0-M20.jar
api-util-1.0.0-M20.jar
asm-3.1.jar
asm-5.0.4.jar
asm-commons-3.1.jar
asm-tree-3.1.jar
avatica-1.8.0.jar
avatica-metrics-1.8.0.jar
avro-1.7.7.jar
bonecp-0.8.0.RELEASE.jar
c3p0-0.9.1.1.jar
calcite-core-1.10.0.jar
calcite-druid-1.10.0.jar
calcite-linq4j-1.10.0.jar
collections-generic-4.01.jar
commons-beanutils-1.9.4.jar
commons-cli-1.2.jar
commons-codec-1.4.jar
commons-collections-3.2.2.jar
commons-compiler-2.7.6.jar
commons-compress-1.9.jar
commons-configuration-1.6.jar
commons-daemon-1.0.13.jar
commons-dbcp-1.4.jar
commons-digester-1.8.jar
commons-el-1.0.jar
commons-httpclient-3.0.1.jar
commons-io-2.1.jar
commons-jexl-2.1.1.jar
commons-lang-2.4.jar
commons-lang3-3.3.2.jar
commons-logging-1.1.jar
commons-math-2.2.jar
commons-math3-3.1.1.jar
commons-net-3.1.jar
commons-pool-1.5.4.jar
curator-client-2.5.0.jar
curator-framework-2.5.0.jar
curator-recipes-2.5.0.jar
curator-x-discovery-2.5.0.jar
datanucleus-api-jdo-4.2.4.jar
datanucleus-core-4.1.17.jar
datanucleus-rdbms-4.1.19.jar
derby-10.10.1.1.jar
disruptor-3.3.0.jar
dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
ehcache-3.3.1.jar
ehcache-core-2.6.3.jar
eigenbase-properties-1.1.5.jar
fastutil-6.5.6.jar
findbugs-annotations-1.3.9-1.jar
fst-2.50.jar
geronimo-annotation_1.0_spec-1.1.1.jar
geronimo-j2ee-management_1.1_spec-1.0.1.jar
geronimo-jaspic_1.0_spec-1.0.jar
geronimo-jcache_1.0_spec-1.0-alpha-1.jar
geronimo-jms_1.1_spec-1.1.1.jar
geronimo-jpa_2.0_spec-1.1.jar
geronimo-jta_1.1_spec-1.1.1.jar
groovy-all-2.4.4.jar
gson-2.7.jar
guava-11.0.2.jar
guice-3.0.jar
guice-assistedinject-3.0.jar
guice-servlet-3.0.jar
hadoop-annotations.jar
hadoop-auth-2.10.0.jar
hadoop-auth.jar
hadoop-common.jar
hadoop-hdfs-client.jar
hadoop-mapreduce-client-app.jar
hadoop-mapreduce-client-common.jar
hadoop-mapreduce-client-core.jar
hadoop-mapreduce-client-jobclient.jar
hadoop-mapreduce-client-shuffle.jar
hadoop-yarn-api.jar
hadoop-yarn-client.jar
hadoop-yarn-common.jar
hadoop-yarn-server-common.jar
hawtbuf-1.11.jar
hbase-annotations-1.1.1.jar
hbase-client-1.1.1.jar
hbase-common-1.1.1-tests.jar
hbase-common-1.1.1.jar
hbase-hadoop-compat-1.1.1.jar
hbase-hadoop2-compat-1.1.1.jar
hbase-prefix-tree-1.1.1.jar
hbase-procedure-1.1.1.jar
hbase-protocol-1.1.1.jar
hbase-server-1.1.1.jar
hive-common-2.3.6.jar
hive-exec-2.3.6-core.jar
hive-exec-2.3.6.jar
hive-hcatalog-core-2.3.6.jar
hive-hcatalog-server-extensions-2.3.6.jar
hive-jdbc-2.3.6.jar
hive-llap-client-2.3.6.jar
hive-llap-common-2.3.6-tests.jar
hive-llap-common-2.3.6.jar
hive-llap-server-2.3.6.jar
hive-llap-tez-2.3.6.jar
hive-metastore-2.3.6.jar
hive-serde-2.3.6.jar
hive-service-2.3.6.jar
hive-service-rpc-2.3.6.jar
hive-shims-0.23-2.3.6.jar
hive-shims-2.3.6.jar
hive-shims-common-2.3.6.jar
hive-shims-scheduler-2.3.6.jar
hive-storage-api-2.4.0.jar
hive-vector-code-gen-2.3.6.jar
hive-webhcat-java-client-2.3.6.jar
hsqldb-1.8.0.10.jar
htrace-core-3.1.0-incubating.jar
htrace-core4-4.1.0-incubating.jar
httpclient-4.3.6.jar
httpcore-4.3.3.jar
ivy-2.4.0.jar
jackson-annotations-2.2.2.jar
jackson-core-2.2.2.jar
jackson-core-asl-1.9.13.jar
jackson-databind-2.2.2.jar
jackson-jaxrs-1.9.13.jar
jackson-mapper-asl-1.9.13.jar
jackson-xc-1.9.13.jar
jakarta.jms-api-2.0.2.jar
jamon-runtime-2.3.1.jar
janino-2.7.6.jar
java-util-1.9.0.jar
javax.inject-1.jar
javax.jdo-3.2.0-m3.jar
javax.servlet-3.0.0.v201112011016.jar
javolution-5.5.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcip-annotations-1.0-1.jar
jcodings-1.0.8.jar
jcommander-1.30.jar
jdo-api-3.0.1.jar
jdom-1.1.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jettison-1.1.jar
jetty-6.1.14.jar
jetty-all-7.6.0.v20120127.jar
jetty-sslengine-6.1.26.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
joda-time-2.1.jar
jol-core-0.2.jar
joni-2.1.2.jar
jpam-1.1.jar
jsch-0.1.50.jar
json-1.8.jar
json-io-2.5.1.jar
json-simple-1.1.jar
json-smart-2.3.jar
jsp-2.1-6.1.14.jar
jsp-api-2.0.jar
jsr305-1.3.9.jar
jta-1.1.jar
jung-algorithms-2.0.1.jar
jung-api-2.0.1.jar
jung-graph-impl-2.0.1.jar
jung-visualization-2.0.1.jar
kryo-2.22.jar
leveldbjni-all-1.8.jar
libfb303-0.9.3.jar
libthrift-0.9.3.jar
log4j-1.2.17.jar
log4j-api-2.6.2.jar
log4j-core-2.6.2.jar
log4j-slf4j-impl-2.6.2.jar
log4j-web-2.6.2.jar
mail-1.4.jar
metrics-core-2.2.0.jar
metrics-core-3.0.2.jar
metrics-core-3.1.2.jar
metrics-ganglia-3.1.2.jar
metrics-graphite-3.1.2.jar
metrics-json-3.0.2.jar
metrics-json-3.1.0.jar
metrics-jvm-3.0.2.jar
metrics-jvm-3.1.0.jar
mssql-jdbc-6.2.1.jre7.jar
netty-3.10.5.Final.jar
netty-3.7.0.Final.jar
netty-all-4.0.23.Final.jar
nimbus-jose-jwt-4.41.1.jar
okhttp-2.7.5.jar
okio-1.6.0.jar
oozie-client-4.3.0.jar
oozie-core-4.3.0.jar
oozie-hadoop-auth-hadoop-2-4.3.0.jar
oozie-hadoop-utils-hadoop-2-4.3.0.jar
oozie-sharelib-distcp-4.3.0.jar
oozie-sharelib-hcatalog-4.3.0.jar
oozie-sharelib-hive-4.3.0.jar
oozie-sharelib-hive2-4.3.0.jar
oozie-sharelib-oozie-4.3.0.jar
oozie-sharelib-spark-4.3.0.jar
oozie-sharelib-sqoop-4.3.0.jar
oozie-sharelib-streaming-4.3.0.jar
opencsv-2.3.jar
openjpa-jdbc-2.4.1.jar
openjpa-kernel-2.4.1.jar
openjpa-lib-2.4.1.jar
openjpa-persistence-2.4.1.jar
openjpa-persistence-jdbc-2.4.1.jar
orc-core-1.3.4.jar
org.eclipse.jgit-3.3.1.201403241930-r.jar
oro-2.0.8.jar
paranamer-2.3.jar
parquet-hadoop-bundle-1.8.1.jar
pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
postgresql-9.0-801.jdbc4.jar
protobuf-java-2.5.0.jar
quartz-2.1.7.jar
serp-1.15.1.jar
servlet-api-2.5-6.1.14.jar
servlet-api-2.5.jar
slf4j-api-1.6.6.jar
slf4j-log4j12-1.6.6.jar
slf4j-simple-1.6.6.jar
slice-0.29.jar
slider-core-0.90.2-incubating.jar
snappy-java-1.0.5.jar
stax-api-1.0-2.jar
stax-api-1.0.1.jar
stax2-api-3.1.4.jar
stringtemplate-3.2.1.jar
tephra-api-0.6.0.jar
tephra-core-0.6.0.jar
tephra-hbase-compat-1.0-0.6.0.jar
transaction-api-1.1.jar
twill-api-0.6.0-incubating.jar
twill-common-0.6.0-incubating.jar
twill-core-0.6.0-incubating.jar
twill-discovery-api-0.6.0-incubating.jar
twill-discovery-core-0.6.0-incubating.jar
twill-zookeeper-0.6.0-incubating.jar
velocity-1.5.jar
woodstox-core-5.0.3.jar
xbean-asm5-shaded-3.17.jar
xercesImpl-2.10.0.jar
xml-apis-1.4.01.jar
xmlenc-0.52.jar
zookeeper-3.4.6.jar
zookeeper-3.4.9.jar
$ docker exec -it 20200929135113r4152_bigtop_1 ls /usr/lib/oozie/lib | wc -l
268
```
I'll continue the investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org