You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Alejandro Fernandez (JIRA)" <ji...@apache.org> on 2016/02/06 01:30:40 UTC
[jira] [Commented] (AMBARI-14950) EU: Oozie service check failed
after downgrade with multiple Oozie servers since rerunning prepare-war
removes webapps/oozie folder
[ https://issues.apache.org/jira/browse/AMBARI-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135345#comment-15135345 ]
Alejandro Fernandez commented on AMBARI-14950:
----------------------------------------------
Actually, looks like it's related to prepare-war and starting oozie server.
{code}
| 2 | DOWNGRADE | 1965 | Oozie | Preparing Oozie Server on os-r6-tcdlus-dalm20tom21secdwngd-5 | COMPLETED | os-r6-tcdlus-dalm20tom21secdwngd-5.novalocal |
| 2 | DOWNGRADE | 1966 | Oozie | Restarting Oozie Server on 3 hosts | COMPLETED | os-r6-tcdlus-dalm20tom21secdwngd-3.novalocal |
| 2 | DOWNGRADE | 1967 | Oozie | Restarting Oozie Server on 3 hosts | COMPLETED | os-r6-tcdlus-dalm20tom21secdwngd-4.novalocal |
| 2 | DOWNGRADE | 1968 | Oozie | Restarting Oozie Server on 3 hosts | COMPLETED | os-r6-tcdlus-dalm20tom21secdwngd-5.novalocal |
{code}
On host 4, it shows this for output-1967.txt
{code}
2016-02-03 04:04:52,973 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] {'not_if': ..., 'user': 'cstm-oozie'}
2016-02-03 04:04:53,031 - Skipping Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] due to not_if
2016-02-03 04:04:53,032 - File['/usr/hdp/current/oozie-server/.hashcode'] {'content': '046a880c90fcbbfea52bec80cb88dd8f', 'mode': 0644}
2016-02-03 04:04:53,129 - File['/usr/hdp/current/oozie-server/.prepare_war_cmd'] {'content': 'cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war ', 'mode': 0644}
...
######## Hortonworks #############
This is MOTD message, added for testing in qe infra
2016-02-03 04:04:54,089 - call['/usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war -d /usr/hdp/current/oozie-server/libext'] {'logoutput': False, 'user': 'cstm-oozie', 'env': {'CATALINA_BASE': '/usr/hdp/current/oozie-server/oozie-server', 'OOZIE_HOME': '/usr/hdp/current/oozie-server'}, 'quiet': False}
2016-02-03 04:05:08,413 - call returned (0, '######## Hortonworks #############
This is MOTD message, added for testing in qe infra
setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-client/conf}
setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-client/oozie-server}
setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting JAVA_HOME=/usr/jdk64/jdk1.7.0_67
setting JRE_HOME=${JAVA_HOME}
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m -XX:MaxPermSize=256m"
setting OOZIE_LOG=/grid/0/log/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting OOZIE_DATA=/grid/0/hadoop/oozie/data
setting OOZIE_HTTP_PORT=11000
setting OOZIE_ADMIN_PORT=11001
setting JAVA_LIBRARY_PATH=/usr/hdp/2.4.0.0-151/hadoop/lib/native/Linux-amd64-64
setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 "
setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-client/conf}
setting CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-client/oozie-server}
setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting JAVA_HOME=/usr/jdk64/jdk1.7.0_67\n setting JRE_HOME=${JAVA_HOME}
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m -XX:MaxPermSize=256m"
setting OOZIE_LOG=/grid/0/log/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting OOZIE_DATA=/grid/0/hadoop/oozie/data
setting OOZIE_HTTP_PORT=11000
setting OOZIE_ADMIN_PORT=11001
setting JAVA_LIBRARY_PATH=/usr/hdp/2.4.0.0-151/hadoop/lib/native/Linux-amd64-64
setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 "
INFO: Adding extension: /usr/hdp/current/oozie-server/libext/falcon-oozie-el-extension-0.6.1.2.3.4.0-3485.jar
INFO: Adding extension: /usr/hdp/current/oozie-server/libext/hadoop-lzo-0.6.0.2.3.4.0-3485.jar
INFO: Adding extension: /usr/hdp/current/oozie-server/libext/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar
INFO: Adding extension: /usr/hdp/current/oozie-server/libext/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar
INFO: Adding extension: /usr/hdp/current/oozie-server/libext/mysql-connector-java.jar
New Oozie WAR file with added \'ExtJS library, JARs\' at /usr/hdp/current/oozie-server/oozie-server/webapps/oozie.war
INFO: Oozie is ready to be started')
2016-02-03 04:05:08,415 - Execute['cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-start.sh'] {'environment': {'OOZIE_CONFIG': '/usr/hdp/current/oozie-server/conf'}, 'not_if': "ambari-sudo.sh su cstm-oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid >/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'", 'user': 'cstm-oozie'}
{code}
The timestamp of /usr/hdp/current/oozie-server/oozie-server/webapps/oozie.war shows it is from Feb 3 04:05, which is right when the oozie servers were all restarted during downgrade.
However, the /usr/hdp/current/oozie-server/oozie-server/webapps/oozie folder was only present after I restarted Oozie Server manually.
> EU: Oozie service check failed after downgrade with multiple Oozie servers since rerunning prepare-war removes webapps/oozie folder
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMBARI-14950
> URL: https://issues.apache.org/jira/browse/AMBARI-14950
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.2.1
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
> Fix For: 2.2.2
>
>
> *Steps*
> # Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, non-HA cluster)
> # Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
> # Perform Downgrade back to original version and let it complete
> # Run service check for Oozie
> *Result*
> Service check failed with error:
> {code}
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf /usr/hdp/current/hadoop-client/bin ambari-qa True /etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM' returned 1. /usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM; source /usr/hdp/current/oozie-client/conf/oozie-env.sh ; /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job -oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
> Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 404, message: Not Found
> Invalid sub-command: Missing argument for option: info
> use 'help [sub-command]' for help details
> Invalid sub-command: Missing argument for option: info
> use 'help [sub-command]' for help details
> workflow_status=
> {code}
> catalina.out prints below
> {code}
> Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext resourcesStart
> SEVERE: Error starting static Resources
> java.lang.IllegalArgumentException: Document base /usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is not a readable directory
> at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
> at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324)
> Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader loadClass
> INFO: Illegal access: this web application instance has been stopped already. Could not load com.google.common.cache.RemovalCause. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
> java.lang.IllegalStateException
> at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
> at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> {code}
> Attached complete logs
> *Note - I did a restart of Oozie service and the issue is no longer seen*
> Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got created as part of Oozie service restart
> {code}
> ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/
> total 136M
> lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs -> /usr/hdp/2.3.4.0-3485/oozie/doc
> lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 -> /usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2
> -rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war
> *drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)