You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Alejandro Fernandez (JIRA)" <ji...@apache.org> on 2016/02/06 01:30:40 UTC
[jira] [Updated] (AMBARI-14950) EU: Oozie service check failed
after downgrade with multiple Oozie servers since rerunning prepare-war
removes webapps/oozie folder
[ https://issues.apache.org/jira/browse/AMBARI-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Fernandez updated AMBARI-14950:
-----------------------------------------
Description:
*Steps*
# Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, non-HA cluster)
# Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
# Perform Downgrade back to original version and let it complete
# Run service check for Oozie
*Result*
Service check failed with error:
{code}
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf /usr/hdp/current/hadoop-client/bin ambari-qa True /etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM' returned 1. /usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM; source /usr/hdp/current/oozie-client/conf/oozie-env.sh ; /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job -oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 404, message: Not Found
Invalid sub-command: Missing argument for option: info
use 'help [sub-command]' for help details
Invalid sub-command: Missing argument for option: info
use 'help [sub-command]' for help details
workflow_status=
{code}
catalina.out prints below
{code}
Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base /usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is not a readable directory
at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324)
Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.google.common.cache.RemovalCause. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
{code}
Attached complete logs
*Note - I did a restart of Oozie service and the issue is no longer seen*
Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got created as part of Oozie service restart
{code}
ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/
total 136M
lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs -> /usr/hdp/2.3.4.0-3485/oozie/doc
lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 -> /usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2
-rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war
*drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
{code}
*Running the prepare-war command multiple times causes it to remove the /usr/hdp/current/oozie-server/oozie-server/webapps/oozie folder.*
Further, there's a bug that the /usr/hdp/current/oozie-server/.prepare_war_cmd marker file is still generated even if the prepare war command was not ran!
was:
*Steps*
# Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, non-HA cluster)
# Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
# Perform Downgrade back to original version and let it complete
# Run service check for Oozie
*Result*
Service check failed with error:
{code}
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf /usr/hdp/current/hadoop-client/bin ambari-qa True /etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM' returned 1. /usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM; source /usr/hdp/current/oozie-client/conf/oozie-env.sh ; /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job -oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 404, message: Not Found
Invalid sub-command: Missing argument for option: info
use 'help [sub-command]' for help details
Invalid sub-command: Missing argument for option: info
use 'help [sub-command]' for help details
workflow_status=
{code}
catalina.out prints below
{code}
Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base /usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is not a readable directory
at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324)
Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load com.google.common.cache.RemovalCause. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
{code}
Attached complete logs
*Note - I did a restart of Oozie service and the issue is no longer seen*
Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got created as part of Oozie service restart
{code}
ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/
total 136M
lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs -> /usr/hdp/2.3.4.0-3485/oozie/doc
lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 -> /usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2
-rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war
*drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
{code}
> EU: Oozie service check failed after downgrade with multiple Oozie servers since rerunning prepare-war removes webapps/oozie folder
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMBARI-14950
> URL: https://issues.apache.org/jira/browse/AMBARI-14950
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.2.1
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
> Fix For: 2.2.2
>
>
> *Steps*
> # Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, non-HA cluster)
> # Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
> # Perform Downgrade back to original version and let it complete
> # Run service check for Oozie
> *Result*
> Service check failed with error:
> {code}
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf /usr/hdp/current/hadoop-client/bin ambari-qa True /etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM' returned 1. /usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab smk_rndQTuaiSErKGKcXD8vmB6AOd5@HWQE.HORTONWORKS.COM; source /usr/hdp/current/oozie-client/conf/oozie-env.sh ; /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job -oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
> Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 404, message: Not Found
> Invalid sub-command: Missing argument for option: info
> use 'help [sub-command]' for help details
> Invalid sub-command: Missing argument for option: info
> use 'help [sub-command]' for help details
> workflow_status=
> {code}
> catalina.out prints below
> {code}
> Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext resourcesStart
> SEVERE: Error starting static Resources
> java.lang.IllegalArgumentException: Document base /usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is not a readable directory
> at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
> at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324)
> Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader loadClass
> INFO: Illegal access: this web application instance has been stopped already. Could not load com.google.common.cache.RemovalCause. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
> java.lang.IllegalStateException
> at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
> at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> {code}
> Attached complete logs
> *Note - I did a restart of Oozie service and the issue is no longer seen*
> Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got created as part of Oozie service restart
> {code}
> ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/
> total 136M
> lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs -> /usr/hdp/2.3.4.0-3485/oozie/doc
> lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 -> /usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2
> -rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war
> *drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
> {code}
> *Running the prepare-war command multiple times causes it to remove the /usr/hdp/current/oozie-server/oozie-server/webapps/oozie folder.*
> Further, there's a bug that the /usr/hdp/current/oozie-server/.prepare_war_cmd marker file is still generated even if the prepare war command was not ran!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)