You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Dmytro Sen (JIRA)" <ji...@apache.org> on 2015/08/12 17:21:45 UTC
[jira] [Updated] (AMBARI-12745) Nodemanagers fail to start because
of wrong recovery.dir property
[ https://issues.apache.org/jira/browse/AMBARI-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Sen updated AMBARI-12745:
--------------------------------
Attachment: AMBARI-12745.patch
> Nodemanagers fail to start because of wrong recovery.dir property
> -----------------------------------------------------------------
>
> Key: AMBARI-12745
> URL: https://issues.apache.org/jira/browse/AMBARI-12745
> Project: Ambari
> Issue Type: Bug
> Components: stacks
> Affects Versions: 2.1.1
> Reporter: Dmytro Sen
> Assignee: Dmytro Sen
> Priority: Blocker
> Fix For: 2.1.1
>
> Attachments: AMBARI-12745.patch
>
>
> $ yarn nodemanager -checkHealth
> {noformat}
> 15/08/07 15:45:24 INFO nodemanager.NodeManager: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NodeManager
> STARTUP_MSG: host = os-u14-chwavu-oozie-ha-1-5/172.22.126.134
> STARTUP_MSG: args = [-checkHealth]
> STARTUP_MSG: version = 2.7.1.2.3.2.0-2602
> STARTUP_MSG: classpath = /usr/hdp/2.3.2.0-2602/hadoop/conf:/usr/hdp/2.3.2.0-2602/hadoop/conf:/usr/hdp/2.3.2.0-2602/hadoop/conf:....
> STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r f66cf95e2e9367a74b0ec88b2df33458b6cff2d0; compiled by 'jenkins' on 2015-08-05T21:42Z
> STARTUP_MSG: java = 1.7.0_79
> ************************************************************/
> 15/08/07 15:45:24 INFO nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]
> 15/08/07 15:45:26 INFO recovery.NMLeveldbStateStoreService: Using state database at /nodemanager/recovery-state/yarn-nm-state for recovery
> 15/08/07 15:45:26 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
> at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
> 15/08/07 15:45:26 INFO service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
> at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 15/08/07 15:45:26 FATAL nodemanager.NodeManager: Error starting NodeManager
> org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
> at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 15/08/07 15:45:26 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NodeManager at os-u14-chwavu-oozie-ha-1-5/172.22.126.134
> ************************************************************/
> yarn@os-u14-chwavu-oozie-ha-1-5:/grid/0/hadoop/yarn$ /usr/hdp/current/hadoop-yarn-nodemanager2015-08-07 01:51:06,160 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NodeManager
> STARTUP_MSG: host = os-u14-chwavu-oozie-ha-1-5/172.22.126.134
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 2.7.1.2.3.2.0-2602
> STARTUP_MSG: classpath = /usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/2.3.2.0-2602/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/hadoop-lzo-0.6.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-common-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpmime-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/xz-1.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-yarn-plugin-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-audit-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-cred-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-hdfs-plugin-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/zookeeper-3.4.6.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/hadoop-common-2.7.1.2.3.2.0-2602.jar:/usr/hd...skipping...
> /sbin/yarn-daemon.sh --config /tmp/hadoopConf start nodemanager
> starting nodemanager, logging to /grid/0/log/hadoop/yarn/yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.out
> yarn@os-u14-chwavu-oozie-ha-1-5:/grid/0/hadoop/yarn$ ll /grid/0/log/hadoop/yarn/yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.
> yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.log
> yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.out
> 2015-08-07 01:51:06,160 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - STARTUP_MSG:
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)