[jira] [Created] (AMBARI-12681) Namenode crashes on "No valid image files found"

["Sumit Mohanty (JIRA)" <ji...@apache.org>]

Sumit Mohanty created AMBARI-12681:
--------------------------------------

             Summary: Namenode crashes on "No valid image files found"
                 Key: AMBARI-12681
                 URL: https://issues.apache.org/jira/browse/AMBARI-12681
             Project: Ambari
          Issue Type: Bug
          Components: stacks
    Affects Versions: 2.1.0
            Reporter: Sumit Mohanty
            Assignee: Sumit Mohanty
            Priority: Critical
             Fix For: 2.1.1


Blue print based deployments start both NN in parallel and occasionally the first attempt to bootstrap may fail. Depending on how it fails the second attempt may not succeed (e.g. it may find the folder and decide that bootstrap succeeded). The solution is to call with -force option but only during initial start.

{code}
About to bootstrap Standby ID nn1 from:\n           Nameservice ID: mycluster\n        Other Namenode ID: nn2\n  Other NN's HTTP address: http://headnode1:30070\n  Other NN's IPC  address: headnode1/10.0.0.10:8020\n             Namespace ID: 1877302436\n            Block pool ID: BP-534536850-10.0.0.10-1438814632688\n               Cluster ID: CID-71ad6e08-e934-4a53-bbf9-0c16b679d848\n           Layout version: -60\n       isUpgradeFinalized: true\n=====================================================\n15/08/05 22:45:13 INFO common.Storage: Storage directory /hadoop/hdfs/namenode has been successfully formatted.\n15/08/05 22:45:16 WARN common.Util: Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\n15/08/05 22:45:16 WARN common.Util: Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\n15/08/05 22:45:16 WARN common.Storage: set restore failed storage to true\n15/08/05 22:45:18 FATAL ha.BootstrapStandby: Unable to read transaction ids 1-6 from the configured shared edits storage qjournal://zookeepernode0:8485;zookeepernode1:8485;zookeepernode2:8485/mycluster. Please copy these logs into the shared edits storage or call saveNamespace on the active node.\nError: Gap in transactions. Expected to be able to read up until at least txid 6 but unable to find any edit logs containing txid 6\n15/08/05 22:45:18 INFO util.ExitUtil: Exiting with status 6\n15/08/05 22:45:18 INFO namenode.NameNode: SHUTDOWN_MSG: \n/************************************************************\nSHUTDOWN_MSG: Shutting down NameNode at headnode0/10.0.0.9\n************************************************************/")
2015-08-05 22:45:18,251 - Bootstrap standby namenode failed with 6 error code. Will retry
{code}

{code}
           Layout version: -60\n       isUpgradeFinalized: true\n=====================================================\nRunning in non-interactive mode, and data appears to exist in Storage Directory /hadoop/hdfs/namenode. Not formatting.\n15/08/05 22:45:28 INFO util.ExitUtil: Exiting with status 5\n15/08/05 22:45:28 INFO namenode.NameNode: SHUTDOWN_MSG: \n/************************************************************\nSHUTDOWN_MSG: Shutting down NameNode at headnode0/10.0.0.9\n************************************************************/"
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)