You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Arpit Gupta (Created) (JIRA)" <ji...@apache.org> on 2011/09/30 02:53:46 UTC

[jira] [Created] (HDFS-2389) restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode

restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode
------------------------------------------------------------------------------------------------------------------------

                 Key: HDFS-2389
                 URL: https://issues.apache.org/jira/browse/HDFS-2389
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 0.20.205.0
            Reporter: Arpit Gupta




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-2389) restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode

Posted by "Arpit Gupta (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118280#comment-13118280 ] 

Arpit Gupta commented on HDFS-2389:
-----------------------------------

with append turned on namenode comes out of safemode without any issues.
                
> restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2389
>                 URL: https://issues.apache.org/jira/browse/HDFS-2389
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20.205.0
>            Reporter: Arpit Gupta
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-2389) restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode

Posted by "Arpit Gupta (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117799#comment-13117799 ] 

Arpit Gupta commented on HDFS-2389:
-----------------------------------

Not sure if this is the cause for this. I Noticed when the namenode was running in a good state in the logs

2011-09-30 00:29:04,932 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successfull for rs/HOST@REALM

2011-09-30 00:29:35,046 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successfull for rs/HOST@REALM


Seems like every 30 seconds its doing authorization with the namenode.
                
> restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2389
>                 URL: https://issues.apache.org/jira/browse/HDFS-2389
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20.205.0
>            Reporter: Arpit Gupta
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-2389) restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode

Posted by "Arpit Gupta (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117798#comment-13117798 ] 

Arpit Gupta commented on HDFS-2389:
-----------------------------------

This is what was noticed

When all the services are up and running restart the hadoop cluster while the hbase services are up. After the namenode restart the namenode fails to come out of safemode

the cluster def nn, snn, jt, hbase master are on independent boxes
12 nodes have dn, tt, region servers running


The following is seen in the namenode logs



2011-09-30 00:55:29,997 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 8020, call renewLease(DFSClient_hb_rs_HOST,60020,1317342295863_1317342296509) from IP:56446: error: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot renew lease for DFSClient_hb_rs_HOST,60020,1317342295863_1317342296509. Name node is in safe mode.
The ratio of reported blocks 0.5556 has not reached the threshold 1.0000. Safe mode will be turned off automatically.
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot renew lease for DFSClient_hb_rs_HOST,60020,1317342295863_1317342296509. Name node is in safe mode.
The ratio of reported blocks 0.5556 has not reached the threshold 1.0000. Safe mode will be turned off automatically.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewLease(FSNamesystem.java:2296)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.renewLease(NameNode.java:789)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)


The above trace keeps on repeating for different hosts.

                
> restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2389
>                 URL: https://issues.apache.org/jira/browse/HDFS-2389
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20.205.0
>            Reporter: Arpit Gupta
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (HDFS-2389) restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode

Posted by "Arpit Gupta (Resolved) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arpit Gupta resolved HDFS-2389.
-------------------------------

    Resolution: Invalid

sorry folks user error append was turned off and was leading to this issue
                
> restart dns and namenode when hbase master and region servers are running and the namenode does not come out of safemode
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2389
>                 URL: https://issues.apache.org/jira/browse/HDFS-2389
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20.205.0
>            Reporter: Arpit Gupta
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira