You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Kang Minwoo <mi...@outlook.com> on 2018/04/24 01:02:07 UTC

Region Server down when hadoop namenode switch

Hello, All

I changed Hadoop name node manually. Because I had some reason.
After that some region server down.
Error logs below

HBase version: 1.26
Hadoop version: 2.7.3

Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file ${hbase hdfs path}. Name node is in safe mode.
The reported blocks 7565873 has reached the threshold 1.0000 of total blocks 7565873. The number of live datanodes 390 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 18 seconds.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1335)
        ... 13 more
        at org.apache.hadoop.ipc.Client.call(Client.java:1475)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy16.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
        at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy17.create(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
        at com.sun.proxy.$Proxy18.create(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1658)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:493)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:489)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:489)
        at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1115)
        at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1091)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
        at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:362)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:734)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:699)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
        at java.lang.Thread.run(Thread.java:748)
2018-04-18 13:55:02,600 FATAL [regionserver/${server host}.logRoller] regionserver.LogRoller: Aborting
java.io.IOException: cannot get log writer
        at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:373)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:734)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:699)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
        at java.lang.Thread.run(Thread.java:748)

I add Hadoop configuration.

<property>
    <name>dfs.client.retry.policy.enabled</name>
    <value>true</value>
</property>

But It had no effect.

Best regards,
Minwoo Kang

Re: Region Server down when hadoop namenode switch

Posted by David Quiroga <qu...@gmail.com>.
SafeMode occurs during NameNode start-up. While in safe mode the NameNode
will refuse to provide any "services", if restarting both NameNodes be sure
to allow time for them to exit safe mode. Status can typically be viewed
from NameNode UI port 50070.

  Safemode for the NameNode is essentially a read-only mode for the HDFS
cluster, where it does not allow any modifications to file system or blocks.

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Safemode


On Mon, Apr 23, 2018 at 8:02 PM, Kang Minwoo <mi...@outlook.com>
wrote:

> Hello, All
>
> I changed Hadoop name node manually. Because I had some reason.
> After that some region server down.
> Error logs below
>
> HBase version: 1.26
> Hadoop version: 2.7.3
>
> Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException:
> Cannot create file ${hbase hdfs path}. Name node is in safe mode.
> The reported blocks 7565873 has reached the threshold 1.0000 of total
> blocks 7565873. The number of live datanodes 390 has reached the minimum
> number 0. In safe mode extension. Safe mode will be turned off
> automatically in 18 seconds.
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> checkNameNodeSafeMode(FSNamesystem.java:1335)
>         ... 13 more
>         at org.apache.hadoop.ipc.Client.call(Client.java:1475)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:229)
>         at com.sun.proxy.$Proxy16.create(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslat
> orPB.java:296)
>         at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:191)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy17.create(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(
> HFileSystem.java:279)
>         at com.sun.proxy.$Proxy18.create(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(
> DFSOutputStream.java:1648)
>         at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
>         at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1658)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$8.
> doCall(DistributedFileSystem.java:493)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$8.
> doCall(DistributedFileSystem.java:489)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.
> createNonRecursive(DistributedFileSystem.java:489)
>         at org.apache.hadoop.fs.FileSystem.createNonRecursive(
> FileSystem.java:1115)
>         at org.apache.hadoop.fs.FileSystem.createNonRecursive(
> FileSystem.java:1091)
>         at org.apache.hadoop.hbase.regionserver.wal.
> ProtobufLogWriter.init(ProtobufLogWriter.java:90)
>         at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(
> DefaultWALProvider.java:362)
>         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.
> createWriterInstance(FSHLog.java:734)
>         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.
> rollWriter(FSHLog.java:699)
>         at org.apache.hadoop.hbase.regionserver.LogRoller.run(
> LogRoller.java:148)
>         at java.lang.Thread.run(Thread.java:748)
> 2018-04-18 13:55:02,600 FATAL [regionserver/${server host}.logRoller]
> regionserver.LogRoller: Aborting
> java.io.IOException: cannot get log writer
>         at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(
> DefaultWALProvider.java:373)
>         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.
> createWriterInstance(FSHLog.java:734)
>         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.
> rollWriter(FSHLog.java:699)
>         at org.apache.hadoop.hbase.regionserver.LogRoller.run(
> LogRoller.java:148)
>         at java.lang.Thread.run(Thread.java:748)
>
> I add Hadoop configuration.
>
> <property>
>     <name>dfs.client.retry.policy.enabled</name>
>     <value>true</value>
> </property>
>
> But It had no effect.
>
> Best regards,
> Minwoo Kang
>