You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Varun Sharma <va...@pinterest.com> on 2013/01/23 18:57:20 UTC

Region server crash while syncing HLog

Hi,

I had 2 region servers crash with this (running hbase-0.94.3 with
hadoop-2.0.0). There is also stuff saying that Memstore was rolled back. Is
this some kind of data loss happening ?

2013-01-23 09:14:23,119 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
close of hlog
java.io.IOException: Reflection
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
        at
org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328)
        at
org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437)
        at
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205)
        at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245)
        at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493)
        at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
        ... 11 more
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
No lease on
/hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725
File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does not
have any open files.
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231)
        at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501)
        at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316)
        at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44082)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)

                                                 4163,64-71     1%

Thanks
Varun

Re: Region server crash while syncing HLog

Posted by Ted Yu <yu...@gmail.com>.
Side note: 0.94.3 is not certified for hadoop 2.0

Cheers

On Wed, Jan 23, 2013 at 10:01 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> This is something to do with HDFS also.  Regarding data loss, the client
> should have known that the put has failed.  So there should not be silent
> data loss as far as i can think of.
> Any way further investigation can make things clearer.
>
> Regards
> Ram
>
> On Wed, Jan 23, 2013 at 11:29 PM, Kevin O'dell <kevin.odell@cloudera.com
> >wrote:
>
> > Varun,
> >
> >   I suspect there might be a FATAL or ERROR above that message?  That
> > message typically means the HLog can't write to HDFS, you may want to
> look
> > at the DN logs around that time.
> >
> > On Wed, Jan 23, 2013 at 12:57 PM, Varun Sharma <va...@pinterest.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I had 2 region servers crash with this (running hbase-0.94.3 with
> > > hadoop-2.0.0). There is also stuff saying that Memstore was rolled
> back.
> > Is
> > > this some kind of data loss happening ?
> > >
> > > 2013-01-23 09:14:23,119 FATAL
> > > org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync.
> Requesting
> > > close of hlog
> > > java.io.IOException: Reflection
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
> > >         at
> > > org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328)
> > >         at
> > > org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493)
> > >         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >         at java.lang.reflect.Method.invoke(Method.java:597)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438)
> > > Caused by: java.lang.reflect.InvocationTargetException
> > >         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >         at java.lang.reflect.Method.invoke(Method.java:597)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
> > >         ... 11 more
> > > Caused by:
> > >
> > >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> > > No lease on
> > >
> > >
> >
> /hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725
> > > File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does
> > not
> > > have any open files.
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44082)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
> > >         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
> > >         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
> > >         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
> > >
> > >                                                  4163,64-71     1%
> > >
> > > Thanks
> > > Varun
> > >
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
> >
>

Re: Region server crash while syncing HLog

Posted by ramkrishna vasudevan <ra...@gmail.com>.
This is something to do with HDFS also.  Regarding data loss, the client
should have known that the put has failed.  So there should not be silent
data loss as far as i can think of.
Any way further investigation can make things clearer.

Regards
Ram

On Wed, Jan 23, 2013 at 11:29 PM, Kevin O'dell <ke...@cloudera.com>wrote:

> Varun,
>
>   I suspect there might be a FATAL or ERROR above that message?  That
> message typically means the HLog can't write to HDFS, you may want to look
> at the DN logs around that time.
>
> On Wed, Jan 23, 2013 at 12:57 PM, Varun Sharma <va...@pinterest.com>
> wrote:
>
> > Hi,
> >
> > I had 2 region servers crash with this (running hbase-0.94.3 with
> > hadoop-2.0.0). There is also stuff saying that Memstore was rolled back.
> Is
> > this some kind of data loss happening ?
> >
> > 2013-01-23 09:14:23,119 FATAL
> > org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
> > close of hlog
> > java.io.IOException: Reflection
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328)
> >         at
> > org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493)
> >         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> >         at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >         at java.lang.reflect.Method.invoke(Method.java:597)
> >         at
> >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> >         at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438)
> > Caused by: java.lang.reflect.InvocationTargetException
> >         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> >         at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >         at java.lang.reflect.Method.invoke(Method.java:597)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
> >         ... 11 more
> > Caused by:
> >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> > No lease on
> >
> >
> /hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725
> > File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does
> not
> > have any open files.
> >         at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308)
> >         at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299)
> >         at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231)
> >         at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501)
> >         at
> >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316)
> >         at
> >
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44082)
> >         at
> >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
> >         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
> >         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
> >         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
> >
> >                                                  4163,64-71     1%
> >
> > Thanks
> > Varun
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

Re: Region server crash while syncing HLog

Posted by Kevin O'dell <ke...@cloudera.com>.
Varun,

  I suspect there might be a FATAL or ERROR above that message?  That
message typically means the HLog can't write to HDFS, you may want to look
at the DN logs around that time.

On Wed, Jan 23, 2013 at 12:57 PM, Varun Sharma <va...@pinterest.com> wrote:

> Hi,
>
> I had 2 region servers crash with this (running hbase-0.94.3 with
> hadoop-2.0.0). There is also stuff saying that Memstore was rolled back. Is
> this some kind of data loss happening ?
>
> 2013-01-23 09:14:23,119 FATAL
> org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
> close of hlog
> java.io.IOException: Reflection
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
>         at
> org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328)
>         at
> org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493)
>         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
>         ... 11 more
> Caused by:
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> No lease on
>
> /hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725
> File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does not
> have any open files.
>         at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308)
>         at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299)
>         at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231)
>         at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501)
>         at
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316)
>         at
>
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44082)
>         at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
>
>                                                  4163,64-71     1%
>
> Thanks
> Varun
>



-- 
Kevin O'Dell
Customer Operations Engineer, Cloudera