You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Murali Krishna. P" <mu...@yahoo.com> on 2011/03/03 14:49:21 UTC

Log rolling failed with ioe

Hi,
I get following error frequently (twice in a day atleast) and the region server 
closes and exits after this. I am on 0.20.6 and we are stuck with this for some 
time. Will be upgrading to 0.90.0 later, but right now, would be great if we can 
have a workaround. Let me know if you need more info.


 WARNING: DFSClient_1483413998 could not complete file 
/user/x/hbase/.logs/host.com,60020,1299145240135/hlog.dat.1299152151
119.  Retrying...
Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.LogRoller run
SEVERE: Log rolling failed with ioe: 
java.io.IOException: DFSClient_1483413998 could not complete file 
/user/x/hbase/.logs/host,60020,1299145240135/hlog.da
t.1299152151119.  Giving up.
        at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3331)

        at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3240)
        at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)

        at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
        at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959)
        at 
org.apache.hadoop.hbase.regionserver.HLog.cleanupCurrentWriter(HLog.java:491)
        at org.apache.hadoop.hbase.regionserver.HLog.rollWriter(HLog.java:342)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:80)
Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.HLog optionalSync

Thanks,
Murali Krishna

Re: Log rolling failed with ioe

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Those logs are _really_ hard to follow since the master one has a
bunch of stuff edited out... I see the master was already doing what
appears to be log splitting before the beginning of that log and then
who knows what happened in the master after the region server got the
exception because that part is also missing.

Anyways it looks like you're just suffering from GC pauses, there
should be something explicit about it in the region server log after
the SEVERE exception.

Learn more about GC pauses with this presentation:
http://tiny.cloudera.com/wrflf

J-D

On Thu, Mar 3, 2011 at 7:07 PM, Murali Krishna. P
<mu...@yahoo.com> wrote:
> Master: http://pastebin.com/kv3Guz71
> RS: http://pastebin.com/9EixG9GB
>
>  Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Sent: Fri, 4 March, 2011 12:23:06 AM
> Subject: Re: Log rolling failed with ioe
>
> The mailing list strips the attachements, can you pastebin them or
> host them on a web server?
>
> Thx,
>
> J-D
>
> On Thu, Mar 3, 2011 at 10:49 AM, Murali Krishna. P
> <mu...@yahoo.com> wrote:
>> No exceptions before that, sending few lines before them in the log file.
>> Also there was a similar retry error in master log also just before this.
>> Attached both master and rs log
>>
>> Thanks,
>> Murali Krishna
>>
>> ________________________________
>> From: Jean-Daniel Cryans <jd...@apache.org>
>> To: user@hbase.apache.org
>> Sent: Thu, 3 March, 2011 8:33:48 PM
>> Subject: Re: Log rolling failed with ioe
>>
>> Anything that comes before that? More exceptions? Can you see what's
>> at the beginning of them? I'm more interested in the symptoms rather
>> than the side effects.
>>
>> Thx,
>>
>> J-D
>>
>> On Thu, Mar 3, 2011 at 5:49 AM, Murali Krishna. P
>> <mu...@yahoo.com> wrote:
>>> Hi,
>>> I get following error frequently (twice in a day atleast) and the region
>>> server
>>> closes and exits after this. I am on 0.20.6 and we are stuck with this for
>>> some
>>> time. Will be upgrading to 0.90.0 later, but right now, would be great if
>>> we can
>>> have a workaround. Let me know if you need more info.
>>>
>>>
>>>  WARNING: DFSClient_1483413998 could not complete file
>>> /user/x/hbase/.logs/host.com,60020,1299145240135/hlog.dat.1299152151
>>> 119.  Retrying...
>>> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.LogRoller run
>>> SEVERE: Log rolling failed with ioe:
>>> java.io.IOException: DFSClient_1483413998 could not complete file
>>> /user/x/hbase/.logs/host,60020,1299145240135/hlog.da
>>> t.1299152151119.  Giving up.
>>>        at
>>>
>>>org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3331)
>>>)
>>>
>>>        at
>>>
>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3240)
>>>        at
>>>
>>>org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>>>)
>>>
>>>        at
>>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>>>        at
>>> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959)
>>>        at
>>>
>>> org.apache.hadoop.hbase.regionserver.HLog.cleanupCurrentWriter(HLog.java:491)
>>>        at
>>> org.apache.hadoop.hbase.regionserver.HLog.rollWriter(HLog.java:342)
>>>        at
>>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:80)
>>> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.HLog
>>> optionalSync
>>>
>>> Thanks,
>>> Murali Krishna
>>>
>>
>

Re: Log rolling failed with ioe

Posted by "Murali Krishna. P" <mu...@yahoo.com>.
Master: http://pastebin.com/kv3Guz71
RS: http://pastebin.com/9EixG9GB

 Thanks,
Murali Krishna




________________________________
From: Jean-Daniel Cryans <jd...@apache.org>
To: user@hbase.apache.org
Sent: Fri, 4 March, 2011 12:23:06 AM
Subject: Re: Log rolling failed with ioe

The mailing list strips the attachements, can you pastebin them or
host them on a web server?

Thx,

J-D

On Thu, Mar 3, 2011 at 10:49 AM, Murali Krishna. P
<mu...@yahoo.com> wrote:
> No exceptions before that, sending few lines before them in the log file.
> Also there was a similar retry error in master log also just before this.
> Attached both master and rs log
>
> Thanks,
> Murali Krishna
>
> ________________________________
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Sent: Thu, 3 March, 2011 8:33:48 PM
> Subject: Re: Log rolling failed with ioe
>
> Anything that comes before that? More exceptions? Can you see what's
> at the beginning of them? I'm more interested in the symptoms rather
> than the side effects.
>
> Thx,
>
> J-D
>
> On Thu, Mar 3, 2011 at 5:49 AM, Murali Krishna. P
> <mu...@yahoo.com> wrote:
>> Hi,
>> I get following error frequently (twice in a day atleast) and the region
>> server
>> closes and exits after this. I am on 0.20.6 and we are stuck with this for
>> some
>> time. Will be upgrading to 0.90.0 later, but right now, would be great if
>> we can
>> have a workaround. Let me know if you need more info.
>>
>>
>>  WARNING: DFSClient_1483413998 could not complete file
>> /user/x/hbase/.logs/host.com,60020,1299145240135/hlog.dat.1299152151
>> 119.  Retrying...
>> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.LogRoller run
>> SEVERE: Log rolling failed with ioe:
>> java.io.IOException: DFSClient_1483413998 could not complete file
>> /user/x/hbase/.logs/host,60020,1299145240135/hlog.da
>> t.1299152151119.  Giving up.
>>        at
>>
>>org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3331)
>>)
>>
>>        at
>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3240)
>>        at
>>
>>org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>>)
>>
>>        at
>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>>        at
>> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HLog.cleanupCurrentWriter(HLog.java:491)
>>        at
>> org.apache.hadoop.hbase.regionserver.HLog.rollWriter(HLog.java:342)
>>        at
>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:80)
>> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.HLog
>> optionalSync
>>
>> Thanks,
>> Murali Krishna
>>
>

Re: Log rolling failed with ioe

Posted by Jean-Daniel Cryans <jd...@apache.org>.
The mailing list strips the attachements, can you pastebin them or
host them on a web server?

Thx,

J-D

On Thu, Mar 3, 2011 at 10:49 AM, Murali Krishna. P
<mu...@yahoo.com> wrote:
> No exceptions before that, sending few lines before them in the log file.
> Also there was a similar retry error in master log also just before this.
> Attached both master and rs log
>
> Thanks,
> Murali Krishna
>
> ________________________________
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: user@hbase.apache.org
> Sent: Thu, 3 March, 2011 8:33:48 PM
> Subject: Re: Log rolling failed with ioe
>
> Anything that comes before that? More exceptions? Can you see what's
> at the beginning of them? I'm more interested in the symptoms rather
> than the side effects.
>
> Thx,
>
> J-D
>
> On Thu, Mar 3, 2011 at 5:49 AM, Murali Krishna. P
> <mu...@yahoo.com> wrote:
>> Hi,
>> I get following error frequently (twice in a day atleast) and the region
>> server
>> closes and exits after this. I am on 0.20.6 and we are stuck with this for
>> some
>> time. Will be upgrading to 0.90.0 later, but right now, would be great if
>> we can
>> have a workaround. Let me know if you need more info.
>>
>>
>>  WARNING: DFSClient_1483413998 could not complete file
>> /user/x/hbase/.logs/host.com,60020,1299145240135/hlog.dat.1299152151
>> 119.  Retrying...
>> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.LogRoller run
>> SEVERE: Log rolling failed with ioe:
>> java.io.IOException: DFSClient_1483413998 could not complete file
>> /user/x/hbase/.logs/host,60020,1299145240135/hlog.da
>> t.1299152151119.  Giving up.
>>        at
>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3331)
>>
>>        at
>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3240)
>>        at
>>
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>>
>>        at
>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>>        at
>> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959)
>>        at
>>
>> org.apache.hadoop.hbase.regionserver.HLog.cleanupCurrentWriter(HLog.java:491)
>>        at
>> org.apache.hadoop.hbase.regionserver.HLog.rollWriter(HLog.java:342)
>>        at
>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:80)
>> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.HLog
>> optionalSync
>>
>> Thanks,
>> Murali Krishna
>>
>

Re: Log rolling failed with ioe

Posted by "Murali Krishna. P" <mu...@yahoo.com>.
No exceptions before that, sending few lines before them in the log file. Also 
there was a similar retry error in master log also just before this. Attached 
both master and rs log

 Thanks,
Murali Krishna




________________________________
From: Jean-Daniel Cryans <jd...@apache.org>
To: user@hbase.apache.org
Sent: Thu, 3 March, 2011 8:33:48 PM
Subject: Re: Log rolling failed with ioe

Anything that comes before that? More exceptions? Can you see what's
at the beginning of them? I'm more interested in the symptoms rather
than the side effects.

Thx,

J-D

On Thu, Mar 3, 2011 at 5:49 AM, Murali Krishna. P
<mu...@yahoo.com> wrote:
> Hi,
> I get following error frequently (twice in a day atleast) and the region 
server
> closes and exits after this. I am on 0.20.6 and we are stuck with this for 
some
> time. Will be upgrading to 0.90.0 later, but right now, would be great if we 
>can
> have a workaround. Let me know if you need more info.
>
>
>  WARNING: DFSClient_1483413998 could not complete file
> /user/x/hbase/.logs/host.com,60020,1299145240135/hlog.dat.1299152151
> 119.  Retrying...
> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.LogRoller run
> SEVERE: Log rolling failed with ioe:
> java.io.IOException: DFSClient_1483413998 could not complete file
> /user/x/hbase/.logs/host,60020,1299145240135/hlog.da
> t.1299152151119.  Giving up.
>        at
>org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3331)
>)
>
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3240)
>        at
>org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>)
>
>        at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>        at 
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959)
>        at
> org.apache.hadoop.hbase.regionserver.HLog.cleanupCurrentWriter(HLog.java:491)
>        at org.apache.hadoop.hbase.regionserver.HLog.rollWriter(HLog.java:342)
>        at 
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:80)
> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.HLog optionalSync
>
> Thanks,
> Murali Krishna
>

Re: Log rolling failed with ioe

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Anything that comes before that? More exceptions? Can you see what's
at the beginning of them? I'm more interested in the symptoms rather
than the side effects.

Thx,

J-D

On Thu, Mar 3, 2011 at 5:49 AM, Murali Krishna. P
<mu...@yahoo.com> wrote:
> Hi,
> I get following error frequently (twice in a day atleast) and the region server
> closes and exits after this. I am on 0.20.6 and we are stuck with this for some
> time. Will be upgrading to 0.90.0 later, but right now, would be great if we can
> have a workaround. Let me know if you need more info.
>
>
>  WARNING: DFSClient_1483413998 could not complete file
> /user/x/hbase/.logs/host.com,60020,1299145240135/hlog.dat.1299152151
> 119.  Retrying...
> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.LogRoller run
> SEVERE: Log rolling failed with ioe:
> java.io.IOException: DFSClient_1483413998 could not complete file
> /user/x/hbase/.logs/host,60020,1299145240135/hlog.da
> t.1299152151119.  Giving up.
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3331)
>
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3240)
>        at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>
>        at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>        at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959)
>        at
> org.apache.hadoop.hbase.regionserver.HLog.cleanupCurrentWriter(HLog.java:491)
>        at org.apache.hadoop.hbase.regionserver.HLog.rollWriter(HLog.java:342)
>        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:80)
> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.HLog optionalSync
>
> Thanks,
> Murali Krishna
>