You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Harold Lim <ro...@yahoo.com> on 2011/04/28 21:52:09 UTC
Write Operations on a Table Hangs
Hi All,
Sometimes when I try to write to a table (I have a program that writes lots of rows to an hbase table), it hangs and then I can't write to the table anymore.
When I look at a region server's log, I see the message
13:49:59,945 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: waiting for cache flush to complete for region usertable
I try to disable/drop the table using the hbase shell but it keeps saying that the table is not disabled.
Any idea what's causing this?
Thanks,
Harold
Re: Write Operations on a Table Hangs
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yes, there are pauses around splits, and it's getting better with
newer versions (0.20 is old). In 0.90 you can pre-split the tables
when importing huge datasets
http://hbase.apache.org/book/important_configurations.html#disable.splitting
You could try jstack'ing the region servers to figure where it's
stuck, but unless you are using a 0.20 that's older than 0.20.6 the
probability of getting a dead lock is extremely low. Please use this
guide to debug HBase issues: http://hbase.apache.org/book/trouble.html
EC2 might be another issue by itself.
J-D
On Thu, Apr 28, 2011 at 2:59 PM, Harold Lim <ro...@yahoo.com> wrote:
> Hi J-D,
>
> What I notice is that, when a region server is going to split the region (create more regions), the write operations kind of paused, and then after a few seconds, the write operations of my program will continue.
>
> However, there are cases where the write operations does not work anymore. I go to the hbase shell and try to do some operations on the table and it does not work too. Operations on other table still seem to work fine.
>
>
> I'm using HBase 0.20.x (manually compiled). It's a 10 node cluster on EC2. No special configurations.
>
>
> Thanks,
> Harold
>
>
>
> --- On Thu, 4/28/11, Jean-Daniel Cryans <jd...@apache.org> wrote:
>
>> From: Jean-Daniel Cryans <jd...@apache.org>
>> Subject: Re: Write Operations on a Table Hangs
>> To: user@hbase.apache.org
>> Date: Thursday, April 28, 2011, 4:55 PM
>> That message only means that the
>> region is being flushed before it's closed.
>>
>> When you say you can't write to it anymore, you mean at the
>> moment of
>> writing or if you let it alone for a few minutes and then
>> come back
>> would you be able to?
>>
>> It's not a behavior that I remember seeing, it might just
>> be your
>> description that's different. In any case, I've never seen
>> disabling a
>> table really doing any good.
>>
>> And which HBase version is that? What kind of cluster? Any
>> special
>> configuration? Anything about "blocking" in the logs?
>>
>> Thx,
>>
>> J-D
>>
>> On Thu, Apr 28, 2011 at 12:52 PM, Harold Lim <ro...@yahoo.com>
>> wrote:
>> > Hi All,
>> >
>> > Sometimes when I try to write to a table (I have a
>> program that writes lots of rows to an hbase table), it
>> hangs and then I can't write to the table anymore.
>> > When I look at a region server's log, I see the
>> message
>> > 13:49:59,945 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: waiting for
>> cache flush to complete for region usertable
>> >
>> > I try to disable/drop the table using the hbase shell
>> but it keeps saying that the table is not disabled.
>> >
>> > Any idea what's causing this?
>> >
>> > Thanks,
>> > Harold
>> >
>> >
>>
>
Re: Write Operations on a Table Hangs
Posted by Harold Lim <ro...@yahoo.com>.
Hi J-D,
What I notice is that, when a region server is going to split the region (create more regions), the write operations kind of paused, and then after a few seconds, the write operations of my program will continue.
However, there are cases where the write operations does not work anymore. I go to the hbase shell and try to do some operations on the table and it does not work too. Operations on other table still seem to work fine.
I'm using HBase 0.20.x (manually compiled). It's a 10 node cluster on EC2. No special configurations.
Thanks,
Harold
--- On Thu, 4/28/11, Jean-Daniel Cryans <jd...@apache.org> wrote:
> From: Jean-Daniel Cryans <jd...@apache.org>
> Subject: Re: Write Operations on a Table Hangs
> To: user@hbase.apache.org
> Date: Thursday, April 28, 2011, 4:55 PM
> That message only means that the
> region is being flushed before it's closed.
>
> When you say you can't write to it anymore, you mean at the
> moment of
> writing or if you let it alone for a few minutes and then
> come back
> would you be able to?
>
> It's not a behavior that I remember seeing, it might just
> be your
> description that's different. In any case, I've never seen
> disabling a
> table really doing any good.
>
> And which HBase version is that? What kind of cluster? Any
> special
> configuration? Anything about "blocking" in the logs?
>
> Thx,
>
> J-D
>
> On Thu, Apr 28, 2011 at 12:52 PM, Harold Lim <ro...@yahoo.com>
> wrote:
> > Hi All,
> >
> > Sometimes when I try to write to a table (I have a
> program that writes lots of rows to an hbase table), it
> hangs and then I can't write to the table anymore.
> > When I look at a region server's log, I see the
> message
> > 13:49:59,945 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: waiting for
> cache flush to complete for region usertable
> >
> > I try to disable/drop the table using the hbase shell
> but it keeps saying that the table is not disabled.
> >
> > Any idea what's causing this?
> >
> > Thanks,
> > Harold
> >
> >
>
Re: Write Operations on a Table Hangs
Posted by Jean-Daniel Cryans <jd...@apache.org>.
That message only means that the region is being flushed before it's closed.
When you say you can't write to it anymore, you mean at the moment of
writing or if you let it alone for a few minutes and then come back
would you be able to?
It's not a behavior that I remember seeing, it might just be your
description that's different. In any case, I've never seen disabling a
table really doing any good.
And which HBase version is that? What kind of cluster? Any special
configuration? Anything about "blocking" in the logs?
Thx,
J-D
On Thu, Apr 28, 2011 at 12:52 PM, Harold Lim <ro...@yahoo.com> wrote:
> Hi All,
>
> Sometimes when I try to write to a table (I have a program that writes lots of rows to an hbase table), it hangs and then I can't write to the table anymore.
> When I look at a region server's log, I see the message
> 13:49:59,945 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: waiting for cache flush to complete for region usertable
>
> I try to disable/drop the table using the hbase shell but it keeps saying that the table is not disabled.
>
> Any idea what's causing this?
>
> Thanks,
> Harold
>
>