You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Iulia Zidaru <iu...@1and1.ro> on 2011/03/23 12:23:10 UTC
Region containing .META. frozen
Hi all,
We are using CDH3B4.
The RegionServer containing .META. stopped working under normal load.
The region server process was up, it could be viewed in hbase admin
interface, but the requests, regions, usedHeap, maxHeap metrics were
constant in time and it couldn't be accessed. Other machines continued
working, but the performance was very bad. A huge log was created on the
machine's HDFS. The cluster came back to normal performance when I had
killed the RegionServer.
In regionServer's log we got only a WARNING
(java.nio.channels.ClosedChannelException) cause by the communication
with the client machine. No logs while it was frozen.
Any idea?
Thank you,
Iulia
--
Iulia Zidaru
Java Developer
1&1 Internet AG - Bucharest/Romania - Web Components Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
iulia.zidaru@1and1.ro
0040 31 223 9153
Re: Region containing .META. frozen
Posted by Iulia Zidaru <iu...@1and1.ro>.
Thank you, St. Ack. We'll take the jstack next time.
On 03/23/2011 04:17 PM, Stack wrote:
> Please take a jstack next time. What version of jvm are you using.
> If older,< u21, and no apparent reason for the RegionServer lockup,
> then it might be a JVM issue. Try adding -XX:+UseMembar to your
> HBASE_OPTS (We've seen at least one case of RegionServer lockup
> because of missed notifications).
>
> St.Ack
>
> On Wed, Mar 23, 2011 at 4:23 AM, Iulia Zidaru<iu...@1and1.ro> wrote:
>> Hi all,
>>
>> We are using CDH3B4.
>> The RegionServer containing .META. stopped working under normal load. The
>> region server process was up, it could be viewed in hbase admin interface,
>> but the requests, regions, usedHeap, maxHeap metrics were constant in time
>> and it couldn't be accessed. Other machines continued working, but the
>> performance was very bad. A huge log was created on the machine's HDFS. The
>> cluster came back to normal performance when I had killed the RegionServer.
>> In regionServer's log we got only a WARNING
>> (java.nio.channels.ClosedChannelException) cause by the communication with
>> the client machine. No logs while it was frozen.
>> Any idea?
>>
>> Thank you,
>> Iulia
>>
>>
>> --
>> Iulia Zidaru
>> Java Developer
>>
>> 1&1 Internet AG - Bucharest/Romania - Web Components Romania
>> 18 Mircea Eliade St
>> Sect 1, Bucharest
>> RO Bucharest, 012015
>> iulia.zidaru@1and1.ro
>> 0040 31 223 9153
>>
>>
>>
--
Iulia Zidaru
Java Developer
1&1 Internet AG - Bucharest/Romania - Web Components Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
iulia.zidaru@1and1.ro
0040 31 223 9153
Re: Region containing .META. frozen
Posted by Stack <st...@duboce.net>.
Please take a jstack next time. What version of jvm are you using.
If older, < u21, and no apparent reason for the RegionServer lockup,
then it might be a JVM issue. Try adding -XX:+UseMembar to your
HBASE_OPTS (We've seen at least one case of RegionServer lockup
because of missed notifications).
St.Ack
On Wed, Mar 23, 2011 at 4:23 AM, Iulia Zidaru <iu...@1and1.ro> wrote:
> Hi all,
>
> We are using CDH3B4.
> The RegionServer containing .META. stopped working under normal load. The
> region server process was up, it could be viewed in hbase admin interface,
> but the requests, regions, usedHeap, maxHeap metrics were constant in time
> and it couldn't be accessed. Other machines continued working, but the
> performance was very bad. A huge log was created on the machine's HDFS. The
> cluster came back to normal performance when I had killed the RegionServer.
> In regionServer's log we got only a WARNING
> (java.nio.channels.ClosedChannelException) cause by the communication with
> the client machine. No logs while it was frozen.
> Any idea?
>
> Thank you,
> Iulia
>
>
> --
> Iulia Zidaru
> Java Developer
>
> 1&1 Internet AG - Bucharest/Romania - Web Components Romania
> 18 Mircea Eliade St
> Sect 1, Bucharest
> RO Bucharest, 012015
> iulia.zidaru@1and1.ro
> 0040 31 223 9153
>
>
>