question about region files

[Jack Levin <ma...@gmail.com>]

We are importing lots and lots of data, 130 TB worth.   If we set
compaction limit to say 128, and blocking limit to say 200.  I know we
expect longer read times unless we use a bloomfilter, however, are
they any other detrimental performance issues to be expected?  Our
flush size limit is set to 100MB.  I notice that REST server sometimes
is unable to talk to regions if there are too many store files, is
that expected behavior?  During massive imports it seems that we run
into problems with REST server freezes (e.g. responds to tcp, but
neither to puts or gets), and CPU goes crazy high, and stays high
until we restart the REST server.  Any ideas?
(I've attached link to jstack trace in my previous email).

-Jack

RE: question about region files

["Gibbon, Robert, VF-Group" <Ro...@vodafone.com>]

This smells of garbage and low memory.

See for ref a similar problem report here - http://kr.forums.oracle.com/forums/thread.jspa?messageID=2146733

How many rest servers do you have loading all of that data? AFAIK they're stateless and loadbalancable

#
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000041260400 nid=0x2412 runnable
#
 
#
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000041261c00 nid=0x2413 runnable
#
 
#
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000412b6c00 nid=0x2414 runnable 

-----Original Message-----
From: Jack Levin [mailto:magnito@gmail.com]
Sent: Sun 10/10/2010 5:24 AM
To: user@hbase.apache.org
Subject: question about region files
 
We are importing lots and lots of data, 130 TB worth.   If we set
compaction limit to say 128, and blocking limit to say 200.  I know we
expect longer read times unless we use a bloomfilter, however, are
they any other detrimental performance issues to be expected?  Our
flush size limit is set to 100MB.  I notice that REST server sometimes
is unable to talk to regions if there are too many store files, is
that expected behavior?  During massive imports it seems that we run
into problems with REST server freezes (e.g. responds to tcp, but
neither to puts or gets), and CPU goes crazy high, and stays high
until we restart the REST server.  Any ideas?
(I've attached link to jstack trace in my previous email).

-Jack

Re: question about region files

[Stack <st...@duboce.net>]

On Sat, Oct 9, 2010 at 8:24 PM, Jack Levin <ma...@gmail.com> wrote:
> We are importing lots and lots of data, 130 TB worth.   If we set
> compaction limit to say 128, and blocking limit to say 200.  I know we
> expect longer read times unless we use a bloomfilter, however, are
> they any other detrimental performance issues to be expected?  Our
> flush size limit is set to 100MB.  I notice that REST server sometimes
> is unable to talk to regions if there are too many store files, is
> that expected behavior?

No.

Might be something to do w/ finding a newly split region; perhaps it
took a while to split?  Can you find which region was problematic then
grep it in master log to see what it was up around this time?


 During massive imports it seems that we run
> into problems with REST server freezes (e.g. responds to tcp, but
> neither to puts or gets), and CPU goes crazy high, and stays high
> until we restart the REST server.  Any ideas?
> (I've attached link to jstack trace in my previous email).
>

There was the OOME issue.  That would likely express itself as lots of
CPU in the run up to the OOME.  Can you find the get or put that is
stalling, figure the region its intended for and again grep master
logs?

(I haven't looked at the jstacks -- will do so later)

St.Ack