You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by steven zhuang <st...@gmail.com> on 2010/03/23 05:53:50 UTC
what does HBase buffer when we do a scan?
hi, all,
I am scanning on a table with 100+ million rows, each has a row key
about 50 bytes, and one column with less than 20 bytes data.
I have set the scancaching to 100000 rows, and with 15 scanners
working the same time, I got OutOfMemory exception, my PC has 6GB ram,
through top I saw the program used more than 1GB ram sometime.
In my calculation, each scanner will buffer 100000*50=5MB, and 15
scanners will buffer 75MB ram, I know each scanner will hold some space,
say, 10MB for each scanner. my question is how come it uses more than 1GB
ram.
anything I have missed?
thanks.
Re: what does HBase buffer when we do a scan?
Posted by steven zhuang <st...@gmail.com>.
no, I havenot, I have done some other trick to make the program work.
On Wed, Mar 24, 2010 at 8:26 AM, ChingShen <ch...@gmail.com> wrote:
> Hi steven,
>
> Have you tuned the java heap size? In HBase, default is 1000(1G).
>
> Shen
>
> On Tue, Mar 23, 2010 at 12:53 PM, steven zhuang <
> steven.zhuang.1984@gmail.com> wrote:
>
> > hi, all,
> > I am scanning on a table with 100+ million rows, each has a row
> key
> > about 50 bytes, and one column with less than 20 bytes data.
> > I have set the scancaching to 100000 rows, and with 15 scanners
> > working the same time, I got OutOfMemory exception, my PC has 6GB ram,
> > through top I saw the program used more than 1GB ram sometime.
> > In my calculation, each scanner will buffer 100000*50=5MB, and 15
> > scanners will buffer 75MB ram, I know each scanner will hold some space,
> > say, 10MB for each scanner. my question is how come it uses more than
> 1GB
> > ram.
> > anything I have missed?
> > thanks.
> >
>
Re: what does HBase buffer when we do a scan?
Posted by ChingShen <ch...@gmail.com>.
Hi steven,
Have you tuned the java heap size? In HBase, default is 1000(1G).
Shen
On Tue, Mar 23, 2010 at 12:53 PM, steven zhuang <
steven.zhuang.1984@gmail.com> wrote:
> hi, all,
> I am scanning on a table with 100+ million rows, each has a row key
> about 50 bytes, and one column with less than 20 bytes data.
> I have set the scancaching to 100000 rows, and with 15 scanners
> working the same time, I got OutOfMemory exception, my PC has 6GB ram,
> through top I saw the program used more than 1GB ram sometime.
> In my calculation, each scanner will buffer 100000*50=5MB, and 15
> scanners will buffer 75MB ram, I know each scanner will hold some space,
> say, 10MB for each scanner. my question is how come it uses more than 1GB
> ram.
> anything I have missed?
> thanks.
>
Re: what does HBase buffer when we do a scan?
Posted by steven zhuang <st...@gmail.com>.
thanks Ryan,
actually when I say 50 bytes per row, I have already included
the row_key&cell value. but beside "data", the in-memory overhead can take
that much memory?
my doubt is that is this all(data and the management overhead)
that the program needs to buffer?
On Tue, Mar 23, 2010 at 1:02 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Alas that is not exactly how it works. You are probably talking about
> 100 bytes per row, plus other in-memory overhead.
>
> Setting scancaching so high is not really the way to go. A few
> thousand is more realistic. You are just trying to amortize the
> per-RPC overhead. 100k rows at a time is kind of... a lot.
>
> The overhead is probably more like 285MB-400MB, not 75MB.
> Unfortunately in-memory representation needs in-memory management as
> well.
>
> -ryan
>
> On Mon, Mar 22, 2010 at 9:53 PM, steven zhuang
> <st...@gmail.com> wrote:
> > hi, all,
> > I am scanning on a table with 100+ million rows, each has a row
> key
> > about 50 bytes, and one column with less than 20 bytes data.
> > I have set the scancaching to 100000 rows, and with 15 scanners
> > working the same time, I got OutOfMemory exception, my PC has 6GB ram,
> > through top I saw the program used more than 1GB ram sometime.
> > In my calculation, each scanner will buffer 100000*50=5MB, and 15
> > scanners will buffer 75MB ram, I know each scanner will hold some space,
> > say, 10MB for each scanner. my question is how come it uses more than
> 1GB
> > ram.
> > anything I have missed?
> > thanks.
> >
>
Re: what does HBase buffer when we do a scan?
Posted by Ryan Rawson <ry...@gmail.com>.
Alas that is not exactly how it works. You are probably talking about
100 bytes per row, plus other in-memory overhead.
Setting scancaching so high is not really the way to go. A few
thousand is more realistic. You are just trying to amortize the
per-RPC overhead. 100k rows at a time is kind of... a lot.
The overhead is probably more like 285MB-400MB, not 75MB.
Unfortunately in-memory representation needs in-memory management as
well.
-ryan
On Mon, Mar 22, 2010 at 9:53 PM, steven zhuang
<st...@gmail.com> wrote:
> hi, all,
> I am scanning on a table with 100+ million rows, each has a row key
> about 50 bytes, and one column with less than 20 bytes data.
> I have set the scancaching to 100000 rows, and with 15 scanners
> working the same time, I got OutOfMemory exception, my PC has 6GB ram,
> through top I saw the program used more than 1GB ram sometime.
> In my calculation, each scanner will buffer 100000*50=5MB, and 15
> scanners will buffer 75MB ram, I know each scanner will hold some space,
> say, 10MB for each scanner. my question is how come it uses more than 1GB
> ram.
> anything I have missed?
> thanks.
>