You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Venkatesh <vr...@aol.com> on 2010/10/12 04:47:48 UTC
Increase region server throughput
I would like to tune region server to increase throughput..On a 10 node cluster,
I'm getting 5 sec per put. (this is unbatched/unbuffered). Other than
region server handler count property is there anything else I can tune
to increase throughput? ( this operation i can't use buffered write without
code change)
thx
venkatesh
Re: Increase region server throughput
Posted by Venkatesh <vr...@aol.com>.
Thanks St.Ack.
I've both those settings autoflush & writebuffer size.
I'll try the new HTable(conf, ..)..(I just have new HTable(table) now)
Right now upto 85% under 10ms..I'm trying to bring the mean down
PS: I can tolerate some loss of data for getting better throughput.
-----Original Message-----
From: Sean Bigdatafun <se...@gmail.com>
To: user@hbase.apache.org
Sent: Thu, Oct 14, 2010 8:11 pm
Subject: Re: Increase region server throughput
Though this setup, setautoflush(false), increases the thoughput, the data
loss rate increases significantly -- there is no way for the client to know
what has been lost and what has gone through. That bothers me.
Sean
On Tue, Oct 12, 2010 at 11:32 AM, Stack <st...@duboce.net> wrote:
> Have you played with these settings HTable API?
>
>
> http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush(boolean)
>
> http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#setWriteBufferSize(long)
>
> There is something seriously wrong if you are seeing 5 seconds per put
> (unless your put is gigabytes in size?).
>
> Are you doing 'new HTable(tablename)' in your client or are you doing
> 'new HTable(conf, tablename)' in your client code? Do the latter if
> not -- share the configuration with HTable instances.
>
> St.Ack
>
> On Mon, Oct 11, 2010 at 10:47 PM, Venkatesh <vr...@aol.com> wrote:
> >
> >
> >
> > I would like to tune region server to increase throughput..On a 10 node
> cluster,
> > I'm getting 5 sec per put. (this is unbatched/unbuffered). Other than
> > region server handler count property is there anything else I can tune
> > to increase throughput? ( this operation i can't use buffered write
> without
> > code change)
> >
> > thx
> > venkatesh
> >
> >
> >
>
Re: Increase region server throughput
Posted by Sean Bigdatafun <se...@gmail.com>.
Though this setup, setautoflush(false), increases the thoughput, the data
loss rate increases significantly -- there is no way for the client to know
what has been lost and what has gone through. That bothers me.
Sean
On Tue, Oct 12, 2010 at 11:32 AM, Stack <st...@duboce.net> wrote:
> Have you played with these settings HTable API?
>
>
> http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush(boolean)
>
> http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#setWriteBufferSize(long)
>
> There is something seriously wrong if you are seeing 5 seconds per put
> (unless your put is gigabytes in size?).
>
> Are you doing 'new HTable(tablename)' in your client or are you doing
> 'new HTable(conf, tablename)' in your client code? Do the latter if
> not -- share the configuration with HTable instances.
>
> St.Ack
>
> On Mon, Oct 11, 2010 at 10:47 PM, Venkatesh <vr...@aol.com> wrote:
> >
> >
> >
> > I would like to tune region server to increase throughput..On a 10 node
> cluster,
> > I'm getting 5 sec per put. (this is unbatched/unbuffered). Other than
> > region server handler count property is there anything else I can tune
> > to increase throughput? ( this operation i can't use buffered write
> without
> > code change)
> >
> > thx
> > venkatesh
> >
> >
> >
>
Re: Increase region server throughput
Posted by Stack <st...@duboce.net>.
Have you played with these settings HTable API?
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush(boolean)
http://hbase.apache.org/docs/r0.20.6/api/org/apache/hadoop/hbase/client/HTable.html#setWriteBufferSize(long)
There is something seriously wrong if you are seeing 5 seconds per put
(unless your put is gigabytes in size?).
Are you doing 'new HTable(tablename)' in your client or are you doing
'new HTable(conf, tablename)' in your client code? Do the latter if
not -- share the configuration with HTable instances.
St.Ack
On Mon, Oct 11, 2010 at 10:47 PM, Venkatesh <vr...@aol.com> wrote:
>
>
>
> I would like to tune region server to increase throughput..On a 10 node cluster,
> I'm getting 5 sec per put. (this is unbatched/unbuffered). Other than
> region server handler count property is there anything else I can tune
> to increase throughput? ( this operation i can't use buffered write without
> code change)
>
> thx
> venkatesh
>
>
>