You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Garrett Wu <wu...@gmail.com> on 2011/08/16 02:12:38 UTC
mini-hbase configuration for tests
I have a bunch of integration tests that spend a lot of time creating and
deleting tables in a mini hbase cluster started using HBaseTestingUtility.
Disabling and deleting a table seems to take a second or two each time
-- I'm guessing it's some sort of heartbeat interval it's waiting for. Is
there a configuration variable I can set to make this go faster?
Garrett
Re: mini-hbase configuration for tests
Posted by Eric Charles <er...@gmail.com>.
Hi JD,
I tried without success to hack our test architecture to evaluate the
effect of those configs. I stopped to avoid diverging to a kind of
'hbase performance testing'.
Definitely good to note that hbase.client.pause and
hbase.master.event.waiting.time can be tuned for tests speed.
Thx.
On 29/08/11 11:08, Jean-Daniel Cryans wrote:
> (cleaning up my unread emails hehe)
>
>> We are using 0.90.x. Which exact timing param can we set? (I'm not sure
>> which one looking http://hbase.apache.org/book/config.files.html#hbase.site)
>>
>> We set configuration via htu.getConfiguration().set... and not via
>> hbase-default.xml. Is it OK to do like that?
>
> As long as the minicluster isn't started when you do those changes
> it's ok, else it's too late.
>
> You could set hbase.client.pause to something smaller than 1000, this
> way the client would be more aggressive.
>
> It seems a smaller value of hbase.master.event.waiting.time would also
> make table changes snappier, the HBase unit tests use a value of 50.
>
> J-D
--
Eric
http://about.echarles.net
Re: mini-hbase configuration for tests
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(cleaning up my unread emails hehe)
> We are using 0.90.x. Which exact timing param can we set? (I'm not sure
> which one looking http://hbase.apache.org/book/config.files.html#hbase.site)
>
> We set configuration via htu.getConfiguration().set... and not via
> hbase-default.xml. Is it OK to do like that?
As long as the minicluster isn't started when you do those changes
it's ok, else it's too late.
You could set hbase.client.pause to something smaller than 1000, this
way the client would be more aggressive.
It seems a smaller value of hbase.master.event.waiting.time would also
make table changes snappier, the HBase unit tests use a value of 50.
J-D
Re: mini-hbase configuration for tests
Posted by Eric Charles <er...@u-mangate.com>.
On 16/08/11 05:33, Stack wrote:
> On Mon, Aug 15, 2011 at 5:12 PM, Garrett Wu<wu...@gmail.com> wrote:
>> I have a bunch of integration tests that spend a lot of time creating and
>> deleting tables in a mini hbase cluster started using HBaseTestingUtility.
>> Disabling and deleting a table seems to take a second or two each time
>> -- I'm guessing it's some sort of heartbeat interval it's waiting for. Is
>> there a configuration variable I can set to make this go faster?
>>
We have the same concern here. It's even worse, cause we start/stop a
minihbasecluster on each test class. We are now going to a solution
where we let cluster run and delete table on each test.
>
> In 0.20, yeah, there was heartbeating. In 0.90.x it should be better.
> You can tweak the timings in src/test/hbase-default.xml params to see
> if you can get it to run faster.
We are using 0.90.x. Which exact timing param can we set? (I'm not sure
which one looking http://hbase.apache.org/book/config.files.html#hbase.site)
We set configuration via htu.getConfiguration().set... and not via
hbase-default.xml. Is it OK to do like that?
> What if you made a new table per
> test? Would things run faster then (w/o deleting the old).
>
> St.Ack
Thx.
--
Eric
http://about.echarles.net
Re: mini-hbase configuration for tests
Posted by Stack <st...@duboce.net>.
On Mon, Aug 15, 2011 at 5:12 PM, Garrett Wu <wu...@gmail.com> wrote:
> I have a bunch of integration tests that spend a lot of time creating and
> deleting tables in a mini hbase cluster started using HBaseTestingUtility.
> Disabling and deleting a table seems to take a second or two each time
> -- I'm guessing it's some sort of heartbeat interval it's waiting for. Is
> there a configuration variable I can set to make this go faster?
>
In 0.20, yeah, there was heartbeating. In 0.90.x it should be better.
You can tweak the timings in src/test/hbase-default.xml params to see
if you can get it to run faster. What if you made a new table per
test? Would things run faster then (w/o deleting the old).
St.Ack
Re: mini-hbase configuration for tests
Posted by Garrett Wu <wu...@gmail.com>.
Thanks for the suggestions. I tweaked jobclient.completion.poll.interval
and hbase.regionserver.msginterval, but that didn't seem to do much. I'll
just not delete the tables, which is fine since they're all in a mini hbase
anyway.
On Mon, Aug 15, 2011 at 5:37 PM, Bill Graham <bi...@gmail.com> wrote:
> Hey Garrett,
>
> I'm not sure about a config setting but in Pig we changed TestHBaseStorage
> to delete all rows of tables instead of truncate them. This was faster
> since
> the tables are typically small in tests. See Dymitriy's note in the
> deleteAllRows method here:
>
>
> http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestHBaseStorage.java
>
> Bill
>
> On Mon, Aug 15, 2011 at 5:12 PM, Garrett Wu <wu...@gmail.com> wrote:
>
> > I have a bunch of integration tests that spend a lot of time creating and
> > deleting tables in a mini hbase cluster started using
> HBaseTestingUtility.
> > Disabling and deleting a table seems to take a second or two each time
> > -- I'm guessing it's some sort of heartbeat interval it's waiting for.
> Is
> > there a configuration variable I can set to make this go faster?
> >
> > Garrett
> >
>
Re: mini-hbase configuration for tests
Posted by Bill Graham <bi...@gmail.com>.
Hey Garrett,
I'm not sure about a config setting but in Pig we changed TestHBaseStorage
to delete all rows of tables instead of truncate them. This was faster since
the tables are typically small in tests. See Dymitriy's note in the
deleteAllRows method here:
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestHBaseStorage.java
Bill
On Mon, Aug 15, 2011 at 5:12 PM, Garrett Wu <wu...@gmail.com> wrote:
> I have a bunch of integration tests that spend a lot of time creating and
> deleting tables in a mini hbase cluster started using HBaseTestingUtility.
> Disabling and deleting a table seems to take a second or two each time
> -- I'm guessing it's some sort of heartbeat interval it's waiting for. Is
> there a configuration variable I can set to make this go faster?
>
> Garrett
>