You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stefan Groschupf <sg...@101tec.com> on 2008/03/07 01:25:40 UTC

test suite

Hi there,
I checked out latest svn and did run the test.

The test suite failed at TestTable, are there known problems today?
Also the test suite took 24 min to run on my little poor box - is  
there any way to speed this up?
Thanks.
Stefan


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com



Re: test suite

Posted by Stefan Groschupf <sg...@101tec.com>.
> There is an issue using local filesystem on windows that we need to  
> figure.  Will try leaning on our windows user to see if we can get  
> more info around the prob.
If it is a problem with unix commands on linux there is a simple  
solution for that.
Just install cygwin on a windows box and add the cygwin bin folder  
where all the *.exe files are to your classpath.
We did run also in issues using the dfs clients on windows but this  
solution fixed that problem.

HTH
Stefan


Re: test suite

Posted by stack <st...@duboce.net>.
You're not missing anything Stefan.  I think there is general agreement 
that 30 mins is too long to wait on tests (Bryan Duxbury, to his credit, 
keeps trying to bring up this fact.  He filed HBASE-498 last night).

There is an issue using local filesystem on windows that we need to 
figure.  Will try leaning on our windows user to see if we can get more 
info around the prob.  Will post to HBASE-498.

St.Ack



Stefan Groschupf wrote:
> I'm sorry if I ask stupid questions, I'm very new to HBase.
> Waiting 30 min to see if a change breaks something or not is painful.
> I understand why you want to use a hadoop minicluster for integration 
> tests but for normal unit tests.
> I would expect that  there is an interface for each hbase component 
> that the server and the clients implements. Similar to the hadoop 
> jobrunner and localJobrunner. Since both components implement this 
> interface it is transparent for the test if we use a distributed mini 
> cluster or a local non tcp ip implementatipon. Also we could have 
> mocks implementing this interface.
> So which puzzle peace I miss that there is the requirement to start up 
> this minicluster every time?
>
>
>
> On Mar 6, 2008, at 8:24 PM, stack wrote:
>
>> Stefan Groschupf wrote:
>>> Is there an hudson that build hbase?
>>> Hudson should run the integration test suite with mini cluster and 
>>> dfs where ant test for the user should only run local fs and local 
>>> map reduce jobrunner.
>> Yeah.  We have our own little zone on hudson: 
>> http://hudson.zones.apache.org/hudson/job/HBase-Patch/.  Needs some 
>> work.   Currently just runs a build on checkin.   Would like to add 
>> the patch testing that happens in hadoop core.
>>
>> Starting dfs cluster from ant script is a little awkward.
>> MiniDFSCluster is nice because it chooses unoccupied ports; important 
>> on a populated build box.  Would need to then pass the chosen dfs 
>> port then to tests and mini dfs would have to be in current process 
>> if mindfs so can shut it down when done.  I tried running in ant 
>> context as distinct task and as wrapper around junit task but ran 
>> into class loading issues.
>>
>> Could call out to a script that starts minidfs but need to do a 
>> little dev so can send it a shutdown when done.  Was not enthusiastic 
>> since replicating the 'real' dfs start/stop scripts.  Alternative 
>> would be to start a 'real' dfs using fixed ports.  This would involve 
>> ant exec'ing.  Would need to deal with *nix and windows paths to make 
>> sure it worked on both.  If I'm not mistaken, even 'local' mode 
>> requires ssh'ing to start datanodes even if on localhost.
>>
>> Maybe the best thing would be to just punt on trying to have the 
>> start/stop of the dfs be run from ant and just do it as part of the 
>> build scripts we run up on the hudson build box?
>>
>> St.Ack
>>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
>
>


RE: test suite

Posted by Jim Kellerman <ji...@powerset.com>.
Sigh. Yes. That test is very problematic.

---
Jim Kellerman, Senior Engineer; Powerset


> -----Original Message-----
> From: Stefan Groschupf [mailto:sg@101tec.com]
> Sent: Friday, March 07, 2008 1:55 PM
> To: hbase-dev@hadoop.apache.org
> Subject: Re: test suite
>
>      [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>      [junit] Test org.apache.hadoop.hbase.mapred.TestTableIndex FAILED
> (timeout)
>
>
> Testsuite: org.apache.hadoop.hbase.mapred.TestTableIndex
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>
> Testcase: unknown took 0.05 sec
>         Caused an ERROR
> Timeout occurred
> junit.framework.AssertionFailedError: Timeout occurred
>
>
>
> Thats all I have, sounds this familiar?
>
> Thanks.
> Stefan
>
>
>
> On Mar 7, 2008, at 1:22 PM, Jim Kellerman wrote:
>
> > If the test suite doesn't pass, that's a problem. What tests failed?
> >
> > And you can run single tests if you want.
> >
> > Wrt Windows, I use cygwin, but the problem is not unix
> commands, but
> > code in HBase that deletes files and directories. FileSystem.delete
> > behaves differently on unix vs Windows. What we should be doing for
> > directories is use FileUtil.deleteFully
> >
> > ---
> > Jim Kellerman, Senior Engineer; Powerset
> >
> >
> >> -----Original Message-----
> >> From: Stefan Groschupf [mailto:sg@101tec.com]
> >> Sent: Friday, March 07, 2008 1:15 PM
> >> To: hbase-dev@hadoop.apache.org
> >> Subject: Re: test suite
> >>
> >>> Since HBase is targeted to run on hdfs and not the local
> >> file system,
> >>> we want to test it in its "native" environment and not a
> >> mock-up which
> >>> might not have all the ideosynchracies of the "real thing".
> >>
> >> Jim, I agree there should be integration tests that run
> against the
> >> hdfs  - that can run on hudon.  However there should be unit test
> >> that test code units of hbase not of hadoop. Those should
> not use the
> >> dfs since the idea of unit tests is not test underlaying
> third party
> >> systems.
> >>
> >> Anyhow just a user feedback the test suite as it is today is not
> >> useful for me personal, since first, it never passed on my box and
> >> second it takes 30 min to run - so I never run it
> >> - what is not the idea of having a test suite and never
> run it during
> >> development.
> >>
> >> Just some feedback...
> >>
> >> Stefan
> >>
> >>
> >> No virus found in this incoming message.
> >> Checked by AVG.
> >> Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release
> >> Date: 3/7/2008 2:01 PM
> >>
> >>
> >
> > No virus found in this outgoing message.
> > Checked by AVG.
> > Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release Date:
> > 3/7/2008 2:01 PM
> >
> >
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
>
>
>
> No virus found in this incoming message.
> Checked by AVG.
> Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release
> Date: 3/7/2008 2:01 PM
>
>

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release Date: 3/7/2008 2:01 PM


Re: test suite

Posted by Stefan Groschupf <sg...@101tec.com>.
     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
     [junit] Test org.apache.hadoop.hbase.mapred.TestTableIndex FAILED  
(timeout)


Testsuite: org.apache.hadoop.hbase.mapred.TestTableIndex
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec

Testcase: unknown took 0.05 sec
	Caused an ERROR
Timeout occurred
junit.framework.AssertionFailedError: Timeout occurred



Thats all I have, sounds this familiar?

Thanks.
Stefan



On Mar 7, 2008, at 1:22 PM, Jim Kellerman wrote:

> If the test suite doesn't pass, that's a problem. What tests failed?
>
> And you can run single tests if you want.
>
> Wrt Windows, I use cygwin, but the problem is not unix commands, but
> code in HBase that deletes files and directories. FileSystem.delete
> behaves differently on unix vs Windows. What we should be doing for
> directories is use FileUtil.deleteFully
>
> ---
> Jim Kellerman, Senior Engineer; Powerset
>
>
>> -----Original Message-----
>> From: Stefan Groschupf [mailto:sg@101tec.com]
>> Sent: Friday, March 07, 2008 1:15 PM
>> To: hbase-dev@hadoop.apache.org
>> Subject: Re: test suite
>>
>>> Since HBase is targeted to run on hdfs and not the local
>> file system,
>>> we want to test it in its "native" environment and not a
>> mock-up which
>>> might not have all the ideosynchracies of the "real thing".
>>
>> Jim, I agree there should be integration tests that run
>> against the hdfs  - that can run on hudon.  However there
>> should be unit test that test code units of hbase not of
>> hadoop. Those should not use the dfs since the idea of unit
>> tests is not test underlaying third party systems.
>>
>> Anyhow just a user feedback the test suite as it is today is
>> not useful for me personal, since first, it never passed on
>> my box and second it takes 30 min to run - so I never run it
>> - what is not the idea of having a test suite and never run
>> it during development.
>>
>> Just some feedback...
>>
>> Stefan
>>
>>
>> No virus found in this incoming message.
>> Checked by AVG.
>> Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release
>> Date: 3/7/2008 2:01 PM
>>
>>
>
> No virus found in this outgoing message.
> Checked by AVG.
> Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release Date:  
> 3/7/2008 2:01 PM
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com



RE: test suite

Posted by Jim Kellerman <ji...@powerset.com>.
If the test suite doesn't pass, that's a problem. What tests failed?

And you can run single tests if you want.

Wrt Windows, I use cygwin, but the problem is not unix commands, but
code in HBase that deletes files and directories. FileSystem.delete
behaves differently on unix vs Windows. What we should be doing for
directories is use FileUtil.deleteFully

---
Jim Kellerman, Senior Engineer; Powerset


> -----Original Message-----
> From: Stefan Groschupf [mailto:sg@101tec.com]
> Sent: Friday, March 07, 2008 1:15 PM
> To: hbase-dev@hadoop.apache.org
> Subject: Re: test suite
>
> > Since HBase is targeted to run on hdfs and not the local
> file system,
> > we want to test it in its "native" environment and not a
> mock-up which
> > might not have all the ideosynchracies of the "real thing".
>
> Jim, I agree there should be integration tests that run
> against the hdfs  - that can run on hudon.  However there
> should be unit test that test code units of hbase not of
> hadoop. Those should not use the dfs since the idea of unit
> tests is not test underlaying third party systems.
>
> Anyhow just a user feedback the test suite as it is today is
> not useful for me personal, since first, it never passed on
> my box and second it takes 30 min to run - so I never run it
> - what is not the idea of having a test suite and never run
> it during development.
>
> Just some feedback...
>
> Stefan
>
>
> No virus found in this incoming message.
> Checked by AVG.
> Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release
> Date: 3/7/2008 2:01 PM
>
>

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release Date: 3/7/2008 2:01 PM


Re: test suite

Posted by Stefan Groschupf <sg...@101tec.com>.
> Since HBase is targeted to run on hdfs and not the local file system,
> we want to test it in its "native" environment and not a mock-up which
> might not have all the ideosynchracies of the "real thing".

Jim, I agree there should be integration tests that run against the  
hdfs  - that can run on hudon.  However there should be unit test that  
test code units of hbase not of hadoop. Those should not use the dfs  
since the idea of unit tests is not test underlaying third party  
systems.

Anyhow just a user feedback the test suite as it is today is not  
useful for me personal, since first, it never passed on my box and  
second it takes 30 min to run - so I never run it - what is not the  
idea of having a test suite and never run it during development.

Just some feedback...

Stefan


RE: test suite

Posted by Jim Kellerman <ji...@powerset.com>.
Since HBase is targeted to run on hdfs and not the local file system,
we want to test it in its "native" environment and not a mock-up which
might not have all the ideosynchracies of the "real thing".

---
Jim Kellerman, Senior Engineer; Powerset


> -----Original Message-----
> From: Stefan Groschupf [mailto:sg@101tec.com]
> Sent: Thursday, March 06, 2008 10:37 PM
> To: hbase-dev@hadoop.apache.org
> Subject: Re: test suite
>
> I'm sorry if I ask stupid questions, I'm very new to HBase.
> Waiting 30 min to see if a change breaks something or not is painful.
> I understand why you want to use a hadoop minicluster for
> integration tests but for normal unit tests.
> I would expect that  there is an interface for each hbase
> component that the server and the clients implements. Similar
> to the hadoop jobrunner and localJobrunner. Since both
> components implement this interface it is transparent for the
> test if we use a distributed mini cluster or a local non tcp
> ip implementatipon. Also we could have mocks implementing
> this interface.
> So which puzzle peace I miss that there is the requirement to
> start up this minicluster every time?
>
>
>
> On Mar 6, 2008, at 8:24 PM, stack wrote:
>
> > Stefan Groschupf wrote:
> >> Is there an hudson that build hbase?
> >> Hudson should run the integration test suite with mini cluster and
> >> dfs where ant test for the user should only run local fs and local
> >> map reduce jobrunner.
> > Yeah.  We have our own little zone on hudson:
> http://hudson.zones.apache.org/hudson/job/HBase-Patch/
> > .  Needs some work.   Currently just runs a build on checkin.
> > Would like to add the patch testing that happens in hadoop core.
> >
> > Starting dfs cluster from ant script is a little awkward.
> > MiniDFSCluster is nice because it chooses unoccupied ports;
> important
> > on a populated build box.  Would need to then pass the
> chosen dfs port
> > then to tests and mini dfs would have to be in current process if
> > mindfs so can shut it down when done.  I tried running in
> ant context
> > as distinct task and as wrapper around junit task but ran
> into class
> > loading issues.
> >
> > Could call out to a script that starts minidfs but need to
> do a little
> > dev so can send it a shutdown when done.  Was not
> enthusiastic since
> > replicating the 'real' dfs start/stop scripts.
> > Alternative would be to start a 'real' dfs using fixed ports.  This
> > would involve ant exec'ing.  Would need to deal with *nix
> and windows
> > paths to make sure it worked on both.  If I'm not mistaken, even
> > 'local' mode requires ssh'ing to start datanodes even if on
> localhost.
> >
> > Maybe the best thing would be to just punt on trying to have the
> > start/stop of the dfs be run from ant and just do it as part of the
> > build scripts we run up on the hudson build box?
> >
> > St.Ack
> >
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
>
>
>
> No virus found in this incoming message.
> Checked by AVG.
> Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release
> Date: 3/7/2008 2:01 PM
>
>

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.518 / Virus Database: 269.21.6/1318 - Release Date: 3/7/2008 2:01 PM


Re: test suite

Posted by Stefan Groschupf <sg...@101tec.com>.
I'm sorry if I ask stupid questions, I'm very new to HBase.
Waiting 30 min to see if a change breaks something or not is painful.
I understand why you want to use a hadoop minicluster for integration  
tests but for normal unit tests.
I would expect that  there is an interface for each hbase component  
that the server and the clients implements. Similar to the hadoop  
jobrunner and localJobrunner. Since both components implement this  
interface it is transparent for the test if we use a distributed mini  
cluster or a local non tcp ip implementatipon. Also we could have  
mocks implementing this interface.
So which puzzle peace I miss that there is the requirement to start up  
this minicluster every time?



On Mar 6, 2008, at 8:24 PM, stack wrote:

> Stefan Groschupf wrote:
>> Is there an hudson that build hbase?
>> Hudson should run the integration test suite with mini cluster and  
>> dfs where ant test for the user should only run local fs and local  
>> map reduce jobrunner.
> Yeah.  We have our own little zone on hudson: http://hudson.zones.apache.org/hudson/job/HBase-Patch/ 
> .  Needs some work.   Currently just runs a build on checkin.    
> Would like to add the patch testing that happens in hadoop core.
>
> Starting dfs cluster from ant script is a little awkward.
> MiniDFSCluster is nice because it chooses unoccupied ports;  
> important on a populated build box.  Would need to then pass the  
> chosen dfs port then to tests and mini dfs would have to be in  
> current process if mindfs so can shut it down when done.  I tried  
> running in ant context as distinct task and as wrapper around junit  
> task but ran into class loading issues.
>
> Could call out to a script that starts minidfs but need to do a  
> little dev so can send it a shutdown when done.  Was not  
> enthusiastic since replicating the 'real' dfs start/stop scripts.   
> Alternative would be to start a 'real' dfs using fixed ports.  This  
> would involve ant exec'ing.  Would need to deal with *nix and  
> windows paths to make sure it worked on both.  If I'm not mistaken,  
> even 'local' mode requires ssh'ing to start datanodes even if on  
> localhost.
>
> Maybe the best thing would be to just punt on trying to have the  
> start/stop of the dfs be run from ant and just do it as part of the  
> build scripts we run up on the hudson build box?
>
> St.Ack
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com



Re: test suite

Posted by stack <st...@duboce.net>.
Stefan Groschupf wrote:
> Is there an hudson that build hbase?
> Hudson should run the integration test suite with mini cluster and dfs 
> where ant test for the user should only run local fs and local map 
> reduce jobrunner.
Yeah.  We have our own little zone on hudson: 
http://hudson.zones.apache.org/hudson/job/HBase-Patch/.  Needs some 
work.   Currently just runs a build on checkin.   Would like to add the 
patch testing that happens in hadoop core.

Starting dfs cluster from ant script is a little awkward. 

MiniDFSCluster is nice because it chooses unoccupied ports; important on 
a populated build box.  Would need to then pass the chosen dfs port then 
to tests and mini dfs would have to be in current process if mindfs so 
can shut it down when done.  I tried running in ant context as distinct 
task and as wrapper around junit task but ran into class loading issues.

Could call out to a script that starts minidfs but need to do a little 
dev so can send it a shutdown when done.  Was not enthusiastic since 
replicating the 'real' dfs start/stop scripts.  Alternative would be to 
start a 'real' dfs using fixed ports.  This would involve ant exec'ing.  
Would need to deal with *nix and windows paths to make sure it worked on 
both.  If I'm not mistaken, even 'local' mode requires ssh'ing to start 
datanodes even if on localhost.

Maybe the best thing would be to just punt on trying to have the 
start/stop of the dfs be run from ant and just do it as part of the 
build scripts we run up on the hudson build box?

St.Ack

Re: test suite

Posted by Stefan Groschupf <sg...@101tec.com>.
Is there an hudson that build hbase?
Hudson should run the integration test suite with mini cluster and dfs  
where ant test for the user should only run local fs and local map  
reduce jobrunner.

On Mar 6, 2008, at 5:23 PM, Bryan Duxbury wrote:

> I also vote for the local file system. However, I understand that  
> there are some issues with local filesystem implementation on  
> Windows, and there are some bugs that will only manifest when run on  
> hdfs, mini or otherwise.
>
> I think the best middle ground is expecting an externally configured  
> dfs, local or otherwise, rather than spinning one up for every test.  
> This would require an extra step on the part of developers running  
> the test suite. Do you think that would make it notably more  
> difficult for new developers to get the test suite going?
>
> Whatever we end up doing regarding filesystem spin up, my testing  
> shows that save us maybe a minute over the entire suite. It's  
> certainly still a worthwhile task to pursue, but won't be a magic  
> bullet.
>
> -Bryan
>
> On Mar 6, 2008, at 5:11 PM, Stefan Groschupf wrote:
>
>> jmock2 is pretty useful, I agree.
>> Also we in pig I submitted a patch that uses the hadoop  
>> localjobrunner (jobtracker=local) - significant faster than the  
>> minicluster.
>> Also in case hbase uses hadoop configuration why not just using the  
>> local file system?
>>
>> On Mar 6, 2008, at 4:45 PM, Bryan Duxbury wrote:
>>
>>> I would add that we have an issue open for speeding up the test  
>>> suite, HBASE-410. Probably the most promising idea for speeding up  
>>> the test suite is converting some of the tests that don't need to  
>>> be integration tests into something more like true unit tests, or  
>>> specs for that matter. Mock objects used to test hyper-focused  
>>> bits of the system would be absolutely ideal, and has the  
>>> advantage of being quite fast.
>>>
>>> If you wanted to make some early contributions, anything in this  
>>> area would be a FANTASTIC place to put them.
>>>
>>> -Bryan
>>>
>>> On Mar 6, 2008, at 4:28 PM, stack wrote:
>>>
>>>> Stefan Groschupf wrote:
>>>>> Hi there,
>>>>> I checked out latest svn and did run the test.
>>>>>
>>>>> The test suite failed at TestTable, are there known problems  
>>>>> today?
>>>>
>>>> Supposed to work.  Usually does.  New hadoop was checked in last  
>>>> night.  That might have been responsible.  Breaks everytime?  If  
>>>> so, open JIRA.
>>>>
>>>>> Also the test suite took 24 min to run on my little poor box -  
>>>>> is there any way to speed this up?
>>>>
>>>> Yeah.  Its a real prob.  Test suite starts up little mini dfs's  
>>>> and mini hbase clusters and sometimes mini mr clusters.  I've  
>>>> spent time on figuring how to do weird things like start the dfs  
>>>> cluster inside of an ant task but ran into classloading issues  
>>>> (joy).  Needs to be fixed.
>>>>
>>>> (Our 'unit tests' are more like 'integration tests' -- one  
>>>> approache would be to call them so and move them out of the unit  
>>>> test suite).
>>>>
>>>> St.Ack
>>>>
>>>>> Thanks.
>>>>> Stefan
>>>>>
>>>>>
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> 101tec Inc.
>>>>> Menlo Park, California, USA
>>>>> http://www.101tec.com
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 101tec Inc.
>> Menlo Park, California, USA
>> http://www.101tec.com
>>
>>
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com



Re: test suite

Posted by Bryan Duxbury <br...@rapleaf.com>.
I also vote for the local file system. However, I understand that  
there are some issues with local filesystem implementation on  
Windows, and there are some bugs that will only manifest when run on  
hdfs, mini or otherwise.

I think the best middle ground is expecting an externally configured  
dfs, local or otherwise, rather than spinning one up for every test.  
This would require an extra step on the part of developers running  
the test suite. Do you think that would make it notably more  
difficult for new developers to get the test suite going?

Whatever we end up doing regarding filesystem spin up, my testing  
shows that save us maybe a minute over the entire suite. It's  
certainly still a worthwhile task to pursue, but won't be a magic  
bullet.

-Bryan

On Mar 6, 2008, at 5:11 PM, Stefan Groschupf wrote:

> jmock2 is pretty useful, I agree.
> Also we in pig I submitted a patch that uses the hadoop  
> localjobrunner (jobtracker=local) - significant faster than the  
> minicluster.
> Also in case hbase uses hadoop configuration why not just using the  
> local file system?
>
> On Mar 6, 2008, at 4:45 PM, Bryan Duxbury wrote:
>
>> I would add that we have an issue open for speeding up the test  
>> suite, HBASE-410. Probably the most promising idea for speeding up  
>> the test suite is converting some of the tests that don't need to  
>> be integration tests into something more like true unit tests, or  
>> specs for that matter. Mock objects used to test hyper-focused  
>> bits of the system would be absolutely ideal, and has the  
>> advantage of being quite fast.
>>
>> If you wanted to make some early contributions, anything in this  
>> area would be a FANTASTIC place to put them.
>>
>> -Bryan
>>
>> On Mar 6, 2008, at 4:28 PM, stack wrote:
>>
>>> Stefan Groschupf wrote:
>>>> Hi there,
>>>> I checked out latest svn and did run the test.
>>>>
>>>> The test suite failed at TestTable, are there known problems today?
>>>
>>> Supposed to work.  Usually does.  New hadoop was checked in last  
>>> night.  That might have been responsible.  Breaks everytime?  If  
>>> so, open JIRA.
>>>
>>>> Also the test suite took 24 min to run on my little poor box -  
>>>> is there any way to speed this up?
>>>
>>> Yeah.  Its a real prob.  Test suite starts up little mini dfs's  
>>> and mini hbase clusters and sometimes mini mr clusters.  I've  
>>> spent time on figuring how to do weird things like start the dfs  
>>> cluster inside of an ant task but ran into classloading issues  
>>> (joy).  Needs to be fixed.
>>>
>>> (Our 'unit tests' are more like 'integration tests' -- one  
>>> approache would be to call them so and move them out of the unit  
>>> test suite).
>>>
>>> St.Ack
>>>
>>>> Thanks.
>>>> Stefan
>>>>
>>>>
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> 101tec Inc.
>>>> Menlo Park, California, USA
>>>> http://www.101tec.com
>>>>
>>>>
>>>
>>
>>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
>
>


Re: test suite

Posted by Stefan Groschupf <sg...@101tec.com>.
jmock2 is pretty useful, I agree.
Also we in pig I submitted a patch that uses the hadoop localjobrunner  
(jobtracker=local) - significant faster than the minicluster.
Also in case hbase uses hadoop configuration why not just using the  
local file system?

On Mar 6, 2008, at 4:45 PM, Bryan Duxbury wrote:

> I would add that we have an issue open for speeding up the test  
> suite, HBASE-410. Probably the most promising idea for speeding up  
> the test suite is converting some of the tests that don't need to be  
> integration tests into something more like true unit tests, or specs  
> for that matter. Mock objects used to test hyper-focused bits of the  
> system would be absolutely ideal, and has the advantage of being  
> quite fast.
>
> If you wanted to make some early contributions, anything in this  
> area would be a FANTASTIC place to put them.
>
> -Bryan
>
> On Mar 6, 2008, at 4:28 PM, stack wrote:
>
>> Stefan Groschupf wrote:
>>> Hi there,
>>> I checked out latest svn and did run the test.
>>>
>>> The test suite failed at TestTable, are there known problems today?
>>
>> Supposed to work.  Usually does.  New hadoop was checked in last  
>> night.  That might have been responsible.  Breaks everytime?  If  
>> so, open JIRA.
>>
>>> Also the test suite took 24 min to run on my little poor box - is  
>>> there any way to speed this up?
>>
>> Yeah.  Its a real prob.  Test suite starts up little mini dfs's and  
>> mini hbase clusters and sometimes mini mr clusters.  I've spent  
>> time on figuring how to do weird things like start the dfs cluster  
>> inside of an ant task but ran into classloading issues (joy).   
>> Needs to be fixed.
>>
>> (Our 'unit tests' are more like 'integration tests' -- one  
>> approache would be to call them so and move them out of the unit  
>> test suite).
>>
>> St.Ack
>>
>>> Thanks.
>>> Stefan
>>>
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> 101tec Inc.
>>> Menlo Park, California, USA
>>> http://www.101tec.com
>>>
>>>
>>
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com



Re: test suite

Posted by Bryan Duxbury <br...@rapleaf.com>.
I would add that we have an issue open for speeding up the test  
suite, HBASE-410. Probably the most promising idea for speeding up  
the test suite is converting some of the tests that don't need to be  
integration tests into something more like true unit tests, or specs  
for that matter. Mock objects used to test hyper-focused bits of the  
system would be absolutely ideal, and has the advantage of being  
quite fast.

If you wanted to make some early contributions, anything in this area  
would be a FANTASTIC place to put them.

-Bryan

On Mar 6, 2008, at 4:28 PM, stack wrote:

> Stefan Groschupf wrote:
>> Hi there,
>> I checked out latest svn and did run the test.
>>
>> The test suite failed at TestTable, are there known problems today?
>
> Supposed to work.  Usually does.  New hadoop was checked in last  
> night.  That might have been responsible.  Breaks everytime?  If  
> so, open JIRA.
>
>> Also the test suite took 24 min to run on my little poor box - is  
>> there any way to speed this up?
>
> Yeah.  Its a real prob.  Test suite starts up little mini dfs's and  
> mini hbase clusters and sometimes mini mr clusters.  I've spent  
> time on figuring how to do weird things like start the dfs cluster  
> inside of an ant task but ran into classloading issues (joy).   
> Needs to be fixed.
>
> (Our 'unit tests' are more like 'integration tests' -- one  
> approache would be to call them so and move them out of the unit  
> test suite).
>
> St.Ack
>
>> Thanks.
>> Stefan
>>
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 101tec Inc.
>> Menlo Park, California, USA
>> http://www.101tec.com
>>
>>
>


Re: test suite

Posted by stack <st...@duboce.net>.
Stefan Groschupf wrote:
> Hi there,
> I checked out latest svn and did run the test.
>
> The test suite failed at TestTable, are there known problems today?

Supposed to work.  Usually does.  New hadoop was checked in last night.  
That might have been responsible.  Breaks everytime?  If so, open JIRA.

> Also the test suite took 24 min to run on my little poor box - is 
> there any way to speed this up?

Yeah.  Its a real prob.  Test suite starts up little mini dfs's and mini 
hbase clusters and sometimes mini mr clusters.  I've spent time on 
figuring how to do weird things like start the dfs cluster inside of an 
ant task but ran into classloading issues (joy).  Needs to be fixed.

(Our 'unit tests' are more like 'integration tests' -- one approache 
would be to call them so and move them out of the unit test suite).

St.Ack

> Thanks.
> Stefan
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
>
>