You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Lukas Vlcek <lu...@gmail.com> on 2008/06/17 09:03:00 UTC

How long is Hadoop full unit test suit expected to run?

Hi,

How long is Hadoop full unit test suit expected to run?
How do you go about running Hadoop tests? I found that it can take hours for
[ant test] target to run which does not seem to be very efficient for
development.
Is there anything I can do to speed up tests (like running Hadoop in a real
cluster)?

Say I would like to fix a bug in Hadoop in ABC.java. Is it OK if I execute
just ABCTest.java (if available) for the development phase before the patch
is attached to JIRA ticket? I don't expect the answer to this question is
positive but I can not think of better workaround for now...

Regards,
Lukas

-- 
http://blog.lukas-vlcek.com/

Re: How long is Hadoop full unit test suit expected to run?

Posted by Enis Soztutar <en...@gmail.com>.
Just standard box like yours (ubuntu). I suspect something must be 
wrong. Could you determine which tests take long time.  On hudson, it 
seems that tests take 1:30 on average. 
http://hudson.zones.apache.org/hudson/view/Hadoop/


Lukas Vlcek wrote:
> Hi,
>
> I am aware of HowToContrib wiki page but in my case [ant test] takes more
> then one hour. I can not tell you how much time it takes because I always
> stopped it after 4-5 hours...
>
> I was running these test on notebook Dell, dual core 1GB of RAM, Windows XP.
> I haven't tried it now after switching to Ubuntu but I think this should not
> make a big difference. What kind of HW are you using for Hadoop testing? I
> would definitely appreciate if [ant test] runs under an hour.
>
> Regards,
> Lukas
>
> On Tue, Jun 17, 2008 at 10:26 AM, Enis Soztutar <en...@gmail.com>
> wrote:
>
>   
>> Lukas Vlcek wrote:
>>
>>     
>>> Hi,
>>>
>>> How long is Hadoop full unit test suit expected to run?
>>> How do you go about running Hadoop tests? I found that it can take hours
>>> for
>>> [ant test] target to run which does not seem to be very efficient for
>>> development.
>>> Is there anything I can do to speed up tests (like running Hadoop in a
>>> real
>>> cluster)?
>>>
>>>
>>>       
>> Hi,
>>
>> Yes, ant test can take up to an hour. On my machine it completes in less
>> than an hour. Previously work has been done to reduce the time that tests
>> take, but some of the tests take a long time by nature such as testing dfs
>> balance, etc.
>> There is an issue to implement ant test-core as a mapred job so that it can
>> be submitted to a cluster. That would help a lot.
>>
>>     
>>> Say I would like to fix a bug in Hadoop in ABC.java. Is it OK if I execute
>>> just ABCTest.java (if available) for the development phase before the
>>> patch
>>> is attached to JIRA ticket? I don't expect the answer to this question is
>>> positive but I can not think of better workaround for now...
>>>
>>>
>>>       
>> This very much depends on the patch. If you are "convinced" by running
>> TestABC, that the patch would be OK, then you can go ahead and submit patch
>> to hudson for QA testing. However, since the resources at Hudson is limited,
>> please do not use it for "regular" tests. As a side note, please run ant
>> test-patch to check your patch.
>>
>> http://wiki.apache.org/hadoop/HowToContribute
>> http://wiki.apache.org/hadoop/CodeReviewChecklist
>>
>> Enis
>>
>>     
>>> Regards,
>>> Lukas
>>>
>>>
>>>
>>>       
>
>
>

Re: How long is Hadoop full unit test suit expected to run?

Posted by Lukas Vlcek <lu...@gmail.com>.
Hi,

I am aware of HowToContrib wiki page but in my case [ant test] takes more
then one hour. I can not tell you how much time it takes because I always
stopped it after 4-5 hours...

I was running these test on notebook Dell, dual core 1GB of RAM, Windows XP.
I haven't tried it now after switching to Ubuntu but I think this should not
make a big difference. What kind of HW are you using for Hadoop testing? I
would definitely appreciate if [ant test] runs under an hour.

Regards,
Lukas

On Tue, Jun 17, 2008 at 10:26 AM, Enis Soztutar <en...@gmail.com>
wrote:

>
> Lukas Vlcek wrote:
>
>> Hi,
>>
>> How long is Hadoop full unit test suit expected to run?
>> How do you go about running Hadoop tests? I found that it can take hours
>> for
>> [ant test] target to run which does not seem to be very efficient for
>> development.
>> Is there anything I can do to speed up tests (like running Hadoop in a
>> real
>> cluster)?
>>
>>
> Hi,
>
> Yes, ant test can take up to an hour. On my machine it completes in less
> than an hour. Previously work has been done to reduce the time that tests
> take, but some of the tests take a long time by nature such as testing dfs
> balance, etc.
> There is an issue to implement ant test-core as a mapred job so that it can
> be submitted to a cluster. That would help a lot.
>
>> Say I would like to fix a bug in Hadoop in ABC.java. Is it OK if I execute
>> just ABCTest.java (if available) for the development phase before the
>> patch
>> is attached to JIRA ticket? I don't expect the answer to this question is
>> positive but I can not think of better workaround for now...
>>
>>
> This very much depends on the patch. If you are "convinced" by running
> TestABC, that the patch would be OK, then you can go ahead and submit patch
> to hudson for QA testing. However, since the resources at Hudson is limited,
> please do not use it for "regular" tests. As a side note, please run ant
> test-patch to check your patch.
>
> http://wiki.apache.org/hadoop/HowToContribute
> http://wiki.apache.org/hadoop/CodeReviewChecklist
>
> Enis
>
>> Regards,
>> Lukas
>>
>>
>>
>


-- 
http://blog.lukas-vlcek.com/

Re: How long is Hadoop full unit test suit expected to run?

Posted by Enis Soztutar <en...@gmail.com>.
Lukas Vlcek wrote:
> Hi,
>
> How long is Hadoop full unit test suit expected to run?
> How do you go about running Hadoop tests? I found that it can take hours for
> [ant test] target to run which does not seem to be very efficient for
> development.
> Is there anything I can do to speed up tests (like running Hadoop in a real
> cluster)?
>   
Hi,

Yes, ant test can take up to an hour. On my machine it completes in less 
than an hour. Previously work has been done to reduce the time that 
tests take, but some of the tests take a long time by nature such as 
testing dfs balance, etc.
There is an issue to implement ant test-core as a mapred job so that it 
can be submitted to a cluster. That would help a lot.
> Say I would like to fix a bug in Hadoop in ABC.java. Is it OK if I execute
> just ABCTest.java (if available) for the development phase before the patch
> is attached to JIRA ticket? I don't expect the answer to this question is
> positive but I can not think of better workaround for now...
>   
This very much depends on the patch. If you are "convinced" by running 
TestABC, that the patch would be OK, then you can go ahead and submit 
patch to hudson for QA testing. However, since the resources at Hudson 
is limited, please do not use it for "regular" tests. As a side note, 
please run ant test-patch to check your patch.

http://wiki.apache.org/hadoop/HowToContribute
http://wiki.apache.org/hadoop/CodeReviewChecklist

Enis
> Regards,
> Lukas
>
>