You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Alan Gates <ga...@hortonworks.com> on 2011/09/02 02:50:59 UTC
Moving to new e2e harness for end-to-end testing
I have gotten the end-to-end test harness to the point where it runs basically all the existing tests and where it can be run from ant. It can be run on either an existing cluster or in Amazon's EC2. There are instructions on how to run it in both settings at https://cwiki.apache.org/confluence/display/PIG/HowToTest. Please try it out to see if it works in your environment.
As proposed in https://cwiki.apache.org/confluence/display/PIG/PigTestProposal, I would like to start migrating the junit tests that are really end-to-end tests (ie the ones that use MiniCluster) to the e2e harness. I'll start with the very long running ones and work down to the shorter ones.
Also, as we contribute new patches these should include end-to-end tests that will run in the new harness rather than in junit. True unit tests should still, of course, be done in junit.
Alan.
RE: Moving to new e2e harness for end-to-end testing
Posted by Santhosh Srinivasan <sm...@yahoo-inc.com>.
I have a suggestion. Since Pig is also available on Amazon's EMR, can we get some kind folks at Amazon to donate computing resources. It's a win-win - Pig is validated against their infrastructure and the community benefits too.
Any thoughts from the Amazon folks on the mailing list?
Thanks,
Santhosh
-----Original Message-----
From: Alan Gates [mailto:gates@hortonworks.com]
Sent: Thursday, September 01, 2011 10:29 PM
To: dev@pig.apache.org
Subject: Re: Moving to new e2e harness for end-to-end testing
One of my goals is to find a way to run public nightly builds for Pig. Since it needs a cluster it's not clear to me Apache's infrastructure is the best choice. I'm also investigating OSU's supercell (http://supercell.osuosl.org/) facilities. Other recommendations are welcome. I'd also like to get it set up so that there is a patch build, where contributors can submit a patch and a list of tests they think should be run, and have it run for them.
For now I am running the tests nightly on a local Jenkins and will report any errors when I see them. And there's EC2 for running your own tests if you don't have a cluster. But this isn't a viable long term solution.
Alan.
On Sep 1, 2011, at 6:49 PM, Dmitriy Ryaboy wrote:
> Alan,
> Great work.
> Any plans for hooking this up to the apache Jenkins instance?
>
> D
>
> On Thu, Sep 1, 2011 at 5:50 PM, Alan Gates <ga...@hortonworks.com> wrote:
>
>> I have gotten the end-to-end test harness to the point where it runs
>> basically all the existing tests and where it can be run from ant.
>> It can be run on either an existing cluster or in Amazon's EC2.
>> There are instructions on how to run it in both settings at
>> https://cwiki.apache.org/confluence/display/PIG/HowToTest. Please
>> try it out to see if it works in your environment.
>>
>> As proposed in
>> https://cwiki.apache.org/confluence/display/PIG/PigTestProposal, I
>> would like to start migrating the junit tests that are really
>> end-to-end tests (ie the ones that use MiniCluster) to the e2e
>> harness. I'll start with the very long running ones and work down to the shorter ones.
>>
>> Also, as we contribute new patches these should include end-to-end
>> tests that will run in the new harness rather than in junit. True
>> unit tests should still, of course, be done in junit.
>>
>> Alan.
Re: Moving to new e2e harness for end-to-end testing
Posted by Alan Gates <ga...@hortonworks.com>.
One of my goals is to find a way to run public nightly builds for Pig. Since it needs a cluster it's not clear to me Apache's infrastructure is the best choice. I'm also investigating OSU's supercell (http://supercell.osuosl.org/) facilities. Other recommendations are welcome. I'd also like to get it set up so that there is a patch build, where contributors can submit a patch and a list of tests they think should be run, and have it run for them.
For now I am running the tests nightly on a local Jenkins and will report any errors when I see them. And there's EC2 for running your own tests if you don't have a cluster. But this isn't a viable long term solution.
Alan.
On Sep 1, 2011, at 6:49 PM, Dmitriy Ryaboy wrote:
> Alan,
> Great work.
> Any plans for hooking this up to the apache Jenkins instance?
>
> D
>
> On Thu, Sep 1, 2011 at 5:50 PM, Alan Gates <ga...@hortonworks.com> wrote:
>
>> I have gotten the end-to-end test harness to the point where it runs
>> basically all the existing tests and where it can be run from ant. It can
>> be run on either an existing cluster or in Amazon's EC2. There are
>> instructions on how to run it in both settings at
>> https://cwiki.apache.org/confluence/display/PIG/HowToTest. Please try it
>> out to see if it works in your environment.
>>
>> As proposed in
>> https://cwiki.apache.org/confluence/display/PIG/PigTestProposal, I would
>> like to start migrating the junit tests that are really end-to-end tests (ie
>> the ones that use MiniCluster) to the e2e harness. I'll start with the very
>> long running ones and work down to the shorter ones.
>>
>> Also, as we contribute new patches these should include end-to-end tests
>> that will run in the new harness rather than in junit. True unit tests
>> should still, of course, be done in junit.
>>
>> Alan.
Re: Moving to new e2e harness for end-to-end testing
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Alan,
Great work.
Any plans for hooking this up to the apache Jenkins instance?
D
On Thu, Sep 1, 2011 at 5:50 PM, Alan Gates <ga...@hortonworks.com> wrote:
> I have gotten the end-to-end test harness to the point where it runs
> basically all the existing tests and where it can be run from ant. It can
> be run on either an existing cluster or in Amazon's EC2. There are
> instructions on how to run it in both settings at
> https://cwiki.apache.org/confluence/display/PIG/HowToTest. Please try it
> out to see if it works in your environment.
>
> As proposed in
> https://cwiki.apache.org/confluence/display/PIG/PigTestProposal, I would
> like to start migrating the junit tests that are really end-to-end tests (ie
> the ones that use MiniCluster) to the e2e harness. I'll start with the very
> long running ones and work down to the shorter ones.
>
> Also, as we contribute new patches these should include end-to-end tests
> that will run in the new harness rather than in junit. True unit tests
> should still, of course, be done in junit.
>
> Alan.
RE: Moving to new e2e harness for end-to-end testing
Posted by Santhosh Srinivasan <sm...@yahoo-inc.com>.
I am super excited. It took years to get to this point where everyone in the community can do this. Great job Alan!
Santhosh
-----Original Message-----
From: Alan Gates [mailto:gates@hortonworks.com]
Sent: Thursday, September 01, 2011 5:51 PM
To:
Subject: Moving to new e2e harness for end-to-end testing
I have gotten the end-to-end test harness to the point where it runs basically all the existing tests and where it can be run from ant. It can be run on either an existing cluster or in Amazon's EC2. There are instructions on how to run it in both settings at https://cwiki.apache.org/confluence/display/PIG/HowToTest. Please try it out to see if it works in your environment.
As proposed in https://cwiki.apache.org/confluence/display/PIG/PigTestProposal, I would like to start migrating the junit tests that are really end-to-end tests (ie the ones that use MiniCluster) to the e2e harness. I'll start with the very long running ones and work down to the shorter ones.
Also, as we contribute new patches these should include end-to-end tests that will run in the new harness rather than in junit. True unit tests should still, of course, be done in junit.
Alan.