Re: scoping integration tests

[Jesse Yates <je...@gmail.com>]

Sorry, about the message with title: "dev Digest of: thread.25579", my
email has been acting up and I forgot to change the subject line when
pulling from the archives.

I'm reposting this to the right thread for the archives and in case anyone
missed it.

-Jesse

On Thu, Dec 1, 2011 at 6:34 PM, Jesse Yates <je...@gmail.com> wrote:

> I'm generally on the same page here, but this is what I'm (currently) thinking about how it could work:
>
>
>
> We need to characterize the unit tests because some are longer than others, and we want to split them up in order of complexity.
>
> Keywal, this next bit is divergent from what I was saying the other day...
>
> The next level up should be testing between integration of components. Again this should be run by the CI; really this is more of a finer grain separation of what was previously just unit tests.
> I've revised my thinking such that this could a different category of unit tests (maybe @Integration that run in their own jvm, in a different step).
> This helps separate concerns between the individual class testing and the testing of multiple pieces working together.
>
>
> Onto using the failsafe plugin.
>
> I like the idea of really leveraging the failsafe plugin to spin up the tests on some of 'real' instance. The (canonical) simple example of using failsafe is to spin up a Jetty server and test whatever stuff you are doing with a 'real'
>
>
> Also, integrating with BigTop to do some of that stuff would also be pretty sweet (and good for both communities).
>
> A good place for these tests would be the hbase-it module discussed in HBASE-4336, since that module can just depend on the rest of the modules (giving full access to the components). Within that package, we could conceivably have the multi-item integration tests (@Integration form above) and then the 'real cluster' tests as well.
>
>
> +1 Having tests that just use the top level API (e.g. what the FB guys are using on their dev cluster after running the full test suite) is really important to make sure we 'real' test cases to ensure functionality.
>
>
> Also, agreeing with Roman, making the plugin to enable that is going to rough, but we should be able to at least find a way to (at least manually at first) have those api level tests run on a real cluster, even if it is a company X and they just post the results. Even then a weekly build/run of that suite could be sufficient at first (though having that for every commit would be must better - anyone know someone at Amazon we can scrounge credits from?).
>
>
> However, I do think that if tests start failing in _any_ of the above phases then the patch that caused it needs to not be committed OR the tests need to be revised (which would also be part of the patch, so my original point stands). That is indicative of broken functionality and trunk should be kept as stable as possible.
>
>
> Yeah, devs shouldn't have to run the full set of tests before submitting a patch, but they should easily be able to run everything up to (but not including) the api-level testing.
> What would be cool though would be that they could even run the api-level testing, but that is just runs on a local MiniCluster, but that's a lot of work out :).
> A lot of that stability comes form having a staged testing cycle as well as helping devs cut down their testing cycles, so anything we do to help cut that time down is good stuff in my book..
>
> -- Jesse
> -------------------
>
>
> ps. I hate to be 'that guy' but we need a consistent way of describing the tests we talking about, but lets push that off until figure out what the heck we actually want to do.
>
>
> Jesse Yates
> 240-888-2200
> @jesse_yates
>
>
>
>> ---------- Forwarded message ----------
>> From: Ted Yu <yu...@gmail.com>
>> To: dev@hbase.apache.org
>> Date: Mon, 28 Nov 2011 12:23:43 -0800
>> Subject: scoping integration tests
>> The following discussion is closely related to HBASE-4712.
>>
>> We should reach general consensus so that the execution of future test
>> strategy is smooth.
>>
>> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com
>> >wrote:
>>
>> > I was considering 'integration tests' as a separate concern from the
>> > large/medium/small _unit_ tests.
>> >
>> > That is, in fact, why the failsafe plugin was added (and is designed
>> for).
>> >
>> > Currently, we have a lot of tests that fall in the realm of integration
>> > tests (testing integration between various pieces, rather than single
>> > functionality alone). The most obvious indicator that something is an
>> > integration test is when it is using the MiniHBaseCluster since that is
>> > really testing the integration between the new feature and the rest of
>> the
>> > system.
>> >
>> > The surefire plugin should really be testing unit tests, and then those
>> > can be classified as small/medium/large. However, the large would be
>> > _extremely_ rare cases.
>> >
>> > If the community is completely moving to just using the annotations for
>> > denoting size of tests, rather than using the integrationTest/unit test
>> > stuff, thats fine. However, I still think that if we do that, we should
>> > have a separate designation for @IntegrationTest (but that is really
>> > already covered in the fact that they would be named differently and
>> > therefore run by the different plugin). The reason we need to split them
>> > out is to have that separate tier of testing - once they pass unit
>> level,
>> > they can be tested at the integration level and then on the acceptance
>> test
>> > level (though we don't have any of those yet).
>> >
>> > This probably means more reclassification of tests.
>> >
>> > Making this explicit is important so people know at what point in the
>> > process their patch breaks (as well as staging the testing being good
>> > practice in general). Therefore, the small/medium/large classifications
>> are
>> > not complete. Further, we should should still be able to use the general
>> > maven testing (mvn test, mvn verify) to run all the tests as expected,
>> > rather than having to know which things extra commands to run. To that
>> end,
>> > we need to integrate HBASE-4712<
>> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book
>> and people know how to work with the current
>> > testing layout.
>> >
>> > I've been a bit behind on the testing discussions on dev@, so didn't
>> get
>> > to put these thoughts in, but I think they can still be added on. This
>> was
>> > also part of what Doug, Stack and I were kind of envisioning in the
>> > original email to the list.
>> >
>> > Thanks.
>> >
>> > -Jesse
>> >
>> >
>> > On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
>> >
>> >> Today's tests are:
>> >>  - 416 small tests, executed in ~3 minutes
>> >>  - 489 medium tests, executed in ~35 minutes (without parallelization)
>> >>  - 280 large tests, executed in ~90 minutes (without parallelization)
>> >>
>> >> So may be you could put this new test as large? Will you want to run
>> this
>> >> test for all central builds? Or do you see it as a specific tests that
>> will
>> >> be executed only sometimes?
>> >>
>> >>
>> >> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
>> >>
>> >>> So I guess it's likely to be filtered as it's "not large".
>> >>>
>> >>>
>> >>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
>> >>>
>> >>>> N:
>> >>>> Currently IntegrationTestConstraint doesn't have @Category
>> annotation.
>> >>>>
>> >>>> FYI
>> >>>>
>> >>>>
>> >>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> Yes, I put the large test as integration tests (may be 3 weeks
>> ago?).
>> >>>>> So if you mix naming pattern with categories, you can end up with
>> no tests
>> >>>>> to do.
>> >>>>>
>> >>>>> It seems logical to me to put large tests in the failsafe module. If
>> >>>>> you want to use the failsafe tests for something else, I can remove
>> then.
>> >>>>>
>> >>>>> see the doc in HBASE-4712<
>> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the
>> patch
>> >>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
>> how
>> >>>>> it designed to work today.
>> >>>>>
>> >>>>> Cheers,
>> >>>>>
>> >>>>> N.
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>>> Copying N.
>> >>>>>>
>> >>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>> >>>>>> jesse.k.yates@gmail.com> wrote:
>> >>>>>>
>> >>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
>> recent
>> >>>>>>> changes N put in.
>> >>>>>>>
>> >>>>>>> I'll look into it.
>> >>>>>>>
>> >>>>>>> Thanks!
>> >>>>>>>
>> >>>>>>> - Jesse Yates
>> >>>>>>>
>> >>>>>>> Sent from my iPhone.
>> >>>>>>>
>> >>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
>> >>>>>>>
>> >>>>>>> > Hi,
>> >>>>>>> > Please categorize new tests for 4605.
>> >>>>>>> >
>> >>>>>>> > When I issued this command:
>> >>>>>>> > mvn failsafe:integration-test
>> >>>>>>> >
>> >>>>>>> > I got:
>> >>>>>>> >
>> >>>>>>> > Results :
>> >>>>>>> >
>> >>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>> >>>>>>> >
>> >>>>>>> > Do you have some idea ?
>> >>>>>>> >
>> >>>>>>> > Thanks
>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>> >
>> > --
>> > -------------------
>> > Jesse Yates
>> > 240-888-2200
>> > @jesse_yates
>> >
>> >
>>
>>
>> ---------- Forwarded message ----------
>> From: Ted Yu <yu...@gmail.com>
>> To: dev@hbase.apache.org
>> Date: Tue, 29 Nov 2011 22:08:30 -0800
>> Subject: Re: scoping integration tests
>> From Stack:
>> -------
>> I think we can have it so everyone gets what they need;
>> small/medium/large are done by surefire and then we have a new
>> intergration test green field that is run by failsafe that can help us
>> get tests for bigtop et al. to run.
>>
>> ------
>>
>> I think the above proposal makes sense.
>>
>> Please share your comments.
>>
>> On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > The following discussion is closely related to HBASE-4712.
>> >
>> > We should reach general consensus so that the execution of future test
>> > strategy is smooth.
>> >
>> > On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com
>> >wrote:
>> >
>> >> I was considering 'integration tests' as a separate concern from the
>> >> large/medium/small _unit_ tests.
>> >>
>> >> That is, in fact, why the failsafe plugin was added (and is designed
>> >> for).
>> >>
>> >> Currently, we have a lot of tests that fall in the realm of integration
>> >> tests (testing integration between various pieces, rather than single
>> >> functionality alone). The most obvious indicator that something is an
>> >> integration test is when it is using the MiniHBaseCluster since that is
>> >> really testing the integration between the new feature and the rest of
>> the
>> >> system.
>> >>
>> >> The surefire plugin should really be testing unit tests, and then those
>> >> can be classified as small/medium/large. However, the large would be
>> >> _extremely_ rare cases.
>> >>
>> >> If the community is completely moving to just using the annotations for
>> >> denoting size of tests, rather than using the integrationTest/unit test
>> >> stuff, thats fine. However, I still think that if we do that, we should
>> >> have a separate designation for @IntegrationTest (but that is really
>> >> already covered in the fact that they would be named differently and
>> >> therefore run by the different plugin). The reason we need to split
>> them
>> >> out is to have that separate tier of testing - once they pass unit
>> level,
>> >> they can be tested at the integration level and then on the acceptance
>> test
>> >> level (though we don't have any of those yet).
>> >>
>> >> This probably means more reclassification of tests.
>> >>
>> >> Making this explicit is important so people know at what point in the
>> >> process their patch breaks (as well as staging the testing being good
>> >> practice in general). Therefore, the small/medium/large
>> classifications are
>> >> not complete. Further, we should should still be able to use the
>> general
>> >> maven testing (mvn test, mvn verify) to run all the tests as expected,
>> >> rather than having to know which things extra commands to run. To that
>> end,
>> >> we need to integrate HBASE-4712<
>> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book
>> and people know how to work with the current
>> >> testing layout.
>> >>
>> >> I've been a bit behind on the testing discussions on dev@, so didn't
>> get
>> >> to put these thoughts in, but I think they can still be added on. This
>> was
>> >> also part of what Doug, Stack and I were kind of envisioning in the
>> >> original email to the list.
>> >>
>> >> Thanks.
>> >>
>> >> -Jesse
>> >>
>> >>
>> >> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
>> >>
>> >>> Today's tests are:
>> >>>  - 416 small tests, executed in ~3 minutes
>> >>>  - 489 medium tests, executed in ~35 minutes (without parallelization)
>> >>>  - 280 large tests, executed in ~90 minutes (without parallelization)
>> >>>
>> >>> So may be you could put this new test as large? Will you want to run
>> >>> this test for all central builds? Or do you see it as a specific
>> tests that
>> >>> will be executed only sometimes?
>> >>>
>> >>>
>> >>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
>> >>>
>> >>>> So I guess it's likely to be filtered as it's "not large".
>> >>>>
>> >>>>
>> >>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
>> >>>>
>> >>>>> N:
>> >>>>> Currently IntegrationTestConstraint doesn't have @Category
>> annotation.
>> >>>>>
>> >>>>> FYI
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>>> Yes, I put the large test as integration tests (may be 3 weeks
>> ago?).
>> >>>>>> So if you mix naming pattern with categories, you can end up with
>> no tests
>> >>>>>> to do.
>> >>>>>>
>> >>>>>> It seems logical to me to put large tests in the failsafe module.
>> If
>> >>>>>> you want to use the failsafe tests for something else, I can
>> remove then.
>> >>>>>>
>> >>>>>> see the doc in HBASE-4712<
>> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the
>> patch
>> >>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
>> how
>> >>>>>> it designed to work today.
>> >>>>>>
>> >>>>>> Cheers,
>> >>>>>>
>> >>>>>> N.
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>>> Copying N.
>> >>>>>>>
>> >>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>> >>>>>>> jesse.k.yates@gmail.com> wrote:
>> >>>>>>>
>> >>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
>> recent
>> >>>>>>>> changes N put in.
>> >>>>>>>>
>> >>>>>>>> I'll look into it.
>> >>>>>>>>
>> >>>>>>>> Thanks!
>> >>>>>>>>
>> >>>>>>>> - Jesse Yates
>> >>>>>>>>
>> >>>>>>>> Sent from my iPhone.
>> >>>>>>>>
>> >>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>> > Hi,
>> >>>>>>>> > Please categorize new tests for 4605.
>> >>>>>>>> >
>> >>>>>>>> > When I issued this command:
>> >>>>>>>> > mvn failsafe:integration-test
>> >>>>>>>> >
>> >>>>>>>> > I got:
>> >>>>>>>> >
>> >>>>>>>> > Results :
>> >>>>>>>> >
>> >>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>> >>>>>>>> >
>> >>>>>>>> > Do you have some idea ?
>> >>>>>>>> >
>> >>>>>>>> > Thanks
>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >> --
>> >> -------------------
>> >> Jesse Yates
>> >> 240-888-2200
>> >> @jesse_yates
>> >>
>> >>
>> >
>>
>>
>> ---------- Forwarded message ----------
>> From: Jesse Yates <je...@gmail.com>
>> To: Ted Yu <yu...@gmail.com>
>> Date: Tue, 29 Nov 2011 22:28:44 -0800
>> Subject: Re: scoping integration tests
>> +1 Overall idea.
>>
>> I don't know if we should consider the integration tests as the primary
>> thing that bigtop runs. Since they are deploying to a small cluster
>> (right?), the tests that FB is supposed to be open sourcing soon should
>> probably be what they run (and lets call those acceptance tests).
>>
>> While integration tests could be run by bigtop, they should definitely be
>> part of our testing of patches as well - no patches should go into trunk
>> that don't pass both the unit tests and integration tests. Then we push in
>> bigger patches or cut releases we make sure that the acceptance tests
>> pass.
>>
>> At least, that's what I was thinking about how testing works.
>>
>> -Jesse
>>
>> On Tue, Nov 29, 2011 at 10:08 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > From Stack:
>> > -------
>> > I think we can have it so everyone gets what they need;
>> > small/medium/large are done by surefire and then we have a new
>> > intergration test green field that is run by failsafe that can help us
>> > get tests for bigtop et al. to run.
>> >
>> > ------
>> >
>> > I think the above proposal makes sense.
>> >
>> > Please share your comments.
>> >
>> >
>> > On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> >> The following discussion is closely related to HBASE-4712.
>> >>
>> >> We should reach general consensus so that the execution of future test
>> >> strategy is smooth.
>> >>
>> >> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com
>> >wrote:
>> >>
>> >>> I was considering 'integration tests' as a separate concern from the
>> >>> large/medium/small _unit_ tests.
>> >>>
>> >>> That is, in fact, why the failsafe plugin was added (and is designed
>> >>> for).
>> >>>
>> >>> Currently, we have a lot of tests that fall in the realm of
>> integration
>> >>> tests (testing integration between various pieces, rather than single
>> >>> functionality alone). The most obvious indicator that something is an
>> >>> integration test is when it is using the MiniHBaseCluster since that
>> is
>> >>> really testing the integration between the new feature and the rest
>> of the
>> >>> system.
>> >>>
>> >>> The surefire plugin should really be testing unit tests, and then
>> those
>> >>> can be classified as small/medium/large. However, the large would be
>> >>> _extremely_ rare cases.
>> >>>
>> >>> If the community is completely moving to just using the annotations
>> for
>> >>> denoting size of tests, rather than using the integrationTest/unit
>> test
>> >>> stuff, thats fine. However, I still think that if we do that, we
>> should
>> >>> have a separate designation for @IntegrationTest (but that is really
>> >>> already covered in the fact that they would be named differently and
>> >>> therefore run by the different plugin). The reason we need to split
>> them
>> >>> out is to have that separate tier of testing - once they pass unit
>> level,
>> >>> they can be tested at the integration level and then on the
>> acceptance test
>> >>> level (though we don't have any of those yet).
>> >>>
>> >>> This probably means more reclassification of tests.
>> >>>
>> >>> Making this explicit is important so people know at what point in the
>> >>> process their patch breaks (as well as staging the testing being good
>> >>> practice in general). Therefore, the small/medium/large
>> classifications are
>> >>> not complete. Further, we should should still be able to use the
>> general
>> >>> maven testing (mvn test, mvn verify) to run all the tests as expected,
>> >>> rather than having to know which things extra commands to run. To
>> that end,
>> >>> we need to integrate HBASE-4712<
>> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book
>> and people know how to work with the current
>> >>> testing layout.
>> >>>
>> >>> I've been a bit behind on the testing discussions on dev@, so didn't
>> >>> get to put these thoughts in, but I think they can still be added on.
>> This
>> >>> was also part of what Doug, Stack and I were kind of envisioning in
>> the
>> >>> original email to the list.
>> >>>
>> >>> Thanks.
>> >>>
>> >>> -Jesse
>> >>>
>> >>>
>> >>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
>> >>>
>> >>>> Today's tests are:
>> >>>>  - 416 small tests, executed in ~3 minutes
>> >>>>  - 489 medium tests, executed in ~35 minutes (without
>> parallelization)
>> >>>>  - 280 large tests, executed in ~90 minutes (without parallelization)
>> >>>>
>> >>>> So may be you could put this new test as large? Will you want to run
>> >>>> this test for all central builds? Or do you see it as a specific
>> tests that
>> >>>> will be executed only sometimes?
>> >>>>
>> >>>>
>> >>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
>> >>>>
>> >>>>> So I guess it's likely to be filtered as it's "not large".
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>>> N:
>> >>>>>> Currently IntegrationTestConstraint doesn't have @Category
>> annotation.
>> >>>>>>
>> >>>>>> FYI
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>>> Yes, I put the large test as integration tests (may be 3 weeks
>> >>>>>>> ago?). So if you mix naming pattern with categories, you can end
>> up with no
>> >>>>>>> tests to do.
>> >>>>>>>
>> >>>>>>> It seems logical to me to put large tests in the failsafe module.
>> If
>> >>>>>>> you want to use the failsafe tests for something else, I can
>> remove then.
>> >>>>>>>
>> >>>>>>> see the doc in HBASE-4712<
>> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the
>> patch
>> >>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
>> >>>>>>> how it designed to work today.
>> >>>>>>>
>> >>>>>>> Cheers,
>> >>>>>>>
>> >>>>>>> N.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >>>>>>>
>> >>>>>>>> Copying N.
>> >>>>>>>>
>> >>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>> >>>>>>>> jesse.k.yates@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
>> >>>>>>>>> recent changes N put in.
>> >>>>>>>>>
>> >>>>>>>>> I'll look into it.
>> >>>>>>>>>
>> >>>>>>>>> Thanks!
>> >>>>>>>>>
>> >>>>>>>>> - Jesse Yates
>> >>>>>>>>>
>> >>>>>>>>> Sent from my iPhone.
>> >>>>>>>>>
>> >>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> > Hi,
>> >>>>>>>>> > Please categorize new tests for 4605.
>> >>>>>>>>> >
>> >>>>>>>>> > When I issued this command:
>> >>>>>>>>> > mvn failsafe:integration-test
>> >>>>>>>>> >
>> >>>>>>>>> > I got:
>> >>>>>>>>> >
>> >>>>>>>>> > Results :
>> >>>>>>>>> >
>> >>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>> >>>>>>>>> >
>> >>>>>>>>> > Do you have some idea ?
>> >>>>>>>>> >
>> >>>>>>>>> > Thanks
>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> -------------------
>> >>> Jesse Yates
>> >>> 240-888-2200
>> >>> @jesse_yates
>> >>>
>> >>>
>> >>
>> >
>>
>>
>> --
>> -------------------
>> Jesse Yates
>> 240-888-2200
>> @jesse_yates
>>
>>
>> ---------- Forwarded message ----------
>> From: N Keywal <nk...@gmail.com>
>> To: Ted Yu <yu...@gmail.com>
>> Date: Wed, 30 Nov 2011 09:33:01 +0100
>> Subject: Re: scoping integration tests
>> ok for me.
>>
>> I will update & the .pom  the doc today
>>
>> On Wed, Nov 30, 2011 at 7:08 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > From Stack:
>> > -------
>> > I think we can have it so everyone gets what they need;
>> > small/medium/large are done by surefire and then we have a new
>> > intergration test green field that is run by failsafe that can help us
>> > get tests for bigtop et al. to run.
>> >
>> > ------
>> >
>> > I think the above proposal makes sense.
>> >
>> > Please share your comments.
>> >
>> >
>> > On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> >> The following discussion is closely related to HBASE-4712.
>> >>
>> >> We should reach general consensus so that the execution of future test
>> >> strategy is smooth.
>> >>
>> >> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com
>> >wrote:
>> >>
>> >>> I was considering 'integration tests' as a separate concern from the
>> >>> large/medium/small _unit_ tests.
>> >>>
>> >>> That is, in fact, why the failsafe plugin was added (and is designed
>> >>> for).
>> >>>
>> >>> Currently, we have a lot of tests that fall in the realm of
>> integration
>> >>> tests (testing integration between various pieces, rather than single
>> >>> functionality alone). The most obvious indicator that something is an
>> >>> integration test is when it is using the MiniHBaseCluster since that
>> is
>> >>> really testing the integration between the new feature and the rest
>> of the
>> >>> system.
>> >>>
>> >>> The surefire plugin should really be testing unit tests, and then
>> those
>> >>> can be classified as small/medium/large. However, the large would be
>> >>> _extremely_ rare cases.
>> >>>
>> >>> If the community is completely moving to just using the annotations
>> for
>> >>> denoting size of tests, rather than using the integrationTest/unit
>> test
>> >>> stuff, thats fine. However, I still think that if we do that, we
>> should
>> >>> have a separate designation for @IntegrationTest (but that is really
>> >>> already covered in the fact that they would be named differently and
>> >>> therefore run by the different plugin). The reason we need to split
>> them
>> >>> out is to have that separate tier of testing - once they pass unit
>> level,
>> >>> they can be tested at the integration level and then on the
>> acceptance test
>> >>> level (though we don't have any of those yet).
>> >>>
>> >>> This probably means more reclassification of tests.
>> >>>
>> >>> Making this explicit is important so people know at what point in the
>> >>> process their patch breaks (as well as staging the testing being good
>> >>> practice in general). Therefore, the small/medium/large
>> classifications are
>> >>> not complete. Further, we should should still be able to use the
>> general
>> >>> maven testing (mvn test, mvn verify) to run all the tests as expected,
>> >>> rather than having to know which things extra commands to run. To
>> that end,
>> >>> we need to integrate HBASE-4712<
>> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book
>> and people know how to work with the current
>> >>> testing layout.
>> >>>
>> >>> I've been a bit behind on the testing discussions on dev@, so didn't
>> >>> get to put these thoughts in, but I think they can still be added on.
>> This
>> >>> was also part of what Doug, Stack and I were kind of envisioning in
>> the
>> >>> original email to the list.
>> >>>
>> >>> Thanks.
>> >>>
>> >>> -Jesse
>> >>>
>> >>>
>> >>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
>> >>>
>> >>>> Today's tests are:
>> >>>>  - 416 small tests, executed in ~3 minutes
>> >>>>  - 489 medium tests, executed in ~35 minutes (without
>> parallelization)
>> >>>>  - 280 large tests, executed in ~90 minutes (without parallelization)
>> >>>>
>> >>>> So may be you could put this new test as large? Will you want to run
>> >>>> this test for all central builds? Or do you see it as a specific
>> tests that
>> >>>> will be executed only sometimes?
>> >>>>
>> >>>>
>> >>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
>> >>>>
>> >>>>> So I guess it's likely to be filtered as it's "not large".
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>>> N:
>> >>>>>> Currently IntegrationTestConstraint doesn't have @Category
>> annotation.
>> >>>>>>
>> >>>>>> FYI
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>>> Yes, I put the large test as integration tests (may be 3 weeks
>> >>>>>>> ago?). So if you mix naming pattern with categories, you can end
>> up with no
>> >>>>>>> tests to do.
>> >>>>>>>
>> >>>>>>> It seems logical to me to put large tests in the failsafe module.
>> If
>> >>>>>>> you want to use the failsafe tests for something else, I can
>> remove then.
>> >>>>>>>
>> >>>>>>> see the doc in HBASE-4712<
>> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the
>> patch
>> >>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
>> >>>>>>> how it designed to work today.
>> >>>>>>>
>> >>>>>>> Cheers,
>> >>>>>>>
>> >>>>>>> N.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >>>>>>>
>> >>>>>>>> Copying N.
>> >>>>>>>>
>> >>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>> >>>>>>>> jesse.k.yates@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
>> >>>>>>>>> recent changes N put in.
>> >>>>>>>>>
>> >>>>>>>>> I'll look into it.
>> >>>>>>>>>
>> >>>>>>>>> Thanks!
>> >>>>>>>>>
>> >>>>>>>>> - Jesse Yates
>> >>>>>>>>>
>> >>>>>>>>> Sent from my iPhone.
>> >>>>>>>>>
>> >>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> > Hi,
>> >>>>>>>>> > Please categorize new tests for 4605.
>> >>>>>>>>> >
>> >>>>>>>>> > When I issued this command:
>> >>>>>>>>> > mvn failsafe:integration-test
>> >>>>>>>>> >
>> >>>>>>>>> > I got:
>> >>>>>>>>> >
>> >>>>>>>>> > Results :
>> >>>>>>>>> >
>> >>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>> >>>>>>>>> >
>> >>>>>>>>> > Do you have some idea ?
>> >>>>>>>>> >
>> >>>>>>>>> > Thanks
>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> -------------------
>> >>> Jesse Yates
>> >>> 240-888-2200
>> >>> @jesse_yates
>> >>>
>> >>>
>> >>
>> >
>>
>>
>> ---------- Forwarded message ----------
>> From: Stack <st...@duboce.net>
>> To: dev@hbase.apache.org
>> Date: Wed, 30 Nov 2011 10:11:34 -0800
>> Subject: Re: scoping integration tests
>> On Tue, Nov 29, 2011 at 10:28 PM, Jesse Yates <je...@gmail.com>
>> wrote:
>> > I don't know if we should consider the integration tests as the primary
>> > thing that bigtop runs. Since they are deploying to a small cluster
>> > (right?), the tests that FB is supposed to be open sourcing soon should
>> > probably be what they run (and lets call those acceptance tests).
>> >
>> > While integration tests could be run by bigtop, they should definitely
>> be
>> > part of our testing of patches as well - no patches should go into trunk
>> > that don't pass both the unit tests and integration tests. Then we push
>> in
>> > bigger patches or cut releases we make sure that the acceptance tests
>> pass.
>> >
>>
>> Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a
>> probably broader goals than perhaps what you've been thinking on
>> Jesse, I think:
>>
>> + IntegrationTests* -- where IntegrationTest* is tests with the
>> IntegrationTest prefix and are that run when  'mvn verify' as per your
>> recent contrib Jesse -- differ from unit tests in that they always run
>> against a cluster
>> + IntegrationTests* do not expect to be able to put their fingers into
>> regionservers and/or master to figure test/pass or fail; they treat
>> the cluster as 'remote'
>> + IntegrationTest* do not concern themselves with cluster setup and
>> teardown -- this is done external to the test code by the failsafe
>> plugin + configs (If we do this and the previous point, then our
>> IntegrationTest* can be run by bigtop if I understood Roman properly
>> yesterday).
>> + Cluster setup and teardown needs to be pluggable (it seems like
>> failsafe can do this for us).  We need this because default cluster
>> will be minicluster but then it needs to be easy for us to plug in a
>> whirr cluster and/or for the likes of our fb homies to plugin their
>> cluster setup, etc.
>> + IntegerationTest* are sizeable. You can pass things like do 1M rows
>> or 100B rows -- and configurable ("Use 'that' cluster")
>>
>> I don't think a patch needs to pass all integration tests before it
>> gets committed.  If a bit of functionality needs an integration-type
>> test, then IMO it shoudl be cast as a 'large' unit test run by
>> surefire.  As per N, devs do 'mvn test' which will run small and
>> medium unit tests and then our CI does small/medium/large.
>>
>> I don't want to get hung up on nomenclature.  If there are long email
>> threads on test taxonomy, then something has gone awry IMO.  A clean
>> surefire for unit tests and failsafe for integration tests where unit
>> tests are whats runnable w/o need of outside dependencies -- e.g. an
>> idling hbase or mr cluster -- and is what devs and our CI does seems
>> like a pretty clean separation to me, one that even us hbase devs
>> might grok.
>>
>> What do you reckon?
>>
>> St.Ack
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Roman Shaposhnik <rv...@apache.org>
>> To: dev@hbase.apache.org
>> Date: Wed, 30 Nov 2011 11:16:21 -0800
>> Subject: Re: scoping integration tests
>> +bigtop-dev
>>
>> On Wed, Nov 30, 2011 at 10:11 AM, Stack <st...@duboce.net> wrote:
>> > Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a
>> > probably broader goals than perhaps what you've been thinking on
>> > Jesse, I think:
>> >
>> > + IntegrationTests* -- where IntegrationTest* is tests with the
>> > IntegrationTest prefix and are that run when  'mvn verify' as per your
>> > recent contrib Jesse -- differ from unit tests in that they always run
>> > against a cluster
>>
>> Small correction: I don't "always run against a cluster" is a useful
>> assumption here. In fact, I really want those tests to be completly
>> agnostic of whether they run against a Mini, Pseudo-distributed
>> or a fully distributed cluster.
>>
>> > + IntegrationTest* do not concern themselves with cluster setup and
>> > teardown -- this is done external to the test code by the failsafe
>> > plugin + configs (If we do this and the previous point, then our
>> > IntegrationTest* can be run by bigtop if I understood Roman properly
>> > yesterday).
>>
>> Correct. An excellent example of how this can be done is pigsmoke
>> artifact with its -Dpigunit.exectype.cluster=true knob (and a few
>> others):
>>     https://issues.apache.org/jira/browse/PIG-1799
>>
>> > + Cluster setup and teardown needs to be pluggable (it seems like
>> > failsafe can do this for us).
>>
>> Only if somebody puts an effort of actually implementing a
>> deployment plugin (perhaps based on Whirr?). This is a non-trivial
>> exercise (although it is a nice project).
>>
>> > I don't want to get hung up on nomenclature.  If there are long email
>> > threads on test taxonomy, then something has gone awry IMO.  A clean
>> > surefire for unit tests and failsafe for integration tests where unit
>> > tests are whats runnable w/o need of outside dependencies -- e.g. an
>> > idling hbase or mr cluster -- and is what devs and our CI does seems
>> > like a pretty clean separation to me, one that even us hbase devs
>> > might grok.
>> >
>> > What do you reckon?
>>
>> +1
>>
>> An extra point that I was trying to articulate very strongly to the GB
>> guys
>> yesterday is this: we need tests NOW even when we don't have the
>> rest of the infrastructure in place (mvn verify, deployment, separation
>> of tests, etc.) If they can start giving me access to the tests they've
>> created I can start running them in Bigtop today.
>>
>> Sure they will be "one step removed" from a casual HBase developer
>> but the bugs that I'm going to file will be quite real.
>>
>> Thanks,
>> Roman.
>>
>>
>>
>
>
> --
> -------------------
> Jesse Yates
> 240-888-2200
> @jesse_yates
>
>