You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jesse Yates <je...@gmail.com> on 2011/12/02 03:34:31 UTC
Re: dev Digest of: thread.25579
I'm generally on the same page here, but this is what I'm (currently)
thinking about how it could work:
We need to characterize the unit tests because some are longer than
others, and we want to split them up in order of complexity.
Keywal, this next bit is divergent from what I was saying the other day...
The next level up should be testing between integration of components.
Again this should be run by the CI; really this is more of a finer
grain separation of what was previously just unit tests.
I've revised my thinking such that this could a different category of
unit tests (maybe @Integration that run in their own jvm, in a
different step).
This helps separate concerns between the individual class testing and
the testing of multiple pieces working together.
Onto using the failsafe plugin.
I like the idea of really leveraging the failsafe plugin to spin up
the tests on some of 'real' instance. The (canonical) simple example
of using failsafe is to spin up a Jetty server and test whatever stuff
you are doing with a 'real'
Also, integrating with BigTop to do some of that stuff would also be
pretty sweet (and good for both communities).
A good place for these tests would be the hbase-it module discussed in
HBASE-4336, since that module can just depend on the rest of the
modules (giving full access to the components). Within that package,
we could conceivably have the multi-item integration tests
(@Integration form above) and then the 'real cluster' tests as well.
+1 Having tests that just use the top level API (e.g. what the FB guys
are using on their dev cluster after running the full test suite) is
really important to make sure we 'real' test cases to ensure
functionality.
Also, agreeing with Roman, making the plugin to enable that is going
to rough, but we should be able to at least find a way to (at least
manually at first) have those api level tests run on a real cluster,
even if it is a company X and they just post the results. Even then a
weekly build/run of that suite could be sufficient at first (though
having that for every commit would be must better - anyone know
someone at Amazon we can scrounge credits from?).
However, I do think that if tests start failing in _any_ of the above
phases then the patch that caused it needs to not be committed OR the
tests need to be revised (which would also be part of the patch, so my
original point stands). That is indicative of broken functionality and
trunk should be kept as stable as possible.
Yeah, devs shouldn't have to run the full set of tests before
submitting a patch, but they should easily be able to run everything
up to (but not including) the api-level testing.
What would be cool though would be that they could even run the
api-level testing, but that is just runs on a local MiniCluster, but
that's a lot of work out :).
A lot of that stability comes form having a staged testing cycle as
well as helping devs cut down their testing cycles, so anything we do
to help cut that time down is good stuff in my book..
-- Jesse
-------------------
ps. I hate to be 'that guy' but we need a consistent way of describing
the tests we talking about, but lets push that off until figure out
what the heck we actually want to do.
Jesse Yates
240-888-2200
@jesse_yates
> ---------- Forwarded message ----------
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org
> Date: Mon, 28 Nov 2011 12:23:43 -0800
> Subject: scoping integration tests
> The following discussion is closely related to HBASE-4712.
>
> We should reach general consensus so that the execution of future test
> strategy is smooth.
>
> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
>
> > I was considering 'integration tests' as a separate concern from the
> > large/medium/small _unit_ tests.
> >
> > That is, in fact, why the failsafe plugin was added (and is designed
> for).
> >
> > Currently, we have a lot of tests that fall in the realm of integration
> > tests (testing integration between various pieces, rather than single
> > functionality alone). The most obvious indicator that something is an
> > integration test is when it is using the MiniHBaseCluster since that is
> > really testing the integration between the new feature and the rest of
> the
> > system.
> >
> > The surefire plugin should really be testing unit tests, and then those
> > can be classified as small/medium/large. However, the large would be
> > _extremely_ rare cases.
> >
> > If the community is completely moving to just using the annotations for
> > denoting size of tests, rather than using the integrationTest/unit test
> > stuff, thats fine. However, I still think that if we do that, we should
> > have a separate designation for @IntegrationTest (but that is really
> > already covered in the fact that they would be named differently and
> > therefore run by the different plugin). The reason we need to split them
> > out is to have that separate tier of testing - once they pass unit level,
> > they can be tested at the integration level and then on the acceptance
> test
> > level (though we don't have any of those yet).
> >
> > This probably means more reclassification of tests.
> >
> > Making this explicit is important so people know at what point in the
> > process their patch breaks (as well as staging the testing being good
> > practice in general). Therefore, the small/medium/large classifications
> are
> > not complete. Further, we should should still be able to use the general
> > maven testing (mvn test, mvn verify) to run all the tests as expected,
> > rather than having to know which things extra commands to run. To that
> end,
> > we need to integrate HBASE-4712<
> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book
> and people know how to work with the current
> > testing layout.
> >
> > I've been a bit behind on the testing discussions on dev@, so didn't get
> > to put these thoughts in, but I think they can still be added on. This
> was
> > also part of what Doug, Stack and I were kind of envisioning in the
> > original email to the list.
> >
> > Thanks.
> >
> > -Jesse
> >
> >
> > On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
> >
> >> Today's tests are:
> >> - 416 small tests, executed in ~3 minutes
> >> - 489 medium tests, executed in ~35 minutes (without parallelization)
> >> - 280 large tests, executed in ~90 minutes (without parallelization)
> >>
> >> So may be you could put this new test as large? Will you want to run
> this
> >> test for all central builds? Or do you see it as a specific tests that
> will
> >> be executed only sometimes?
> >>
> >>
> >> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
> >>
> >>> So I guess it's likely to be filtered as it's "not large".
> >>>
> >>>
> >>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>
> >>>> N:
> >>>> Currently IntegrationTestConstraint doesn't have @Category annotation.
> >>>>
> >>>> FYI
> >>>>
> >>>>
> >>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com> wrote:
> >>>>
> >>>>> Yes, I put the large test as integration tests (may be 3 weeks ago?).
> >>>>> So if you mix naming pattern with categories, you can end up with no
> tests
> >>>>> to do.
> >>>>>
> >>>>> It seems logical to me to put large tests in the failsafe module. If
> >>>>> you want to use the failsafe tests for something else, I can remove
> then.
> >>>>>
> >>>>> see the doc in HBASE-4712<
> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the
> patch
> >>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on how
> >>>>> it designed to work today.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> N.
> >>>>>
> >>>>>
> >>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>
> >>>>>> Copying N.
> >>>>>>
> >>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
> >>>>>> jesse.k.yates@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hmmm, that's odd. It was working last week, maybe it was the recent
> >>>>>>> changes N put in.
> >>>>>>>
> >>>>>>> I'll look into it.
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> - Jesse Yates
> >>>>>>>
> >>>>>>> Sent from my iPhone.
> >>>>>>>
> >>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> > Hi,
> >>>>>>> > Please categorize new tests for 4605.
> >>>>>>> >
> >>>>>>> > When I issued this command:
> >>>>>>> > mvn failsafe:integration-test
> >>>>>>> >
> >>>>>>> > I got:
> >>>>>>> >
> >>>>>>> > Results :
> >>>>>>> >
> >>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> >>>>>>> >
> >>>>>>> > Do you have some idea ?
> >>>>>>> >
> >>>>>>> > Thanks
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
> >
> > --
> > -------------------
> > Jesse Yates
> > 240-888-2200
> > @jesse_yates
> >
> >
>
>
> ---------- Forwarded message ----------
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org
> Date: Tue, 29 Nov 2011 22:08:30 -0800
> Subject: Re: scoping integration tests
> From Stack:
> -------
> I think we can have it so everyone gets what they need;
> small/medium/large are done by surefire and then we have a new
> intergration test green field that is run by failsafe that can help us
> get tests for bigtop et al. to run.
>
> ------
>
> I think the above proposal makes sense.
>
> Please share your comments.
>
> On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > The following discussion is closely related to HBASE-4712.
> >
> > We should reach general consensus so that the execution of future test
> > strategy is smooth.
> >
> > On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
> >
> >> I was considering 'integration tests' as a separate concern from the
> >> large/medium/small _unit_ tests.
> >>
> >> That is, in fact, why the failsafe plugin was added (and is designed
> >> for).
> >>
> >> Currently, we have a lot of tests that fall in the realm of integration
> >> tests (testing integration between various pieces, rather than single
> >> functionality alone). The most obvious indicator that something is an
> >> integration test is when it is using the MiniHBaseCluster since that is
> >> really testing the integration between the new feature and the rest of
> the
> >> system.
> >>
> >> The surefire plugin should really be testing unit tests, and then those
> >> can be classified as small/medium/large. However, the large would be
> >> _extremely_ rare cases.
> >>
> >> If the community is completely moving to just using the annotations for
> >> denoting size of tests, rather than using the integrationTest/unit test
> >> stuff, thats fine. However, I still think that if we do that, we should
> >> have a separate designation for @IntegrationTest (but that is really
> >> already covered in the fact that they would be named differently and
> >> therefore run by the different plugin). The reason we need to split them
> >> out is to have that separate tier of testing - once they pass unit
> level,
> >> they can be tested at the integration level and then on the acceptance
> test
> >> level (though we don't have any of those yet).
> >>
> >> This probably means more reclassification of tests.
> >>
> >> Making this explicit is important so people know at what point in the
> >> process their patch breaks (as well as staging the testing being good
> >> practice in general). Therefore, the small/medium/large classifications
> are
> >> not complete. Further, we should should still be able to use the general
> >> maven testing (mvn test, mvn verify) to run all the tests as expected,
> >> rather than having to know which things extra commands to run. To that
> end,
> >> we need to integrate HBASE-4712<
> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book
> and people know how to work with the current
> >> testing layout.
> >>
> >> I've been a bit behind on the testing discussions on dev@, so didn't
> get
> >> to put these thoughts in, but I think they can still be added on. This
> was
> >> also part of what Doug, Stack and I were kind of envisioning in the
> >> original email to the list.
> >>
> >> Thanks.
> >>
> >> -Jesse
> >>
> >>
> >> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
> >>
> >>> Today's tests are:
> >>> - 416 small tests, executed in ~3 minutes
> >>> - 489 medium tests, executed in ~35 minutes (without parallelization)
> >>> - 280 large tests, executed in ~90 minutes (without parallelization)
> >>>
> >>> So may be you could put this new test as large? Will you want to run
> >>> this test for all central builds? Or do you see it as a specific tests
> that
> >>> will be executed only sometimes?
> >>>
> >>>
> >>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
> >>>
> >>>> So I guess it's likely to be filtered as it's "not large".
> >>>>
> >>>>
> >>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>> N:
> >>>>> Currently IntegrationTestConstraint doesn't have @Category
> annotation.
> >>>>>
> >>>>> FYI
> >>>>>
> >>>>>
> >>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com>
> wrote:
> >>>>>
> >>>>>> Yes, I put the large test as integration tests (may be 3 weeks
> ago?).
> >>>>>> So if you mix naming pattern with categories, you can end up with
> no tests
> >>>>>> to do.
> >>>>>>
> >>>>>> It seems logical to me to put large tests in the failsafe module. If
> >>>>>> you want to use the failsafe tests for something else, I can remove
> then.
> >>>>>>
> >>>>>> see the doc in HBASE-4712<
> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the
> patch
> >>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
> how
> >>>>>> it designed to work today.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> N.
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Copying N.
> >>>>>>>
> >>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
> >>>>>>> jesse.k.yates@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
> recent
> >>>>>>>> changes N put in.
> >>>>>>>>
> >>>>>>>> I'll look into it.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> - Jesse Yates
> >>>>>>>>
> >>>>>>>> Sent from my iPhone.
> >>>>>>>>
> >>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> > Hi,
> >>>>>>>> > Please categorize new tests for 4605.
> >>>>>>>> >
> >>>>>>>> > When I issued this command:
> >>>>>>>> > mvn failsafe:integration-test
> >>>>>>>> >
> >>>>>>>> > I got:
> >>>>>>>> >
> >>>>>>>> > Results :
> >>>>>>>> >
> >>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> >>>>>>>> >
> >>>>>>>> > Do you have some idea ?
> >>>>>>>> >
> >>>>>>>> > Thanks
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> -------------------
> >> Jesse Yates
> >> 240-888-2200
> >> @jesse_yates
> >>
> >>
> >
>
>
> ---------- Forwarded message ----------
> From: Jesse Yates <je...@gmail.com>
> To: Ted Yu <yu...@gmail.com>
> Date: Tue, 29 Nov 2011 22:28:44 -0800
> Subject: Re: scoping integration tests
> +1 Overall idea.
>
> I don't know if we should consider the integration tests as the primary
> thing that bigtop runs. Since they are deploying to a small cluster
> (right?), the tests that FB is supposed to be open sourcing soon should
> probably be what they run (and lets call those acceptance tests).
>
> While integration tests could be run by bigtop, they should definitely be
> part of our testing of patches as well - no patches should go into trunk
> that don't pass both the unit tests and integration tests. Then we push in
> bigger patches or cut releases we make sure that the acceptance tests pass.
>
> At least, that's what I was thinking about how testing works.
>
> -Jesse
>
> On Tue, Nov 29, 2011 at 10:08 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > From Stack:
> > -------
> > I think we can have it so everyone gets what they need;
> > small/medium/large are done by surefire and then we have a new
> > intergration test green field that is run by failsafe that can help us
> > get tests for bigtop et al. to run.
> >
> > ------
> >
> > I think the above proposal makes sense.
> >
> > Please share your comments.
> >
> >
> > On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> The following discussion is closely related to HBASE-4712.
> >>
> >> We should reach general consensus so that the execution of future test
> >> strategy is smooth.
> >>
> >> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
> >>
> >>> I was considering 'integration tests' as a separate concern from the
> >>> large/medium/small _unit_ tests.
> >>>
> >>> That is, in fact, why the failsafe plugin was added (and is designed
> >>> for).
> >>>
> >>> Currently, we have a lot of tests that fall in the realm of integration
> >>> tests (testing integration between various pieces, rather than single
> >>> functionality alone). The most obvious indicator that something is an
> >>> integration test is when it is using the MiniHBaseCluster since that is
> >>> really testing the integration between the new feature and the rest of
> the
> >>> system.
> >>>
> >>> The surefire plugin should really be testing unit tests, and then those
> >>> can be classified as small/medium/large. However, the large would be
> >>> _extremely_ rare cases.
> >>>
> >>> If the community is completely moving to just using the annotations for
> >>> denoting size of tests, rather than using the integrationTest/unit test
> >>> stuff, thats fine. However, I still think that if we do that, we should
> >>> have a separate designation for @IntegrationTest (but that is really
> >>> already covered in the fact that they would be named differently and
> >>> therefore run by the different plugin). The reason we need to split
> them
> >>> out is to have that separate tier of testing - once they pass unit
> level,
> >>> they can be tested at the integration level and then on the acceptance
> test
> >>> level (though we don't have any of those yet).
> >>>
> >>> This probably means more reclassification of tests.
> >>>
> >>> Making this explicit is important so people know at what point in the
> >>> process their patch breaks (as well as staging the testing being good
> >>> practice in general). Therefore, the small/medium/large
> classifications are
> >>> not complete. Further, we should should still be able to use the
> general
> >>> maven testing (mvn test, mvn verify) to run all the tests as expected,
> >>> rather than having to know which things extra commands to run. To that
> end,
> >>> we need to integrate HBASE-4712<
> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book
> and people know how to work with the current
> >>> testing layout.
> >>>
> >>> I've been a bit behind on the testing discussions on dev@, so didn't
> >>> get to put these thoughts in, but I think they can still be added on.
> This
> >>> was also part of what Doug, Stack and I were kind of envisioning in the
> >>> original email to the list.
> >>>
> >>> Thanks.
> >>>
> >>> -Jesse
> >>>
> >>>
> >>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
> >>>
> >>>> Today's tests are:
> >>>> - 416 small tests, executed in ~3 minutes
> >>>> - 489 medium tests, executed in ~35 minutes (without parallelization)
> >>>> - 280 large tests, executed in ~90 minutes (without parallelization)
> >>>>
> >>>> So may be you could put this new test as large? Will you want to run
> >>>> this test for all central builds? Or do you see it as a specific
> tests that
> >>>> will be executed only sometimes?
> >>>>
> >>>>
> >>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
> >>>>
> >>>>> So I guess it's likely to be filtered as it's "not large".
> >>>>>
> >>>>>
> >>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>
> >>>>>> N:
> >>>>>> Currently IntegrationTestConstraint doesn't have @Category
> annotation.
> >>>>>>
> >>>>>> FYI
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Yes, I put the large test as integration tests (may be 3 weeks
> >>>>>>> ago?). So if you mix naming pattern with categories, you can end
> up with no
> >>>>>>> tests to do.
> >>>>>>>
> >>>>>>> It seems logical to me to put large tests in the failsafe module.
> If
> >>>>>>> you want to use the failsafe tests for something else, I can
> remove then.
> >>>>>>>
> >>>>>>> see the doc in HBASE-4712<
> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the
> patch
> >>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
> >>>>>>> how it designed to work today.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> N.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>>> Copying N.
> >>>>>>>>
> >>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
> >>>>>>>> jesse.k.yates@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
> >>>>>>>>> recent changes N put in.
> >>>>>>>>>
> >>>>>>>>> I'll look into it.
> >>>>>>>>>
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>> - Jesse Yates
> >>>>>>>>>
> >>>>>>>>> Sent from my iPhone.
> >>>>>>>>>
> >>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> > Hi,
> >>>>>>>>> > Please categorize new tests for 4605.
> >>>>>>>>> >
> >>>>>>>>> > When I issued this command:
> >>>>>>>>> > mvn failsafe:integration-test
> >>>>>>>>> >
> >>>>>>>>> > I got:
> >>>>>>>>> >
> >>>>>>>>> > Results :
> >>>>>>>>> >
> >>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> >>>>>>>>> >
> >>>>>>>>> > Do you have some idea ?
> >>>>>>>>> >
> >>>>>>>>> > Thanks
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> -------------------
> >>> Jesse Yates
> >>> 240-888-2200
> >>> @jesse_yates
> >>>
> >>>
> >>
> >
>
>
> --
> -------------------
> Jesse Yates
> 240-888-2200
> @jesse_yates
>
>
> ---------- Forwarded message ----------
> From: N Keywal <nk...@gmail.com>
> To: Ted Yu <yu...@gmail.com>
> Date: Wed, 30 Nov 2011 09:33:01 +0100
> Subject: Re: scoping integration tests
> ok for me.
>
> I will update & the .pom the doc today
>
> On Wed, Nov 30, 2011 at 7:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > From Stack:
> > -------
> > I think we can have it so everyone gets what they need;
> > small/medium/large are done by surefire and then we have a new
> > intergration test green field that is run by failsafe that can help us
> > get tests for bigtop et al. to run.
> >
> > ------
> >
> > I think the above proposal makes sense.
> >
> > Please share your comments.
> >
> >
> > On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> The following discussion is closely related to HBASE-4712.
> >>
> >> We should reach general consensus so that the execution of future test
> >> strategy is smooth.
> >>
> >> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
> >>
> >>> I was considering 'integration tests' as a separate concern from the
> >>> large/medium/small _unit_ tests.
> >>>
> >>> That is, in fact, why the failsafe plugin was added (and is designed
> >>> for).
> >>>
> >>> Currently, we have a lot of tests that fall in the realm of integration
> >>> tests (testing integration between various pieces, rather than single
> >>> functionality alone). The most obvious indicator that something is an
> >>> integration test is when it is using the MiniHBaseCluster since that is
> >>> really testing the integration between the new feature and the rest of
> the
> >>> system.
> >>>
> >>> The surefire plugin should really be testing unit tests, and then those
> >>> can be classified as small/medium/large. However, the large would be
> >>> _extremely_ rare cases.
> >>>
> >>> If the community is completely moving to just using the annotations for
> >>> denoting size of tests, rather than using the integrationTest/unit test
> >>> stuff, thats fine. However, I still think that if we do that, we should
> >>> have a separate designation for @IntegrationTest (but that is really
> >>> already covered in the fact that they would be named differently and
> >>> therefore run by the different plugin). The reason we need to split
> them
> >>> out is to have that separate tier of testing - once they pass unit
> level,
> >>> they can be tested at the integration level and then on the acceptance
> test
> >>> level (though we don't have any of those yet).
> >>>
> >>> This probably means more reclassification of tests.
> >>>
> >>> Making this explicit is important so people know at what point in the
> >>> process their patch breaks (as well as staging the testing being good
> >>> practice in general). Therefore, the small/medium/large
> classifications are
> >>> not complete. Further, we should should still be able to use the
> general
> >>> maven testing (mvn test, mvn verify) to run all the tests as expected,
> >>> rather than having to know which things extra commands to run. To that
> end,
> >>> we need to integrate HBASE-4712<
> https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book
> and people know how to work with the current
> >>> testing layout.
> >>>
> >>> I've been a bit behind on the testing discussions on dev@, so didn't
> >>> get to put these thoughts in, but I think they can still be added on.
> This
> >>> was also part of what Doug, Stack and I were kind of envisioning in the
> >>> original email to the list.
> >>>
> >>> Thanks.
> >>>
> >>> -Jesse
> >>>
> >>>
> >>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
> >>>
> >>>> Today's tests are:
> >>>> - 416 small tests, executed in ~3 minutes
> >>>> - 489 medium tests, executed in ~35 minutes (without parallelization)
> >>>> - 280 large tests, executed in ~90 minutes (without parallelization)
> >>>>
> >>>> So may be you could put this new test as large? Will you want to run
> >>>> this test for all central builds? Or do you see it as a specific
> tests that
> >>>> will be executed only sometimes?
> >>>>
> >>>>
> >>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
> >>>>
> >>>>> So I guess it's likely to be filtered as it's "not large".
> >>>>>
> >>>>>
> >>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>
> >>>>>> N:
> >>>>>> Currently IntegrationTestConstraint doesn't have @Category
> annotation.
> >>>>>>
> >>>>>> FYI
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Yes, I put the large test as integration tests (may be 3 weeks
> >>>>>>> ago?). So if you mix naming pattern with categories, you can end
> up with no
> >>>>>>> tests to do.
> >>>>>>>
> >>>>>>> It seems logical to me to put large tests in the failsafe module.
> If
> >>>>>>> you want to use the failsafe tests for something else, I can
> remove then.
> >>>>>>>
> >>>>>>> see the doc in HBASE-4712<
> https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the
> patch
> >>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
> >>>>>>> how it designed to work today.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> N.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>>> Copying N.
> >>>>>>>>
> >>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
> >>>>>>>> jesse.k.yates@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
> >>>>>>>>> recent changes N put in.
> >>>>>>>>>
> >>>>>>>>> I'll look into it.
> >>>>>>>>>
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>> - Jesse Yates
> >>>>>>>>>
> >>>>>>>>> Sent from my iPhone.
> >>>>>>>>>
> >>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> > Hi,
> >>>>>>>>> > Please categorize new tests for 4605.
> >>>>>>>>> >
> >>>>>>>>> > When I issued this command:
> >>>>>>>>> > mvn failsafe:integration-test
> >>>>>>>>> >
> >>>>>>>>> > I got:
> >>>>>>>>> >
> >>>>>>>>> > Results :
> >>>>>>>>> >
> >>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> >>>>>>>>> >
> >>>>>>>>> > Do you have some idea ?
> >>>>>>>>> >
> >>>>>>>>> > Thanks
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> -------------------
> >>> Jesse Yates
> >>> 240-888-2200
> >>> @jesse_yates
> >>>
> >>>
> >>
> >
>
>
> ---------- Forwarded message ----------
> From: Stack <st...@duboce.net>
> To: dev@hbase.apache.org
> Date: Wed, 30 Nov 2011 10:11:34 -0800
> Subject: Re: scoping integration tests
> On Tue, Nov 29, 2011 at 10:28 PM, Jesse Yates <je...@gmail.com>
> wrote:
> > I don't know if we should consider the integration tests as the primary
> > thing that bigtop runs. Since they are deploying to a small cluster
> > (right?), the tests that FB is supposed to be open sourcing soon should
> > probably be what they run (and lets call those acceptance tests).
> >
> > While integration tests could be run by bigtop, they should definitely be
> > part of our testing of patches as well - no patches should go into trunk
> > that don't pass both the unit tests and integration tests. Then we push
> in
> > bigger patches or cut releases we make sure that the acceptance tests
> pass.
> >
>
> Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a
> probably broader goals than perhaps what you've been thinking on
> Jesse, I think:
>
> + IntegrationTests* -- where IntegrationTest* is tests with the
> IntegrationTest prefix and are that run when 'mvn verify' as per your
> recent contrib Jesse -- differ from unit tests in that they always run
> against a cluster
> + IntegrationTests* do not expect to be able to put their fingers into
> regionservers and/or master to figure test/pass or fail; they treat
> the cluster as 'remote'
> + IntegrationTest* do not concern themselves with cluster setup and
> teardown -- this is done external to the test code by the failsafe
> plugin + configs (If we do this and the previous point, then our
> IntegrationTest* can be run by bigtop if I understood Roman properly
> yesterday).
> + Cluster setup and teardown needs to be pluggable (it seems like
> failsafe can do this for us). We need this because default cluster
> will be minicluster but then it needs to be easy for us to plug in a
> whirr cluster and/or for the likes of our fb homies to plugin their
> cluster setup, etc.
> + IntegerationTest* are sizeable. You can pass things like do 1M rows
> or 100B rows -- and configurable ("Use 'that' cluster")
>
> I don't think a patch needs to pass all integration tests before it
> gets committed. If a bit of functionality needs an integration-type
> test, then IMO it shoudl be cast as a 'large' unit test run by
> surefire. As per N, devs do 'mvn test' which will run small and
> medium unit tests and then our CI does small/medium/large.
>
> I don't want to get hung up on nomenclature. If there are long email
> threads on test taxonomy, then something has gone awry IMO. A clean
> surefire for unit tests and failsafe for integration tests where unit
> tests are whats runnable w/o need of outside dependencies -- e.g. an
> idling hbase or mr cluster -- and is what devs and our CI does seems
> like a pretty clean separation to me, one that even us hbase devs
> might grok.
>
> What do you reckon?
>
> St.Ack
>
>
>
> ---------- Forwarded message ----------
> From: Roman Shaposhnik <rv...@apache.org>
> To: dev@hbase.apache.org
> Date: Wed, 30 Nov 2011 11:16:21 -0800
> Subject: Re: scoping integration tests
> +bigtop-dev
>
> On Wed, Nov 30, 2011 at 10:11 AM, Stack <st...@duboce.net> wrote:
> > Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a
> > probably broader goals than perhaps what you've been thinking on
> > Jesse, I think:
> >
> > + IntegrationTests* -- where IntegrationTest* is tests with the
> > IntegrationTest prefix and are that run when 'mvn verify' as per your
> > recent contrib Jesse -- differ from unit tests in that they always run
> > against a cluster
>
> Small correction: I don't "always run against a cluster" is a useful
> assumption here. In fact, I really want those tests to be completly
> agnostic of whether they run against a Mini, Pseudo-distributed
> or a fully distributed cluster.
>
> > + IntegrationTest* do not concern themselves with cluster setup and
> > teardown -- this is done external to the test code by the failsafe
> > plugin + configs (If we do this and the previous point, then our
> > IntegrationTest* can be run by bigtop if I understood Roman properly
> > yesterday).
>
> Correct. An excellent example of how this can be done is pigsmoke
> artifact with its -Dpigunit.exectype.cluster=true knob (and a few
> others):
> https://issues.apache.org/jira/browse/PIG-1799
>
> > + Cluster setup and teardown needs to be pluggable (it seems like
> > failsafe can do this for us).
>
> Only if somebody puts an effort of actually implementing a
> deployment plugin (perhaps based on Whirr?). This is a non-trivial
> exercise (although it is a nice project).
>
> > I don't want to get hung up on nomenclature. If there are long email
> > threads on test taxonomy, then something has gone awry IMO. A clean
> > surefire for unit tests and failsafe for integration tests where unit
> > tests are whats runnable w/o need of outside dependencies -- e.g. an
> > idling hbase or mr cluster -- and is what devs and our CI does seems
> > like a pretty clean separation to me, one that even us hbase devs
> > might grok.
> >
> > What do you reckon?
>
> +1
>
> An extra point that I was trying to articulate very strongly to the GB guys
> yesterday is this: we need tests NOW even when we don't have the
> rest of the infrastructure in place (mvn verify, deployment, separation
> of tests, etc.) If they can start giving me access to the tests they've
> created I can start running them in Bigtop today.
>
> Sure they will be "one step removed" from a casual HBase developer
> but the bugs that I'm going to file will be quite real.
>
> Thanks,
> Roman.
>
>
>
--
-------------------
Jesse Yates
240-888-2200
@jesse_yates
Re: dev Digest of: thread.25579
Posted by Jesse Yates <je...@gmail.com>.
I may be splitting hairs while beating a dead horse, but just to get
closure on the issue, my last (probably) 2 cents, in line.
On Thu, Dec 1, 2011 at 10:30 PM, Stack <st...@duboce.net> wrote:
> On Thu, Dec 1, 2011 at 6:34 PM, Jesse Yates <je...@gmail.com>
> wrote:
> > We need to characterize the unit tests because some are longer than
> > others, and we want to split them up in order of complexity.
> >
>
> Nkeywal has done this, no? He's made them into small/medium/large.
> Some tests may need to be moved between categories but the hard graft
> has been done here?
>
> You introduced the IntegrationTest category that is run with failsafe.
> Some of the current large unit tests might get moved over to this new
> IntegrationTest category.
>
>
I feel like most of those would actually go in the new @IntegrationTest
category.
You don't mean that they would be run by the failsafe plugin though, right?
I'm fine with making sure they are more frequently run than the api-level
tests.
>
> > Keywal, this next bit is divergent from what I was saying the other
> day...
> >
> > The next level up should be testing between integration of components.
>
> I'm not sure I grok this. Next level up from unit tests?
Next level up in complexity was my intention here. Medium and Large kinda
cover this.
> But unit
> tests, for both medium and large, are allowed spin up mini clusters so
> they are already testing integration of components? If you are
> suggesting that these kind of minicluster tests get moved out to
> IntegrationTest category, then I'd disagree; or, I disagree that
> should happen any time soon (too much work, too disruptive).
>
> > Again this should be run by the CI;
>
> If you are saying the IntegrationTests should be run the CI, I'm
> thinking not. I'm thinking they'd be run on occasion when trying to
> proof a build or release candidate or trying to verify an hotfix
> before production deploy
>
This is why I think semantics are good groundwork :)
I would argue that unit tests really do test the individual unit (class)
and then integration tests each multiple components. The reason I make the
differentiation is that often times to test a single component properly a
bigger test (possibly on a mini cluster) is needed, but it is not
necessarily testing multiple pieces.
Based on you definition of Integration Test, then no, CI would not be the
way to go. However, based on my above with the @Integration annonation,
then they are not big enough to be a real problem.
Howver, they do kind of work as the 'large' category too. Maybe that should
just be renamed? This is not a sticking point for me - though I have been
harping on it - if the word 'integration' is becoming overloaded.
I'm okay with the current labels, the labeling sematics just feel a little
heavy handed.
+/- 0
>
> > I've revised my thinking such that this could a different category of
> > unit tests (maybe @Integration that run in their own jvm, in a
> > different step).
> > This helps separate concerns between the individual class testing and
> > the testing of multiple pieces working together.
> >
>
> I don't think this viable.
>
> What I do think viable is committing the last nkeywal suggestion of
> 'mvn test' running current small and medium tests but then we have the
> CI run small/medium/large. Then, IntegrationTests are a new category
> altogether of tests that go against the API only and that presume a
> cluster (built by failsafe or otherwise); the tests do not do cluster
> setup/teardown.
>
I worry about code breaking those situations and we only find out when
running the bigger tests when we cut the releases or choose to do
verification. Unless that choice comes weekly, its is going to be a much
bigger pain trying to figure out which patch caused breakage.
Again, a semantic issue where I would have most of the tests that spin up
miniclusters as integration tests not the api tests.
> > Yeah, devs shouldn't have to run the full set of tests before
> > submitting a patch, but they should easily be able to run everything
> > up to (but not including) the api-level testing.
>
> Yes. Nkeywal has this going on in his patch.
>
> I'd say too that the failsafe plugin should out of the box let devs
> run all IntegrationTest. failsafe default might be to spin up a mini
> hbase cluster.
>
>
+1 on that being the default and then having a profile that bigtop uses to
deploy out to a real cluster. Given that the default would be to spin the
minicluster, it would be sweet if CI could run those too. Its another level
of assurance (it runs on the mini cluster so I'm _pretty sure_ it works)
and one that I would be super comfortable pushing things into the codebase
with (without it feels a little risky).
Maybe we could have a nightly/weekly jenkins build that just pulls the
current codebase and runs those, rather than for every commit?
That would keep the load on the system as low as possible and at the same
time help us minimize the time to find the tests.
So either way, as long as we have a fairly easy way to run them, it should
alright :)
-- Jesse
Re: dev Digest of: thread.25579
Posted by Stack <st...@duboce.net>.
On Thu, Dec 1, 2011 at 6:34 PM, Jesse Yates <je...@gmail.com> wrote:
> We need to characterize the unit tests because some are longer than
> others, and we want to split them up in order of complexity.
>
Nkeywal has done this, no? He's made them into small/medium/large.
Some tests may need to be moved between categories but the hard graft
has been done here?
You introduced the IntegrationTest category that is run with failsafe.
Some of the current large unit tests might get moved over to this new
IntegrationTest category.
> Keywal, this next bit is divergent from what I was saying the other day...
>
> The next level up should be testing between integration of components.
I'm not sure I grok this. Next level up from unit tests? But unit
tests, for both medium and large, are allowed spin up mini clusters so
they are already testing integration of components? If you are
suggesting that these kind of minicluster tests get moved out to
IntegrationTest category, then I'd disagree; or, I disagree that
should happen any time soon (too much work, too disruptive).
> Again this should be run by the CI;
If you are saying the IntegrationTests should be run the CI, I'm
thinking not. I'm thinking they'd be run on occasion when trying to
proof a build or release candidate or trying to verify an hotfix
before production deploy
> I've revised my thinking such that this could a different category of
> unit tests (maybe @Integration that run in their own jvm, in a
> different step).
> This helps separate concerns between the individual class testing and
> the testing of multiple pieces working together.
>
I don't think this viable.
What I do think viable is committing the last nkeywal suggestion of
'mvn test' running current small and medium tests but then we have the
CI run small/medium/large. Then, IntegrationTests are a new category
altogether of tests that go against the API only and that presume a
cluster (built by failsafe or otherwise); the tests do not do cluster
setup/teardown.
> A good place for these tests would be the hbase-it module discussed in
> HBASE-4336, since that module can just depend on the rest of the
> modules (giving full access to the components). Within that package,
> we could conceivably have the multi-item integration tests
> (@Integration form above) and then the 'real cluster' tests as well.
>
I'd be down w/ this. Currently IntegrationTest* are mixed in w/ unit
tests but it would be best if they were off in a module of their own.
> +1 Having tests that just use the top level API (e.g. what the FB guys
> are using on their dev cluster after running the full test suite) is
> really important to make sure we 'real' test cases to ensure
> functionality.
>
> Also, agreeing with Roman, making the plugin to enable that is going
> to rough, but we should be able to at least find a way to (at least
> manually at first) have those api level tests run on a real cluster,
> even if it is a company X and they just post the results. Even then a
> weekly build/run of that suite could be sufficient at first (though
> having that for every commit would be must better - anyone know
> someone at Amazon we can scrounge credits from?).
>
+1 on above too.
> However, I do think that if tests start failing in _any_ of the above
> phases then the patch that caused it needs to not be committed OR the
> tests need to be revised (which would also be part of the patch, so my
> original point stands). That is indicative of broken functionality and
> trunk should be kept as stable as possible.
>
Depends. Patch may have been committed before an IntegationTest is
run. I think we should allow this. For sure, if IntegrationTests are
broke, we need to figure which patch broke things.
> Yeah, devs shouldn't have to run the full set of tests before
> submitting a patch, but they should easily be able to run everything
> up to (but not including) the api-level testing.
Yes. Nkeywal has this going on in his patch.
I'd say too that the failsafe plugin should out of the box let devs
run all IntegrationTest. failsafe default might be to spin up a mini
hbase cluster.
> What would be cool though would be that they could even run the
> api-level testing, but that is just runs on a local MiniCluster, but
> that's a lot of work out :).
I don't think so. In fact I think it a prereq for easy IntegrationTest dev'ing.
>
> ps. I hate to be 'that guy' but we need a consistent way of describing
> the tests we talking about, but lets push that off until figure out
> what the heck we actually want to do.
>
The way I see it, I think we have a proposal on what to do. Nkeywal
has done a refactoring of our unit tests into categories but we need a
new category of tests, tests that are optionally long-running against
clusters of variable size doing scales beyond what would be tenable on
Apache's or anyone's CI. These are the tests we'd give our bigtop
brothers and that we'd run when we wanted to do a verification beyond
what our unit tests can do.
Good on you Jesse,
St.Ack