You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2011/11/28 21:23:43 UTC
scoping integration tests
The following discussion is closely related to HBASE-4712.
We should reach general consensus so that the execution of future test
strategy is smooth.
On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <je...@gmail.com>wrote:
> I was considering 'integration tests' as a separate concern from the
> large/medium/small _unit_ tests.
>
> That is, in fact, why the failsafe plugin was added (and is designed for).
>
> Currently, we have a lot of tests that fall in the realm of integration
> tests (testing integration between various pieces, rather than single
> functionality alone). The most obvious indicator that something is an
> integration test is when it is using the MiniHBaseCluster since that is
> really testing the integration between the new feature and the rest of the
> system.
>
> The surefire plugin should really be testing unit tests, and then those
> can be classified as small/medium/large. However, the large would be
> _extremely_ rare cases.
>
> If the community is completely moving to just using the annotations for
> denoting size of tests, rather than using the integrationTest/unit test
> stuff, thats fine. However, I still think that if we do that, we should
> have a separate designation for @IntegrationTest (but that is really
> already covered in the fact that they would be named differently and
> therefore run by the different plugin). The reason we need to split them
> out is to have that separate tier of testing - once they pass unit level,
> they can be tested at the integration level and then on the acceptance test
> level (though we don't have any of those yet).
>
> This probably means more reclassification of tests.
>
> Making this explicit is important so people know at what point in the
> process their patch breaks (as well as staging the testing being good
> practice in general). Therefore, the small/medium/large classifications are
> not complete. Further, we should should still be able to use the general
> maven testing (mvn test, mvn verify) to run all the tests as expected,
> rather than having to know which things extra commands to run. To that end,
> we need to integrate HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book and people know how to work with the current
> testing layout.
>
> I've been a bit behind on the testing discussions on dev@, so didn't get
> to put these thoughts in, but I think they can still be added on. This was
> also part of what Doug, Stack and I were kind of envisioning in the
> original email to the list.
>
> Thanks.
>
> -Jesse
>
>
> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
>
>> Today's tests are:
>> - 416 small tests, executed in ~3 minutes
>> - 489 medium tests, executed in ~35 minutes (without parallelization)
>> - 280 large tests, executed in ~90 minutes (without parallelization)
>>
>> So may be you could put this new test as large? Will you want to run this
>> test for all central builds? Or do you see it as a specific tests that will
>> be executed only sometimes?
>>
>>
>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
>>
>>> So I guess it's likely to be filtered as it's "not large".
>>>
>>>
>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> N:
>>>> Currently IntegrationTestConstraint doesn't have @Category annotation.
>>>>
>>>> FYI
>>>>
>>>>
>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com> wrote:
>>>>
>>>>> Yes, I put the large test as integration tests (may be 3 weeks ago?).
>>>>> So if you mix naming pattern with categories, you can end up with no tests
>>>>> to do.
>>>>>
>>>>> It seems logical to me to put large tests in the failsafe module. If
>>>>> you want to use the failsafe tests for something else, I can remove then.
>>>>>
>>>>> see the doc in HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the patch
>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on how
>>>>> it designed to work today.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> N.
>>>>>
>>>>>
>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> Copying N.
>>>>>>
>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>>>>>> jesse.k.yates@gmail.com> wrote:
>>>>>>
>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the recent
>>>>>>> changes N put in.
>>>>>>>
>>>>>>> I'll look into it.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> - Jesse Yates
>>>>>>>
>>>>>>> Sent from my iPhone.
>>>>>>>
>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>> > Hi,
>>>>>>> > Please categorize new tests for 4605.
>>>>>>> >
>>>>>>> > When I issued this command:
>>>>>>> > mvn failsafe:integration-test
>>>>>>> >
>>>>>>> > I got:
>>>>>>> >
>>>>>>> > Results :
>>>>>>> >
>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>>>>>>> >
>>>>>>> > Do you have some idea ?
>>>>>>> >
>>>>>>> > Thanks
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> -------------------
> Jesse Yates
> 240-888-2200
> @jesse_yates
>
>
Re: scoping integration tests
Posted by Roman Shaposhnik <rv...@apache.org>.
+bigtop-dev
On Wed, Nov 30, 2011 at 10:11 AM, Stack <st...@duboce.net> wrote:
> Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a
> probably broader goals than perhaps what you've been thinking on
> Jesse, I think:
>
> + IntegrationTests* -- where IntegrationTest* is tests with the
> IntegrationTest prefix and are that run when 'mvn verify' as per your
> recent contrib Jesse -- differ from unit tests in that they always run
> against a cluster
Small correction: I don't "always run against a cluster" is a useful
assumption here. In fact, I really want those tests to be completly
agnostic of whether they run against a Mini, Pseudo-distributed
or a fully distributed cluster.
> + IntegrationTest* do not concern themselves with cluster setup and
> teardown -- this is done external to the test code by the failsafe
> plugin + configs (If we do this and the previous point, then our
> IntegrationTest* can be run by bigtop if I understood Roman properly
> yesterday).
Correct. An excellent example of how this can be done is pigsmoke
artifact with its -Dpigunit.exectype.cluster=true knob (and a few
others):
https://issues.apache.org/jira/browse/PIG-1799
> + Cluster setup and teardown needs to be pluggable (it seems like
> failsafe can do this for us).
Only if somebody puts an effort of actually implementing a
deployment plugin (perhaps based on Whirr?). This is a non-trivial
exercise (although it is a nice project).
> I don't want to get hung up on nomenclature. If there are long email
> threads on test taxonomy, then something has gone awry IMO. A clean
> surefire for unit tests and failsafe for integration tests where unit
> tests are whats runnable w/o need of outside dependencies -- e.g. an
> idling hbase or mr cluster -- and is what devs and our CI does seems
> like a pretty clean separation to me, one that even us hbase devs
> might grok.
>
> What do you reckon?
+1
An extra point that I was trying to articulate very strongly to the GB guys
yesterday is this: we need tests NOW even when we don't have the
rest of the infrastructure in place (mvn verify, deployment, separation
of tests, etc.) If they can start giving me access to the tests they've
created I can start running them in Bigtop today.
Sure they will be "one step removed" from a casual HBase developer
but the bugs that I'm going to file will be quite real.
Thanks,
Roman.
Re: scoping integration tests
Posted by Roman Shaposhnik <rv...@apache.org>.
+bigtop-dev
On Wed, Nov 30, 2011 at 10:11 AM, Stack <st...@duboce.net> wrote:
> Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a
> probably broader goals than perhaps what you've been thinking on
> Jesse, I think:
>
> + IntegrationTests* -- where IntegrationTest* is tests with the
> IntegrationTest prefix and are that run when 'mvn verify' as per your
> recent contrib Jesse -- differ from unit tests in that they always run
> against a cluster
Small correction: I don't "always run against a cluster" is a useful
assumption here. In fact, I really want those tests to be completly
agnostic of whether they run against a Mini, Pseudo-distributed
or a fully distributed cluster.
> + IntegrationTest* do not concern themselves with cluster setup and
> teardown -- this is done external to the test code by the failsafe
> plugin + configs (If we do this and the previous point, then our
> IntegrationTest* can be run by bigtop if I understood Roman properly
> yesterday).
Correct. An excellent example of how this can be done is pigsmoke
artifact with its -Dpigunit.exectype.cluster=true knob (and a few
others):
https://issues.apache.org/jira/browse/PIG-1799
> + Cluster setup and teardown needs to be pluggable (it seems like
> failsafe can do this for us).
Only if somebody puts an effort of actually implementing a
deployment plugin (perhaps based on Whirr?). This is a non-trivial
exercise (although it is a nice project).
> I don't want to get hung up on nomenclature. If there are long email
> threads on test taxonomy, then something has gone awry IMO. A clean
> surefire for unit tests and failsafe for integration tests where unit
> tests are whats runnable w/o need of outside dependencies -- e.g. an
> idling hbase or mr cluster -- and is what devs and our CI does seems
> like a pretty clean separation to me, one that even us hbase devs
> might grok.
>
> What do you reckon?
+1
An extra point that I was trying to articulate very strongly to the GB guys
yesterday is this: we need tests NOW even when we don't have the
rest of the infrastructure in place (mvn verify, deployment, separation
of tests, etc.) If they can start giving me access to the tests they've
created I can start running them in Bigtop today.
Sure they will be "one step removed" from a casual HBase developer
but the bugs that I'm going to file will be quite real.
Thanks,
Roman.
Re: scoping integration tests
Posted by Stack <st...@duboce.net>.
On Tue, Nov 29, 2011 at 10:28 PM, Jesse Yates <je...@gmail.com> wrote:
> I don't know if we should consider the integration tests as the primary
> thing that bigtop runs. Since they are deploying to a small cluster
> (right?), the tests that FB is supposed to be open sourcing soon should
> probably be what they run (and lets call those acceptance tests).
>
> While integration tests could be run by bigtop, they should definitely be
> part of our testing of patches as well - no patches should go into trunk
> that don't pass both the unit tests and integration tests. Then we push in
> bigger patches or cut releases we make sure that the acceptance tests pass.
>
Chatting w/ Roman and Mikhail yesterday at the Hackathon, and with a
probably broader goals than perhaps what you've been thinking on
Jesse, I think:
+ IntegrationTests* -- where IntegrationTest* is tests with the
IntegrationTest prefix and are that run when 'mvn verify' as per your
recent contrib Jesse -- differ from unit tests in that they always run
against a cluster
+ IntegrationTests* do not expect to be able to put their fingers into
regionservers and/or master to figure test/pass or fail; they treat
the cluster as 'remote'
+ IntegrationTest* do not concern themselves with cluster setup and
teardown -- this is done external to the test code by the failsafe
plugin + configs (If we do this and the previous point, then our
IntegrationTest* can be run by bigtop if I understood Roman properly
yesterday).
+ Cluster setup and teardown needs to be pluggable (it seems like
failsafe can do this for us). We need this because default cluster
will be minicluster but then it needs to be easy for us to plug in a
whirr cluster and/or for the likes of our fb homies to plugin their
cluster setup, etc.
+ IntegerationTest* are sizeable. You can pass things like do 1M rows
or 100B rows -- and configurable ("Use 'that' cluster")
I don't think a patch needs to pass all integration tests before it
gets committed. If a bit of functionality needs an integration-type
test, then IMO it shoudl be cast as a 'large' unit test run by
surefire. As per N, devs do 'mvn test' which will run small and
medium unit tests and then our CI does small/medium/large.
I don't want to get hung up on nomenclature. If there are long email
threads on test taxonomy, then something has gone awry IMO. A clean
surefire for unit tests and failsafe for integration tests where unit
tests are whats runnable w/o need of outside dependencies -- e.g. an
idling hbase or mr cluster -- and is what devs and our CI does seems
like a pretty clean separation to me, one that even us hbase devs
might grok.
What do you reckon?
St.Ack
Re: scoping integration tests
Posted by Jesse Yates <je...@gmail.com>.
+1 Overall idea.
I don't know if we should consider the integration tests as the primary
thing that bigtop runs. Since they are deploying to a small cluster
(right?), the tests that FB is supposed to be open sourcing soon should
probably be what they run (and lets call those acceptance tests).
While integration tests could be run by bigtop, they should definitely be
part of our testing of patches as well - no patches should go into trunk
that don't pass both the unit tests and integration tests. Then we push in
bigger patches or cut releases we make sure that the acceptance tests pass.
At least, that's what I was thinking about how testing works.
-Jesse
On Tue, Nov 29, 2011 at 10:08 PM, Ted Yu <yu...@gmail.com> wrote:
> From Stack:
> -------
> I think we can have it so everyone gets what they need;
> small/medium/large are done by surefire and then we have a new
> intergration test green field that is run by failsafe that can help us
> get tests for bigtop et al. to run.
>
> ------
>
> I think the above proposal makes sense.
>
> Please share your comments.
>
>
> On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> The following discussion is closely related to HBASE-4712.
>>
>> We should reach general consensus so that the execution of future test
>> strategy is smooth.
>>
>> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <je...@gmail.com>wrote:
>>
>>> I was considering 'integration tests' as a separate concern from the
>>> large/medium/small _unit_ tests.
>>>
>>> That is, in fact, why the failsafe plugin was added (and is designed
>>> for).
>>>
>>> Currently, we have a lot of tests that fall in the realm of integration
>>> tests (testing integration between various pieces, rather than single
>>> functionality alone). The most obvious indicator that something is an
>>> integration test is when it is using the MiniHBaseCluster since that is
>>> really testing the integration between the new feature and the rest of the
>>> system.
>>>
>>> The surefire plugin should really be testing unit tests, and then those
>>> can be classified as small/medium/large. However, the large would be
>>> _extremely_ rare cases.
>>>
>>> If the community is completely moving to just using the annotations for
>>> denoting size of tests, rather than using the integrationTest/unit test
>>> stuff, thats fine. However, I still think that if we do that, we should
>>> have a separate designation for @IntegrationTest (but that is really
>>> already covered in the fact that they would be named differently and
>>> therefore run by the different plugin). The reason we need to split them
>>> out is to have that separate tier of testing - once they pass unit level,
>>> they can be tested at the integration level and then on the acceptance test
>>> level (though we don't have any of those yet).
>>>
>>> This probably means more reclassification of tests.
>>>
>>> Making this explicit is important so people know at what point in the
>>> process their patch breaks (as well as staging the testing being good
>>> practice in general). Therefore, the small/medium/large classifications are
>>> not complete. Further, we should should still be able to use the general
>>> maven testing (mvn test, mvn verify) to run all the tests as expected,
>>> rather than having to know which things extra commands to run. To that end,
>>> we need to integrate HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book and people know how to work with the current
>>> testing layout.
>>>
>>> I've been a bit behind on the testing discussions on dev@, so didn't
>>> get to put these thoughts in, but I think they can still be added on. This
>>> was also part of what Doug, Stack and I were kind of envisioning in the
>>> original email to the list.
>>>
>>> Thanks.
>>>
>>> -Jesse
>>>
>>>
>>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
>>>
>>>> Today's tests are:
>>>> - 416 small tests, executed in ~3 minutes
>>>> - 489 medium tests, executed in ~35 minutes (without parallelization)
>>>> - 280 large tests, executed in ~90 minutes (without parallelization)
>>>>
>>>> So may be you could put this new test as large? Will you want to run
>>>> this test for all central builds? Or do you see it as a specific tests that
>>>> will be executed only sometimes?
>>>>
>>>>
>>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
>>>>
>>>>> So I guess it's likely to be filtered as it's "not large".
>>>>>
>>>>>
>>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> N:
>>>>>> Currently IntegrationTestConstraint doesn't have @Category annotation.
>>>>>>
>>>>>> FYI
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com> wrote:
>>>>>>
>>>>>>> Yes, I put the large test as integration tests (may be 3 weeks
>>>>>>> ago?). So if you mix naming pattern with categories, you can end up with no
>>>>>>> tests to do.
>>>>>>>
>>>>>>> It seems logical to me to put large tests in the failsafe module. If
>>>>>>> you want to use the failsafe tests for something else, I can remove then.
>>>>>>>
>>>>>>> see the doc in HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the patch
>>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
>>>>>>> how it designed to work today.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> N.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Copying N.
>>>>>>>>
>>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>>>>>>>> jesse.k.yates@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
>>>>>>>>> recent changes N put in.
>>>>>>>>>
>>>>>>>>> I'll look into it.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> - Jesse Yates
>>>>>>>>>
>>>>>>>>> Sent from my iPhone.
>>>>>>>>>
>>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> > Hi,
>>>>>>>>> > Please categorize new tests for 4605.
>>>>>>>>> >
>>>>>>>>> > When I issued this command:
>>>>>>>>> > mvn failsafe:integration-test
>>>>>>>>> >
>>>>>>>>> > I got:
>>>>>>>>> >
>>>>>>>>> > Results :
>>>>>>>>> >
>>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>>>>>>>>> >
>>>>>>>>> > Do you have some idea ?
>>>>>>>>> >
>>>>>>>>> > Thanks
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> -------------------
>>> Jesse Yates
>>> 240-888-2200
>>> @jesse_yates
>>>
>>>
>>
>
--
-------------------
Jesse Yates
240-888-2200
@jesse_yates
Re: scoping integration tests
Posted by N Keywal <nk...@gmail.com>.
ok for me.
I will update & the .pom the doc today
On Wed, Nov 30, 2011 at 7:08 AM, Ted Yu <yu...@gmail.com> wrote:
> From Stack:
> -------
> I think we can have it so everyone gets what they need;
> small/medium/large are done by surefire and then we have a new
> intergration test green field that is run by failsafe that can help us
> get tests for bigtop et al. to run.
>
> ------
>
> I think the above proposal makes sense.
>
> Please share your comments.
>
>
> On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> The following discussion is closely related to HBASE-4712.
>>
>> We should reach general consensus so that the execution of future test
>> strategy is smooth.
>>
>> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <je...@gmail.com>wrote:
>>
>>> I was considering 'integration tests' as a separate concern from the
>>> large/medium/small _unit_ tests.
>>>
>>> That is, in fact, why the failsafe plugin was added (and is designed
>>> for).
>>>
>>> Currently, we have a lot of tests that fall in the realm of integration
>>> tests (testing integration between various pieces, rather than single
>>> functionality alone). The most obvious indicator that something is an
>>> integration test is when it is using the MiniHBaseCluster since that is
>>> really testing the integration between the new feature and the rest of the
>>> system.
>>>
>>> The surefire plugin should really be testing unit tests, and then those
>>> can be classified as small/medium/large. However, the large would be
>>> _extremely_ rare cases.
>>>
>>> If the community is completely moving to just using the annotations for
>>> denoting size of tests, rather than using the integrationTest/unit test
>>> stuff, thats fine. However, I still think that if we do that, we should
>>> have a separate designation for @IntegrationTest (but that is really
>>> already covered in the fact that they would be named differently and
>>> therefore run by the different plugin). The reason we need to split them
>>> out is to have that separate tier of testing - once they pass unit level,
>>> they can be tested at the integration level and then on the acceptance test
>>> level (though we don't have any of those yet).
>>>
>>> This probably means more reclassification of tests.
>>>
>>> Making this explicit is important so people know at what point in the
>>> process their patch breaks (as well as staging the testing being good
>>> practice in general). Therefore, the small/medium/large classifications are
>>> not complete. Further, we should should still be able to use the general
>>> maven testing (mvn test, mvn verify) to run all the tests as expected,
>>> rather than having to know which things extra commands to run. To that end,
>>> we need to integrate HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book and people know how to work with the current
>>> testing layout.
>>>
>>> I've been a bit behind on the testing discussions on dev@, so didn't
>>> get to put these thoughts in, but I think they can still be added on. This
>>> was also part of what Doug, Stack and I were kind of envisioning in the
>>> original email to the list.
>>>
>>> Thanks.
>>>
>>> -Jesse
>>>
>>>
>>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
>>>
>>>> Today's tests are:
>>>> - 416 small tests, executed in ~3 minutes
>>>> - 489 medium tests, executed in ~35 minutes (without parallelization)
>>>> - 280 large tests, executed in ~90 minutes (without parallelization)
>>>>
>>>> So may be you could put this new test as large? Will you want to run
>>>> this test for all central builds? Or do you see it as a specific tests that
>>>> will be executed only sometimes?
>>>>
>>>>
>>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
>>>>
>>>>> So I guess it's likely to be filtered as it's "not large".
>>>>>
>>>>>
>>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> N:
>>>>>> Currently IntegrationTestConstraint doesn't have @Category annotation.
>>>>>>
>>>>>> FYI
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com> wrote:
>>>>>>
>>>>>>> Yes, I put the large test as integration tests (may be 3 weeks
>>>>>>> ago?). So if you mix naming pattern with categories, you can end up with no
>>>>>>> tests to do.
>>>>>>>
>>>>>>> It seems logical to me to put large tests in the failsafe module. If
>>>>>>> you want to use the failsafe tests for something else, I can remove then.
>>>>>>>
>>>>>>> see the doc in HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the patch
>>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on
>>>>>>> how it designed to work today.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> N.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Copying N.
>>>>>>>>
>>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>>>>>>>> jesse.k.yates@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the
>>>>>>>>> recent changes N put in.
>>>>>>>>>
>>>>>>>>> I'll look into it.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> - Jesse Yates
>>>>>>>>>
>>>>>>>>> Sent from my iPhone.
>>>>>>>>>
>>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> > Hi,
>>>>>>>>> > Please categorize new tests for 4605.
>>>>>>>>> >
>>>>>>>>> > When I issued this command:
>>>>>>>>> > mvn failsafe:integration-test
>>>>>>>>> >
>>>>>>>>> > I got:
>>>>>>>>> >
>>>>>>>>> > Results :
>>>>>>>>> >
>>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>>>>>>>>> >
>>>>>>>>> > Do you have some idea ?
>>>>>>>>> >
>>>>>>>>> > Thanks
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> -------------------
>>> Jesse Yates
>>> 240-888-2200
>>> @jesse_yates
>>>
>>>
>>
>
Re: scoping integration tests
Posted by Ted Yu <yu...@gmail.com>.
>From Stack:
-------
I think we can have it so everyone gets what they need;
small/medium/large are done by surefire and then we have a new
intergration test green field that is run by failsafe that can help us
get tests for bigtop et al. to run.
------
I think the above proposal makes sense.
Please share your comments.
On Mon, Nov 28, 2011 at 12:23 PM, Ted Yu <yu...@gmail.com> wrote:
> The following discussion is closely related to HBASE-4712.
>
> We should reach general consensus so that the execution of future test
> strategy is smooth.
>
> On Mon, Nov 28, 2011 at 11:50 AM, Jesse Yates <je...@gmail.com>wrote:
>
>> I was considering 'integration tests' as a separate concern from the
>> large/medium/small _unit_ tests.
>>
>> That is, in fact, why the failsafe plugin was added (and is designed
>> for).
>>
>> Currently, we have a lot of tests that fall in the realm of integration
>> tests (testing integration between various pieces, rather than single
>> functionality alone). The most obvious indicator that something is an
>> integration test is when it is using the MiniHBaseCluster since that is
>> really testing the integration between the new feature and the rest of the
>> system.
>>
>> The surefire plugin should really be testing unit tests, and then those
>> can be classified as small/medium/large. However, the large would be
>> _extremely_ rare cases.
>>
>> If the community is completely moving to just using the annotations for
>> denoting size of tests, rather than using the integrationTest/unit test
>> stuff, thats fine. However, I still think that if we do that, we should
>> have a separate designation for @IntegrationTest (but that is really
>> already covered in the fact that they would be named differently and
>> therefore run by the different plugin). The reason we need to split them
>> out is to have that separate tier of testing - once they pass unit level,
>> they can be tested at the integration level and then on the acceptance test
>> level (though we don't have any of those yet).
>>
>> This probably means more reclassification of tests.
>>
>> Making this explicit is important so people know at what point in the
>> process their patch breaks (as well as staging the testing being good
>> practice in general). Therefore, the small/medium/large classifications are
>> not complete. Further, we should should still be able to use the general
>> maven testing (mvn test, mvn verify) to run all the tests as expected,
>> rather than having to know which things extra commands to run. To that end,
>> we need to integrate HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>ASAP so its in the book and people know how to work with the current
>> testing layout.
>>
>> I've been a bit behind on the testing discussions on dev@, so didn't get
>> to put these thoughts in, but I think they can still be added on. This was
>> also part of what Doug, Stack and I were kind of envisioning in the
>> original email to the list.
>>
>> Thanks.
>>
>> -Jesse
>>
>>
>> On Mon, Nov 28, 2011 at 11:29 AM, N Keywal <nk...@gmail.com> wrote:
>>
>>> Today's tests are:
>>> - 416 small tests, executed in ~3 minutes
>>> - 489 medium tests, executed in ~35 minutes (without parallelization)
>>> - 280 large tests, executed in ~90 minutes (without parallelization)
>>>
>>> So may be you could put this new test as large? Will you want to run
>>> this test for all central builds? Or do you see it as a specific tests that
>>> will be executed only sometimes?
>>>
>>>
>>> On Mon, Nov 28, 2011 at 8:24 PM, N Keywal <nk...@gmail.com> wrote:
>>>
>>>> So I guess it's likely to be filtered as it's "not large".
>>>>
>>>>
>>>> On Mon, Nov 28, 2011 at 8:17 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> N:
>>>>> Currently IntegrationTestConstraint doesn't have @Category annotation.
>>>>>
>>>>> FYI
>>>>>
>>>>>
>>>>> On Mon, Nov 28, 2011 at 11:08 AM, N Keywal <nk...@gmail.com> wrote:
>>>>>
>>>>>> Yes, I put the large test as integration tests (may be 3 weeks ago?).
>>>>>> So if you mix naming pattern with categories, you can end up with no tests
>>>>>> to do.
>>>>>>
>>>>>> It seems logical to me to put large tests in the failsafe module. If
>>>>>> you want to use the failsafe tests for something else, I can remove then.
>>>>>>
>>>>>> see the doc in HBASE-4712<https://issues.apache.org/jira/browse/HBASE-4712>(last comment) and the patch
>>>>>> HBASE-4847 <https://issues.apache.org/jira/browse/HBASE-4847> on how
>>>>>> it designed to work today.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> N.
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 28, 2011 at 8:03 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> Copying N.
>>>>>>>
>>>>>>> On Mon, Nov 28, 2011 at 10:36 AM, Jesse Yates <
>>>>>>> jesse.k.yates@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hmmm, that's odd. It was working last week, maybe it was the recent
>>>>>>>> changes N put in.
>>>>>>>>
>>>>>>>> I'll look into it.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> - Jesse Yates
>>>>>>>>
>>>>>>>> Sent from my iPhone.
>>>>>>>>
>>>>>>>> On Nov 28, 2011, at 9:51 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> > Hi,
>>>>>>>> > Please categorize new tests for 4605.
>>>>>>>> >
>>>>>>>> > When I issued this command:
>>>>>>>> > mvn failsafe:integration-test
>>>>>>>> >
>>>>>>>> > I got:
>>>>>>>> >
>>>>>>>> > Results :
>>>>>>>> >
>>>>>>>> > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
>>>>>>>> >
>>>>>>>> > Do you have some idea ?
>>>>>>>> >
>>>>>>>> > Thanks
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> -------------------
>> Jesse Yates
>> 240-888-2200
>> @jesse_yates
>>
>>
>