You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Andrew Purtell <ap...@apache.org> on 2012/12/20 20:54:56 UTC
Move TestReplication out of LargeTests into hbase-it instead?
See
https://issues.apache.org/jira/browse/HBASE-5778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537252#comment-13537252
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Enis Söztutar <en...@gmail.com>.
No it is not expected to fail actually. Could you please open a bug.
Enis
On Tue, Jan 8, 2013 at 12:23 PM, Stack <st...@duboce.net> wrote:
> IntegrationTestRebalanceAndKillServersTargeted
>
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Stack <st...@duboce.net>.
On Fri, Dec 21, 2012 at 10:37 AM, Stack <st...@duboce.net> wrote:
> On Thu, Dec 20, 2012 at 3:55 PM, Andrew Purtell <ap...@apache.org>wrote:
>
>> Any interest in setting up a consortium for a HBase testing facility? We
>> can't make point contributions to ASF to go to a specific project, but
>> it's
>> not unreasonable to suggest a couple HBase dev shops could pitch in a few
>> bucks a month for project administered EC2 account for spinning up
>> hbase-it. (This would be minimum viable I think, EC2 is ok for functional
>> and smoke testing at least.)
>>
>>
> One suggestion that has been made, and that I will follow up on, is that
> we ask if we can use the bigtop infrastructure. They have an
> infrastructure already in place (sponsored by Cloudera) that is apart and
> unrelated to Apache dedicated to bigtop building. It is at
> http://bigtop01.cloudera.org:8080 Let me go investigate and report back.
> Pluses are that it is already set up. Cons are we would be competing w/
> bigtop builds but it should be possible to get extra resources dedicated if
> too much competition.
>
>
I wrote bigtop mailing list asking about the above [1] and just talked to
Roman about what would be involved getting hbase it tests running on bigtop
infra. Roman talked up what you'd expect, that bigtop infrastructure is a
community resource/effort so running big long tests w/ chaos monkey's doing
their damnedest is all well and good but only if someone is actually paying
attention to the running cluster and making use of the result. We talked
about trying a pilot effort to get something small going first and then
working up from there. Let me work at getting something to try on bigtop
infra with their permission.
Toward that end, I just ran mvn verify in the hbase-it module as per the
doc [2]. My thought was that we could get the base set of it tests running
over on bigtop if only on a single node for now. The tests failed though:
Running org.apache.hadoop.hbase.IntegrationTestDataIngestWithChaosMonkey
2013-01-08 11:35:51.209 java[51577:1703] Unable to load realm info from
SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 346.926 sec
Running org.apache.hadoop.hbase.IntegrationTestManyRegions
2013-01-08 11:41:38.782 java[51635:1703] Unable to load realm info from
SCDynamicStore
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 145.428 sec
Running
org.apache.hadoop.hbase.IntegrationTestRebalanceAndKillServersTargeted
2013-01-08 11:44:04.811 java[51672:1703] Unable to load realm info from
SCDynamicStore
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 230.595 sec
<<< FAILURE!
Is that to be expected? Is IntegrationTestRebalanceAndKillServersTargeted
expected to fail currently?
Thanks,
St.Ack
1.
http://mail-archives.apache.org/mod_mbox/bigtop-dev/201212.mbox/%3CCA+RK=_Dn=viqWkmRtfa=_2SetExMCDOuB8Ut0Pm8vdABiyFCYA@mail.gmail.com%3E
2. http://hbase.apache.org/book.html#integration.tests
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Stack <st...@duboce.net>.
On Thu, Dec 20, 2012 at 3:55 PM, Andrew Purtell <ap...@apache.org> wrote:
> Any interest in setting up a consortium for a HBase testing facility? We
> can't make point contributions to ASF to go to a specific project, but it's
> not unreasonable to suggest a couple HBase dev shops could pitch in a few
> bucks a month for project administered EC2 account for spinning up
> hbase-it. (This would be minimum viable I think, EC2 is ok for functional
> and smoke testing at least.)
>
>
One suggestion that has been made, and that I will follow up on, is that we
ask if we can use the bigtop infrastructure. They have an infrastructure
already in place (sponsored by Cloudera) that is apart and unrelated to
Apache dedicated to bigtop building. It is at
http://bigtop01.cloudera.org:8080 Let me go investigate and report back.
Pluses are that it is already set up. Cons are we would be competing w/
bigtop builds but it should be possible to get extra resources dedicated if
too much competition.
Otherwise, if this does not work out, for hbase integration tests, yes,
lets try and organize distributing paying for integration test
infrastructure.
St.Ack
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Enis Söztutar <en...@gmail.com>.
Given that this is the replication test, I would expect that it needs at
least two clusters. It is definitely possible to port this to hbase-it, but
some work might be needed.
On Thu, Dec 20, 2012 at 3:55 PM, Andrew Purtell <ap...@apache.org> wrote:
> Any interest in setting up a consortium for a HBase testing facility? We
> can't make point contributions to ASF to go to a specific project, but it's
> not unreasonable to suggest a couple HBase dev shops could pitch in a few
> bucks a month for project administered EC2 account for spinning up
> hbase-it. (This would be minimum viable I think, EC2 is ok for functional
> and smoke testing at least.)
>
>
> On Thu, Dec 20, 2012 at 3:52 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > This test is obviously not helpful as is. We're not losing coverage by
> > moving it out to integration tests. We're gaining sanity though.
> >
> >
> > On Thu, Dec 20, 2012 at 3:49 PM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
> >
> >> On Thu, Dec 20, 2012 at 1:31 PM, Stack <st...@duboce.net> wrote:
> >>
> >> > On Thu, Dec 20, 2012 at 12:01 PM, Jesse Yates <
> jesse.k.yates@gmail.com
> >> > >wrote:
> >> >
> >> > > Not being intimately familiar with the replication tests, how well
> >> > covered
> >> > > on that stuff are we by the remaining unit tests? Also, since the
> >> > hbase-it
> >> > > tests still get run, are we really gaining anything in terms of CI
> >> > > reliability?
> >> > >
> >> >
> >> > The hbase-it tests are not run up on jenkins Jesse, not at the moment
> at
> >> > least.
> >> >
> >> >
> >> I'm just a bit worried that we will break things and not have it caught
> on
> >> check-in (the real arbiter of what's a valid patch), meaning we will
> break
> >> the branch without realizing. :-/ We need some way to ensure that the
> >> underlying code is covered.
> >>
> >> At the minimum it needs to be part of the release checklist that we run
> >> the
> >> replication IT test on two real clusters (assuming this is a black-box
> >> test
> >> and not messing with things too much). I don't expect needing more than
> >> small functional clusters (say max 5 nodes?) to test this adequately.
> The
> >> Jenkins machines don't seem sufficient for this, so my gut feel is that
> it
> >> will have to be a release item that the RM needs to verify works (either
> >> personally or by proxy). This could even apply to all the hbase-it, for
> >> releases going forward
> >>
> >> Ideally, we will also have some unit tests that subsume some of the
> tested
> >> functionality when/if we move them to a more infrequent tests, though
> its
> >> hard to say how possible/useful this would be to manage in practice.
> >>
> >> Short term, maybe we disable them and file a Jira to get them rock solid
> >> (or on -it and tested regularly somehow)? This goes back to the long
> >> standing discussion that we should disable flappers until they tell us
> >> something useful.
> >>
> >> Again don't know those tests very well, so these are just general
> >> platitudes :)
> >>
> >>
> >> > Agree to moving TestReplication and perhaps some of the mapreduce
> tests
> >> out
> >> > of unit test core.
> >> >
> >> > That means we should set up a regular run of hbase-it tests somewhere
> >> > though?
> >> >
> >> > St.Ack
> >> >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Andrew Purtell <ap...@apache.org>.
Any interest in setting up a consortium for a HBase testing facility? We
can't make point contributions to ASF to go to a specific project, but it's
not unreasonable to suggest a couple HBase dev shops could pitch in a few
bucks a month for project administered EC2 account for spinning up
hbase-it. (This would be minimum viable I think, EC2 is ok for functional
and smoke testing at least.)
On Thu, Dec 20, 2012 at 3:52 PM, Andrew Purtell <ap...@apache.org> wrote:
> This test is obviously not helpful as is. We're not losing coverage by
> moving it out to integration tests. We're gaining sanity though.
>
>
> On Thu, Dec 20, 2012 at 3:49 PM, Jesse Yates <je...@gmail.com>wrote:
>
>> On Thu, Dec 20, 2012 at 1:31 PM, Stack <st...@duboce.net> wrote:
>>
>> > On Thu, Dec 20, 2012 at 12:01 PM, Jesse Yates <jesse.k.yates@gmail.com
>> > >wrote:
>> >
>> > > Not being intimately familiar with the replication tests, how well
>> > covered
>> > > on that stuff are we by the remaining unit tests? Also, since the
>> > hbase-it
>> > > tests still get run, are we really gaining anything in terms of CI
>> > > reliability?
>> > >
>> >
>> > The hbase-it tests are not run up on jenkins Jesse, not at the moment at
>> > least.
>> >
>> >
>> I'm just a bit worried that we will break things and not have it caught on
>> check-in (the real arbiter of what's a valid patch), meaning we will break
>> the branch without realizing. :-/ We need some way to ensure that the
>> underlying code is covered.
>>
>> At the minimum it needs to be part of the release checklist that we run
>> the
>> replication IT test on two real clusters (assuming this is a black-box
>> test
>> and not messing with things too much). I don't expect needing more than
>> small functional clusters (say max 5 nodes?) to test this adequately. The
>> Jenkins machines don't seem sufficient for this, so my gut feel is that it
>> will have to be a release item that the RM needs to verify works (either
>> personally or by proxy). This could even apply to all the hbase-it, for
>> releases going forward
>>
>> Ideally, we will also have some unit tests that subsume some of the tested
>> functionality when/if we move them to a more infrequent tests, though its
>> hard to say how possible/useful this would be to manage in practice.
>>
>> Short term, maybe we disable them and file a Jira to get them rock solid
>> (or on -it and tested regularly somehow)? This goes back to the long
>> standing discussion that we should disable flappers until they tell us
>> something useful.
>>
>> Again don't know those tests very well, so these are just general
>> platitudes :)
>>
>>
>> > Agree to moving TestReplication and perhaps some of the mapreduce tests
>> out
>> > of unit test core.
>> >
>> > That means we should set up a regular run of hbase-it tests somewhere
>> > though?
>> >
>> > St.Ack
>> >
>>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Andrew Purtell <ap...@apache.org>.
This test is obviously not helpful as is. We're not losing coverage by
moving it out to integration tests. We're gaining sanity though.
On Thu, Dec 20, 2012 at 3:49 PM, Jesse Yates <je...@gmail.com>wrote:
> On Thu, Dec 20, 2012 at 1:31 PM, Stack <st...@duboce.net> wrote:
>
> > On Thu, Dec 20, 2012 at 12:01 PM, Jesse Yates <jesse.k.yates@gmail.com
> > >wrote:
> >
> > > Not being intimately familiar with the replication tests, how well
> > covered
> > > on that stuff are we by the remaining unit tests? Also, since the
> > hbase-it
> > > tests still get run, are we really gaining anything in terms of CI
> > > reliability?
> > >
> >
> > The hbase-it tests are not run up on jenkins Jesse, not at the moment at
> > least.
> >
> >
> I'm just a bit worried that we will break things and not have it caught on
> check-in (the real arbiter of what's a valid patch), meaning we will break
> the branch without realizing. :-/ We need some way to ensure that the
> underlying code is covered.
>
> At the minimum it needs to be part of the release checklist that we run the
> replication IT test on two real clusters (assuming this is a black-box test
> and not messing with things too much). I don't expect needing more than
> small functional clusters (say max 5 nodes?) to test this adequately. The
> Jenkins machines don't seem sufficient for this, so my gut feel is that it
> will have to be a release item that the RM needs to verify works (either
> personally or by proxy). This could even apply to all the hbase-it, for
> releases going forward
>
> Ideally, we will also have some unit tests that subsume some of the tested
> functionality when/if we move them to a more infrequent tests, though its
> hard to say how possible/useful this would be to manage in practice.
>
> Short term, maybe we disable them and file a Jira to get them rock solid
> (or on -it and tested regularly somehow)? This goes back to the long
> standing discussion that we should disable flappers until they tell us
> something useful.
>
> Again don't know those tests very well, so these are just general
> platitudes :)
>
>
> > Agree to moving TestReplication and perhaps some of the mapreduce tests
> out
> > of unit test core.
> >
> > That means we should set up a regular run of hbase-it tests somewhere
> > though?
> >
> > St.Ack
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Jesse Yates <je...@gmail.com>.
On Thu, Dec 20, 2012 at 1:31 PM, Stack <st...@duboce.net> wrote:
> On Thu, Dec 20, 2012 at 12:01 PM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
>
> > Not being intimately familiar with the replication tests, how well
> covered
> > on that stuff are we by the remaining unit tests? Also, since the
> hbase-it
> > tests still get run, are we really gaining anything in terms of CI
> > reliability?
> >
>
> The hbase-it tests are not run up on jenkins Jesse, not at the moment at
> least.
>
>
I'm just a bit worried that we will break things and not have it caught on
check-in (the real arbiter of what's a valid patch), meaning we will break
the branch without realizing. :-/ We need some way to ensure that the
underlying code is covered.
At the minimum it needs to be part of the release checklist that we run the
replication IT test on two real clusters (assuming this is a black-box test
and not messing with things too much). I don't expect needing more than
small functional clusters (say max 5 nodes?) to test this adequately. The
Jenkins machines don't seem sufficient for this, so my gut feel is that it
will have to be a release item that the RM needs to verify works (either
personally or by proxy). This could even apply to all the hbase-it, for
releases going forward
Ideally, we will also have some unit tests that subsume some of the tested
functionality when/if we move them to a more infrequent tests, though its
hard to say how possible/useful this would be to manage in practice.
Short term, maybe we disable them and file a Jira to get them rock solid
(or on -it and tested regularly somehow)? This goes back to the long
standing discussion that we should disable flappers until they tell us
something useful.
Again don't know those tests very well, so these are just general
platitudes :)
> Agree to moving TestReplication and perhaps some of the mapreduce tests out
> of unit test core.
>
> That means we should set up a regular run of hbase-it tests somewhere
> though?
>
> St.Ack
>
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Nick Dimiduk <nd...@gmail.com>.
What size of cluster would be necessary to make such regular runs useful?
How much resources are we talking about here?
-n
On Thu, Dec 20, 2012 at 1:33 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. we should set up a regular run of hbase-it tests somewhere though
>
> +1 on above.
>
> On Thu, Dec 20, 2012 at 1:31 PM, Stack <st...@duboce.net> wrote:
>
> > On Thu, Dec 20, 2012 at 12:01 PM, Jesse Yates <jesse.k.yates@gmail.com
> > >wrote:
> >
> > > Not being intimately familiar with the replication tests, how well
> > covered
> > > on that stuff are we by the remaining unit tests? Also, since the
> > hbase-it
> > > tests still get run, are we really gaining anything in terms of CI
> > > reliability?
> > >
> >
> > The hbase-it tests are not run up on jenkins Jesse, not at the moment at
> > least.
> >
> > Agree to moving TestReplication and perhaps some of the mapreduce tests
> out
> > of unit test core.
> >
> > That means we should set up a regular run of hbase-it tests somewhere
> > though?
> >
> > St.Ack
> >
>
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Ted Yu <yu...@gmail.com>.
bq. we should set up a regular run of hbase-it tests somewhere though
+1 on above.
On Thu, Dec 20, 2012 at 1:31 PM, Stack <st...@duboce.net> wrote:
> On Thu, Dec 20, 2012 at 12:01 PM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
>
> > Not being intimately familiar with the replication tests, how well
> covered
> > on that stuff are we by the remaining unit tests? Also, since the
> hbase-it
> > tests still get run, are we really gaining anything in terms of CI
> > reliability?
> >
>
> The hbase-it tests are not run up on jenkins Jesse, not at the moment at
> least.
>
> Agree to moving TestReplication and perhaps some of the mapreduce tests out
> of unit test core.
>
> That means we should set up a regular run of hbase-it tests somewhere
> though?
>
> St.Ack
>
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Stack <st...@duboce.net>.
On Thu, Dec 20, 2012 at 12:01 PM, Jesse Yates <je...@gmail.com>wrote:
> Not being intimately familiar with the replication tests, how well covered
> on that stuff are we by the remaining unit tests? Also, since the hbase-it
> tests still get run, are we really gaining anything in terms of CI
> reliability?
>
The hbase-it tests are not run up on jenkins Jesse, not at the moment at
least.
Agree to moving TestReplication and perhaps some of the mapreduce tests out
of unit test core.
That means we should set up a regular run of hbase-it tests somewhere
though?
St.Ack
Re: Move TestReplication out of LargeTests into hbase-it instead?
Posted by Jesse Yates <je...@gmail.com>.
Not being intimately familiar with the replication tests, how well covered
on that stuff are we by the remaining unit tests? Also, since the hbase-it
tests still get run, are we really gaining anything in terms of CI
reliability?
But yeah, that test is massive and could definitely benefit from running on
a real clusters, rather than just miniclusters.
My $0.02
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Thu, Dec 20, 2012 at 11:54 AM, Andrew Purtell <ap...@apache.org>wrote:
> See
>
> https://issues.apache.org/jira/browse/HBASE-5778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537252#comment-13537252
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>