You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hbase.apache.org by Nick Dimiduk <nd...@gmail.com> on 2015/05/01 01:10:18 UTC

Flapping tests on 1.1.0RC0 branch

Hi folks,

I've been struggling to get green test runs on branch-1.1. I believe some
of these apply to 1.0 and 0.98 as well (HBASE-13143, HBASE-13391). I filed
tickets for a couple of these earlier in the week (HBASE-13591,
HBASE-13587, HBASE-13590), and then disabled them in search of build
stability. It would be great if we can swarm on these tests and either fix
the bug or fix the test. Remember HBaseCon is just a week away at this
point.

Thanks a lot,
Nick

https://builds.apache.org/job/HBase-1.1.0RC0-JDK7/

org.apache.hadoop.hbase.master.TestSplitLogManager.testGetPreviousRecoveryMode
54 55 57 58 59 60 61 62
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery
59 60 61
org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testLotsOfRegionReplicas[1]
59 60 61
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testRecovery
59 60 61
org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testLotsOfRegionReplicas[0]
60

https://builds.apache.org/job/HBase-1.1.0RC0-JDK8/

org.apache.hadoop.hbase.master.TestSplitLogManager.testGetPreviousRecoveryMode
54 55 57 58 59 60 61 62
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster.org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
58
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery
62
org.apache.hadoop.hbase.regionserver.TestRowTooBig.org.apache.hadoop.hbase.regionserver.TestRowTooBig
58
org.apache.hadoop.hbase.regionserver.TestSCVFWithMiniCluster.org.apache.hadoop.hbase.regionserver.TestSCVFWithMiniCluster
58
org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload.org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload
58
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
58
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor.org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
58
org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testReplicationEndpointReturnsFalseOnReplicate
55
org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed.org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
58
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
58
org.apache.hadoop.hbase.namespace.TestZKLessNamespaceAuditor.org.apache.hadoop.hbase.namespace.TestZKLessNamespaceAuditor
58
org.apache.hadoop.hbase.regionserver.TestRegionReplicas.org.apache.hadoop.hbase.regionserver.TestRegionReplicas
58
org.apache.hadoop.hbase.regionserver.TestEncryptionRandomKeying.org.apache.hadoop.hbase.regionserver.TestEncryptionRandomKeying
58
org.apache.hadoop.hbase.regionserver.TestCompactionState.org.apache.hadoop.hbase.regionserver.TestCompactionState
58
org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay.org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay
58
org.apache.hadoop.hbase.regionserver.TestRegionServerOnlineConfigChange.org.apache.hadoop.hbase.regionserver.TestRegionServerOnlineConfigChange
58
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testRecovery
62
org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod.org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod
58

Re: Flapping tests on 1.1.0RC0 branch

Posted by Nick Dimiduk <nd...@gmail.com>.

Looks like we're using a mix of maven versions. I've had problems with
3.0.x series, maybe bumping to "maven (latest)" across the board will help?

On Thu, Apr 30, 2015 at 4:27 PM, Andrew Purtell <ap...@apache.org> wrote:

> I don't see any of the flakiness of the builds that run on Apache Jenkins
> when running the suite locally for every release candidate, not even after
> 20 repetitions, not even after 20 repetitions on three different JVMs (6,
> 7, and 8). I have the luxury of an EC2 instance type that is big enough to
> be a single tenant on its hardware host. The Apache Jenkins worker pool in
> contrast is made up of VMs running on loaded up hosts and some workers in
> the pool just won't work, we've had to exclude them. (We might need to
> exclude more.) For what it's worth as a data point, as 0.98 RM I've written
> off ASF Jenkins and just use my own resources.
>
> It's also curious that the precommit builds seem to do better than the
> others.
>
>
> On Thu, Apr 30, 2015 at 4:10 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > Hi folks,
> >
> > I've been struggling to get green test runs on branch-1.1. I believe some
> > of these apply to 1.0 and 0.98 as well (HBASE-13143, HBASE-13391). I
> filed
> > tickets for a couple of these earlier in the week (HBASE-13591,
> > HBASE-13587, HBASE-13590), and then disabled them in search of build
> > stability. It would be great if we can swarm on these tests and either
> fix
> > the bug or fix the test. Remember HBaseCon is just a week away at this
> > point.
> >
> > Thanks a lot,
> > Nick
> >
> > https://builds.apache.org/job/HBase-1.1.0RC0-JDK7/
> >
> >
> >
> org.apache.hadoop.hbase.master.TestSplitLogManager.testGetPreviousRecoveryMode
> > 54 55 57 58 59 60 61 62
> >
> >
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery
> > 59 60 61
> >
> >
> org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testLotsOfRegionReplicas[1]
> > 59 60 61
> >
> >
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testRecovery
> > 59 60 61
> >
> >
> org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testLotsOfRegionReplicas[0]
> > 60
> >
> > https://builds.apache.org/job/HBase-1.1.0RC0-JDK8/
> >
> >
> >
> org.apache.hadoop.hbase.master.TestSplitLogManager.testGetPreviousRecoveryMode
> > 54 55 57 58 59 60 61 62
> >
> >
> org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster.org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
> > 58
> >
> >
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery
> > 62
> >
> >
> org.apache.hadoop.hbase.regionserver.TestRowTooBig.org.apache.hadoop.hbase.regionserver.TestRowTooBig
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.TestSCVFWithMiniCluster.org.apache.hadoop.hbase.regionserver.TestSCVFWithMiniCluster
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload.org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
> > 58
> >
> >
> org.apache.hadoop.hbase.namespace.TestNamespaceAuditor.org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
> > 58
> >
> >
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testReplicationEndpointReturnsFalseOnReplicate
> > 55
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed.org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
> > 58
> >
> >
> org.apache.hadoop.hbase.namespace.TestZKLessNamespaceAuditor.org.apache.hadoop.hbase.namespace.TestZKLessNamespaceAuditor
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.TestRegionReplicas.org.apache.hadoop.hbase.regionserver.TestRegionReplicas
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.TestEncryptionRandomKeying.org.apache.hadoop.hbase.regionserver.TestEncryptionRandomKeying
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.TestCompactionState.org.apache.hadoop.hbase.regionserver.TestCompactionState
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay.org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay
> > 58
> >
> >
> org.apache.hadoop.hbase.regionserver.TestRegionServerOnlineConfigChange.org.apache.hadoop.hbase.regionserver.TestRegionServerOnlineConfigChange
> > 58
> >
> >
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testRecovery
> > 62
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod.org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod
> > 58
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: Flapping tests on 1.1.0RC0 branch

Posted by Andrew Purtell <ap...@apache.org>.

I don't see any of the flakiness of the builds that run on Apache Jenkins
when running the suite locally for every release candidate, not even after
20 repetitions, not even after 20 repetitions on three different JVMs (6,
7, and 8). I have the luxury of an EC2 instance type that is big enough to
be a single tenant on its hardware host. The Apache Jenkins worker pool in
contrast is made up of VMs running on loaded up hosts and some workers in
the pool just won't work, we've had to exclude them. (We might need to
exclude more.) For what it's worth as a data point, as 0.98 RM I've written
off ASF Jenkins and just use my own resources.

It's also curious that the precommit builds seem to do better than the
others.


On Thu, Apr 30, 2015 at 4:10 PM, Nick Dimiduk <nd...@gmail.com> wrote:

> Hi folks,
>
> I've been struggling to get green test runs on branch-1.1. I believe some
> of these apply to 1.0 and 0.98 as well (HBASE-13143, HBASE-13391). I filed
> tickets for a couple of these earlier in the week (HBASE-13591,
> HBASE-13587, HBASE-13590), and then disabled them in search of build
> stability. It would be great if we can swarm on these tests and either fix
> the bug or fix the test. Remember HBaseCon is just a week away at this
> point.
>
> Thanks a lot,
> Nick
>
> https://builds.apache.org/job/HBase-1.1.0RC0-JDK7/
>
>
> org.apache.hadoop.hbase.master.TestSplitLogManager.testGetPreviousRecoveryMode
> 54 55 57 58 59 60 61 62
>
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery
> 59 60 61
>
> org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testLotsOfRegionReplicas[1]
> 59 60 61
>
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testRecovery
> 59 60 61
>
> org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover.testLotsOfRegionReplicas[0]
> 60
>
> https://builds.apache.org/job/HBase-1.1.0RC0-JDK8/
>
>
> org.apache.hadoop.hbase.master.TestSplitLogManager.testGetPreviousRecoveryMode
> 54 55 57 58 59 60 61 62
>
> org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster.org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
> 58
>
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery
> 62
>
> org.apache.hadoop.hbase.regionserver.TestRowTooBig.org.apache.hadoop.hbase.regionserver.TestRowTooBig
> 58
>
> org.apache.hadoop.hbase.regionserver.TestSCVFWithMiniCluster.org.apache.hadoop.hbase.regionserver.TestSCVFWithMiniCluster
> 58
>
> org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload.org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload
> 58
>
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
> 58
>
> org.apache.hadoop.hbase.namespace.TestNamespaceAuditor.org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
> 58
>
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testReplicationEndpointReturnsFalseOnReplicate
> 55
>
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed.org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
> 58
>
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
> 58
>
> org.apache.hadoop.hbase.namespace.TestZKLessNamespaceAuditor.org.apache.hadoop.hbase.namespace.TestZKLessNamespaceAuditor
> 58
>
> org.apache.hadoop.hbase.regionserver.TestRegionReplicas.org.apache.hadoop.hbase.regionserver.TestRegionReplicas
> 58
>
> org.apache.hadoop.hbase.regionserver.TestEncryptionRandomKeying.org.apache.hadoop.hbase.regionserver.TestEncryptionRandomKeying
> 58
>
> org.apache.hadoop.hbase.regionserver.TestCompactionState.org.apache.hadoop.hbase.regionserver.TestCompactionState
> 58
>
> org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay.org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay
> 58
>
> org.apache.hadoop.hbase.regionserver.TestRegionServerOnlineConfigChange.org.apache.hadoop.hbase.regionserver.TestRegionServerOnlineConfigChange
> 58
>
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testRecovery
> 62
>
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod.org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod
> 58
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)