[jira] [Commented] (HDDS-2964) Fix @Ignore-d integration tests

["Attila Doroszlai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038617#comment-17038617 ] 

Attila Doroszlai commented on HDDS-2964:
----------------------------------------

Thanks [~shashikant] and [~msingh] for working on these fixes.

Note that the disabled tests are intermittent.  We had cases where a test failed after 15 completely clean runs.

I would like to propose the following process to give us more confidence in the tests being enabled in a shorter timeframe:

# create a branch in own fork for the fix (nothing unusual here)
# disable all irrelevant tests in a separate commit (to get faster feedback)
# commit the fix, push to own fork
# incrementally push empty commits to the branch to get independent test results
# after N successful runs (eg. N = 20 or so), revert commit from step #2
# wait for full CI run in fork
# create PR

> Fix @Ignore-d integration tests
> -------------------------------
>
>                 Key: HDDS-2964
>                 URL: https://issues.apache.org/jira/browse/HDDS-2964
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>          Components: test
>            Reporter: Marton Elek
>            Priority: Major
>
> We marked all the intermittent unit tests with @Ignore to get reliable feedback from CI builds.
> Before HDDS-2833 we had 21 @Ignore annotations, HDDS-2833 introduced 34 new one.
> We need to review all of these tests and either fix, or delete or convert them to real unit tests.
> The current list of ignore tests:
> {code:java}
> hadoop-hdds/server-scm org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java:  @Ignore
> hadoop-hdds/server-scm org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java:  @Ignore("Tracked by HDDS-2508.")
> hadoop-hdds/server-scm org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java:  @Ignore
> hadoop-hdds/server-scm org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java:  @Ignore
> hadoop-ozone/integration-test org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java:  @Ignore("TODO:HDDS-1159")
> hadoop-ozone/integration-test org/apache/hadoop/hdds/scm/pipeline/TestNodeFailure.java:  @Ignore
> hadoop-ozone/integration-test org/apache/hadoop/hdds/scm/pipeline/TestNodeFailure.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineCreateAndDestroy.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeWithPipelineRules.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestCloseContainerHandlingByClient.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestCloseContainerHandlingByClient.java:  @Ignore // test needs to be fixed after close container is handled for
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestContainerReplicationEndToEnd.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailures.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestContainerStateMachine.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestDeleteWithSlowFollower.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClient.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestMultiBlockWritesWithDnFailures.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestOzoneAtRestEncryption.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnException.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java:  @Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java:  @Ignore("Debug Jenkins Timeout")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java:@Ignore("Fix this after adding audit support for HA Acl code. This will be " +
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientWithRatis.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestSecureOzoneRpcClient.java:  @Ignore("Needs to be moved out of this class as  client setup is static")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestBlockDeletion.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerByPipeline.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/common/transport/server/ratis/TestCSMMetrics.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainerRatis.java:@Ignore("Disabling Ratis tests for pipeline work.")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainerWithTLS.java:@Ignore("TODO:HDDS-1157")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/ozoneimpl/TestRatisManager.java:@Ignore("Disabling Ratis tests for pipeline work.")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/server/TestContainerServer.java:@Ignore("Takes too long to run this test. Ignoring for time being.")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/server/TestSecureContainerServer.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/container/TestContainerReplication.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/freon/TestFreonWithDatanodeFastRestart.java:  @Ignore("TODO:HDDS-1160")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/freon/TestFreonWithDatanodeRestart.java:  @Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/freon/TestFreonWithPipelineDestroy.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/om/TestOmMetrics.java:  @Ignore("Test failing because of table cache. Revisit later.")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/om/TestOmSQLCli.java:  @Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/om/TestOzoneManagerHA.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/om/TestOzoneManagerRestart.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/om/TestScmSafeMode.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/scm/node/TestQueryNode.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/scm/pipeline/TestSCMPipelineMetrics.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/scm/TestSCMContainerPlacementPolicyMetrics.java:@Ignore("HDDS-2961")
> hadoop-ozone/integration-test org/apache/hadoop/ozone/TestMiniOzoneCluster.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/TestStorageContainerManager.java:@Ignore
> hadoop-ozone/ozone-manager org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java:@Ignore("HDDS-2648")
> hadoop-ozone/ozone-manager org/apache/hadoop/ozone/om/TestBucketManagerImpl.java:@Ignore("Bucket Manager does not use cache, Disable it for now.") {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org