You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "xichen01 (via GitHub)" <gi...@apache.org> on 2023/12/29 12:13:20 UTC

[PR] HDDS-9962. Intermittent timeout in TestBlockDeletion.testBlockDeletion [ozone]

xichen01 opened a new pull request, #5886:
URL: https://github.com/apache/ozone/pull/5886

   ## What changes were proposed in this pull request?
   Fix unstable integration tests.
   
   Fix unstable integration tests.
   
   Multiple tests have uncovered a number of things that can cause `TestBlockDeletion` to fail
   1. `hdds.datanode.block.delete.queue.limit` defaults to 5, which may cause tasks to be discarded if they can't be added to the queue, thus causing the test to time out. 
   2. `restartHddsDatanode` in `TestBlockDeletion.testBlockDeletion` can sometimes cause the DN to be restarted before sending the `DeleteBlockTransactionResult` to the SCM.
   https://github.com/apache/ozone/blob/4eca52b6b76b0737bfb2d6ed94097969a0737a45/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestBlockDeletion.java#L299-L310
   
   3. `SCMBlockDeletingService#notifyStatusChanged` in `SCMBlockDeletingService` may be executed several times, resulting in the `serviceStatus` being set from `RUNNING` to `PAUSING`, which leads to `SCMBlockDeletingService` does not work.
   ```bash
   2023-12-28 10:42:20,496 [EventQueue-OpenPipelineForHealthyPipelineSafeModeRule] INFO  safemode.SCMSafeModeManager (SCMSafeModeManager.java:exitSafeMode(244)) - SCM exiting safe mode.  <<--- First
   2023-12-28 10:42:20,496 [EventQueue-OpenPipelineForHealthyPipelineSafeModeRule] INFO  ha.SCMContext (SCMContext.java:updateSafeModeStatus(230)) - Update SafeModeStatus from SafeModeStatus{safeModeStatus=true, preCheckPassed=true} to SafeModeStatus{safeModeStatus=false, preCheckPassed=true}.
   //...
   2023-12-28 10:42:20,497 [EventQueue-OpenPipelineForHealthyPipelineSafeModeRule] DEBUG ha.SCMServiceManager (SCMServiceManager.java:notifyStatusChanged(51)) - Notify service:SCMBlockDeletingService.
   //...
   2023-12-28 10:42:20,499 [EventQueue-PipelineReportForOneReplicaPipelineSafeModeRule] INFO  safemode.SCMSafeModeManager (SCMSafeModeManager.java:validateSafeModeExitRules(215)) - ScmSafeModeManager, all rules are successfully validated
   2023-12-28 10:42:20,499 [EventQueue-PipelineReportForOneReplicaPipelineSafeModeRule] INFO  safemode.SCMSafeModeManager (SCMSafeModeManager.java:exitSafeMode(244)) - SCM exiting safe mode.   <<--- Second
   2023-12-28 10:42:20,499 [EventQueue-PipelineReportForOneReplicaPipelineSafeModeRule] INFO  ha.SCMContext (SCMContext.java:updateSafeModeStatus(230)) - Update SafeModeStatus from SafeModeStatus{safeModeStatus=false, preCheckPassed=true} to SafeModeStatus{safeModeStatus=false, preCheckPassed=true}.
   //...
   2023-12-28 10:42:20,500 [EventQueue-PipelineReportForOneReplicaPipelineSafeModeRule] DEBUG ha.SCMServiceManager (SCMServiceManager.java:notifyStatusChanged(51)) - Notify service:SCMBlockDeletingService.
   //...
   ```
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-9962
   
   ## How was this patch tested?
   
   Existing Test.
   Twice 25 * 15 Tests all successful
   https://github.com/xichen01/ozone/actions/runs/7353980620/attempts/1
   https://github.com/xichen01/ozone/actions/runs/7353980620
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9962. Intermittent timeout in TestBlockDeletion.testBlockDeletion [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #5886:
URL: https://github.com/apache/ozone/pull/5886


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org