[jira] [Commented] (HDDS-15802) Add integration/system tests to certify per-volume push replication thread pools on DataNode (HDDS-15412).

["Wei-Chiu Chuang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-15802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095407#comment-18095407 ] 

Wei-Chiu Chuang commented on HDDS-15802:
----------------------------------------

Update – this PR will focus on P0 items.

 

P1 items goes to HDDS-15815

P2 items goes to HDDS-15816

> Add integration/system tests to certify per-volume push replication thread pools on DataNode (HDDS-15412).
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-15802
>                 URL: https://issues.apache.org/jira/browse/HDDS-15802
>             Project: Apache Ozone
>          Issue Type: Sub-task
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>            Priority: Major
>              Labels: pull-request-available
>
> h2. Background
> [HDDS-15412|https://github.com/apache/ozone/pull/10693] adds optional per-volume push replication thread pools on the DataNode:
>  * {{hdds.datanode.replication.per.volume.enabled}} (default: {{{}false{}}})
>  * {{hdds.datanode.replication.per.volume.streams.limit}} (default: {{{}1{}}}, reconfigurable)
> When enabled, push replication is routed to a volume-specific executor so slow replication on one disk does not block replication on other disks. Pull replication, EC reconstruction, and other tasks continue using the global replication pool. Failed volumes shut down their per-volume pool via {{{}shutdownFailedVolumePools(){}}}.
> Unit tests in {{TestReplicationSupervisor}} cover init, routing, isolation, resize, and volume-failure shutdown. Cluster-level certification is still missing.
> h2. Motivation
> Operators enable this feature primarily during decommission/maintenance on multi-volume datanodes. We need integration tests that prove:
>  # Basic replication still works with the feature enabled.
>  # The feature delivers its intended isolation benefit under realistic cluster load.
>  # The datanode recovers correctly from volume failure and SCM failover while replication is in flight.
> Without these tests, we rely on unit coverage and manual validation before enabling the feature in production.
> h2. Scope
> Add integration tests under {{{}hadoop-ozone/integration-test{}}}, preferably in a new class:
> {{org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication}}
> Extend existing suites where appropriate:
>  * {{TestDatanodeReconfiguration}} — per-volume pool resize via reconfig
>  * Optionally {{TestDatanodeHddsVolumeFailureDetection}} / volume-failure patterns
> h3. In scope
> ||Priority||Test||Description||
> |P0|Push/pull regression with per-volume enabled|Multi-volume DN ({{{}setNumDataVolumes(2){}}}), feature ON. Verify existing push/pull replication from {{TestContainerReplication}} still succeeds. Assert volume pools are created (count matches DN volume count).|
> |P0|Volume failure: healthy-volume replication continues|Fail one volume on a multi-volume DN while per-volume pools are enabled. Assert failed-volume pool is removed; push replication for containers on healthy volumes still succeeds; DN remains within toleration limits.|
> |P0|Decommission E2E with per-volume pools|Multi-volume DN with Ratis and/or EC data; feature ON with {{{}streams.limit=1{}}}. Decommission the DN. Assert node reaches {{{}DECOMMISSIONED{}}}, under-replicated containers recover, and cluster remains healthy.|
> |P1|Runtime reconfiguration of per-volume pool size|Extend {{{}TestDatanodeReconfiguration{}}}: assert {{hdds.datanode.replication.per.volume.streams.limit}} is reconfigurable; reconfigure (e.g. 1 → 3) and verify all volume pool sizes update.|
> |P1|Cross-volume push isolation under concurrent load|Containers on volume A and B on same source DN; {{{}streams.limit=1{}}}. Slow push on A should not block push completion on B.|
> |P1|SCM failover during active replication|SCM HA cluster (3 SCMs), per-volume enabled, under-replication in progress (decommission or DN stop). {{transferLeadership()}} mid-replication. Assert containers eventually recover, DN {{getTotalInFlightReplications()}} returns to 0, and volume pools remain functional.|
> |P2|Default-off backward compatibility|Same regression tests with {{{}per.volume.enabled=false{}}}; assert {{getVolumeReplicationThreadPools()}} is null and behavior matches pre-feature baseline.|
> |P2|Pull replication uses global pool|With feature ON, pull replication succeeds; optional log/assertion that pull tasks are not routed to per-volume executors.|
> |P2|Decommission/maintenance pool scaling|DN enters {{{}DECOMMISSIONING{}}}; assert per-volume pool size scales per {{outofservice.limit.factor}} (mirrors unit test {{{}perVolumePoolResizeOnNodeStateChange{}}}).|
> h3. Out of scope
>  * Re-testing unit-level behavior already covered in {{TestReplicationSupervisor}} (init logging, config validation, non-push routing).
>  * SCM-side reconstruction counter behavior on failover (HDDS-15071).
>  * Toggling {{per.volume.enabled}} at runtime without restart (not reconfigurable).
>  * Acceptance/compose-cluster tests (can be a follow-up if mini-cluster coverage is insufficient).
>  * Performance/benchmark tests.
> h2. Acceptance criteria
>  *  New integration test class (or clearly scoped extensions to existing classes) covers all P0 scenarios.
>  *  At least one P1 scenario implemented (recommend: SCM failover or cross-volume isolation).
>  *  All new tests pass in CI with {{{}-DskipShade -DskipRecon -DskipDocs{}}}.
>  *  Tests use short SCM/DN report intervals consistent with existing replication integration tests.
>  *  No new third-party dependencies.
>  *  Checkstyle and RAT pass.
> h2. Implementation notes
>  * Use {{MiniOzoneCluster.newBuilder(conf).setNumDataVolumes(2)}} (see {{{}TestDatanodeHddsVolumeFailureToleration{}}}).
>  * For SCM failover, use {{MiniOzoneHAClusterImpl}} and {{transferLeadership()}} (see {{{}TestFailoverWithSCMHA{}}}, {{{}TestDatanodeSCMNodesReconfiguration{}}}).
>  * Access supervisor via {{{}datanode.getDatanodeStateMachine().getSupervisor(){}}}.
>  * Pool assertions via {{supervisor.getVolumeReplicationThreadPools()}} and {{{}getPoolSize(volumeRoot){}}}.
>  * Reuse helpers from {{TestContainerReplication}} for direct push/pull command injection where appropriate.
>  * For decommission E2E, follow patterns from {{TestDecommissionAndMaintenance}} / {{{}TestReplicationManagerIntegration{}}}.
>  * Consider schema V2/V3 parameterization if other DN integration tests do ({{{}@ContainerLayoutTestInfo{}}}).
> h2. Minimum viable delivery
> If timeboxed, land P0 only:
>  # Push/pull regression with feature ON
>  # Volume failure continuity
>  # Decommission E2E
> Add SCM failover as the first P1 follow-up.
> h2. References
>  * Implementation PR: [#10693|https://github.com/apache/ozone/pull/10693]
>  * Parent/feature JIRA: HDDS-15412
>  * Related design context: EC decommission reconstruction (per-volume DN pools are Phase 3 enabler)
>  * Existing integration patterns:
>  ** {{TestContainerReplication}}
>  ** {{TestDatanodeHddsVolumeFailureDetection}}
>  ** {{TestDecommissionAndMaintenance}}
>  ** {{TestFailoverWithSCMHA}}
>  ** {{TestDatanodeReconfiguration}}
> h2. Test plan (for PR description)
> mvn -pl :ozone-integration-test test \
> -Dtest=TestPerVolumePushReplication\
> -DskipShade-DskipRecon-DskipDocs
> Also run extended {{TestDatanodeReconfiguration}} if modified.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org