You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2024/11/22 22:02:00 UTC

[jira] [Updated] (HDDS-11773) Frequent DataNode Ratis snapshotting

     [ https://issues.apache.org/jira/browse/HDDS-11773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wei-Chiu Chuang updated HDDS-11773:
-----------------------------------
    Description: 
On a cluster with heavy HBase workload (more than 1000 hsync/WriteChunk+PutBlock) request, it is observed Ratis is taking snapshots every 5-8 seconds.

 

It looks too aggressive and should be tuned to avoid too much overhead penalty.
{noformat}
2024-11-21 16:46:13,617 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4727280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280
2024-11-21 16:46:13,619 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4727280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280 took: 3 ms
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4677268
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4727280
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4717277 -> 4727280
2024-11-21 16:46:19,244 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4737280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4737280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280 took: 2 ms
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4687272
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4737280
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4727280 -> 4737280
2024-11-21 16:46:24,739 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4747283) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283
2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4747283) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283 took: 2 ms
2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4697275
2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4747283
2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4737280 -> 4747283 {noformat}
 

In the DataNode source code, it intends to set the default value of 
hdds.ratis.snapshot.threshold to be 100000 (100k),
[https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java#L132]

 
however, the actual value is 10000 (10k) [https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/resources/ozone-default.xml#L281] 
 

I suggest to update the actual value in ozone-default.xml to 100k. The default value is causing too much snapshottng for Hbase workloads. Because DataNode does thousands of transactions per second, it is snapshotting every few seconds.

 

hdds.container.ratis.statemachine.max.pending.apply-transactions may need an update as well.

 

  was:
On a cluster with heavy HBase workload (more than 1000 hsync/WriteChunk+PutBlock) request, it is observed Ratis is taking snapshots every 5-8 seconds.

 

It looks too aggressive and should be tuned to avoid too much overhead penalty.

 

cc: [~smeng] I suspect this is related to the small size per checksum causing client having to transmit longer metadata.

 
{noformat}
2024-11-21 16:46:13,617 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4727280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280
2024-11-21 16:46:13,619 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4727280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280 took: 3 ms
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4677268
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4727280
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4717277 -> 4727280
2024-11-21 16:46:19,244 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4737280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4737280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280 took: 2 ms
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4687272
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4737280
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4727280 -> 4737280
2024-11-21 16:46:24,739 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4747283) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283
2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4747283) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283 took: 2 ms
2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4697275
2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4747283
2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4737280 -> 4747283 {noformat}


> Frequent DataNode Ratis snapshotting
> ------------------------------------
>
>                 Key: HDDS-11773
>                 URL: https://issues.apache.org/jira/browse/HDDS-11773
>             Project: Apache Ozone
>          Issue Type: Task
>            Reporter: Wei-Chiu Chuang
>            Priority: Major
>
> On a cluster with heavy HBase workload (more than 1000 hsync/WriteChunk+PutBlock) request, it is observed Ratis is taking snapshots every 5-8 seconds.
>  
> It looks too aggressive and should be tuned to avoid too much overhead penalty.
> {noformat}
> 2024-11-21 16:46:13,617 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4727280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280
> 2024-11-21 16:46:13,619 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4727280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280 took: 3 ms
> 2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4677268
> 2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4727280
> 2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4717277 -> 4727280
> 2024-11-21 16:46:19,244 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4737280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280
> 2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4737280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280 took: 2 ms
> 2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4687272
> 2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4737280
> 2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4727280 -> 4737280
> 2024-11-21 16:46:24,739 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4747283) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283
> 2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4747283) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283 took: 2 ms
> 2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4697275
> 2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4747283
> 2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4737280 -> 4747283 {noformat}
>  
> In the DataNode source code, it intends to set the default value of 
> hdds.ratis.snapshot.threshold to be 100000 (100k),
> [https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java#L132]
>  
> however, the actual value is 10000 (10k) [https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/resources/ozone-default.xml#L281] 
>  
> I suggest to update the actual value in ozone-default.xml to 100k. The default value is causing too much snapshottng for Hbase workloads. Because DataNode does thousands of transactions per second, it is snapshotting every few seconds.
>  
> hdds.container.ratis.statemachine.max.pending.apply-transactions may need an update as well.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org