You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2024/11/22 22:02:00 UTC
[jira] [Updated] (HDDS-11773) Frequent DataNode Ratis snapshotting
[ https://issues.apache.org/jira/browse/HDDS-11773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HDDS-11773:
-----------------------------------
Description:
On a cluster with heavy HBase workload (more than 1000 hsync/WriteChunk+PutBlock) request, it is observed Ratis is taking snapshots every 5-8 seconds.
It looks too aggressive and should be tuned to avoid too much overhead penalty.
{noformat}
2024-11-21 16:46:13,617 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4727280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280
2024-11-21 16:46:13,619 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4727280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280 took: 3 ms
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4677268
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4727280
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4717277 -> 4727280
2024-11-21 16:46:19,244 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4737280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4737280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280 took: 2 ms
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4687272
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4737280
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4727280 -> 4737280
2024-11-21 16:46:24,739 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4747283) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283
2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4747283) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283 took: 2 ms
2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4697275
2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4747283
2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4737280 -> 4747283 {noformat}
In the DataNode source code, it intends to set the default value of
hdds.ratis.snapshot.threshold to be 100000 (100k),
[https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java#L132]
however, the actual value is 10000 (10k) [https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/resources/ozone-default.xml#L281]
I suggest to update the actual value in ozone-default.xml to 100k. The default value is causing too much snapshottng for Hbase workloads. Because DataNode does thousands of transactions per second, it is snapshotting every few seconds.
hdds.container.ratis.statemachine.max.pending.apply-transactions may need an update as well.
was:
On a cluster with heavy HBase workload (more than 1000 hsync/WriteChunk+PutBlock) request, it is observed Ratis is taking snapshots every 5-8 seconds.
It looks too aggressive and should be tuned to avoid too much overhead penalty.
cc: [~smeng] I suspect this is related to the small size per checksum causing client having to transmit longer metadata.
{noformat}
2024-11-21 16:46:13,617 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4727280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280
2024-11-21 16:46:13,619 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4727280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280 took: 3 ms
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4677268
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4727280
2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4717277 -> 4727280
2024-11-21 16:46:19,244 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4737280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4737280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280 took: 2 ms
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4687272
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4737280
2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4727280 -> 4737280
2024-11-21 16:46:24,739 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4747283) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283
2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4747283) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283 took: 2 ms
2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4697275
2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4747283
2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4737280 -> 4747283 {noformat}
> Frequent DataNode Ratis snapshotting
> ------------------------------------
>
> Key: HDDS-11773
> URL: https://issues.apache.org/jira/browse/HDDS-11773
> Project: Apache Ozone
> Issue Type: Task
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> On a cluster with heavy HBase workload (more than 1000 hsync/WriteChunk+PutBlock) request, it is observed Ratis is taking snapshots every 5-8 seconds.
>
> It looks too aggressive and should be tuned to avoid too much overhead penalty.
> {noformat}
> 2024-11-21 16:46:13,617 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4727280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280
> 2024-11-21 16:46:13,619 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4727280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4727280 took: 3 ms
> 2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4677268
> 2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4727280
> 2024-11-21 16:46:13,620 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4717277 -> 4727280
> 2024-11-21 16:46:19,244 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4737280) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280
> 2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4737280) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4737280 took: 2 ms
> 2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4687272
> 2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4737280
> 2024-11-21 16:46:19,246 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4727280 -> 4737280
> 2024-11-21 16:46:24,739 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:171, i:4747283) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283
> 2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:171, i:4747283) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4747283 took: 2 ms
> 2024-11-21 16:46:24,741 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.statemachine.impl.SimpleStateMachineStorage: Deleting old snapshot at /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/snapshot.171_4697275
> 2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: Took a snapshot at index 4747283
> 2024-11-21 16:46:24,742 INFO [7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater]-org.apache.ratis.server.impl.StateMachineUpdater: 7cc563b3-14b5-4334-820b-5c3bbecffad8@group-D595E2D0A206-StateMachineUpdater: snapshotIndex: updateIncreasingly 4737280 -> 4747283 {noformat}
>
> In the DataNode source code, it intends to set the default value of
> hdds.ratis.snapshot.threshold to be 100000 (100k),
> [https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java#L132]
>
> however, the actual value is 10000 (10k) [https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/resources/ozone-default.xml#L281]
>
> I suggest to update the actual value in ozone-default.xml to 100k. The default value is causing too much snapshottng for Hbase workloads. Because DataNode does thousands of transactions per second, it is snapshotting every few seconds.
>
> hdds.container.ratis.statemachine.max.pending.apply-transactions may need an update as well.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org