You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Tanvi Penumudy (Jira)" <ji...@apache.org> on 2024/03/25 07:09:00 UTC
[jira] [Updated] (HDDS-10586) Avoid loading network topology layer schema file for every read
[ https://issues.apache.org/jira/browse/HDDS-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tanvi Penumudy updated HDDS-10586:
----------------------------------
Description:
* After the change implemented in: [https://github.com/apache/ozone/pull/5391], the network topology layer schema file is now loaded for every key-level read operation, flooding the OM logs (please find the below logs for reference).
* This is because a new NetworkTopologyImpl object is being instantiated for each read. As part of this instantiation, NodeSchemaManager is initialized each time resulting in the loading of the schema file each time.
* To optimize this process, a new NetworkTopologyImpl object is now generated only when the fetched network topology cluster tree differs from the current cluster tree.
* This object generation is now bound by the rate specified by the ozone.om.network.topology.refresh.duration configuration which defaults to 1 hour.
* This ensures that the schema file is also loaded at the same rate, thereby reducing the overhead during read operations.
*Synthetic read workload:* ozone freon ockrw -n 10 --percentage-read 100 --percentage-list 0 -v vol1 -b buck1 -p sample
*OM logs:*
{code:java}
2024-03-25 11:25:57 2024-03-25 05:55:57,738 [IPC Server handler 97 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 95 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 40 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 40 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 95 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 71 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
{code}
was:
* After the change implemented in: [https://github.com/apache/ozone/pull/5391], the network topology layer schema file is now loaded for every key-level read operation.
* This is because a new NetworkTopologyImpl object is being instantiated for each read. As part of this instantiation, NodeSchemaManager is initialized each time resulting in the loading of the schema file each time.
* To optimize this process, a new NetworkTopologyImpl object is now generated only when the fetched network topology cluster tree differs from the current cluster tree.
* This object generation is now bound by the rate specified by the ozone.om.network.topology.refresh.duration configuration which defaults to 1 hour.
* This ensures that the schema file is also loaded at the same rate, thereby reducing the overhead during read operations.
{code:java}
2024-03-21 23:17:30,064 INFO [IPC Server handler 34 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-21 23:17:30,111 INFO [IPC Server handler 48 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from [jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,111 INFO [IPC Server handler 48 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-21 23:17:30,112 INFO [IPC Server handler 48 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from [jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,112 INFO [IPC Server handler 48 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-21 23:17:30,135 INFO [IPC Server handler 60 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from [jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,135 INFO [IPC Server handler 60 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-21 23:17:30,136 INFO [IPC Server handler 60 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from [jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,136 INFO [IPC Server handler 60 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-21 23:17:32,716 INFO [IPC Server handler 36 on 9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from [jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
{code}
> Avoid loading network topology layer schema file for every read
> ---------------------------------------------------------------
>
> Key: HDDS-10586
> URL: https://issues.apache.org/jira/browse/HDDS-10586
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: OM, SCM
> Reporter: Tanvi Penumudy
> Assignee: Tanvi Penumudy
> Priority: Major
>
> * After the change implemented in: [https://github.com/apache/ozone/pull/5391], the network topology layer schema file is now loaded for every key-level read operation, flooding the OM logs (please find the below logs for reference).
> * This is because a new NetworkTopologyImpl object is being instantiated for each read. As part of this instantiation, NodeSchemaManager is initialized each time resulting in the loading of the schema file each time.
> * To optimize this process, a new NetworkTopologyImpl object is now generated only when the fetched network topology cluster tree differs from the current cluster tree.
> * This object generation is now bound by the rate specified by the ozone.om.network.topology.refresh.duration configuration which defaults to 1 hour.
> * This ensures that the schema file is also loaded at the same rate, thereby reducing the overhead during read operations.
> *Synthetic read workload:* ozone freon ockrw -n 10 --percentage-read 100 --percentage-list 0 -v vol1 -b buck1 -p sample
> *OM logs:*
> {code:java}
> 2024-03-25 11:25:57 2024-03-25 05:55:57,738 [IPC Server handler 97 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 95 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 40 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 40 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 95 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 71 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org