You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Tanvi Penumudy (Jira)" <ji...@apache.org> on 2023/09/22 13:43:00 UTC

[jira] [Created] (HDDS-9343) Shift sortDatanodes logic to OM

Tanvi Penumudy created HDDS-9343:
------------------------------------

             Summary: Shift sortDatanodes logic to OM
                 Key: HDDS-9343
                 URL: https://issues.apache.org/jira/browse/HDDS-9343
             Project: Apache Ozone
          Issue Type: Sub-task
            Reporter: Tanvi Penumudy
            Assignee: Tanvi Penumudy


Motivation behind this change: 
 * When OM has to call SCM then the performance of the objects on the read-path can affect SCM’s scaling requirements, so SCM needs to perform at the same level as that of OM for the read-path - harming the performance of OM by going to SCM for every read. 
 * Instead, a more efficient approach would be to perform sorting within OM itself, eliminating the need to rely on SCM for every read.

Steps how this can be achieved:

I. Add API in SCM:
 * The current sorting logic of datanodes in the container pipelines is present in SCM (SCMBlockProtocolServer).
 * SCM holds a configuration regarding the layout of the datanodes (similar to that of Hadoop) as part of an .xml file (NodeSchemaManager#init) - Configuration key: ozone.scm.network.topology.schema.file, value: network-topology-default.xml
 * The first step would be to add an API for SCM to serve this .xml file as part of an RPC call to OM.
 * We can do this such that OM has enough information to take the client location and the layout information for sorting the datanodes.

 

II. OM Cache and Refresh:
 * OM will need to cache the layout information and periodically refresh it.
 * The periodic refreshes would involve refetching the updated layout information from SCM.


III. Refactor NetworkTopology + Sort:
 * The NetworkTopology calculation can be refactored and moved to a common location so that both OM and SCM can utilize it.
 * On top of the NetworkTopology object obtained, NetworkTopology#sortByDistanceCost could be called.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org