[jira] [Created] (HDDS-5384) OM refreshPipeline should not invoke the expensive OmKeyLocationInfoGroup.getLocationList()

["Wei-Chiu Chuang (Jira)" <ji...@apache.org>]

Wei-Chiu Chuang created HDDS-5384:
-------------------------------------

             Summary: OM refreshPipeline should not invoke the expensive OmKeyLocationInfoGroup.getLocationList()
                 Key: HDDS-5384
                 URL: https://issues.apache.org/jira/browse/HDDS-5384
             Project: Apache Ozone
          Issue Type: Improvement
          Components: OM
            Reporter: Wei-Chiu Chuang
         Attachments: om_listatus_alloc_after.svg, om_liststatus_alloc_before.svg, om_liststatus_cpu_after.svg, om_liststatus_cpu_before.svg

The OM's refreshPipeline (used by liststatus) implementation iterates over OmKeyLocationInfoGroup.getLocationList(), which is a very expensive call. It iterate over a collection of list of objects, allocates a new list, perform operations on each of them. In short, it's an O(n) method in terms of space and time complexity.

There are many places in the Ozone code that uses this method. Most usages iterates over the generated list, without modifying the list. We should instead return the collection of lists, which is O(1).

I have a client that issues many listStatus calls to examine the effect. Before the change, refreshPipeline costs 8.65% of heap usage. After: 1.95%.
CPU cost: before: 8.18% after: 4.22%

We should refrain from invoking getLocationList() as much as possible. But given the wide usage in the code, I elect not to remove the usage completely to avoid destabilizing it. Instead, I changed the usage in refreshPipeline to demonstrate its impact.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org