You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "mingchao zhao (Jira)" <ji...@apache.org> on 2022/12/21 08:40:00 UTC
[jira] [Updated] (HDDS-3466) Improve filterViableNodes performance in pipeline creation
[ https://issues.apache.org/jira/browse/HDDS-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mingchao zhao updated HDDS-3466:
--------------------------------
Target Version/s: 1.4.0 (was: 1.3.0)
> Improve filterViableNodes performance in pipeline creation
> ----------------------------------------------------------
>
> Key: HDDS-3466
> URL: https://issues.apache.org/jira/browse/HDDS-3466
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM
> Affects Versions: 0.5.0
> Reporter: Li Cheng
> Priority: Major
> Labels: TriagePending
>
> Per [~sodonnell]'s investigation, pipeline creation may have performance issue once the load-sorting algorithm in HDDS-3139.
>
> This task is to track potential performance bottleneck caused by this sorting operation for pipeline creation in large scale cluster.
>
> I am a little concerned about the expense of forming the list of healthy nodes on large clusters. We have to do quite a lot of work to form a list and then only use 3 nodes from the list. Even the method {{currentPipelineCount()}} needs to do a few map lookups per node to get the current pipeline count. This is the case even before this change. Creating a pipeline on a large cluster would be expensive already, but this change probably makes it worse, due to the sort needed. I know it was me who suggested the sort.
> I think the code as it is will work OK upto about 1000 nodes, and then the performance will drop off as the number of nodes goes toward 10k.
> Eg here are some benchmarks I created using this test code, which is similar to what we are doing in {{filterViableNodes()}}:
>
> {{ public List<Object> sortingWithMap(BenchmarkState state) \{
> return state.otherList.stream()
> .map(o -> new Mock(o, state.rand.nextInt(20)))
> .filter(o -> o.getSize() <= 20)
> .sorted(Comparator.comparingInt(Mock::getSize))
> .map(o -> o.getObject())
> .collect(Collectors.toList());
> }}}
> The OPs per second for various list sizes are:
>
> {{Benchmark (listSize) Mode Cnt Score Error Units
> Sorting.sortingWithMap 100 thrpt 3 113948.345 ± 446.426 ops/s
> Sorting.sortingWithMap 1000 thrpt 3 9468.507 ± 894.138 ops/s
> Sorting.sortingWithMap 5000 thrpt 3 1931.612 ± 263.919 ops/s
> Sorting.sortingWithMap 10000 thrpt 3 970.745 ± 25.823 ops/s
> Sorting.sortingWithMap 100000 thrpt 3 87.684 ± 35.438 ops/s}}
> For a 1000 node cluster, with 10 pipelines per node, we would be looking at about 1 second to form all the piplines.
> For a 5k node cluster, it would be about 25 seconds
> For a 10k node cluster it would be 103 seconds, but even here, that would be at close to 1000 pipelines per second.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org