[jira] [Comment Edited] (RATIS-1234) Compare the performance between Ozone and DataStreamApi

["runzhiwang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RATIS-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248532#comment-17248532 ] 

runzhiwang edited comment on RATIS-1234 at 12/13/20, 8:48 AM:
--------------------------------------------------------------

   [~szetszwo] I test it again. In my test, Ozone has 3 pipelines, each datanode has one leader. Both ozone and ratis streaming use 3 clients on 3 machines to write file. Network bandwidth is also 10000Mb/s. When start cluster, the jvm options is -Xms4000m -Xmx20000m -Xmn15000m.
   The workload is 3 clients * (200 files,400 files,600 files,800 files) * 128MB. 
   1. The time cost of ratis streaming increase linearly when increase the work load, but the time cost of ozone double every time. 
   2. The CPU cost are both high, about 500%. 
   3. Ozone use higher memory even when the work load is 3 clients * 200 files * 128MB.
   4. Ozone can not work normaly when work load is above 3 clients * 800 files * 128MB, but ratis streaming work normaly.
 !screenshot-1.png! 


was (Author: yjxxtd):
   [~szetszwo] I test it again. In my test, Ozone has 3 pipelines, each datanode has one leader. Both ozone and ratis streaming use 3 clients on 3 machines to write file. Network bandwidth is also 10000Mb/s. When start cluster, the jvm options is -Xms4000m -Xmx20000m -Xmn15000m.
   The workload is 3 clients * (200 files,400 files,600 files,800 files) * 128MB. 
   1. The time cost of ratis streaming increase linearly when increase the work load, but the time cost of ozone double every time. 
   2. The CPU cost are both high, about 500%. 
   3. Ozone use higher memory even when the work load is 3 clients * 200 files * 128MB.
 !screenshot-1.png! 

> Compare the performance between Ozone and DataStreamApi
> -------------------------------------------------------
>
>                 Key: RATIS-1234
>                 URL: https://issues.apache.org/jira/browse/RATIS-1234
>             Project: Ratis
>          Issue Type: Sub-task
>            Reporter: runzhiwang
>            Priority: Major
>         Attachments: screenshot-1.png
>
>
> [~szetszwo] Hi, my network bandwidth is only 10000Mb / s, because primary need to stream data to two peers, the max IO speed for peer is only 5000Mb / s, about 625MB / s. So I can not test the max throughput of ratis streaming and ozone cluster.
> Besides, ozone has the balance leader feature, for example, if ozone cluster has 6 pipeline, 3 datanodes, each datanode has 2 leader, the 10000Mb /s can be used fully for each datanode. So with this 10000Mb / s network bandwidth, we can not compare ozone and DataStreamApi, unless we change DataStreamApi to primary -> peer -> peer, or we need machine with bigger network bandwidth, but I do not have.
> Actually, I test the performance of ozone and DataStreamApi with high load, they are almost similar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)