[jira] [Commented] (RATIS-1312) Compare the performance between HDFS and DataStreamApi

["runzhiwang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RATIS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17280482#comment-17280482 ] 

runzhiwang commented on RATIS-1312:
-----------------------------------

[~szetszwo] Hi, let's discuss here.

I have test the performance of HDFS and Ratis streaming. But Ratis streaming is about 30% slower than HDFS. When 2 clients * 128MB * 200 files, Ratis streaming cost about 80 seconds, HDFS cost about 60 seconds. Not sure why.

In my test, as the following code shows, I did not call startTransaction to avoid submitting raft request, so that we can only compare the performance of network and write disk.

I have attached the flame graph of hdfs and ratis streaming.

composeAsync(info.getPrevious(), requestExecutor, n -> JavaUtils.allOf(remoteWrites)
        .thenCombineAsync(localWrite, (v, bytesWritten) -> {
          if (request.getType() == Type.STREAM_HEADER
              || (request.getType() == Type.STREAM_DATA && !close)) {
            sendReply(remoteWrites, request, bytesWritten, ctx);
          } else if (close) {
            sendReply(remoteWrites, request, bytesWritten, ctx);
  //            if (info.isPrimary()) {
  //              // after all server close stream, primary server start transaction
  //              startTransaction(info, request, bytesWritten, ctx);
  //            } else {
 //              sendReply(remoteWrites, request, bytesWritten, ctx);
 //            }
          } else {
            throw new IllegalStateException(this + ": Unexpected type " + request.getType() + ", request=" + request);
          }
          return null;
        }, requestExecutor)).whenComplete((v, exception) -> {

> Compare the performance between HDFS and DataStreamApi
> ------------------------------------------------------
>
>                 Key: RATIS-1312
>                 URL: https://issues.apache.org/jira/browse/RATIS-1312
>             Project: Ratis
>          Issue Type: Sub-task
>            Reporter: runzhiwang
>            Priority: Major
>         Attachments: hdfs.svg, streaming.svg
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)