You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2009/04/27 22:03:31 UTC

[jira] Created: (AVRO-24) benchmark bulk data

benchmark bulk data
-------------------

                 Key: AVRO-24
                 URL: https://issues.apache.org/jira/browse/AVRO-24
             Project: Avro
          Issue Type: Task
          Components: java
            Reporter: Doug Cutting
            Assignee: Doug Cutting
             Fix For: 1.0


It would be good to validate that the RPC wire format is capable of transmitting bulk data efficiently.  In particular, to be used for HDFS file access, it must be able to, when including file data in an RPC response, or writing file data in an RPC request:
 - saturate a disk's throughput or a network interface; and
 - not consume much CPU.
In other words, Avro's RPC should not be a bottleneck in the transfer of file data from a remote disk to an application or vice versa, and moreover it should leave the vast majority of the CPU for the application.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (AVRO-24) benchmark bulk data

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AVRO-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724603#action_12724603 ] 

Doug Cutting commented on AVRO-24:
----------------------------------

> Reading 64KB frames might be enough to mask 1ms latency between frames. Otherwise it might need pipelining of multiple frames.

I agree.  I hope 64k frames will mean we don't need to add an async RPC API.  Note however that the wire format should not change with async: we'll tag requests and responses with a call ID anyway so that we can multiplex over a single connection.

> do multiple simultaneous transfers use different connections?

The plan is to make Hadoop's Client and Server implement Avro's Transciever interface, so the transport will be the same as Hadoop's current RPC transport.  This caches a single connection per host, so simultaneous transfers from or to a host share a connection.  We'll see how this goes.

> Server side : Datanode : Is the disk data fetched inside RPC handler?

If we use transferTo, the disk would be accessed in the Server.Responder thread, otherwise it will be accessed in a Server.Handler thread (of which there are many).  So, if we implement transferTo, we'd probably need to change Server to support more responder threads too, but, if we don't, I think the existing handler pool will work well.  When there are more active requests than spindles, the server will slow down, as expected.  A slow spindle will mostly only affect requests to that spindle, since data will be fully buffered before the connection is accessed.

> benchmark bulk data
> -------------------
>
>                 Key: AVRO-24
>                 URL: https://issues.apache.org/jira/browse/AVRO-24
>             Project: Avro
>          Issue Type: Task
>          Components: java
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.0.0
>
>
> It would be good to validate that the RPC wire format is capable of transmitting bulk data efficiently.  In particular, to be used for HDFS file access, it must be able to, when including file data in an RPC response, or writing file data in an RPC request:
>  - saturate a disk's throughput or a network interface; and
>  - not consume much CPU.
> In other words, Avro's RPC should not be a bottleneck in the transfer of file data from a remote disk to an application or vice versa, and moreover it should leave the vast majority of the CPU for the application.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (AVRO-24) benchmark bulk data

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AVRO-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724203#action_12724203 ] 

Raghu Angadi commented on AVRO-24:
----------------------------------


Are bulk transfers already part of the spec? I am interested in learning more about if and how they differ from non-bulk RPCs. 

Sometime back I briefly looked at Avro RPC server and noticed that it already avoids an extra copy compared to Hadoop while replying.

> benchmark bulk data
> -------------------
>
>                 Key: AVRO-24
>                 URL: https://issues.apache.org/jira/browse/AVRO-24
>             Project: Avro
>          Issue Type: Task
>          Components: java
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.0.0
>
>
> It would be good to validate that the RPC wire format is capable of transmitting bulk data efficiently.  In particular, to be used for HDFS file access, it must be able to, when including file data in an RPC response, or writing file data in an RPC request:
>  - saturate a disk's throughput or a network interface; and
>  - not consume much CPU.
> In other words, Avro's RPC should not be a bottleneck in the transfer of file data from a remote disk to an application or vice versa, and moreover it should leave the vast majority of the CPU for the application.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (AVRO-24) benchmark bulk data

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AVRO-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724369#action_12724369 ] 

Raghu Angadi commented on AVRO-24:
----------------------------------


Thanks Doug. I agree benchmarking first would be the way to go. 

I would not be much concerned about minimizing the copies (at least initially)... cpu usuallly is not a bottleneck. When Datanode's CPU usage while serving data went down nearly 10 times between 0.16 and 0.18, hardly anyone noticed (same, when it doubled between 0.13 and 0.14).

Some of the cases I am interested in with bulk transfers (not not be addressed here, benchmarks help in assessing tthese) :

   * How the streaming reads are handled. Depending on frame size, client-side might not need pipelining. Reading 64KB frames might be enough to mask 1ms latency between frames. Otherwise it might need pipelining of multiple frames.
   * Connection management : do multiple simultaneous transfers use different connections? Do normal RPCs also share it. Hadoop does not yet have a case with lot of bulk and non-bulk RPCs at the same time.
   * Server side : Datanode : Is the disk data fetched inside RPC handler? Since data is directly written to client socket, will a slow client hold the handler.
      ** If one slower disk inhibits serving data from other disks. Larger number of handlers can mask this.


   

> benchmark bulk data
> -------------------
>
>                 Key: AVRO-24
>                 URL: https://issues.apache.org/jira/browse/AVRO-24
>             Project: Avro
>          Issue Type: Task
>          Components: java
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.0.0
>
>
> It would be good to validate that the RPC wire format is capable of transmitting bulk data efficiently.  In particular, to be used for HDFS file access, it must be able to, when including file data in an RPC response, or writing file data in an RPC request:
>  - saturate a disk's throughput or a network interface; and
>  - not consume much CPU.
> In other words, Avro's RPC should not be a bottleneck in the transfer of file data from a remote disk to an application or vice versa, and moreover it should leave the vast majority of the CPU for the application.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (AVRO-24) benchmark bulk data

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/AVRO-24?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting resolved AVRO-24.
------------------------------

       Resolution: Fixed
    Fix Version/s: 1.2.0

I just committed a simple benchmark that uses the new HTTP transport.  This defines a protocol with two methods, one sends a ByteBuffer and one that receives a ByteBuffer.

Running it on my laptop, making 100k requests each of 100 bytes, reports the following:

{noformat}
java  -Dtest.count=100000 -Dtest.size=100 org.apache.avro.TestBulkData
READ
seconds = 18
requests/second = 5367
MB = 9
MB/second = 0
WRITE
seconds = 15
requests/second = 6655
MB = 9
MB/second = 0
{noformat}

Increasing the bytes-per-request to 100k slows the request rate:

{noformat}
java -Dtest.count=100000 -Dtest.size=100000 org.apache.avro.TestBulkData
READ
seconds = 40
requests/second = 2460
MB = 9536
MB/second = 234
WRITE
seconds = 61
requests/second = 1637
MB = 9536
MB/second = 156
{noformat}


> benchmark bulk data
> -------------------
>
>                 Key: AVRO-24
>                 URL: https://issues.apache.org/jira/browse/AVRO-24
>             Project: Avro
>          Issue Type: Task
>          Components: java
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.2.0
>
>
> It would be good to validate that the RPC wire format is capable of transmitting bulk data efficiently.  In particular, to be used for HDFS file access, it must be able to, when including file data in an RPC response, or writing file data in an RPC request:
>  - saturate a disk's throughput or a network interface; and
>  - not consume much CPU.
> In other words, Avro's RPC should not be a bottleneck in the transfer of file data from a remote disk to an application or vice versa, and moreover it should leave the vast majority of the CPU for the application.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (AVRO-24) benchmark bulk data

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/AVRO-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724252#action_12724252 ] 

Doug Cutting commented on AVRO-24:
----------------------------------

> Are bulk transfers already part of the spec?

The idea is that bulk transfers can be efficiently implemented by just using the 'bytes' type in parameter, field and/or return values.  When a large value of type bytes is transmitted, it generates a separate frame at the transport layer.  Clients can then read and write such large values without copying.  On write, if one passes a large ByteBuffer as a parameter, field or return value, a reference is passed down and it is written directly to the socket.  Similarly, on read, the ByteBuffer that's read from the socket is directly returned to the client as the value of the field, parameter or method.

http://people.apache.org/~cutting/avro/spec.html#Message+Framing

This is not yet perfect.  First, while Avro permits object reuse, its RPC framework does not.  So, if an RPC method returns a ByteBuffer, a new ByteBuffer will be allocated per call.  However we could easily add a pool here to address this.

Second, sendfile is not yet supported.  This would require using an alternate representation for values of type bytes.  One might define something like:

interface ByteChannelable {
  int write(WritableByteChannel c);
  int read(ReadableByteChannel c);
  byte[] bytes();
  void bytes(byte[]);
  ByteBuffer buffer();
}

Then one could implement a version of this that contains a FileChannel and a start and end position whose read and write methods would call transferFrom and transferTo.

We could switch to such a representation by default, instead of using ByteBuffer (which unfortunately cannot be extended).  Note that any Requestor and Responder can easily be extended to use a different DatumReader, so we would not have to make this the default.

But first, I thought we'd benchmark things without these changes to get a baseline.

> benchmark bulk data
> -------------------
>
>                 Key: AVRO-24
>                 URL: https://issues.apache.org/jira/browse/AVRO-24
>             Project: Avro
>          Issue Type: Task
>          Components: java
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.0.0
>
>
> It would be good to validate that the RPC wire format is capable of transmitting bulk data efficiently.  In particular, to be used for HDFS file access, it must be able to, when including file data in an RPC response, or writing file data in an RPC request:
>  - saturate a disk's throughput or a network interface; and
>  - not consume much CPU.
> In other words, Avro's RPC should not be a bottleneck in the transfer of file data from a remote disk to an application or vice versa, and moreover it should leave the vast majority of the CPU for the application.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.