[jira] [Comment Edited] (KUDU-90) Add a header checksum to our RPC protocol

["Jun He (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15776991#comment-15776991 ] 

Jun He edited comment on KUDU-90 at 12/26/16 2:04 AM:
------------------------------------------------------

[~tlipcon] [~jdcryans]

I added 4-byte checksum into the RPC protocol. Checksum is inserted immediately after total_size. The new message framing will be

{quote}
  total_size: (32-bit big-endian integer)
    the size of the rest of the message, including 4-byte checksum, and not including this 4-byte header
{color:red}
  header_checksum: (32-bit big-endian integer)  CRC-32
    the checksum of header (excluding varint-prefixed header size field)
    -  CRC-32 computation in C++ uses `boost/crc.hpp`.
    - CRC-32 computation in Java is based on hadoop-common class, `org.apache.hadoop.util.PureJavaCrc32`
{color}

  header: varint-prefixed header protobuf
    - client->server messages use the RequestHeader protobuf
    - server->client messages use the ResponseHeader protobuf

...
{quote}

Note that this is a non-backward compatible change.

Another option is to add an optional field into header proto file and then we can save checksum in this field. But this will complicate the checksum computation and also will add 4-byte data in header object. Please let me know your comments.

Code review: https://gerrit.cloudera.org/#/c/5578/



was (Author: junhe):
[~tlipcon] [~jdcryans]

I added 4-byte checksum into the RPC protocol. Checksum is inserted immediately after total_size. The new message framing will be

{quote}
  total_size: (32-bit big-endian integer)
    the size of the rest of the message, not including this 4-byte header and also not including 4-byte checksum
{color:red}
  header_checksum: (32-bit big-endian integer)  CRC-32
    the checksum of header (excluding varint-prefixed header size field)
    -  CRC-32 computation in C++ uses `boost/crc.hpp`.
    - CRC-32 computation in Java is based on hadoop-common class, `org.apache.hadoop.util.PureJavaCrc32`
{color}

  header: varint-prefixed header protobuf
    - client->server messages use the RequestHeader protobuf
    - server->client messages use the ResponseHeader protobuf

...
{quote}

Note that this is a non-backward compatible change.

Another option is to add an optional field into header proto file and then we can save checksum in this field. But this will complicate the checksum computation and also will add 4-byte data in header object. Please let me know your comments.

Code review: https://gerrit.cloudera.org/#/c/5578/


> Add a header checksum to our RPC protocol
> -----------------------------------------
>
>                 Key: KUDU-90
>                 URL: https://issues.apache.org/jira/browse/KUDU-90
>             Project: Kudu
>          Issue Type: Bug
>          Components: rpc
>    Affects Versions: M4
>            Reporter: Jean-Daniel Cryans
>              Labels: newbie
>
> See the context for this here: http://gerrit.ent.cloudera.com:8080/?l=259#/c/1077/8/java/kudu-client/src/main/java/kudu/rpc/KuduRpc.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)