[jira] [Comment Edited] (KUDU-2305) Local variables can overflow when serializing a 2GB message

["Michael Ho (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393584#comment-16393584 ] 

Michael Ho edited comment on KUDU-2305 at 3/9/18 11:19 PM:
-----------------------------------------------------------

It appears that there are still few places in the code which may overflow if the combined size of the entire message exceeds 2GB.

{{addition_size}} in {{SerializeMessage()}} is a 32-bit signed integer. However, the caller in {{OutboundCall::SetRequestPayload()}} may be passing a value larger than UINT_MAX. In which case, we will be truncating the value., leading to incorrect {{total_size}} being computed in {{SerializeMessage()}}

{noformat}
void SerializeMessage(const MessageLite& message, faststring* param_buf, int additional_size, bool use_cached_size) {
{noformat}

Along the same line, even when the {{total_size}} is larger than {{FLAGS_max_message_size}}, it appears that the client may keep sending the payload anyway and the receiver will reject it in {{InboundTransfer::ReceiveBuffer(}} path. It's possible that the sender may try to send the entire message before the receiver gets a chance to reject it and close the connection. In that sense, it's possible for {{total_size}} of a message to be larger than INT_MAX and some local variables in {{OutboundTransfer::SendBuffer}}, {{TlsSocket::Writev()}}, {{Socket::Writev()}} may overflow too. So, we may need to fix those places too.


was (Author: kwho):
It appears that there are still few places in the code which may overflow if the combined size of the entire message exceeds 2GB.

{{addition_size}} in {{SerializeMessage()}} is a 32-bit signed integer. However, the caller in {{OutboundCall::SetRequestPayload()}} may be passing a value larger than INT_MAX. In which case, we will be truncating the value., leading to incorrect {{total_size}} being computed in {{SerializeMessage()}}

{noformat}
void SerializeMessage(const MessageLite& message, faststring* param_buf, int additional_size, bool use_cached_size) {
{noformat}

Along the same line, even when the {{total_size}} is larger than {{FLAGS_max_message_size}}, it appears that the client may keep sending the payload anyway and the receiver will reject it in {{InboundTransfer::ReceiveBuffer(}} path. It's possible that the sender may try to send the entire message before the receiver gets a chance to reject it and close the connection. In that sense, it's possible for {{total_size}} of a message to be larger than INT_MAX and some local variables in {{OutboundTransfer::SendBuffer}}, {{TlsSocket::Writev()}}, {{Socket::Writev()}} may overflow too. So, we may need to fix those places too.

> Local variables can overflow when serializing a 2GB message
> -----------------------------------------------------------
>
>                 Key: KUDU-2305
>                 URL: https://issues.apache.org/jira/browse/KUDU-2305
>             Project: Kudu
>          Issue Type: Bug
>          Components: rpc
>    Affects Versions: 1.6.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Major
>             Fix For: 1.7.0
>
>
> When rpc_max_message_size is set to its maximum of INT_MAX (2147483647), certain local variables in SerializeMessage can overflow as messages approach this size. Specifically, recorded_size, size_with_delim, and total_size are 4 byte signed integers and could overflow when additional_size becomes large.
> Since INT_MAX is the largest allowable value for rpc_max_message_size (a 4 byte signed integer), these variables will not overflow if changed to 4 byte unsigned integers. This would eliminate the potential problem for serialization.
> A similar problem exists in the InboundTransfer::ReceiveBuffer() and similar codepaths. Changing those variables to unsigned integers should resolve the issue.
> This does not impact existing systems, because the default value of rpc_max_message_size is 50MB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)