[kudu-CR] rpc: bump default max message size to 50MB

["Todd Lipcon (Code Review)" <ge...@cloudera.org>]

Hello Mike Percy,

I'd like you to do a code review.  Please visit

    http://gerrit.cloudera.org:8080/2889

to review the following change.

Change subject: rpc: bump default max message size to 50MB
......................................................................

rpc: bump default max message size to 50MB

It seems that the current default of only 8MB can be problematic
in a few cases:

* If tables contain larger cells, it's not hard to overflow the 8MB for scanner
  results.
* When writing, if users aren't very careful about managing batch sizes,
  it's also possible to overflow.

Of course, it's also possible to overflow 50MB, so in some cases this
commit just "delays the inevitable". But, I did a survey of other RPC
systems and found that 8MB is far lower than what those expect:

- GRPC max frame size is 100MB by default, and allows the user to increase
  it if necessary[1]
- Hadoop RPC defaults to 64MB and is configurable[2]. Responses larger than
  1MB will be logged by default. Note that HDFS does not use RPC for
  data transfer, so >1MB RPCs are unlikely.
- HBase RPC sets the maximum request to 256MB and warns for responses
  larger than 100MB.[3]

This commit also changes the remote bootstrap code path to aim for 4MB
chunks regardless of the max RPC message size, since it doesn't make
sense to do IO in 50MB units.

[1] https://github.com/grpc/grpc-java/issues/917
[2] https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java#L77
[3] https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java#L268

Change-Id: I1df399c5dd28ec9539fbd701d26ff2c7467991c8
---
M src/kudu/rpc/transfer.cc
M src/kudu/tserver/remote_bootstrap_client.cc
M src/kudu/tserver/remote_bootstrap_session.cc
3 files changed, 22 insertions(+), 17 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/89/2889/1
-- 
To view, visit http://gerrit.cloudera.org:8080/2889
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1df399c5dd28ec9539fbd701d26ff2c7467991c8
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>