You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Henry Robinson (Code Review)" <ge...@cloudera.org> on 2017/02/03 01:24:01 UTC

[Impala-ASF-CR] IMPALA-4874: Increase the maximum KRPC message size to 4GB

Henry Robinson has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/5887

Change subject: IMPALA-4874: Increase the maximum KRPC message size to 4GB
......................................................................

IMPALA-4874: Increase the maximum KRPC message size to 4GB

KRPC restricts the maximum message size with --rpc_max_message_size. The
Protobuf API used to deserialize messages also has a maximum size. Both
can be raised, and should be as a stop-gap until Impala has mechanisms
to prevent all its RPC payloads getting too large.

Note that some statestore payloads may still be larger than the 4GB
max. IMPALA-4875 will handle that for the statestore case particularly.

TESTING
-------

* This allows test_very_large_string to pass with KRPC.
* Tests added to statestore-test and rpc-mgr-test for large payloads.

Change-Id: Idb97dd4604548c404278fe8094db2e419021a7c3
---
M be/src/kudu/rpc/serialization.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc_test.proto
M be/src/statestore/statestore-test.cc
6 files changed, 88 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/5887/1
-- 
To view, visit http://gerrit.cloudera.org:8080/5887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Idb97dd4604548c404278fe8094db2e419021a7c3
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <he...@cloudera.com>

[Impala-ASF-CR] IMPALA-4874: Increase the maximum KRPC message size to 4GB

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-4874: Increase the maximum KRPC message size to 4GB
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/5887/1/be/src/kudu/rpc/serialization.cc
File be/src/kudu/rpc/serialization.cc:

PS1, Line 119: buf.size(
Is it reasonable to DCHECK(BitUtil::IsNonNegative32Bit(buf.size()) here given the 4GB limit imposed ?


http://gerrit.cloudera.org:8080/#/c/5887/1/be/src/statestore/statestore-test.cc
File be/src/statestore/statestore-test.cc:

Line 49:   TestSubscriber(int id, RpcMgr* rpc_mgr, const TNetworkAddress& statestore_address, int32_t port)
nit: long line.


-- 
To view, visit http://gerrit.cloudera.org:8080/5887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Idb97dd4604548c404278fe8094db2e419021a7c3
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4874: Increase the maximum KRPC message size to 4GB

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.

Change subject: IMPALA-4874: Increase the maximum KRPC message size to 4GB
......................................................................


Patch Set 1:

Does the 4GB payload limit include sidecar already or is it set separately ?

-- 
To view, visit http://gerrit.cloudera.org:8080/5887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Idb97dd4604548c404278fe8094db2e419021a7c3
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4874: Increase the maximum KRPC message size to 4GB

Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has uploaded a new patch set (#2).

Change subject: IMPALA-4874: Increase the maximum KRPC message size to 4GB
......................................................................

IMPALA-4874: Increase the maximum KRPC message size to 4GB

KRPC restricts the maximum message size with --rpc_max_message_size. The
Protobuf API used to deserialize messages also has a maximum size. Both
can be raised, and should be as a stop-gap until Impala has mechanisms
to prevent all its RPC payloads getting too large.

Note that some statestore payloads may still be larger than the 4GB
max. IMPALA-4875 will handle that for the statestore case particularly.

TESTING
-------

* This allows test_very_large_string to pass with KRPC.
* Tests added to statestore-test and rpc-mgr-test for large payloads.

Change-Id: Idb97dd4604548c404278fe8094db2e419021a7c3
---
M be/src/kudu/rpc/serialization.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc_test.proto
M be/src/statestore/statestore-test.cc
6 files changed, 87 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/5887/2
-- 
To view, visit http://gerrit.cloudera.org:8080/5887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb97dd4604548c404278fe8094db2e419021a7c3
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>

[Impala-ASF-CR] IMPALA-4874: Increase the maximum KRPC message size to 4GB

Posted by "Henry Robinson (Code Review)" <ge...@cloudera.org>.
Henry Robinson has abandoned this change.

Change subject: IMPALA-4874: Increase the maximum KRPC message size to 4GB
......................................................................


Abandoned

-- 
To view, visit http://gerrit.cloudera.org:8080/5887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: abandon
Gerrit-Change-Id: Idb97dd4604548c404278fe8094db2e419021a7c3
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>