You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by interma <gi...@git.apache.org> on 2018/06/21 02:29:42 UTC
[GitHub] incubator-hawq pull request #1377: HAWQ-1627. Support setting the max protoc...
GitHub user interma opened a pull request:
https://github.com/apache/incubator-hawq/pull/1377
HAWQ-1627. Support setting the max protocol message size when talking
with HDFS
**Fix what?**
User can set the max protocol message size, e.g.
In _etc/hdfs-client.xml_:
```
<property>
<name>ipc.maximum.data.length</name>
<value>67108864</value>
</property>
```
**How to fix?**
Use `CodedInputStream.SetTotalBytesLimit()` set the max size.
**Test?**
No test, reason:
It can be covered by former tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/interma/interma-hawq ipc_maxdata
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1377.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1377
----
commit 8ac2663668e353078fff02f48debfd8223911e7c
Author: interma <in...@...>
Date: 2018-06-20T05:02:44Z
HAWQ-1627. Support setting the max protocol message size when talking
with HDFS
----
---
[GitHub] incubator-hawq issue #1377: HAWQ-1627. Support setting the max protocol mess...
Posted by linwen <gi...@git.apache.org>.
Github user linwen commented on the issue:
https://github.com/apache/incubator-hawq/pull/1377
+1
---
[GitHub] incubator-hawq issue #1377: HAWQ-1627. Support setting the max protocol mess...
Posted by interma <gi...@git.apache.org>.
Github user interma commented on the issue:
https://github.com/apache/incubator-hawq/pull/1377
Merged
---
[GitHub] incubator-hawq issue #1377: HAWQ-1627. Support setting the max protocol mess...
Posted by amyrazz44 <gi...@git.apache.org>.
Github user amyrazz44 commented on the issue:
https://github.com/apache/incubator-hawq/pull/1377
LGTM +1
---
[GitHub] incubator-hawq pull request #1377: HAWQ-1627. Support setting the max protoc...
Posted by interma <gi...@git.apache.org>.
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1377#discussion_r197324064
--- Diff: depends/libhdfs3/src/rpc/RpcChannel.cpp ---
@@ -768,7 +771,15 @@ void RpcChannelImpl::readOneResponse(bool writeLock) {
buffer.resize(headerSize);
in->readFully(&buffer[0], headerSize, readTimeout);
- if (!curRespHeader.ParseFromArray(&buffer[0], headerSize)) {
+ // use CodedInputStream around the buffer, so we can set TotalBytesLimit on it
+ ArrayInputStream ais(&buffer[0], headerSize);
+ CodedInputStream cis(&ais);
+ cis.SetTotalBytesLimit(maxLength, maxLength/2);
+
+ // use ParseFromCodedStream instead of ParseFromArray, so it can consume the above CodedInputStream
+ //
+ // if just use ParseFromArray, we have on chance to set TotalBytesLimit (64MB default)
--- End diff --
Thanks! I will fix.
---
[GitHub] incubator-hawq issue #1377: HAWQ-1627. Support setting the max protocol mess...
Posted by interma <gi...@git.apache.org>.
Github user interma commented on the issue:
https://github.com/apache/incubator-hawq/pull/1377
@amyrazz44 @wengyanqing help to review, thanks.
---
[GitHub] incubator-hawq pull request #1377: HAWQ-1627. Support setting the max protoc...
Posted by linwen <gi...@git.apache.org>.
Github user linwen commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1377#discussion_r196996343
--- Diff: depends/libhdfs3/src/rpc/RpcChannel.cpp ---
@@ -768,7 +771,15 @@ void RpcChannelImpl::readOneResponse(bool writeLock) {
buffer.resize(headerSize);
in->readFully(&buffer[0], headerSize, readTimeout);
- if (!curRespHeader.ParseFromArray(&buffer[0], headerSize)) {
+ // use CodedInputStream around the buffer, so we can set TotalBytesLimit on it
+ ArrayInputStream ais(&buffer[0], headerSize);
+ CodedInputStream cis(&ais);
+ cis.SetTotalBytesLimit(maxLength, maxLength/2);
+
+ // use ParseFromCodedStream instead of ParseFromArray, so it can consume the above CodedInputStream
+ //
+ // if just use ParseFromArray, we have on chance to set TotalBytesLimit (64MB default)
--- End diff --
"on" should be "no"?
---
[GitHub] incubator-hawq pull request #1377: HAWQ-1627. Support setting the max protoc...
Posted by interma <gi...@git.apache.org>.
Github user interma closed the pull request at:
https://github.com/apache/incubator-hawq/pull/1377
---