You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Himanshu Vashishtha (JIRA)" <ji...@apache.org> on 2012/08/06 08:27:02 UTC
[jira] [Created] (HBASE-6515) Setting request size with protobuf
Himanshu Vashishtha created HBASE-6515:
------------------------------------------
Summary: Setting request size with protobuf
Key: HBASE-6515
URL: https://issues.apache.org/jira/browse/HBASE-6515
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Himanshu Vashishtha
While running replication on upstream code, I am hitting the size-limit exception while sending WALEdits to a different cluster.
{code}
com.google.protobuf.InvalidProtocolBufferException: IPC server unable to read call parameters: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
{code}
Do we have a property to set some max size or something?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6515) Setting request size with protobuf
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429360#comment-13429360 ]
Todd Lipcon commented on HBASE-6515:
------------------------------------
Did you look into what the default max size is?
I don't think we should arbitrarily raise the limit. Instead, if replication sends too-large RPCs, we should figure out how to make it do smaller batches to fit within the limit. RPC payloads in the 10s or 100s of MBs are not good.
> Setting request size with protobuf
> ----------------------------------
>
> Key: HBASE-6515
> URL: https://issues.apache.org/jira/browse/HBASE-6515
> Project: HBase
> Issue Type: Bug
> Components: ipc, replication
> Affects Versions: 0.96.0
> Reporter: Himanshu Vashishtha
> Priority: Critical
>
> While running replication on upstream code, I am hitting the size-limit exception while sending WALEdits to a different cluster.
> {code}
> com.google.protobuf.InvalidProtocolBufferException: IPC server unable to read call parameters: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
> {code}
> Do we have a property to set some max size or something?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6515) Setting request size with protobuf
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-6515:
----------------------------------
Component/s: replication
ipc
Priority: Critical (was: Major)
> Setting request size with protobuf
> ----------------------------------
>
> Key: HBASE-6515
> URL: https://issues.apache.org/jira/browse/HBASE-6515
> Project: HBase
> Issue Type: Bug
> Components: ipc, replication
> Affects Versions: 0.96.0
> Reporter: Himanshu Vashishtha
> Priority: Critical
>
> While running replication on upstream code, I am hitting the size-limit exception while sending WALEdits to a different cluster.
> {code}
> com.google.protobuf.InvalidProtocolBufferException: IPC server unable to read call parameters: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
> {code}
> Do we have a property to set some max size or something?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6515) Setting request size with protobuf
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430463#comment-13430463 ]
Himanshu Vashishtha commented on HBASE-6515:
--------------------------------------------
One can set batch size of the WALEdit shipment by "replication.source.size.capacity" property, with its default being 64MB.
It looks like we should add a note that the maximum batchsize should be < 64MB, and change the default to a lower value, ...8MB?
> Setting request size with protobuf
> ----------------------------------
>
> Key: HBASE-6515
> URL: https://issues.apache.org/jira/browse/HBASE-6515
> Project: HBase
> Issue Type: Bug
> Components: ipc, replication
> Affects Versions: 0.96.0
> Reporter: Himanshu Vashishtha
> Priority: Critical
>
> While running replication on upstream code, I am hitting the size-limit exception while sending WALEdits to a different cluster.
> {code}
> com.google.protobuf.InvalidProtocolBufferException: IPC server unable to read call parameters: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
> {code}
> Do we have a property to set some max size or something?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6515) Setting request size with protobuf
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429661#comment-13429661 ]
Himanshu Vashishtha commented on HBASE-6515:
--------------------------------------------
Not yet, but yes, will test in that direction.
While using ycsb, I see it does support 6MB waledit object.
HTable's write buffer can also play interesting role here; it creates one WALEdit object per table flush. Setting it to a higher value may cause this exception. Given that current rpc can handle 64 MB arrays, this is something to look for.
I will report back the default limit soon.
> Setting request size with protobuf
> ----------------------------------
>
> Key: HBASE-6515
> URL: https://issues.apache.org/jira/browse/HBASE-6515
> Project: HBase
> Issue Type: Bug
> Components: ipc, replication
> Affects Versions: 0.96.0
> Reporter: Himanshu Vashishtha
> Priority: Critical
>
> While running replication on upstream code, I am hitting the size-limit exception while sending WALEdits to a different cluster.
> {code}
> com.google.protobuf.InvalidProtocolBufferException: IPC server unable to read call parameters: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
> {code}
> Do we have a property to set some max size or something?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6515) Setting request size with protobuf
Posted by "Himanshu Vashishtha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430421#comment-13430421 ]
Himanshu Vashishtha commented on HBASE-6515:
--------------------------------------------
The default max value is 64MB.
> Setting request size with protobuf
> ----------------------------------
>
> Key: HBASE-6515
> URL: https://issues.apache.org/jira/browse/HBASE-6515
> Project: HBase
> Issue Type: Bug
> Components: ipc, replication
> Affects Versions: 0.96.0
> Reporter: Himanshu Vashishtha
> Priority: Critical
>
> While running replication on upstream code, I am hitting the size-limit exception while sending WALEdits to a different cluster.
> {code}
> com.google.protobuf.InvalidProtocolBufferException: IPC server unable to read call parameters: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
> {code}
> Do we have a property to set some max size or something?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira