You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Aravindan Vijayan (Jira)" <ji...@apache.org> on 2020/06/08 21:09:00 UTC
[jira] [Commented] (HDDS-3126) DBUpdateResponse message could be
much larger than ipc.maximum.data.length
[ https://issues.apache.org/jira/browse/HDDS-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128655#comment-17128655 ]
Aravindan Vijayan commented on HDDS-3126:
-----------------------------------------
If the RPC response size exceeds max length, Recon switches to get full snapshot instead. After getting the full snapshot, the existing requests in the OM RocksDB WAL will not be added to the response thereafter. Fixing this on RPC layer would mean a more complex protocol that keeps tracks of offsets. Since the Recon-OM data transfer contract is able to recover from this, closing as WAD.
> DBUpdateResponse message could be much larger than ipc.maximum.data.length
> --------------------------------------------------------------------------
>
> Key: HDDS-3126
> URL: https://issues.apache.org/jira/browse/HDDS-3126
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Recon
> Reporter: Xiaoyu Yao
> Assignee: Aravindan Vijayan
> Priority: Major
> Labels: TriagePending
>
> HDDS-1391 introduce a new OM RPC to allow Recon server to get delta of OM metadata update. However, the delta itself could be large. This causes ERROR on OM like below.
> Should we consider sending update in chunks over hadoop RPC instead of all in one piece (1.5 GB in this case)?
> 4:34:56.403 PM WARN OzoneManagerProtocolServerSideTranslatorPB
> ##Response for request DBUpdates is too big size 1584040343
> 4:34:57.022 PM WARN Server
> Error serializing call response for call Call#12 Retry#15 org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest from 10.17.112.109:58674
> com.google.protobuf.CodedOutputStream$OutOfSpaceException: CodedOutputStream was writing to a flat byte array and ran out of space.
> at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:828)
> at com.google.protobuf.CodedOutputStream.writeRawBytes(CodedOutputStream.java:959)
> at com.google.protobuf.CodedOutputStream.writeRawBytes(CodedOutputStream.java:905)
> at com.google.protobuf.CodedOutputStream.writeBytesNoTag(CodedOutputStream.java:386)
> at com.google.protobuf.CodedOutputStream.writeBytes(CodedOutputStream.java:229)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$DBUpdatesResponse.writeTo(OzoneManagerProtocolProtos.java)
> at com.google.protobuf.CodedOutputStream.writeMessageNoTag(CodedOutputStream.java:380)
> at com.google.protobuf.CodedOutputStream.writeMessage(CodedOutputStream.java:222)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMResponse.writeTo(OzoneManagerProtocolProtos.java:15959)
> at org.apache.hadoop.ipc.Server.setupResponseForProtobuf(Server.java:3216)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:3165)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:3141)
> at org.apache.hadoop.ipc.Server.access$200(Server.java:139)
> at org.apache.hadoop.ipc.Server$RpcCall.doResponse(Server.java:1061)
> at org.apache.hadoop.ipc.Server$Call.doResponse(Server.java:858)
> at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:844)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1001)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:912)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org