[jira] [Commented] (TEZ-3784) Submitting very large DAG throws com.google.protobuf.CodedInputStream exception

["Rajesh Balamohan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071820#comment-16071820 ] 

Rajesh Balamohan commented on TEZ-3784:
---------------------------------------

Appears that hive has hit this issue as well HIVE-11592.

> Submitting very large DAG throws com.google.protobuf.CodedInputStream exception
> -------------------------------------------------------------------------------
>
>                 Key: TEZ-3784
>                 URL: https://issues.apache.org/jira/browse/TEZ-3784
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>
> {noformat}
> Dag submit failed due to Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
>         at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
>         at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
>         at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
>         at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
>         at org.apache.tez.dag.api.records.DAGProtos$TezUserPayloadProto.<init>(DAGProtos.java:8282)
>         at org.apache.tez.dag.api.records.DAGProtos$TezUserPayloadProto.<init>(DAGProtos.java:8246)
>         at org.apache.tez.dag.api.records.DAGProtos$TezUserPayloadProto$1.parsePartialFrom(DAGProtos.java:8334)
>         at org.apache.tez.dag.api.records.DAGProtos$TezUserPayloadProto$1.parsePartialFrom(DAGProtos.java:8329)
>         at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
>         at org.apache.tez.dag.api.records.DAGProtos$TezEntityDescriptorProto.<init>(DAGProtos.java:7520)
>         at org.apache.tez.dag.api.records.DAGProtos$TezEntityDescriptorProto.<init>(DAGProtos.java:7462)
>         at org.apache.tez.dag.api.records.DAGProtos$TezEntityDescriptorProto$1.parsePartialFrom(DAGProtos.java:7563)
>         at org.apache.tez.dag.api.records.DAGProtos$TezEntityDescriptorProto$1.parsePartialFrom(DAGProtos.java:7558)
>         at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
>         at org.apache.tez.dag.api.records.DAGProtos$RootInputLeafOutputProto.<init>(DAGProtos.java:8854)
>         at org.apache.tez.dag.api.records.DAGProtos$RootInputLeafOutputProto.<init>(DAGProtos.java:8796)
>         at org.apache.tez.dag.api.records.DAGProtos$RootInputLeafOutputProto$1.parsePartialFrom(DAGProtos.java:8905)
>         at org.apache.tez.dag.api.records.DAGProtos$RootInputLeafOutputProto$1.parsePartialFrom(DAGProtos.java:8900)
>         at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
>         at org.apache.tez.dag.api.records.DAGProtos$VertexPlan.<init>(DAGProtos.java:11012)
>         at org.apache.tez.dag.api.records.DAGProtos$VertexPlan.<init>(DAGProtos.java:10893)
>         at org.apache.tez.dag.api.records.DAGProtos$VertexPlan$1.parsePartialFrom(DAGProtos.java:11107)
>         at org.apache.tez.dag.api.records.DAGProtos$VertexPlan$1.parsePartialFrom(DAGProtos.java:11102)
>         at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
>         at org.apache.tez.dag.api.records.DAGProtos$DAGPlan.<init>(DAGProtos.java:22507)
>         at org.apache.tez.dag.api.records.DAGProtos$DAGPlan.<init>(DAGProtos.java:22449)
>         at org.apache.tez.dag.api.records.DAGProtos$DAGPlan$1.parsePartialFrom(DAGProtos.java:22638)
>         at org.apache.tez.dag.api.records.DAGProtos$DAGPlan$1.parsePartialFrom(DAGProtos.java:22633)
>         at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
>         at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$SubmitDAGRequestProto.<init>(DAGClientAMProtocolRPC.java:4673)
>         at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$SubmitDAGRequestProto.<init>(DAGClientAMProtocolRPC.java:4620)
>         at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$SubmitDAGRequestProto$1.parsePartialFrom(DAGClientAMProtocolRPC.java:4729)
>         at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$SubmitDAGRequestProto$1.parsePartialFrom(DAGClientAMProtocolRPC.java:4724)
>         at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
>         at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
>         at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
>         at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
>         at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$SubmitDAGRequestProto.parseFrom(DAGClientAMProtocolRPC.java:4976)
>         at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:169)
>         at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7636)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> {noformat}
> Appears that protobuf used in tez has a default size of 64 MB. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)