com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.

[Bernard Quizon <be...@cheetahdigital.com>]

Hi.

We're using Hive 3.0.1 and we're currently experiencing this issue:




















































*Error while processing statement: FAILED: Execution Error, return code 2
from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
vertexName=Map 1, vertexId=vertex_1581309524541_0094_14_00,
diagnostics=[Vertex vertex_1581309524541_0094_14_00 [Map 1] killed/failed
due to:INIT_FAILURE, Fail to create InputInitializerManager,
org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class
with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator at
org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)
at
org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4122)
at
org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:207)
at
org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2932)
at
org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2879)
at
org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2861)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at
org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1957)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:206)
at
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2317)
at
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2303)
at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)
at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115) at
java.lang.Thread.run(Thread.java:745)Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
... 25 moreCaused by: com.google.protobuf.InvalidProtocolBufferException:
Protocol message was too large.  May be malicious.  Use
CodedInputStream.setSizeLimit() to increase the size limit. at
com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
at
com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
at
org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19294)
at
org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19258)
at
org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19360)
at
org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19355)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217) at
com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223) at
com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at
org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.parseFrom(DAGProtos.java:19552)
at
org.apache.tez.common.TezUtils.createConfFromByteString(TezUtils.java:116)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:130)
... 30 more]Vertex killed, vertexName=Reducer 2,
vertexId=vertex_1581309524541_0094_14_01, diagnostics=[Vertex received Kill
in NEW state., Vertex vertex_1581309524541_0094_14_01 [Reducer 2]
killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to
VERTEX_FAILURE. failedVertices:1 killedVertices:1*

Our tables are in ORC format and we probably have 1k Hive table in that
cluster.
I see these issues below but I don't think any of them are related to what
we're experiencing.
https://issues.apache.org/jira/browse/HIVE-11592
https://issues.apache.org/jira/browse/HIVE-11268

I see that the *sizeLimit* for CodedInputStream is set to
*PROTOBUF_MESSAGE_MAX_LIMIT* = *1073741824*.
Is there any workaround for this? And would guys know how we can lower the
protobuf message size per request?

BTW, we've tried deleting the contents of TAB_COL_STATS but it didn't help.

Thanks,
Bernard