You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kylin.apache.org by Zhou Kang <zh...@outlook.com> on 2019/09/09 07:28:37 UTC

KYLIN-3973: InvalidProtocolBufferException

Hi, guys

I reproduce this problem: https://issues.apache.org/jira/browse/KYLIN-3973

My env is Kylin 2.5.2 + HBase 2.2. After reading the code of hbase, I found the problem but don’t know the best method to resolve it.

in hbase coprocessor rpc, data pass to hbase client by CodedInputStream.newInstance(). in hbase 0.98 the input is byte[], and hbase 2.2 is InputStream. When protobuf read the input data, byte[] will not trigger the size check in function refillBuffer. But in hbase 2.2, the inputStream will refill many times and finnaly hit the limit of sizeLimit(DEFAULT_SIZE_LIMIT). So the solution is enlarge this value.

Extra, DEFAULT_SIZE_LIMIT is 64M in protobuf 2.5.0, which is changed to Integer.MAX_VALUE in protobuf3.


A simple way to fix bug is useing input.setSizeLimit() to modify this limit. But coprocessor/endpoint/generated/CubeVisitProtos.java is an auto generated file.

  Another way is changing com.google<http://com.google>.protobuf:2.5.0 to a shaded package. Some other lib(such as hbase) choose this way.


I need your suggestion ~


Thank you.