You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Kyle Purtell (Jira)" <ji...@apache.org> on 2022/07/01 02:05:00 UTC
[jira] [Comment Edited] (HBASE-27097) SimpleRpcServer is broken
[ https://issues.apache.org/jira/browse/HBASE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561231#comment-17561231 ]
Andrew Kyle Purtell edited comment on HBASE-27097 at 7/1/22 2:04 AM:
---------------------------------------------------------------------
[~vjasani] This is holding up 2.5.0 so I wanted to determine if the issue is possibly environmental.
Built HBase 2.5.1-SNAPSHOT with -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.3
I set up a VM... Configured BIND to serve DNS. Ensured forward and backward DNS resolution for the hostname. Set up Kerberos and initialized a realm. Added a principal. Created a keytab. Configured Hadoop and HBase daemons to use kerberos authentication. SASL+Kerberos and hbase.rpc.protection=privacy. Confirmed the configuration is secure and requires krb auth, e.g.
{noformat}
$ klist
klist: No credentials cache found (filename: /tmp/krb5cc_1000)
$ ./hbase-2.5.1-SNAPSHOT/bin/hbase shell
hbase:001:0> list
TABLE
ERROR: Found no valid authentication method from options
$ kinit
Password for apurtell@LOCALDOMAIN: XXXXXXXXXXXX
$ ./hbase-2.5.1-SNAPSHOT/bin/hbase shell
hbase:001:0> list
TABLE
0 row(s)
Took 0.0162 seconds
=> []
hbase:002:0> list_security_capabilities
SECURE_AUTHENTICATION
Took 0.0323 seconds
=> ["SECURE_AUTHENTICATION"]
{noformat}
Tested with NettyRpcServer. All good.
{noformat}
./hbase-2.5.1-SNAPSHOT/bin/hbase ltt -read 100:10 -write 1:100:10 -num_keys 1000000
{noformat}
Switched the regionserver config to SimpleRpcServer. This is relevant output from the regionserver log:
{noformat}
2022-06-30T18:51:26,102 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService, hbase.pb.ClientMetaService
{noformat}
Ran LTT again.
{noformat}
./hbase-2.5.1-SNAPSHOT/bin/hbase ltt -read 100:10 -write 1:100:10 -num_keys 1000000
{noformat}
No issues.
Last lines of LTT output included below:
{noformat}
2022-06-30T18:58:22,046 INFO [MultiThreadedAction-ProgressReporter-1656640437023] util.MultiThreadedAction: [W:11] Keys=991869, cols=2.4 M, time=00:04:25 Overall: [keys/s= 3742, latency=2.66 ms] Current: [keys/s=3176, latency=3.14 ms], wroteUpTo=991855, wroteQSize=3
2022-06-30T18:58:22,059 INFO [MultiThreadedAction-ProgressReporter-1656640437039] util.MultiThreadedAction: [R:10] Keys=4425274, cols=14.8 M, time=00:04:25 Overall: [keys/s= 16697, latency=0.04 ms] Current: [keys/s=14045, latency=0.15 ms], verified=4425274
Failed to write keys: 0
{noformat}
~~Conclusion: SimpleRpcServer is not broken when used with SASL+Kerberos and hbase.rpc.protection=privacy.~~
was (Author: apurtell):
[~vjasani] This is holding up 2.5.0 so I wanted to determine if the issue is possibly environmental.
Built HBase 2.5.1-SNAPSHOT with -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.3
I set up a VM... Configured BIND to serve DNS. Ensured forward and backward DNS resolution for the hostname. Set up Kerberos and initialized a realm. Added a principal. Created a keytab. Configured Hadoop and HBase daemons to use kerberos authentication. SASL+Kerberos and hbase.rpc.protection=privacy. Confirmed the configuration is secure and requires krb auth, e.g.
{noformat}
$ klist
klist: No credentials cache found (filename: /tmp/krb5cc_1000)
$ ./hbase-2.5.1-SNAPSHOT/bin/hbase shell
hbase:001:0> list
TABLE
ERROR: Found no valid authentication method from options
$ kinit
Password for apurtell@LOCALDOMAIN: XXXXXXXXXXXX
$ ./hbase-2.5.1-SNAPSHOT/bin/hbase shell
hbase:001:0> list
TABLE
0 row(s)
Took 0.0162 seconds
=> []
hbase:002:0> list_security_capabilities
SECURE_AUTHENTICATION
Took 0.0323 seconds
=> ["SECURE_AUTHENTICATION"]
{noformat}
Tested with NettyRpcServer. All good.
{noformat}
./hbase-2.5.1-SNAPSHOT/bin/hbase ltt -read 100:10 -write 1:100:10 -num_keys 1000000
{noformat}
Switched the regionserver config to SimpleRpcServer. This is relevant output from the regionserver log:
{noformat}
2022-06-30T18:51:26,102 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService, hbase.pb.ClientMetaService
{noformat}
Ran LTT again.
{noformat}
./hbase-2.5.1-SNAPSHOT/bin/hbase ltt -read 100:10 -write 1:100:10 -num_keys 1000000
{noformat}
No issues.
Last lines of LTT output included below:
{noformat}
2022-06-30T18:58:22,046 INFO [MultiThreadedAction-ProgressReporter-1656640437023] util.MultiThreadedAction: [W:11] Keys=991869, cols=2.4 M, time=00:04:25 Overall: [keys/s= 3742, latency=2.66 ms] Current: [keys/s=3176, latency=3.14 ms], wroteUpTo=991855, wroteQSize=3
2022-06-30T18:58:22,059 INFO [MultiThreadedAction-ProgressReporter-1656640437039] util.MultiThreadedAction: [R:10] Keys=4425274, cols=14.8 M, time=00:04:25 Overall: [keys/s= 16697, latency=0.04 ms] Current: [keys/s=14045, latency=0.15 ms], verified=4425274
Failed to write keys: 0
{noformat}
Conclusion: SimpleRpcServer is not broken when used with SASL+Kerberos and hbase.rpc.protection=privacy.
> SimpleRpcServer is broken
> -------------------------
>
> Key: HBASE-27097
> URL: https://issues.apache.org/jira/browse/HBASE-27097
> Project: HBase
> Issue Type: Bug
> Components: rpc
> Affects Versions: 2.5.0
> Reporter: Andrew Kyle Purtell
> Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-4, 2.4.14
>
> Attachments: MultiByteBuff.patch
>
>
> Concerns about SimpleRpcServer are not new, and not new to 2.5. @chenxu noticed a problem on HBASE-23917 back in 2020. After some simple evaluations it seems quite broken.
> When I run an async version of ITLCC against a 2.5.0 cluster configured with hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls because there are too many in flight requests. The logic to pause with too many in flight requests is my own. That's not important. Looking at the server logs it is apparent that SimpleRpcServer is quite broken. Handlers suffer frequent protobuf parse errors and do not properly return responses to the client. This is what stalls my test client. Rather quickly all available request slots are full of requests that will have to time out on the client side.
> Exceptions have three patterns but they all have in common SimpleServerRpcConnection#process. It seems likely the root cause is mismatched expectations or bugs in connection buffer handling in SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that process and parse the buffers. It also seems likely that changes were made to downstream classes like ServerRpcConnection expecting NettyRpcServer's particulars without updating SimpleServerRpcConnection and/or SimpleRpcServer. That said, this is just a superficial analysis.
> 1) "Protocol message end-group tag did not match expected tag"
> {noformat}
> 2022-06-07T16:44:04,625 WARN [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
> at org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10520) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10464) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9981) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9910) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14190) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15304) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15298) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14860) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14651) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2638) ~[hbase-client-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processRequest(ServerRpcConnection.java:644) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:444) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.process(SimpleServerRpcConnection.java:285) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.readAndProcess(SimpleServerRpcConnection.java:251) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.doRead(SimpleRpcServer.java:318) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:180) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:153) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
> {noformat}
> 2) "Protocol message tag had invalid wire type."
> {noformat}
> 2022-06-07T16:44:04,705 WARN [Reader=6,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
> at org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:134) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:527) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:320) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:10700) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:10620) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:11481) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:11475) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10520) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10464) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9981) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9910) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14190) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15304) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15298) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14860) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14651) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2638) ~[hbase-client-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processRequest(ServerRpcConnection.java:644) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:444) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.process(SimpleServerRpcConnection.java:285) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.readAndProcess(SimpleServerRpcConnection.java:251) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.doRead(SimpleRpcServer.java:318) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:180) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:153) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
> at java.lang.Thread.run(Thread.java:829) ~[?:?]
> {noformat}
> 3) "While parsing a protocol message, the input ended unexpectedly in the middle of a field."
> {noformat}
> 2022-06-07T16:44:04,885 WARN [Reader=9,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.
> at org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:107) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readRawLittleEndian64(CodedInputStream.java:4478) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readFixed64(CodedInputStream.java:4167) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:511) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:320) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:10700) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue.<init>(ClientProtos.java:10620) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:11481) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue$1.parsePartialFrom(ClientProtos.java:11475) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10520) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.<init>(ClientProtos.java:10464) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9981) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.<init>(ClientProtos.java:9910) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14251) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.<init>(ClientProtos.java:14190) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15304) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$1.parsePartialFrom(ClientProtos.java:15298) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14860) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest$Builder.mergeFrom(ClientProtos.java:14651) ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:420) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:317) ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
> at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.mergeFrom(ProtobufUtil.java:2638) ~[hbase-client-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processRequest(ServerRpcConnection.java:644) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:444) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.process(SimpleServerRpcConnection.java:285) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleServerRpcConnection.readAndProcess(SimpleServerRpcConnection.java:251) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener.doRead(SimpleRpcServer.java:318) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.doRunLoop(SimpleRpcServer.java:180) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer$Listener$Reader.run(SimpleRpcServer.java:153) ~[hbase-server-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
> at java.lang.Thread.run(Thread.java:829) ~[?:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)