You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Michael Smith (Jira)" <ji...@apache.org> on 2024/06/03 23:25:00 UTC
[jira] (KUDU-3582) Incomplete sidecar data returned by RpcContext::GetInboundSidecar()
[ https://issues.apache.org/jira/browse/KUDU-3582 ]
Michael Smith deleted comment on KUDU-3582:
-------------------------------------
was (Author: JIRAUSER288956):
I think this is probably a case where we tried to use sidecars after calling DiscardTransfer - https://github.com/apache/kudu/blob/1.17.0/src/kudu/rpc/inbound_call.h#L218-L220 - which has some big caveats.
> Incomplete sidecar data returned by RpcContext::GetInboundSidecar()
> -------------------------------------------------------------------
>
> Key: KUDU-3582
> URL: https://issues.apache.org/jira/browse/KUDU-3582
> Project: Kudu
> Issue Type: Bug
> Components: rpc
> Reporter: Wenzhe Zhou
> Priority: Major
>
> Impala executor calls KRPC sidecar API RpcContext::GetInboundSidecar() to read serialized thrift object from KRPC, then do thrift deserialization. (See GetSidecar() at
> https://github.com/apache/impala/blob/master/be/src/rpc/sidecar-util.h#L60-L67)
> In a customer reported cases, extra workloads were added to Impala cluster, which caused long delay for KRPCs between Impala daemons. The long delay caused KRPCs been cancelled, hence impala query failures.
> {code:java}
> impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.383988 1182322 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.243.38.160:27000 (fragment_instance_id=9940332ce09828fd:b751966300000632): took 59m57s. Error: Aborted: EndDataStream RPC to 10.243.38.160:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.384006 1182322 kudu-status-util.h:55] EndDataStream() to 10.243.38.160:27000 failed: Aborted: EndDataStream RPC to 10.243.38.160:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.384631 1182314 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.34.163.32:27000 (fragment_instance_id=9940332ce09828fd:b751966300000735): took 59m57s. Error: Aborted: EndDataStream RPC to 10.34.163.32:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.384668 1182314 kudu-status-util.h:55] EndDataStream() to 10.34.163.32:27000 failed: Aborted: EndDataStream RPC to 10.34.163.32:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.420662 1182313 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.243.36.21:27000 (fragment_instance_id=9940332ce09828fd:b75196630000033a): took 1h. Error: Aborted: EndDataStream RPC to 10.243.36.21:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.420683 1182313 kudu-status-util.h:55] EndDataStream() to 10.243.36.21:27000 failed: Aborted: EndDataStream RPC to 10.243.36.21:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.420779 1182322 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.243.38.160:27000 (fragment_instance_id=9940332ce09828fd:b75196630000033b): took 1h. Error: Aborted: EndDataStream RPC to 10.243.38.160:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.420799 1182322 kudu-status-util.h:55] EndDataStream() to 10.243.38.160:27000 failed: Aborted: EndDataStream RPC to 10.243.38.160:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.421937 1182314 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.34.163.32:27000 (fragment_instance_id=9940332ce09828fd:b75196630000043e): took 1h. Error: Aborted:
> {code}
> Then extra workloads were removed and Impala cluster was restarted. During restarting Impala cluster, lots of Impala daemon crashed. The stacktraces of core files and log messages shows that impala daemons received incomplete data from KRPC sidecar. The incomplete data did not cause thrift deserialization failure so the valid but incomplete data was not captured and handled properly.
> See impala Jira: IMPALA-13107. The issue could not be re-produced locally.
> A quick fixing from Impala side was merged to mitigate the crash issue. Need to look into this issue further from KRPC internal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)