You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Alexey Serbin (Jira)" <ji...@apache.org> on 2024/05/30 00:54:00 UTC
[jira] [Comment Edited] (KUDU-3582) Incomplete sidecar data returned by RpcContext::GetInboundSidecar()
[ https://issues.apache.org/jira/browse/KUDU-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850533#comment-17850533 ]
Alexey Serbin edited comment on KUDU-3582 at 5/30/24 12:53 AM:
---------------------------------------------------------------
Hi [~wzhou],
I'm not sure there is enough information in the description to understand what this report is about. IIUC, the summary states that the sidecar data returned by {{RpcContext}} wasn't complete, but it's also stated that the corresponding RPC has been cancelled. Why would one expect complete data from a cancelled RPC?
Could you please clarify on the following:
# What was the expected KRPC's behavior in your scenario?
# What is the actual KRPC's behavior you observed?
In other words, what's the essence of the bug reported here?
Thank you!
was (Author: aserbin):
Hi [~wzhou],
I'm not sure there is enough information in the description to understand what this report is about.
Could you please clarify on the following:
# What was the expected KRPC's behavior in your scenario?
# What is the actual KRPC's behavior you observed?
In other words, what's the essence of the bug reported here?
Thank you!
> Incomplete sidecar data returned by RpcContext::GetInboundSidecar()
> -------------------------------------------------------------------
>
> Key: KUDU-3582
> URL: https://issues.apache.org/jira/browse/KUDU-3582
> Project: Kudu
> Issue Type: Bug
> Components: rpc
> Reporter: Wenzhe Zhou
> Priority: Major
>
> Impala executor calls KRPC sidecar API RpcContext::GetInboundSidecar() to read serialized thrift object from KRPC, then do thrift deserialization. (See GetSidecar() at
> https://github.com/apache/impala/blob/master/be/src/rpc/sidecar-util.h#L60-L67)
> In a customer reported cases, extra workloads were added to Impala cluster, which caused long delay for KRPCs between Impala daemons. The long delay caused KRPCs been cancelled, hence impala query failures.
> {code:java}
> impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.383988 1182322 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.243.38.160:27000 (fragment_instance_id=9940332ce09828fd:b751966300000632): took 59m57s. Error: Aborted: EndDataStream RPC to 10.243.38.160:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.384006 1182322 kudu-status-util.h:55] EndDataStream() to 10.243.38.160:27000 failed: Aborted: EndDataStream RPC to 10.243.38.160:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.384631 1182314 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.34.163.32:27000 (fragment_instance_id=9940332ce09828fd:b751966300000735): took 59m57s. Error: Aborted: EndDataStream RPC to 10.34.163.32:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.384668 1182314 kudu-status-util.h:55] EndDataStream() to 10.34.163.32:27000 failed: Aborted: EndDataStream RPC to 10.34.163.32:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.420662 1182313 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.243.36.21:27000 (fragment_instance_id=9940332ce09828fd:b75196630000033a): took 1h. Error: Aborted: EndDataStream RPC to 10.243.36.21:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.420683 1182313 kudu-status-util.h:55] EndDataStream() to 10.243.36.21:27000 failed: Aborted: EndDataStream RPC to 10.243.36.21:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.420779 1182322 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.243.38.160:27000 (fragment_instance_id=9940332ce09828fd:b75196630000033b): took 1h. Error: Aborted: EndDataStream RPC to 10.243.38.160:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.420799 1182322 kudu-status-util.h:55] EndDataStream() to 10.243.38.160:27000 failed: Aborted: EndDataStream RPC to 10.243.38.160:27000 is cancelled in state ON_OUTBOUND_QUEUE impalad.bdnyr019x21t1.nam.nsroot.net.impala.log.INFO.20240520-210047.1181040:I0521 05:40:09.421937 1182314 krpc-data-stream-sender.cc:405] Slow EndDataStream RPC to 10.34.163.32:27000 (fragment_instance_id=9940332ce09828fd:b75196630000043e): took 1h. Error: Aborted:
> {code}
> Then extra workloads were removed and Impala cluster was restarted. During restarting Impala cluster, lots of Impala daemon crashed. The stacktraces of core files and log messages shows that impala daemons received incomplete data from KRPC sidecar. The incomplete data did not cause thrift deserialization failure so the valid but incomplete data was not captured and handled properly.
> See impala Jira: IMPALA-13107. The issue could not be re-produced locally.
> A quick fixing from Impala side was merged to mitigate the crash issue. Need to look into this issue further from KRPC internal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)