You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Michael Ho (JIRA)" <ji...@apache.org> on 2019/02/27 04:19:00 UTC
[jira] [Resolved] (IMPALA-8027) KRPC datastream timing out on both
the receiver and sender side even in a minicluster
[ https://issues.apache.org/jira/browse/IMPALA-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Ho resolved IMPALA-8027.
--------------------------------
Resolution: Duplicate
Haven't happened in a while. The long term fix is IMPALA-6618.
> KRPC datastream timing out on both the receiver and sender side even in a minicluster
> -------------------------------------------------------------------------------------
>
> Key: IMPALA-8027
> URL: https://issues.apache.org/jira/browse/IMPALA-8027
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
> Affects Versions: Impala 3.2.0
> Reporter: Bikramjeet Vig
> Assignee: Michael Ho
> Priority: Critical
> Labels: broken-build
>
> krpc datastreams seem to time out at the same time at both sender and receiver causing two running queries to fail. This happened while running core tests on s3.
> Logs from coordinator:
> {noformat}
> I1228 05:18:56.202587 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1 timed out waiting for receiver fragment instance: 8f46b2518734bef1:6ef2d40400000000, dest node: 2
> I1228 05:18:56.203061 13396 rpcz_store.cc:265] Call impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id 11274) took 120782ms. Request Metrics: {}
> I1228 05:18:56.203114 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1 timed out waiting for receiver fragment instance: 194f5b70907ac97c:84a116d600000000, dest node: 2
> I1228 05:18:56.203136 13396 rpcz_store.cc:265] Call impala.DataStreamService.TransmitData from 127.0.0.1:53110 (request call id 8637) took 123811ms. Request Metrics: {}
> I1228 05:18:56.203155 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1 timed out waiting for receiver fragment instance: 194f5b70907ac97c:84a116d600000000, dest node: 2
> I1228 05:18:56.203167 13396 rpcz_store.cc:265] Call impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id 11273) took 123776ms. Request Metrics: {}
> I1228 05:18:56.203181 13396 krpc-data-stream-mgr.cc:408] Reduced stream ID cache from 413 items, to 410, eviction took: 1ms
> I1228 05:18:56.204746 13377 coordinator.cc:707] Backend completed: host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001 remaining=2 query_id=8f46b2518734bef1:6ef2d40400000000
> I1228 05:18:56.204756 13377 coordinator-backend-state.cc:262] query_id=8f46b2518734bef1:6ef2d40400000000: first in-progress backend: impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000
> I1228 05:18:56.204769 13377 coordinator.cc:522] ExecState: query id=8f46b2518734bef1:6ef2d40400000000 finstance=8f46b2518734bef1:6ef2d40400000001 on host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001 (EXECUTING -> ERROR) status=Sender 127.0.0.1 timed out waiting for receiver fragment instance: 8f46b2518734bef1:6ef2d40400000000, dest node: 2
> {noformat}
> Logs from executor:
> {noformat}
> E1228 05:18:56.203181 26715 krpc-data-stream-sender.cc:343] channel send to 127.0.0.1:27000 failed: (fragment_instance_id=8f46b2518734bef1:6ef2d40400000000): Sender 127.0.0.1 timed out waiting for receiver fragment instance: 8f46b2518734bef1:6ef2d40400000000, dest node: 2
> E1228 05:18:56.203256 26682 krpc-data-stream-sender.cc:343] channel send to 127.0.0.1:27000 failed: (fragment_instance_id=194f5b70907ac97c:84a116d600000000): Sender 127.0.0.1 timed out waiting for receiver fragment instance: 194f5b70907ac97c:84a116d600000000, dest node: 2
> I1228 05:18:56.203451 26715 query-state.cc:576] Instance completed. instance_id=8f46b2518734bef1:6ef2d40400000001 #in-flight=3 status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for receiver fragment instance: 8f46b2518734bef1:6ef2d40400000000, dest node: 2
> I1228 05:18:56.203485 26713 query-state.cc:249] UpdateBackendExecState(): last report for 8f46b2518734bef1:6ef2d40400000000
> I1228 05:18:56.203514 26682 query-state.cc:576] Instance completed. instance_id=194f5b70907ac97c:84a116d600000003 #in-flight=2 status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for receiver fragment instance: 194f5b70907ac97c:84a116d600000000, dest node: 2
> I1228 05:18:56.203536 26680 query-state.cc:249] UpdateBackendExecState(): last report for 194f5b70907ac97c:84a116d600000000
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)