You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Sailesh Mukil (JIRA)" <ji...@apache.org> on 2017/04/12 05:50:41 UTC

[jira] [Created] (IMPALA-5199) Impala may hang on empty row batch exchange

Sailesh Mukil created IMPALA-5199:
-------------------------------------

             Summary: Impala may hang on empty row batch exchange
                 Key: IMPALA-5199
                 URL: https://issues.apache.org/jira/browse/IMPALA-5199
             Project: IMPALA
          Issue Type: Bug
          Components: Distributed Exec
    Affects Versions: Impala 2.5.0
            Reporter: Sailesh Mukil
            Assignee: Sailesh Mukil
            Priority: Critical


Hang reproduction:

Start impala with the following stress timeouts (the same as those in test_exchange_delays):
{code}
bin/start-impala-cluster.py --impalad_args=--stress_datastream_recvr_delay_ms=10000 --impalad_args=--datastream_sender_timeout_ms=5000
{code}

Do a join where 0 rows will be exchanged:
{code}
select count(*) from tpch.lineitem inner join tpch.orders on l_orderkey=o_orderkey where o_orderkey < 0
{code}

When the data-stream-sender sends 0 rows, it sends it with params.row_batch.num_rows=0 and params.eos=true.

The receiver hits the following case:
https://github.com/apache/incubator-impala/blob/master/be/src/service/impala-server.cc#L1133
It ultimately times out from FindRecvrOrWait() after datastream_sender_timeout_ms.
https://github.com/apache/incubator-impala/blob/0d0c93ec8c4949940ec113192731f2adb66a0c5e/be/src/runtime/data-stream-mgr.cc#L123

Due to stress_datastream_recvr_delay_ms, the exchange node is Prepare()'d even later.
https://github.com/apache/incubator-impala/blob/a50c344077f6c9bbea3d3cbaa2e9146ba20ac9a9/be/src/exec/exchange-node.cc#L74

Although this is a testing flag, this behavior is reasonable to expect in a real world setting under heavy stress, since the stress_datastream_recvr_delay_ms flag was added into testing after some users experienced nodes receiving data from DataStreamSenders and timing out after datastream_sender_timeout_ms, and before the ExchangeNodes were created. Thus, the CloseSender() function is called before the exchange node is created.

After stress_datastream_recvr_delay_ms, the ExchangeNode is created and it calls CreateRecvr().
https://github.com/apache/incubator-impala/blob/a50c344077f6c9bbea3d3cbaa2e9146ba20ac9a9/be/src/exec/exchange-node.cc#L80

Then ExchangeNode::Open() calls DataStreamRecvr::CreateMerger() which makes a few threads wait on DataStreamRecvr::SenderQueue::GetBatch().
https://github.com/apache/incubator-impala/blob/0d0c93ec8c4949940ec113192731f2adb66a0c5e/be/src/runtime/data-stream-recvr.cc#L124

However, since the DataStreamSender has already received the timeout error from earlier, the receiver threads in GetBatch() will wait indefinitely until the query is cancelled by the user.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)