[jira] [Created] (IMPALA-6294) Concurrent hung with lots of fragments and spilling make slow progress due to blocking in DataStreamRecvr and DataStreamSender

["Mostafa Mokhtar (JIRA)" <ji...@apache.org>]

Mostafa Mokhtar created IMPALA-6294:
---------------------------------------

             Summary: Concurrent hung with lots of fragments and spilling make slow progress due to blocking in DataStreamRecvr and DataStreamSender
                 Key: IMPALA-6294
                 URL: https://issues.apache.org/jira/browse/IMPALA-6294
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
    Affects Versions: Impala 2.11.0
            Reporter: Mostafa Mokhtar
            Assignee: Mostafa Mokhtar
            Priority: Critical
         Attachments: IMPALA-6285 TPCDS Q3 slow broadcast, slow_broadcast_q3_reciever.txt, slow_broadcast_q3_sender.txt

While running a highly concurrent spilling workload on a large cluster queries start running slower, even light weight queries that are not running are affected by this slow down. 

{code}
          EXCHANGE_NODE (id=9):(Total: 3m1s, non-child: 3m1s, % non-child: 100.00%)
             - ConvertRowBatchTime: 999.990us
             - PeakMemoryUsage: 0
             - RowsReturned: 108.00K (108001)
             - RowsReturnedRate: 593.00 /sec
            DataStreamReceiver:
              BytesReceived(4s000ms): 254.47 KB, 338.82 KB, 338.82 KB, 852.43 KB, 1.32 MB, 1.33 MB, 1.50 MB, 2.53 MB, 2.99 MB, 3.00 MB, 3.00 MB, 3.00 MB, 3.00 MB, 3.00 MB, 3.00 MB, 3.00 MB, 3.00 MB, 3.00 MB, 3.16 MB, 3.49 MB, 3.80 MB, 4.15 MB, 4.55 MB, 4.84 MB, 4.99 MB, 5.07 MB, 5.41 MB, 5.75 MB, 5.92 MB, 6.00 MB, 6.00 MB, 6.00 MB, 6.07 MB, 6.28 MB, 6.33 MB, 6.43 MB, 6.67 MB, 6.91 MB, 7.29 MB, 8.03 MB, 9.12 MB, 9.68 MB, 9.90 MB, 9.97 MB, 10.44 MB, 11.25 MB
               - BytesReceived: 11.73 MB (12301692)
               - DeserializeRowBatchTimer: 957.990ms
               - FirstBatchArrivalWaitTime: 0.000ns
               - PeakMemoryUsage: 644.44 KB (659904)
               - SendersBlockedTimer: 0.000ns
               - SendersBlockedTotalTimer(*): 0.000ns
{code}

{code}
        DataStreamSender (dst_id=9):(Total: 1s819ms, non-child: 1s819ms, % non-child: 100.00%)
           - BytesSent: 234.64 MB (246033840)
           - NetworkThroughput(*): 139.58 MB/sec
           - OverallThroughput: 128.92 MB/sec
           - PeakMemoryUsage: 33.12 KB (33920)
           - RowsReturned: 108.00K (108001)
           - SerializeBatchTime: 133.998ms
           - TransmitDataRPCTime: 1s680ms
           - UncompressedRowBatchSize: 446.42 MB (468102200)
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)