[jira] [Created] (IMPALA-6279) concurrent performance is bad

["Zhangyi Lu (JIRA)" <ji...@apache.org>]

Zhangyi Lu created IMPALA-6279:
----------------------------------

             Summary: concurrent performance is bad
                 Key: IMPALA-6279
                 URL: https://issues.apache.org/jira/browse/IMPALA-6279
             Project: IMPALA
          Issue Type: Story
    Affects Versions: Impala 2.7.0
            Reporter: Zhangyi Lu
            Priority: Critical
         Attachments: profile.txt, s1.png, s2.png

Version: Impala Shell v2.7.0-cdh5.9.1 (24ad6df) built on Wed Jan 11 13:39:25 PST 2017
Impala daemon number: 7
memory: 120G per daemon
I ran a query in nonconcurrent mode. It took 19.56s
When I ran the same query with other 31 queries concurrently it took 3 minutes 7 seconds. The concurrent performance was reduced dramatically.
I attached two screenshots of some key metrics. In attached screenshot s1.png. This query took 3min7s in total. But the threads network receive and send wait time are 1.7 hours and 1.2 hours respectively. And planning wait time is longer than when this query was executed in non-concurrent mode.
Most of the fragments’ log of this query is like screenshot s2.png, you can see the sum of system and user thread time is far less than total wall clock time. That means wait time took an extreme high percent of total time.
Could someone help point out why the thread network receive and send wait time are so long?
Here I attached a full profile of this query for your reference.
Thanks.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)