[jira] [Created] (IMPALA-6431) Disable codegen if query kudu table with PK predicate

["Juan Yu (JIRA)" <ji...@apache.org>]

Juan Yu created IMPALA-6431:
-------------------------------

             Summary: Disable codegen if query kudu table with PK predicate
                 Key: IMPALA-6431
                 URL: https://issues.apache.org/jira/browse/IMPALA-6431
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend
            Reporter: Juan Yu


In near real-time use cases, lots of kudu queries are filtered by primary keys so it only needs to process a small amount of data. In such scenario, codegen could consume a big part of CPU compare to the total query CPU usage. when running with high concurrency, this can easily saturate CPU and impact throughput. Disable codegen will not only improve throughput, also reduce CPU usage.

Here are some test results:

Create table test(
...
PRIMARY KEY (pickup_datetime, medallion, hack_license, vendor_id)
)
PARTITION BY HASH (medallion, vendor_id) PARTITIONS 10
stored as KUDU;

Q1 - select count(*) from trip_data_kudu where pickup_datetime='2013-01-09 20:33:00';

Q2 - select passenger_count, avg(trip_time_in_secs), count(1) from trip_data_kudu where pickup_datetime='2013-01-09 20:33:00' group by 1 order by 2 desc;

Q3 - select * from trip_data_kudu where pickup_datetime='2013-01-09 20:33:00' and vendor_id='CMT';

 
|16 concurrency|QPS|CPU usage|
|Q1 (codegen enabled)|42|76%|
|Q1 (codegen disabled)|250|58%|
| | | |
|Q2 (codegen enabled)|7.7|85%|
|Q2 (codegen disabled)|78|65%|
| | | |
|Q3 (codegen enabled)|52|75%|
|Q3 (codegen disabled)|185|60%|

 

Note that Impala doesn't have per partition cardinality stats for Kudu table, query cannot benefit from DISABLE_CODEGEN_ROWS_THRESHOLD optimization.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)