[jira] [Created] (DRILL-4365) Performance with lots of small parquet files

["Aman Sinha (JIRA)" <ji...@apache.org>]

Aman Sinha created DRILL-4365:
---------------------------------

             Summary: Performance with lots of small parquet files
                 Key: DRILL-4365
                 URL: https://issues.apache.org/jira/browse/DRILL-4365
             Project: Apache Drill
          Issue Type: Bug
          Components: Storage - Parquet
    Affects Versions: 1.5.0
            Reporter: Aman Sinha


I am seeing a performance degradation on 1.5.0 compared to 1.4.0 with a query over 968 small parquet files where the total # rows is only 1000, so just about 1 row per file.  The profile shows parquet scan is slower.  With bigger  tables, I haven't seen the same issue yet (although need confirmation from the full performance run).  

Note: this is with default slice_target of 100K so only 1 scan fragment was used.   I will attach the dataset to this JIRA if anyone wants to repro. 

On 1.4.0: (with multiple runs):

{noformat}
0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
+---------+
| EXPR$0  |
+---------+
| 39      |
+---------+
1 row selected (2.544 seconds)
0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
+---------+
| EXPR$0  |
+---------+
| 39      |
+---------+
1 row selected (2.434 seconds)
{noformat}

On 1.5.0:  (multiple runs):

{noformat}
0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
+---------+
| EXPR$0  |
+---------+
| 39      |
+---------+
1 row selected (3.851 seconds)
0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
+---------+
| EXPR$0  |
+---------+
| 39      |
+---------+
1 row selected (3.61 seconds)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)