[jira] [Created] (DRILL-1549) Support partition pruning for simple IN filters

["Aman Sinha (JIRA)" <ji...@apache.org>]

Aman Sinha created DRILL-1549:
---------------------------------

             Summary: Support partition pruning for simple IN filters
                 Key: DRILL-1549
                 URL: https://issues.apache.org/jira/browse/DRILL-1549
             Project: Apache Drill
          Issue Type: Bug
          Components: Query Planning & Optimization
    Affects Versions: 0.6.0
            Reporter: Aman Sinha
            Assignee: Aman Sinha
             Fix For: 0.7.0


Queries containing directory filters of the form 'dir0 IN (1, 2)' currently don't get considered for partition pruning due to a bug inside the visitor of DirPathBuilder.  This JIRA is to address the issue for basic IN filters, which actually is converted to an OR condition by the optimizer (dir0 = 1 OR dir1 = 2).  Note that this JIRA does not cover the more general IN filters which are combined with other ANDs etc.

To illustrate, here's the restricted case that we want to fix here: (the explain shows that the scan is reading all directories, not just 1995, 1996): 

explain plan for select * from dfs.`/Users/asinha/data/multilevel/parquet` where dir0 in (1995, 1996);

...
 "graph" : [ {
    "pop" : "parquet-scan",
    "@id" : 4,
    "entries" : [ {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1994/Q1/orders_94_q1.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1994/Q2/orders_94_q2.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1994/Q3/orders_94_q3.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1994/Q4/orders_94_q4.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1995/Q1/orders_95_q1.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1995/Q2/orders_95_q2.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1995/Q3/orders_95_q3.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1995/Q4/orders_95_q4.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1996/Q1/orders_96_q1.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1996/Q2/orders_96_q2.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1996/Q3/orders_96_q3.parquet"
    }, {
      "path" : "file:/Users/asinha/data/multilevel/parquet/1996/Q4/orders_96_q4.parquet"
    } ],



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)