You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Aditya Kishore <ad...@gmail.com> on 2014/04/29 00:15:04 UTC
Review Request 20804: DRILL-494: Filter Pushdown into Scan Operator
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/
-----------------------------------------------------------
Review request for drill, Jinfeng Ni and Steven Phillips.
Bugs: DRILL-494
https://issues.apache.org/jira/browse/DRILL-494
Repository: drill-git
Description
-------
The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
Diffs
-----
exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
Diff: https://reviews.apache.org/r/20804/diff/
Testing
-------
Thanks,
Aditya Kishore
Re: Review Request 20804: DRILL-494: Filter Pushdown into Scan Operator
Posted by Aditya Kishore <ad...@gmail.com>.
> On April 28, 2014, 5:59 p.m., Timothy Chen wrote:
> > The title and jira is not quite close to what the patch is?
You are correct. Initially this was suppose to be one single feature which would include filter pushdown into HBase. Later it was broken into two separate issues and the HBase filter pushdown is handled in DRILL-571.
I'll update the JIRA title to match this.
> On April 28, 2014, 5:59 p.m., Timothy Chen wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java, line 46
> > <https://reviews.apache.org/r/20804/diff/1/?file=569732#file569732line46>
> >
> > What's the use of this?
This is use to describe a particular GroupScan to optique so that it can differentiate between two scans.
> On April 28, 2014, 5:59 p.m., Timothy Chen wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java, line 120
> > <https://reviews.apache.org/r/20804/diff/1/?file=569741#file569741line120>
> >
> > So how does each storage plugin's optimizer rule is able to distinguished in Optiq?
> >
> > Does each plugin itself from each storage engine config has to also check that the Scan is set to be using the storage engine of itself?
> >
> > It seems like perhaps we can do something better if this is the case.
This just adds a set of new optimizer rules to planner. Individual rules will define the condition on which the rule will be matched by Optique. Please look at the HBasePushFilterIntoScan.java in DRILL-571 for example.
- Aditya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/#review41658
-----------------------------------------------------------
On April 28, 2014, 3:15 p.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20804/
> -----------------------------------------------------------
>
> (Updated April 28, 2014, 3:15 p.m.)
>
>
> Review request for drill, Jinfeng Ni and Steven Phillips.
>
>
> Bugs: DRILL-494
> https://issues.apache.org/jira/browse/DRILL-494
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
> exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
> exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
> exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
> exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
> exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
>
> Diff: https://reviews.apache.org/r/20804/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>
Re: Review Request 20804: DRILL-494: Filter Pushdown into Scan Operator
Posted by Timothy Chen <tn...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/#review41658
-----------------------------------------------------------
The title and jira is not quite close to what the patch is?
exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java
<https://reviews.apache.org/r/20804/#comment75215>
What's the use of this?
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java
<https://reviews.apache.org/r/20804/#comment75217>
So how does each storage plugin's optimizer rule is able to distinguished in Optiq?
Does each plugin itself from each storage engine config has to also check that the Scan is set to be using the storage engine of itself?
It seems like perhaps we can do something better if this is the case.
- Timothy Chen
On April 28, 2014, 10:15 p.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20804/
> -----------------------------------------------------------
>
> (Updated April 28, 2014, 10:15 p.m.)
>
>
> Review request for drill, Jinfeng Ni and Steven Phillips.
>
>
> Bugs: DRILL-494
> https://issues.apache.org/jira/browse/DRILL-494
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
> exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
> exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
> exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
> exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
> exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
>
> Diff: https://reviews.apache.org/r/20804/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>
Re: Review Request 20804: DRILL-494: Support for Storage Plugin Optimizer
Rules
Posted by Steven Phillips <sp...@maprtech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/#review41707
-----------------------------------------------------------
Ship it!
Ship It!
- Steven Phillips
On April 29, 2014, 2:29 a.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20804/
> -----------------------------------------------------------
>
> (Updated April 29, 2014, 2:29 a.m.)
>
>
> Review request for drill, Jinfeng Ni and Steven Phillips.
>
>
> Bugs: DRILL-494
> https://issues.apache.org/jira/browse/DRILL-494
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
>
>
> Diffs
> -----
>
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseGroupScan.java b8b6af4
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
> exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
> exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
> exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
> exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
> exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
>
> Diff: https://reviews.apache.org/r/20804/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>
Re: Review Request 20804: DRILL-494: Support for Storage Plugin Optimizer
Rules
Posted by Aditya Kishore <ad...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/
-----------------------------------------------------------
(Updated April 28, 2014, 7:29 p.m.)
Review request for drill, Jinfeng Ni and Steven Phillips.
Changes
-------
Addressed review comments.
Summary (updated)
-----------------
DRILL-494: Support for Storage Plugin Optimizer Rules
Bugs: DRILL-494
https://issues.apache.org/jira/browse/DRILL-494
Repository: drill-git
Description
-------
The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
Diffs (updated)
-----
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseGroupScan.java b8b6af4
exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
Diff: https://reviews.apache.org/r/20804/diff/
Testing
-------
Thanks,
Aditya Kishore
Re: Review Request 20804: DRILL-494: Filter Pushdown into Scan Operator
Posted by Aditya Kishore <ad...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/#review41662
-----------------------------------------------------------
In the process of separating the changes as two separate task, some of the changes which should be in this patch are missed. Will fix this in the updated patch shortly. Please wait for it for further review.
- Aditya Kishore
On April 28, 2014, 3:15 p.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20804/
> -----------------------------------------------------------
>
> (Updated April 28, 2014, 3:15 p.m.)
>
>
> Review request for drill, Jinfeng Ni and Steven Phillips.
>
>
> Bugs: DRILL-494
> https://issues.apache.org/jira/browse/DRILL-494
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
> exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
> exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
> exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
> exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
> exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
>
> Diff: https://reviews.apache.org/r/20804/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>
Re: Review Request 20804: DRILL-494: Filter Pushdown into Scan Operator
Posted by Aditya Kishore <ad...@gmail.com>.
> On April 28, 2014, 5:49 p.m., Jinfeng Ni wrote:
> >
>
> Jinfeng Ni wrote:
> Also, I feel the title of this JIRA probably is a bit misleading, since the most of the change is not about filter pushdown. In stead, the change is to provide the interface necessary to allow each storage plugin to provide a pluggable ruleset to the optimizer. I assume the code change for filter pushdown is in Drill-571 (?).
Yes you are correct, Timothy also pointed it out. Will update the JIRA and title of this review.
- Aditya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/#review41655
-----------------------------------------------------------
On April 28, 2014, 3:15 p.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20804/
> -----------------------------------------------------------
>
> (Updated April 28, 2014, 3:15 p.m.)
>
>
> Review request for drill, Jinfeng Ni and Steven Phillips.
>
>
> Bugs: DRILL-494
> https://issues.apache.org/jira/browse/DRILL-494
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
> exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
> exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
> exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
> exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
> exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
>
> Diff: https://reviews.apache.org/r/20804/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>
Re: Review Request 20804: DRILL-494: Filter Pushdown into Scan Operator
Posted by Aditya Kishore <ad...@gmail.com>.
> On April 28, 2014, 5:49 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java, line 77
> > <https://reviews.apache.org/r/20804/diff/1/?file=569734#file569734line77>
> >
> > I assume we will revisit the cost formula here, after Aman's cost model is integrated. Here, getNetwork()* getDisk() probably need revision.
Yes, I'll add the TODO tag here.
> On April 28, 2014, 5:49 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java, line 82
> > <https://reviews.apache.org/r/20804/diff/1/?file=569734#file569734line82>
> >
> > Since ScanPrel has a private field "scan", we can return this "scan" directly, in stead of getting from drillTable.
Thanks, will fix this.
> On April 28, 2014, 5:49 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java, line 96
> > <https://reviews.apache.org/r/20804/diff/1/?file=569734#file569734line96>
> >
> > It would be better we use scan.getDigest(), in stead of scan.toString(), since we now have getDigest() GroupScan interface.
Aah, that was the intention, missed doing that :(
- Aditya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/#review41655
-----------------------------------------------------------
On April 28, 2014, 3:15 p.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20804/
> -----------------------------------------------------------
>
> (Updated April 28, 2014, 3:15 p.m.)
>
>
> Review request for drill, Jinfeng Ni and Steven Phillips.
>
>
> Bugs: DRILL-494
> https://issues.apache.org/jira/browse/DRILL-494
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
> exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
> exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
> exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
> exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
> exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
>
> Diff: https://reviews.apache.org/r/20804/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>
Re: Review Request 20804: DRILL-494: Filter Pushdown into Scan Operator
Posted by Jinfeng Ni <jn...@maprtech.com>.
> On April 28, 2014, 5:49 p.m., Jinfeng Ni wrote:
> >
Also, I feel the title of this JIRA probably is a bit misleading, since the most of the change is not about filter pushdown. In stead, the change is to provide the interface necessary to allow each storage plugin to provide a pluggable ruleset to the optimizer. I assume the code change for filter pushdown is in Drill-571 (?).
- Jinfeng
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/#review41655
-----------------------------------------------------------
On April 28, 2014, 3:15 p.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20804/
> -----------------------------------------------------------
>
> (Updated April 28, 2014, 3:15 p.m.)
>
>
> Review request for drill, Jinfeng Ni and Steven Phillips.
>
>
> Bugs: DRILL-494
> https://issues.apache.org/jira/browse/DRILL-494
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
> exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
> exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
> exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
> exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
> exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
>
> Diff: https://reviews.apache.org/r/20804/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>
Re: Review Request 20804: DRILL-494: Filter Pushdown into Scan Operator
Posted by Jinfeng Ni <jn...@maprtech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20804/#review41655
-----------------------------------------------------------
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java
<https://reviews.apache.org/r/20804/#comment75207>
I assume we will revisit the cost formula here, after Aman's cost model is integrated. Here, getNetwork()* getDisk() probably need revision.
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java
<https://reviews.apache.org/r/20804/#comment75214>
Since ScanPrel has a private field "scan", we can return this "scan" directly, in stead of getting from drillTable.
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java
<https://reviews.apache.org/r/20804/#comment75206>
It would be better we use scan.getDigest(), in stead of scan.toString(), since we now have getDigest() GroupScan interface.
- Jinfeng Ni
On April 28, 2014, 3:15 p.m., Aditya Kishore wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20804/
> -----------------------------------------------------------
>
> (Updated April 28, 2014, 3:15 p.m.)
>
>
> Review request for drill, Jinfeng Ni and Steven Phillips.
>
>
> Bugs: DRILL-494
> https://issues.apache.org/jira/browse/DRILL-494
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> The patch allows individual storage plugins to add custom optimizer rules which can be used to rewrite the query plan, for example a qualified filter can be pushed into the scan operator if the storage engine supports such a capbility.
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/GroupScan.java 314e889
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 1492a28
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java a945129
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrule.java 8985a58
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java 8892a8f
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java 3081b46
> exec/java-exec/src/main/java/org/apache/drill/exec/store/QueryOptimizerRule.java ec6465e
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java 67f6df3
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginOptimizerRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 7a88098
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FormatPlugin.java 73e414c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/ReadEntryWithPath.java bf1d762
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java 6e87da5
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 68fee34
> exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java cfbeb83
> exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveScan.java bf33805
> exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java b8b66dc
> exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java b71784b
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java d9e6795
> exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java cd7575d
>
> Diff: https://reviews.apache.org/r/20804/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aditya Kishore
>
>