You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Jason Altekruse <al...@gmail.com> on 2015/02/06 00:02:09 UTC
Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/
-----------------------------------------------------------
Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
Bugs: DRILL-2173
https://issues.apache.org/jira/browse/DRILL-2173
Repository: drill-git
Description
-------
Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
Diffs
-----
exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94
exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestConstantFolding.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 3b7adca
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/OptiqForkedConstantReduxRule.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java db6c0c7
Diff: https://reviews.apache.org/r/30701/diff/
Testing
-------
Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
Thanks,
Jason Altekruse
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jinfeng Ni <jn...@maprtech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review78964
-----------------------------------------------------------
Ship it!
Ship It!
- Jinfeng Ni
On April 3, 2015, 4:12 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated April 3, 2015, 4:12 p.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, Parth Chandra, and Venki Korukanti.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that can query against partition information and then scan a subset of data. Example use case, find the most recent directory and only that partition worth of data.
>
>
> Diffs
> -----
>
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java 7b76092
> contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveSchemaFactory.java 023517b
> contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/schema/MongoSchemaFactory.java 32c42ba
> exec/java-exec/src/main/codegen/templates/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java ab121b0
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 35c35ec
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java a4ac724
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 3b51a69
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java f7a1a04
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractSchema.java 90e3ef4
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorerImpl.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/SubSchemaWrapper.java 2c0d8b8
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java 4a3eba9
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/SubDirectoryList.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java aeff09b
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
> exec/java-exec/src/test/java/org/apache/drill/exec/planner/TestDirectoryExplorerUDFs.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Test have been run on a very recent version, made a few minor cleanup edits since, waiting on another run, but do not anticipate issues.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/
-----------------------------------------------------------
(Updated April 3, 2015, 11:12 p.m.)
Review request for drill, Jacques Nadeau, Mehant Baid, Parth Chandra, and Venki Korukanti.
Changes
-------
Added UDFs for both case sensitivities as well as finding minimum directory (full list: maxdir,imaxdir,mindir,imindir), Now generated with freemarker. Changed interface to take List instead of Collection to enforce ordering (as these lists are expected to be used in parellel). Removed unused imports and some general cleanup. SubDirectoryList was shared between the workspace and filesystem schemas, so I pulled it out into a separate class.
Bugs: DRILL-2173
https://issues.apache.org/jira/browse/DRILL-2173
Repository: drill-git
Description
-------
Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that can query against partition information and then scan a subset of data. Example use case, find the most recent directory and only that partition worth of data.
Diffs (updated)
-----
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java 7b76092
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveSchemaFactory.java 023517b
contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/schema/MongoSchemaFactory.java 32c42ba
exec/java-exec/src/main/codegen/templates/DirectoryExplorers.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java ab121b0
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 35c35ec
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java a4ac724
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 3b51a69
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java f7a1a04
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractSchema.java 90e3ef4
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorerImpl.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaPartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/SubSchemaWrapper.java 2c0d8b8
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java 4a3eba9
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/SubDirectoryList.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java aeff09b
exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
exec/java-exec/src/test/java/org/apache/drill/exec/planner/TestDirectoryExplorerUDFs.java PRE-CREATION
Diff: https://reviews.apache.org/r/30701/diff/
Testing
-------
Test have been run on a very recent version, made a few minor cleanup edits since, waiting on another run, but do not anticipate issues.
Thanks,
Jason Altekruse
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jinfeng Ni <jn...@maprtech.com>.
> On April 1, 2015, 1 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java, line 82
> > <https://reviews.apache.org/r/30701/diff/6/?file=906263#file906263line82>
> >
> > Not sure if the case-insensitive comparison should be used as default, or it should depend on the schema (i.e HBase schema could use a different sensitive policy from FileSystemSchema, etc), or it should be passed in as a parameter of udf maxdir(). In the query
> > " select *
> > from dfs.my_workspace.data_directory
> > where dir0 in (select MAX(dir0) from dfs.my_workspace.data_directory)"
> >
> > Aggregate function max() could use case sensitive string comparison. If this maxdir UDF chooses to use case-insensitive, then after partition pruning, it might return different query results.
>
> Jason Altekruse wrote:
> The primary use case we had in mind with this feature was actually just finding recent data, so all of the partition names were numeric. For the sake of date formats that are arranged such that a string comparison can give the corret result, ie YYYY-MM-DD or similar, the case sensitivity wouldn't matter. I think there are a lot of possibilities of ways that users might want to query there partition information, and I think it might be best to leave open the interface for writing custom UDFs in these cases. I could pass a flag to this UDF, or wriate another to do the same operation but case-sensitively.
Make sense either add a flag, or provide another UDF which will use case-sensitive compare. (similar to the function implementation "like" and "ilike").
- Jinfeng
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review78562
-----------------------------------------------------------
On March 25, 2015, 5:54 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated March 25, 2015, 5:54 p.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, Parth Chandra, and Venki Korukanti.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that can query against partition information and then scan a subset of data. Example use case, find the most recent directory and only that partition worth of data.
>
>
> Diffs
> -----
>
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java 7b76092
> contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveSchemaFactory.java 023517b
> contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/schema/MongoSchemaFactory.java 32c42ba
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java ab121b0
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 35c35ec
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 5e31e5c
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 3b51a69
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java f7a1a04
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractSchema.java 90e3ef4
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorerImpl.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/SubSchemaWrapper.java 2c0d8b8
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java 4a3eba9
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java 7c8d9b3
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Test have been run on a very recent version, made a few minor cleanup edits since, waiting on another run, but do not anticipate issues.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
> On April 1, 2015, 8 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java, line 82
> > <https://reviews.apache.org/r/30701/diff/6/?file=906263#file906263line82>
> >
> > Not sure if the case-insensitive comparison should be used as default, or it should depend on the schema (i.e HBase schema could use a different sensitive policy from FileSystemSchema, etc), or it should be passed in as a parameter of udf maxdir(). In the query
> > " select *
> > from dfs.my_workspace.data_directory
> > where dir0 in (select MAX(dir0) from dfs.my_workspace.data_directory)"
> >
> > Aggregate function max() could use case sensitive string comparison. If this maxdir UDF chooses to use case-insensitive, then after partition pruning, it might return different query results.
The primary use case we had in mind with this feature was actually just finding recent data, so all of the partition names were numeric. For the sake of date formats that are arranged such that a string comparison can give the corret result, ie YYYY-MM-DD or similar, the case sensitivity wouldn't matter. I think there are a lot of possibilities of ways that users might want to query there partition information, and I think it might be best to leave open the interface for writing custom UDFs in these cases. I could pass a flag to this UDF, or wriate another to do the same operation but case-sensitively.
> On April 1, 2015, 8 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java, line 101
> > <https://reviews.apache.org/r/30701/diff/6/?file=906270#file906270line101>
> >
> > What's the purpose of passing partitionColumns and partitionValues? In FileSystemSchema or WorkspaceSchema getSubPartitions, those two parameters are not used. UDF DirectoryExplorers just passes two empty list. I'm not clear why the interface need these two additional parameters, on top of "schema" and "table".
These columns were added for use with storage systems that track partition column names. It is the case that they are not used for the only two current implementations of the interface in the file system and workspace schemas. These are primarily useful for Hive, as we can do partition pruning currently on partition columns. Adding this to the interface allowed generalizing this functionality to enable future use in Hive. It would be possible that would could have two different interface to avoid confusion in the cases where the partion columns are not needed.
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review78562
-----------------------------------------------------------
On March 26, 2015, 12:54 a.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated March 26, 2015, 12:54 a.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, Parth Chandra, and Venki Korukanti.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that can query against partition information and then scan a subset of data. Example use case, find the most recent directory and only that partition worth of data.
>
>
> Diffs
> -----
>
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java 7b76092
> contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveSchemaFactory.java 023517b
> contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/schema/MongoSchemaFactory.java 32c42ba
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java ab121b0
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 35c35ec
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 5e31e5c
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 3b51a69
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java f7a1a04
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractSchema.java 90e3ef4
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorerImpl.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/SubSchemaWrapper.java 2c0d8b8
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java 4a3eba9
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java 7c8d9b3
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Test have been run on a very recent version, made a few minor cleanup edits since, waiting on another run, but do not anticipate issues.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jinfeng Ni <jn...@maprtech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review78562
-----------------------------------------------------------
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java
<https://reviews.apache.org/r/30701/#comment127434>
Not sure if the case-insensitive comparison should be used as default, or it should depend on the schema (i.e HBase schema could use a different sensitive policy from FileSystemSchema, etc), or it should be passed in as a parameter of udf maxdir(). In the query
" select *
from dfs.my_workspace.data_directory
where dir0 in (select MAX(dir0) from dfs.my_workspace.data_directory)"
Aggregate function max() could use case sensitive string comparison. If this maxdir UDF chooses to use case-insensitive, then after partition pruning, it might return different query results.
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java
<https://reviews.apache.org/r/30701/#comment127432>
What's the purpose of passing partitionColumns and partitionValues? In FileSystemSchema or WorkspaceSchema getSubPartitions, those two parameters are not used. UDF DirectoryExplorers just passes two empty list. I'm not clear why the interface need these two additional parameters, on top of "schema" and "table".
- Jinfeng Ni
On March 25, 2015, 5:54 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated March 25, 2015, 5:54 p.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, Parth Chandra, and Venki Korukanti.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that can query against partition information and then scan a subset of data. Example use case, find the most recent directory and only that partition worth of data.
>
>
> Diffs
> -----
>
> contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java 7b76092
> contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveSchemaFactory.java 023517b
> contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/schema/MongoSchemaFactory.java 32c42ba
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java ab121b0
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 35c35ec
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 5e31e5c
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 3b51a69
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java f7a1a04
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractSchema.java 90e3ef4
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorerImpl.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/SubSchemaWrapper.java 2c0d8b8
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java 4a3eba9
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java 7c8d9b3
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Test have been run on a very recent version, made a few minor cleanup edits since, waiting on another run, but do not anticipate issues.
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/
-----------------------------------------------------------
(Updated March 26, 2015, 12:54 a.m.)
Review request for drill, Jacques Nadeau, Mehant Baid, Parth Chandra, and Venki Korukanti.
Changes
-------
Adding venki
Bugs: DRILL-2173
https://issues.apache.org/jira/browse/DRILL-2173
Repository: drill-git
Description (updated)
-------
Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that can query against partition information and then scan a subset of data. Example use case, find the most recent directory and only that partition worth of data.
Diffs
-----
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java 7b76092
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveSchemaFactory.java 023517b
contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/schema/MongoSchemaFactory.java 32c42ba
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java ab121b0
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 35c35ec
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 5e31e5c
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 3b51a69
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java f7a1a04
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractSchema.java 90e3ef4
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorerImpl.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaPartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/SubSchemaWrapper.java 2c0d8b8
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java 4a3eba9
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java 7c8d9b3
exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
Diff: https://reviews.apache.org/r/30701/diff/
Testing (updated)
-------
Test have been run on a very recent version, made a few minor cleanup edits since, waiting on another run, but do not anticipate issues.
Thanks,
Jason Altekruse
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/
-----------------------------------------------------------
(Updated March 26, 2015, 12:52 a.m.)
Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
Changes
-------
Changed the interface to attach it to schema instead of storage plugins. Schemas are created for the lifecycle of a query, unlike storage plugins that exist from when they are registered throughout the lifecycle of the drillbit. To enable features in the future like login/session specific permissions on tables, any element of a query that wants to access data or metadata should go through the schema to ensure that it is accessible from the current session, not the storage plugin directly as was done previously.
Bugs: DRILL-2173
https://issues.apache.org/jira/browse/DRILL-2173
Repository: drill-git
Description
-------
Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
Diffs (updated)
-----
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseSchemaFactory.java 7b76092
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/schema/HiveSchemaFactory.java 023517b
contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/schema/MongoSchemaFactory.java 32c42ba
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java ab121b0
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 35c35ec
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 5e31e5c
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java 3b51a69
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java f7a1a04
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractSchema.java 90e3ef4
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorerImpl.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaPartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/SubSchemaWrapper.java 2c0d8b8
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemSchemaFactory.java 4a3eba9
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java 7c8d9b3
exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
Diff: https://reviews.apache.org/r/30701/diff/
Testing
-------
Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
Thanks,
Jason Altekruse
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/
-----------------------------------------------------------
(Updated March 16, 2015, 9:04 p.m.)
Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
Changes
-------
Address Chris' review comments
Bugs: DRILL-2173
https://issues.apache.org/jira/browse/DRILL-2173
Repository: drill-git
Description
-------
Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
Diffs (updated)
-----
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctionHelpers.java 3b1d7ef
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 108f5bb
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java 7a1f61e
exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/ExpressionInterpreterTest.java PRE-CREATION
exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
Diff: https://reviews.apache.org/r/30701/diff/
Testing
-------
Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
Thanks,
Jason Altekruse
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java, line 71
> > <https://reviews.apache.org/r/30701/diff/3/?file=853387#file853387line71>
> >
> > Is there a reason to use java.io. here?
In the compiled classes we do not include the merged imports list (I do not know the reason for this). You can see references to static helper methods, for example several of the UDFs defined in StringFunctions refernce regex classes/utilties or a Drill specific toStringFromUTF8 method from Drill using the fully qualified name.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java, line 79
> > <https://reviews.apache.org/r/30701/diff/3/?file=853387#file853387line79>
> >
> > Why use this fully qualified name instead of just MAXDIR_NAME? Same below.
see above.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java, line 50
> > <https://reviews.apache.org/r/30701/diff/3/?file=853393#file853393line50>
> >
> > If this could be the set of all the files in a directory, it could be really large. Would it be better to define this to return an Iterator<String>? That doesn't necessarily mean the implementations have to change (the interator might just iterate over an array that's generated internally), but it would give implementations that would have a large result set more flexibility in how they work, and not necessarily require materializing a large result set all at once.
That sounds good to me. To allow for use of the extended for loop by consumers of the interface I will have it return Iterable instead.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java, line 106
> > <https://reviews.apache.org/r/30701/diff/3/?file=853387#file853387line106>
> >
> > No blank line here.
fixed
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java, line 50
> > <https://reviews.apache.org/r/30701/diff/3/?file=853396#file853396line50>
> >
> > Same comment as before re the array of Strings.
Fixed
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review74629
-----------------------------------------------------------
On Feb. 28, 2015, 12:18 a.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2015, 12:18 a.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java 7a1f61e
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/ExpressionInterpreterTest.java PRE-CREATION
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java, line 67
> > <https://reviews.apache.org/r/30701/diff/3/?file=853390#file853390line67>
> >
> > To avoid future confusion with these, can we add units to the ends of the names, e.g., INITIAL_OFF_HEAP_ALLOCATION_MB, or whatever it is?
I had to move this change into an earlier patch. I have made the correction for the name with appropriate units there
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java, line 31
> > <https://reviews.apache.org/r/30701/diff/3/?file=853391#file853391line31>
> >
> > Formatting: after the opening brace, use a newline, and line these up one after the other with regular function-style indenting:
> >
> > public static final Class[] INJECTABLE_TYPES = {
> > DrillBuf.class,
> > QueryDateTimeInfo.class,
> > PartitionExplorer.class,
> > };
fixed
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java, line 24
> > <https://reviews.apache.org/r/30701/diff/3/?file=853393#file853393line24>
> >
> > Can we have an interface level comment that describes what the interface is for?
Added an explanation with the complete use case (might be a bit much, but the lack of related docs to refer to makes this a bit hard to describe succinctly).
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java, line 25
> > <https://reviews.apache.org/r/30701/diff/3/?file=853393#file853393line25>
> >
> > No blank line here.
Fixed
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java, line 48
> > <https://reviews.apache.org/r/30701/diff/3/?file=853393#file853393line48>
> >
> > There should also be an @throws line for the exception you declared, explaining the conditions which will cause the exception.
fixed
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java, line 23
> > <https://reviews.apache.org/r/30701/diff/3/?file=853394#file853394line23>
> >
> > There's no need to explicitly show super().
fixed
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java, line 49
> > <https://reviews.apache.org/r/30701/diff/3/?file=853396#file853396line49>
> >
> > @throws PartitionNotFoundException when ....
fixed
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java, line 306
> > <https://reviews.apache.org/r/30701/diff/3/?file=853397#file853397line306>
> >
> > Make these two values final.
Fixed
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java, line 309
> > <https://reviews.apache.org/r/30701/diff/3/?file=853397#file853397line309>
> >
> > If this try block is really just to protect yourself against a failure of 'new String(temp, "UTF-8")', then you should just protect that in order to avoid causing confusion if plugins.get() might also be able to throw UnsupportedEncodingException:
> >
> > String pluginName;
> > try {
> > pluginName = new String(temp, "UTF-8");
> > } catch (UnsupportedEncodingException e) {
> > throw new RuntimeException("UTF-8 encoding unavailable", e);
> > }
> >
> > return plugins.get(pluginName).getSubpartitions(...);
Figred out if I actually pass a charset I don't get the exception. simplifies this in a number of places.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java, line 121
> > <https://reviews.apache.org/r/30701/diff/3/?file=853398#file853398line121>
> >
> > make currentInput and temp final.
refactored, added finals where appropriate in new code.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java, line 130
> > <https://reviews.apache.org/r/30701/diff/3/?file=853398#file853398line130>
> >
> > Same comment as for the handling of this in the StoragePluginRegistry. Since the pattern repeats here two more times, it would be best to capture that and reuse it. I suggest a public static function, possibly on StoragePluginRegistry (or some similar appropriate place) that does this:
> >
> > public static String getStringFromHolder(final VarCharHolder holder) {
> > final byte[] temp = new byte[currentInput.end - currentInput.start];
> > currentInput.buffer.getBytes(0, temp, 0, currentInput.end - currentInput.start);
> > String s;
> > try {
> > s = ...;
> > } catch (...) {
> > ...
> > }
> > return s;
> > }
> >
> > Then, in this, and the function above, you can use
> > final String pluginName = getStringFromHolder(workspace); // or whatever
Found out there was a function similar to what I wanted in StringFucntionHelpers, I defined a utilty there to convert a VarCharHolder to a string and added it in the various places where this pattern appeared before.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java, line 134
> > <https://reviews.apache.org/r/30701/diff/3/?file=853398#file853398line134>
> >
> > Can this be final?
yes, fixed
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java, line 144
> > <https://reviews.apache.org/r/30701/diff/3/?file=853398#file853398line144>
> >
> > Move the declaration to here, and make it final.
refactored, put finals in new iterator class that replaces this interface.
> On Feb. 28, 2015, 12:26 a.m., Chris Westin wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java, line 107
> > <https://reviews.apache.org/r/30701/diff/3/?file=853388#file853388line107>
> >
> > Capture f.getType() once in a final local before the first conditional. Then, it should be ok to directly check for equality using == against the classes -- they'll be the same class reference in the class loader. No need to use equals() in this case.
Fixed
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review74629
-----------------------------------------------------------
On Feb. 28, 2015, 12:18 a.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2015, 12:18 a.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java 7a1f61e
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/ExpressionInterpreterTest.java PRE-CREATION
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Chris Westin <ch...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review74629
-----------------------------------------------------------
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java
<https://reviews.apache.org/r/30701/#comment121249>
Is there a reason to use java.io. here?
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java
<https://reviews.apache.org/r/30701/#comment121250>
Why use this fully qualified name instead of just MAXDIR_NAME? Same below.
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java
<https://reviews.apache.org/r/30701/#comment121251>
No blank line here.
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java
<https://reviews.apache.org/r/30701/#comment121252>
Capture f.getType() once in a final local before the first conditional. Then, it should be ok to directly check for equality using == against the classes -- they'll be the same class reference in the class loader. No need to use equals() in this case.
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java
<https://reviews.apache.org/r/30701/#comment121253>
To avoid future confusion with these, can we add units to the ends of the names, e.g., INITIAL_OFF_HEAP_ALLOCATION_MB, or whatever it is?
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java
<https://reviews.apache.org/r/30701/#comment121256>
Formatting: after the opening brace, use a newline, and line these up one after the other with regular function-style indenting:
public static final Class[] INJECTABLE_TYPES = {
DrillBuf.class,
QueryDateTimeInfo.class,
PartitionExplorer.class,
};
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java
<https://reviews.apache.org/r/30701/#comment121258>
Can we have an interface level comment that describes what the interface is for?
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java
<https://reviews.apache.org/r/30701/#comment121259>
No blank line here.
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java
<https://reviews.apache.org/r/30701/#comment121260>
There should also be an @throws line for the exception you declared, explaining the conditions which will cause the exception.
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java
<https://reviews.apache.org/r/30701/#comment121276>
If this could be the set of all the files in a directory, it could be really large. Would it be better to define this to return an Iterator<String>? That doesn't necessarily mean the implementations have to change (the interator might just iterate over an array that's generated internally), but it would give implementations that would have a large result set more flexibility in how they work, and not necessarily require materializing a large result set all at once.
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java
<https://reviews.apache.org/r/30701/#comment121261>
There's no need to explicitly show super().
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java
<https://reviews.apache.org/r/30701/#comment121262>
@throws PartitionNotFoundException when ....
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java
<https://reviews.apache.org/r/30701/#comment121277>
Same comment as before re the array of Strings.
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java
<https://reviews.apache.org/r/30701/#comment121264>
Make these two values final.
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java
<https://reviews.apache.org/r/30701/#comment121263>
If this try block is really just to protect yourself against a failure of 'new String(temp, "UTF-8")', then you should just protect that in order to avoid causing confusion if plugins.get() might also be able to throw UnsupportedEncodingException:
String pluginName;
try {
pluginName = new String(temp, "UTF-8");
} catch (UnsupportedEncodingException e) {
throw new RuntimeException("UTF-8 encoding unavailable", e);
}
return plugins.get(pluginName).getSubpartitions(...);
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
<https://reviews.apache.org/r/30701/#comment121267>
make currentInput and temp final.
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
<https://reviews.apache.org/r/30701/#comment121269>
Same comment as for the handling of this in the StoragePluginRegistry. Since the pattern repeats here two more times, it would be best to capture that and reuse it. I suggest a public static function, possibly on StoragePluginRegistry (or some similar appropriate place) that does this:
public static String getStringFromHolder(final VarCharHolder holder) {
final byte[] temp = new byte[currentInput.end - currentInput.start];
currentInput.buffer.getBytes(0, temp, 0, currentInput.end - currentInput.start);
String s;
try {
s = ...;
} catch (...) {
...
}
return s;
}
Then, in this, and the function above, you can use
final String pluginName = getStringFromHolder(workspace); // or whatever
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
<https://reviews.apache.org/r/30701/#comment121270>
Can this be final?
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
<https://reviews.apache.org/r/30701/#comment121271>
Move the declaration to here, and make it final.
- Chris Westin
On Feb. 27, 2015, 4:18 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2015, 4:18 p.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java 7a1f61e
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/ExpressionInterpreterTest.java PRE-CREATION
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/
-----------------------------------------------------------
(Updated Feb. 28, 2015, 12:18 a.m.)
Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
Bugs: DRILL-2173
https://issues.apache.org/jira/browse/DRILL-2173
Repository: drill-git
Description
-------
Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
Diffs (updated)
-----
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java 7a1f61e
exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/ExpressionInterpreterTest.java PRE-CREATION
exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
Diff: https://reviews.apache.org/r/30701/diff/
Testing
-------
Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
Thanks,
Jason Altekruse
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/
-----------------------------------------------------------
(Updated Feb. 7, 2015, 1:24 a.m.)
Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
Changes
-------
Code cleanup
Bugs: DRILL-2173
https://issues.apache.org/jira/browse/DRILL-2173
Repository: drill-git
Description
-------
Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
Diffs (updated)
-----
exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94
exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestConstantFolding.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java 7a1f61e
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 99c6ab8
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 378e81a
Diff: https://reviews.apache.org/r/30701/diff/
Testing
-------
Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
Thanks,
Jason Altekruse
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
> On Feb. 6, 2015, 9:21 p.m., Jason Altekruse wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java, line 35
> > <https://reviews.apache.org/r/30701/diff/2/?file=851938#file851938line35>
> >
> > I am considering making it a requirement that implimentors of this interface return a partition description that can be fed back into the method unmodified to continue to traverse down the partition tree.
I had originally declared this interface to take in and return Strings, but this required boilerplate in the UDF to convert the input holders to strings. So I changed the interface to take the VarCharHolders and started to change the return type to be a VarCharHolder array. Unfortunately this would have forced me to expose off heap allocation to the storage plugin, which seemed undesireable. Please let me know your thoughts.
Changing the docs as proposed above would allow for more generic code with just boilerplate to convert the returned strings from the first call to the interface back into varchar holders in subsequent calls to traverse down the heirarchy. It would remove the need for storge plugin specific decisions about whether or not the original partition could/must be concatenated with the sub-partition and what characters UDF writers should glue them together with.
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review71361
-----------------------------------------------------------
On Feb. 6, 2015, 2:05 a.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated Feb. 6, 2015, 2:05 a.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
>
>
> Diffs
> -----
>
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestConstantFolding.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 3b7adca
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/OptiqForkedConstantReduxRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java db6c0c7
> exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 99c6ab8
> exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 378e81a
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
> On Feb. 6, 2015, 9:21 p.m., Jason Altekruse wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java, line 35
> > <https://reviews.apache.org/r/30701/diff/2/?file=851938#file851938line35>
> >
> > I am considering making it a requirement that implimentors of this interface return a partition description that can be fed back into the method unmodified to continue to traverse down the partition tree.
>
> Jason Altekruse wrote:
> I had originally declared this interface to take in and return Strings, but this required boilerplate in the UDF to convert the input holders to strings. So I changed the interface to take the VarCharHolders and started to change the return type to be a VarCharHolder array. Unfortunately this would have forced me to expose off heap allocation to the storage plugin, which seemed undesireable. Please let me know your thoughts.
>
> Changing the docs as proposed above would allow for more generic code with just boilerplate to convert the returned strings from the first call to the interface back into varchar holders in subsequent calls to traverse down the heirarchy. It would remove the need for storge plugin specific decisions about whether or not the original partition could/must be concatenated with the sub-partition and what characters UDF writers should glue them together with.
fixed
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review71361
-----------------------------------------------------------
On Feb. 28, 2015, 12:18 a.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2015, 12:18 a.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
>
>
> Diffs
> -----
>
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java 7a1f61e
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/ExpressionInterpreterTest.java PRE-CREATION
> exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/TestConstantFolding.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review71361
-----------------------------------------------------------
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java
<https://reviews.apache.org/r/30701/#comment117059>
I am considering making it a requirement that implimentors of this interface return a partition description that can be fed back into the method unmodified to continue to traverse down the partition tree.
- Jason Altekruse
On Feb. 6, 2015, 2:05 a.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated Feb. 6, 2015, 2:05 a.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
>
>
> Diffs
> -----
>
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestConstantFolding.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 3b7adca
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/OptiqForkedConstantReduxRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java db6c0c7
> exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 99c6ab8
> exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 378e81a
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/
-----------------------------------------------------------
(Updated Feb. 6, 2015, 2:05 a.m.)
Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
Changes
-------
First udf to use the new interface, maxdir, is working properly.
Bugs: DRILL-2173
https://issues.apache.org/jira/browse/DRILL-2173
Repository: drill-git
Description
-------
Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
Diffs (updated)
-----
exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94
exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestConstantFolding.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 3b7adca
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/OptiqForkedConstantReduxRule.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java db6c0c7
exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 99c6ab8
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 378e81a
Diff: https://reviews.apache.org/r/30701/diff/
Testing
-------
Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
Thanks,
Jason Altekruse
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review71359
-----------------------------------------------------------
Intermediate work branch: https://github.com/jaltekruse/incubator-drill/tree/2173-partition-queries-with-pruning
- Jason Altekruse
On Feb. 5, 2015, 11:02 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated Feb. 5, 2015, 11:02 p.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
>
>
> Diffs
> -----
>
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestConstantFolding.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 3b7adca
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/OptiqForkedConstantReduxRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java db6c0c7
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
>
>
> Thanks,
>
> Jason Altekruse
>
>
Re: Review Request 30701: DRILL-2173 partition queries for dynamic
partition pruning
Posted by Jason Altekruse <al...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30701/#review71334
-----------------------------------------------------------
exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java
<https://reviews.apache.org/r/30701/#comment117014>
Looks like I messed up while copying on of the other tests to make my own, will fix this.
- Jason Altekruse
On Feb. 5, 2015, 11:02 p.m., Jason Altekruse wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30701/
> -----------------------------------------------------------
>
> (Updated Feb. 5, 2015, 11:02 p.m.)
>
>
> Review request for drill, Jacques Nadeau, Mehant Baid, and Parth Chandra.
>
>
> Bugs: DRILL-2173
> https://issues.apache.org/jira/browse/DRILL-2173
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Adds a new interface for UDFs to access partition information. Together with 2060 which allows constant expression folding this will allow UDFs that
>
>
> Diffs
> -----
>
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestConstantFolding.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DirectoryExplorers.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 3b7adca
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/OptiqForkedConstantReduxRule.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/AbstractStoragePlugin.java b032fce
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/PartitionNotFoundException.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePlugin.java ef5978c
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginPartitionExplorer.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/store/StoragePluginRegistry.java 5d0eed6
> exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java db6c0c7
>
> Diff: https://reviews.apache.org/r/30701/diff/
>
>
> Testing
> -------
>
> Some unit tests on the functionality have been run, still a work in progress so no full mvn build run yet
>
>
> Thanks,
>
> Jason Altekruse
>
>