You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Navis Ryu <na...@nexr.com> on 2014/06/24 11:33:07 UTC
Re: Review Request 16747: Add file pruning into Hive
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16747/
-----------------------------------------------------------
(Updated June 24, 2014, 9:33 a.m.)
Review request for hive.
Changes
-------
Rebased to trunk
Bugs: HIVE-1662
https://issues.apache.org/jira/browse/HIVE-1662
Repository: hive-git
Description
-------
now hive support filename virtual column.
if a file name filter presents in a query, hive should be able to only add files which passed the filter to input paths.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7932a3d
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPathName.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a80feb9
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5e5cf97
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 179ad29
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 2ce4dbd
ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 683618f
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java c3a83d4
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 61cc874
ql/src/java/org/apache/hadoop/hive/ql/metadata/FilePruningPredicateHandler.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java 7d7c764
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java 33ef581
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 5c6751c
ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 703c9d1
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cb284d7
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java f293c43
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 9945dea
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java f3203bf
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 699b476
ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 7aaf455
ql/src/test/queries/clientpositive/file_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/file_pruning.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/16747/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 16747: Add file pruning into Hive
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16747/
-----------------------------------------------------------
(Updated July 4, 2014, 12:13 a.m.)
Review request for hive.
Changes
-------
Fixed test fails
Bugs: HIVE-1662
https://issues.apache.org/jira/browse/HIVE-1662
Repository: hive-git
Description
-------
now hive support filename virtual column.
if a file name filter presents in a query, hive should be able to only add files which passed the filter to input paths.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0256ec9
itests/qtest/testconfiguration.properties 1462ecd
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPathName.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/Context.java abc4290
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a80feb9
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 622ee45
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 29d59a4
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1095173
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java a9869f7
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 949bcfb
ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 683618f
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java c3a83d4
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 61cc874
ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java 409de7c
ql/src/java/org/apache/hadoop/hive/ql/metadata/FilePruningPredicateHandler.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java 7d7c764
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java 33ef581
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 5c6751c
ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 703c9d1
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 399f92a
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java f293c43
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 9945dea
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java f3203bf
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 699b476
ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java e7db370
ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 41243fe
ql/src/test/queries/clientpositive/file_pruning.q PRE-CREATION
ql/src/test/results/clientnegative/index_compact_entry_limit.q.out 85614ca
ql/src/test/results/clientnegative/index_compact_size_limit.q.out 7c6bb0a
ql/src/test/results/clientpositive/file_pruning.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/file_pruning.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/16747/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 16747: Add file pruning into Hive
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16747/
-----------------------------------------------------------
(Updated June 25, 2014, 1:27 a.m.)
Review request for hive.
Changes
-------
Fixed tez test fails
Bugs: HIVE-1662
https://issues.apache.org/jira/browse/HIVE-1662
Repository: hive-git
Description
-------
now hive support filename virtual column.
if a file name filter presents in a query, hive should be able to only add files which passed the filter to input paths.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7932a3d
itests/qtest/pom.xml dc4519a
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPathName.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a80feb9
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 622ee45
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5e5cf97
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 179ad29
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 2ce4dbd
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 949bcfb
ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 683618f
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java c3a83d4
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 61cc874
ql/src/java/org/apache/hadoop/hive/ql/metadata/FilePruningPredicateHandler.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java 7d7c764
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java 33ef581
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 5c6751c
ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 703c9d1
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 83d09c0
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java f293c43
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 9945dea
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java f3203bf
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 699b476
ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 7aaf455
ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 41243fe
ql/src/test/queries/clientpositive/file_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/file_pruning.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/file_pruning.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/16747/diff/
Testing
-------
Thanks,
Navis Ryu