You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2018/06/07 03:04:05 UTC
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Hello Vuk Ercegovac,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/10631
to review the following change.
Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading
......................................................................
IMPALA-7141. Extract interfaces for partition pruning without loading
This adds a new PrunablePartition interface which HdfsPartition
implements. The interface includes only the partition ID and the
partition key/values.
For the case of the current catalog implementation, this provides no
benefit. However, for LocalCatalog, we want to defer loading partition
information until after pruning. With this interface, we can construct
PrunablePartition objects using just the partition names, and then load
the partitions once pruning is complete.
Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java
18 files changed, 181 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/1
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 7
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Jun 2018 20:09:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10631
to look at the new patch set (#5).
Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading
......................................................................
IMPALA-7141. Extract interfaces for partition pruning without loading
This adds a new PrunablePartition interface which HdfsPartition
implements. The interface includes only the partition ID and the
partition key/values.
For the case of the current catalog implementation, this provides no
benefit. However, for LocalCatalog, we want to defer loading partition
information until after pruning. With this interface, we can construct
PrunablePartition objects using just the partition names, and then load
the partitions once pruning is complete.
Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java
18 files changed, 179 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/5
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10631
to look at the new patch set (#2).
Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading
......................................................................
IMPALA-7141. Extract interfaces for partition pruning without loading
This adds a new PrunablePartition interface which HdfsPartition
implements. The interface includes only the partition ID and the
partition key/values.
For the case of the current catalog implementation, this provides no
benefit. However, for LocalCatalog, we want to defer loading partition
information until after pruning. With this interface, we can construct
PrunablePartition objects using just the partition names, and then load
the partitions once pruning is complete.
Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java
18 files changed, 181 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/2
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10631
to look at the new patch set (#6).
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
This adds a new PrunablePartition interface which HdfsPartition
implements. The interface includes only the partition ID and the
partition key/values.
For the case of the current catalog implementation, this provides no
benefit. However, for LocalCatalog, we want to defer loading partition
information until after pruning. With this interface, we can construct
PrunablePartition objects using just the partition names, and then load
the partitions once pruning is complete.
Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java
18 files changed, 173 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/6
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 6
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading
......................................................................
Patch Set 5:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@136
PS5, Line 136: hdfs
nit: fsTable, or just table
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@141
PS5, Line 141: assert
precondition
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@120
PS5, Line 120: @return
nit: Returns
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@138
PS5, Line 138: @return
nit: Returns
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java
File fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@30
PS5, Line 30: complet
nit: complete
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@35
PS5, Line 35: @return
nit: Returns
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@41
PS5, Line 41: @return
nit: Returns
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@a158
PS5, Line 158:
this will have that default partition whereas matchingPatitionIds will not have it since getPatitionIds (L150), which is the only place I see here it can sneak in. why not add qualifying partitions as done here instead of getting all and filtering?
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@101
PS5, Line 101: If 'allowEmpty' is False, empty partitions are not returned.
heh... how many negations are here?
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:06:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@a158
PS5, Line 158:
> this will have that default partition whereas matchingPatitionIds will not
hm, the issue I was facing wasn't with this, so much as with the assertions in CatalogTest and with EXPLAIN output where we assume that 'getPartitionsMap()' returns an extra partition.
I found the whole "dummy partition" thing really confusing, with special cases in ~10-15 places around the code to skip over them, so I took a whack at a patch to clean that stuff up a bit. I'll put that patch in order before this one in the patch series. Let me know what you think.
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:14:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading
......................................................................
Patch Set 4:
seems like this may have broken some existing behavior. Worth a look at a high level but will need a bit of tweaking before being committable.
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 03:20:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2709/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 10
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jun 2018 21:17:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
This adds a new PrunablePartition interface which HdfsPartition
implements. The interface includes only the partition ID and the
partition key/values.
For the case of the current catalog implementation, this provides no
benefit. However, for LocalCatalog, we want to defer loading partition
information until after pruning. With this interface, we can construct
PrunablePartition objects using just the partition names, and then load
the partitions once pruning is complete.
Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Reviewed-on: http://gerrit.cloudera.org:8080/10631
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Todd Lipcon <to...@apache.org>
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java
18 files changed, 175 insertions(+), 52 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Todd Lipcon: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 11
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
Patch Set 10: Code-Review+2
Forwarding +2
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 10
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jun 2018 01:15:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
Patch Set 10: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 10
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jun 2018 00:51:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@a158
PS5, Line 158:
> hm, the issue I was facing wasn't with this, so much as with the assertions
any cleanup/simplification is a good thing. hopefully this is not one of those threads that unravels too many other things.
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:28:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading
......................................................................
Patch Set 5:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@136
PS5, Line 136: hdfs
> nit: fsTable, or just table
Done
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@141
PS5, Line 141: assert
> precondition
it's not really a precondition, though - just an internal sanity check. I reorganized the method a bit and had it throw AssertionError so it's always enabled.
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@120
PS5, Line 120: @return
> nit: Returns
Done
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@138
PS5, Line 138: @return
> nit: Returns
Done
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java
File fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@30
PS5, Line 30: complet
> nit: complete
Done
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@35
PS5, Line 35: @return
> nit: Returns
Done
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@41
PS5, Line 41: @return
> nit: Returns
Done
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java:
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@a158
PS5, Line 158:
> any cleanup/simplification is a good thing. hopefully this is not one of th
didn't end up being a huge patch. Assuming it passes the exhaustive tests I think it won't be too much yak-shaving
http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@101
PS5, Line 101: If 'allowEmpty' is False, empty partitions are not returned.
> heh... how many negations are here?
:)
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 22:58:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 6
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jun 2018 23:35:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2702/
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 9
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jun 2018 06:26:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10631
to look at the new patch set (#7).
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
This adds a new PrunablePartition interface which HdfsPartition
implements. The interface includes only the partition ID and the
partition key/values.
For the case of the current catalog implementation, this provides no
benefit. However, for LocalCatalog, we want to defer loading partition
information until after pruning. With this interface, we can construct
PrunablePartition objects using just the partition names, and then load
the partitions once pruning is complete.
Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java
18 files changed, 175 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/7
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 7
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 )
Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2702/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/10631
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa
Gerrit-Change-Number: 10631
Gerrit-PatchSet: 9
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jun 2018 03:15:34 +0000
Gerrit-HasComments: No