You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Pranay Singh (Code Review)" <ge...@cloudera.org> on 2018/05/18 15:48:44 UTC
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Pranay Singh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10450
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
......................................................................
IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Catalog currently loads the HMS data even, if only the file metadata load
is requested. This change detects some of those updates and tries to skip
the loading of HMS data if only the file metadata load is requested.
Testing:
Ran the core test without failures.
Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
1 file changed, 39 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/10450/1
--
To view, visit http://gerrit.cloudera.org:8080/10450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
Gerrit-Change-Number: 10450
Gerrit-PatchSet: 1
Gerrit-Owner: Pranay Singh
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
......................................................................
Patch Set 2:
> > (2 comments)
> >
> > Just interested in this optimization. May I ask some questions?
> >
> > Looks like we are optimizing the case when partitionsToUpdate !=
> > null and partitions were neither dropped, created. Can we
> optimize
> > the case that partitionsToUpdate != null and some partitions are
> > dropped? For example when an INSERT OVERWRITE statement updates
> the
> > majority of the partitions and only drops few of them.
>
> Will this case not introduce inconsistency between HMS and Impala?
The problem with optimization is that introduces inconsistency, something which my change introduces too, when ALTER TABLE is done and HMS crashes.
--
To view, visit http://gerrit.cloudera.org:8080/10450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
Gerrit-Change-Number: 10450
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 19 May 2018 18:18:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
......................................................................
Patch Set 2:
(2 comments)
> (2 comments)
>
> Just interested in this optimization. May I ask some questions?
>
> Looks like we are optimizing the case when partitionsToUpdate !=
> null and partitions were neither dropped, created. Can we optimize
> the case that partitionsToUpdate != null and some partitions are
> dropped? For example when an INSERT OVERWRITE statement updates the
> majority of the partitions and only drops few of them.
Will this case not cause introduce inconsistency between HMS and Impala ?
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1409
PS2, Line 1409: size() == 0
> nit: can be simplified by isEmpty()
OK
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1410
PS2, Line 1410: partitionsToUpdateFileMdByPath = getPartitionsByPath(partitionsToUpdate);
: loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true);
> Looks like the original codes perform the same as these two lines. Since dr
The new change behaves very much like the old code except for the case when dirtyPartitions exist in that case there is an overhead of dropping and loading the dirty partitions from Metastore.
--
To view, visit http://gerrit.cloudera.org:8080/10450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
Gerrit-Change-Number: 10450
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 19 May 2018 18:14:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1410
PS2, Line 1410: partitionsToUpdateFileMdByPath = getPartitionsByPath(partitionsToUpdate);
: loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true);
> I don't think so. In line 1404, the dirtyPartitions are all added to partit
Yes missed on that one, I'll rollback this change.
--
To view, visit http://gerrit.cloudera.org:8080/10450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
Gerrit-Change-Number: 10450
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 21 May 2018 15:53:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
......................................................................
Patch Set 2:
> (2 comments)
>
> Just interested in this optimization. May I ask some questions?
>
> Looks like we are optimizing the case when partitionsToUpdate !=
> null and partitions were neither dropped, created. Can we optimize
> the case that partitionsToUpdate != null and some partitions are
> dropped? For example when an INSERT OVERWRITE statement updates the
> majority of the partitions and only drops few of them.
Will this case not introduce inconsistency between HMS and Impala?
--
To view, visit http://gerrit.cloudera.org:8080/10450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
Gerrit-Change-Number: 10450
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 19 May 2018 18:15:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
......................................................................
Patch Set 2:
(2 comments)
Just interested in this optimization. May I ask some questions?
Looks like we are optimizing the case when partitionsToUpdate != null and partitions were neither dropped, created. Can we optimize the case that partitionsToUpdate != null and some partitions are dropped? For example when an INSERT OVERWRITE statement updates the majority of the partitions and only drops few of them.
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1409
PS2, Line 1409: size() == 0
nit: can be simplified by isEmpty()
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1410
PS2, Line 1410: partitionsToUpdateFileMdByPath = getPartitionsByPath(partitionsToUpdate);
: loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true);
Looks like the original codes perform the same as these two lines. Since dropPartitions, loadPartitionsFromMetastore and loadPartitionsFromMetastore will return fast when their first parameter is empty. partitionExist is not empty means partitionsToUpdate != null, so the original codes will perform as these two lines.
My question is what are the HMS requests we save?
--
To view, visit http://gerrit.cloudera.org:8080/10450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
Gerrit-Change-Number: 10450
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 19 May 2018 13:17:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1410
PS2, Line 1410: partitionsToUpdateFileMdByPath = getPartitionsByPath(partitionsToUpdate);
: loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true);
> The new change behaves very much like the old code except for the case when
I don't think so. In line 1404, the dirtyPartitions are all added to partitionsToRemove. When dirtyPartitions exist, partitionsToRemove won't be empty. Thus the if-branch won't be chosen and the else-branch performs the same as the old codes.
--
To view, visit http://gerrit.cloudera.org:8080/10450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
Gerrit-Change-Number: 10450
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 20 May 2018 01:27:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
Posted by "Pranay Singh (Code Review)" <ge...@cloudera.org>.
Pranay Singh has abandoned this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations.
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/10450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a
Gerrit-Change-Number: 10450
Gerrit-PatchSet: 2
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>