You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org> on 2022/03/16 03:14:06 UTC

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Yu-Wen Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18324


Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................

IMPALA-11181: Improving performance of compaction checking

After HIVE-25753, we don't need to explicitly set all partitions' name
to get the latest compaction id. Besides, we can also send the last
compaction id over to HMS so that HMS will send back compaction info
only if there are newer compactions. In this way, we can avoid
unnecessary data transmitted between HMS and Catalogd.

Testing:
existing tests

Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
3 files changed, 24 insertions(+), 21 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 3: Code-Review+1

@Yu-Wen: Thanks for addressing all my comments. Patch looks good to me.


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 23:02:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Mar 2022 00:11:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7933/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Mar 2022 03:15:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10351/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 05:10:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7988/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 5
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 01:23:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10362/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 16:38:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 4: Code-Review+2

LGTM. Thanks for working on this!


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 01:22:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7933/


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Mar 2022 08:10:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................

IMPALA-11181: Improving performance of compaction checking

After HIVE-25753, we don't need to explicitly set all partitions' name
to get the latest compaction id. Besides, we can also send the last
compaction id over to HMS so that HMS will send back compaction info
only if there are newer compactions. In this way, we can avoid
unnecessary data transmitted between HMS and Catalogd.

Testing:
existing tests

Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
3 files changed, 30 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/3
-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@705
PS3, Line 705:       long latestCompactionId = partNameToCompactionId.getOrDefault(
             :           entry.getKey().getName(), -1L);
             :       if (latestCompactionId > 0) {
nit: Can we optimize this to the following case?

 if (partNameToCompactionId.containsKey(entry.getKey().getName()))


http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@295
PS3, Line 295:   // The last committed compaction id in the table level. It will be sent as a filter to
nit: Could you add a blank line before this?



-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 02:04:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 5
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 05:50:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10294/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Mar 2022 03:34:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................

IMPALA-11181: Improving performance of compaction checking

After HIVE-25753, we don't need to explicitly set all partitions' name
to get the latest compaction id. Besides, we can also send the last
compaction id over to HMS so that HMS will send back compaction info
only if there are newer compactions. In this way, we can avoid
unnecessary data transmitted between HMS and Catalogd.

Testing:
existing tests

Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Reviewed-on: http://gerrit.cloudera.org:8080/18324
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
3 files changed, 31 insertions(+), 25 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 6
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................

IMPALA-11181: Improving performance of compaction checking

After HIVE-25753, we don't need to explicitly set all partitions' name
to get the latest compaction id. Besides, we can also send the last
compaction id over to HMS so that HMS will send back compaction info
only if there are newer compactions. In this way, we can avoid
unnecessary data transmitted between HMS and Catalogd.

Testing:
existing tests

Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
3 files changed, 29 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10317/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Mar 2022 21:56:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@705
PS3, Line 705:       if (partNameToCompactionId.containsKey(entry.getKey().getName())) {
             :         stalePartitions.add(entry.getKey());
             :         iter.remove();
> nit: Can we optimize this to the following case?
Done


http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@295
PS3, Line 295: 
> nit: Could you add a blank line before this?
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 16:20:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................

IMPALA-11181: Improving performance of compaction checking

After HIVE-25753, we don't need to explicitly set all partitions' name
to get the latest compaction id. Besides, we can also send the last
compaction id over to HMS so that HMS will send back compaction info
only if there are newer compactions. In this way, we can avoid
unnecessary data transmitted between HMS and Catalogd.

Testing:
existing tests

Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
3 files changed, 31 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/4
-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 2:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@297
PS2, Line 297:   private long lastCompactionId_ = -1;
Please add some more description on the use of this variable, how it is supposed to get updated etc


http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@974
PS2, Line 974:     if (partition.getLastCompactionId() > 0) {
nit: we can skip the check since the following condition is sufficient: 
lastCompactionId_ = Math.max(lastCompactionId_, partition.getLastCompactionId());


http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@485
PS2, Line 485:       // Compaction id is not provided with DirectMetaProvider implementation
Why are we changing the old behavior?


http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@538
PS2, Line 538:     if (metas.isEmpty()) {
nit: We can skip this check since checkLatestCompaction() already checks metas size.


http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@543
PS2, Line 543:     List<PartitionRef> stalePartitions = MetastoreShim.checkLatestCompaction(
nit: if we want to log the time that checkLatestCompaction() takes, it would be better to move the Stopwatch check in MetastoreShim.checkLatestCompaction()



-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Mar 2022 00:16:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7955/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Mar 2022 19:22:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7950/


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Mar 2022 02:33:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7950/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Mar 2022 21:37:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7978/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 04:50:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Mar 2022 09:39:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 )

Change subject: IMPALA-11181: Improving performance of compaction checking
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 5
Gerrit-Owner: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 01:23:02 +0000
Gerrit-HasComments: No