You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2022/08/29 14:11:41 UTC

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18926


Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................

IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload

ALTER_TABLE event could trigger full refresh on the table, which is
time consuming and can lead to event-processor lagging behind. We need
logs to show the difference between the before and after hmsTables so we
can further distinguish the cases that actully don't need a full table
refresh, e.g. changing the owner, changing tblproperties, etc.

Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
1 file changed, 34 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18926/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has removed a vote on this change.

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Sep 2022 15:22:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Yu-Wen Lai, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18926

to look at the new patch set (#2).

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................

IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload

ALTER_TABLE event could trigger full refresh on the table, which is
time consuming and can lead to event-processor lagging behind. We need
logs to show the difference between the before and after hmsTables so we
can further distinguish the cases that actully don't need a full table
refresh, e.g. changing the owner, changing tblproperties, etc.

Tests:
 - Attach the process using IntelliJ to pause the metadata loading for a
   while. So the metadata reloading of ALTER_TABLE event can exceed the
   threshold. Manually verified the logs.

Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
1 file changed, 49 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18926/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18926/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/18926/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1462
PS3, Line 1462:       long duration = System.currentTimeMillis() - start_ms;
Hi quanlong, System.nanoTime() is more accurate than System.currentTimeMillis(), and the result has same unit with LOADING_WARNING_TIME_NS.



-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 02:49:36 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8552/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 22:23:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8542/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 00:05:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 5: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 15:38:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Yu-Wen Lai (Code Review)" <ge...@cloudera.org>.
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 1: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 12:03:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 09:06:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11281/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Sun, 04 Sep 2022 05:38:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11311/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 08:29:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8552/


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Sep 2022 03:12:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 6: Verified+1

> Patch Set 6: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8552/

The failure is https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/6288
while is due to a flaky test: IMPALA-11508.
Merging this.


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Sep 2022 05:10:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 6: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 22:23:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Yu-Wen Lai, Impala Public Jenkins, Xiang Yang, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18926

to look at the new patch set (#5).

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................

IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload

ALTER_TABLE event could trigger full refresh on the table, which is
time consuming and can lead to event-processor lagging behind. We need
logs to show the difference between the before and after hmsTables so we
can further distinguish the cases that actully don't need a full table
refresh, e.g. changing the owner, changing tblproperties, etc.

Tests:
 - Attach the process using IntelliJ to pause the metadata loading for a
   while. So the metadata reloading of ALTER_TABLE event can exceed the
   threshold. Manually verified the logs.

Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
1 file changed, 20 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18926/5
-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8542/


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 04:48:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11312/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 08:38:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11240/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 14:32:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 1: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 11:50:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Yu-Wen Lai, Impala Public Jenkins, Xiang Yang, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18926

to look at the new patch set (#4).

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................

IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload

ALTER_TABLE event could trigger full refresh on the table, which is
time consuming and can lead to event-processor lagging behind. We need
logs to show the difference between the before and after hmsTables so we
can further distinguish the cases that actully don't need a full table
refresh, e.g. changing the owner, changing tblproperties, etc.

Tests:
 - Attach the process using IntelliJ to pause the metadata loading for a
   while. So the metadata reloading of ALTER_TABLE event can exceed the
   threshold. Manually verified the logs.

Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
1 file changed, 20 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18926/4
-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................

IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload

ALTER_TABLE event could trigger full refresh on the table, which is
time consuming and can lead to event-processor lagging behind. We need
logs to show the difference between the before and after hmsTables so we
can further distinguish the cases that actully don't need a full table
refresh, e.g. changing the owner, changing tblproperties, etc.

Tests:
 - Attach the process using IntelliJ to pause the metadata loading for a
   while. So the metadata reloading of ALTER_TABLE event can exceed the
   threshold. Manually verified the logs.

Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Reviewed-on: http://gerrit.cloudera.org:8080/18926
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Quanlong Huang <hu...@gmail.com>
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
1 file changed, 20 insertions(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved
  Quanlong Huang: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 3:

(2 comments)

> Patch Set 3: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8542/

The failure is due to compilation error on Apache Hive 3.x:
https://jenkins.impala.io/job/all-build-options-ub1604/8722/console

http://gerrit.cloudera.org:8080/#/c/18926/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/18926/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1462
PS3, Line 1462:       long duration = System.currentTimeMillis() - start_ms;
> Hi quanlong, System.nanoTime() is more accurate than System.currentTimeMill
Done


http://gerrit.cloudera.org:8080/#/c/18926/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1474
PS3, Line 1474:       if (tableBefore_.isSetColStats() || tableAfter_.isSetColStats()) {
The colStats field is added in HIVE-21078 which is not in Hive-3.1.2. This causes compilation error when building with Apache Hive 3.x.

In my experiments, the colStats is always null in the AlterTable events. It seems it's not set into the event, at least for the oldTable object:

https://github.com/apache/hive/blob/4e4e39c471094567dcdfd9840edbd99d7eafc230/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L6117
https://github.infra.cloudera.com/CDH/hive/blob/64234e1217b105ac74216e7877a75a14b141ee6d/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L3733

Hive also have codes logging the Table#toString() directly, e.g.
https://github.com/apache/hive/blob/4e4e39c471094567dcdfd9840edbd99d7eafc230/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L708
https://github.com/apache/hive/blob/4e4e39c471094567dcdfd9840edbd99d7eafc230/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L766

BTW, I think even if we log the stats, the complexity is the same as logging the column schema. There are no large binaries like incremental stats of Impala.

So I think it's ok to ignore these checks.



-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 07:42:06 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11540: Add logs for ALTER TABLE events that trigger slow metadata reload

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )

Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18926
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce
Gerrit-Change-Number: 18926
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Sep 2022 00:05:24 +0000
Gerrit-HasComments: No