You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2024/08/09 17:06:04 UTC
[Impala-ASF-CR] WIP IMPALA-13293: Fix too long wait for initial catalog update
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21655
Change subject: WIP IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
WIP IMPALA-13293: Fix too long wait for initial catalog update
Makes starting impala cluster faster:
ImpaladCatalog: 12s->11s
LocalCatalog: 11->10s
Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
---
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
7 files changed, 47 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/21655/1
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 3: Code-Review+1
(1 comment)
Thanks for working on this! The change looks good to me, but I have some confusion at one part.
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1889
PS3, Line 1889: // Avoid calling getCatalog again in this attempt as it may return a different
: // ImpaladCatalog if a full topic update has arrived.
Wouldn't it be better to wait on the new catalog then?
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 16:54:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 4: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1889
PS3, Line 1889: while (true) {
: // Avoid calling getCatalog again in this attempt as
> I thought that it is easier to understand this way.
Thanks for the explanation (though I might have understood only half of it). I'm OK with keeping this change minimal, which is probably safer as well, though it'd be nice to make this logic clearer/simpler in the future.
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Aug 2024 13:49:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16681/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Aug 2024 08:27:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: WIP IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16650/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Aug 2024 17:40:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: WIP IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Aug 2024 22:16:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21655/1/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/21655/1/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@72
PS1, Line 72: * Allows waiting for isReady() to become true. May return early, so
> nit: early
Done
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 05:55:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
IMPALA-13293: Fix too long wait for initial catalog update
Coordinators always waited 2s in Frontend.WaitForCatalog() for the
first catalog update to arrive, even if the update arrived quicker.
This was obvious in case of LocalCatalog, but ImpaladCatalog looked
as if it would wake up when the update arrives by notifying
catalogUpdateEventNotifier_. In reality the arrival of the catalog
update led FeCatalogManager to create a new ImpaladCatalog instance
while Frontend was still waiting for the old instance which never
got notified.
The change fixes the wait/notify mechanism in ImpaladCatalog and
implements a similar logic for LocalCatalog in CatalogdMetaProvider.
Makes starting impala cluster faster:
ImpaladCatalog: ~12s->~11s
LocalCatalog: ~11->~10s
Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Reviewed-on: http://gerrit.cloudera.org:8080/21655
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
7 files changed, 54 insertions(+), 13 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Aug 2024 18:05:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16667/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 06:25:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Aug 2024 13:03:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 4: Code-Review+2
LGTM. Carry Zoltan's +1.
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Aug 2024 12:16:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21655/1/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/21655/1/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@72
PS1, Line 72: * Allows waiting for isReady() to become true. May return earlier, so
nit: early
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 20:56:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21655
to look at the new patch set (#3).
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
IMPALA-13293: Fix too long wait for initial catalog update
Coordinators always waited 2s in Frontend.WaitForCatalog() for the
first catalog update to arrive, even if the update arrived quicker.
This was obvious in case of LocalCatalog, but ImpaladCatalog looked
as if it would wake up when the update arrives by notifying
catalogUpdateEventNotifier_. In reality the arrival of the catalog
update led FeCatalogManager to create a new ImpaladCatalog instance
while Frontend was still waiting for the old instance which never
got notified.
The change fixes the wait/notify mechanism in ImpaladCatalog and
implements a similar logic for LocalCatalog in CatalogdMetaProvider.
Makes starting impala cluster faster:
ImpaladCatalog: ~12s->~11s
LocalCatalog: ~11->~10s
Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
---
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
7 files changed, 49 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/21655/3
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10864/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Aug 2024 13:03:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16662/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 19:34:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21655
to look at the new patch set (#4).
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
IMPALA-13293: Fix too long wait for initial catalog update
Coordinators always waited 2s in Frontend.WaitForCatalog() for the
first catalog update to arrive, even if the update arrived quicker.
This was obvious in case of LocalCatalog, but ImpaladCatalog looked
as if it would wake up when the update arrives by notifying
catalogUpdateEventNotifier_. In reality the arrival of the catalog
update led FeCatalogManager to create a new ImpaladCatalog instance
while Frontend was still waiting for the old instance which never
got notified.
The change fixes the wait/notify mechanism in ImpaladCatalog and
implements a similar logic for LocalCatalog in CatalogdMetaProvider.
Makes starting impala cluster faster:
ImpaladCatalog: ~12s->~11s
LocalCatalog: ~11->~10s
Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
---
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
7 files changed, 54 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/21655/4
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1882
PS3, Line 1882: *
> nit: we can also add CatalogdMetaProvider#isReady() in this comment
done, also extended the function comment to mention how some tests do this differently
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1889
PS3, Line 1889: while (true) {
: // Avoid calling getCatalog again in this attempt as
> Wouldn't it be better to wait on the new catalog then?
I thought that it is easier to understand this way.
By getting the new catalog there is a very small chance to be faster if the catalog is changed between the two getCatalog() calls. But even that is not perfect - the catalog update can arrive between isReady() and waitForCatalogUpdate(), which can lead to an unnecessary wait of 2s if the catalogReadyNotifier_/catalogUpdateEventNotifier_ are already notified when waitForCatalogUpdate() starts to wait for them. This likely to be very rare and (thanks to the timeout) it can only hang for a limited time.
I think that the clearer solution would be to refactor this:
- add waitForReadyCatalog() and setIsReady () to FeCatalogManager
- use waitForReadyCatalog() above instead of waiting for the FeCatalog directly
- call FeCatalogManager.setIsReady() above in JniFrontend instead of calling FeCatalog.setIsReady() directly
As FeCatalogManager knows when a catalog update arrives, it could avoid the timing issue in the first paragraph.
My plan was to make this change minimal, but I can do small refactor if needed.
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Aug 2024 08:12:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: WIP IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10851/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Aug 2024 17:06:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21655
to look at the new patch set (#2).
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
IMPALA-13293: Fix too long wait for initial catalog update
Coordinators always waited 2s in Frontend.WaitForCatalog() for the
first catalog update to arrive, even if the update arrived quicker.
This was obvious in case of LocalCatalog, but ImpaladCatalog looked
as if it would wake up when the update arrives by notifying
catalogUpdateEventNotifier_. In reality the arrival of the catalog
update led FeCatalogManager to create a new ImpaladCatalog instance
while Frontend was still waiting for the old instance which never
got notified.
The change fixes the wait/notify mechanism in ImpaladCatalog and
implements a similar logic for LocalCatalog in CatalogdMetaProvider.
Makes starting impala cluster faster:
ImpaladCatalog: ~12s->~11s
LocalCatalog: ~11->~10s
Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
---
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
7 files changed, 49 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/21655/2
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalog.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/FeCatalog.java@103
PS3, Line 103: void waitForCatalogUpdate(long timeoutMs);
Can we add a comment saying this is only used in startup? If it's used after impala is ready, things might be a bit complex in dealing with catalogd restarts or catalogd HA failover.
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@425
PS3, Line 425: }
Can we add a Preconditions check to make sure isReady() is true at the end?
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 07:49:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalog.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/FeCatalog.java@103
PS3, Line 103: void waitForCatalogUpdate(long timeoutMs);
> Can we add a comment saying this is only used in startup? If it's used afte
It is actually used after startup at a single call site:
https://github.com/apache/impala/blob/d8a8412c2b750937a3577b08d81ffd9a16269b83/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java#L240
I tried to not change the behavior in that case - LocalCatalog will return immediately as it is already ready, while ImpaladCatalog will wait for the next catalog update to notify it. If a full catalog update is received after startup, I think the same speedup will apply as during startup.
I think that it is confusing at this point, as in ImpaladCatalog this is really meant to wait for an update, while in LocalCatalog it is NOOP unless it is the initial startup (or probably it will also wait after INVALIDATE METADATA or catalog service id change?). I was thinking about refactoring it, but it seemed better to wait for non-local catalog support to be dropped before doing it (probably the original authors thought the same :) )
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@425
PS3, Line 425: }
> Can we add a Preconditions check to make sure isReady() is true at the end?
If the timeout is reached, then it may not be ready at this point. Both places that call waitForCatalogUpdate() do it in a loop and will wait again for 2s for the catalog to be ready.
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 13:40:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13293: Fix too long wait for initial catalog update
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21655 )
Change subject: IMPALA-13293: Fix too long wait for initial catalog update
......................................................................
Patch Set 3: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalog.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/FeCatalog.java@103
PS3, Line 103: void waitForCatalogUpdate(long timeoutMs);
> It is actually used after startup at a single call site:
Ack
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@425
PS3, Line 425: }
> If the timeout is reached, then it may not be ready at this point. Both pla
Ack
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/21655/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1882
PS3, Line 1882: * @see ImpaladCatalog#isReady()
nit: we can also add CatalogdMetaProvider#isReady() in this comment
--
To view, visit http://gerrit.cloudera.org:8080/21655
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4da1d0c0d25f80b02c39be863431201d1b913c5d
Gerrit-Change-Number: 21655
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Aug 2024 00:15:18 +0000
Gerrit-HasComments: Yes