You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2023/09/21 02:37:23 UTC
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20500
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
In the /metrics webUI, coordinator shows metrics of
"catalog.num-databases" and "catalog.num-tables" for its local catalog
cache. They are updated at the end of each query execution, via
Frontend.getCatalogMetrics().
In LocalCatalog mode, there is no need for every coordinator to have the
full list of tables of every database. However, getCatalogMetrics ends
up iterating over every DB and fetching these lists (if uncached) in
order to provide a count. This introduces unnecessary catalog RPCs at
the end of each query execution. When catalogd is slow/hanging in
processing such coordinator RPCs, simple queries will also be hanging.
This patch removes tracking the db/table count metrics from coordinator
side in LocalCatalog mode. They will always be -1. The count isn't
particularly relevant – if someone wants to keep track of the size of
their catalog they are better off looking at that metric from catalogd.
Tests:
- test_non_compact_catalog_topic_updates uses these two metrics to
detect new catalog updates. Changed it to use "catalog.curr-version"
instead.
Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
---
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_compact_catalog_updates.py
3 files changed, 11 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/20500/1
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20500
to look at the new patch set (#2).
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
In the /metrics webUI, coordinator shows metrics of
"catalog.num-databases" and "catalog.num-tables" for its local catalog
cache. They are updated at the end of each query execution, via
Frontend.getCatalogMetrics().
In LocalCatalog mode, there is no need for every coordinator to have the
full list of tables of every database. However, getCatalogMetrics ends
up iterating over every DB and fetching these lists (if uncached) in
order to provide a count. This introduces unnecessary catalog RPCs at
the end of each query execution. When catalogd is slow/hanging in
processing such coordinator RPCs, simple queries will also be hanging.
This patch removes tracking the db/table count metrics from coordinator
side in LocalCatalog mode. They will always be -1. The count isn't
particularly relevant – if someone wants to keep track of the size of
their catalog they are better off looking at that metric from catalogd.
Tests:
- test_non_compact_catalog_topic_updates uses these two metrics to
detect new catalog updates. Changed it to use "catalog.curr-version"
instead.
Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
---
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_compact_catalog_updates.py
3 files changed, 19 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/20500/2
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14044/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 03:25:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14053/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 20:20:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
In the /metrics webUI, coordinator shows metrics of
"catalog.num-databases" and "catalog.num-tables" for its local catalog
cache. They are updated at the end of each query execution, via
Frontend.getCatalogMetrics().
In LocalCatalog mode, there is no need for every coordinator to have the
full list of tables of every database. However, getCatalogMetrics ends
up iterating over every DB and fetching these lists (if uncached) in
order to provide a count. This introduces unnecessary catalog RPCs at
the end of each query execution. When catalogd is slow/hanging in
processing such coordinator RPCs, simple queries will also be hanging.
This patch removes tracking the db/table count metrics from coordinator
side in LocalCatalog mode. They will always be -1. The count isn't
particularly relevant – if someone wants to keep track of the size of
their catalog they are better off looking at that metric from catalogd.
Tests:
- test_non_compact_catalog_topic_updates uses these two metrics to
detect new catalog updates. Changed it to use "catalog.curr-version"
instead.
Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Reviewed-on: http://gerrit.cloudera.org:8080/20500
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
Tested-by: Wenzhe Zhou <wz...@cloudera.com>
---
0 files changed, 0 insertions(+), 0 deletions(-)
Approvals:
Wenzhe Zhou: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20500/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/20500/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1067
PS2, Line 1067: FeCatalogUtils.populateCacheMetrics(getCatalog(), resp);
Do we need to call FeCatalogUtils.populateCacheMetrics() for non local catalog mode?
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 03:25:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has removed a vote on this change.
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20500/2/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java:
http://gerrit.cloudera.org:8080/#/c/20500/2/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@481
PS2, Line 481: cacheEntrySize.getMedian()
median size was affected when setting resp.num_dbs = -1 and
resp.num_tables = -1
http://gerrit.cloudera.org:8080/#/c/20500/3/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/20500/3/tests/custom_cluster/test_local_catalog.py@520
PS3, Line 520: 1000
this test failed. Should change cache_entry_median_size < 3000
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 19:49:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Wenzhe Zhou, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20500
to look at the new patch set (#3).
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
In the /metrics webUI, coordinator shows metrics of
"catalog.num-databases" and "catalog.num-tables" for its local catalog
cache. They are updated at the end of each query execution, via
Frontend.getCatalogMetrics().
In LocalCatalog mode, there is no need for every coordinator to have the
full list of tables of every database. However, getCatalogMetrics ends
up iterating over every DB and fetching these lists (if uncached) in
order to provide a count. This introduces unnecessary catalog RPCs at
the end of each query execution. When catalogd is slow/hanging in
processing such coordinator RPCs, simple queries will also be hanging.
This patch removes tracking the db/table count metrics from coordinator
side in LocalCatalog mode. They will always be -1. The count isn't
particularly relevant – if someone wants to keep track of the size of
their catalog they are better off looking at that metric from catalogd.
Tests:
- test_non_compact_catalog_topic_updates uses these two metrics to
detect new catalog updates. Changed it to use "catalog.curr-version"
instead.
Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
---
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_compact_catalog_updates.py
3 files changed, 20 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/20500/3
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14043/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 03:04:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9740/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 02:59:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9740/
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 07:33:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14045/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 04:08:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Sep 2023 00:54:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20500/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/20500/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1067
PS2, Line 1067: }
> It's a no-op for the legacy catalog mode. We can move it into the first if-
Since we already check if it's local catalog mode here, we can skip to call the function for non local catalog mode. Please move it to the first if-branch
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 03:50:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 03:56:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#4) to the change originally created by Quanlong Huang. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
In the /metrics webUI, coordinator shows metrics of
"catalog.num-databases" and "catalog.num-tables" for its local catalog
cache. They are updated at the end of each query execution, via
Frontend.getCatalogMetrics().
In LocalCatalog mode, there is no need for every coordinator to have the
full list of tables of every database. However, getCatalogMetrics ends
up iterating over every DB and fetching these lists (if uncached) in
order to provide a count. This introduces unnecessary catalog RPCs at
the end of each query execution. When catalogd is slow/hanging in
processing such coordinator RPCs, simple queries will also be hanging.
This patch removes tracking the db/table count metrics from coordinator
side in LocalCatalog mode. They will always be -1. The count isn't
particularly relevant – if someone wants to keep track of the size of
their catalog they are better off looking at that metric from catalogd.
Tests:
- test_non_compact_catalog_topic_updates uses these two metrics to
detect new catalog updates. Changed it to use "catalog.curr-version"
instead.
Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
---
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_compact_catalog_updates.py
M tests/custom_cluster/test_local_catalog.py
4 files changed, 21 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/20500/4
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Sep 2023 00:38:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9746/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 20:29:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9746/
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Sep 2023 00:45:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 3:
this looks relevant:
https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/433/testReport/junit/custom_cluster.test_local_catalog/TestObservability/test_cache_metrics/
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 11:56:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20500/2/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/20500/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1067
PS2, Line 1067: }
> Do we need to call FeCatalogUtils.populateCacheMetrics() for non local cata
It's a no-op for the legacy catalog mode. We can move it into the first if-branch.
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Sep 2023 03:41:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has removed a vote on this change.
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Removed Code-Review+2 by Wenzhe Zhou <wz...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20500 )
Change subject: IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
......................................................................
Patch Set 4:
one in-relevant test failure, hit IMPALA-11396
--
To view, visit http://gerrit.cloudera.org:8080/20500
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Gerrit-Change-Number: 20500
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Sep 2023 00:53:50 +0000
Gerrit-HasComments: No