You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org> on 2018/09/25 20:27:13 UTC
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11511
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
IMPALA-7531: Daemon level catalog cache metrics
This patch adds the aggregated CatalogdMetaProvider cache stats to
the catalog metrics on the coordinators. They can be accessed under
<coordinator>:<web-port>/metrics#catalog.
These metrics are refreshed at the end of planning, for each query run.
Testing:
-------
Visual inspection by running a few queries locally and making sure
stats are updated. Also modified existing tests to account for this
behavior.
Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
---
M be/src/service/impala-server.cc
M be/src/util/impalad-metrics.cc
M be/src/util/impalad-metrics.h
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_local_catalog.py
10 files changed, 391 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/11511/1
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/852/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Sep 2018 22:04:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/service/impala-server.cc@1212
PS1, Line 1212: e_miss_count
> if there's a miss, there's a load? just wondering it they're redundant. I s
Yea, this is interesting. I'm pretty sure that this related to the way we are loading partitions. I ran a bunch of other queries and the hit/load was 1:1 and then I loaded a partitioned table and it diverged.
We load partitions differently from other metadata. We make an initial pass which checks if the cache has given partitions (hits/misses recorded) and then we bulk load the missing ones and put them into cache one at a time. It is not super clear to me how Guava accounts for this. Not super sure if puts() this way are incremented for loadSuccess.
https://github.com/google/guava/blob/v14.0/guava/src/com/google/common/cache/LocalCache.java#L2885
I can dig into it, but will keep it as is for now.
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/util/impalad-metrics.h
File be/src/util/impalad-metrics.h:
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/util/impalad-metrics.h@134
PS1, Line 134: /// Ratio of Impalad Catalog cache requests that were hits.
> Is this since the beginning of time?
Yes, clarified.
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/util/impalad-metrics.cc
File be/src/util/impalad-metrics.cc:
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/util/impalad-metrics.cc@211
PS1, Line 211: catalog_metrics->AddDoubleGauge(ImpaladMetricKeys::CATALOG_CACHE_AVG_LOAD_TIME, 0);
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/11511/1/common/thrift/metrics.json
File common/thrift/metrics.json:
http://gerrit.cloudera.org:8080/#/c/11511/1/common/thrift/metrics.json@1697
PS1, Line 1697: "key": "catalog.cache.average-load-time"
> I think it's super confusing that a metric for Impalad starts with "catalog
Doesn't the context help here? We have been doing it like this for a few other similar metrics, say catalog.ready, catalog.curr-version, catalog.num-tables etc... I'm afraid changing all of them might break some monitoring tools?
http://gerrit.cloudera.org:8080/#/c/11511/1/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java:
http://gerrit.cloudera.org:8080/#/c/11511/1/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@370
PS1, Line 370: if (!(provider instanceof CatalogdMetaProvider)) return;
> Not now, but there is another one called DirectMetaProvider that's easy to
Yep, I added this check for DirectMetaProvider.
http://gerrit.cloudera.org:8080/#/c/11511/1/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11511/1/tests/custom_cluster/test_local_catalog.py@35
PS1, Line 35: def get_debug_page(self, page_url):
: """Returns the content of the debug page 'page_url' as json."""
: response = requests.get(page_url)
: assert response.status_code == requests.codes.ok
: return json.loads(response.text)
> use get_debug_webpage_json from impala_service.py
Done
http://gerrit.cloudera.org:8080/#/c/11511/1/tests/custom_cluster/test_local_catalog.py@50
PS1, Line 50: \
> flake8: E502 the backslash is redundant between brackets
Done
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 22:58:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 5: Code-Review+2
(1 comment)
Carrying +2.
http://gerrit.cloudera.org:8080/#/c/11511/4/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11511/4/tests/custom_cluster/test_local_catalog.py@253
PS4, Line 253: "create table %s (a int)" % test_table_name,
> Not a big deal, but I think we have some annotation for temporary tables ("
Done
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 5
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Sep 2018 22:22:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Hello Philip Zeyliger, Impala Public Jenkins, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11511
to look at the new patch set (#3).
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
IMPALA-7531: Daemon level catalog cache metrics
This patch adds the aggregated CatalogdMetaProvider cache stats to
the catalog metrics on the coordinators. They can be accessed under
<coordinator>:<web-port>/metrics#catalog.
These metrics are refreshed at the end of planning, for each query run.
Testing:
-------
Visual inspection by running a few queries locally and making sure
stats are updated. Also modified existing tests to account for this
behavior.
Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
---
M be/src/service/impala-server.cc
M be/src/util/impalad-metrics.cc
M be/src/util/impalad-metrics.h
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_local_catalog.py
10 files changed, 387 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/11511/3
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 5
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Sep 2018 02:15:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/801/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 23:36:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/util/impalad-metrics.h
File be/src/util/impalad-metrics.h:
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/util/impalad-metrics.h@134
PS1, Line 134: /// Ratio of Impalad Catalog cache requests that were hits.
Is this since the beginning of time?
http://gerrit.cloudera.org:8080/#/c/11511/1/common/thrift/metrics.json
File common/thrift/metrics.json:
http://gerrit.cloudera.org:8080/#/c/11511/1/common/thrift/metrics.json@1697
PS1, Line 1697: "key": "catalog.cache.average-load-time"
I think it's super confusing that a metric for Impalad starts with "catalog". I worry that it'll cause some confusion. Perhaps start them with "impalad_catalog" or something?
http://gerrit.cloudera.org:8080/#/c/11511/1/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java:
http://gerrit.cloudera.org:8080/#/c/11511/1/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@370
PS1, Line 370: if (!(provider instanceof CatalogdMetaProvider)) return;
Is this possible?
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 20:41:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 3: Code-Review+2
(1 comment)
comment about sync'ing the text in the .h and json files, otherwise, lgtm.
http://gerrit.cloudera.org:8080/#/c/11511/3/common/thrift/metrics.json
File common/thrift/metrics.json:
http://gerrit.cloudera.org:8080/#/c/11511/3/common/thrift/metrics.json@1720
PS3, Line 1720: "description": "Ratio of Impalad Catalog cache requests that were hits.",
are you sync'ing these to the comments in the .h file? This one was updated.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Sep 2018 00:08:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/800/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 23:35:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11511/1/common/thrift/metrics.json
File common/thrift/metrics.json:
http://gerrit.cloudera.org:8080/#/c/11511/1/common/thrift/metrics.json@1697
PS1, Line 1697: "key": "catalog.cache.average-load-time"
> Doesn't the context help here? We have been doing it like this for a few ot
Yeah, looks like we've already crossed that bridge on trunk.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Sep 2018 21:06:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/854/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 5
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Sep 2018 22:56:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Hello Philip Zeyliger, Impala Public Jenkins, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11511
to look at the new patch set (#5).
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
IMPALA-7531: Daemon level catalog cache metrics
This patch adds the aggregated CatalogdMetaProvider cache stats to
the catalog metrics on the coordinators. They can be accessed under
<coordinator>:<web-port>/metrics#catalog.
These metrics are refreshed at the end of planning, for each query run.
Testing:
-------
Visual inspection by running a few queries locally and making sure
stats are updated. Also modified existing tests to account for this
behavior.
Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/util/impalad-metrics.cc
M be/src/util/impalad-metrics.h
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_local_catalog.py
11 files changed, 402 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/11511/5
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 5
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/service/impala-server.cc@1212
PS1, Line 1212: e_miss_count
if there's a miss, there's a load? just wondering it they're redundant. I see from the screenshot that there are 16 misses, but 14 loads. What happened with the other two misses?
http://gerrit.cloudera.org:8080/#/c/11511/1/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java:
http://gerrit.cloudera.org:8080/#/c/11511/1/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@370
PS1, Line 370: if (!(provider instanceof CatalogdMetaProvider)) return;
> Is this possible?
Not now, but there is another one called DirectMetaProvider that's easy to swap in. I think it makes sense to keep this here as-is or move it to the interface.
http://gerrit.cloudera.org:8080/#/c/11511/1/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11511/1/tests/custom_cluster/test_local_catalog.py@35
PS1, Line 35: def get_debug_page(self, page_url):
: """Returns the content of the debug page 'page_url' as json."""
: response = requests.get(page_url)
: assert response.status_code == requests.codes.ok
: return json.loads(response.text)
use get_debug_webpage_json from impala_service.py
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 21:09:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 1:
Example screenshot: https://pasteboard.co/HFyOX0G.png
Will address flake warnings in the next PS.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 20:28:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/793/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 21:00:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 3:
(3 comments)
Took another pass. I had to scratch my head a little about how the updates work. I think this wouldn't update metrics for "create table" statements, or "explain" statements, and I think that's probably a bug. (Or likely I'm missing something.)
http://gerrit.cloudera.org:8080/#/c/11511/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11511/3//COMMIT_MSG@13
PS3, Line 13: These metrics are refreshed at the end of planning, for each query run.
This is pretty weird. Can you update impala-server.h to include a note on the docs for UpdateCatalogMetrics() about when they're updated? I think they're updated at the end of planning as well as when there are statestore callbacks?
http://gerrit.cloudera.org:8080/#/c/11511/3/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/11511/3/common/thrift/Frontend.thrift@116
PS3, Line 116: 12: optional double cache_hit_rate
You answered my question about whether this is since beginning of time; could you answer it here as a comment?
Thanks!
http://gerrit.cloudera.org:8080/#/c/11511/3/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11511/3/tests/custom_cluster/test_local_catalog.py@250
PS3, Line 250: query = "select count(*) from functional.alltypes"
Different query types have slightly different paths from impala-server.cc.
(e.g.,:
if (result.stmt_type == TStmtType::DDL) {
Status status = UpdateCatalogMetrics();
)
Do we need to check that these are getting updated for a variety of different queries? E.g., "explain", "select", and "create table"?
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Sep 2018 21:19:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Hello Philip Zeyliger, Impala Public Jenkins, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11511
to look at the new patch set (#2).
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
IMPALA-7531: Daemon level catalog cache metrics
This patch adds the aggregated CatalogdMetaProvider cache stats to
the catalog metrics on the coordinators. They can be accessed under
<coordinator>:<web-port>/metrics#catalog.
These metrics are refreshed at the end of planning, for each query run.
Testing:
-------
Visual inspection by running a few queries locally and making sure
stats are updated. Also modified existing tests to account for this
behavior.
Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
---
M be/src/service/impala-server.cc
M be/src/util/impalad-metrics.cc
M be/src/util/impalad-metrics.h
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_local_catalog.py
10 files changed, 389 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/11511/2
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11511/2/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11511/2/tests/custom_cluster/test_local_catalog.py@20
PS2, Line 20: import json
flake8: F401 'json' imported but unused
http://gerrit.cloudera.org:8080/#/c/11511/2/tests/custom_cluster/test_local_catalog.py@23
PS2, Line 23: import requests
flake8: F401 'requests' imported but unused
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 2
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 23:00:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 3:
(1 comment)
Phil, do you have any comments?
http://gerrit.cloudera.org:8080/#/c/11511/3/common/thrift/metrics.json
File common/thrift/metrics.json:
http://gerrit.cloudera.org:8080/#/c/11511/3/common/thrift/metrics.json@1720
PS3, Line 1720: "description": "Ratio of Impalad Catalog cache requests that were hits.",
> are you sync'ing these to the comments in the .h file? This one was updated
I'll keep it this way since the description pops up in the web UI and it should be concise. The header comment was more for those reading the code.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Sep 2018 00:56:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 4: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11511/4/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11511/4/tests/custom_cluster/test_local_catalog.py@253
PS4, Line 253: "create table functional.%s (a int)" % test_table_name,
Not a big deal, but I think we have some annotation for temporary tables ("unique_database") that we should maybe be using to avoid conflicts.
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Sep 2018 22:08:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/util/impalad-metrics.cc
File be/src/util/impalad-metrics.cc:
http://gerrit.cloudera.org:8080/#/c/11511/1/be/src/util/impalad-metrics.cc@211
PS1, Line 211: catalog_metrics->AddDoubleGauge(ImpaladMetricKeys::CATALOG_CACHE_AVG_LOAD_TIME, 0);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/11511/1/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11511/1/tests/custom_cluster/test_local_catalog.py@50
PS1, Line 50: \
flake8: E502 the backslash is redundant between brackets
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 1
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Sep 2018 20:27:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11511/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11511/3//COMMIT_MSG@13
PS3, Line 13: These metrics are refreshed at the end of planning, for each query run.
> This is pretty weird. Can you update impala-server.h to include a note on t
Done, I made it to update metrics regardless of the type of query.
http://gerrit.cloudera.org:8080/#/c/11511/3/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/11511/3/common/thrift/Frontend.thrift@116
PS3, Line 116: 12: optional double cache_hit_rate
> You answered my question about whether this is since beginning of time; cou
Added it in the header, updated here too.
http://gerrit.cloudera.org:8080/#/c/11511/3/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/11511/3/tests/custom_cluster/test_local_catalog.py@250
PS3, Line 250: # A mix of queries of various types.
> Different query types have slightly different paths from impala-server.cc.
Done
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Sep 2018 21:31:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3239/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 5
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Sep 2018 22:22:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Hello Philip Zeyliger, Impala Public Jenkins, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11511
to look at the new patch set (#4).
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
IMPALA-7531: Daemon level catalog cache metrics
This patch adds the aggregated CatalogdMetaProvider cache stats to
the catalog metrics on the coordinators. They can be accessed under
<coordinator>:<web-port>/metrics#catalog.
These metrics are refreshed at the end of planning, for each query run.
Testing:
-------
Visual inspection by running a few queries locally and making sure
stats are updated. Also modified existing tests to account for this
behavior.
Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/util/impalad-metrics.cc
M be/src/util/impalad-metrics.h
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_local_catalog.py
11 files changed, 402 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/11511/4
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7531: Daemon level catalog cache metrics
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11511 )
Change subject: IMPALA-7531: Daemon level catalog cache metrics
......................................................................
IMPALA-7531: Daemon level catalog cache metrics
This patch adds the aggregated CatalogdMetaProvider cache stats to
the catalog metrics on the coordinators. They can be accessed under
<coordinator>:<web-port>/metrics#catalog.
These metrics are refreshed at the end of planning, for each query run.
Testing:
-------
Visual inspection by running a few queries locally and making sure
stats are updated. Also modified existing tests to account for this
behavior.
Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Reviewed-on: http://gerrit.cloudera.org:8080/11511
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/util/impalad-metrics.cc
M be/src/util/impalad-metrics.h
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_local_catalog.py
11 files changed, 402 insertions(+), 30 deletions(-)
Approvals:
Bharath Vissapragada: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/11511
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I23c131b77ca84aa4df8919213bbd83944fa112a5
Gerrit-Change-Number: 11511
Gerrit-PatchSet: 6
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>