You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tamas Mate (Code Review)" <ge...@cloudera.org> on 2019/07/08 13:51:34 UTC
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13806
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663 Expose current DDL metrics on WebUI
This change adds two new tables to the Catalog WebUI's '/catalog' page.
These tables are exposing the number of running catalog operations,
which helps to better understand the catalog load and reason when the
catalog is overloaded.
A new 'monitor' package is created to collect the monitoring
sub-classes. The new counters are called just after JNI methods are
invoked, so it can notice the exceptions. The existing thrift structure
is extended with the new counters, these values are unwrapped and
displayed on the WebUI.
Before/after pictures of the /catalog page can be found in the Jira.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
---
M be/src/catalog/catalog-server.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogUsageMonitor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M www/catalog.tmpl
11 files changed, 360 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/3
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5404/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 16
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2020 13:35:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 11:12:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663 Expose current DDL metrics on WebUI
This change adds two new tables to the Catalog WebUI's '/catalog' page.
These tables are exposing the number of running catalog operations,
which helps to better understand the catalog load and reason when the
catalog is overloaded.
A new 'monitor' package is created to collect the monitoring
sub-classes. The new counters are called just after JNI methods are
invoked, so it can notice the exceptions. The existing thrift structure
is extended with the new counters, these values are unwrapped and
displayed on the WebUI.
Before/after pictures of the '/catalog' page can be found in the Jira.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
---
M be/src/catalog/catalog-server.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogUsageMonitor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M www/catalog.tmpl
11 files changed, 367 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/7
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 14: Code-Review+2
Thanks for applying the changes!
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 14
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Feb 2020 13:47:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 17: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 17
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2020 17:46:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13806/12/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/13806/12/tests/webserver/test_web_pages.py@766
PS12, Line 766:
flake8: W391 blank line at end of file
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 12
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 07 Feb 2020 14:46:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3846/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 4
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jul 2019 07:23:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663: Expose current DDL metrics on WebUI
This change adds a new '/operations' tab to the Catalog WebUI. On this
page the users can see two summary tables, the first report is based on
how many catalog operations are running and which tables are involved.
The second report shows the number of catalog operations on table level.
A new 'monitor' package is created to collect the monitoring classes.
Each DDL/FinalizeDML/ResetMetada request get a class with common base
class. The output of these classes are summarized and can be accessed
with a new JNI call.
A screenshot of the new '/operations' page can be found in the Jira.
Testing:
Unit tests were added to test the counters and a web server test to
verify the availability of the page.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java
M tests/webserver/test_web_pages.py
A www/catalog_operations.tmpl
22 files changed, 1,261 insertions(+), 157 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/12
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 12
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 13:
(4 comments)
Only found a few nits, other then that LGTM
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java@34
PS13, Line 34: super();
I think it's implicitly called by Java.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java@39
PS13, Line 39: AtomicLong
nit: since everything is in 'synchronized' I think a simple Long is enough
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java@77
PS13, Line 77: summarizTable.javaes
typos
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java@65
PS13, Line 65: Optional<TTableName> tTableName =
: req.table_name != null ? Optional.of(req.table_name) : Optional.empty();
: if (req.is_refresh) {
: decrementCounter(ResetMetadataType.REFRESH.toString(), getTableName(tTableName));
: } else if (tTableName.isPresent()) {
: decrementCounter(
: ResetMetadataType.INVALIDATE_METADATA.toString(), getTableName(tTableName));
: } else {
: decrementCounter(ResetMetadataType.INVALIDATE_METADATA_GLOBAL.toString(),
: getTableName(tTableName));
nit: this is mostly duplicated from increment. Seems like it wouldn't be too much effort to refactor the duplicated parts into a helper method.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 13
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Feb 2020 13:33:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4646/
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2019 21:48:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4588/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 5
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jul 2019 12:43:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 16: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 16
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2020 12:54:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4597/
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 6
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Jul 2019 11:00:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
The first build today failed due to a maven build problem. The previous builds failed because the CatalogOpRequestCounter.incrementDdlTypeCounter was not threadsafe, it is fixed now. Please let me know your thoughts on this change.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 11:41:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 17: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 17
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2020 12:54:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4588/
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 5
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jul 2019 14:21:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3837/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jul 2019 14:32:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
(1 comment)
I have started adding the table names to this change, will need some additional time to finish.
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@41
PS7, Line 41: private AtomicLong resetMetadataCounter_;
:
: private AtomicLong dmlOperationCounter_;
:
: private AtomicLong syncDdlOperationCounter_;
> Can you add a one line comment on each of these counters as to what do they
DML in this context is when an INSERT/CTAS is being finalized, it is triggered here:
https://github.com/apache/impala/blob/branch-3.3.0/be/src/service/client-request-state.cc#L1071
The current name is not very descriptive.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Aug 2019 16:20:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 13:
(5 comments)
Thank you for the review, additionally to the comments there was a test failure due to COMMENT_ON requests, which was fixed as well.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java@34
PS13, Line 34: super();
> I think it's implicitly called by Java.
Done
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java@39
PS13, Line 39: AtomicLong
> nit: since everything is in 'synchronized' I think a simple Long is enough
Right, moved from AtomicLong to MutableLong.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java@77
PS13, Line 77: summarizTable.javaes
> typos
Done
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java@65
PS13, Line 65: Optional<TTableName> tTableName =
: req.table_name != null ? Optional.of(req.table_name) : Optional.empty();
: if (req.is_refresh) {
: decrementCounter(ResetMetadataType.REFRESH.toString(), getTableName(tTableName));
: } else if (tTableName.isPresent()) {
: decrementCounter(
: ResetMetadataType.INVALIDATE_METADATA.toString(), getTableName(tTableName));
: } else {
: decrementCounter(ResetMetadataType.INVALIDATE_METADATA_GLOBAL.toString(),
: getTableName(tTableName));
> nit: this is mostly duplicated from increment. Seems like it wouldn't be to
Refactored, created a new method.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
File fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java@34
PS13, Line 34: private final String TEST_UPDATE_SQL = "UPDATE table SET c3 = "
: + "'test';";
> nit: this string could be written without concatenation.
Done
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 13
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Feb 2020 14:07:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
Hi Vihang, thank your for the review.
These metrics could be useful to identify if the CatalogD starts to become the bottleneck, for example when the metadata operations start piling up due to IMPALA-6671. Although, this change will not help to identify which tables are causing problem, it could give the user an idea on how busy/stuck the CatalogD. To have further understanding on the load I believe we would need historical data.
Displaying the table names could be a useful addition, will add a third column to this patch.
What do you think?
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Aug 2019 07:24:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5156/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 13
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 07 Feb 2020 16:01:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5154/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 12
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 07 Feb 2020 15:32:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5445/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 17
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2020 12:54:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
(6 comments)
Found a couple of nits, other than that lgtm. It will be even more useful when you add the table names to the metrics.
http://gerrit.cloudera.org:8080/#/c/13806/7/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/13806/7/be/src/catalog/catalog-server.cc@545
PS7, Line 545: int i = 0; i < catalog_usage_result.ddl_type_counter.size(); ++i
nit: range-based for loop can be used instead:
for (const auto& ddl_type_counter : catalog_usage_result.ddl_type_counter) {
..
}
http://gerrit.cloudera.org:8080/#/c/13806/7/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/13806/7/common/thrift/JniCatalog.thrift@718
PS7, Line 718: dll
typo: ddl
http://gerrit.cloudera.org:8080/#/c/13806/7/common/thrift/JniCatalog.thrift@720
PS7, Line 720:
nit: unnecessary blank line
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@62
PS7, Line 62: if (isSyncDdl) {
: syncDdlOperationCounter_.incrementAndGet();
: }
nit: this and the other 'if (isSyncDdl)' statements in this file would fit in a single line.
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java@26
PS7, Line 26: import com.google.common.base.Function;
it's also imported at L20.
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/service/JniCatalog.java@95
PS7, Line 95: catalogUsageMonitor
nit: maybe you could create a reference to the CatalogOpRequestCounter object since you only use that later.
This way all the increments and decrements will fit in a single line.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Oct 2019 12:35:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2019 15:57:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4597/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 6
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Jul 2019 09:22:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4598/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 6
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Jul 2019 15:19:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13806/3/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/13806/3/fe/src/main/java/org/apache/impala/catalog/Table.java@186
PS3, Line 186: CatalogUsageMonitor.INSTANCE.getCatalogTableMetrics().updateFrequentlyAccessedTables(this);
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/13806/3/fe/src/main/java/org/apache/impala/catalog/Table.java@193
PS3, Line 193: CatalogUsageMonitor.INSTANCE.getCatalogTableMetrics().updateHighFileCountTables(this);
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jul 2019 13:52:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5153/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 11
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 07 Feb 2020 15:23:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4587/
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 4
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jul 2019 09:03:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663: Expose current DDL metrics on WebUI
This change adds a new '/operations' tab to the Catalog WebUI. On this
page the users can see two summary tables, the first report is based on
how many catalog operations are running and which tables are involved.
The second report shows the number of catalog operations on table level.
A new 'monitor' package is created to collect the monitoring classes.
Each DDL/FinalizeDML/ResetMetada request get a class with common base
class. The output of these classes are summarized and can be accessed
with a new JNI call.
A screenshot of the new '/operations' page can be found in the Jira.
Testing:
Unit tests were added to test the counters and a web server test to
verify the availability of the page.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java
M tests/webserver/test_web_pages.py
A www/catalog_operations.tmpl
22 files changed, 1,202 insertions(+), 158 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/16
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 16
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Norbert Luksa (Code Review)" <ge...@cloudera.org>.
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 13: Code-Review+1
(2 comments)
Left a small nit, but lgtm.
http://gerrit.cloudera.org:8080/#/c/13806/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13806/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@427
PS8, Line 427: case DROP_FUNCTION:
> It is possible that I misunderstood the concerns here, but I will try to an
I see, I misunderstood something, thanks for clarifying.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
File fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java@34
PS13, Line 34: private final String TEST_UPDATE_SQL = "UPDATE table SET c3 = "
: + "'test';";
nit: this string could be written without concatenation.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 13
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Feb 2020 11:41:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5328/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 14
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 Feb 2020 14:57:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 15: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5421/
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 15
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Feb 2020 18:15:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5421/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 15
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Feb 2020 13:47:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
(1 comment)
> (1 comment)
>
> I have started adding the table names to this change, will need
> some additional time to finish.
I will be on vacation for a month. I would not be able to take a look at this patch until I come back. May ask Bharath to take a look?
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/service/JniCatalog.java@190
PS7, Line 190: incrementResetMetadataCounter
there are quite a few variations of reset metadata (global invalidate, refresh table, refresh functions, refresh authorization). I think it will be useful to have them track them individually
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Aug 2019 20:08:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 15: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 15
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Feb 2020 13:47:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663: Expose current DDL metrics on WebUI
This change adds a new '/operations' tab to the Catalog WebUI. On this
page the users can see two summary tables, the first report is based on
how many catalog operations are running and which tables are involved.
The second report shows the number of catalog operations on table level.
A new 'monitor' package is created to collect the monitoring classes.
Each DDL/FinalizeDML/ResetMetada request get a class with common base
class. The output of these classes are summarized and can be accessed
with a new JNI call.
A screenshot of the new '/operations' page can be found in the Jira.
Testing:
Unit tests were added to test the counters and a web server test to
verify the availability of the page.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java
M tests/webserver/test_web_pages.py
A www/catalog_operations.tmpl
22 files changed, 1,260 insertions(+), 157 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/11
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 11
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663: Expose current DDL metrics on WebUI
This change adds a new '/operations' tab to the Catalog WebUI. On this
page the users can see two summary tables, the first report is based on
how many catalog operations are running and which tables are involved.
The second report shows the number of catalog operations on table level.
A new 'monitor' package is created to collect the monitoring classes.
Each DDL/FinalizeDML/ResetMetada request get a class with common base
class. The output of these classes are summarized and can be accessed
with a new JNI call.
A screenshot of the new '/operations' page can be found in the Jira.
Testing:
Unit tests were added to test the counters and a web server test to
verify the availability of the page.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java
M tests/webserver/test_web_pages.py
A www/catalog_operations.tmpl
22 files changed, 1,216 insertions(+), 158 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/14
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 14
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4587/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 4
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jul 2019 07:24:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
(4 comments)
I think this in general is a very useful addition. Can you also specify the motivation of why we need to track the total number of DDLs instead of tracking them on a per table level. Certain metadata operations like invalidate metadata as indeed global and cannot be tracked at table. However, I think it useful to track how many refresh or invalidates were called on a given table and finding top-n such tables. Similarly for DMLs, instead of a getting a total count of INSERTS, would it be more useful to get top-N tables which are inserted into?
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@41
PS7, Line 41: private AtomicLong resetMetadataCounter_;
:
: private AtomicLong dmlOperationCounter_;
:
: private AtomicLong syncDdlOperationCounter_;
Can you add a one line comment on each of these counters as to what do they represent? Also, would be great if you could specify what operations constitute a DML in this context.
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@70
PS7, Line 70: decrementDdlTypeCounter
Is decrement ever needed? same for the other counters?
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@109
PS7, Line 109: getResetMetadataCounter
By returning the AtomicLong object this class is leaking the private field and it is subject to modification by other classes. A more common pattern is to return its value (return type long).
http://gerrit.cloudera.org:8080/#/c/13806/7/www/catalog.tmpl
File www/catalog.tmpl:
http://gerrit.cloudera.org:8080/#/c/13806/7/www/catalog.tmpl@202
PS7, Line 202: Running
Is this user-visible text? Running seems to suggest these operations are currently being run.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2019 17:24:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 6:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4598/
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 6
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Jul 2019 16:55:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663: Expose current DDL metrics on WebUI
This change adds a new '/operations' tab to the Catalog WebUI. On this
page the users can see two summary tables, the first report is based on
how many catalog operations are running and which tables are involved.
The second report shows the number of catalog operations on table level.
A new 'monitor' package is created to collect the monitoring classes.
Each DDL/FinalizeDML/ResetMetada request get a class with common base
class. The output of these classes are summarized and can be accessed
with a new JNI call.
A screenshot of the new '/operations' page can be found in the Jira.
Testing:
Unit tests were added to test the counters and a web server test to
verify the availability of the page.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java
M tests/webserver/test_web_pages.py
A www/catalog_operations.tmpl
22 files changed, 1,260 insertions(+), 157 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/13
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 13
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4655/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 04:53:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663: Expose current DDL metrics on WebUI
This change adds a new '/operations' tab to the Catalog WebUI. On this
page the users can see two summary tables, the first report is based on
how many catalog operations are running and which tables are involved.
The second report shows the number of catalog operations on table level.
A new 'monitor' package is created to collect the monitoring classes.
Each DDL/FinalizeDML/ResetMetada request get a class with common base
class. The output of these classes are summarized and can be accessed
with a new JNI call.
A screenshot of the new '/operations' page can be found in the Jira.
Testing:
Unit tests were added to test the counters and a web server test to
verify the availability of the page.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Reviewed-on: http://gerrit.cloudera.org:8080/13806
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java
M tests/webserver/test_web_pages.py
A www/catalog_operations.tmpl
22 files changed, 1,202 insertions(+), 158 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 18
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4646/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2019 15:16:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
......................................................................
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13806/11/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/13806/11/tests/webserver/test_web_pages.py@765
PS11, Line 765:
flake8: W292 no newline at end of file
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 11
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 07 Feb 2020 14:37:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
Patch Set 7:
(2 comments)
> (4 comments)
>
> I think this in general is a very useful addition. Can you also
> specify the motivation of why we need to track the total number of
> DDLs instead of tracking them on a per table level. Certain
> metadata operations like invalidate metadata as indeed global and
> cannot be tracked at table. However, I think it useful to track how
> many refresh or invalidates were called on a given table and
> finding top-n such tables. Similarly for DMLs, instead of a getting
> a total count of INSERTS, would it be more useful to get top-N
> tables which are inserted into?
Took a second pass and realized that you are indeed tracking currently running DDL ops. The question in such a case is how do we determine if seeing 3 resetMetadata operations on 3 small tables is better or worse than 1 resetMetadata operation on a large table. Can you please comment on how should the end user interpret these values to determine the load on the catalog?
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@70
PS7, Line 70: decrementDdlTypeCounter
> Is decrement ever needed? same for the other counters?
you may disregard this comment since I thought you were only tracking the global count of operations instead of running ones.
http://gerrit.cloudera.org:8080/#/c/13806/7/www/catalog.tmpl
File www/catalog.tmpl:
http://gerrit.cloudera.org:8080/#/c/13806/7/www/catalog.tmpl@202
PS7, Line 202: Running
> Is this user-visible text? Running seems to suggest these operations are cu
I think I misunderstood the patch a little bit. Looks like we are indeed tracking the running operations. You may discard this comment.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2019 17:32:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
......................................................................
IMPALA-6663 Expose current DDL metrics on WebUI
This change adds two new tables to the Catalog WebUI's '/catalog' page.
These tables are exposing the number of running catalog operations,
which helps to better understand the catalog load and reason when the
catalog is overloaded.
A new 'monitor' package is created to collect the monitoring
sub-classes. The new counters are called just after JNI methods are
invoked, so it can notice the exceptions. The existing thrift structure
is extended with the new counters, these values are unwrapped and
displayed on the WebUI.
Before/after pictures of the '/catalog' page can be found in the Jira.
Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
---
M be/src/catalog/catalog-server.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java
R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogUsageMonitor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M www/catalog.tmpl
11 files changed, 362 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/4
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 4
Gerrit-Owner: Tamas Mate <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>