You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2022/08/26 00:26:00 UTC
[Impala-ASF-CR](branch-4.1.1) IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/18916
to review the following change.
Change subject: IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
......................................................................
IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
We've seen catalogd throws OutOfMemoryError when serializing large
responses (i.e. size > 2GB). However, the related table names are
missing in the logs. Admins would like to get the table names and
blacklist those tables until they are optimized (e.g. reducing
partitions).
To improve the supportability, this patch adds logs in the Catalogd RPC
code paths to log some details of the request, also add thread
annotations to improve readability of jstacks.
Tests:
- Add unit tests for short descriptions of requests.
- Manually add codes to throw OutOfMemoryError and verify the logs
shown as expected.
- Run test_concurrent_ddls.py and metadata tests. Capture jstacks and
verify the thread annotations are shown.
- Run CORE tests
Change-Id: Iac7f2eda8b95643a3d3c3bef64ea71b67b20595a
Reviewed-on: http://gerrit.cloudera.org:8080/18772
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/ColumnName.java
M fe/src/main/java/org/apache/impala/analysis/FunctionName.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableName.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
A fe/src/main/java/org/apache/impala/util/CatalogOpUtil.java
A fe/src/test/java/org/apache/impala/util/CatalogOpUtilTest.java
9 files changed, 513 insertions(+), 36 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/18916/1
--
To view, visit http://gerrit.cloudera.org:8080/18916
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iac7f2eda8b95643a3d3c3bef64ea71b67b20595a
Gerrit-Change-Number: 18916
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR](branch-4.1.1) IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18916 )
Change subject: IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18916
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Iac7f2eda8b95643a3d3c3bef64ea71b67b20595a
Gerrit-Change-Number: 18916
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Fri, 26 Aug 2022 11:40:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18916 )
Change subject: IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
......................................................................
Patch Set 1: Verified+1
This is a clean cherrypick. Verified in https://jenkins.impala.io/job/gerrit-verify-dryrun/8498/ (ignored the flaky test failure).
--
To view, visit http://gerrit.cloudera.org:8080/18916
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Iac7f2eda8b95643a3d3c3bef64ea71b67b20595a
Gerrit-Change-Number: 18916
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Fri, 26 Aug 2022 05:41:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR](branch-4.1.1) IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18916 )
Change subject: IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
......................................................................
IMPALA-11401,IMPALA-10794: Add logs and thread names for catalogd RPCs
We've seen catalogd throws OutOfMemoryError when serializing large
responses (i.e. size > 2GB). However, the related table names are
missing in the logs. Admins would like to get the table names and
blacklist those tables until they are optimized (e.g. reducing
partitions).
To improve the supportability, this patch adds logs in the Catalogd RPC
code paths to log some details of the request, also add thread
annotations to improve readability of jstacks.
Tests:
- Add unit tests for short descriptions of requests.
- Manually add codes to throw OutOfMemoryError and verify the logs
shown as expected.
- Run test_concurrent_ddls.py and metadata tests. Capture jstacks and
verify the thread annotations are shown.
- Run CORE tests
Change-Id: Iac7f2eda8b95643a3d3c3bef64ea71b67b20595a
Reviewed-on: http://gerrit.cloudera.org:8080/18772
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-on: http://gerrit.cloudera.org:8080/18916
Tested-by: Quanlong Huang <hu...@gmail.com>
---
M fe/src/main/java/org/apache/impala/analysis/ColumnName.java
M fe/src/main/java/org/apache/impala/analysis/FunctionName.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableName.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
A fe/src/main/java/org/apache/impala/util/CatalogOpUtil.java
A fe/src/test/java/org/apache/impala/util/CatalogOpUtilTest.java
9 files changed, 513 insertions(+), 36 deletions(-)
Approvals:
Quanlong Huang: Verified
Csaba Ringhofer: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18916
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: merged
Gerrit-Change-Id: Iac7f2eda8b95643a3d3c3bef64ea71b67b20595a
Gerrit-Change-Number: 18916
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>