You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2021/05/12 08:32:16 UTC

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response 

Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17427


Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................

IMPALA-10702: Add warning logs for slow or large catalogd response

It'd be helpful to log the slow or large responses of catalogd in
debugging scalability issues. This patch adds these warning logs in
JniCatalog, where we serialize thrift responses. See some example
outputs in the jira description.

Responses that have size larger than 50MB or take more than 60s to
finish will be logged with the request. Add flags for these two
thredshold in case users found the warnings too verbose and want to
increase the thresholds.

Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
5 files changed, 127 insertions(+), 23 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/17427/1
-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17427

to look at the new patch set (#2).

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................

IMPALA-10702: Add warning logs for slow or large catalogd response

It'd be helpful to log the slow or large responses of catalogd in
debugging scalability issues. This patch adds these warning logs in
JniCatalog, where we serialize thrift responses. See some example
outputs in the jira description.

Responses that have size larger than 50MB or take more than 60s to
finish will be logged with the request. Add flags for these two
thredshold in case users found the warnings too verbose and want to
increase the thresholds.

Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
5 files changed, 129 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/17427/2
-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7151/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 May 2021 00:31:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 May 2021 00:31:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7155/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 May 2021 14:19:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8752/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 May 2021 06:35:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17427/1/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:

http://gerrit.cloudera.org:8080/#/c/17427/1/fe/src/main/java/org/apache/impala/service/BackendConfig.java@273
PS1, Line 273:   public long getWarnCatalogResponseDurationMs() { return warnCatalogResponseDurationMs_; }
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 May 2021 08:33:15 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8702/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 May 2021 08:53:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8728/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 May 2021 10:39:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17427

to look at the new patch set (#5).

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................

IMPALA-10702: Add warning logs for slow or large catalogd response

It'd be helpful to log the slow or large responses of catalogd in
debugging scalability issues. This patch adds these warning logs in
JniCatalog, where we serialize thrift responses. See some example
outputs in the jira description.

Responses that have size larger than 50MB or take more than 60s to
finish will be logged with the request. Add flags for these two
thredshold in case users found the warnings too verbose and want to
increase the thresholds.

Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
5 files changed, 129 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/17427/5
-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7151/


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 May 2021 06:28:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 May 2021 12:36:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 May 2021 20:26:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 1:

Thanks for working on this. LGTM


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 May 2021 17:50:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7165/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 May 2021 06:41:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17427

to look at the new patch set (#4).

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................

IMPALA-10702: Add warning logs for slow or large catalogd response

It'd be helpful to log the slow or large responses of catalogd in
debugging scalability issues. This patch adds these warning logs in
JniCatalog, where we serialize thrift responses. See some example
outputs in the jira description.

Responses that have size larger than 50MB or take more than 60s to
finish will be logged with the request. Add flags for these two
thredshold in case users found the warnings too verbose and want to
increase the thresholds.

Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
5 files changed, 129 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/17427/4
-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................

IMPALA-10702: Add warning logs for slow or large catalogd response

It'd be helpful to log the slow or large responses of catalogd in
debugging scalability issues. This patch adds these warning logs in
JniCatalog, where we serialize thrift responses. See some example
outputs in the jira description.

Responses that have size larger than 50MB or take more than 60s to
finish will be logged with the request. Add flags for these two
thredshold in case users found the warnings too verbose and want to
increase the thresholds.

Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Reviewed-on: http://gerrit.cloudera.org:8080/17427
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
5 files changed, 129 insertions(+), 23 deletions(-)

Approvals:
  Quanlong Huang: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 5: Code-Review+2

Resolved conflicts due to flags added by other patches. Carry on Vihang's +2.


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 May 2021 06:40:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8748/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 May 2021 01:56:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 3:

> Patch Set 3: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7151/

Failed by IMPALA-10704


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 May 2021 07:56:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10702: Add warning logs for slow or large catalogd response

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17427 )

Change subject: IMPALA-10702: Add warning logs for slow or large catalogd response
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/17427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icffcfcaad2a718aebf79e2331efb05ca7a9a7671
Gerrit-Change-Number: 17427
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 May 2021 17:50:55 +0000
Gerrit-HasComments: No