You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2023/03/15 02:55:00 UTC
[jira] [Commented] (IMPALA-12003) Request logs of getPartialCatalogObject are too verbose
[ https://issues.apache.org/jira/browse/IMPALA-12003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700472#comment-17700472 ]
Quanlong Huang commented on IMPALA-12003:
-----------------------------------------
Take the e2e tests in local-catalog mode as an example: [https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/7134/]
{noformat}
# wc -l catalogd.58a20e90a469.invalid-user.log.INFO.20230314-020629.1
315395 catalogd.58a20e90a469.invalid-user.log.INFO.20230314-020629.1
# grep getPartialCatalogObject catalogd.58a20e90a469.invalid-user.log.INFO.20230314-020629.1 | wc -l
111060{noformat}
1/3 of the logs are about getPartialCatalogObject requests. Most of the getPartialCatalogObject requests have empty "Time spent" (meaning 0ms). Few of them are longer than 10ms.
{noformat}
# grep getPartialCatalogObject catalogd.58a20e90a469.invalid-user.log.INFO.20230314-020629.1 | grep -e 'Time spent' | wc -l
55528
# grep getPartialCatalogObject catalogd.58a20e90a469.invalid-user.log.INFO.20230314-020629.1 | grep -e 'Time spent: $' | wc -l
46700
# grep getPartialCatalogObject catalogd.58a20e90a469.invalid-user.log.INFO.20230314-020629.1 | grep -e 'Time spent: [0-9]ms' | wc -l
6007{noformat}
> Request logs of getPartialCatalogObject are too verbose
> -------------------------------------------------------
>
> Key: IMPALA-12003
> URL: https://issues.apache.org/jira/browse/IMPALA-12003
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Priority: Major
>
> When local-catalog mode is enabled, there are lots of trivial logs for getPartialCatalogObject requests:
> {noformat}
> I0314 02:06:33.130956 104 JniUtil.java:162] getPartialCatalogObject request: Getting partial catalog object of CATALOG_SERVICE_ID
> I0314 02:06:33.136337 104 JniUtil.java:169] Finished getPartialCatalogObject request: Getting partial catalog object of CATALOG_SERVICE_ID. Time spent: 6ms
> I0314 02:06:33.158417 104 JniUtil.java:162] getPartialCatalogObject request: Getting partial catalog object of DATABASE:default
> I0314 02:06:33.161764 104 JniUtil.java:169] Finished getPartialCatalogObject request: Getting partial catalog object of DATABASE:default. Time spent: 3ms
> I0314 02:06:33.173619 104 JniUtil.java:162] getPartialCatalogObject request: Getting partial catalog object of DATABASE:functional
> I0314 02:06:33.173954 104 JniUtil.java:169] Finished getPartialCatalogObject request: Getting partial catalog object of DATABASE:functional. Time spent:
> I0314 02:06:33.176931 104 JniUtil.java:162] getPartialCatalogObject request: Getting partial catalog object of DATABASE:functional_avro
> I0314 02:06:33.177160 104 JniUtil.java:169] Finished getPartialCatalogObject request: Getting partial catalog object of DATABASE:functional_avro. Time spent: 1ms
> I0314 02:06:33.178148 104 JniUtil.java:162] getPartialCatalogObject request: Getting partial catalog object of DATABASE:functional_avro_def
> I0314 02:06:33.178380 104 JniUtil.java:169] Finished getPartialCatalogObject request: Getting partial catalog object of DATABASE:functional_avro_def. Time spent:
> I0314 02:06:33.179805 104 JniUtil.java:162] getPartialCatalogObject request: Getting partial catalog object of DATABASE:functional_avro_snap
> I0314 02:06:33.180004 104 JniUtil.java:169] Finished getPartialCatalogObject request: Getting partial catalog object of DATABASE:functional_avro_snap. Time spent:{noformat}
> To start the minicluster in local-catalog mode in the dev env:
> {code:java}
> bin/start-impala-cluster.py --catalogd_args="--catalog_topic_mode=minimal" --impalad_args="--use_local_catalog" {code}
> We'd better lower the log lever to TRACE. They don't really help in supportability. CC [~prozsa]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org