You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Sai Hemanth Gantasala (Jira)" <ji...@apache.org> on 2023/11/08 00:40:00 UTC
[jira] [Updated] (IMPALA-12547) Set the default worker threads for compaction to 1
[ https://issues.apache.org/jira/browse/IMPALA-12547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sai Hemanth Gantasala updated IMPALA-12547:
-------------------------------------------
Description:
Set the default worker threads for compaction to 1.
Currently worker threads for compaction to 4, as a result, in the external frontend mode, there are a lot of calls to HMS in idle state. So, it is flooding the catalogD log with unnecessary calls to HMS.
{code:java}
I0503 14:13:26.944278 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_current_notificationEventId
I0503 14:13:26.948185 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_config_value
I0503 14:13:26.949983 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_next_notification
I0503 14:13:41.494956 15300 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
I0503 14:13:41.501145 15303 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
I0503 14:13:41.507638 15306 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
I0503 14:13:41.541407 15309 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2{code}
After the above-proposed values for the configs, this is the log info in an idle state.
{code:java}
I1107 15:58:03.130424 533568 CatalogMetastoreServer.java:215] Invoking HMS API: get_next_notification
I1107 15:58:03.157773 533460 CatalogMetastoreServer.java:215] Invoking HMS API: scheduled_query_poll
I1107 15:58:03.178020 533480 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2 {code}
was:
Set the default worker threads for compaction to 1. Also, set the worker sleep time to 60000ms.
Currently worker threads for compaction to 4, as a result, in the external frontend mode, there are a lot of calls to HMS in idle state. So, it is flooding the catalogD log with unnecessary calls to HMS.
{code:java}
I0503 14:13:26.944278 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_current_notificationEventId
I0503 14:13:26.948185 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_config_value
I0503 14:13:26.949983 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_next_notification
I0503 14:13:41.494956 15300 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
I0503 14:13:41.501145 15303 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
I0503 14:13:41.507638 15306 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
I0503 14:13:41.541407 15309 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2{code}
After the above-proposed values for the configs, this is the log info in an idle state.
{code:java}
I1107 15:58:03.130424 533568 CatalogMetastoreServer.java:215] Invoking HMS API: get_next_notification
I1107 15:58:03.157773 533460 CatalogMetastoreServer.java:215] Invoking HMS API: scheduled_query_poll
I1107 15:58:03.178020 533480 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2 {code}
> Set the default worker threads for compaction to 1
> --------------------------------------------------
>
> Key: IMPALA-12547
> URL: https://issues.apache.org/jira/browse/IMPALA-12547
> Project: IMPALA
> Issue Type: Wish
> Components: fe
> Reporter: Sai Hemanth Gantasala
> Assignee: Sai Hemanth Gantasala
> Priority: Major
>
> Set the default worker threads for compaction to 1.
> Currently worker threads for compaction to 4, as a result, in the external frontend mode, there are a lot of calls to HMS in idle state. So, it is flooding the catalogD log with unnecessary calls to HMS.
> {code:java}
> I0503 14:13:26.944278 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_current_notificationEventId
> I0503 14:13:26.948185 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_config_value
> I0503 14:13:26.949983 15874 CatalogMetastoreServer.java:215] Invoking HMS API: get_next_notification
> I0503 14:13:41.494956 15300 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
> I0503 14:13:41.501145 15303 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
> I0503 14:13:41.507638 15306 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2
> I0503 14:13:41.541407 15309 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2{code}
> After the above-proposed values for the configs, this is the log info in an idle state.
> {code:java}
> I1107 15:58:03.130424 533568 CatalogMetastoreServer.java:215] Invoking HMS API: get_next_notification
> I1107 15:58:03.157773 533460 CatalogMetastoreServer.java:215] Invoking HMS API: scheduled_query_poll
> I1107 15:58:03.178020 533480 CatalogMetastoreServer.java:215] Invoking HMS API: find_next_compact2 {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org