You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hive.apache.org by "Karen Coppage (Jira)" <ji...@apache.org> on 2021/08/05 10:10:00 UTC

[jira] [Assigned] (HIVE-25429) Delta metrics collection may cause number of tez counters to exceed tez.counters.max limit

     [ https://issues.apache.org/jira/browse/HIVE-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karen Coppage reassigned HIVE-25429:
------------------------------------


> Delta metrics collection may cause number of tez counters to exceed tez.counters.max limit
> ------------------------------------------------------------------------------------------
>
>                 Key: HIVE-25429
>                 URL: https://issues.apache.org/jira/browse/HIVE-25429
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive
>    Affects Versions: 4.0.0
>            Reporter: Karen Coppage
>            Assignee: Karen Coppage
>            Priority: Major
>
> There's a limit to the number of tez counters allowed (tez.counters.max). Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 counters for each partition touched by a given query, which can result in a huge number of counters, which is unnecessary because we're only interested in n number of partitions with the most deltas. This change limits the number of counters created to hive.txn.acid.metrics.max.cache.size*3.
> Also when tez.counters.max is reached a LimitExceededException is thrown but isn't caught on the Hive side and causes the query to fail. We should catch this and skip delta metrics collection in this case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)