[jira] [Updated] (HIVE-4518) Counter Strike: Operation Operator

["Jason Dere (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Dere updated HIVE-4518:
-----------------------------

    Release Note: Added config setting hive.counters.group.name: counter group name for counters used during query execution.

> Counter Strike: Operation Operator
> ----------------------------------
>
>                 Key: HIVE-4518
>                 URL: https://issues.apache.org/jira/browse/HIVE-4518
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Gunther Hagleitner
>            Assignee: Gunther Hagleitner
>             Fix For: 0.13.0
>
>         Attachments: HIVE-4518.1.patch, HIVE-4518.10.patch, HIVE-4518.11.patch, HIVE-4518.2.patch, HIVE-4518.3.patch, HIVE-4518.4.patch, HIVE-4518.5.patch, HIVE-4518.6.patch.txt, HIVE-4518.7.patch, HIVE-4518.8.patch, HIVE-4518.9.patch
>
>
> Queries of the form:
> from foo
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> Generate a huge amount of counters. The reason is that task.progress is turned on for dynamic partitioning queries.
> The counters not only make queries slower than necessary (up to 50%) you will also eventually run out. That's because we're wrapping them in enum values to comply with hadoop 0.17.
> The real reason we turn task.progress on is that we need CREATED_FILES and FATAL counters to ensure dynamic partitioning queries don't go haywire.
> The counters have counter-intuitive names like C1 through C1000 and don't seem really useful by themselves.
> With hadoop 20+ you don't need to wrap the counters anymore, each operator can simply create and increment counters. That should simplify the code a lot.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)