You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2015/03/19 21:02:38 UTC
[jira] [Commented] (TEZ-2203) Intern strings in tez counters
[ https://issues.apache.org/jira/browse/TEZ-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370006#comment-14370006 ]
Bikas Saha commented on TEZ-2203:
---------------------------------
Added test for the interning. Added another test (that is ignored) that allows testing for memory overhead with counters.
The numbers below show the trend of memory growth with counters and counter groups. The numbers were taken from a modified version of the ignored memory test. So they not be reproducible exactly but the trend should be representative.
{noformat}10K tasks
Empty Counters
##### Heap utilization statistics [MB] for testBasicCounters
##### Used Memory:40
##### Free Memory:366
##### Total Memory:407
##### Max Memory:910
One big group and counter
##### Heap utilization statistics [MB] for testBasicCounters
##### Used Memory:47
##### Free Memory:359
##### Total Memory:407
##### Max Memory:910
One big group and 100 counters
##### Heap utilization statistics [MB] for testBasicCounters
###### Used Memory:130
###### Free Memory:372
###### Total Memory:503
###### Max Memory:910
One big group and 100 counters. No interning
##### Heap utilization statistics [MB] for testBasicCounters
###### Used Memory:282
###### Free Memory:448
###### Total Memory:731
###### Max Memory:910
100 big groups and 1 counter
##### Heap utilization statistics [MB] for testBasicCounters
##### Used Memory:639
##### Free Memory:365
##### Total Memory:1005
##### Max Memory:1005
100 big groups and 1 counter. No interning
##### Heap utilization statistics [MB] for testBasicCounters
###### Used Memory:793
###### Free Memory:962
###### Total Memory:1756
###### Max Memory:2688 {noformat}
> Intern strings in tez counters
> ------------------------------
>
> Key: TEZ-2203
> URL: https://issues.apache.org/jira/browse/TEZ-2203
> Project: Apache Tez
> Issue Type: Task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2203.1.patch
>
>
> Getting per IO counters is possible today. This jira tracks work needed to enabled them by default. Internalizing strings to save memory is one item needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)