[jira] [Updated] (SAMZA-834) Heap usage increases dramatically after container timer metrics change to nanosecond

["Xinyu Liu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xinyu Liu updated SAMZA-834:
----------------------------
    Summary: Heap usage increases dramatically after container timer metrics change to nanosecond  (was: Heap usage increases dramatically after container timer metrics changes to nanosecond)

> Heap usage increases dramatically after container timer metrics change to nanosecond
> ------------------------------------------------------------------------------------
>
>                 Key: SAMZA-834
>                 URL: https://issues.apache.org/jira/browse/SAMZA-834
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Xinyu Liu
>            Assignee: Xinyu Liu
>
> After the change of container timer metrics (chooseNs, windowNs, processNs, and commitNs) from millisecond to nanosecond, we noticed a dramatic increase of memory heap usage in one of our production job. After investigation we found that the SlidingTimeWindowReservoir.update(duration) will be called much more frequently due to the duration is non-zero after the nanosecond change (In contrast, it is often zero when using millisecond). Within the 5-minute window, the storage inside SlidingTimeWindowReservoir increases a lot for a high qps job (for our job with around 10K qps, it increases the heap from <5M to 100M). It causes long GCs and degrades the job performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)