[jira] [Updated] (APEXMALHAR-2332) StateTracker should free memory after committed

["bright chen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/APEXMALHAR-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

bright chen updated APEXMALHAR-2332:
------------------------------------
    Remaining Estimate: 120h  (was: 48h)
     Original Estimate: 120h  (was: 48h)
           Description: 
Current StateTracker free memory was triggered by a timer. The default the timer value was DAGContext.STREAMING_WINDOW_SIZE_MILLIS.defaultValue * OperatorContext.APPLICATION_WINDOW_COUNT.defaultValue. It would have memory leak if the process with operator thread and memory release thread as following:

bucket1: put(), put() ... put()
bucket2: put(), put() ... put()
freeMemory(): {bucket removed from bucketHeap and bucketAccessTimes}
commit: bucket1, bucket2

in this case,  nothing was freed and the bucket can't be freed any more

And the default value of free memory could large and memory used up even before get chance of free memory. 

  was:
StateTracker#bucketAccessTimes keep the bucket access time order by access time. It was used by free memory thread to decide which bucket can be freed. So, each access to bucket include put and get should update the access time.

As bucketAccessTimes and bucketHeap are shared by two thread. update them for each operation could impact the performance. It better to update period. As Bucket don't support window operation. I am going to keep the update time and update when time out.


               Summary: StateTracker should free memory after committed  (was: StateTracker#bucketAccessed should be called each time access to the bucket)

> StateTracker should free memory after committed
> -----------------------------------------------
>
>                 Key: APEXMALHAR-2332
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2332
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: bright chen
>            Assignee: bright chen
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Current StateTracker free memory was triggered by a timer. The default the timer value was DAGContext.STREAMING_WINDOW_SIZE_MILLIS.defaultValue * OperatorContext.APPLICATION_WINDOW_COUNT.defaultValue. It would have memory leak if the process with operator thread and memory release thread as following:
> bucket1: put(), put() ... put()
> bucket2: put(), put() ... put()
> freeMemory(): {bucket removed from bucketHeap and bucketAccessTimes}
> commit: bucket1, bucket2
> in this case,  nothing was freed and the bucket can't be freed any more
> And the default value of free memory could large and memory used up even before get chance of free memory. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)