[jira] [Created] (APEXMALHAR-2335) Problems on StateTracker#bucketAccessed

["bright chen (JIRA)" <ji...@apache.org>]

bright chen created APEXMALHAR-2335:
---------------------------------------

             Summary: Problems on StateTracker#bucketAccessed
                 Key: APEXMALHAR-2335
                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2335
             Project: Apache Apex Malhar
          Issue Type: Bug
            Reporter: bright chen
            Assignee: bright chen


The intension of StateTracker#bucketAccessed as I understand was to keep the last access time of each bucket so freeMemory can free the buckets which not accessed recently.
But here are some problems

- StateTracker#bucketAccessed() will be called each time access to the bucket. Current implementation use Concurrent map and set to handle it(bucketHeap and bucketAccessTimes). it could be pretty heavy.

- StateTracker#run use bucketHeap to get the bucket which need to free memory, but at the same time bucketHeap was constantly changed by bucketAccessed(). It could cause concurrent issue event bucketHeap is a Current set. And even more the while loop could never end. Following two bug probably related to this:
https://issues.apache.org/jira/browse/APEXMALHAR-2333
https://issues.apache.org/jira/browse/APEXMALHAR-2334

Following are my thought to solve these issue.
- It's not necessary to keep very accurate of update time, it probably only need to be updated once for each stream window. But StateTracker don't support endWindow() (in fact endWindow() was triggered at end of application window). So, we can keep the bucket which updated in this period and update only at the end of the period.

- keep two sets of bucketHeap, one for bucketAccessed() and one for run(). so the lock only need when switch



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)