[jira] [Commented] (APEXMALHAR-2130) implement scalable windowed storage

["bright chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356232#comment-15356232 ] 

bright chen commented on APEXMALHAR-2130:
-----------------------------------------

Here are some of thoughts:
There are two type of actor for this storage, one is the operator which receive/handle the data( which need operation like query, add, modify and delete). Another type operator is query/manage the result only( need operation query, maybe need operation delete). The first type of operator need response as soon as possible and the second type of operator need accurate result.

The storage can have a cache layer and persistent layer

for the first type of operator, all the operation only apply to cache layer. The second type of operator apply to persistent layer.

There has separate thread(s) which merge the cache data to persistent data(one way only).

use the version to manage the conflict.

When do the query from query operator, it's better to sync all cached data to the persistent first and then do the query. But there need a mechanism to notify all operators which have the cache. Maybe think about this issue in future. 


> implement scalable windowed storage
> -----------------------------------
>
>                 Key: APEXMALHAR-2130
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: bright chen
>            Assignee: bright chen
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)