You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Thomas Weise (JIRA)" <ji...@apache.org> on 2017/03/28 05:26:41 UTC
[jira] [Updated] (APEXMALHAR-2130) Scalable windowed storage
[ https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise updated APEXMALHAR-2130:
-------------------------------------
Issue Type: New Feature (was: Task)
> Scalable windowed storage
> -------------------------
>
> Key: APEXMALHAR-2130
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
> Project: Apache Apex Malhar
> Issue Type: New Feature
> Reporter: bright chen
> Assignee: David Yan
> Labels: roadmap
> Fix For: 3.7.0
>
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the checkpointing window id. This should be done incrementally (ManagedState) to avoid wasting space with unchanged data
> 3. When recovering, it takes the recovery window id and restores to that snapshot
> 4. When a window is committed, all windows with a lower ID should be purged from the store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage interfaces, and because of 2 and 3, we may want to add methods to the WindowedStorage interface so that the implementation of WindowedOperator can notify the storage of checkpointing, recovering and committing of a window.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)