You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@apex.apache.org by "David Yan (JIRA)" <ji...@apache.org> on 2016/11/30 22:47:58 UTC

[jira] [Commented] (APEXMALHAR-2361) Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710094#comment-15710094 ] 

David Yan commented on APEXMALHAR-2361:
---------------------------------------

It makes sense. This potentially skips a lot of lookups because we can simply delete the windows and not worry about the keys in the windows.

> Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance
> --------------------------------------------------------------------------------
>
>                 Key: APEXMALHAR-2361
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2361
>             Project: Apache Apex Malhar
>          Issue Type: Improvement
>            Reporter: bright chen
>            Assignee: bright chen
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Currently, SpillableWindowedKeyedStorage remove(Window) will go through each key and mark all of them as deleted. It would be expensive when there are lots of keys and especially these entry already spill out of memory (this the common case when remove() was called).
> Suggest to mark whole window as deleted. When the window was marked as deleted, it will not allowed to add/update any entry of this window ( this should match the requirement as remove(Window) only be called after allowed lateness



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)