[jira] [Commented] (APEXCORE-609) Container runs out of memory when buffer spooling is disabled

["Pramod Immaneni (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/APEXCORE-609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823203#comment-15823203 ] 

Pramod Immaneni commented on APEXCORE-609:
------------------------------------------

Today, when spooling is disabled, because old data needs to be kept around for failure recovery till the purge from commit, new memory is allocated to store the new published data and this causes the container to run out of memory. New memory is allocated so that the processing can move forward, otherwise a new checkpoint will not happen and old data cannot be purged resulting in a deadlock. One idea is, in the spooling disabled case, when buffer is full, drop old data that subscribers have already (even though it is needed for recovery) and mark the upstream container dirty for recovery so in case there is a failure of the downstream the upstream is also recovered along with downstream. The advantage with this approach is that in scenarios where there is no failure or when there is a failure but data between two checkpoints fits completely in the buffer there is no performance penalty to pay and all the benefits of buffer server.

> Container runs out of memory when buffer spooling is disabled
> -------------------------------------------------------------
>
>                 Key: APEXCORE-609
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-609
>             Project: Apache Apex Core
>          Issue Type: Bug
>            Reporter: Pramod Immaneni
>            Assignee: Pramod Immaneni
>
> When buffer spooling is disabled and the specified buffer in memory limit is reached, because of fault tolerance requirements, additional memory is allocated to store the new data. This causes container to run out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)