You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/12/14 15:18:00 UTC
[jira] [Commented] (SAMZA-2044) EOSMessage causes out of memory
Exceptions related to WindowOperatorImpl
[ https://issues.apache.org/jira/browse/SAMZA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721519#comment-16721519 ]
ASF GitHub Bot commented on SAMZA-2044:
---------------------------------------
GitHub user georgantasp opened a pull request:
https://github.com/apache/samza/pull/862
SAMZA-2044 - Change api to avoid OOMem Exceptions
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/georgantasp/samza SAMZA-2044
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/862.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #862
----
commit 0844025d7b037150d42d71c692abb9484945957b
Author: Peter Georgantas <pe...@...>
Date: 2018-12-14T15:14:58Z
SAMZA-2044 - change api to avoid OOMem Exceptions
----
> EOSMessage causes out of memory Exceptions related to WindowOperatorImpl
> ------------------------------------------------------------------------
>
> Key: SAMZA-2044
> URL: https://issues.apache.org/jira/browse/SAMZA-2044
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.14.0, 1.0
> Reporter: Peter Georgantas
> Priority: Major
>
> The contract of the handleEndOfStream method dictates that a collection of results be returned. In the case of WindowOperatorImpl which has a backing RocksDB store, this effectively causes the entirety of the store to be pulled into memory. In many cases, this will cause out of memory exceptions (otherwise why not keep the store in memory in the first place).
> Since this is a protected api, I have a relatively simple change to propose which could allow data to be consumed downstream as it is brought out of the store:
> {{protected void handleEndOfStream(Consumer<WindowPane<K, Object>> consumer, MessageCollector collector, TaskCoordinator coordinator)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)