You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/02/10 18:08:41 UTC
[jira] [Commented] (APEXCORE-641) Subscribers/DataListeners may not
be scheduled to execute even when they have data to process
[ https://issues.apache.org/jira/browse/APEXCORE-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861613#comment-15861613 ]
ASF GitHub Bot commented on APEXCORE-641:
-----------------------------------------
GitHub user vrozov opened a pull request:
https://github.com/apache/apex-core/pull/470
APEXCORE-641 Subscribers/DataListeners may not be scheduled to execute even when they have data to process
@PramodSSImmaneni, @tweise, @sandeshh Please review.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vrozov/apex-core APEXCORE-641
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-core/pull/470.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #470
----
commit aae4a97559c30c4b0ab610270b02489345996b5b
Author: Vlad Rozov <v....@datatorrent.com>
Date: 2017-02-08T18:15:20Z
APEXCORE-641 Subscribers/DataListeners may not be scheduled to execute even when they have data to process
----
> Subscribers/DataListeners may not be scheduled to execute even when they have data to process
> ---------------------------------------------------------------------------------------------
>
> Key: APEXCORE-641
> URL: https://issues.apache.org/jira/browse/APEXCORE-641
> Project: Apache Apex Core
> Issue Type: Bug
> Components: Buffer Server
> Affects Versions: 3.2.0, 3.3.0, 3.2.1, 3.4.0, 3.5.0, 3.6.0
> Reporter: Vlad Rozov
> Assignee: Vlad Rozov
>
> Buffer server iterates over DataListeners aka LogicalNodes and each LogicalNode tries to send to it's downstream all data that Publisher added to the DataList. When an output port is connected to multiple partitions or downstream operators (2 or more DataListeners/LogicalNodes) there may be more data published to the DataList after first few DataListeners in the listeners set iterated over DataList and reached the last block published so far. The data published while the last DataListeners sends data to it's downstream will not be processed by other DataListeners until Publisher adds more data to the DataList. This may lead to blocked operators, as Buffer server may stop processing data completely in case Publisher fills more than one Data block while a single DataListener sends data to it's downstream and there are more Subscribers/DataListeners than number of in memory blocks allowed (8). In such case, Publisher will be suspended, and there will be no task scheduled to process data already published to the DataList.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)