[jira] [Commented] (FLINK-1489) Failing JobManager due to blocking calls in Execution.scheduleOrUpdateConsumers

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312430#comment-14312430 ] 

ASF GitHub Bot commented on FLINK-1489:
---------------------------------------

GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/378

    [FLINK-1489] Fixes blocking scheduleOrUpdateConsumers message calls

    Replaces the blocking calls with futures which in case of an exception let the respective task fail. Furthermore, the PartitionInfos are buffered on the JobManager in case that some of the consumers are not yet scheduled. Once the state of the consumers switched to running, all buffered partition infos are sent to the consumers.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink fixScheduleOrUpdateConsumers

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/378.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #378
    
----
commit d17f15ac966d59444aed86ed7d1c9cc1a2716152
Author: Till Rohrmann <tr...@apache.org>
Date:   2015-02-06T14:13:28Z

    [FLINK-1489] Replaces blocking scheduleOrUpdateConsumers message calls with asynchronous futures. Buffers PartitionInfos at the JobManager in case that the respective consumer has not been scheduled.
    
    Conflicts:
    	flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala

----


> Failing JobManager due to blocking calls in Execution.scheduleOrUpdateConsumers
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-1489
>                 URL: https://issues.apache.org/jira/browse/FLINK-1489
>             Project: Flink
>          Issue Type: Bug
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>
> [~Zentol] reported that the JobManager failed to execute his python job. The reason is that the the JobManager executes blocking calls in the actor thread in the method {{Execution.sendUpdateTaskRpcCall}} as a result to receiving a {{ScheduleOrUpdateConsumers}} message. 
> Every TaskManager possibly sends a {{ScheduleOrUpdateConsumers}} to the JobManager to notify the consumers about available data. The JobManager then sends to each TaskManager the respective update call {{Execution.sendUpdateTaskRpcCall}}. By blocking the actor thread, we effectively execute the update calls sequentially. Due to the ever accumulating delay, some of the initial timeouts on the TaskManager side in {{IntermediateResultParititon.scheduleOrUpdateConsumers}} fail. As a result the execution of the respective Tasks fails.
> A solution would be to make the call non-blocking.
> A general caveat for actor programming is: We should never block the actor thread, otherwise we seriously jeopardize the scalability of the system. Or even worse, the system simply fails.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)