You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Mariia Mykhailova (JIRA)" <ji...@apache.org> on 2016/05/24 00:42:12 UTC
[jira] [Created] (REEF-1403) Deadlock between
ContextRuntime.StartTask and HeartBeatManager.OnNext(Alarm)
Mariia Mykhailova created REEF-1403:
---------------------------------------
Summary: Deadlock between ContextRuntime.StartTask and HeartBeatManager.OnNext(Alarm)
Key: REEF-1403
URL: https://issues.apache.org/jira/browse/REEF-1403
Project: REEF
Issue Type: Bug
Components: REEF.NET Evaluator
Reporter: Mariia Mykhailova
We have a potential deadlock in task start/timed heartbeats.
{{ContextRuntime.StartTask}} does the following:
1. Acquires lock on {{ContextRuntime._contextLifeCycle}}
2. Calls {{TaskRuntime.RunTask()}} which calls {{TaskStatus.setInit}} which calls {{HeartBeatManager.onNext(TaskStatusProto)}} which acquires lock on {{HeartBeatManager}} itself.
At the same time {{HeartBeatManager.onNext(Alarm)}} gets called every 4 seconds since evaluator start, and it:
1. Acquires lock on {{HeartBeatManager}}.
2. Calls {{GetEvaluatorHeartbeatProto}} which calls {{ContextManager.GetTaskStatus()}} which calls {{ContextManager.GetTaskStatus()}} which acquires lock on {{ContextRuntime._contextLifeCycle}}.
If task will be starting at the same time as heartbeat kick in, these two actions will deadlock each other. I encountered a repro on our tests when working on REEF-1388, but in general case it's not impossible that second or third task that starts on evaluator runs into a heartbeat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)