[jira] [Commented] (REEF-1480) Increase the retry count for task registration to high value

["Julia (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402650#comment-15402650 ] 

Julia commented on REEF-1480:
-----------------------------

I agree for now we should simply call Cache() method in IInputPartition from Context layer. The work originally was defined in REEF-1405. This would separate data downloading from task initialization and reduce the task initialization time dramatically.

Separate GC initialization in another context layer would be much more complex and may conflict with the current FT design. Because currently in every retry in FT, we create a new communication group at driver. At Evaluator side, we create a new IGroupCommClient and NetworkService at task side. Any instance sharing about those must be very carefully evaluated for reuse during the retry. In failure case, we might have to recreate those instances anyway. So I would suggest to leave this proposal later. 

Once the data downloading is separated, then the timeout discussed in this Jira doesn't need to be too long. If any error happens during the task initialization, we don't need to wait for too long before recovery. 

> Increase the retry count for task registration to high value
> ------------------------------------------------------------
>
>                 Key: REEF-1480
>                 URL: https://issues.apache.org/jira/browse/REEF-1480
>             Project: REEF
>          Issue Type: Improvement
>          Components: REEF.NET
>         Environment: C#
>            Reporter: Dhruv Mahajan
>
> Currently, the default retry count in Group communication to wait for registration is set so that error is thrown after around 4 minutes. For IMRU tasks, if data downloading takes a lot of time error gets thrown. In general this can be the issue for any other application also since it is too lower level parameter to expose via application interfaces, for example {{IMRUJobDefinition}}. Like hadoop MapReduce, we can take a configuration file and then read these parameters from over there. For now, we would like to set the default to a very high value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)