[jira] [Commented] (SAMZA-375) Investigate Mesos Job Support

["Konstantin Alexandroff (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315010#comment-14315010 ] 

Konstantin Alexandroff commented on SAMZA-375:
----------------------------------------------

[~criccomini], one question bothers me.

Is there any guaranteed behavior (when using YARN) regarding task commit (state/checkpoint) happening upon:
a. Normal task completion
b. Unhandled exception
c. Process termination
?

Mesos normally terminates tasks by sending SIGTERM to process root. We handle these signals using shutdown hook in StreamJob to close Mesos tasks cleanly. But we do nothing for Samza task containers and they seem to just die abruptly (without proper shutdown). I don't see YARN adapter doing anything special regarding this as well. Is this by design or YARN terminates tasks differently?

We would prefer tasks to commit upon process termination. This would allow us to update jobs as frequently as needed. If this is not a default behavior then I suppose we shouldn't break it in Mesos adapter. But what's the best way to override it on project level, by implementing custom SamzaContainer (with custom RunLoop perhaps)?

> Investigate Mesos Job Support
> -----------------------------
>
>                 Key: SAMZA-375
>                 URL: https://issues.apache.org/jira/browse/SAMZA-375
>             Project: Samza
>          Issue Type: Bug
>          Components: hello-samza
>            Reporter: Jon Bringhurst
>            Assignee: Jon Bringhurst
>              Labels: mesos, project
>         Attachments: SAMZA-375.patch, Screen Shot 2014-08-23 at 5.51.39 PM.png, Screen Shot 2014-09-22 at 8.59.12 AM.png, samza-mesos-multiple-jobs.jpg
>
>
> It would be nice if Samza had support for Mesos (https://mesos.apache.org/).
> The current plan is to create a MesosJob and MesosJobFactory, then look into what it would take to allow the AM code to act as a Mesos scheduler.
> The feasibility of this landing in trunk will be better understood after a rough prototype has been created.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)