[jira] [Comment Edited] (MESOS-2198) Scheduler#statusUpdate should not be called multiple times for the same status update

["Benjamin Mahler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14252569#comment-14252569 ] 

Benjamin Mahler edited comment on MESOS-2198 at 1/8/15 9:31 PM:
----------------------------------------------------------------

Hi Robert!

We can only _guarantee_ at-least-once delivery or at-most-once delivery. Since you want to receive the status update, we're left with at-least-once. That's the only guarantee we can provide. I tried to find a related writeup, hope this one helps: http://antirez.com/news/78

In order to be robust, frameworks should be applying status updates in an idempotent manner.

Also note that at-most-once is not only impractical, it's hard. You need replicated state (since frameworks are HA), and you need a scheme to ensure that you can avoid duplicates without keeping the full history of updates (can't use infinite memory!).


was (Author: bmahler):
Hi Robert!

We can only _guarantee_ at-least-once delivery or at-most-once delivery. Since you want to receive the status update, we're left with at-least-once. That's the only guarantee we can provide. I tried to find a related writeup, hope this one helps: http://antirez.com/news/78

In order to be robust, frameworks should be applying status updates in an idempotent manner.

Also note that at-least-once is not only impractical, it's hard. You need replicated state (since frameworks are HA), and you need a scheme to ensure that you can avoid duplicates without keeping the full history of updates (can't use infinite memory!).

> Scheduler#statusUpdate should not be called multiple times for the same status update
> -------------------------------------------------------------------------------------
>
>                 Key: MESOS-2198
>                 URL: https://issues.apache.org/jira/browse/MESOS-2198
>             Project: Mesos
>          Issue Type: Bug
>          Components: framework
>            Reporter: Robert Lacroix
>
> Currently Scheduler#statusUpdate can be called multiple times for the same status update, for example when the slave retransmits a status update because it's not acknowledged in time. Especially for terminal status updates this can lead to unexpected scheduler behavior when task id's are being reused.
> Consider this scenario:
> * Scheduler schedules task
> * Task fails, slave sends TASK_FAILED
> * Scheduler is busy and libmesos doesn't acknowledge update in time
> * Slave retransmits TASK_FAILED
> * Scheduler eventually receives first TASK_FAILED and reschedules task
> * Second TASK_FAILED triggers statusUpdate again and the scheduler can't determine if the TASK_FAILED belongs to the first or second run of the task.
> It would be a lot better if libmesos would dedupe status updates and only call Scheduler#statusUpdate once per status update it received. Retries with the same UUID shouldn't cause Scheduler#statusUpdate to be executed again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)