[jira] [Commented] (MESOS-2122) MesosSchedulerDriver stop causes resource offer exhaustion

["Benjamin Mahler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218426#comment-14218426 ] 

Benjamin Mahler commented on MESOS-2122:
----------------------------------------

Hm, I can't seem to dig up the ticket related to this. It's a long standing limitation in libprocess: we don't notify termination when libprocess stays up but a Process terminates:
https://github.com/apache/mesos/blob/0.21.0/3rdparty/libprocess/TODO#L9

To my knowledge, we've been skirting this issue because for the most part, frameworks will be failing over when calling stop(failover=true). At which point, a new instantiation of the framework will re-register and we'll treat the old one as having gone away.

> MesosSchedulerDriver stop causes resource offer exhaustion
> ----------------------------------------------------------
>
>                 Key: MESOS-2122
>                 URL: https://issues.apache.org/jira/browse/MESOS-2122
>             Project: Mesos
>          Issue Type: Bug
>    Affects Versions: 0.20.0, 0.20.1
>         Environment: x86_64 Debian Wheezy (w/ mesosphere repos, packages)
>            Reporter: Zach Carlson
>         Attachments: mesos_2122.py
>
>
> For additional consideration, see https://github.com/airbnb/chronos/issues/290 and https://github.com/mesosphere/marathon/issues/787
> When the SchedulerProcess managed by the MesosSchedulerDriver detects a master, it performs a link() to the master. Libprocess proceeds to establish the link. Once the scheduler has performed all the work necessary, it may call MesosSchedulerDriver.stop(failover = true). 
> This is where things go awry: at this point, the SchedulerProcess schedules a termination event for itself. When libprocess's schedule thread rolls through, it performs a cleanup() of the SchedulerProcess, as expected. Part of the cleanup() is calling SocketManager::exited() on the SchedulerProcess. The problem with this is that SocketManager::exited() cleans up the links from the link map, but does not actually close the sockets. Now, since MesosSchedulerDriver::stop() was called with failover = true, no DeregisterFramework message was sent, so the Mesos master believes that the connection (which is still active) is still valid with a registered framework listening for events. It sends resourceOffers to the 'valid' framework... and since there's nothing actually listening for events, no response is sent, no offers are accepted or declined, and Mesos will grind to a halt (*until version 0.21.0, which will (according to release notes) rescind un-responded offers after a configurable timeout) -- no further offers made to any framework, and when all current framework work has completed, no further work will be performed due to the offers being wasted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)