You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Meng Zhu (JIRA)" <ji...@apache.org> on 2018/12/07 05:11:00 UTC

[jira] [Created] (MESOS-9460) Speculative operations may make master and agent resource views out of sync.

Meng Zhu created MESOS-9460:
-------------------------------

             Summary: Speculative operations may make master and agent resource views out of sync.
                 Key: MESOS-9460
                 URL: https://issues.apache.org/jira/browse/MESOS-9460
             Project: Mesos
          Issue Type: Bug
    Affects Versions: 1.7.0, 1.6.1, 1.5.1
            Reporter: Meng Zhu


This bug could happen with the following sequence of events:

- agent (re)registers with the master
- speculative operation calls are made to the master
- the allocator is speculatively updated in https://github.com/apache/mesos/blob/master/src/master/master.cpp#L11315
- before agent resource gets updated, it sends `UpdateSlaveMessage` when getting the (re)registered message if it has the capability `RESOURCE_PROVIDER` or oversubscription is used (https://github.com/apache/mesos/blob/master/src/slave/slave.cpp#L1551 and https://github.com/apache/mesos/blob/master/src/slave/slave.cpp#L1633)
- the `UpdateSlaveMessage` triggers allocator to update the total resources with STALE info sent from the agent https://github.com/apache/mesos/blob/master/src/master/master.cpp#L8205, thus the update from the previous operation is overwritten and LOST
- agent finishes the operation and informs the master through `UpdateOperationStatusMessage` but for the speculative operation, we do not update the allocator https://github.com/apache/mesos/blob/master/src/master/master.cpp#L11177
- The resource views of master and agent are out of sync.

This caused MESOS-7971 and likely MESOS-9458 as well. 

[~chhsia0] proposes to use `resource_version_uuid` to fix this (https://issues.apache.org/jira/browse/MESOS-7971?focusedCommentId=16712278&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16712278).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)