You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Chun-Hung Hsiao (JIRA)" <ji...@apache.org> on 2018/12/07 22:42:00 UTC
[jira] [Comment Edited] (MESOS-9293) If a framework looses
operation information it cannot reconcile to acknowledge updates.
[ https://issues.apache.org/jira/browse/MESOS-9293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710685#comment-16710685 ]
Chun-Hung Hsiao edited comment on MESOS-9293 at 12/7/18 10:41 PM:
------------------------------------------------------------------
{noformat}
commit 93373344dab0ba0966871ae1470b8ba95a48e320
Author: Benjamin Bannier benjamin.bannier@mesosphere.io
Date: Wed Dec 5 13:02:59 2018 -0800
Made agent state consistent with forwarded updates.
When the agent handles an `UpdateOperationStatusMessage` from a resource
provider, it injects its own ID which is (at least conceptually) unknown
to the resource provider before forwarding the message to the master,
and also updates its own tracking for the operation.
This patch makes sure that we first mutate the message before handing it
on for updating the internal operation tracking, while previously we
used the unmodified message. Always using the same message reduces error
potential if in future changes we e.g., introduce agent operation status
update managers.
Review: https://reviews.apache.org/r/69458/{noformat}
{noformat}
commit d5577c52587f9bad0bc98eba4b5cf0206022c42a
Author: Benjamin Bannier <be...@mesosphere.io>
Date: Wed Dec 5 13:03:06 2018 -0800
Added agent and resource provider IDs to operation status messages.
This patch adds agent and resource provider IDs to
`UpdateOperationStatus` and `UpdateOperationStatusMessage`. With that
frameworks are able to reconcile enough information after failover to
construct operation acknowledgements.
We will add code to populate these fields in a follow-up patch.
Review: https://reviews.apache.org/r/69162/{noformat}
{noformat}
commit be52c980598406a6f41c6e3f6323b02ecaccc450
Author: Benjamin Bannier <be...@mesosphere.io>
Date: Wed Dec 5 13:03:09 2018 -0800
Set agent and/or resource provider ID in operation status updates.
This patch sets agent and/or resource provider ID operation status
update messages. This is not always possible, e.g., some operations
might fail validation so that no corresponding IDs can be extracted.
Since operations failing validation are currently directly rejected by
the master without going through a status update manager, they are not
retried either. If a master status update manager for operations is
introduced at a later point it should be possible to forward
acknowledgements for updates to the master's update manager.
Review: https://reviews.apache.org/r/69163/{noformat}
Backported to 1.7.x:
{noformat}
commit 3c7c4568b1b1d75d43cdf71871599557c54c7656
commit 58c7ff57b70e43da479ef2b4ea7f635c2c4a8b73
commit 9fd65a6f8862e52a6d8c9ed970498f4a8db113f7{noformat}
was (Author: chhsia0):
{noformat}
commit d5577c52587f9bad0bc98eba4b5cf0206022c42a
Author: Benjamin Bannier <be...@mesosphere.io>
Date: Wed Dec 5 13:03:06 2018 -0800
Added agent and resource provider IDs to operation status messages.
This patch adds agent and resource provider IDs to
`UpdateOperationStatus` and `UpdateOperationStatusMessage`. With that
frameworks are able to reconcile enough information after failover to
construct operation acknowledgements.
We will add code to populate these fields in a follow-up patch.
Review: https://reviews.apache.org/r/69162/{noformat}
{noformat}
commit be52c980598406a6f41c6e3f6323b02ecaccc450
Author: Benjamin Bannier <be...@mesosphere.io>
Date: Wed Dec 5 13:03:09 2018 -0800
Set agent and/or resource provider ID in operation status updates.
This patch sets agent and/or resource provider ID operation status
update messages. This is not always possible, e.g., some operations
might fail validation so that no corresponding IDs can be extracted.
Since operations failing validation are currently directly rejected by
the master without going through a status update manager, they are not
retried either. If a master status update manager for operations is
introduced at a later point it should be possible to forward
acknowledgements for updates to the master's update manager.
Review: https://reviews.apache.org/r/69163/{noformat}
Backported to 1.7.x:
{noformat}
commit 58c7ff57b70e43da479ef2b4ea7f635c2c4a8b73
commit 9fd65a6f8862e52a6d8c9ed970498f4a8db113f7{noformat}
> If a framework looses operation information it cannot reconcile to acknowledge updates.
> ---------------------------------------------------------------------------------------
>
> Key: MESOS-9293
> URL: https://issues.apache.org/jira/browse/MESOS-9293
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 1.7.0
> Reporter: James DeFelice
> Assignee: Benjamin Bannier
> Priority: Major
> Labels: mesosphere, operation-feedback
> Fix For: 1.7.1, 1.8.0
>
>
> Normally, frameworks are expected to checkpoint agent ID and resource provider ID before accepting an offer with an OfferOperation. From this expectation comes the requirement in the v1 scheduler API that a framework must provide the agent ID and resource provider ID when acknowledging an offer operation status update. However, this expectation breaks down:
> 1. the framework might lose its checkpointed data; it no longer remembers the agent ID or the resource provider ID
> 2. even if the framework checkpoints data, it could be sent a stale update: maybe the original ACK it sent to Mesos was lost, and it needs to re-ACK. If a framework deleted its checkpointed data after sending the ACK (that's dropped) then upon replay of the status update it no longer has the agent ID or resource provider ID for the operation.
> An easy remedy would be to add the agent ID and resource provider ID to the OperationStatus message received by the scheduler so that a framework can build a proper ACK for the update, even if it doesn't have access to its previously checkpointed information.
> I'm filing this as a BUG because there's no way to reliably use the offer operation status API until this has been fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)