You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by gr...@apache.org on 2019/02/05 23:34:39 UTC
[mesos] branch master updated: Added more documentation for
operation feedback.
This is an automated email from the ASF dual-hosted git repository.
grag pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
The following commit(s) were added to refs/heads/master by this push:
new ae7e733 Added more documentation for operation feedback.
ae7e733 is described below
commit ae7e7330d7fc7bbe2ce923025025577e087f8348
Author: Greg Mann <gr...@mesosphere.io>
AuthorDate: Tue Feb 5 08:31:33 2019 -0800
Added more documentation for operation feedback.
Review: https://reviews.apache.org/r/69871/
---
docs/high-availability-framework-guide.md | 71 +++++++++++++++++++++++++------
docs/reconciliation.md | 27 ++++++++++++
docs/scheduler-http-api.md | 23 +++++++++-
3 files changed, 107 insertions(+), 14 deletions(-)
diff --git a/docs/high-availability-framework-guide.md b/docs/high-availability-framework-guide.md
index 1bf58e1..0e7f9f8 100644
--- a/docs/high-availability-framework-guide.md
+++ b/docs/high-availability-framework-guide.md
@@ -63,18 +63,19 @@ availability:
followed by M2 -- it will _not_ receive M2 followed by M1.
* As a convenience for framework authors, Mesos provides reliable delivery of
- task status updates. The agent persists task status updates to disk and then
- forwards them to the master. The master sends status updates to the
- appropriate framework scheduler. When a scheduler acknowledges a status
- update, the master forwards the acknowledgment back to the agent, which
- allows the stored status update to be garbage collected. If the agent does
- not receive an acknowledgment for a task status update within a certain
- amount of time, it will repeatedly resend the status update to the master,
- which will again forward the update to the scheduler. Hence, task status
- updates will be delivered "at least once", assuming that the agent and the
- scheduler both remain available. To handle the fact that task status updates
- might be delivered more than once, it can be helpful to make the framework
- logic that processes them [idempotent](https://en.wikipedia.org/wiki/Idempotence).
+ task status updates and operation status updates. The agent persists these
+ updates to disk and then forwards them to the master. The master sends
+ status updates to the appropriate framework scheduler. When a scheduler
+ acknowledges a status update, the master forwards the acknowledgment back to
+ the agent, which allows the stored status update to be garbage collected. If
+ the agent does not receive an acknowledgment for a status update within a
+ certain amount of time, it will repeatedly resend the update to the master,
+ which will again forward the update to the scheduler. Hence, task and
+ operation status updates will be delivered "at least once", assuming that
+ the agent and the scheduler both remain available. To handle the fact that
+ task and operation status updates might be delivered more than once, it can
+ be helpful to make the framework logic that processes them
+ [idempotent](https://en.wikipedia.org/wiki/Idempotence).
* The Mesos master stores information about the active tasks and registered
frameworks _in memory_: it does not persist it to disk or attempt to ensure
@@ -234,6 +235,52 @@ task description is syntactically malformed (e.g., the task ID contains an
invalid character). The `reason` field of the `TaskStatus` message can be used
to disambiguate between such situations.
+## Performing operations on offered resources
+
+The scheduler API provides a number of operations which can be applied to
+resources included in offers sent to a framework scheduler. Schedulers which use
+the [v1 scheduler API](scheduler-http-api.md) may set the `id` field in an offer
+operation in order to request feedback for the operation. When this is done, the
+scheduler will receive `UPDATE_OPERATION_STATUS` events on its HTTP event stream
+when the operation transitions to a new state. Additionally, the scheduler may
+use the `RECONCILE_OPERATIONS` call to perform explicit or implicit
+[reconciliation](reconciliation.md) of its operations' states, similar to task
+state reconciliation.
+
+Unlike tasks, which occur as the result of `LAUNCH` or `LAUNCH_GROUP`
+operations, other operations do not currently have intermediate states that they
+transition through:
+
+* An operation begins in the `OPERATION_PENDING` state. In the absence of any
+ system failures, it remains in this state until it transitions to a terminal
+ state.
+
+* There exist several terminal states that an operation may transition to:
+
+ * `OPERATION_FINISHED` is used when an operation completes successfully.
+ * `OPERATION_FAILED` is used when an operation was attempted but failed to
+ complete.
+ * `OPERATION_ERROR` is used when an operation failed because it was not
+ specified correctly and was thus never attempted.
+ * `OPERATION_DROPPED` is used when an operation was not successfully delivered
+ to the agent.
+
+* When performing operation reconciliation, the scheduler may encounter other
+ non-terminal states due to various failures in the system:
+
+ * `OPERATION_UNREACHABLE` is used when an operation was previously pending on
+ an agent which is not currently reachable by the Mesos master.
+ * `OPERATION_RECOVERING` is used when an operation was previously pending on
+ an agent which has been recovered from the master's checkpointed state after
+ a master failover, but which has not yet reregistered.
+ * `OPERATION_UNKNOWN` is used when Mesos does not recognize an operation ID
+ included in an explicit reconciliation request. This may be because an
+ operation with that ID was never received by the master, or because the
+ operation state is gone due to garbage collection or a system/network
+ failure.
+ * `OPERATION_GONE_BY_OPERATOR` is used when an operation was previously
+ pending on an agent which was marked as "gone" by an operator.
+
## Dealing with Partitioned or Failed Agents
The Mesos master tracks the availability and health of the registered agents
diff --git a/docs/reconciliation.md b/docs/reconciliation.md
index bfb117c..150a145 100644
--- a/docs/reconciliation.md
+++ b/docs/reconciliation.md
@@ -126,3 +126,30 @@ Offers are reconciled automatically after a failure:
* Offers do not persist beyond the lifetime of a Master.
* If a disconnection occurs, offers are no longer valid.
* Offers are rescinded and regenerated each time the framework (re-)registers.
+
+
+# Operation Reconciliation
+
+When a scheduler specifies an `id` on an offer operation, the master will
+provide updates on the status of that operation. If the scheduler needs to
+reconcile its view of the current states of operations with the master's view,
+it can do so via the `RECONCILE_OPERATIONS` call in the v1 scheduler API.
+
+Operation reconciliation is similar to task reconciliation in that the scheduler
+can perform either explicit or implicit reconciliation by specifying particular
+operation IDs or by leaving the `operations` field unset, respectively.
+
+Operation reconciliation differs from task reconciliation in that the
+`RECONCILE_OPERATIONS` call provides a single HTTP response which contains all
+operation status updates, rather than an asynchronous stream of individual
+updates on the event stream. NOTE, however, that this API is experimental and
+we plan to change this in the near future so that operation reconciliation will
+resemble task reconciliation in this respect.
+
+In order to explicitly reconcile particular operations, the scheduler should
+include in the `RECONCILE_OPERATIONS` call a list of operations, specifying an
+operation ID, agent ID, and resource provider ID (if applicable) for each one.
+
+Similar to task reconciliation, we recommend that schedulers implement a
+periodic reconciliation loop for operations in order to defend against network
+failures and bugs in the scheduler and/or Mesos master.
diff --git a/docs/scheduler-http-api.md b/docs/scheduler-http-api.md
index 84f6187..8384336 100644
--- a/docs/scheduler-http-api.md
+++ b/docs/scheduler-http-api.md
@@ -137,8 +137,6 @@ NOTE: Mesos will cap `Filters.refuse_seconds` at 31536000 seconds (365 days).
The master will send task status updates in response to `LAUNCH` and `LAUNCH_GROUP` operations. For other types of operations, if an operation ID is specified, the master will send operation status updates in response.
-NOTE: For the time being, an operation ID can only be set if the operation affects resources provided by a [resource provider](csi.md#resource-providers). See [MESOS-8194](https://issues.apache.org/jira/browse/MESOS-8371) for more details.
-
```
ACCEPT Request (JSON):
POST /api/v1/scheduler HTTP/1.1
@@ -597,6 +595,27 @@ UPDATE Event (JSON)
}
```
+### UPDATE_OPERATION_STATUS
+Sent by the master whenever there is an update to the state of an operation for which the scheduler requested feedback by setting the operation's `id` field. It is the responsibility of the scheduler to explicitly acknowledge the receipt of any status updates which have their `uuid` field set, as this indicates that the update will be retried until acknowledgement is received. This ensures that such updates are delivered reliably. See `ACKNOWLEDGE_OPERATION_STATUS` in the Calls section a [...]
+
+```
+UPDATE_OPERATION_STATUS Event (JSON)
+
+<event-length>
+{
+ "type" : "UPDATE_OPERATION_STATUS",
+ "update_operation_status" : {
+ "status" : {
+ "operation_id" : { "value" : "operation-1234"},
+ "state" : "OPERATION_FAILED",
+ "uuid" : "adfadfadbhgvjayd23r2uahj",
+ "agent_id" : { "value" : "12214-23523-S235235"},
+ "resource_provider_id" : { "value" : "83978-17885-1089645"}
+ }
+ }
+}
+```
+
### MESSAGE
A custom message generated by the executor that is forwarded to the scheduler by the master. This message is not interpreted by Mesos and is only forwarded (without reliability guarantees) to the scheduler. It is up to the executor to retry if the message is dropped for any reason. The `data` field contains raw bytes encoded as Base64.