You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Joseph Wu <jo...@mesosphere.io> on 2019/02/13 23:23:11 UTC
Review Request 69980: Modified when master responds to operation
status updates.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/
-----------------------------------------------------------
Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
Bugs: MESOS-9542
https://issues.apache.org/jira/browse/MESOS-9542
Repository: mesos
Description
-------
When dealing with orphaned operation status updates, there are two
cases the master must deal with:
- The simple case is when the master knows the framework is completed.
These status updates can be acknowledged by the master.
- However, a completed framework can be rotated out of the master's
memory. In addition, after master failover, if an agent reregisters
before the framework, an operation can appear to be orphaned until
the framework reregisters.
This adds a fixed delay between agent reregistration and when the
master acknowledges operation status updates from unknown frameworks.
The delay should give frameworks ample time to reregister.
The delay is based on agent reregistration in order to mitigate the
delay of acknowledging status updates of frameworks rotated out of
the completed frameworks buffer.
Diffs
-----
src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
src/master/master.cpp 014e0e053cdf5c53a5ef8d63300205a121bed319
Diff: https://reviews.apache.org/r/69980/diff/1/
Testing
-------
TODO: This case needs unit tests.
Thanks,
Joseph Wu
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review212818
-----------------------------------------------------------
PASS: Mesos patch 69980 was successfully built and tested.
Reviews applied: `['69960', '69961', '69962', '69963', '69967', '69980']`
All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2887/mesos-review-69980
- Mesos Reviewbot Windows
On Feb. 13, 2019, 3:23 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
>
> (Updated Feb. 13, 2019, 3:23 p.m.)
>
>
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
>
>
> Bugs: MESOS-9542
> https://issues.apache.org/jira/browse/MESOS-9542
>
>
> Repository: mesos
>
>
> Description
> -------
>
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
> These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
> memory. In addition, after master failover, if an agent reregisters
> before the framework, an operation can appear to be orphaned until
> the framework reregisters.
>
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
>
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
>
>
> Diffs
> -----
>
> src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
> src/master/master.cpp 014e0e053cdf5c53a5ef8d63300205a121bed319
>
>
> Diff: https://reviews.apache.org/r/69980/diff/1/
>
>
> Testing
> -------
>
> TODO: This case needs unit tests.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review212936
-----------------------------------------------------------
FAIL: Some of the unit tests failed. Please check the relevant logs.
Reviews applied: `['69960', '69961', '69962', '69963', '69967', '69980']`
Failed command: `Start-MesosCITesting`
All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2902/mesos-review-69980
Relevant logs:
- [mesos-tests.log](http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2902/mesos-review-69980/logs/mesos-tests.log):
```
I0220 01:54:20.080921 60676 ex832fb4-0000 because it is terminating
I0220 01:54:20.081923 56988 master.cpp:1269] Agent ae10102e-10e4-48ec-a6f2-9674e6832fb4-S0 at slave(491)@192.10.1.6:56719 (windows-02.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net) disconnected
I0220 01:54:20.082947 56988 master.cpp:3292] Disconnecting agent ae10102e-10e4-48ec-a6f2-9674e6832fb4-S0 at slave(491)@192.10.1.6:56719 (windows-02.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net)
I0220 01:54:20.082947 56988 master.cpp:3311] Deactivating agent ae10102e-10e4-48ec-a6f2-9674e6832fb4-S0 at slave(491)@192.10.1.6:56719 (windows-02.chtsmhjxogyevckjfayqqcnjda.xx.internal.cloudapp.net)
I0220 01:54:20.082947 60756 hierarchical.cpp:390] Removed framework ae10102e-10e4-48ec-a6f2-9674e6832fb4-0000
I0220 01:54:20.082947 60756 hierarchical.cpp:827] Agent ae10102e-10e4-48ec-a6f2-9674e6832fb4-S0 deactivated
I0220 01:54:20.084924 56988 containerizer.cpp:2526] Destroying container 8c59f777-acc4-44f0-9a79-9f3c92f58614 in RUNNING state
I0220 01:54:20.084924 56988 containerize[ OK ] IsolationFlag/MemoryIsolatorTest.ROOT_MemUsage/0 (685 ms)
[----------] 1 test from IsolationFlag/MemoryIsolatorTest (704 ms total)
[----------] Global test environment tear-down
[==========] 1112 tests from 105 test cases ran. (557867 ms total)
[ PASSED ] 1111 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] DockerFetcherPluginTest.INTERNET_CURL_FetchImage
1 FAILED TEST
YOU HAVE 232 DISABLED TESTS
r.cpp:3193] Transitioning the state of container 8c59f777-acc4-44f0-9a79-9f3c92f58614 from RUNNING to DESTROYING
I0220 01:54:20.084924 56988 launcher.cpp:161] Asked to destroy container 8c59f777-acc4-44f0-9a79-9f3c92f58614
W0220 01:54:20.085935 61824 process.cpp:1423] Failed to recv on socket WindowsFD::Type::SOCKET=5756 to peer '192.10.1.6:58703': IO failed with error code: The specified network name is no longer available.
W0220 01:54:20.086922 61824 process.cpp:838] Failed to recv on socket WindowsFD::Type::SOCKET=5720 to peer '192.10.1.6:58704': IO failed with error code: The specified network name is no longer available.
I0220 01:54:20.105926 56988 containerizer.cpp:3032] Container 8c59f777-acc4-44f0-9a79-9f3c92f58614 has exited
I0220 01:54:20.133921 53304 master.cpp:1109] Master terminating
I0220 01:54:20.135931 59500 hierarchical.cpp:678] Removed agent ae10102e-10e4-48ec-a6f2-9674e6832fb4-S0
I0220 01:54:20.938930 61824 process.cpp:927] Stopped the socket accept loop
```
- Mesos Reviewbot Windows
On Feb. 19, 2019, 4:47 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
>
> (Updated Feb. 19, 2019, 4:47 p.m.)
>
>
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
>
>
> Bugs: MESOS-9542
> https://issues.apache.org/jira/browse/MESOS-9542
>
>
> Repository: mesos
>
>
> Description
> -------
>
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
> These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
> memory. In addition, after master failover, if an agent reregisters
> before the framework, an operation can appear to be orphaned until
> the framework reregisters.
>
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
>
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
>
>
> Diffs
> -----
>
> src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
> src/master/master.cpp 106d924bf16231b3bda3fb719db68c01d73644ee
>
>
> Diff: https://reviews.apache.org/r/69980/diff/2/
>
>
> Testing
> -------
>
> TODO: This case needs unit tests.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Greg Mann <gr...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review213176
-----------------------------------------------------------
Ship it!
Ship It!
- Greg Mann
On Feb. 22, 2019, 12:07 a.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
>
> (Updated Feb. 22, 2019, 12:07 a.m.)
>
>
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
>
>
> Bugs: MESOS-9542
> https://issues.apache.org/jira/browse/MESOS-9542
>
>
> Repository: mesos
>
>
> Description
> -------
>
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
> These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
> memory. In addition, after master failover, if an agent reregisters
> before the framework, an operation can appear to be orphaned until
> the framework reregisters.
>
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
>
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
>
>
> Diffs
> -----
>
> src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
> src/master/master.cpp 106d924bf16231b3bda3fb719db68c01d73644ee
>
>
> Diff: https://reviews.apache.org/r/69980/diff/3/
>
>
> Testing
> -------
>
> See last patch in chain.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Joseph Wu <jo...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/
-----------------------------------------------------------
(Updated Feb. 21, 2019, 4:07 p.m.)
Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
Changes
-------
Fixed the conditional telling the master to adopt orphan operations of unknown frameworks, with operation IDs.
Bugs: MESOS-9542
https://issues.apache.org/jira/browse/MESOS-9542
Repository: mesos
Description
-------
When dealing with orphaned operation status updates, there are two
cases the master must deal with:
- The simple case is when the master knows the framework is completed.
These status updates can be acknowledged by the master.
- However, a completed framework can be rotated out of the master's
memory. In addition, after master failover, if an agent reregisters
before the framework, an operation can appear to be orphaned until
the framework reregisters.
This adds a fixed delay between agent reregistration and when the
master acknowledges operation status updates from unknown frameworks.
The delay should give frameworks ample time to reregister.
The delay is based on agent reregistration in order to mitigate the
delay of acknowledging status updates of frameworks rotated out of
the completed frameworks buffer.
Diffs (updated)
-----
src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
src/master/master.cpp 106d924bf16231b3bda3fb719db68c01d73644ee
Diff: https://reviews.apache.org/r/69980/diff/3/
Changes: https://reviews.apache.org/r/69980/diff/2-3/
Testing (updated)
-------
See last patch in chain.
Thanks,
Joseph Wu
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Greg Mann <gr...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review213007
-----------------------------------------------------------
src/master/master.cpp
Lines 8753 (patched)
<https://reviews.apache.org/r/69980/#comment298859>
Do we really want to check `!operation->info().has_id()` here? Should it be `operation->info().has_id()` instead?
I think we want to wait until the timeout elapses to ACK orphans of non-completed frameworks when they DO request feedback, to ensure that the framework receives the requested update after it reregisters?
- Greg Mann
On Feb. 20, 2019, 12:47 a.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
>
> (Updated Feb. 20, 2019, 12:47 a.m.)
>
>
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
>
>
> Bugs: MESOS-9542
> https://issues.apache.org/jira/browse/MESOS-9542
>
>
> Repository: mesos
>
>
> Description
> -------
>
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
> These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
> memory. In addition, after master failover, if an agent reregisters
> before the framework, an operation can appear to be orphaned until
> the framework reregisters.
>
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
>
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
>
>
> Diffs
> -----
>
> src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
> src/master/master.cpp 106d924bf16231b3bda3fb719db68c01d73644ee
>
>
> Diff: https://reviews.apache.org/r/69980/diff/2/
>
>
> Testing
> -------
>
> TODO: This case needs unit tests.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Joseph Wu <jo...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/
-----------------------------------------------------------
(Updated Feb. 19, 2019, 4:47 p.m.)
Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
Changes
-------
Modified comment per suggestion.
Bugs: MESOS-9542
https://issues.apache.org/jira/browse/MESOS-9542
Repository: mesos
Description
-------
When dealing with orphaned operation status updates, there are two
cases the master must deal with:
- The simple case is when the master knows the framework is completed.
These status updates can be acknowledged by the master.
- However, a completed framework can be rotated out of the master's
memory. In addition, after master failover, if an agent reregisters
before the framework, an operation can appear to be orphaned until
the framework reregisters.
This adds a fixed delay between agent reregistration and when the
master acknowledges operation status updates from unknown frameworks.
The delay should give frameworks ample time to reregister.
The delay is based on agent reregistration in order to mitigate the
delay of acknowledging status updates of frameworks rotated out of
the completed frameworks buffer.
Diffs (updated)
-----
src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
src/master/master.cpp 106d924bf16231b3bda3fb719db68c01d73644ee
Diff: https://reviews.apache.org/r/69980/diff/2/
Changes: https://reviews.apache.org/r/69980/diff/1-2/
Testing
-------
TODO: This case needs unit tests.
Thanks,
Joseph Wu
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Feb. 14, 2019, 12:31 p.m., Greg Mann wrote:
> > src/master/master.cpp
> > Lines 8686-8708 (original), 8686-8712 (patched)
> > <https://reviews.apache.org/r/69980/diff/1/?file=2125184#file2125184line8686>
> >
> > Consider the case of a terminal-but-unacknowledged operation which has been sent to the master by a reregistered agent and which has its ID set. Since we only place non-terminal operations in `orphanedOperations`, we will get `frameworkWillAcknowledge == true` here. If this framework never reregisters, the I think we could end up in a state where the agent retries terminal updates for that operation forever.
> >
> > For such updates, I think the master needs to either:
> > 1) have a way to determine that this is a terminal-but-unacknowledged orphaned operation (i.e. place it in `orphanedOperations`), or
> > 2) fall back to default behavior of acknowledging updates for operations that it doesn't recognize.
> >
> > WDYT?
>
> Joseph Wu wrote:
> This is a good point. Orphan operations must be include terminal and non-terminal operations. With the chain as it is right now, it is only possible to produce a terminal orphan operation by adding a non-terminal orphan, and then receiving a terminal update. This transition is a bit weird, since we have a case where an UpdateSlaveMessage can contain a terminal operation, belonging to an unknown framework. As long as the `Master::updateSlave()` method marks this as an orphan, the master will be able to adopt the orphan. (This is choice 1).
Fixed with a change here: https://reviews.apache.org/r/69960/diff/3/
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review212838
-----------------------------------------------------------
On Feb. 19, 2019, 4:47 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
>
> (Updated Feb. 19, 2019, 4:47 p.m.)
>
>
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
>
>
> Bugs: MESOS-9542
> https://issues.apache.org/jira/browse/MESOS-9542
>
>
> Repository: mesos
>
>
> Description
> -------
>
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
> These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
> memory. In addition, after master failover, if an agent reregisters
> before the framework, an operation can appear to be orphaned until
> the framework reregisters.
>
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
>
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
>
>
> Diffs
> -----
>
> src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
> src/master/master.cpp 106d924bf16231b3bda3fb719db68c01d73644ee
>
>
> Diff: https://reviews.apache.org/r/69980/diff/2/
>
>
> Testing
> -------
>
> TODO: This case needs unit tests.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Joseph Wu <jo...@mesosphere.io>.
> On Feb. 14, 2019, 12:31 p.m., Greg Mann wrote:
> > src/master/master.cpp
> > Lines 8686-8708 (original), 8686-8712 (patched)
> > <https://reviews.apache.org/r/69980/diff/1/?file=2125184#file2125184line8686>
> >
> > Consider the case of a terminal-but-unacknowledged operation which has been sent to the master by a reregistered agent and which has its ID set. Since we only place non-terminal operations in `orphanedOperations`, we will get `frameworkWillAcknowledge == true` here. If this framework never reregisters, the I think we could end up in a state where the agent retries terminal updates for that operation forever.
> >
> > For such updates, I think the master needs to either:
> > 1) have a way to determine that this is a terminal-but-unacknowledged orphaned operation (i.e. place it in `orphanedOperations`), or
> > 2) fall back to default behavior of acknowledging updates for operations that it doesn't recognize.
> >
> > WDYT?
This is a good point. Orphan operations must be include terminal and non-terminal operations. With the chain as it is right now, it is only possible to produce a terminal orphan operation by adding a non-terminal orphan, and then receiving a terminal update. This transition is a bit weird, since we have a case where an UpdateSlaveMessage can contain a terminal operation, belonging to an unknown framework. As long as the `Master::updateSlave()` method marks this as an orphan, the master will be able to adopt the orphan. (This is choice 1).
- Joseph
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review212838
-----------------------------------------------------------
On Feb. 13, 2019, 3:23 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
>
> (Updated Feb. 13, 2019, 3:23 p.m.)
>
>
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
>
>
> Bugs: MESOS-9542
> https://issues.apache.org/jira/browse/MESOS-9542
>
>
> Repository: mesos
>
>
> Description
> -------
>
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
> These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
> memory. In addition, after master failover, if an agent reregisters
> before the framework, an operation can appear to be orphaned until
> the framework reregisters.
>
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
>
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
>
>
> Diffs
> -----
>
> src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
> src/master/master.cpp 014e0e053cdf5c53a5ef8d63300205a121bed319
>
>
> Diff: https://reviews.apache.org/r/69980/diff/1/
>
>
> Testing
> -------
>
> TODO: This case needs unit tests.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Greg Mann <gr...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review212838
-----------------------------------------------------------
src/master/master.cpp
Lines 8686-8708 (original), 8686-8712 (patched)
<https://reviews.apache.org/r/69980/#comment298729>
Consider the case of a terminal-but-unacknowledged operation which has been sent to the master by a reregistered agent and which has its ID set. Since we only place non-terminal operations in `orphanedOperations`, we will get `frameworkWillAcknowledge == true` here. If this framework never reregisters, the I think we could end up in a state where the agent retries terminal updates for that operation forever.
For such updates, I think the master needs to either:
1) have a way to determine that this is a terminal-but-unacknowledged orphaned operation (i.e. place it in `orphanedOperations`), or
2) fall back to default behavior of acknowledging updates for operations that it doesn't recognize.
WDYT?
src/master/master.cpp
Lines 8721-8722 (patched)
<https://reviews.apache.org/r/69980/#comment298728>
Instead of "The master will treat these...", I would suggest:
"The master will drop these status updates so that subsequent retries can be handled by the scheduler if it reregisters."
- Greg Mann
On Feb. 13, 2019, 11:23 p.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
>
> (Updated Feb. 13, 2019, 11:23 p.m.)
>
>
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
>
>
> Bugs: MESOS-9542
> https://issues.apache.org/jira/browse/MESOS-9542
>
>
> Repository: mesos
>
>
> Description
> -------
>
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
> These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
> memory. In addition, after master failover, if an agent reregisters
> before the framework, an operation can appear to be orphaned until
> the framework reregisters.
>
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
>
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
>
>
> Diffs
> -----
>
> src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
> src/master/master.cpp 014e0e053cdf5c53a5ef8d63300205a121bed319
>
>
> Diff: https://reviews.apache.org/r/69980/diff/1/
>
>
> Testing
> -------
>
> TODO: This case needs unit tests.
>
>
> Thanks,
>
> Joseph Wu
>
>
Re: Review Request 69980: Modified when master responds to operation
status updates.
Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review212823
-----------------------------------------------------------
Patch looks great!
Reviews applied: [69968, 69960, 69961, 69962, 69963, 69967, 69980]
Passed command: export OS='ubuntu:14.04' BUILDTOOL='autotools' COMPILER='gcc' CONFIGURATION='--verbose --disable-libtool-wrappers --disable-parallel-test-execution' ENVIRONMENT='GLOG_v=1 MESOS_VERBOSE=1'; ./support/docker-build.sh
- Mesos Reviewbot
On Feb. 14, 2019, 12:23 a.m., Joseph Wu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
>
> (Updated Feb. 14, 2019, 12:23 a.m.)
>
>
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
>
>
> Bugs: MESOS-9542
> https://issues.apache.org/jira/browse/MESOS-9542
>
>
> Repository: mesos
>
>
> Description
> -------
>
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
> These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
> memory. In addition, after master failover, if an agent reregisters
> before the framework, an operation can appear to be orphaned until
> the framework reregisters.
>
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
>
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
>
>
> Diffs
> -----
>
> src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4
> src/master/master.cpp 014e0e053cdf5c53a5ef8d63300205a121bed319
>
>
> Diff: https://reviews.apache.org/r/69980/diff/1/
>
>
> Testing
> -------
>
> TODO: This case needs unit tests.
>
>
> Thanks,
>
> Joseph Wu
>
>