You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Megha Sharma <ms...@apple.com> on 2017/11/27 22:02:31 UTC
Re: Review Request 61473: Send status updates when agent re-registers.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/
-----------------------------------------------------------
(Updated Nov. 27, 2017, 10:02 p.m.)
Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
Changes
-------
Split into multiple commits.
Summary (updated)
-----------------
Send status updates when agent re-registers.
Bugs: MESOS-7215
https://issues.apache.org/jira/browse/MESOS-7215
Repository: mesos
Description (updated)
-------
Master will send task status updates to frameworks when an agent
re-registers.
Diffs (updated)
-----
src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
Diff: https://reviews.apache.org/r/61473/diff/18/
Changes: https://reviews.apache.org/r/61473/diff/17-18/
Testing
-------
make check
Thanks,
Megha Sharma
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Jiang Yan Xu <ya...@jxu.me>.
> On Nov. 27, 2017, 4:25 p.m., Ilya Pronin wrote:
> > src/master/http.cpp
> > Lines 343-345 (patched)
> > <https://reviews.apache.org/r/61473/diff/20/?file=1902099#file1902099line343>
> >
> > I may be ignorant of the discussion behind this, but since we don't treat these tasks as completed anymore, do we really need to maintain backwards compatibility here, in stats and metrics? This kind of hides the real state of things.
>
> Megha Sharma wrote:
> Copying @xujyan to the discussion.
It's true that this isn't perfect and my first intuition was to export the real state. We chatted with Vinod as well. Ultimately we think it's worse to export information inconsistently. More than metrics there are v0 http endpoints, the webUI and v1 operator APIs that expose the task's state (in addition to the scheduler API). It would be weird for the task's state to show up differently via different APIs.
Secondly, the mechanism implemented by Mesos to handle the backwards compatibility w.r.t TASK_LOST vs. new states is to decide which to use when the task status is **created**, unlike how we handle old/new reservation formats and framework API devolve/evolve, for example. This means we lose the original state in upon creation and have no way to export it later. We had to use a `bool unreachable` to *remember* the original state for unreachable tasks but this cannot be generalized to all "new states" and I don't think it's worth treating unreachable tasks specically.
Does it make sense?
- Jiang Yan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/#review191674
-----------------------------------------------------------
On Nov. 28, 2017, 9:28 a.m., Megha Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61473/
> -----------------------------------------------------------
>
> (Updated Nov. 28, 2017, 9:28 a.m.)
>
>
> Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
>
>
> Bugs: MESOS-7215
> https://issues.apache.org/jira/browse/MESOS-7215
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Master will not kill the tasks for non-Partition aware frameworks
> when an unreachable agent re-registers with the master.
> Master used to send a ShutdownFrameworkMessages to the agent
> to kill the tasks from non partition aware frameworks including
> the ones that are still registered which was problematic because
> the offer from this agent could still go to the same framework which
> could then launch new tasks. The agent would then receive tasks
> of the same framework and ignore them because it thinks the
> framework is shutting down. The framework is not shutting down of
> course, so from the master and the scheduler's perspective the task
> is pending in STAGING forever until the next agent reregistration,
> which could happen much later. This commit fixes the problem by
> not shutting down the non-partition aware frameworks on such an
> agent.
>
>
> Diffs
> -----
>
> include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
> include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
> src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
> src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
> src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
> src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
>
>
> Diff: https://reviews.apache.org/r/61473/diff/25/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Megha Sharma
>
>
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Ilya Pronin <ip...@twopensource.com>.
> On Nov. 27, 2017, 4:25 p.m., Ilya Pronin wrote:
> > src/master/master.cpp
> > Line 9370 (original), 9322 (patched)
> > <https://reviews.apache.org/r/61473/diff/20/?file=1902101#file1902101line9370>
> >
> > I would suggest moving this closer to `newTaskState` selection logic for readability.
>
> Megha Sharma wrote:
> I feel its more readable if we populate the value for `bool unreachable` just before its usage but I am open to moving it if you feel strongly about it.
Not stongly, but when I was reading this part I was like "This looks weird, why do we only check for `TASK_GONE_BY_OPERATOR` here? What about other terminal states?". Then I scrolled up and found that `newTaskState` can only be `TASK_LOST`, `TASK_UNREACHABLE` or `TASK_GONE_BY_OPERATOR`.
- Ilya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/#review191674
-----------------------------------------------------------
On Nov. 27, 2017, 4:56 p.m., Megha Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61473/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2017, 4:56 p.m.)
>
>
> Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
>
>
> Bugs: MESOS-7215
> https://issues.apache.org/jira/browse/MESOS-7215
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Master will not kill the tasks for non-Partition aware frameworks
> when an unreachable agent re-registers with the master.
> Master used to send a ShutdownFrameworkMessages to the agent
> to kill the tasks from non partition aware frameworks including
> the ones that are still registered which was problematic because
> the offer from this agent could still go to the same framework which
> could then launch new tasks. The agent would then receive tasks
> of the same framework and ignore them because it thinks the
> framework is shutting down. The framework is not shutting down of
> course, so from the master and the scheduler's perspective the task
> is pending in STAGING forever until the next agent reregistration,
> which could happen much later. This commit fixes the problem by
> not shutting down the non-partition aware frameworks on such an
> agent.
>
>
> Diffs
> -----
>
> include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
> include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
> src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
> src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
> src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
> src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
>
>
> Diff: https://reviews.apache.org/r/61473/diff/22/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Megha Sharma
>
>
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Ilya Pronin <ip...@twopensource.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/#review191674
-----------------------------------------------------------
The approach looks good to me.
src/master/http.cpp
Line 320 (original), 320 (patched)
<https://reviews.apache.org/r/61473/#comment269882>
Style: should be `JSON::ArrayWriter* writer`.
src/master/http.cpp
Lines 322 (patched)
<https://reviews.apache.org/r/61473/#comment269883>
Style: `&` should be attached to the type.
src/master/http.cpp
Lines 343-345 (patched)
<https://reviews.apache.org/r/61473/#comment269885>
I may be ignorant of the discussion behind this, but since we don't treat these tasks as completed anymore, do we really need to maintain backwards compatibility here, in stats and metrics? This kind of hides the real state of things.
src/master/master.cpp
Line 9370 (original), 9322 (patched)
<https://reviews.apache.org/r/61473/#comment269878>
I would suggest moving this closer to `newTaskState` selection logic for readability.
src/master/master.cpp
Line 10318 (original), 10271-10272 (patched)
<https://reviews.apache.org/r/61473/#comment269912>
Is there any reason not to iterate like this?
```cpp
foreachvalue (const Owned<Task>& task, framework->unreachableTasks) {
// ...
}
```
- Ilya Pronin
On Nov. 27, 2017, 2:08 p.m., Megha Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61473/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2017, 2:08 p.m.)
>
>
> Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
>
>
> Bugs: MESOS-7215
> https://issues.apache.org/jira/browse/MESOS-7215
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Master will not kill the tasks for non-Partition aware frameworks
> when an unreachable agent re-registers with the master.
> Master used to send a ShutdownFrameworkMessages to the agent
> to kill the tasks from non partition aware frameworks including
> the ones that are still registered which was problematic because
> the offer from this agent could still go to the same framework which
> could then launch new tasks. The agent would then receive tasks
> of the same framework and ignore them because it thinks the
> framework is shutting down. The framework is not shutting down of
> course, so from the master and the scheduler's perspective the task
> is pending in STAGING forever until the next agent reregistration,
> which could happen much later. This commit fixes the problem by
> not shutting down the non-partition aware frameworks on such an
> agent.
>
>
> Diffs
> -----
>
> include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
> include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
> src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
> src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
> src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
> src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
>
>
> Diff: https://reviews.apache.org/r/61473/diff/20/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Megha Sharma
>
>
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Jiang Yan Xu <ya...@jxu.me>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/#review192185
-----------------------------------------------------------
Ship it!
Committing with some small tweaks below and for the commit description. Please resolve all issues from other reviewers with comments.
include/mesos/mesos.proto
Line 345 (original), 343 (patched)
<https://reviews.apache.org/r/61473/#comment270233>
Use a `NOTE: `.
include/mesos/v1/mesos.proto
Line 343 (original), 341 (patched)
<https://reviews.apache.org/r/61473/#comment270234>
Use a `NOTE: `.
src/master/http.cpp
Lines 336-338 (patched)
<https://reviews.apache.org/r/61473/#comment270235>
Put it above `if (!authorizeTask_->accept(*task, framework_->info)) {` line so the order is consistent with the block below.
Also add a small comment about this check:
```
// There could be TASK_LOST tasks in this map. See comment for
// `unreachableTasks`.
```
src/master/master.cpp
Lines 9322-9323 (patched)
<https://reviews.apache.org/r/61473/#comment270231>
Tweak comment so it's less jagged.
src/master/master.cpp
Lines 9614-9616 (original), 9565-9568 (patched)
<https://reviews.apache.org/r/61473/#comment270232>
Reorder the comments and the CHECK.
- Jiang Yan Xu
On Nov. 28, 2017, 4:59 p.m., Megha Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61473/
> -----------------------------------------------------------
>
> (Updated Nov. 28, 2017, 4:59 p.m.)
>
>
> Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
>
>
> Bugs: MESOS-7215
> https://issues.apache.org/jira/browse/MESOS-7215
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Master will not kill the tasks for non-Partition aware frameworks
> when an unreachable agent re-registers with the master.
> Master used to send a ShutdownFrameworkMessages to the agent
> to kill the tasks from non partition aware frameworks including
> the ones that are still registered which was problematic because
> the offer from this agent could still go to the same framework which
> could then launch new tasks. The agent would then receive tasks
> of the same framework and ignore them because it thinks the
> framework is shutting down. The framework is not shutting down of
> course, so from the master and the scheduler's perspective the task
> is pending in STAGING forever until the next agent reregistration,
> which could happen much later. This commit fixes the problem by
> not shutting down the non-partition aware frameworks on such an
> agent.
>
>
> Diffs
> -----
>
> include/mesos/mesos.proto b1ebfe25301549397a48468a02882e971213d45c
> include/mesos/v1/mesos.proto d535eb40b205fc176730937eed4ce84ea7a369af
> src/master/http.cpp 9dcdcbeeea6135091db5aa21dd54bc14d84f33fc
> src/master/master.hpp 1c6a86fb37dee7a2ff4d564f4641a42af6206bb2
> src/master/master.cpp 7bcdb743659435847db6cdea917afc497e641582
> src/tests/partition_tests.cpp 067529acc2b3a1d7f0713c602d5f680ea19b6de8
>
>
> Diff: https://reviews.apache.org/r/61473/diff/29/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Megha Sharma
>
>
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Megha Sharma <ms...@apple.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/
-----------------------------------------------------------
(Updated Nov. 29, 2017, 12:59 a.m.)
Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
Changes
-------
Tested with make check.
Bugs: MESOS-7215
https://issues.apache.org/jira/browse/MESOS-7215
Repository: mesos
Description
-------
Master will not kill the tasks for non-Partition aware frameworks
when an unreachable agent re-registers with the master.
Master used to send a ShutdownFrameworkMessages to the agent
to kill the tasks from non partition aware frameworks including
the ones that are still registered which was problematic because
the offer from this agent could still go to the same framework which
could then launch new tasks. The agent would then receive tasks
of the same framework and ignore them because it thinks the
framework is shutting down. The framework is not shutting down of
course, so from the master and the scheduler's perspective the task
is pending in STAGING forever until the next agent reregistration,
which could happen much later. This commit fixes the problem by
not shutting down the non-partition aware frameworks on such an
agent.
Diffs (updated)
-----
include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
Diff: https://reviews.apache.org/r/61473/diff/26/
Changes: https://reviews.apache.org/r/61473/diff/25-26/
Testing
-------
make check
Thanks,
Megha Sharma
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Megha Sharma <ms...@apple.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/
-----------------------------------------------------------
(Updated Nov. 28, 2017, 5:28 p.m.)
Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
Bugs: MESOS-7215
https://issues.apache.org/jira/browse/MESOS-7215
Repository: mesos
Description
-------
Master will not kill the tasks for non-Partition aware frameworks
when an unreachable agent re-registers with the master.
Master used to send a ShutdownFrameworkMessages to the agent
to kill the tasks from non partition aware frameworks including
the ones that are still registered which was problematic because
the offer from this agent could still go to the same framework which
could then launch new tasks. The agent would then receive tasks
of the same framework and ignore them because it thinks the
framework is shutting down. The framework is not shutting down of
course, so from the master and the scheduler's perspective the task
is pending in STAGING forever until the next agent reregistration,
which could happen much later. This commit fixes the problem by
not shutting down the non-partition aware frameworks on such an
agent.
Diffs (updated)
-----
include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
Diff: https://reviews.apache.org/r/61473/diff/23/
Changes: https://reviews.apache.org/r/61473/diff/22-23/
Testing
-------
make check
Thanks,
Megha Sharma
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Ilya Pronin <ip...@twopensource.com>.
> On Nov. 28, 2017, 12:46 a.m., Jiang Yan Xu wrote:
> > src/master/master.cpp
> > Line 9370 (original), 9322 (patched)
> > <https://reviews.apache.org/r/61473/diff/22/?file=1902260#file1902260line9370>
> >
> > I have the similar feeling as Ilya, without context it's hard to understand this line. Although the context is not too far above, why not just define it within the context?
> >
> >
> > ```
> > TaskState newTaskState = TASK_UNREACHABLE;
> > TaskStatus::Reason newTaskReason = TaskStatus::REASON_SLAVE_REMOVED;
> >
> > // Needed to convey task unreachablility because we lose this information
> > // from the task state if `TASK_LOST` is used.
> > bool unreachable = true;
> >
> > if (!framework->capabilities.partitionAware) {
> > newTaskState = TASK_LOST;
> > } else if (unreachableTime.isNone()) {
> > unreachable = false;
> > newTaskState = TASK_GONE_BY_OPERATOR;
> > newTaskReason = TaskStatus::REASON_SLAVE_REMOVED_BY_OPERATOR;
> > }
> > ```
> >
> > Note that I simplied the nested if else structure. Does it look right?
>
> Megha Sharma wrote:
> Looks good, thanks Yan!
That line is still there :) `unreachable` declared on it shadows the one outside of the loop.
- Ilya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/#review191976
-----------------------------------------------------------
On Nov. 28, 2017, 9:28 a.m., Megha Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61473/
> -----------------------------------------------------------
>
> (Updated Nov. 28, 2017, 9:28 a.m.)
>
>
> Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
>
>
> Bugs: MESOS-7215
> https://issues.apache.org/jira/browse/MESOS-7215
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Master will not kill the tasks for non-Partition aware frameworks
> when an unreachable agent re-registers with the master.
> Master used to send a ShutdownFrameworkMessages to the agent
> to kill the tasks from non partition aware frameworks including
> the ones that are still registered which was problematic because
> the offer from this agent could still go to the same framework which
> could then launch new tasks. The agent would then receive tasks
> of the same framework and ignore them because it thinks the
> framework is shutting down. The framework is not shutting down of
> course, so from the master and the scheduler's perspective the task
> is pending in STAGING forever until the next agent reregistration,
> which could happen much later. This commit fixes the problem by
> not shutting down the non-partition aware frameworks on such an
> agent.
>
>
> Diffs
> -----
>
> include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
> include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
> src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
> src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
> src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
> src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
>
>
> Diff: https://reviews.apache.org/r/61473/diff/24/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Megha Sharma
>
>
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Jiang Yan Xu <ya...@jxu.me>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/#review191976
-----------------------------------------------------------
Almost there!
include/mesos/mesos.proto
Lines 344-346 (original), 342 (patched)
<https://reviews.apache.org/r/61473/#comment269950>
Given the potential confusion about the chagned behavior, at the API level I think there's value in explaining the history a little bit:
```
Prior to Mesos 1.5, such tasks will be killed by Mesos when the agent reregisters (unless the master has failed over). However due to the lack of benefit in maintaining different behaviors depending on whether the master has failed over (see MESOS-7215), as of 1.5, Mesos will not kill these tasks in either case.
```
How does it soud?
include/mesos/v1/mesos.proto
Lines 342-344 (original), 340 (patched)
<https://reviews.apache.org/r/61473/#comment269957>
Ditto.
src/master/http.cpp
Line 322 (original), 323 (patched)
<https://reviews.apache.org/r/61473/#comment269983>
Do we need a
```
if (task.get()->state() != TASK_UNREACHABLE) {
continue;
}
```
here?
src/master/master.hpp
Line 873 (original), 873 (patched)
<https://reviews.apache.org/r/61473/#comment269994>
We need the comment about `unreachable` here as well?
src/master/master.hpp
Line 2644 (original), 2645 (patched)
<https://reviews.apache.org/r/61473/#comment269995>
Are there two cases here or `unreachable` is the only variable that matters?
We can assert the following invariant inside an `if (unreachable) {}`:
```
CHECK(task->state() == TASK_UNREACHABLE || task->state() == TASK_LOST) << task->state();
```
src/master/master.hpp
Line 3160 (original), 3157 (patched)
<https://reviews.apache.org/r/61473/#comment269996>
s/all tasks/tasks runing/
src/master/master.cpp
Lines 6804-6807 (original), 6779-6782 (patched)
<https://reviews.apache.org/r/61473/#comment269997>
Fix indentation.
src/master/master.cpp
Line 6849 (original), 6820 (patched)
<https://reviews.apache.org/r/61473/#comment269998>
s/Determine which frameworks on the slave to shutdown, if any.//
Previously when there are two cases, this sentence is a summary. Now that there is only one case, this sentence feels redudant.
src/master/master.cpp
Line 9370 (original), 9322 (patched)
<https://reviews.apache.org/r/61473/#comment270000>
I have the similar feeling as Ilya, without context it's hard to understand this line. Although the context is not too far above, why not just define it within the context?
```
TaskState newTaskState = TASK_UNREACHABLE;
TaskStatus::Reason newTaskReason = TaskStatus::REASON_SLAVE_REMOVED;
// Needed to convey task unreachablility because we lose this information
// from the task state if `TASK_LOST` is used.
bool unreachable = true;
if (!framework->capabilities.partitionAware) {
newTaskState = TASK_LOST;
} else if (unreachableTime.isNone()) {
unreachable = false;
newTaskState = TASK_GONE_BY_OPERATOR;
newTaskReason = TaskStatus::REASON_SLAVE_REMOVED_BY_OPERATOR;
}
```
Note that I simplied the nested if else structure. Does it look right?
src/master/master.cpp
Line 9582 (original), 9535 (patched)
<https://reviews.apache.org/r/61473/#comment270001>
Assert this invariant inside this `if`?
```
CHECK(!unreachable) << task->task_id();
```
src/master/master.cpp
Line 10318 (original), 10271 (patched)
<https://reviews.apache.org/r/61473/#comment270002>
Use `foreachvalue`. `values()` call will copy out the values.
- Jiang Yan Xu
On Nov. 27, 2017, 4:56 p.m., Megha Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61473/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2017, 4:56 p.m.)
>
>
> Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
>
>
> Bugs: MESOS-7215
> https://issues.apache.org/jira/browse/MESOS-7215
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Master will not kill the tasks for non-Partition aware frameworks
> when an unreachable agent re-registers with the master.
> Master used to send a ShutdownFrameworkMessages to the agent
> to kill the tasks from non partition aware frameworks including
> the ones that are still registered which was problematic because
> the offer from this agent could still go to the same framework which
> could then launch new tasks. The agent would then receive tasks
> of the same framework and ignore them because it thinks the
> framework is shutting down. The framework is not shutting down of
> course, so from the master and the scheduler's perspective the task
> is pending in STAGING forever until the next agent reregistration,
> which could happen much later. This commit fixes the problem by
> not shutting down the non-partition aware frameworks on such an
> agent.
>
>
> Diffs
> -----
>
> include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
> include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
> src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
> src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
> src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
> src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
>
>
> Diff: https://reviews.apache.org/r/61473/diff/22/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Megha Sharma
>
>
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Megha Sharma <ms...@apple.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/
-----------------------------------------------------------
(Updated Nov. 28, 2017, 12:56 a.m.)
Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
Bugs: MESOS-7215
https://issues.apache.org/jira/browse/MESOS-7215
Repository: mesos
Description
-------
Master will not kill the tasks for non-Partition aware frameworks
when an unreachable agent re-registers with the master.
Master used to send a ShutdownFrameworkMessages to the agent
to kill the tasks from non partition aware frameworks including
the ones that are still registered which was problematic because
the offer from this agent could still go to the same framework which
could then launch new tasks. The agent would then receive tasks
of the same framework and ignore them because it thinks the
framework is shutting down. The framework is not shutting down of
course, so from the master and the scheduler's perspective the task
is pending in STAGING forever until the next agent reregistration,
which could happen much later. This commit fixes the problem by
not shutting down the non-partition aware frameworks on such an
agent.
Diffs (updated)
-----
include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
Diff: https://reviews.apache.org/r/61473/diff/22/
Changes: https://reviews.apache.org/r/61473/diff/21-22/
Testing
-------
make check
Thanks,
Megha Sharma
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Megha Sharma <ms...@apple.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/
-----------------------------------------------------------
(Updated Nov. 27, 2017, 10:08 p.m.)
Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
Bugs: MESOS-7215
https://issues.apache.org/jira/browse/MESOS-7215
Repository: mesos
Description
-------
Master will not kill the tasks for non-Partition aware frameworks
when an unreachable agent re-registers with the master.
Master used to send a ShutdownFrameworkMessages to the agent
to kill the tasks from non partition aware frameworks including
the ones that are still registered which was problematic because
the offer from this agent could still go to the same framework which
could then launch new tasks. The agent would then receive tasks
of the same framework and ignore them because it thinks the
framework is shutting down. The framework is not shutting down of
course, so from the master and the scheduler's perspective the task
is pending in STAGING forever until the next agent reregistration,
which could happen much later. This commit fixes the problem by
not shutting down the non-partition aware frameworks on such an
agent.
Diffs (updated)
-----
include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
Diff: https://reviews.apache.org/r/61473/diff/20/
Changes: https://reviews.apache.org/r/61473/diff/19-20/
Testing
-------
make check
Thanks,
Megha Sharma
Re: Review Request 61473: Do not kill non partition aware tasks.
Posted by Megha Sharma <ms...@apple.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61473/
-----------------------------------------------------------
(Updated Nov. 27, 2017, 10:05 p.m.)
Review request for mesos, James Peach, Vinod Kone, and Jiang Yan Xu.
Summary (updated)
-----------------
Do not kill non partition aware tasks.
Bugs: MESOS-7215
https://issues.apache.org/jira/browse/MESOS-7215
Repository: mesos
Description (updated)
-------
Master will not kill the tasks for non-Partition aware frameworks
when an unreachable agent re-registers with the master.
Master used to send a ShutdownFrameworkMessages to the agent
to kill the tasks from non partition aware frameworks including
the ones that are still registered which was problematic because
the offer from this agent could still go to the same framework which
could then launch new tasks. The agent would then receive tasks
of the same framework and ignore them because it thinks the
framework is shutting down. The framework is not shutting down of
course, so from the master and the scheduler's perspective the task
is pending in STAGING forever until the next agent reregistration,
which could happen much later. This commit fixes the problem by
not shutting down the non-partition aware frameworks on such an
agent.
Diffs (updated)
-----
include/mesos/mesos.proto e194093e490741acc552fd3ad328fd710b4b4435
include/mesos/v1/mesos.proto 6fb1139683952877667abbcf8bf84b5b31bcd29e
src/master/http.cpp 10084125deb839a9846a4f64d2e433ff02754c02
src/master/master.hpp a309fc78ee2613762f3d5d22ac7559afc7aac4a3
src/master/master.cpp 2ddd67ada3731803b00883b6a1f32b20c1bb238f
src/tests/partition_tests.cpp e49c474167076b4136a161ed29b11db9a13455a7
Diff: https://reviews.apache.org/r/61473/diff/19/
Changes: https://reviews.apache.org/r/61473/diff/18-19/
Testing
-------
make check
Thanks,
Megha Sharma