You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Benjamin Mahler <bm...@apache.org> on 2017/01/26 21:42:00 UTC
Review Request 55971: Updated the agent to be MULTI_ROLE capable.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/
-----------------------------------------------------------
Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
Bugs: MESOS-6964
https://issues.apache.org/jira/browse/MESOS-6964
Repository: mesos
Description
-------
With the addition of MULTI_ROLE support, the agent needs to ensure
that allocated resources reported to the master have the
`Resource.AllocationInfo` set. The approach taken here is to set
it only in memory upon recovering tasks and executors. Note that
once we allow a framework to modify its roles, we need to update
the agent to re-persist the resources with injected allocation
info (rather than just setting it in memory).
Diffs
-----
src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
Diff: https://reviews.apache.org/r/55971/diff/
Testing
-------
The tests pass at the end of this review chain.
Thanks,
Benjamin Mahler
Re: Review Request 55971: Updated the agent to be MULTI_ROLE capable.
Posted by Benjamin Mahler <bm...@apache.org>.
> On Feb. 1, 2017, 9:01 p.m., Michael Park wrote:
> > src/slave/slave.cpp, lines 5227-5229
> > <https://reviews.apache.org/r/55971/diff/2/?file=1617172#file1617172line5227>
> >
> > Could we add a note here about how checking for `MULTI_ROLE` capabitility isn't sufficient since there could be tasks persisted from before a framework upgrade? I find it pretty tricky, and worth noting.
Good point, will do.
- Benjamin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/#review163876
-----------------------------------------------------------
On Jan. 27, 2017, 12:27 a.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55971/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2017, 12:27 a.m.)
>
>
> Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
>
>
> Bugs: MESOS-6964
> https://issues.apache.org/jira/browse/MESOS-6964
>
>
> Repository: mesos
>
>
> Description
> -------
>
> With the addition of MULTI_ROLE support, the agent needs to ensure
> that allocated resources reported to the master have the
> `Resource.AllocationInfo` set. The approach taken here is to set
> it only in memory upon recovering tasks and executors. Note that
> once we allow a framework to modify its roles, we need to update
> the agent to re-persist the resources with injected allocation
> info (rather than just setting it in memory).
>
>
> Diffs
> -----
>
> src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
> src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
>
> Diff: https://reviews.apache.org/r/55971/diff/
>
>
> Testing
> -------
>
> The tests pass at the end of this review chain.
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 55971: Updated the agent to be MULTI_ROLE capable.
Posted by Michael Park <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/#review163876
-----------------------------------------------------------
Ship it!
src/slave/slave.cpp (lines 5227 - 5229)
<https://reviews.apache.org/r/55971/#comment235358>
Could we add a note here about how checking for `MULTI_ROLE` capabitility isn't sufficient since there could be tasks persisted from before a framework upgrade? I find it pretty tricky, and worth noting.
- Michael Park
On Jan. 26, 2017, 4:27 p.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55971/
> -----------------------------------------------------------
>
> (Updated Jan. 26, 2017, 4:27 p.m.)
>
>
> Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
>
>
> Bugs: MESOS-6964
> https://issues.apache.org/jira/browse/MESOS-6964
>
>
> Repository: mesos
>
>
> Description
> -------
>
> With the addition of MULTI_ROLE support, the agent needs to ensure
> that allocated resources reported to the master have the
> `Resource.AllocationInfo` set. The approach taken here is to set
> it only in memory upon recovering tasks and executors. Note that
> once we allow a framework to modify its roles, we need to update
> the agent to re-persist the resources with injected allocation
> info (rather than just setting it in memory).
>
>
> Diffs
> -----
>
> src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
> src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
>
> Diff: https://reviews.apache.org/r/55971/diff/
>
>
> Testing
> -------
>
> The tests pass at the end of this review chain.
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 55971: Updated the agent to be MULTI_ROLE capable.
Posted by Benjamin Mahler <bm...@apache.org>.
> On Feb. 1, 2017, 9:01 p.m., Michael Park wrote:
> > src/slave/slave.cpp, line 5235
> > <https://reviews.apache.org/r/55971/diff/2/?file=1617172#file1617172line5235>
> >
> > We keep the `{` on the same line for lambdas.
Yeah, perhaps we should look into changing that to be consistent with non-lambda functions.
- Benjamin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/#review163877
-----------------------------------------------------------
On Jan. 27, 2017, 12:27 a.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55971/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2017, 12:27 a.m.)
>
>
> Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
>
>
> Bugs: MESOS-6964
> https://issues.apache.org/jira/browse/MESOS-6964
>
>
> Repository: mesos
>
>
> Description
> -------
>
> With the addition of MULTI_ROLE support, the agent needs to ensure
> that allocated resources reported to the master have the
> `Resource.AllocationInfo` set. The approach taken here is to set
> it only in memory upon recovering tasks and executors. Note that
> once we allow a framework to modify its roles, we need to update
> the agent to re-persist the resources with injected allocation
> info (rather than just setting it in memory).
>
>
> Diffs
> -----
>
> src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
> src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
>
> Diff: https://reviews.apache.org/r/55971/diff/
>
>
> Testing
> -------
>
> The tests pass at the end of this review chain.
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 55971: Updated the agent to be MULTI_ROLE capable.
Posted by Michael Park <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/#review163877
-----------------------------------------------------------
src/slave/slave.cpp (line 5235)
<https://reviews.apache.org/r/55971/#comment235359>
We keep the `{` on the same line for lambdas.
- Michael Park
On Jan. 26, 2017, 4:27 p.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55971/
> -----------------------------------------------------------
>
> (Updated Jan. 26, 2017, 4:27 p.m.)
>
>
> Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
>
>
> Bugs: MESOS-6964
> https://issues.apache.org/jira/browse/MESOS-6964
>
>
> Repository: mesos
>
>
> Description
> -------
>
> With the addition of MULTI_ROLE support, the agent needs to ensure
> that allocated resources reported to the master have the
> `Resource.AllocationInfo` set. The approach taken here is to set
> it only in memory upon recovering tasks and executors. Note that
> once we allow a framework to modify its roles, we need to update
> the agent to re-persist the resources with injected allocation
> info (rather than just setting it in memory).
>
>
> Diffs
> -----
>
> src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
> src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
>
> Diff: https://reviews.apache.org/r/55971/diff/
>
>
> Testing
> -------
>
> The tests pass at the end of this review chain.
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 55971: Updated the agent to be MULTI_ROLE capable.
Posted by Benjamin Mahler <bm...@apache.org>.
> On Jan. 30, 2017, 1:46 a.m., Guangya Liu wrote:
> > src/slave/slave.cpp, lines 1874-1878
> > <https://reviews.apache.org/r/55971/diff/2/?file=1617172#file1617172line1874>
> >
> > Ditto as comments in previous patches, move this to a resources util.
Added a TODO for now.
> On Jan. 30, 2017, 1:46 a.m., Guangya Liu wrote:
> > src/slave/slave.cpp, lines 5241-5249
> > <https://reviews.apache.org/r/55971/diff/2/?file=1617172#file1617172line5241>
> >
> > Since we do not support store mix allocated and unallocated resources, how about optimize the logic a big as following?
> >
> > ```
> > if (resource->has_allocation_info()) {
> > break;
> > }
> >
> > if (roles.size() != 1) {
> > LOG(FATAL) << "Missing 'Resource.AllocationInfo' for resources"
> > << " allocated to MULTI_ROLE framework"
> > << " '" << frameworkInfo.name() << "'";
> > }
> >
> > resource->mutable_allocation_info()->set_role(*roles.begin());
> > ```
I will drop for now since if we wrote this code we should probably be CHECKing the invariant, no? Would like to figure out what to do about this! :)
> On Jan. 30, 2017, 1:46 a.m., Guangya Liu wrote:
> > src/slave/slave.cpp, line 5244
> > <https://reviews.apache.org/r/55971/diff/2/?file=1617172#file1617172line5244>
> >
> > How about include `resources` in the log message?
Hm.. I'm having a hard time seeing how that would be helpful here, any thoughts?
- Benjamin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/#review163428
-----------------------------------------------------------
On Jan. 27, 2017, 12:27 a.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55971/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2017, 12:27 a.m.)
>
>
> Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
>
>
> Bugs: MESOS-6964
> https://issues.apache.org/jira/browse/MESOS-6964
>
>
> Repository: mesos
>
>
> Description
> -------
>
> With the addition of MULTI_ROLE support, the agent needs to ensure
> that allocated resources reported to the master have the
> `Resource.AllocationInfo` set. The approach taken here is to set
> it only in memory upon recovering tasks and executors. Note that
> once we allow a framework to modify its roles, we need to update
> the agent to re-persist the resources with injected allocation
> info (rather than just setting it in memory).
>
>
> Diffs
> -----
>
> src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
> src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
>
> Diff: https://reviews.apache.org/r/55971/diff/
>
>
> Testing
> -------
>
> The tests pass at the end of this review chain.
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 55971: Updated the agent to be MULTI_ROLE capable.
Posted by Guangya Liu <gy...@gmail.com>.
> On \u4e00\u6708 30, 2017, 1:46 a.m., Guangya Liu wrote:
> > src/slave/slave.cpp, line 5244
> > <https://reviews.apache.org/r/55971/diff/2/?file=1617172#file1617172line5244>
> >
> > How about include `resources` in the log message?
>
> Benjamin Mahler wrote:
> Hm.. I'm having a hard time seeing how that would be helpful here, any thoughts?
My intention is having `resources` in the log so that we can know which resources do not allocation info in this framework.
- Guangya
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/#review163428
-----------------------------------------------------------
On \u4e00\u6708 27, 2017, 12:27 a.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55971/
> -----------------------------------------------------------
>
> (Updated \u4e00\u6708 27, 2017, 12:27 a.m.)
>
>
> Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
>
>
> Bugs: MESOS-6964
> https://issues.apache.org/jira/browse/MESOS-6964
>
>
> Repository: mesos
>
>
> Description
> -------
>
> With the addition of MULTI_ROLE support, the agent needs to ensure
> that allocated resources reported to the master have the
> `Resource.AllocationInfo` set. The approach taken here is to set
> it only in memory upon recovering tasks and executors. Note that
> once we allow a framework to modify its roles, we need to update
> the agent to re-persist the resources with injected allocation
> info (rather than just setting it in memory).
>
>
> Diffs
> -----
>
> src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
> src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
>
> Diff: https://reviews.apache.org/r/55971/diff/
>
>
> Testing
> -------
>
> The tests pass at the end of this review chain.
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 55971: Updated the agent to be MULTI_ROLE capable.
Posted by Guangya Liu <gy...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/#review163428
-----------------------------------------------------------
src/slave/slave.cpp (lines 1874 - 1878)
<https://reviews.apache.org/r/55971/#comment234888>
Ditto as comments in previous patches, move this to a resources util.
src/slave/slave.cpp (lines 5241 - 5249)
<https://reviews.apache.org/r/55971/#comment234887>
Since we do not support store mix allocated and unallocated resources, how about optimize the logic a big as following?
```
if (resource->has_allocation_info()) {
break;
}
if (roles.size() != 1) {
LOG(FATAL) << "Missing 'Resource.AllocationInfo' for resources"
<< " allocated to MULTI_ROLE framework"
<< " '" << frameworkInfo.name() << "'";
}
resource->mutable_allocation_info()->set_role(*roles.begin());
```
src/slave/slave.cpp (line 5244)
<https://reviews.apache.org/r/55971/#comment234886>
How about include `resources` in the log message?
- Guangya Liu
On \u4e00\u6708 27, 2017, 12:27 a.m., Benjamin Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55971/
> -----------------------------------------------------------
>
> (Updated \u4e00\u6708 27, 2017, 12:27 a.m.)
>
>
> Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
>
>
> Bugs: MESOS-6964
> https://issues.apache.org/jira/browse/MESOS-6964
>
>
> Repository: mesos
>
>
> Description
> -------
>
> With the addition of MULTI_ROLE support, the agent needs to ensure
> that allocated resources reported to the master have the
> `Resource.AllocationInfo` set. The approach taken here is to set
> it only in memory upon recovering tasks and executors. Note that
> once we allow a framework to modify its roles, we need to update
> the agent to re-persist the resources with injected allocation
> info (rather than just setting it in memory).
>
>
> Diffs
> -----
>
> src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
> src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
>
> Diff: https://reviews.apache.org/r/55971/diff/
>
>
> Testing
> -------
>
> The tests pass at the end of this review chain.
>
>
> Thanks,
>
> Benjamin Mahler
>
>
Re: Review Request 55971: Updated the agent to be MULTI_ROLE capable.
Posted by Benjamin Mahler <bm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55971/
-----------------------------------------------------------
(Updated Jan. 27, 2017, 12:27 a.m.)
Review request for mesos, Benjamin Bannier, Jay Guo, Guangya Liu, and Michael Park.
Changes
-------
Fixed an incorrect CHECK assumption.
Bugs: MESOS-6964
https://issues.apache.org/jira/browse/MESOS-6964
Repository: mesos
Description
-------
With the addition of MULTI_ROLE support, the agent needs to ensure
that allocated resources reported to the master have the
`Resource.AllocationInfo` set. The approach taken here is to set
it only in memory upon recovering tasks and executors. Note that
once we allow a framework to modify its roles, we need to update
the agent to re-persist the resources with injected allocation
info (rather than just setting it in memory).
Diffs (updated)
-----
src/slave/resource_estimators/fixed.cpp 2c1268c3423091c6a419020a3af97255de55db0a
src/slave/slave.cpp 0548b04073c0ba4adfc4433d75fd18c2ba79d891
Diff: https://reviews.apache.org/r/55971/diff/
Testing
-------
The tests pass at the end of this review chain.
Thanks,
Benjamin Mahler