You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Vinod Kone <vi...@gmail.com> on 2013/01/04 23:37:54 UTC
Review Request: Slave Restart (Part 11): Garbagecollect meta directories
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8837/
-----------------------------------------------------------
Review request for mesos, Benjamin Hindman and Ben Mahler.
Description
-------
Currently we only gc task meta directories.
TODO: gc'ing executor directories. this is tricky because of the async nature status updates manager.
Diffs
-----
src/slave/slave.cpp 9755b46f97173d6fcc9ab1fd63e0e4814b3bc018
src/slave/status_update_manager.cpp PRE-CREATION
Diff: https://reviews.apache.org/r/8837/diff/
Testing
-------
make check
Thanks,
Vinod Kone
Re: Review Request: Slave Restart (Part 11): Garbagecollect meta directories
Posted by Vinod Kone <vi...@gmail.com>.
> On Feb. 23, 2013, 11:57 p.m., Benjamin Hindman wrote:
> > src/slave/status_update_manager.cpp, line 451
> > <https://reviews.apache.org/r/8837/diff/3/?file=259980#file259980line451>
> >
> > Why? Isn't this just good archive information to have (and possibly show)?
My main intention here (and with this review) is to curb the growth of meta directories. For example for an executor with 1000s of completed tasks, it would be wasteful for recovery to go through them, because its not going to use that info for recovery. Recovery is mainly interested in "non-terminal" states (of tasks, executors, frameworks) to be able to re-concile.
Ideally, I want to delete a task directory (when it reaches a terminal state and all updates are acked), an executor directory (when its terminated), framework directory (framework is shutdown) etc. This would make recovery fast and have a small memory footprint for the recovered state.
Thoughts?
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8837/#review16992
-----------------------------------------------------------
On Feb. 19, 2013, 8:09 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8837/
> -----------------------------------------------------------
>
> (Updated Feb. 19, 2013, 8:09 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Currently we only gc task meta directories.
>
> TODO: gc'ing executor directories. this is tricky because of the async nature status updates manager.
>
>
> Diffs
> -----
>
> src/slave/slave.cpp d4721c3eb51db87278d05f6fbe2eadb8a3a9b4dd
> src/slave/status_update_manager.cpp PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8837/diff/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Slave Restart (Part 11): Garbagecollect meta directories
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8837/#review16992
-----------------------------------------------------------
Ship it!
src/slave/status_update_manager.cpp
<https://reviews.apache.org/r/8837/#comment36031>
Why? Isn't this just good archive information to have (and possibly show)?
- Benjamin Hindman
On Feb. 19, 2013, 8:09 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8837/
> -----------------------------------------------------------
>
> (Updated Feb. 19, 2013, 8:09 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Currently we only gc task meta directories.
>
> TODO: gc'ing executor directories. this is tricky because of the async nature status updates manager.
>
>
> Diffs
> -----
>
> src/slave/slave.cpp d4721c3eb51db87278d05f6fbe2eadb8a3a9b4dd
> src/slave/status_update_manager.cpp PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8837/diff/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Slave Restart (Part 11): Garbagecollect meta directories
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8837/#review17253
-----------------------------------------------------------
Ship it!
src/slave/status_update_manager.cpp
<https://reviews.apache.org/r/8837/#comment36684>
Ok. Just keep in mind that you may need to debug things and you'll lose information by doing this as well. Could archive? But then we need to think about gc..
- Ben Mahler
On Feb. 19, 2013, 8:09 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8837/
> -----------------------------------------------------------
>
> (Updated Feb. 19, 2013, 8:09 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Currently we only gc task meta directories.
>
> TODO: gc'ing executor directories. this is tricky because of the async nature status updates manager.
>
>
> Diffs
> -----
>
> src/slave/slave.cpp d4721c3eb51db87278d05f6fbe2eadb8a3a9b4dd
> src/slave/status_update_manager.cpp PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8837/diff/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Slave Restart (Part 11): Garbagecollect meta directories
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8837/
-----------------------------------------------------------
(Updated March 13, 2013, 6:14 a.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
rebased off trunk for posterity. no need for review.
Description
-------
Currently we only gc task meta directories.
TODO: gc'ing executor directories. this is tricky because of the async nature status updates manager.
Diffs (updated)
-----
src/slave/status_update_manager.hpp PRE-CREATION
src/slave/status_update_manager.cpp PRE-CREATION
Diff: https://reviews.apache.org/r/8837/diff/
Testing
-------
make check
Thanks,
Vinod Kone
Re: Review Request: Slave Restart (Part 11): Garbagecollect meta directories
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8837/
-----------------------------------------------------------
(Updated Feb. 19, 2013, 8:09 p.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
rebased off trunk.
Description
-------
Currently we only gc task meta directories.
TODO: gc'ing executor directories. this is tricky because of the async nature status updates manager.
Diffs (updated)
-----
src/slave/slave.cpp d4721c3eb51db87278d05f6fbe2eadb8a3a9b4dd
src/slave/status_update_manager.cpp PRE-CREATION
Diff: https://reviews.apache.org/r/8837/diff/
Testing
-------
make check
Thanks,
Vinod Kone
Re: Review Request: Slave Restart (Part 11): Garbagecollect meta directories
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8837/
-----------------------------------------------------------
(Updated Jan. 29, 2013, 11:10 p.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
rebased
Description
-------
Currently we only gc task meta directories.
TODO: gc'ing executor directories. this is tricky because of the async nature status updates manager.
Diffs (updated)
-----
src/slave/slave.cpp 9755b46f97173d6fcc9ab1fd63e0e4814b3bc018
src/slave/status_update_manager.cpp PRE-CREATION
Diff: https://reviews.apache.org/r/8837/diff/
Testing
-------
make check
Thanks,
Vinod Kone