You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Kevin Klues <kl...@gmail.com> on 2016/01/08 02:29:22 UTC
Review Request 42053: Add flags to set size of completed
task/framework history.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
Review request for mesos and Ben Mahler.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Diffs
-----
src/master/constants.hpp f1624e1ed5091f08f57d4382b344cab8b05ba840
src/master/constants.cpp 589f2ee55a24d7e8b9352a4c8f94a3785fd1bef4
src/master/flags.hpp 9af6c68eef6bcf39d5776809fab6c66dc95da6b2
src/master/flags.cpp f864419a6276356c97a0a4fe29e5cfce6c4660c4
src/master/master.hpp f764915ff8bbf74b99a617e3c4735d38fc13e5f0
src/master/master.cpp 145c5932610bd019eb090947569b608df6815c3a
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
Thanks,
Kevin Klues
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Neil Conway <ne...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113391
-----------------------------------------------------------
Overall looks good!
Is it worth adding a unit test for this, I wonder?
- Neil Conway
On Jan. 8, 2016, 1:29 a.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 1:29 a.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
>
> Diffs
> -----
>
> src/master/constants.hpp f1624e1ed5091f08f57d4382b344cab8b05ba840
> src/master/constants.cpp 589f2ee55a24d7e8b9352a4c8f94a3785fd1bef4
> src/master/flags.hpp 9af6c68eef6bcf39d5776809fab6c66dc95da6b2
> src/master/flags.cpp f864419a6276356c97a0a4fe29e5cfce6c4660c4
> src/master/master.hpp f764915ff8bbf74b99a617e3c4735d38fc13e5f0
> src/master/master.cpp 145c5932610bd019eb090947569b608df6815c3a
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
- Kevin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113428
-----------------------------------------------------------
On Jan. 11, 2016, 9:39 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 11, 2016, 9:39 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
- Kevin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113428
-----------------------------------------------------------
On Jan. 11, 2016, 9:39 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 11, 2016, 9:39 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
> On Jan. 8, 2016, 7:16 a.m., Anand Mazumdar wrote:
> > LGTM. Two minor issues:
> >
> > 1. May be we can escape without passing the `Master` pointer around?
> > 2. Would be great to have a test to go with this?
I am submitting the test as a separate patch.
> On Jan. 8, 2016, 7:16 a.m., Anand Mazumdar wrote:
> > src/master/master.hpp, line 1356
> > <https://reviews.apache.org/r/42053/diff/1/?file=1186664#file1186664line1356>
> >
> > hmmm.. Do we need to pass an pointer to `Master` here? Looks like a overkill to me.
> >
> > Can't we either pass the `Flags` object or just pass on the value of `flags.max_completed_frameworks` here?
I've changed it to take a flags parameter instead of a pointer to master.
- Kevin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113428
-----------------------------------------------------------
On Jan. 11, 2016, 9:39 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 11, 2016, 9:39 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Anand Mazumdar <ma...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113428
-----------------------------------------------------------
LGTM. Two minor issues:
1. May be we can escape without passing the `Master` pointer around?
2. Would be great to have a test to go with this?
src/master/master.hpp (line 1356)
<https://reviews.apache.org/r/42053/#comment174051>
hmmm.. Do we need to pass an pointer to `Master` here? Looks like a overkill to me.
Can't we either pass the `Flags` object or just pass on the value of `flags.max_completed_frameworks` here?
src/master/master.hpp (line 1696)
<https://reviews.apache.org/r/42053/#comment174052>
Ditto as above. What do you think about just passing the `size_t` directly here?
If not, let's do: (since we already have the `Master` here as an argument)
s/`_master->`/`master->`, the `master` member would already have been initialized by now.
- Anand Mazumdar
On Jan. 8, 2016, 1:29 a.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 1:29 a.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
>
> Diffs
> -----
>
> src/master/constants.hpp f1624e1ed5091f08f57d4382b344cab8b05ba840
> src/master/constants.cpp 589f2ee55a24d7e8b9352a4c8f94a3785fd1bef4
> src/master/flags.hpp 9af6c68eef6bcf39d5776809fab6c66dc95da6b2
> src/master/flags.cpp f864419a6276356c97a0a4fe29e5cfce6c4660c4
> src/master/master.hpp f764915ff8bbf74b99a617e3c4735d38fc13e5f0
> src/master/master.cpp 145c5932610bd019eb090947569b608df6815c3a
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113418
-----------------------------------------------------------
Bad patch!
Reviews applied: [42053]
Failed command: ./support/apply-review.sh -n -r 42053
Error:
2016-01-08 06:02:12 URL:https://reviews.apache.org/r/42053/diff/raw/ [4818/4818] -> "42053.patch" [1]
error: patch failed: src/master/flags.hpp:78
error: src/master/flags.hpp: patch does not apply
error: patch failed: src/master/flags.cpp:444
error: src/master/flags.cpp: patch does not apply
- Mesos ReviewBot
On Jan. 8, 2016, 1:29 a.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 1:29 a.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
>
> Diffs
> -----
>
> src/master/constants.hpp f1624e1ed5091f08f57d4382b344cab8b05ba840
> src/master/constants.cpp 589f2ee55a24d7e8b9352a4c8f94a3785fd1bef4
> src/master/flags.hpp 9af6c68eef6bcf39d5776809fab6c66dc95da6b2
> src/master/flags.cpp f864419a6276356c97a0a4fe29e5cfce6c4660c4
> src/master/master.hpp f764915ff8bbf74b99a617e3c4735d38fc13e5f0
> src/master/master.cpp 145c5932610bd019eb090947569b608df6815c3a
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113547
-----------------------------------------------------------
Patch looks great!
Reviews applied: [42053]
Passed command: export OS=ubuntu:14.04;export CONFIGURATION="--verbose";export COMPILER=gcc; ./support/docker_build.sh
- Mesos ReviewBot
On Jan. 8, 2016, 5:47 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 5:47 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
>
> Diffs
> -----
>
> src/master/constants.hpp f1624e1ed5091f08f57d4382b344cab8b05ba840
> src/master/constants.cpp 589f2ee55a24d7e8b9352a4c8f94a3785fd1bef4
> src/master/flags.hpp 9af6c68eef6bcf39d5776809fab6c66dc95da6b2
> src/master/flags.cpp f864419a6276356c97a0a4fe29e5cfce6c4660c4
> src/master/master.hpp f764915ff8bbf74b99a617e3c4735d38fc13e5f0
> src/master/master.cpp 145c5932610bd019eb090947569b608df6815c3a
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
> On Jan. 12, 2016, 1:38 a.m., Ben Mahler wrote:
> > Looks great, thanks! Just a few trivial comments below, and we can commit this once the unit test is in place.
I've addressed the two issues below and pushed a new patch. Patch for unit test forthcoming.
- Kevin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113896
-----------------------------------------------------------
On Jan. 12, 2016, 6:37 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 6:37 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113896
-----------------------------------------------------------
Looks great, thanks! Just a few trivial comments below, and we can commit this once the unit test is in place.
src/master/flags.cpp (lines 464 - 472)
<https://reviews.apache.org/r/42053/#comment174680>
I'm a bit hesitant to say "in the cache" to the user, as it sounds like an implementation detail? Perhaps we should say:
"Maximum number of completed frameworks to store in memory."
Thoughts?
src/master/master.hpp (line 1352)
<https://reviews.apache.org/r/42053/#comment174682>
2 space indent :)
- Ben Mahler
On Jan. 11, 2016, 9:39 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 11, 2016, 9:39 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review113944
-----------------------------------------------------------
Patch looks great!
Reviews applied: [42053]
Passed command: export OS=ubuntu:14.04;export CONFIGURATION="--verbose";export COMPILER=gcc; ./support/docker_build.sh
- Mesos ReviewBot
On Jan. 11, 2016, 9:39 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 11, 2016, 9:39 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
> On Jan. 12, 2016, 10:33 p.m., Adam B wrote:
> > src/master/master.hpp, line 1683
> > <https://reviews.apache.org/r/42053/diff/5/?file=1194904#file1194904line1683>
> >
> > Could you call this `masterFlags` so it doesn't get confused with FrameworkInfo parameters?
> > Even better would be to pass only the specific flag you care about.
> >
> > Or, maybe we should finally separate the caching into a common class so you only have to set the sizes for the circular buffer once. We long ago discussed refactoring the circular buffer so we could enable storage/retrieval to/from an archive of the data that is no longer cached in memory.
> >
> > I'd be ok with any of these, so I'll let you and your Shepherd choose.
Me and Ben (my shepherd) talked about this and decided that passing flags might be better in case we decide to add more flags we want to pass through in the future. Renaming this field makes sense. Done.
> On Jan. 12, 2016, 10:33 p.m., Adam B wrote:
> > src/master/flags.cpp, lines 464-472
> > <https://reviews.apache.org/r/42053/diff/5/?file=1194903#file1194903line464>
> >
> > Since you're adding new command-line flags, you should also update `docs/configuration.md` so the 0.27 release managers don't have to do it for you.
Done. Website changes while under review can be viewed here:
http://c99.millennium.berkeley.edu:4567/documentation/latest/configuration/
- Kevin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review114077
-----------------------------------------------------------
On Jan. 12, 2016, 11:07 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 11:07 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> docs/configuration.md cbe7f5a338a0fc350c4b6c0e2f1f48bd0869ac34
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp 5268408fc63a28afabc27cba96d3ecb360608a65
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
> There is alos now a unit test for this as reviewed here:
> https://reviews.apache.org/r/42053/
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Adam B <ad...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review114077
-----------------------------------------------------------
Just a couple of suggestions.
src/master/flags.cpp (lines 464 - 472)
<https://reviews.apache.org/r/42053/#comment174866>
Since you're adding new command-line flags, you should also update `docs/configuration.md` so the 0.27 release managers don't have to do it for you.
src/master/master.hpp (line 1683)
<https://reviews.apache.org/r/42053/#comment174869>
Could you call this `masterFlags` so it doesn't get confused with FrameworkInfo parameters?
Even better would be to pass only the specific flag you care about.
Or, maybe we should finally separate the caching into a common class so you only have to set the sizes for the circular buffer once. We long ago discussed refactoring the circular buffer so we could enable storage/retrieval to/from an archive of the data that is no longer cached in memory.
I'd be ok with any of these, so I'll let you and your Shepherd choose.
- Adam B
On Jan. 12, 2016, 2:15 p.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 2:15 p.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp 5268408fc63a28afabc27cba96d3ecb360608a65
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
> There is alos now a unit test for this as reviewed here:
> https://reviews.apache.org/r/42053/
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 14, 2016, 11:25 p.m.)
Review request for mesos and Ben Mahler.
Changes
-------
Update based on bmahlers whitespace issue
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description (updated)
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Diffs (updated)
-----
docs/configuration.md 7e0eb9584f3cb24a388c44d7bc81cd87dbb4072e
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp 863a11c82d322f56db1ccf25b73a4ffff1304a46
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
There is alos now a unit test for this as reviewed here:
https://reviews.apache.org/r/42053/
Thanks,
Kevin Klues
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/#review114594
-----------------------------------------------------------
Ship it!
docs/configuration.md (lines 487 - 488)
<https://reviews.apache.org/r/42053/#comment175435>
Could you avoid the tab character here?
- Ben Mahler
On Jan. 13, 2016, 10:43 a.m., Kevin Klues wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42053/
> -----------------------------------------------------------
>
> (Updated Jan. 13, 2016, 10:43 a.m.)
>
>
> Review request for mesos and Ben Mahler.
>
>
> Bugs: MESOS-3307
> https://issues.apache.org/jira/browse/MESOS-3307
>
>
> Repository: mesos
>
>
> Description
> -------
>
> The default size of the buffers used to hold the state of completed
> tasks/frameworks is very large. However, many frameworks don't care much
> about this information when requesting a master's state. Moreover, if a
> large number of frameworks request this state simultaneously, the
> master can quickly become overwhelmed because the process of generating
> this state both blocks the master and takes up a lot of cycles. By
> allowing the master to configure the size of the buffers used to hold
> this state, we give it the power to significantly reduce the amount of
> state it needs to maintain.
>
> This change allows the master to limit the size of this state via
> command line flags.
>
> This commit is based on a pull request generated by Felix Bechstein at:
> https://github.com/apache/mesos/pull/82
>
> Review: https://reviews.apache.org/r/42053
>
>
> Diffs
> -----
>
> docs/configuration.md 7e0eb9584f3cb24a388c44d7bc81cd87dbb4072e
> src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
> src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
> src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
> src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
> src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
> src/master/master.cpp c122c30d943813fc3ce9e7025783c7231809b022
>
> Diff: https://reviews.apache.org/r/42053/diff/
>
>
> Testing
> -------
>
> On Darwin I launched a master with:
>
> ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
>
> and a slave with:
>
> ./bin/mesos-slave.sh --master=127.0.0.1:5050
>
> and then ran a bunch of instances of:
>
> ./src/test-framework --master=127.0.0.1:5050
>
> each of which runs 5 tasks to completion
>
> I then ran:
>
> curl http://localhost:5050/tasks
>
> and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
>
> There is alos now a unit test for this as reviewed here:
> https://reviews.apache.org/r/42053/
>
>
> Thanks,
>
> Kevin Klues
>
>
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 13, 2016, 10:43 a.m.)
Review request for mesos and Ben Mahler.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs
-----
docs/configuration.md 7e0eb9584f3cb24a388c44d7bc81cd87dbb4072e
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp c122c30d943813fc3ce9e7025783c7231809b022
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
There is alos now a unit test for this as reviewed here:
https://reviews.apache.org/r/42053/
Thanks,
Kevin Klues
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 13, 2016, 10:42 a.m.)
Review request for mesos and Ben Mahler.
Changes
-------
Missed one change from flags->masterFlags per Adam's previous comments
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs (updated)
-----
docs/configuration.md 7e0eb9584f3cb24a388c44d7bc81cd87dbb4072e
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp c122c30d943813fc3ce9e7025783c7231809b022
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
There is alos now a unit test for this as reviewed here:
https://reviews.apache.org/r/42053/
Thanks,
Kevin Klues
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 12, 2016, 11:07 p.m.)
Review request for mesos and Ben Mahler.
Changes
-------
Addressed Adam's issues.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs (updated)
-----
docs/configuration.md cbe7f5a338a0fc350c4b6c0e2f1f48bd0869ac34
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp 5268408fc63a28afabc27cba96d3ecb360608a65
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
There is alos now a unit test for this as reviewed here:
https://reviews.apache.org/r/42053/
Thanks,
Kevin Klues
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 12, 2016, 10:15 p.m.)
Review request for mesos and Ben Mahler.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs
-----
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp 5268408fc63a28afabc27cba96d3ecb360608a65
Diff: https://reviews.apache.org/r/42053/diff/
Testing (updated)
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
There is alos now a unit test for this as reviewed here:
https://reviews.apache.org/r/42053/
Thanks,
Kevin Klues
Re: Review Request 42053: Added flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 12, 2016, 10:06 p.m.)
Review request for mesos and Ben Mahler.
Summary (updated)
-----------------
Added flags to set size of completed task/framework history.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs
-----
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp 5268408fc63a28afabc27cba96d3ecb360608a65
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
Thanks,
Kevin Klues
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 12, 2016, 10:05 p.m.)
Review request for mesos and Ben Mahler.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs (updated)
-----
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp 5268408fc63a28afabc27cba96d3ecb360608a65
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
Thanks,
Kevin Klues
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 12, 2016, 6:37 p.m.)
Review request for mesos and Ben Mahler.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs
-----
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
Thanks,
Kevin Klues
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 12, 2016, 6:36 p.m.)
Review request for mesos and Ben Mahler.
Changes
-------
Fixed bmahler's isses related to whitespace and wording of new flag descriptions.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs (updated)
-----
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
Thanks,
Kevin Klues
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 11, 2016, 9:39 p.m.)
Review request for mesos and Ben Mahler.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description (updated)
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Review: https://reviews.apache.org/r/42053
Diffs (updated)
-----
src/master/constants.hpp ebab341e58035d4b579828add752c1ee37efeb95
src/master/constants.cpp 77dd31430776e4f24e6e074c1470edcb19e58449
src/master/flags.hpp d923b1b0444d7e9023f1db4cbc4f7d4b84c20ff5
src/master/flags.cpp 88909590ff421421659e6faac7f3444bdc57b630
src/master/master.hpp f02d165874fa8023675e545793de699aeecae29b
src/master/master.cpp 2d9b7f9540574aa3ef9e5af3b2b8922dffeebac8
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
Thanks,
Kevin Klues
Re: Review Request 42053: Add flags to set size of completed
task/framework history.
Posted by Kevin Klues <kl...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42053/
-----------------------------------------------------------
(Updated Jan. 8, 2016, 5:47 p.m.)
Review request for mesos and Ben Mahler.
Changes
-------
This change doesn't resolve any issues and simply fixes a merge conflict due to added code via a rebase.
Bugs: MESOS-3307
https://issues.apache.org/jira/browse/MESOS-3307
Repository: mesos
Description
-------
The default size of the buffers used to hold the state of completed
tasks/frameworks is very large. However, many frameworks don't care much
about this information when requesting a master's state. Moreover, if a
large number of frameworks request this state simultaneously, the
master can quickly become overwhelmed because the process of generating
this state both blocks the master and takes up a lot of cycles. By
allowing the master to configure the size of the buffers used to hold
this state, we give it the power to significantly reduce the amount of
state it needs to maintain.
This change allows the master to limit the size of this state via
command line flags.
This commit is based on a pull request generated by Felix Bechstein at:
https://github.com/apache/mesos/pull/82
Diffs (updated)
-----
src/master/constants.hpp f1624e1ed5091f08f57d4382b344cab8b05ba840
src/master/constants.cpp 589f2ee55a24d7e8b9352a4c8f94a3785fd1bef4
src/master/flags.hpp 9af6c68eef6bcf39d5776809fab6c66dc95da6b2
src/master/flags.cpp f864419a6276356c97a0a4fe29e5cfce6c4660c4
src/master/master.hpp f764915ff8bbf74b99a617e3c4735d38fc13e5f0
src/master/master.cpp 145c5932610bd019eb090947569b608df6815c3a
Diff: https://reviews.apache.org/r/42053/diff/
Testing
-------
On Darwin I launched a master with:
./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos --max_completed_tasks_per_framework=2 --max_completed_frameworks=1
and a slave with:
./bin/mesos-slave.sh --master=127.0.0.1:5050
and then ran a bunch of instances of:
./src/test-framework --master=127.0.0.1:5050
each of which runs 5 tasks to completion
I then ran:
curl http://localhost:5050/tasks
and verified that only 1 framework and 2 of its completed tasks were given back to me in the json that was returned. I repeated this for a number of other configurations with max_completed_frameworks=0..2 and max_completed_tasks_per_framework=0..5 and verified visually that the proper number of tasks/frameworks were being returned by the /tasks endpoint.
Thanks,
Kevin Klues