You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Dmitriy Shirchenko <ca...@gmail.com> on 2017/07/05 21:50:52 UTC
Re: Review Request 57487: Implementation of Dynamic Reservations
Proposal
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57487/
-----------------------------------------------------------
(Updated July 5, 2017, 9:50 p.m.)
Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.
Changes
-------
Full patch running in our prod. Note: if you use dynamic resource estimator (for oversubscription), use Mesos > 1.2.1. Also, this patch contains small orthogonal fixes: allowing rootfs and changing task id length. I believe no harm in adding them since I don't want to unresolve merge conflicts.
Repository: aurora
Description
-------
Esteemed reviewers, here is the latest iteration on the implementation of dynamic reservations. Some changes are merging of the patches into a single one, updated design document with a more high level overview and user stories told from an operator’s point of view. Unit TESTS are going to be done as soon as we agree on the approach, as I have tested this patch on local vagrant and a multi-node dev cluster. Jenkins build is expected to fail as tested are incomplete.
For reference, here are previous two patches which feedback I addressed in this new single patch.
Previous 2 patches:
https://reviews.apache.org/r/56690/
https://reviews.apache.org/r/56691/
RFC document: https://docs.google.com/document/d/15n29HSQPXuFrnxZAgfVINTRP1Iv47_jfcstJNuMwr5A
Design Doc [UPDATED]: https://docs.google.com/document/d/1L2EKEcKKBPmuxRviSUebyuqiNwaO-2hsITBjt3SgWvE
Diffs (updated)
-----
examples/vagrant/mesos_config/etc_mesos-slave/resources aa0e97e1c4a6c1a76cc712549159db9336d051eb
examples/vagrant/upstart/aurora-scheduler.conf 63fcc87be653835cb3c3f25dae4164f1d7c8d4da
src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java f2296a9d7a88be7e43124370edecfe64415df00f
src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java c81387f24d554bcb2ee73e49028ba068ad11e4d6
src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 6f2ca35c5d83dde29c24865b4826d4932e96da80
src/main/java/org/apache/aurora/scheduler/HostOffer.java bc40d0798f40003cab5bf6efe607217e4d5de9f1
src/main/java/org/apache/aurora/scheduler/TaskVars.java 676dfd9f9d7ee0633c05424f788fd0ab116976bb
src/main/java/org/apache/aurora/scheduler/TierInfo.java c45b949ae7946fc92d7e62f94696ddc4f0790cfa
src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java b12ac83168401c15fb1d30179ea8e4816f09cd3d
src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java f0b148cd158d61cd89cc51dca9f3fa4c6feb1b49
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 754fde0fdc976b673d78ae15d8ccd8c85b792373
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java 4dac9757a65e144142d36ee921b85a02a5311fe5
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java 5c987fd051728486172c8afd34219e86d56f00d5
src/main/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilter.java f6c759f03c4152ae93317692fc9db202fe251122
src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 36608a9f027c95723c31f9915852112beb367223
src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java df51d4cf4893899613683603ab4aa9aefa88faa6
src/main/java/org/apache/aurora/scheduler/mesos/FrameworkInfoFactory.java 8ad9d216ad14534ec53ab506e92b669632db891a
src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 5a5281aeaea1e2a4e0eab67069605838ee809c6c
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 0d639f66db456858278b0485c91c40975c3b45ac
src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java e1cd81e6fbd98f23046e6e775be268be4310c62a
src/main/java/org/apache/aurora/scheduler/mesos/VersionedSchedulerDriverService.java 5e86504c70083065278864e6ab1cc85c83a45a28
src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 78255e6dfa31c4920afc0221ee60ec4f8c2a12c4
src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java adf7f33e4a72d87c3624f84dfe4998e20dc75fdc
src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 317a2d26d8bfa27988c60a7706b9fb3aa9b4e2a2
src/main/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilter.java 5ed578cc4c11b49f607db5f7e516d9e6022a926c
src/main/java/org/apache/aurora/scheduler/resources/AcceptedOffer.java 291d5c95916915afc48a7143759e523fccd52feb
src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 9aa263a9cfae03a9a0c5bc7fe3a1405397d3009c
src/main/java/org/apache/aurora/scheduler/resources/ResourceMapper.java 375f93c5277a78666fc4823382c82ac4d179f39d
src/main/java/org/apache/aurora/scheduler/scheduling/ReservationTimeoutCalculator.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/scheduling/SchedulingModule.java 03a0e8485d1a392f107fda5b4af05b7f8f6067c6
src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java 203f62bacc47470545d095e4d25f7e0f25990ed9
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java a177b301203143539b052524d14043ec8a85a46d
src/main/java/org/apache/aurora/scheduler/stats/AsyncStatsModule.java 40451e91aed45866c2030d901160cc4e084834df
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 059fbb86a575f5b3d78a63c9a7b5a9eebb6cb3ae
src/main/resources/org/apache/aurora/scheduler/tiers.json 34ddb1dc769a73115c209c9b2ee158cd364392d8
src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 82e40d509d84c37a19b6a9ef942283d908833840
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java a7c9c83eebbbea7ae8a6c807f98d3ce8bd050137
src/test/java/org/apache/aurora/scheduler/base/InstanceKeysTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 50d7499f4332a3feb0e2301cb707f2cea6bb2e98
src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java bf9c2b46b186125061c29ac777134a871c727066
src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java d2fe7a3d78c9c12ca3d0b40843eb7351217fe1fb
src/test/java/org/apache/aurora/scheduler/http/OffersTest.java 30699596a1c95199df7504f62c5c18cab1be1c6c
src/test/java/org/apache/aurora/scheduler/mesos/FrameworkInfoFactoryImplTest.java b22f047ba8ead5d840e13e379ec4471288b21118
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 93cc34cf8393f969087cd0fd6f577228c00170e9
src/test/java/org/apache/aurora/scheduler/offers/HostOffers.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java d7addc0effb60c196cf339081ad81de541d05385
src/test/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilterTest.java c39b00d65c8c2f2f74c836c6117a651fb7f9cd05
src/test/java/org/apache/aurora/scheduler/resources/AcceptedOfferTest.java dded9c34749cf599d197ed312ffb6bf63b6033f1
src/test/java/org/apache/aurora/scheduler/resources/PortMapperTest.java cef795bff131df46a50b632eaf2261df0b8dc3e7
src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java b8b8edb1a21ba89b8b60f8f8451c8c776fc23ae8
src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 676d305d257585e53f0a05b359ba7eb11f1b23be
src/test/java/org/apache/aurora/scheduler/scheduling/ReservationTimeoutCalculatorTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerImplTest.java fa1a81785802b82542030e1aae786fe9570d9827
src/test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 78f440f7546de9ed6842cb51db02b3bddc9a74ff
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java cf2d25ec2e407df7159e0021ddb44adf937e1777
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 016859ca3bf83f64d2576b4c7109729770f9e25c
src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 0361b36be3bc1f7d925b616d67dde5c64fd7a909
src/test/sh/org/apache/aurora/e2e/http/http_example_bad_healthcheck.aurora c826a54809ea8bcd25fa04a8fa15ecd414a53e30
src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora b85907a955a1ba83012a26912938b6466f0476ac
src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 1a81dc5dcde0400510c5576ac523f4d3f14424ca
Diff: https://reviews.apache.org/r/57487/diff/8/
Changes: https://reviews.apache.org/r/57487/diff/7-8/
Testing
-------
Tested on local vagrant for following scenarios:
Reserving a task
Making sure returned offer comes back
Making sure offer is unreserved
Thanks,
Dmitriy Shirchenko
Re: Review Request 57487: Implementation of Dynamic Reservations
Proposal
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57487/#review179708
-----------------------------------------------------------
This patch does not apply cleanly against master (a922b05), do you need to rebase?
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On July 5, 2017, 2:50 p.m., Dmitriy Shirchenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57487/
> -----------------------------------------------------------
>
> (Updated July 5, 2017, 2:50 p.m.)
>
>
> Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Esteemed reviewers, here is the latest iteration on the implementation of dynamic reservations. Some changes are merging of the patches into a single one, updated design document with a more high level overview and user stories told from an operator’s point of view. Unit TESTS are going to be done as soon as we agree on the approach, as I have tested this patch on local vagrant and a multi-node dev cluster. Jenkins build is expected to fail as tested are incomplete.
>
> For reference, here are previous two patches which feedback I addressed in this new single patch.
> Previous 2 patches:
> https://reviews.apache.org/r/56690/
> https://reviews.apache.org/r/56691/
>
> RFC document: https://docs.google.com/document/d/15n29HSQPXuFrnxZAgfVINTRP1Iv47_jfcstJNuMwr5A
> Design Doc [UPDATED]: https://docs.google.com/document/d/1L2EKEcKKBPmuxRviSUebyuqiNwaO-2hsITBjt3SgWvE
>
>
> Diffs
> -----
>
> examples/vagrant/mesos_config/etc_mesos-slave/resources aa0e97e1c4a6c1a76cc712549159db9336d051eb
> examples/vagrant/upstart/aurora-scheduler.conf 63fcc87be653835cb3c3f25dae4164f1d7c8d4da
> src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java f2296a9d7a88be7e43124370edecfe64415df00f
> src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java c81387f24d554bcb2ee73e49028ba068ad11e4d6
> src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 6f2ca35c5d83dde29c24865b4826d4932e96da80
> src/main/java/org/apache/aurora/scheduler/HostOffer.java bc40d0798f40003cab5bf6efe607217e4d5de9f1
> src/main/java/org/apache/aurora/scheduler/TaskVars.java 676dfd9f9d7ee0633c05424f788fd0ab116976bb
> src/main/java/org/apache/aurora/scheduler/TierInfo.java c45b949ae7946fc92d7e62f94696ddc4f0790cfa
> src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java b12ac83168401c15fb1d30179ea8e4816f09cd3d
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java f0b148cd158d61cd89cc51dca9f3fa4c6feb1b49
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 754fde0fdc976b673d78ae15d8ccd8c85b792373
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java 4dac9757a65e144142d36ee921b85a02a5311fe5
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java 5c987fd051728486172c8afd34219e86d56f00d5
> src/main/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilter.java f6c759f03c4152ae93317692fc9db202fe251122
> src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 36608a9f027c95723c31f9915852112beb367223
> src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java df51d4cf4893899613683603ab4aa9aefa88faa6
> src/main/java/org/apache/aurora/scheduler/mesos/FrameworkInfoFactory.java 8ad9d216ad14534ec53ab506e92b669632db891a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 5a5281aeaea1e2a4e0eab67069605838ee809c6c
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 0d639f66db456858278b0485c91c40975c3b45ac
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java e1cd81e6fbd98f23046e6e775be268be4310c62a
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedSchedulerDriverService.java 5e86504c70083065278864e6ab1cc85c83a45a28
> src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 78255e6dfa31c4920afc0221ee60ec4f8c2a12c4
> src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java adf7f33e4a72d87c3624f84dfe4998e20dc75fdc
> src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 317a2d26d8bfa27988c60a7706b9fb3aa9b4e2a2
> src/main/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilter.java 5ed578cc4c11b49f607db5f7e516d9e6022a926c
> src/main/java/org/apache/aurora/scheduler/resources/AcceptedOffer.java 291d5c95916915afc48a7143759e523fccd52feb
> src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 9aa263a9cfae03a9a0c5bc7fe3a1405397d3009c
> src/main/java/org/apache/aurora/scheduler/resources/ResourceMapper.java 375f93c5277a78666fc4823382c82ac4d179f39d
> src/main/java/org/apache/aurora/scheduler/scheduling/ReservationTimeoutCalculator.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/scheduling/SchedulingModule.java 03a0e8485d1a392f107fda5b4af05b7f8f6067c6
> src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java 203f62bacc47470545d095e4d25f7e0f25990ed9
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java a177b301203143539b052524d14043ec8a85a46d
> src/main/java/org/apache/aurora/scheduler/stats/AsyncStatsModule.java 40451e91aed45866c2030d901160cc4e084834df
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 059fbb86a575f5b3d78a63c9a7b5a9eebb6cb3ae
> src/main/resources/org/apache/aurora/scheduler/tiers.json 34ddb1dc769a73115c209c9b2ee158cd364392d8
> src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 82e40d509d84c37a19b6a9ef942283d908833840
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java a7c9c83eebbbea7ae8a6c807f98d3ce8bd050137
> src/test/java/org/apache/aurora/scheduler/base/InstanceKeysTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 50d7499f4332a3feb0e2301cb707f2cea6bb2e98
> src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java bf9c2b46b186125061c29ac777134a871c727066
> src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java d2fe7a3d78c9c12ca3d0b40843eb7351217fe1fb
> src/test/java/org/apache/aurora/scheduler/http/OffersTest.java 30699596a1c95199df7504f62c5c18cab1be1c6c
> src/test/java/org/apache/aurora/scheduler/mesos/FrameworkInfoFactoryImplTest.java b22f047ba8ead5d840e13e379ec4471288b21118
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 93cc34cf8393f969087cd0fd6f577228c00170e9
> src/test/java/org/apache/aurora/scheduler/offers/HostOffers.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java d7addc0effb60c196cf339081ad81de541d05385
> src/test/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilterTest.java c39b00d65c8c2f2f74c836c6117a651fb7f9cd05
> src/test/java/org/apache/aurora/scheduler/resources/AcceptedOfferTest.java dded9c34749cf599d197ed312ffb6bf63b6033f1
> src/test/java/org/apache/aurora/scheduler/resources/PortMapperTest.java cef795bff131df46a50b632eaf2261df0b8dc3e7
> src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java b8b8edb1a21ba89b8b60f8f8451c8c776fc23ae8
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 676d305d257585e53f0a05b359ba7eb11f1b23be
> src/test/java/org/apache/aurora/scheduler/scheduling/ReservationTimeoutCalculatorTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerImplTest.java fa1a81785802b82542030e1aae786fe9570d9827
> src/test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 78f440f7546de9ed6842cb51db02b3bddc9a74ff
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java cf2d25ec2e407df7159e0021ddb44adf937e1777
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 016859ca3bf83f64d2576b4c7109729770f9e25c
> src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 0361b36be3bc1f7d925b616d67dde5c64fd7a909
> src/test/sh/org/apache/aurora/e2e/http/http_example_bad_healthcheck.aurora c826a54809ea8bcd25fa04a8fa15ecd414a53e30
> src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora b85907a955a1ba83012a26912938b6466f0476ac
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 1a81dc5dcde0400510c5576ac523f4d3f14424ca
>
>
> Diff: https://reviews.apache.org/r/57487/diff/8/
>
>
> Testing
> -------
>
> Tested on local vagrant for following scenarios:
> Reserving a task
> Making sure returned offer comes back
> Making sure offer is unreserved
>
>
> Thanks,
>
> Dmitriy Shirchenko
>
>