You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Jordan Ly <jo...@gmail.com> on 2018/01/18 19:13:11 UTC
Review Request 65225: Allow custom data structures in HostOffers
injected via Guice modules
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65225/
-----------------------------------------------------------
Review request for Aurora, David McLaughlin and Bill Farner.
Repository: aurora
Description
-------
Following along with the custom OfferOrder work done in https://reviews.apache.org/r/59480/ and https://reviews.apache.org/r/59698/, as well as the custom selector work in https://reviews.apache.org/r/63973/.
It would be helpful to be able to inject a custom data structure to hold offers that takes advantage of different orderings.
For example: if we used a bin-packing ordering and we wanted to select an empty host quickly for certain jobs, we could inject a data structure that also holds all empty slots for quick retrieval as opposed to iterating through the whole list as we are currently required to do.
I've created a `FilterableCollection` in order to generalize the idea that we have a collection that has filters applied at different levels (in our case, we apply filters in `HostOffers` and in `TaskAssignerImpl` before we iterate for selection. I've also created a thin `FilterableCollectionImpl` that mimics the current functionality in master (a wrapper around `ConcurrentSkipListSet` with filters applied for the necessary functions.
Additionally, I've added the `offer_collection_modules` so cluster operators can inject their own custom data structures.
Diffs
-----
src/main/java/org/apache/aurora/scheduler/offers/FilterableCollection.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/offers/FilterableCollectionImpl.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/offers/HostOffers.java 2ea7a01085b87c8ed6765537a8005e2349784ab0
src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 8f9e33d81be9087821784a8a08079a1736d9cb63
src/main/java/org/apache/aurora/scheduler/offers/OfferManagerImpl.java 084b48c6b77bb38cf0ed2709cc8a4bffec68597a
src/main/java/org/apache/aurora/scheduler/offers/OfferManagerModule.java de16c14c887d4225e0629b1580eb5e740798f1f7
src/main/java/org/apache/aurora/scheduler/offers/OfferOrderBuilder.java 1260ef19506acb8e8a937d4fd7b7152361bd3c40
src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java 1e36b2c3094e99f05aa4a4f098a48df2293b4320
Diff: https://reviews.apache.org/r/65225/diff/1/
Testing
-------
TODO: I am planning on adding tests for `FilterableCollection`, just wanted to put this out there for quick comments.
I've done some ad-hoc testing of injecting custom collection types.
Thanks,
Jordan Ly
Re: Review Request 65225: Allow custom data structures in HostOffers
injected via Guice modules
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65225/#review195746
-----------------------------------------------------------
Master (8a5dcac) is red with this patch.
./build-support/jenkins/build.sh
^
constructor OfferSettings.OfferSettings(Amount<Long,Time>,List<OfferOrder>,FilterableCollectionFactory,Amount<Long,Time>,long,Ticker) is not applicable
(actual and formal argument lists differ in length)
constructor OfferSettings.OfferSettings(Amount<Long,Time>,Ordering<HostOffer>,FilterableCollectionFactory,Amount<Long,Time>,long,Ticker) is not applicable
(actual and formal argument lists differ in length)
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java:387: error: no suitable constructor found for OfferSettings(Amount<Long,Time>,List<OfferOrder>,Amount<Long,Time>,long,FakeTicker)
new OfferSettings(
^
constructor OfferSettings.OfferSettings(Amount<Long,Time>,List<OfferOrder>,FilterableCollectionFactory,Amount<Long,Time>,long,Ticker) is not applicable
(actual and formal argument lists differ in length)
constructor OfferSettings.OfferSettings(Amount<Long,Time>,Ordering<HostOffer>,FilterableCollectionFactory,Amount<Long,Time>,long,Ticker) is not applicable
(actual and formal argument lists differ in length)
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java:562: error: no suitable constructor found for OfferSettings(Amount<Long,Time>,ImmutableList<OfferOrder>,Amount<Long,Time>,long,FakeTicker)
OfferSettings settings = new OfferSettings(
^
constructor OfferSettings.OfferSettings(Amount<Long,Time>,List<OfferOrder>,FilterableCollectionFactory,Amount<Long,Time>,long,Ticker) is not applicable
(actual and formal argument lists differ in length)
constructor OfferSettings.OfferSettings(Amount<Long,Time>,Ordering<HostOffer>,FilterableCollectionFactory,Amount<Long,Time>,long,Ticker) is not applicable
(actual and formal argument lists differ in length)
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/storage/durability/DurableStorageTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: Some messages have been simplified; recompile with -Xdiags:verbose to get full output
7 errors
FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':compileTestJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 5m 16s
28 actionable tasks: 22 executed, 6 up-to-date
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On Jan. 18, 2018, 7:13 p.m., Jordan Ly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65225/
> -----------------------------------------------------------
>
> (Updated Jan. 18, 2018, 7:13 p.m.)
>
>
> Review request for Aurora, David McLaughlin and Bill Farner.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Following along with the custom OfferOrder work done in https://reviews.apache.org/r/59480/ and https://reviews.apache.org/r/59698/, as well as the custom selector work in https://reviews.apache.org/r/63973/.
>
> It would be helpful to be able to inject a custom data structure to hold offers that takes advantage of different orderings.
>
> For example: if we used a bin-packing ordering and we wanted to select an empty host quickly for certain jobs, we could inject a data structure that also holds all empty slots for quick retrieval as opposed to iterating through the whole list as we are currently required to do.
>
> I've created a `FilterableCollection` in order to generalize the idea that we have a collection that has filters applied at different levels (in our case, we apply filters in `HostOffers` and in `TaskAssignerImpl` before we iterate for selection. I've also created a thin `FilterableCollectionImpl` that mimics the current functionality in master (a wrapper around `ConcurrentSkipListSet` with filters applied for the necessary functions.
>
> Additionally, I've added the `offer_collection_modules` so cluster operators can inject their own custom data structures.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/offers/FilterableCollection.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/offers/FilterableCollectionImpl.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/offers/HostOffers.java 2ea7a01085b87c8ed6765537a8005e2349784ab0
> src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 8f9e33d81be9087821784a8a08079a1736d9cb63
> src/main/java/org/apache/aurora/scheduler/offers/OfferManagerImpl.java 084b48c6b77bb38cf0ed2709cc8a4bffec68597a
> src/main/java/org/apache/aurora/scheduler/offers/OfferManagerModule.java de16c14c887d4225e0629b1580eb5e740798f1f7
> src/main/java/org/apache/aurora/scheduler/offers/OfferOrderBuilder.java 1260ef19506acb8e8a937d4fd7b7152361bd3c40
> src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java 1e36b2c3094e99f05aa4a4f098a48df2293b4320
>
>
> Diff: https://reviews.apache.org/r/65225/diff/1/
>
>
> Testing
> -------
>
> TODO: I am planning on adding tests for `FilterableCollection`, just wanted to put this out there for quick comments.
>
> I've done some ad-hoc testing of injecting custom collection types.
>
>
> Thanks,
>
> Jordan Ly
>
>