You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Maxim Khutornenko <ma...@apache.org> on 2016/04/21 23:57:24 UTC
Review Request 46459: Schema changes for resource management
refactoring
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/
-----------------------------------------------------------
Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
Repository: aurora
Description
-------
This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
A few design/implementation notes:
- All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
- The requested ports have been flatmapped together with other resource types to simplify handling
- Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
- `IResource` immutable wrapper got couple of new methods:
- `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
- `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
Notes on data migration/backfilling:
- The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
- Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
- `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
- `LogStorage` backfilling covers transaction replays
- `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
Diffs
-----
api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
Diff: https://reviews.apache.org/r/46459/diff/
Testing
-------
Manual scheduler upgrades/rollbacks in Vagrant.
Thanks,
Maxim Khutornenko
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
> On April 22, 2016, 3:15 p.m., Joshua Cohen wrote:
> > Overall looks good to me. One question though: what happens in the scenario where we receive a TaskConfig that has both the individual task-level resources populated as well as the new resource collection. Further, what happens in that scenario if the values don't match? I don't think we can rely on the client to send us valid data because there are lots of users who generate thrift requests directly, rather than using the client.
Doesn't `ThriftBackfill.backfillTask` already address this problem? If `TaskConfig.resources` is not empty the field-level fields are unconditionally rewritten to match. I suppose we could throw an error in this case but I think it's more robust to ignore deprecated fields and default to the new replacement fields instead. I plan to mark all fields as deprecated and add release notes when scheduler is fully moved to use `TaskConfig.resources`.
> On April 22, 2016, 3:15 p.m., Joshua Cohen wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml, line 161
> > <https://reviews.apache.org/r/46459/diff/2/?file=1356072#file1356072line161>
> >
> > Should this be included as part of the initial query rather than requring a sub-select?
That's what I started with but eventually had to change it to subselect as join did not work well with a case of empty resources (which is still possible to hit when scheduler restarts to a new version and data is about to be migrated). I presume this is the exact reason why the optional `taskLinks` has also been populated via a sub-select.
We can revisit it in the next version when all resources are migrated and old fields are being removed.
> On April 22, 2016, 3:15 p.m., Joshua Cohen wrote:
> > src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java, lines 49-52
> > <https://reviews.apache.org/r/46459/diff/2/?file=1356057#file1356057line49>
> >
> > Would it make sense to make this a default method on the interface?
This is an excellent idea. Done.
> On April 22, 2016, 3:15 p.m., Joshua Cohen wrote:
> > src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java, line 322
> > <https://reviews.apache.org/r/46459/diff/2/?file=1356054#file1356054line322>
> >
> > Do you think it would make sense to find all instances of resources with multiple values rather than stopping at the first? That way if someone specified multiple invalid resources they can avoid multiple round trips to correct each instance?
> >
> > (I suppose it does potentially open up the ability for a malicious user to craft a config that could cause the scheduler to perform extra work, but I'm not sure we're really protected against that scenario in general)
We usually just error out on the first error found instead of trying to validate the entire config. In this particular case though I suppose it's easy to validate all resources without any extra risk. Changed.
- Maxim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130096
-----------------------------------------------------------
On April 21, 2016, 11:03 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 21, 2016, 11:03 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Joshua Cohen <jc...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130096
-----------------------------------------------------------
Overall looks good to me. One question though: what happens in the scenario where we receive a TaskConfig that has both the individual task-level resources populated as well as the new resource collection. Further, what happens in that scenario if the values don't match? I don't think we can rely on the client to send us valid data because there are lots of users who generate thrift requests directly, rather than using the client.
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java (line 322)
<https://reviews.apache.org/r/46459/#comment193745>
Do you think it would make sense to find all instances of resources with multiple values rather than stopping at the first? That way if someone specified multiple invalid resources they can avoid multiple round trips to correct each instance?
(I suppose it does potentially open up the ability for a malicious user to craft a config that could cause the scheduler to perform extra work, but I'm not sure we're really protected against that scenario in general)
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java (lines 49 - 52)
<https://reviews.apache.org/r/46459/#comment193746>
Would it make sense to make this a default method on the interface?
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml (line 161)
<https://reviews.apache.org/r/46459/#comment193757>
Should this be included as part of the initial query rather than requring a sub-select?
- Joshua Cohen
On April 21, 2016, 11:03 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 21, 2016, 11:03 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review129986
-----------------------------------------------------------
Ship it!
Master (95dcca5) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 21, 2016, 11:03 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 21, 2016, 11:03 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
> On April 22, 2016, 6:55 p.m., Zameer Manji wrote:
> > src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java, line 210
> > <https://reviews.apache.org/r/46459/diff/3/?file=1357528#file1357528line210>
> >
> > Instead of having a switch statement here, perhaps it would be cleaner if we hace a private static ImmutableMap which mapped the thrift to json types.
I don't find this indirection any easier to follow. It's also consistent with the internal thrift type handling.
> On April 22, 2016, 6:55 p.m., Zameer Manji wrote:
> > src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java, line 143
> > <https://reviews.apache.org/r/46459/diff/3/?file=1357529#file1357529line143>
> >
> > To prevent human errors when adding more resouces to this enum perhaps we should make the type here `Resources$_Fields` and do the `getThriftFieldId()` inside the constructor?
That's a fine suggestion! It also allowed for single line definitions (at least until we have more params there).
> On April 22, 2016, 6:55 p.m., Zameer Manji wrote:
> > src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java, line 37
> > <https://reviews.apache.org/r/46459/diff/3/?file=1357530#file1357530line37>
> >
> > Shouldn't this be `T value` ?
Unfortunately, it can't. It has to handle `IResource.getRawValue()`, which returns an opaque thrift union value. This is the only way to handle type conversions in a generic way.
> On April 22, 2016, 6:55 p.m., Zameer Manji wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java, line 48
> > <https://reviews.apache.org/r/46459/diff/3/?file=1357535#file1357535line48>
> >
> > Perhaps to reduce duplication we could use `ResourceType.<resource>.getAuroraName()` here instead of hard coding in CPUS, RAM_MB, etc?
It has to be the enum name here, not the `getAuroraName()` but the suggestion still makes sense.
- Maxim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130140
-----------------------------------------------------------
On April 22, 2016, 10:48 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 22, 2016, 10:48 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Zameer Manji <zm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130140
-----------------------------------------------------------
src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java (line 210)
<https://reviews.apache.org/r/46459/#comment193814>
Instead of having a switch statement here, perhaps it would be cleaner if we hace a private static ImmutableMap which mapped the thrift to json types.
src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java (line 141)
<https://reviews.apache.org/r/46459/#comment193824>
To prevent human errors when adding more resouces to this enum perhaps we should make the type here `Resources$_Fields` and do the `getThriftFieldId()` inside the constructor?
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java (line 37)
<https://reviews.apache.org/r/46459/#comment193831>
Shouldn't this be `T value` ?
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java (line 48)
<https://reviews.apache.org/r/46459/#comment193835>
Perhaps to reduce duplication we could use `ResourceType.<resource>.getAuroraName()` here instead of hard coding in CPUS, RAM_MB, etc?
- Zameer Manji
On April 22, 2016, 9:45 a.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 22, 2016, 9:45 a.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130120
-----------------------------------------------------------
Master (95dcca5) is red with this patch.
./build-support/jenkins/build.sh
[1m with temporary_dir() as checkpoint_root:[0m
[1m te = AuroraExecutor([0m
[1m> runner_provider=make_provider(checkpoint_root),[0m
[1m sandbox_provider=DefaultTestSandboxProvider())[0m
src/test/python/apache/aurora/executor/test_thermos_executor.py:580:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/test/python/apache/aurora/executor/test_thermos_executor.py:193: in make_provider
[1m pex_location=thermos_runner_path(),[0m
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
build = True
[1m def thermos_runner_path(build=True):[0m
[1m if not build:[0m
[1m return getattr(thermos_runner_path, 'value', None)[0m
[1m [0m
[1m if not hasattr(thermos_runner_path, 'value'):[0m
[1m pex_dir = safe_mkdtemp()[0m
[1m> assert subprocess.call(["./pants", "--pants-distdir=%s" % pex_dir, "binary",[0m
[1m "src/main/python/apache/thermos/runner:thermos_runner"]) == 0[0m
[1m[31mE assert 1 == 0[0m
[1m[31mE + where 1 = <function call at 0x7f244cd04b18>(['./pants', '--pants-distdir=/tmp/user/10021/tmpITHJ7v', 'binary', 'src/main/python/apache/thermos/runner:thermos_runner'])[0m
[1m[31mE + where <function call at 0x7f244cd04b18> = subprocess.call[0m
src/test/python/apache/aurora/executor/test_thermos_executor.py:185: AssertionError
-------------- Captured stderr call --------------
Traceback (most recent call last):
File "/home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/0.0.80/bin/pants", line 7, in <module>
from pants.bin.pants_exe import main
ImportError: No module named pants.bin.pants_exe
generated xml file: /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml
[1m[31m 16 failed, 632 passed, 6 skipped, 1 warnings, 8 error in 154.50 seconds [0m
FAILURE
[32m
Waiting for background workers to finish.[0m
17:03:58 03:27 [complete][31m
FAILURE[0m
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 22, 2016, 4:45 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 22, 2016, 4:45 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
> On April 24, 2016, 4:01 p.m., Bill Farner wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql, line 114
> > <https://reviews.apache.org/r/46459/diff/5/?file=1358264#file1358264line114>
> >
> > Generalizing all resources to varchar type is quite unfortunate. Have you considered using a column per resource type to at least avoid the string conversion?
Yes, I have considered this approach and ruled it out due to the following:
- To ensure data integrity (e.g. only one column with a non-null value per row) we'd have to write a very ugly check constaint on the `task_resource` table with a combinatorial explosion in complexity, e.g.:
```
...
CHECK (
(doubleType IS NOT NULL AND longType IS NULL AND stringType IS NULL)
OR (longType IS NOT NULL AND doubleType IS NULL AND stringType IS NULL)
OR (stringType IS NOT NULL AND doubleType IS NULL AND longType IS NULL)
...
);
```
- Cumbersome logic relying on hardcoded `ResourceType` enum values in MyBatis to map values to table columns on insert and select OR a non-standard way of doing the same in the java layer instead.
- Schema migrations and mapper/java code changes any time a new java type is introduced.
The current approach is much simpler as it does not require ANY changes to the SQL layer when a new resource type/kind is introduced. The only code change needed is a new `ResourceTypeConverter` interface implementation, which is mostly a single liner.
- Maxim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130308
-----------------------------------------------------------
On April 23, 2016, 12:16 a.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 23, 2016, 12:16 a.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Bill Farner <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130308
-----------------------------------------------------------
src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql (line 114)
<https://reviews.apache.org/r/46459/#comment194022>
Generalizing all resources to varchar type is quite unfortunate. Have you considered using a column per resource type to at least avoid the string conversion?
- Bill Farner
On April 22, 2016, 5:16 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 22, 2016, 5:16 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/
-----------------------------------------------------------
(Updated April 25, 2016, 11:18 p.m.)
Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
Changes
-------
Joshua's comments.
Repository: aurora
Description
-------
This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
A few design/implementation notes:
- All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
- The requested ports have been flatmapped together with other resource types to simplify handling
- Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
- `IResource` immutable wrapper got couple of new methods:
- `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
- `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
Notes on data migration/backfilling:
- The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
- Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
- `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
- `LogStorage` backfilling covers transaction replays
- `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
Diffs (updated)
-----
api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
Diff: https://reviews.apache.org/r/46459/diff/
Testing
-------
Manual scheduler upgrades/rollbacks in Vagrant.
Thanks,
Maxim Khutornenko
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
> On April 25, 2016, 2:34 p.m., Joshua Cohen wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java, line 107
> > <https://reviews.apache.org/r/46459/diff/5/?file=1358250#file1358250line107>
> >
> > Should we pull `ResourceType.findByResource(e)` up to a var, rather than calling it twice?
Since it's a small map lookup, there is not much to gain in perf here, so I'd prefer having a more concise current form. I realized 'find' sounded a bit heavy for its purpose though and changed it to better reflect the meaning.
> On April 25, 2016, 2:34 p.m., Joshua Cohen wrote:
> > src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java, line 32
> > <https://reviews.apache.org/r/46459/diff/5/?file=1358248#file1358248line32>
> >
> > Fix javadoc to drop reference to type parameter?
Changed to @code.
> On April 25, 2016, 2:34 p.m., Joshua Cohen wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml, lines 131-132
> > <https://reviews.apache.org/r/46459/diff/5/?file=1358263#file1358263line131>
> >
> > nit: fix indentation
Done.
- Maxim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130417
-----------------------------------------------------------
On April 23, 2016, 12:16 a.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 23, 2016, 12:16 a.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Joshua Cohen <jc...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130417
-----------------------------------------------------------
Ship it!
lgtm, just a few nits below.
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java (line 32)
<https://reviews.apache.org/r/46459/#comment194158>
Fix javadoc to drop reference to type parameter?
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java (line 107)
<https://reviews.apache.org/r/46459/#comment194159>
Should we pull `ResourceType.findByResource(e)` up to a var, rather than calling it twice?
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml (lines 131 - 132)
<https://reviews.apache.org/r/46459/#comment194161>
nit: fix indentation
- Joshua Cohen
On April 23, 2016, 12:16 a.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 23, 2016, 12:16 a.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130229
-----------------------------------------------------------
Ship it!
Master (d339036) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 23, 2016, 12:16 a.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 23, 2016, 12:16 a.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Bill Farner <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130507
-----------------------------------------------------------
Ship it!
Ship It!
- Bill Farner
On April 22, 2016, 5:16 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 22, 2016, 5:16 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Zameer Manji <zm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130506
-----------------------------------------------------------
Ship it!
Ship It!
- Zameer Manji
On April 22, 2016, 5:16 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 22, 2016, 5:16 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/
-----------------------------------------------------------
(Updated April 23, 2016, 12:16 a.m.)
Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
Changes
-------
Fixing merge key and adding unique constraint into task_resource table.
Repository: aurora
Description
-------
This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
A few design/implementation notes:
- All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
- The requested ports have been flatmapped together with other resource types to simplify handling
- Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
- `IResource` immutable wrapper got couple of new methods:
- `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
- `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
Notes on data migration/backfilling:
- The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
- Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
- `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
- `LogStorage` backfilling covers transaction replays
- `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
Diffs (updated)
-----
api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
Diff: https://reviews.apache.org/r/46459/diff/
Testing
-------
Manual scheduler upgrades/rollbacks in Vagrant.
Thanks,
Maxim Khutornenko
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review130215
-----------------------------------------------------------
Ship it!
Master (95dcca5) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 22, 2016, 10:48 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 22, 2016, 10:48 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/
-----------------------------------------------------------
(Updated April 22, 2016, 10:48 p.m.)
Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
Changes
-------
Zameer's comments.
Repository: aurora
Description
-------
This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
A few design/implementation notes:
- All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
- The requested ports have been flatmapped together with other resource types to simplify handling
- Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
- `IResource` immutable wrapper got couple of new methods:
- `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
- `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
Notes on data migration/backfilling:
- The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
- Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
- `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
- `LogStorage` backfilling covers transaction replays
- `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
Diffs (updated)
-----
api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
Diff: https://reviews.apache.org/r/46459/diff/
Testing
-------
Manual scheduler upgrades/rollbacks in Vagrant.
Thanks,
Maxim Khutornenko
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/
-----------------------------------------------------------
(Updated April 22, 2016, 4:45 p.m.)
Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
Changes
-------
Joshua's comments.
Repository: aurora
Description
-------
This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
A few design/implementation notes:
- All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
- The requested ports have been flatmapped together with other resource types to simplify handling
- Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
- `IResource` immutable wrapper got couple of new methods:
- `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
- `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
Notes on data migration/backfilling:
- The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
- Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
- `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
- `LogStorage` backfilling covers transaction replays
- `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
Diffs (updated)
-----
api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
Diff: https://reviews.apache.org/r/46459/diff/
Testing
-------
Manual scheduler upgrades/rollbacks in Vagrant.
Thanks,
Maxim Khutornenko
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Maxim Khutornenko <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/
-----------------------------------------------------------
(Updated April 21, 2016, 11:03 p.m.)
Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
Changes
-------
Fixing unit test.
Repository: aurora
Description
-------
This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
A few design/implementation notes:
- All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
- The requested ports have been flatmapped together with other resource types to simplify handling
- Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
- `IResource` immutable wrapper got couple of new methods:
- `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
- `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
Notes on data migration/backfilling:
- The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
- Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
- `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
- `LogStorage` backfilling covers transaction replays
- `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
Diffs (updated)
-----
api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
src/test/python/apache/aurora/client/cli/test_inspect.py 452699c08d642e7166903d7797c1650c735266db
src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
Diff: https://reviews.apache.org/r/46459/diff/
Testing
-------
Manual scheduler upgrades/rollbacks in Vagrant.
Thanks,
Maxim Khutornenko
Re: Review Request 46459: Schema changes for resource management
refactoring
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review129971
-----------------------------------------------------------
Master (95dcca5) is red with this patch.
./build-support/jenkins/build.sh
src/test/python/apache/thermos/common/test_pathspec.py::test_exception_on_none_keys [32mPASSED[0m
src/test/python/apache/thermos/common/test_planner.py::test_planner_empty [32mPASSED[0m
src/test/python/apache/thermos/common/test_planner.py::test_planner_unordered [32mPASSED[0m
src/test/python/apache/thermos/common/test_planner.py::test_planner_ordered [32mPASSED[0m
src/test/python/apache/thermos/common/test_planner.py::test_planner_mixed [32mPASSED[0m
src/test/python/apache/thermos/common/test_planner.py::test_planner_unsatisfiables [32mPASSED[0m
==================== FAILURES ====================
____ TestInspectCommand.test_inspect_job_raw _____
self = <aurora.client.cli.test_inspect.TestInspectCommand testMethod=test_inspect_job_raw>
[1m def test_inspect_job_raw(self):[0m
[1m mock_stdout = [][0m
[1m def mock_print_out(msg, indent=0):[0m
[1m indent_str = " " * indent[0m
[1m mock_stdout.append("%s%s" % (indent_str, msg))[0m
[1m job_config = self.get_job_config()[0m
[1m with contextlib.nested([0m
[1m patch('apache.aurora.client.cli.context.AuroraCommandContext.print_out',[0m
[1m side_effect=mock_print_out),[0m
[1m patch('apache.aurora.client.cli.context.AuroraCommandContext.get_job_config',[0m
[1m return_value=job_config)):[0m
[1m cmd = AuroraCommandLine()[0m
[1m assert cmd.execute(['job', 'inspect', '--raw', 'west/bozo/test/hello', 'config.aurora']) == 0[0m
[1m output = '\n'.join(mock_stdout)[0m
[1m> assert output == str(job_config.job())[0m
[1m[31mE AssertionError: assert "JobConfigura...r='jenkins'))" == "JobConfigurat...r='jenkins'))"[0m
[1m[31mE Detailed information truncated, use "-vv" to show[0m
src/test/python/apache/aurora/client/cli/test_inspect.py:92: AssertionError
generated xml file: /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml
[1m[31m 1 failed, 655 passed, 6 skipped, 1 warnings in 197.57 seconds [0m
FAILURE
[32m
Waiting for background workers to finish.[0m
22:17:21 04:03 [complete][31m
FAILURE[0m
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 21, 2016, 9:57 p.m., Maxim Khutornenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
>
> (Updated April 21, 2016, 9:57 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
>
> Apologies for the large diff but this is the minimal logically correct set of changes to ensure proper schema migration/backfilling.
>
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way we can handle additional resource types by just adding a `ResourceTypeConverter` for the new type. Among other benefits the increased human readability of SQL data simplifying ad-hoc analysis
> - The requested ports have been flatmapped together with other resource types to simplify handling
> - Resource details are described in `ResourceType` that provides translation bridge between generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
> - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource values in a generic way when reloading resources from DB;
> - `public Object getRawValue()`: for accessing resource value in a generic way when storing it in DB;
>
>
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB (e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore` disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server combinations
>
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486
> src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59
> src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java ed561c65947d41861d80229ad459392a4bceadf1
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1
> src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION
> src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be
> src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 1ccfe01111e6c7c18046a8193245e537ae207a13
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0
> src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88
> src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java e6e79a0547cafeee80d656dff09d7adb52e4c420
> src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e
>
> Diff: https://reviews.apache.org/r/46459/diff/
>
>
> Testing
> -------
>
> Manual scheduler upgrades/rollbacks in Vagrant.
>
>
> Thanks,
>
> Maxim Khutornenko
>
>