You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Stephan Erb <se...@apache.org> on 2017/08/03 21:50:19 UTC
Re: Review Request 60942: Remove task level resource fields from
thrift interface and db
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60942/#review182164
-----------------------------------------------------------
Thanks for the patch! Good work with the cleanup.
I am not an expert with the DB migrations so it would be great to hear feedback from others as well.
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java
Lines 33-42 (original), 32-34 (patched)
<https://reviews.apache.org/r/60942/#comment258028>
Isn't that the table we have keep around? Or is it safe because we assume users will update version by version and the table will always exist before they upgrade to 0.19 with this patch?
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml
Lines 174-178 (original), 165-169 (patched)
<https://reviews.apache.org/r/60942/#comment258031>
The comment is outdated now.
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml
Lines 228-237 (original), 216-226 (patched)
<https://reviews.apache.org/r/60942/#comment258035>
This is why we should never wonder why the DBTaskStore is so slow :-D
(this has nothing to do with the patch)
- Stephan Erb
On July 18, 2017, 4:43 p.m., Nicolás Donatucci wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60942/
> -----------------------------------------------------------
>
> (Updated July 18, 2017, 4:43 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Zameer Manji.
>
>
> Bugs: AURORA-1707
> https://issues.apache.org/jira/browse/AURORA-1707
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Removed task level resource fields from the DB and the thrift-API.
> To do this, a new DB migration was added. When upgrading, it just drops the task level resource fields. When downgrading, it creates the fields again and populates them with information from the task_resource table.
>
> IMPORTANT: One of the python client tests is failing (test_cron_diff). This is not serious, I think it is a problem with the ordering of the elements of Resources (had similar problems with other python client tests that instead of comparing the Resources as a set, compared them as lists and thus order mattered). Nevertheless, I could not fully understand the code of that test. I was hoping someone could give me a hand with that.
> But then again, it is a smaller issue and so the patch can start being reviewed.
>
> Issue Related: AURORA-1707
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 3749531b5412d7ca217736aa85eed8e6606225ad
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 186fa1b3a4780c0536fb486d50a33133258110cd
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java d2eb6aa6e4a155b2d28debab2ca10dfc76d57213
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java cda55c55680a19ed421299a8949299b21949787b
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java af106a8a9ee8c14122e98bcc0ec44b616f21d61f
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V011_DropResourceFields.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java 138cd5316adc73eed24fc7accc53885dd5d5bee5
> src/main/python/apache/aurora/client/cli/diff_formatter.py 78717774aa3fbaf83a5fb850bc9f9f4a4038d70f
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 5422183e4a1fe122fc0e1aa871aa75ae102e322d
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 7a86f47af67adb3e488381d30ddf424549deefbc
> src/test/java/org/apache/aurora/scheduler/http/TestUtils.java 689482c9f6c49bcca781834566edeb975d2f3af2
> src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java caaba9b6dff46ff0b037759f1c817a321ae15ee4
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 43e32eede27bbf26363a3fd1ca34ffe6f8c01a73
> src/test/java/org/apache/aurora/scheduler/updater/UpdateAgentReserverImplTest.java 1bc2a778ad3f1543a055023f9ec3fe9e4a9523e3
> src/test/python/apache/aurora/client/cli/test_status.py b0b7f96d8148f0dd1f6f45d1c5c6f830cabcfd5d
> src/test/python/apache/aurora/client/cli/util.py 43db828ca1cccd91c88f016b4c994fef33182fbf
> src/test/python/apache/aurora/config/test_resources.py 25a20f0b702189744a26b85053db34c37ff5b03c
>
>
> Diff: https://reviews.apache.org/r/60942/diff/2/
>
>
> Testing
> -------
>
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
>
> 1 - Created a new VM with a fresh Aurora 0.18.0, ran the tests so that the db would have something. Then upgraded to patched version and ran the tests again. Everything worked and task level resource fields were no longer there.
> 2 - Created a new VM with a fresh patched version, ran the tests so that the db would have something. Then downgraded to 0.18.0 and ran the tests again. Everything worked and task level resources were there, with the correct backfilled values.
>
>
> Thanks,
>
> Nicolás Donatucci
>
>
Re: Review Request 60942: Remove task level resource fields from
thrift interface and db
Posted by Nicolás Donatucci <nd...@medallia.com>.
> On Aug. 3, 2017, 9:50 p.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java
> > Lines 33-42 (original), 32-34 (patched)
> > <https://reviews.apache.org/r/60942/diff/2/?file=1778637#file1778637line33>
> >
> > Isn't that the table we have keep around? Or is it safe because we assume users will update version by version and the table will always exist before they upgrade to 0.19 with this patch?
Yes. The thing is that the migration script will not work anymore. At some point it backfills the new fields using the old ones, and as the old ones are no longer in the schema, it wouldn't compile. So without the migration script working, the whole migration kind of loses its meaning.
In order for things to work one should first update to any version in which migration V004 is applied (0.14.0), and then this will work.
> On Aug. 3, 2017, 9:50 p.m., Stephan Erb wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml
> > Lines 174-178 (original), 165-169 (patched)
> > <https://reviews.apache.org/r/60942/diff/2/?file=1778641#file1778641line174>
> >
> > The comment is outdated now.
I'll remove it
> On Aug. 3, 2017, 9:50 p.m., Stephan Erb wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml
> > Lines 228-237 (original), 216-226 (patched)
> > <https://reviews.apache.org/r/60942/diff/2/?file=1778641#file1778641line231>
> >
> > This is why we should never wonder why the DBTaskStore is so slow :-D
> >
> > (this has nothing to do with the patch)
:p
- Nicolás
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60942/#review182164
-----------------------------------------------------------
On July 18, 2017, 2:43 p.m., Nicolás Donatucci wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60942/
> -----------------------------------------------------------
>
> (Updated July 18, 2017, 2:43 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Zameer Manji.
>
>
> Bugs: AURORA-1707
> https://issues.apache.org/jira/browse/AURORA-1707
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Removed task level resource fields from the DB and the thrift-API.
> To do this, a new DB migration was added. When upgrading, it just drops the task level resource fields. When downgrading, it creates the fields again and populates them with information from the task_resource table.
>
> IMPORTANT: One of the python client tests is failing (test_cron_diff). This is not serious, I think it is a problem with the ordering of the elements of Resources (had similar problems with other python client tests that instead of comparing the Resources as a set, compared them as lists and thus order mattered). Nevertheless, I could not fully understand the code of that test. I was hoping someone could give me a hand with that.
> But then again, it is a smaller issue and so the patch can start being reviewed.
>
> Issue Related: AURORA-1707
>
>
> Diffs
> -----
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift 3749531b5412d7ca217736aa85eed8e6606225ad
> src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 186fa1b3a4780c0536fb486d50a33133258110cd
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java d2eb6aa6e4a155b2d28debab2ca10dfc76d57213
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java cda55c55680a19ed421299a8949299b21949787b
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java af106a8a9ee8c14122e98bcc0ec44b616f21d61f
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V011_DropResourceFields.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java 138cd5316adc73eed24fc7accc53885dd5d5bee5
> src/main/python/apache/aurora/client/cli/diff_formatter.py 78717774aa3fbaf83a5fb850bc9f9f4a4038d70f
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 5422183e4a1fe122fc0e1aa871aa75ae102e322d
> src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 7a86f47af67adb3e488381d30ddf424549deefbc
> src/test/java/org/apache/aurora/scheduler/http/TestUtils.java 689482c9f6c49bcca781834566edeb975d2f3af2
> src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java caaba9b6dff46ff0b037759f1c817a321ae15ee4
> src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 43e32eede27bbf26363a3fd1ca34ffe6f8c01a73
> src/test/java/org/apache/aurora/scheduler/updater/UpdateAgentReserverImplTest.java 1bc2a778ad3f1543a055023f9ec3fe9e4a9523e3
> src/test/python/apache/aurora/client/cli/test_status.py b0b7f96d8148f0dd1f6f45d1c5c6f830cabcfd5d
> src/test/python/apache/aurora/client/cli/util.py 43db828ca1cccd91c88f016b4c994fef33182fbf
> src/test/python/apache/aurora/config/test_resources.py 25a20f0b702189744a26b85053db34c37ff5b03c
>
>
> Diff: https://reviews.apache.org/r/60942/diff/2/
>
>
> Testing
> -------
>
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
>
> 1 - Created a new VM with a fresh Aurora 0.18.0, ran the tests so that the db would have something. Then upgraded to patched version and ran the tests again. Everything worked and task level resource fields were no longer there.
> 2 - Created a new VM with a fresh patched version, ran the tests so that the db would have something. Then downgraded to 0.18.0 and ran the tests again. Everything worked and task level resources were there, with the correct backfilled values.
>
>
> Thanks,
>
> Nicolás Donatucci
>
>