You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@aurora.apache.org by "Joe Smith (JIRA)" <ji...@apache.org> on 2015/04/07 20:47:13 UTC
[jira] [Created] (AURORA-1258) Improve procedure for adding
instances to a job
Joe Smith created AURORA-1258:
---------------------------------
Summary: Improve procedure for adding instances to a job
Key: AURORA-1258
URL: https://issues.apache.org/jira/browse/AURORA-1258
Project: Aurora
Issue Type: Story
Components: Reliability, Usability
Reporter: Joe Smith
The current process for adding instances to a job is highly manual, and potentially dangerous.
1. Take a config for a job with 10 instances, update it to 20 instances.
2. The batch size will be increased, and users will need to specify shards 10 to 19.
3. After this update is complete, users will need to manually update shards 0-9 again.
There may be other changes pulled in as part of this update other than just increasing the number of instances, which could further complicate things.
One possible improvement would be to change the updater from 'under-provision' where it kills instances first, then schedules new instances, to an 'over-provision' where it adds on new instances, then backpedals and kills the old instances.
Overall, a single command or process for a user to take an already-existing job and increase the number of instances would reduce overhead and fat-fingering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jira] [Created] (AURORA-1258) Improve procedure for adding
instances to a job
Posted by Josh Adams <jo...@gmail.com>.
+1
On Tue, Apr 7, 2015 at 12:59 PM, Suman Karumuri <
skarumuri@twitter.com.invalid> wrote:
> +1 for scale command.
>
> On Tue, Apr 7, 2015 at 12:45 PM, Kevin Sweeney
> <ksweeney@twitter.com.invalid
> > wrote:
>
> > How about a scale command somewhere that will check that the only
> operation
> > it's doing is "scaling" and not introducing new code. This could be
> > accomplished by diffing the config for the new instances with that of the
> > old instances. For cases where it's not a pure scale operation, maybe
> have
> > an override flag like --allow-heterogenous. Also, the existing updater
> > should handle this case fine if the config hasn't changed - it won't
> > attempt to modify unchanged instances. Maybe we need to update diff to
> give
> > an easy way to ensure that a proposed change is a pure scale operation
> and
> > not a combination of scaling and a code deploy.
> >
> > On Tuesday, April 7, 2015, Joe Smith (JIRA) <ji...@apache.org> wrote:
> >
> > > Joe Smith created AURORA-1258:
> > > ---------------------------------
> > >
> > > Summary: Improve procedure for adding instances to a job
> > > Key: AURORA-1258
> > > URL:
> https://issues.apache.org/jira/browse/AURORA-1258
> > > Project: Aurora
> > > Issue Type: Story
> > > Components: Reliability, Usability
> > > Reporter: Joe Smith
> > >
> > >
> > > The current process for adding instances to a job is highly manual, and
> > > potentially dangerous.
> > >
> > > 1. Take a config for a job with 10 instances, update it to 20
> instances.
> > > 2. The batch size will be increased, and users will need to specify
> > shards
> > > 10 to 19.
> > > 3. After this update is complete, users will need to manually update
> > > shards 0-9 again.
> > >
> > > There may be other changes pulled in as part of this update other than
> > > just increasing the number of instances, which could further complicate
> > > things.
> > >
> > > One possible improvement would be to change the updater from
> > > 'under-provision' where it kills instances first, then schedules new
> > > instances, to an 'over-provision' where it adds on new instances, then
> > > backpedals and kills the old instances.
> > >
> > > Overall, a single command or process for a user to take an
> > > already-existing job and increase the number of instances would reduce
> > > overhead and fat-fingering.
> > >
> > >
> > >
> > > --
> > > This message was sent by Atlassian JIRA
> > > (v6.3.4#6332)
> > >
> >
> >
> > --
> > Sent from Gmail Mobile
> >
>
Re: [jira] [Created] (AURORA-1258) Improve procedure for adding instances to a job
Posted by Joseph Smith <ya...@gmail.com>.
Kevin, I think your line of thought is on the money, and sounds like something which would be pretty useful :)
Thanks!
Joe
> On Apr 7, 2015, at 1:06 PM, Erb, Stephan <St...@blue-yonder.com> wrote:
>
> +1 as well.
>
> We are implementing an external auto-scaler for certain types of jobs which would greatly benefit from such an command. The scaling can be simulated using the updater, but the latter always needs to know about the job configuration which makes the entire implementation more complicated and coupled than it should be.
>
> Best Regards,
> Stephan
>
> ________________________________________
> From: Suman Karumuri <sk...@twitter.com.INVALID>
> Sent: Tuesday, April 7, 2015 9:59 PM
> To: dev@aurora.apache.org
> Subject: Re: [jira] [Created] (AURORA-1258) Improve procedure for adding instances to a job
>
> +1 for scale command.
>
> On Tue, Apr 7, 2015 at 12:45 PM, Kevin Sweeney <ksweeney@twitter.com.invalid
>> wrote:
>
>> How about a scale command somewhere that will check that the only operation
>> it's doing is "scaling" and not introducing new code. This could be
>> accomplished by diffing the config for the new instances with that of the
>> old instances. For cases where it's not a pure scale operation, maybe have
>> an override flag like --allow-heterogenous. Also, the existing updater
>> should handle this case fine if the config hasn't changed - it won't
>> attempt to modify unchanged instances. Maybe we need to update diff to give
>> an easy way to ensure that a proposed change is a pure scale operation and
>> not a combination of scaling and a code deploy.
>>
>> On Tuesday, April 7, 2015, Joe Smith (JIRA) <ji...@apache.org> wrote:
>>
>>> Joe Smith created AURORA-1258:
>>> ---------------------------------
>>>
>>> Summary: Improve procedure for adding instances to a job
>>> Key: AURORA-1258
>>> URL: https://issues.apache.org/jira/browse/AURORA-1258
>>> Project: Aurora
>>> Issue Type: Story
>>> Components: Reliability, Usability
>>> Reporter: Joe Smith
>>>
>>>
>>> The current process for adding instances to a job is highly manual, and
>>> potentially dangerous.
>>>
>>> 1. Take a config for a job with 10 instances, update it to 20 instances.
>>> 2. The batch size will be increased, and users will need to specify
>> shards
>>> 10 to 19.
>>> 3. After this update is complete, users will need to manually update
>>> shards 0-9 again.
>>>
>>> There may be other changes pulled in as part of this update other than
>>> just increasing the number of instances, which could further complicate
>>> things.
>>>
>>> One possible improvement would be to change the updater from
>>> 'under-provision' where it kills instances first, then schedules new
>>> instances, to an 'over-provision' where it adds on new instances, then
>>> backpedals and kills the old instances.
>>>
>>> Overall, a single command or process for a user to take an
>>> already-existing job and increase the number of instances would reduce
>>> overhead and fat-fingering.
>>>
>>>
>>>
>>> --
>>> This message was sent by Atlassian JIRA
>>> (v6.3.4#6332)
>>>
>>
>>
>> --
>> Sent from Gmail Mobile
>>
Re: [jira] [Created] (AURORA-1258) Improve procedure for adding
instances to a job
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
+1 as well.
We are implementing an external auto-scaler for certain types of jobs which would greatly benefit from such an command. The scaling can be simulated using the updater, but the latter always needs to know about the job configuration which makes the entire implementation more complicated and coupled than it should be.
Best Regards,
Stephan
________________________________________
From: Suman Karumuri <sk...@twitter.com.INVALID>
Sent: Tuesday, April 7, 2015 9:59 PM
To: dev@aurora.apache.org
Subject: Re: [jira] [Created] (AURORA-1258) Improve procedure for adding instances to a job
+1 for scale command.
On Tue, Apr 7, 2015 at 12:45 PM, Kevin Sweeney <ksweeney@twitter.com.invalid
> wrote:
> How about a scale command somewhere that will check that the only operation
> it's doing is "scaling" and not introducing new code. This could be
> accomplished by diffing the config for the new instances with that of the
> old instances. For cases where it's not a pure scale operation, maybe have
> an override flag like --allow-heterogenous. Also, the existing updater
> should handle this case fine if the config hasn't changed - it won't
> attempt to modify unchanged instances. Maybe we need to update diff to give
> an easy way to ensure that a proposed change is a pure scale operation and
> not a combination of scaling and a code deploy.
>
> On Tuesday, April 7, 2015, Joe Smith (JIRA) <ji...@apache.org> wrote:
>
> > Joe Smith created AURORA-1258:
> > ---------------------------------
> >
> > Summary: Improve procedure for adding instances to a job
> > Key: AURORA-1258
> > URL: https://issues.apache.org/jira/browse/AURORA-1258
> > Project: Aurora
> > Issue Type: Story
> > Components: Reliability, Usability
> > Reporter: Joe Smith
> >
> >
> > The current process for adding instances to a job is highly manual, and
> > potentially dangerous.
> >
> > 1. Take a config for a job with 10 instances, update it to 20 instances.
> > 2. The batch size will be increased, and users will need to specify
> shards
> > 10 to 19.
> > 3. After this update is complete, users will need to manually update
> > shards 0-9 again.
> >
> > There may be other changes pulled in as part of this update other than
> > just increasing the number of instances, which could further complicate
> > things.
> >
> > One possible improvement would be to change the updater from
> > 'under-provision' where it kills instances first, then schedules new
> > instances, to an 'over-provision' where it adds on new instances, then
> > backpedals and kills the old instances.
> >
> > Overall, a single command or process for a user to take an
> > already-existing job and increase the number of instances would reduce
> > overhead and fat-fingering.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>
>
> --
> Sent from Gmail Mobile
>
Re: [jira] [Created] (AURORA-1258) Improve procedure for adding
instances to a job
Posted by Suman Karumuri <sk...@twitter.com.INVALID>.
+1 for scale command.
On Tue, Apr 7, 2015 at 12:45 PM, Kevin Sweeney <ksweeney@twitter.com.invalid
> wrote:
> How about a scale command somewhere that will check that the only operation
> it's doing is "scaling" and not introducing new code. This could be
> accomplished by diffing the config for the new instances with that of the
> old instances. For cases where it's not a pure scale operation, maybe have
> an override flag like --allow-heterogenous. Also, the existing updater
> should handle this case fine if the config hasn't changed - it won't
> attempt to modify unchanged instances. Maybe we need to update diff to give
> an easy way to ensure that a proposed change is a pure scale operation and
> not a combination of scaling and a code deploy.
>
> On Tuesday, April 7, 2015, Joe Smith (JIRA) <ji...@apache.org> wrote:
>
> > Joe Smith created AURORA-1258:
> > ---------------------------------
> >
> > Summary: Improve procedure for adding instances to a job
> > Key: AURORA-1258
> > URL: https://issues.apache.org/jira/browse/AURORA-1258
> > Project: Aurora
> > Issue Type: Story
> > Components: Reliability, Usability
> > Reporter: Joe Smith
> >
> >
> > The current process for adding instances to a job is highly manual, and
> > potentially dangerous.
> >
> > 1. Take a config for a job with 10 instances, update it to 20 instances.
> > 2. The batch size will be increased, and users will need to specify
> shards
> > 10 to 19.
> > 3. After this update is complete, users will need to manually update
> > shards 0-9 again.
> >
> > There may be other changes pulled in as part of this update other than
> > just increasing the number of instances, which could further complicate
> > things.
> >
> > One possible improvement would be to change the updater from
> > 'under-provision' where it kills instances first, then schedules new
> > instances, to an 'over-provision' where it adds on new instances, then
> > backpedals and kills the old instances.
> >
> > Overall, a single command or process for a user to take an
> > already-existing job and increase the number of instances would reduce
> > overhead and fat-fingering.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>
>
> --
> Sent from Gmail Mobile
>
Re: [jira] [Created] (AURORA-1258) Improve procedure for adding
instances to a job
Posted by Brian Brazil <br...@boxever.com>.
On 7 April 2015 at 20:45, Kevin Sweeney <ks...@twitter.com.invalid>
wrote:
> How about a scale command somewhere that will check that the only operation
> it's doing is "scaling" and not introducing new code. This could be
> accomplished by diffing the config for the new instances with that of the
> old instances. For cases where it's not a pure scale operation, maybe have
> an override flag like --allow-heterogenous. Also, the existing updater
> should handle this case fine if the config hasn't changed - it won't
> attempt to modify unchanged instances. Maybe we need to update diff to give
> an easy way to ensure that a proposed change is a pure scale operation and
> not a combination of scaling and a code deploy.
>
There are other types of change that it'd be good to do without changing
code/flags, such as giving a job more resources if it's OOMing.
Some size changes may also require changing flags (e.g. if there's sharding
of some form and each task needs to know the size of the job), so I think
Joe's idea of adding new instances first when the size of the job in
increasing is good as it makes it harder to accidentally take out the
entire job when upsizing.
Brian
>
> On Tuesday, April 7, 2015, Joe Smith (JIRA) <ji...@apache.org> wrote:
>
> > Joe Smith created AURORA-1258:
> > ---------------------------------
> >
> > Summary: Improve procedure for adding instances to a job
> > Key: AURORA-1258
> > URL: https://issues.apache.org/jira/browse/AURORA-1258
> > Project: Aurora
> > Issue Type: Story
> > Components: Reliability, Usability
> > Reporter: Joe Smith
> >
> >
> > The current process for adding instances to a job is highly manual, and
> > potentially dangerous.
> >
> > 1. Take a config for a job with 10 instances, update it to 20 instances.
> > 2. The batch size will be increased, and users will need to specify
> shards
> > 10 to 19.
> > 3. After this update is complete, users will need to manually update
> > shards 0-9 again.
> >
> > There may be other changes pulled in as part of this update other than
> > just increasing the number of instances, which could further complicate
> > things.
> >
> > One possible improvement would be to change the updater from
> > 'under-provision' where it kills instances first, then schedules new
> > instances, to an 'over-provision' where it adds on new instances, then
> > backpedals and kills the old instances.
> >
> > Overall, a single command or process for a user to take an
> > already-existing job and increase the number of instances would reduce
> > overhead and fat-fingering.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>
>
> --
> Sent from Gmail Mobile
>
Re: [jira] [Created] (AURORA-1258) Improve procedure for adding
instances to a job
Posted by Kevin Sweeney <ks...@twitter.com.INVALID>.
How about a scale command somewhere that will check that the only operation
it's doing is "scaling" and not introducing new code. This could be
accomplished by diffing the config for the new instances with that of the
old instances. For cases where it's not a pure scale operation, maybe have
an override flag like --allow-heterogenous. Also, the existing updater
should handle this case fine if the config hasn't changed - it won't
attempt to modify unchanged instances. Maybe we need to update diff to give
an easy way to ensure that a proposed change is a pure scale operation and
not a combination of scaling and a code deploy.
On Tuesday, April 7, 2015, Joe Smith (JIRA) <ji...@apache.org> wrote:
> Joe Smith created AURORA-1258:
> ---------------------------------
>
> Summary: Improve procedure for adding instances to a job
> Key: AURORA-1258
> URL: https://issues.apache.org/jira/browse/AURORA-1258
> Project: Aurora
> Issue Type: Story
> Components: Reliability, Usability
> Reporter: Joe Smith
>
>
> The current process for adding instances to a job is highly manual, and
> potentially dangerous.
>
> 1. Take a config for a job with 10 instances, update it to 20 instances.
> 2. The batch size will be increased, and users will need to specify shards
> 10 to 19.
> 3. After this update is complete, users will need to manually update
> shards 0-9 again.
>
> There may be other changes pulled in as part of this update other than
> just increasing the number of instances, which could further complicate
> things.
>
> One possible improvement would be to change the updater from
> 'under-provision' where it kills instances first, then schedules new
> instances, to an 'over-provision' where it adds on new instances, then
> backpedals and kills the old instances.
>
> Overall, a single command or process for a user to take an
> already-existing job and increase the number of instances would reduce
> overhead and fat-fingering.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
--
Sent from Gmail Mobile