You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by "Erb, Stephan" <St...@blue-yonder.com> on 2015/09/02 22:06:25 UTC
Meaning and usage of job environments
Hi everyone,
I have been wondering about the environment part of Aurora jobkeys (devel, test, staging, staging1, ...prod).
I guess you made the environment a first class citizen in order to enforce some kind of standardization. How is this working out for you in practice?
IIRC, the current set of supported environment names is enforced by the client. In general, would you be opposed to moving this validation to the scheduler, so that it can be configured cluster wide? This would enable us to introduce environment names which are closer to our domain and organization. Or is this even somewhat covered in the upcoming job tier implementation?
Thanks for your input,
Stephan
Re: Meaning and usage of job environments
Posted by Maxim Khutornenko <ma...@apache.org>.
I agree that env story is not well defined now. The good news is that
we have to improve it now that we introduced task tier concept. This
problem is listed as an open question in oversubscription design doc
[1] and I expect another round of discussions before we define the
right way forward. If you feel you can drive this brainstorming that
would be great!
[1] - https://docs.google.com/document/d/1r1WCHgmPJp5wbrqSZLsgtxPNj3sULfHrSFmxp2GyPTo
On Wed, Sep 2, 2015 at 2:42 PM, Zameer Manji <zm...@apache.org> wrote:
> +1 to moving the validation to the scheduler. I think a reasonable step
> forward would be to set the defaults on the scheduler to be the same as
> they are in the client and remove the code in the client.
>
> On Wed, Sep 2, 2015 at 2:07 PM, Bill Farner <te...@gmail.com> wrote:
>
>> Environments were introduced as a first-class concept for namespacing. In
>> theory, it's useful so you can have something like a 'test' namespace that
>> contains all the same jobs as the 'prod' namespace. Prior to this, users
>> were namespacing within the job name, which was not so nice. There have
>> also been discussions around implied policy with environment names, but
>> that has not materialized.
>>
>> In general, would you be opposed to moving this validation to the
>> > scheduler, so that it can be configured cluster wide?
>> >
>>
>> I've wanted to see this happen, an would be in favor.
>>
>> This would enable us to introduce environment names which are closer to our
>> > domain and organization. Or is this even somewhat covered in the
>> upcoming
>> > job tier implementation?
>> >
>>
>> It's not covered by tiers. Tiers are loosely related to environments, but
>> the current plan is to keep them decoupled. I would encourage you to
>> explore the scheduler-side enforcement.
>>
>>
>> On Wed, Sep 2, 2015 at 1:06 PM, Erb, Stephan <St...@blue-yonder.com>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > I have been wondering about the environment part of Aurora jobkeys
>> (devel,
>> > test, staging, staging1, ...prod).
>> >
>> > I guess you made the environment a first class citizen in order to
>> enforce
>> > some kind of standardization. How is this working out for you in
>> practice?
>> >
>> > IIRC, the current set of supported environment names is enforced by the
>> > client. In general, would you be opposed to moving this validation to the
>> > scheduler, so that it can be configured cluster wide? This would enable
>> us
>> > to introduce environment names which are closer to our domain and
>> > organization. Or is this even somewhat covered in the upcoming job tier
>> > implementation?
>> >
>> > Thanks for your input,
>> > Stephan
>>
>
>
>
> --
> Zameer Manji
Re: Meaning and usage of job environments
Posted by Zameer Manji <zm...@apache.org>.
+1 to moving the validation to the scheduler. I think a reasonable step
forward would be to set the defaults on the scheduler to be the same as
they are in the client and remove the code in the client.
On Wed, Sep 2, 2015 at 2:07 PM, Bill Farner <te...@gmail.com> wrote:
> Environments were introduced as a first-class concept for namespacing. In
> theory, it's useful so you can have something like a 'test' namespace that
> contains all the same jobs as the 'prod' namespace. Prior to this, users
> were namespacing within the job name, which was not so nice. There have
> also been discussions around implied policy with environment names, but
> that has not materialized.
>
> In general, would you be opposed to moving this validation to the
> > scheduler, so that it can be configured cluster wide?
> >
>
> I've wanted to see this happen, an would be in favor.
>
> This would enable us to introduce environment names which are closer to our
> > domain and organization. Or is this even somewhat covered in the
> upcoming
> > job tier implementation?
> >
>
> It's not covered by tiers. Tiers are loosely related to environments, but
> the current plan is to keep them decoupled. I would encourage you to
> explore the scheduler-side enforcement.
>
>
> On Wed, Sep 2, 2015 at 1:06 PM, Erb, Stephan <St...@blue-yonder.com>
> wrote:
>
> > Hi everyone,
> >
> > I have been wondering about the environment part of Aurora jobkeys
> (devel,
> > test, staging, staging1, ...prod).
> >
> > I guess you made the environment a first class citizen in order to
> enforce
> > some kind of standardization. How is this working out for you in
> practice?
> >
> > IIRC, the current set of supported environment names is enforced by the
> > client. In general, would you be opposed to moving this validation to the
> > scheduler, so that it can be configured cluster wide? This would enable
> us
> > to introduce environment names which are closer to our domain and
> > organization. Or is this even somewhat covered in the upcoming job tier
> > implementation?
> >
> > Thanks for your input,
> > Stephan
>
--
Zameer Manji
Re: Meaning and usage of job environments
Posted by Bill Farner <te...@gmail.com>.
Environments were introduced as a first-class concept for namespacing. In
theory, it's useful so you can have something like a 'test' namespace that
contains all the same jobs as the 'prod' namespace. Prior to this, users
were namespacing within the job name, which was not so nice. There have
also been discussions around implied policy with environment names, but
that has not materialized.
In general, would you be opposed to moving this validation to the
> scheduler, so that it can be configured cluster wide?
>
I've wanted to see this happen, an would be in favor.
This would enable us to introduce environment names which are closer to our
> domain and organization. Or is this even somewhat covered in the upcoming
> job tier implementation?
>
It's not covered by tiers. Tiers are loosely related to environments, but
the current plan is to keep them decoupled. I would encourage you to
explore the scheduler-side enforcement.
On Wed, Sep 2, 2015 at 1:06 PM, Erb, Stephan <St...@blue-yonder.com>
wrote:
> Hi everyone,
>
> I have been wondering about the environment part of Aurora jobkeys (devel,
> test, staging, staging1, ...prod).
>
> I guess you made the environment a first class citizen in order to enforce
> some kind of standardization. How is this working out for you in practice?
>
> IIRC, the current set of supported environment names is enforced by the
> client. In general, would you be opposed to moving this validation to the
> scheduler, so that it can be configured cluster wide? This would enable us
> to introduce environment names which are closer to our domain and
> organization. Or is this even somewhat covered in the upcoming job tier
> implementation?
>
> Thanks for your input,
> Stephan