You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by Mark Chu-Carroll <mc...@apache.org> on 2014/05/22 20:48:37 UTC
Updated proposal: Aurora Shorthands
The bulk of the feedback from the first round of comments on this could be
summed up, roughly "get rid of the stupid command shortcuts you dope,
they're a menace!". Message received: they're gone.
But I still think that the configurable defaults are valuable. Based on
some feedback, I've revised the proposal a bit. This includes trying to
strengthen the motivation/problem statement, and adding ways to provide
default parameters only for specific commands, plus many small changes.
The updated proposal is attached below.
# Defaults and Shorthands in the Aurora Client
## Motivation
Most of the time, users are doing the same thing, over and over again.
They're working mainly on one particular service, in one particular
workspace. But they need to repeat the same parameters, over and over
again, and they need to remember what those parameters are, what order
they occur in, what format they use, and other details that can be
difficult to remember.
In order to avoid these problems, users set up custom scripts
that make it easy for them to run commands like "`myservice start`",
instead of "`aurora job create west/markcc/prod/myserver
src/main/aurora/myservice/myservice.aurora`"
To reduce this, we'd like to support a way for users to set up a
configuration file that defines defaults and shorthands for their
everyday work. With shorthands, a user that only works with a single
service could say "aurora job create", instead of needing to spell
out the full jobkey and configuration file location; a user working with
multiple datacenters could say "`aurora job create @east`" or
"`aurora job create @west`" to select the correct jobkey.
## Proposal: Aurora Init Files
To allow users to customize shorthands, we'll provide a
builtin capability to allow users to provide a configuration
file, from which their customizations will be loaded.
Many applications use a simple pattern to solve similar problems.
Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
don't specifically tell the tool where to find a configuration, it looks
in the current directory or one of its parents to find a file named
"Vagrantfile".
We'd like to follow a similar pattern, and create an "AuroraInit" file.
The aurora init file is found by searching the following locations, in
order:
* the contents of the "--init-file" parameter.
* if the "--init-file" parameter is unspecified, then look in the current
directory for a file named "AuroraInit".
* if no "AuroraInit" file exists in the current directory, then look in
the users
home directory for an init file named ".AuroraInit".
### What goes into an init file?
We should support the following kinds of things:
1. Universal defaults - user-defined default settings that will be
applied to
all commands. For example, if there is a default config file that
should always
be used if the user doesn't specify one, that would be a universal
default.
2. Command specific defaults - users should be able to specify that they
always want
to use certain parameter settings for a specific command. For example,
if they
want to always use a default batch size of 10 for updates, but don't
want to affect
batch sizes for other commands like kill, they could use a command
specific default.
3. Aliases - shorthand names for longer parameters. A user could
specify shorthands
"east" and "west" for full jobkeys in two different datacenters.
### Using aliases and defaults
A default specifies a set of _bindings_. If a parameter is omitted
from a command, and there's a binding for that parameter, it will be
automatically inserted into the command as if the user had typed it.
An alias is a short equivalent for a parameter. When a command line is
provided by a user, aliases will be expanded inline. A user can
specifically
mark an alias for expansion by prefixing it with "@"; if an alias
appears on the command-line surrounded by whitespace, it will be
replaced even if it isn't marked with an "@".
### Defining Shorthands and Defaults
The init file is, like aurora job configurations, a python source file
using a Pystachio
schema. The schema is loaded, and an `Init` object is retrieved from the
top-level
variable `init` in that file.
The pystachio schema for init files is:
class CommandDefaults(Struct)
command = Required(String)
defaults=Map(String, String)
class Init(Struct):
defaults=Map(String, String)
command_defaults = Optional(CommandDefaults)
aliases=Map(String, String)
For example, an init file could contain the following:
init=Init(
defaults={
"jobspec": "west/markcc/test/myservice",
"config_file": "./src/aurora/myservice.aurora"
},
aliases={
"east": "east/markcc/test/myservice",
"c": "east",
"me": "markchucarroll"
},
command_defaults(command="job update",
defaults={"--batch-size": 10}
))
With this configuration file, if the user ran "`aurora job create`" without
any parameters,
it would automatically be expanded to "`aurora job create
west/markcc/test/myservice ./src/aurora/myservice.aurora`".
If a user ran "`aurora job create east`", the alias `east` would be
expanded to "east/markcc/test/myservice", and the missing config file
parameter would
be instantiated using the default, to create a command-line:
"`aurora job create east/markcc/test/myservice
./src/aurora/myservice.aurora`".
If a user ran "`aurora job create @c/@me/test/myservice`", the two aliases
would
be expanded, and the omitted configuration parameter would be added:
"`aurora job create east/markchucarroll/test/myservice
./src/aurora/myservice.aurora`".
If a user ran "`aurora job update`", then the `jobspec` and `config_file`
parameters would
get inserted from the global defaults, and the "--batch-size=10" would be
inserted from
the command defaults for "aurora job update".
Re: Updated proposal: Aurora Shorthands
Posted by Kevin Sweeney <ke...@apache.org>.
On Thu, May 29, 2014 at 1:23 PM, Mark Chu-Carroll <mc...@apache.org>
wrote:
> The reason that I use an alias for rbt isn't because I'm trying to save
> keystrokes. It's because "-o", "-g" and "-r" mean nothing to me. More
> keystrokes is fine, if it comes with clarity - that's why I went along with
> dropping the automatic shorthands.
>
Meta-point: rbt does have long options - I just find it excessively verbose
to type --open --guess-fields and --review-request-id.
>
> The stuff that you can do with Brian's aliases proposal does work equally
> well in mine; any alias is automatically substituted, with or without "@",
> if it's used in bare form (that is, not as a part of another string in the
> command line.) Substituting inline, though, I still believe is useful. If
> we implement it, and find that no one uses it, I'll happily remove it.
>
> -Mark
>
>
> On Thu, May 29, 2014 at 4:10 PM, David McLaughlin <da...@dmclaughlin.com>
> wrote:
>
> > I see the point about having global client defaults, but I still think we
> > should avoid inferring the jobkey from the home directory. I'd suggest
> only
> > inferring command_defaults, or even storing them separately - but this
> also
> > adds complexity. Interested in what other people think here.
> >
> > With regards to symbols and the general theme of keystrokes, specifically
> > this point:
> >
> > "I also think that you're over-estimating the degree of memory required
> to
> > > use it successfully; I've certainly seen
> > > many cases of people using stuff like that."
> >
> >
> > Look at rbt, which you mentioned you wrote a wrapper around.
> >
> > This posts a new review:
> >
> > ./rbt post -o -g
> >
> > This updates an existing review:
> >
> > ./rbt post -o -r <id>
> >
> > This has very few key strokes. It has only one parameter change between
> the
> > two commands that you need to remember. It is very simple when you look
> at
> > it in an isolated fashion. Yet people will still write wrappers around it
> > (or just hope it's in my bash history like I do).
> >
> > In this case I can't imagine constructing partial job keys from variables
> > would save you anything. I think if I'm running the commands often enough
> > that I'd remember the aliases, I'd either most likely be able to remember
> > my cluster and role names too - or more realistically, I'd just add the
> > full jobkey path to my AuroraInit file and be done with it.
> >
> > I'm only just catching on to the fact that you can't define more than one
> > job key in the AuroraInit file and aliases might be the way around that.
> To
> > solve that, I actually liked Brian's suggestion in the other thread:
> >
> > > Perhaps a stop-gap would be to require explicit aliases, e.g.
> > >
> > > [aliases]
> > > test-west = wickman/west/test/service
> > > test-east = wickman/east/test/service
> > > prod-west = wickman/west/test/service
> > > prod-east = wickman/east/test/service
> > >
> >
> > Maybe we could ask users to define pystachio structs with all their
> > options, then have a profile (or target) option like:
> >
> > ./aurora job update --profile=prod
> > ./aurora job kill --profile=prod
> >
> > etc.
> >
> >
> > On Thu, May 29, 2014 at 10:24 AM, Mark Chu-Carroll <
> mchucarroll@apache.org
> > >
> > wrote:
> >
> > > WRT to the home directory fallback: I'm not hugely committed to it, but
> > > I've discussed this with a couple of people, and all have liked it, for
> > one
> > > big reason: they're using git in the twitter codebase - so the working
> > > directory that they use contains a ton of different projects. They
> can't
> > > just put an AuroraInit file in the root directory of the workspace,
> > because
> > > it won't work. So they want to be able to write one, put it in git in
> > some
> > > directory, and then put a link to it into their homedir. Then they have
> > > defaults in place.
> > >
> > > WRT to separate init file: I think it's very important that the two be
> > > different. The config file is a shared entity used by multiple people,
> > and
> > > possibly used in different ways by different people. For example, one
> > > member of a team may typically only work with one of a team's services,
> > > while all are defined in the same config file. The user would to have
> > > defaults to work with the one service that they actually work on - but
> > > making it a default for all users of the config file would not work.
> > >
> > > WRT @symbols: yes, we could do substitution via the shell or something
> > > similar. But I think that the argument you made above about putting
> > things
> > > into the config file applies here: having all of the shorthands and
> > > defaults that the user works with be stored in one place is better that
> > > scattering among multiple files. I also think that you're
> > over-estimating
> > > the degree of memory required to use it successfully; I've certainly
> seen
> > > many cases of people using stuff like that. (Think about how people use
> > > git; they know the names of their branches, and they know the order of
> > > things for the commands they use most. For normal operation, that's
> fine.
> > > Back in my google days, we had a crazy git->perforce bridge, and it
> used
> > > aliases in exactly this way. It was hugely successful among engineers.
> I
> > > think that this is similar.)
> > >
> > > Finally, about the @ symbol: I'm not in love with it, but given my
> > argument
> > > above, if we want that functionality, we have to use something. My
> first
> > > choice would have been to use pystachio-style stashes - but the braces
> > are
> > > already used by bash; in fact, all of the bracket characters already
> are
> > > used. For single-character prefixes, there aren't many characters that
> > > aren't already used by the shell - we can't use !, #, $, ^, &, *, ~, or
> > ?.
> > > For easy to type stuff, that left very little other than @. And @ is
> used
> > > as a variable prefix in perl and ruby, so it's not without precedent or
> > > familiarity to many people.
> > >
> > > -Mark
> > >
> > >
> > > On Tue, May 27, 2014 at 7:19 PM, David McLaughlin <
> david@dmclaughlin.com
> > >
> > > wrote:
> > >
> > > > I am a huge +1 on the idea of storing my run configurations in config
> > > files
> > > > so that I don't need to type them out every time. I also really like
> > the
> > > > idea of command defaults.
> > > >
> > > > I do however have a couple of comments about the details of the
> > proposal.
> > > >
> > > > I'm a little concerned about the fallback to the user's home
> > directory. I
> > > > feel like these config files will be per-project and checked in next
> to
> > > the
> > > > aurora config file for all to share. Given that things like update
> > batch
> > > > sizes probably *should* be defined per-project and not per-user, I at
> > > least
> > > > think we should encourage this. It's also not uncommon to run
> commands
> > > in a
> > > > different cwd than you intended, and having such broad fallbacks
> seems
> > to
> > > > be just asking for trouble. I think it's perfectly fine to expect
> > > > AuroraInit to either be in the correct working directory or for the
> > user
> > > to
> > > > supply an explicit path to the file.
> > > >
> > > > I also think once you get to that stage, you can make the observation
> > > that
> > > > most production aurora configs will have an AuroraInit file next to
> > them.
> > > > So I wonder if we can just simplify here and let the user roll these
> > into
> > > > one file? It'd be nice to have a naming convention for aurora files
> so
> > > they
> > > > could be dropped from the parameters too.
> > > >
> > > >
> > > > The other concern I have is the syntax for injecting variables into
> > > > commands with the @symbol. It seems like there are already tools for
> > > > solving this - bash, python, etc. that should we just concede defeat
> > to.
> > > If
> > > > I'm typing my commands manually and relying on aliases I'd make a bet
> > > that
> > > > I'm probably not going to be remembering what I alias'd different
> parts
> > > of
> > > > my jobpath to and again, what the order of the jobpath bits are.
> > > >
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > >
> > > >
> > > > On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> > > > <mc...@apache.org>wrote:
> > > >
> > > > > The bulk of the feedback from the first round of comments on this
> > could
> > > > be
> > > > > summed up, roughly "get rid of the stupid command shortcuts you
> dope,
> > > > > they're a menace!". Message received: they're gone.
> > > > >
> > > > > But I still think that the configurable defaults are valuable.
> Based
> > on
> > > > > some feedback, I've revised the proposal a bit. This includes
> trying
> > to
> > > > > strengthen the motivation/problem statement, and adding ways to
> > provide
> > > > > default parameters only for specific commands, plus many small
> > changes.
> > > > >
> > > > > The updated proposal is attached below.
> > > > >
> > > > >
> > > > > # Defaults and Shorthands in the Aurora Client
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > Most of the time, users are doing the same thing, over and over
> > again.
> > > > > They're working mainly on one particular service, in one particular
> > > > > workspace. But they need to repeat the same parameters, over and
> over
> > > > > again, and they need to remember what those parameters are, what
> > order
> > > > > they occur in, what format they use, and other details that can be
> > > > > difficult to remember.
> > > > >
> > > > > In order to avoid these problems, users set up custom scripts
> > > > > that make it easy for them to run commands like "`myservice
> start`",
> > > > > instead of "`aurora job create west/markcc/prod/myserver
> > > > > src/main/aurora/myservice/myservice.aurora`"
> > > > >
> > > > > To reduce this, we'd like to support a way for users to set up a
> > > > > configuration file that defines defaults and shorthands for their
> > > > > everyday work. With shorthands, a user that only works with a
> single
> > > > > service could say "aurora job create", instead of needing to spell
> > > > > out the full jobkey and configuration file location; a user working
> > > with
> > > > > multiple datacenters could say "`aurora job create @east`" or
> > > > > "`aurora job create @west`" to select the correct jobkey.
> > > > >
> > > > > ## Proposal: Aurora Init Files
> > > > >
> > > > > To allow users to customize shorthands, we'll provide a
> > > > > builtin capability to allow users to provide a configuration
> > > > > file, from which their customizations will be loaded.
> > > > > Many applications use a simple pattern to solve similar problems.
> > > > > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if
> you
> > > > > don't specifically tell the tool where to find a configuration, it
> > > looks
> > > > > in the current directory or one of its parents to find a file named
> > > > > "Vagrantfile".
> > > > >
> > > > > We'd like to follow a similar pattern, and create an "AuroraInit"
> > file.
> > > > > The aurora init file is found by searching the following locations,
> > in
> > > > > order:
> > > > >
> > > > > * the contents of the "--init-file" parameter.
> > > > > * if the "--init-file" parameter is unspecified, then look in
> the
> > > > > current
> > > > > directory for a file named "AuroraInit".
> > > > > * if no "AuroraInit" file exists in the current directory, then
> > look
> > > > in
> > > > > the users
> > > > > home directory for an init file named ".AuroraInit".
> > > > >
> > > > >
> > > > > ### What goes into an init file?
> > > > >
> > > > > We should support the following kinds of things:
> > > > >
> > > > > 1. Universal defaults - user-defined default settings that will
> be
> > > > > applied to
> > > > > all commands. For example, if there is a default config file
> > that
> > > > > should always
> > > > > be used if the user doesn't specify one, that would be a
> > universal
> > > > > default.
> > > > > 2. Command specific defaults - users should be able to specify
> > that
> > > > they
> > > > > always want
> > > > > to use certain parameter settings for a specific command. For
> > > > example,
> > > > > if they
> > > > > want to always use a default batch size of 10 for updates, but
> > > don't
> > > > > want to affect
> > > > > batch sizes for other commands like kill, they could use a
> > command
> > > > > specific default.
> > > > > 3. Aliases - shorthand names for longer parameters. A user
> could
> > > > > specify shorthands
> > > > > "east" and "west" for full jobkeys in two different
> > datacenters.
> > > > >
> > > > > ### Using aliases and defaults
> > > > >
> > > > > A default specifies a set of _bindings_. If a parameter is omitted
> > > > > from a command, and there's a binding for that parameter, it will
> be
> > > > > automatically inserted into the command as if the user had typed
> it.
> > > > >
> > > > > An alias is a short equivalent for a parameter. When a command line
> > is
> > > > > provided by a user, aliases will be expanded inline. A user can
> > > > > specifically
> > > > > mark an alias for expansion by prefixing it with "@"; if an alias
> > > > > appears on the command-line surrounded by whitespace, it will be
> > > > > replaced even if it isn't marked with an "@".
> > > > >
> > > > >
> > > > > ### Defining Shorthands and Defaults
> > > > >
> > > > > The init file is, like aurora job configurations, a python source
> > file
> > > > > using a Pystachio
> > > > > schema. The schema is loaded, and an `Init` object is retrieved
> from
> > > the
> > > > > top-level
> > > > > variable `init` in that file.
> > > > >
> > > > > The pystachio schema for init files is:
> > > > >
> > > > > class CommandDefaults(Struct)
> > > > > command = Required(String)
> > > > > defaults=Map(String, String)
> > > > >
> > > > > class Init(Struct):
> > > > > defaults=Map(String, String)
> > > > > command_defaults = Optional(CommandDefaults)
> > > > > aliases=Map(String, String)
> > > > >
> > > > >
> > > > > For example, an init file could contain the following:
> > > > >
> > > > > init=Init(
> > > > > defaults={
> > > > > "jobspec": "west/markcc/test/myservice",
> > > > > "config_file": "./src/aurora/myservice.aurora"
> > > > > },
> > > > > aliases={
> > > > > "east": "east/markcc/test/myservice",
> > > > > "c": "east",
> > > > > "me": "markchucarroll"
> > > > > },
> > > > > command_defaults(command="job update",
> > > > > defaults={"--batch-size": 10}
> > > > > ))
> > > > >
> > > > >
> > > > > With this configuration file, if the user ran "`aurora job create`"
> > > > without
> > > > > any parameters,
> > > > > it would automatically be expanded to "`aurora job create
> > > > > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> > > > >
> > > > > If a user ran "`aurora job create east`", the alias `east` would be
> > > > > expanded to "east/markcc/test/myservice", and the missing config
> file
> > > > > parameter would
> > > > > be instantiated using the default, to create a command-line:
> > > > > "`aurora job create east/markcc/test/myservice
> > > > > ./src/aurora/myservice.aurora`".
> > > > >
> > > > > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> > > > aliases
> > > > > would
> > > > > be expanded, and the omitted configuration parameter would be
> added:
> > > > > "`aurora job create east/markchucarroll/test/myservice
> > > > > ./src/aurora/myservice.aurora`".
> > > > >
> > > > > If a user ran "`aurora job update`", then the `jobspec` and
> > > `config_file`
> > > > > parameters would
> > > > > get inserted from the global defaults, and the "--batch-size=10"
> > would
> > > be
> > > > > inserted from
> > > > > the command defaults for "aurora job update".
> > > > >
> > > >
> > >
> >
>
Re: Updated proposal: Aurora Shorthands
Posted by Kevin Sweeney <ks...@twitter.com>.
On Thu, May 29, 2014 at 1:23 PM, Mark Chu-Carroll <mc...@apache.org>
wrote:
> The reason that I use an alias for rbt isn't because I'm trying to save
> keystrokes. It's because "-o", "-g" and "-r" mean nothing to me. More
> keystrokes is fine, if it comes with clarity - that's why I went along with
> dropping the automatic shorthands.
>
> Meta-point: rbt does have long options - I just find it excessively
verbose to type --open --guess-fields and --review-request-id.
> The stuff that you can do with Brian's aliases proposal does work equally
> well in mine; any alias is automatically substituted, with or without "@",
> if it's used in bare form (that is, not as a part of another string in the
> command line.) Substituting inline, though, I still believe is useful. If
> we implement it, and find that no one uses it, I'll happily remove it.
>
> -Mark
>
>
> On Thu, May 29, 2014 at 4:10 PM, David McLaughlin <da...@dmclaughlin.com>
> wrote:
>
> > I see the point about having global client defaults, but I still think we
> > should avoid inferring the jobkey from the home directory. I'd suggest
> only
> > inferring command_defaults, or even storing them separately - but this
> also
> > adds complexity. Interested in what other people think here.
> >
> > With regards to symbols and the general theme of keystrokes, specifically
> > this point:
> >
> > "I also think that you're over-estimating the degree of memory required
> to
> > > use it successfully; I've certainly seen
> > > many cases of people using stuff like that."
> >
> >
> > Look at rbt, which you mentioned you wrote a wrapper around.
> >
> > This posts a new review:
> >
> > ./rbt post -o -g
> >
> > This updates an existing review:
> >
> > ./rbt post -o -r <id>
> >
> > This has very few key strokes. It has only one parameter change between
> the
> > two commands that you need to remember. It is very simple when you look
> at
> > it in an isolated fashion. Yet people will still write wrappers around it
> > (or just hope it's in my bash history like I do).
> >
> > In this case I can't imagine constructing partial job keys from variables
> > would save you anything. I think if I'm running the commands often enough
> > that I'd remember the aliases, I'd either most likely be able to remember
> > my cluster and role names too - or more realistically, I'd just add the
> > full jobkey path to my AuroraInit file and be done with it.
> >
> > I'm only just catching on to the fact that you can't define more than one
> > job key in the AuroraInit file and aliases might be the way around that.
> To
> > solve that, I actually liked Brian's suggestion in the other thread:
> >
> > > Perhaps a stop-gap would be to require explicit aliases, e.g.
> > >
> > > [aliases]
> > > test-west = wickman/west/test/service
> > > test-east = wickman/east/test/service
> > > prod-west = wickman/west/test/service
> > > prod-east = wickman/east/test/service
> > >
> >
> > Maybe we could ask users to define pystachio structs with all their
> > options, then have a profile (or target) option like:
> >
> > ./aurora job update --profile=prod
> > ./aurora job kill --profile=prod
> >
> > etc.
> >
> >
> > On Thu, May 29, 2014 at 10:24 AM, Mark Chu-Carroll <
> mchucarroll@apache.org
> > >
> > wrote:
> >
> > > WRT to the home directory fallback: I'm not hugely committed to it, but
> > > I've discussed this with a couple of people, and all have liked it, for
> > one
> > > big reason: they're using git in the twitter codebase - so the working
> > > directory that they use contains a ton of different projects. They
> can't
> > > just put an AuroraInit file in the root directory of the workspace,
> > because
> > > it won't work. So they want to be able to write one, put it in git in
> > some
> > > directory, and then put a link to it into their homedir. Then they have
> > > defaults in place.
> > >
> > > WRT to separate init file: I think it's very important that the two be
> > > different. The config file is a shared entity used by multiple people,
> > and
> > > possibly used in different ways by different people. For example, one
> > > member of a team may typically only work with one of a team's services,
> > > while all are defined in the same config file. The user would to have
> > > defaults to work with the one service that they actually work on - but
> > > making it a default for all users of the config file would not work.
> > >
> > > WRT @symbols: yes, we could do substitution via the shell or something
> > > similar. But I think that the argument you made above about putting
> > things
> > > into the config file applies here: having all of the shorthands and
> > > defaults that the user works with be stored in one place is better that
> > > scattering among multiple files. I also think that you're
> > over-estimating
> > > the degree of memory required to use it successfully; I've certainly
> seen
> > > many cases of people using stuff like that. (Think about how people use
> > > git; they know the names of their branches, and they know the order of
> > > things for the commands they use most. For normal operation, that's
> fine.
> > > Back in my google days, we had a crazy git->perforce bridge, and it
> used
> > > aliases in exactly this way. It was hugely successful among engineers.
> I
> > > think that this is similar.)
> > >
> > > Finally, about the @ symbol: I'm not in love with it, but given my
> > argument
> > > above, if we want that functionality, we have to use something. My
> first
> > > choice would have been to use pystachio-style stashes - but the braces
> > are
> > > already used by bash; in fact, all of the bracket characters already
> are
> > > used. For single-character prefixes, there aren't many characters that
> > > aren't already used by the shell - we can't use !, #, $, ^, &, *, ~, or
> > ?.
> > > For easy to type stuff, that left very little other than @. And @ is
> used
> > > as a variable prefix in perl and ruby, so it's not without precedent or
> > > familiarity to many people.
> > >
> > > -Mark
> > >
> > >
> > > On Tue, May 27, 2014 at 7:19 PM, David McLaughlin <
> david@dmclaughlin.com
> > >
> > > wrote:
> > >
> > > > I am a huge +1 on the idea of storing my run configurations in config
> > > files
> > > > so that I don't need to type them out every time. I also really like
> > the
> > > > idea of command defaults.
> > > >
> > > > I do however have a couple of comments about the details of the
> > proposal.
> > > >
> > > > I'm a little concerned about the fallback to the user's home
> > directory. I
> > > > feel like these config files will be per-project and checked in next
> to
> > > the
> > > > aurora config file for all to share. Given that things like update
> > batch
> > > > sizes probably *should* be defined per-project and not per-user, I at
> > > least
> > > > think we should encourage this. It's also not uncommon to run
> commands
> > > in a
> > > > different cwd than you intended, and having such broad fallbacks
> seems
> > to
> > > > be just asking for trouble. I think it's perfectly fine to expect
> > > > AuroraInit to either be in the correct working directory or for the
> > user
> > > to
> > > > supply an explicit path to the file.
> > > >
> > > > I also think once you get to that stage, you can make the observation
> > > that
> > > > most production aurora configs will have an AuroraInit file next to
> > them.
> > > > So I wonder if we can just simplify here and let the user roll these
> > into
> > > > one file? It'd be nice to have a naming convention for aurora files
> so
> > > they
> > > > could be dropped from the parameters too.
> > > >
> > > >
> > > > The other concern I have is the syntax for injecting variables into
> > > > commands with the @symbol. It seems like there are already tools for
> > > > solving this - bash, python, etc. that should we just concede defeat
> > to.
> > > If
> > > > I'm typing my commands manually and relying on aliases I'd make a bet
> > > that
> > > > I'm probably not going to be remembering what I alias'd different
> parts
> > > of
> > > > my jobpath to and again, what the order of the jobpath bits are.
> > > >
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > >
> > > >
> > > > On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> > > > <mc...@apache.org>wrote:
> > > >
> > > > > The bulk of the feedback from the first round of comments on this
> > could
> > > > be
> > > > > summed up, roughly "get rid of the stupid command shortcuts you
> dope,
> > > > > they're a menace!". Message received: they're gone.
> > > > >
> > > > > But I still think that the configurable defaults are valuable.
> Based
> > on
> > > > > some feedback, I've revised the proposal a bit. This includes
> trying
> > to
> > > > > strengthen the motivation/problem statement, and adding ways to
> > provide
> > > > > default parameters only for specific commands, plus many small
> > changes.
> > > > >
> > > > > The updated proposal is attached below.
> > > > >
> > > > >
> > > > > # Defaults and Shorthands in the Aurora Client
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > Most of the time, users are doing the same thing, over and over
> > again.
> > > > > They're working mainly on one particular service, in one particular
> > > > > workspace. But they need to repeat the same parameters, over and
> over
> > > > > again, and they need to remember what those parameters are, what
> > order
> > > > > they occur in, what format they use, and other details that can be
> > > > > difficult to remember.
> > > > >
> > > > > In order to avoid these problems, users set up custom scripts
> > > > > that make it easy for them to run commands like "`myservice
> start`",
> > > > > instead of "`aurora job create west/markcc/prod/myserver
> > > > > src/main/aurora/myservice/myservice.aurora`"
> > > > >
> > > > > To reduce this, we'd like to support a way for users to set up a
> > > > > configuration file that defines defaults and shorthands for their
> > > > > everyday work. With shorthands, a user that only works with a
> single
> > > > > service could say "aurora job create", instead of needing to spell
> > > > > out the full jobkey and configuration file location; a user working
> > > with
> > > > > multiple datacenters could say "`aurora job create @east`" or
> > > > > "`aurora job create @west`" to select the correct jobkey.
> > > > >
> > > > > ## Proposal: Aurora Init Files
> > > > >
> > > > > To allow users to customize shorthands, we'll provide a
> > > > > builtin capability to allow users to provide a configuration
> > > > > file, from which their customizations will be loaded.
> > > > > Many applications use a simple pattern to solve similar problems.
> > > > > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if
> you
> > > > > don't specifically tell the tool where to find a configuration, it
> > > looks
> > > > > in the current directory or one of its parents to find a file named
> > > > > "Vagrantfile".
> > > > >
> > > > > We'd like to follow a similar pattern, and create an "AuroraInit"
> > file.
> > > > > The aurora init file is found by searching the following locations,
> > in
> > > > > order:
> > > > >
> > > > > * the contents of the "--init-file" parameter.
> > > > > * if the "--init-file" parameter is unspecified, then look in
> the
> > > > > current
> > > > > directory for a file named "AuroraInit".
> > > > > * if no "AuroraInit" file exists in the current directory, then
> > look
> > > > in
> > > > > the users
> > > > > home directory for an init file named ".AuroraInit".
> > > > >
> > > > >
> > > > > ### What goes into an init file?
> > > > >
> > > > > We should support the following kinds of things:
> > > > >
> > > > > 1. Universal defaults - user-defined default settings that will
> be
> > > > > applied to
> > > > > all commands. For example, if there is a default config file
> > that
> > > > > should always
> > > > > be used if the user doesn't specify one, that would be a
> > universal
> > > > > default.
> > > > > 2. Command specific defaults - users should be able to specify
> > that
> > > > they
> > > > > always want
> > > > > to use certain parameter settings for a specific command. For
> > > > example,
> > > > > if they
> > > > > want to always use a default batch size of 10 for updates, but
> > > don't
> > > > > want to affect
> > > > > batch sizes for other commands like kill, they could use a
> > command
> > > > > specific default.
> > > > > 3. Aliases - shorthand names for longer parameters. A user
> could
> > > > > specify shorthands
> > > > > "east" and "west" for full jobkeys in two different
> > datacenters.
> > > > >
> > > > > ### Using aliases and defaults
> > > > >
> > > > > A default specifies a set of _bindings_. If a parameter is omitted
> > > > > from a command, and there's a binding for that parameter, it will
> be
> > > > > automatically inserted into the command as if the user had typed
> it.
> > > > >
> > > > > An alias is a short equivalent for a parameter. When a command line
> > is
> > > > > provided by a user, aliases will be expanded inline. A user can
> > > > > specifically
> > > > > mark an alias for expansion by prefixing it with "@"; if an alias
> > > > > appears on the command-line surrounded by whitespace, it will be
> > > > > replaced even if it isn't marked with an "@".
> > > > >
> > > > >
> > > > > ### Defining Shorthands and Defaults
> > > > >
> > > > > The init file is, like aurora job configurations, a python source
> > file
> > > > > using a Pystachio
> > > > > schema. The schema is loaded, and an `Init` object is retrieved
> from
> > > the
> > > > > top-level
> > > > > variable `init` in that file.
> > > > >
> > > > > The pystachio schema for init files is:
> > > > >
> > > > > class CommandDefaults(Struct)
> > > > > command = Required(String)
> > > > > defaults=Map(String, String)
> > > > >
> > > > > class Init(Struct):
> > > > > defaults=Map(String, String)
> > > > > command_defaults = Optional(CommandDefaults)
> > > > > aliases=Map(String, String)
> > > > >
> > > > >
> > > > > For example, an init file could contain the following:
> > > > >
> > > > > init=Init(
> > > > > defaults={
> > > > > "jobspec": "west/markcc/test/myservice",
> > > > > "config_file": "./src/aurora/myservice.aurora"
> > > > > },
> > > > > aliases={
> > > > > "east": "east/markcc/test/myservice",
> > > > > "c": "east",
> > > > > "me": "markchucarroll"
> > > > > },
> > > > > command_defaults(command="job update",
> > > > > defaults={"--batch-size": 10}
> > > > > ))
> > > > >
> > > > >
> > > > > With this configuration file, if the user ran "`aurora job create`"
> > > > without
> > > > > any parameters,
> > > > > it would automatically be expanded to "`aurora job create
> > > > > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> > > > >
> > > > > If a user ran "`aurora job create east`", the alias `east` would be
> > > > > expanded to "east/markcc/test/myservice", and the missing config
> file
> > > > > parameter would
> > > > > be instantiated using the default, to create a command-line:
> > > > > "`aurora job create east/markcc/test/myservice
> > > > > ./src/aurora/myservice.aurora`".
> > > > >
> > > > > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> > > > aliases
> > > > > would
> > > > > be expanded, and the omitted configuration parameter would be
> added:
> > > > > "`aurora job create east/markchucarroll/test/myservice
> > > > > ./src/aurora/myservice.aurora`".
> > > > >
> > > > > If a user ran "`aurora job update`", then the `jobspec` and
> > > `config_file`
> > > > > parameters would
> > > > > get inserted from the global defaults, and the "--batch-size=10"
> > would
> > > be
> > > > > inserted from
> > > > > the command defaults for "aurora job update".
> > > > >
> > > >
> > >
> >
>
--
Kevin Sweeney
@kts
Re: Updated proposal: Aurora Shorthands
Posted by Mark Chu-Carroll <mc...@apache.org>.
The reason that I use an alias for rbt isn't because I'm trying to save
keystrokes. It's because "-o", "-g" and "-r" mean nothing to me. More
keystrokes is fine, if it comes with clarity - that's why I went along with
dropping the automatic shorthands.
The stuff that you can do with Brian's aliases proposal does work equally
well in mine; any alias is automatically substituted, with or without "@",
if it's used in bare form (that is, not as a part of another string in the
command line.) Substituting inline, though, I still believe is useful. If
we implement it, and find that no one uses it, I'll happily remove it.
-Mark
On Thu, May 29, 2014 at 4:10 PM, David McLaughlin <da...@dmclaughlin.com>
wrote:
> I see the point about having global client defaults, but I still think we
> should avoid inferring the jobkey from the home directory. I'd suggest only
> inferring command_defaults, or even storing them separately - but this also
> adds complexity. Interested in what other people think here.
>
> With regards to symbols and the general theme of keystrokes, specifically
> this point:
>
> "I also think that you're over-estimating the degree of memory required to
> > use it successfully; I've certainly seen
> > many cases of people using stuff like that."
>
>
> Look at rbt, which you mentioned you wrote a wrapper around.
>
> This posts a new review:
>
> ./rbt post -o -g
>
> This updates an existing review:
>
> ./rbt post -o -r <id>
>
> This has very few key strokes. It has only one parameter change between the
> two commands that you need to remember. It is very simple when you look at
> it in an isolated fashion. Yet people will still write wrappers around it
> (or just hope it's in my bash history like I do).
>
> In this case I can't imagine constructing partial job keys from variables
> would save you anything. I think if I'm running the commands often enough
> that I'd remember the aliases, I'd either most likely be able to remember
> my cluster and role names too - or more realistically, I'd just add the
> full jobkey path to my AuroraInit file and be done with it.
>
> I'm only just catching on to the fact that you can't define more than one
> job key in the AuroraInit file and aliases might be the way around that. To
> solve that, I actually liked Brian's suggestion in the other thread:
>
> > Perhaps a stop-gap would be to require explicit aliases, e.g.
> >
> > [aliases]
> > test-west = wickman/west/test/service
> > test-east = wickman/east/test/service
> > prod-west = wickman/west/test/service
> > prod-east = wickman/east/test/service
> >
>
> Maybe we could ask users to define pystachio structs with all their
> options, then have a profile (or target) option like:
>
> ./aurora job update --profile=prod
> ./aurora job kill --profile=prod
>
> etc.
>
>
> On Thu, May 29, 2014 at 10:24 AM, Mark Chu-Carroll <mchucarroll@apache.org
> >
> wrote:
>
> > WRT to the home directory fallback: I'm not hugely committed to it, but
> > I've discussed this with a couple of people, and all have liked it, for
> one
> > big reason: they're using git in the twitter codebase - so the working
> > directory that they use contains a ton of different projects. They can't
> > just put an AuroraInit file in the root directory of the workspace,
> because
> > it won't work. So they want to be able to write one, put it in git in
> some
> > directory, and then put a link to it into their homedir. Then they have
> > defaults in place.
> >
> > WRT to separate init file: I think it's very important that the two be
> > different. The config file is a shared entity used by multiple people,
> and
> > possibly used in different ways by different people. For example, one
> > member of a team may typically only work with one of a team's services,
> > while all are defined in the same config file. The user would to have
> > defaults to work with the one service that they actually work on - but
> > making it a default for all users of the config file would not work.
> >
> > WRT @symbols: yes, we could do substitution via the shell or something
> > similar. But I think that the argument you made above about putting
> things
> > into the config file applies here: having all of the shorthands and
> > defaults that the user works with be stored in one place is better that
> > scattering among multiple files. I also think that you're
> over-estimating
> > the degree of memory required to use it successfully; I've certainly seen
> > many cases of people using stuff like that. (Think about how people use
> > git; they know the names of their branches, and they know the order of
> > things for the commands they use most. For normal operation, that's fine.
> > Back in my google days, we had a crazy git->perforce bridge, and it used
> > aliases in exactly this way. It was hugely successful among engineers. I
> > think that this is similar.)
> >
> > Finally, about the @ symbol: I'm not in love with it, but given my
> argument
> > above, if we want that functionality, we have to use something. My first
> > choice would have been to use pystachio-style stashes - but the braces
> are
> > already used by bash; in fact, all of the bracket characters already are
> > used. For single-character prefixes, there aren't many characters that
> > aren't already used by the shell - we can't use !, #, $, ^, &, *, ~, or
> ?.
> > For easy to type stuff, that left very little other than @. And @ is used
> > as a variable prefix in perl and ruby, so it's not without precedent or
> > familiarity to many people.
> >
> > -Mark
> >
> >
> > On Tue, May 27, 2014 at 7:19 PM, David McLaughlin <david@dmclaughlin.com
> >
> > wrote:
> >
> > > I am a huge +1 on the idea of storing my run configurations in config
> > files
> > > so that I don't need to type them out every time. I also really like
> the
> > > idea of command defaults.
> > >
> > > I do however have a couple of comments about the details of the
> proposal.
> > >
> > > I'm a little concerned about the fallback to the user's home
> directory. I
> > > feel like these config files will be per-project and checked in next to
> > the
> > > aurora config file for all to share. Given that things like update
> batch
> > > sizes probably *should* be defined per-project and not per-user, I at
> > least
> > > think we should encourage this. It's also not uncommon to run commands
> > in a
> > > different cwd than you intended, and having such broad fallbacks seems
> to
> > > be just asking for trouble. I think it's perfectly fine to expect
> > > AuroraInit to either be in the correct working directory or for the
> user
> > to
> > > supply an explicit path to the file.
> > >
> > > I also think once you get to that stage, you can make the observation
> > that
> > > most production aurora configs will have an AuroraInit file next to
> them.
> > > So I wonder if we can just simplify here and let the user roll these
> into
> > > one file? It'd be nice to have a naming convention for aurora files so
> > they
> > > could be dropped from the parameters too.
> > >
> > >
> > > The other concern I have is the syntax for injecting variables into
> > > commands with the @symbol. It seems like there are already tools for
> > > solving this - bash, python, etc. that should we just concede defeat
> to.
> > If
> > > I'm typing my commands manually and relying on aliases I'd make a bet
> > that
> > > I'm probably not going to be remembering what I alias'd different parts
> > of
> > > my jobpath to and again, what the order of the jobpath bits are.
> > >
> > >
> > > Thanks,
> > > David
> > >
> > >
> > >
> > > On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> > > <mc...@apache.org>wrote:
> > >
> > > > The bulk of the feedback from the first round of comments on this
> could
> > > be
> > > > summed up, roughly "get rid of the stupid command shortcuts you dope,
> > > > they're a menace!". Message received: they're gone.
> > > >
> > > > But I still think that the configurable defaults are valuable. Based
> on
> > > > some feedback, I've revised the proposal a bit. This includes trying
> to
> > > > strengthen the motivation/problem statement, and adding ways to
> provide
> > > > default parameters only for specific commands, plus many small
> changes.
> > > >
> > > > The updated proposal is attached below.
> > > >
> > > >
> > > > # Defaults and Shorthands in the Aurora Client
> > > >
> > > > ## Motivation
> > > >
> > > > Most of the time, users are doing the same thing, over and over
> again.
> > > > They're working mainly on one particular service, in one particular
> > > > workspace. But they need to repeat the same parameters, over and over
> > > > again, and they need to remember what those parameters are, what
> order
> > > > they occur in, what format they use, and other details that can be
> > > > difficult to remember.
> > > >
> > > > In order to avoid these problems, users set up custom scripts
> > > > that make it easy for them to run commands like "`myservice start`",
> > > > instead of "`aurora job create west/markcc/prod/myserver
> > > > src/main/aurora/myservice/myservice.aurora`"
> > > >
> > > > To reduce this, we'd like to support a way for users to set up a
> > > > configuration file that defines defaults and shorthands for their
> > > > everyday work. With shorthands, a user that only works with a single
> > > > service could say "aurora job create", instead of needing to spell
> > > > out the full jobkey and configuration file location; a user working
> > with
> > > > multiple datacenters could say "`aurora job create @east`" or
> > > > "`aurora job create @west`" to select the correct jobkey.
> > > >
> > > > ## Proposal: Aurora Init Files
> > > >
> > > > To allow users to customize shorthands, we'll provide a
> > > > builtin capability to allow users to provide a configuration
> > > > file, from which their customizations will be loaded.
> > > > Many applications use a simple pattern to solve similar problems.
> > > > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> > > > don't specifically tell the tool where to find a configuration, it
> > looks
> > > > in the current directory or one of its parents to find a file named
> > > > "Vagrantfile".
> > > >
> > > > We'd like to follow a similar pattern, and create an "AuroraInit"
> file.
> > > > The aurora init file is found by searching the following locations,
> in
> > > > order:
> > > >
> > > > * the contents of the "--init-file" parameter.
> > > > * if the "--init-file" parameter is unspecified, then look in the
> > > > current
> > > > directory for a file named "AuroraInit".
> > > > * if no "AuroraInit" file exists in the current directory, then
> look
> > > in
> > > > the users
> > > > home directory for an init file named ".AuroraInit".
> > > >
> > > >
> > > > ### What goes into an init file?
> > > >
> > > > We should support the following kinds of things:
> > > >
> > > > 1. Universal defaults - user-defined default settings that will be
> > > > applied to
> > > > all commands. For example, if there is a default config file
> that
> > > > should always
> > > > be used if the user doesn't specify one, that would be a
> universal
> > > > default.
> > > > 2. Command specific defaults - users should be able to specify
> that
> > > they
> > > > always want
> > > > to use certain parameter settings for a specific command. For
> > > example,
> > > > if they
> > > > want to always use a default batch size of 10 for updates, but
> > don't
> > > > want to affect
> > > > batch sizes for other commands like kill, they could use a
> command
> > > > specific default.
> > > > 3. Aliases - shorthand names for longer parameters. A user could
> > > > specify shorthands
> > > > "east" and "west" for full jobkeys in two different
> datacenters.
> > > >
> > > > ### Using aliases and defaults
> > > >
> > > > A default specifies a set of _bindings_. If a parameter is omitted
> > > > from a command, and there's a binding for that parameter, it will be
> > > > automatically inserted into the command as if the user had typed it.
> > > >
> > > > An alias is a short equivalent for a parameter. When a command line
> is
> > > > provided by a user, aliases will be expanded inline. A user can
> > > > specifically
> > > > mark an alias for expansion by prefixing it with "@"; if an alias
> > > > appears on the command-line surrounded by whitespace, it will be
> > > > replaced even if it isn't marked with an "@".
> > > >
> > > >
> > > > ### Defining Shorthands and Defaults
> > > >
> > > > The init file is, like aurora job configurations, a python source
> file
> > > > using a Pystachio
> > > > schema. The schema is loaded, and an `Init` object is retrieved from
> > the
> > > > top-level
> > > > variable `init` in that file.
> > > >
> > > > The pystachio schema for init files is:
> > > >
> > > > class CommandDefaults(Struct)
> > > > command = Required(String)
> > > > defaults=Map(String, String)
> > > >
> > > > class Init(Struct):
> > > > defaults=Map(String, String)
> > > > command_defaults = Optional(CommandDefaults)
> > > > aliases=Map(String, String)
> > > >
> > > >
> > > > For example, an init file could contain the following:
> > > >
> > > > init=Init(
> > > > defaults={
> > > > "jobspec": "west/markcc/test/myservice",
> > > > "config_file": "./src/aurora/myservice.aurora"
> > > > },
> > > > aliases={
> > > > "east": "east/markcc/test/myservice",
> > > > "c": "east",
> > > > "me": "markchucarroll"
> > > > },
> > > > command_defaults(command="job update",
> > > > defaults={"--batch-size": 10}
> > > > ))
> > > >
> > > >
> > > > With this configuration file, if the user ran "`aurora job create`"
> > > without
> > > > any parameters,
> > > > it would automatically be expanded to "`aurora job create
> > > > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job create east`", the alias `east` would be
> > > > expanded to "east/markcc/test/myservice", and the missing config file
> > > > parameter would
> > > > be instantiated using the default, to create a command-line:
> > > > "`aurora job create east/markcc/test/myservice
> > > > ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> > > aliases
> > > > would
> > > > be expanded, and the omitted configuration parameter would be added:
> > > > "`aurora job create east/markchucarroll/test/myservice
> > > > ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job update`", then the `jobspec` and
> > `config_file`
> > > > parameters would
> > > > get inserted from the global defaults, and the "--batch-size=10"
> would
> > be
> > > > inserted from
> > > > the command defaults for "aurora job update".
> > > >
> > >
> >
>
Re: Updated proposal: Aurora Shorthands
Posted by David McLaughlin <da...@dmclaughlin.com>.
I see the point about having global client defaults, but I still think we
should avoid inferring the jobkey from the home directory. I'd suggest only
inferring command_defaults, or even storing them separately - but this also
adds complexity. Interested in what other people think here.
With regards to symbols and the general theme of keystrokes, specifically
this point:
"I also think that you're over-estimating the degree of memory required to
> use it successfully; I've certainly seen
> many cases of people using stuff like that."
Look at rbt, which you mentioned you wrote a wrapper around.
This posts a new review:
./rbt post -o -g
This updates an existing review:
./rbt post -o -r <id>
This has very few key strokes. It has only one parameter change between the
two commands that you need to remember. It is very simple when you look at
it in an isolated fashion. Yet people will still write wrappers around it
(or just hope it's in my bash history like I do).
In this case I can't imagine constructing partial job keys from variables
would save you anything. I think if I'm running the commands often enough
that I'd remember the aliases, I'd either most likely be able to remember
my cluster and role names too - or more realistically, I'd just add the
full jobkey path to my AuroraInit file and be done with it.
I'm only just catching on to the fact that you can't define more than one
job key in the AuroraInit file and aliases might be the way around that. To
solve that, I actually liked Brian's suggestion in the other thread:
> Perhaps a stop-gap would be to require explicit aliases, e.g.
>
> [aliases]
> test-west = wickman/west/test/service
> test-east = wickman/east/test/service
> prod-west = wickman/west/test/service
> prod-east = wickman/east/test/service
>
Maybe we could ask users to define pystachio structs with all their
options, then have a profile (or target) option like:
./aurora job update --profile=prod
./aurora job kill --profile=prod
etc.
On Thu, May 29, 2014 at 10:24 AM, Mark Chu-Carroll <mc...@apache.org>
wrote:
> WRT to the home directory fallback: I'm not hugely committed to it, but
> I've discussed this with a couple of people, and all have liked it, for one
> big reason: they're using git in the twitter codebase - so the working
> directory that they use contains a ton of different projects. They can't
> just put an AuroraInit file in the root directory of the workspace, because
> it won't work. So they want to be able to write one, put it in git in some
> directory, and then put a link to it into their homedir. Then they have
> defaults in place.
>
> WRT to separate init file: I think it's very important that the two be
> different. The config file is a shared entity used by multiple people, and
> possibly used in different ways by different people. For example, one
> member of a team may typically only work with one of a team's services,
> while all are defined in the same config file. The user would to have
> defaults to work with the one service that they actually work on - but
> making it a default for all users of the config file would not work.
>
> WRT @symbols: yes, we could do substitution via the shell or something
> similar. But I think that the argument you made above about putting things
> into the config file applies here: having all of the shorthands and
> defaults that the user works with be stored in one place is better that
> scattering among multiple files. I also think that you're over-estimating
> the degree of memory required to use it successfully; I've certainly seen
> many cases of people using stuff like that. (Think about how people use
> git; they know the names of their branches, and they know the order of
> things for the commands they use most. For normal operation, that's fine.
> Back in my google days, we had a crazy git->perforce bridge, and it used
> aliases in exactly this way. It was hugely successful among engineers. I
> think that this is similar.)
>
> Finally, about the @ symbol: I'm not in love with it, but given my argument
> above, if we want that functionality, we have to use something. My first
> choice would have been to use pystachio-style stashes - but the braces are
> already used by bash; in fact, all of the bracket characters already are
> used. For single-character prefixes, there aren't many characters that
> aren't already used by the shell - we can't use !, #, $, ^, &, *, ~, or ?.
> For easy to type stuff, that left very little other than @. And @ is used
> as a variable prefix in perl and ruby, so it's not without precedent or
> familiarity to many people.
>
> -Mark
>
>
> On Tue, May 27, 2014 at 7:19 PM, David McLaughlin <da...@dmclaughlin.com>
> wrote:
>
> > I am a huge +1 on the idea of storing my run configurations in config
> files
> > so that I don't need to type them out every time. I also really like the
> > idea of command defaults.
> >
> > I do however have a couple of comments about the details of the proposal.
> >
> > I'm a little concerned about the fallback to the user's home directory. I
> > feel like these config files will be per-project and checked in next to
> the
> > aurora config file for all to share. Given that things like update batch
> > sizes probably *should* be defined per-project and not per-user, I at
> least
> > think we should encourage this. It's also not uncommon to run commands
> in a
> > different cwd than you intended, and having such broad fallbacks seems to
> > be just asking for trouble. I think it's perfectly fine to expect
> > AuroraInit to either be in the correct working directory or for the user
> to
> > supply an explicit path to the file.
> >
> > I also think once you get to that stage, you can make the observation
> that
> > most production aurora configs will have an AuroraInit file next to them.
> > So I wonder if we can just simplify here and let the user roll these into
> > one file? It'd be nice to have a naming convention for aurora files so
> they
> > could be dropped from the parameters too.
> >
> >
> > The other concern I have is the syntax for injecting variables into
> > commands with the @symbol. It seems like there are already tools for
> > solving this - bash, python, etc. that should we just concede defeat to.
> If
> > I'm typing my commands manually and relying on aliases I'd make a bet
> that
> > I'm probably not going to be remembering what I alias'd different parts
> of
> > my jobpath to and again, what the order of the jobpath bits are.
> >
> >
> > Thanks,
> > David
> >
> >
> >
> > On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> > <mc...@apache.org>wrote:
> >
> > > The bulk of the feedback from the first round of comments on this could
> > be
> > > summed up, roughly "get rid of the stupid command shortcuts you dope,
> > > they're a menace!". Message received: they're gone.
> > >
> > > But I still think that the configurable defaults are valuable. Based on
> > > some feedback, I've revised the proposal a bit. This includes trying to
> > > strengthen the motivation/problem statement, and adding ways to provide
> > > default parameters only for specific commands, plus many small changes.
> > >
> > > The updated proposal is attached below.
> > >
> > >
> > > # Defaults and Shorthands in the Aurora Client
> > >
> > > ## Motivation
> > >
> > > Most of the time, users are doing the same thing, over and over again.
> > > They're working mainly on one particular service, in one particular
> > > workspace. But they need to repeat the same parameters, over and over
> > > again, and they need to remember what those parameters are, what order
> > > they occur in, what format they use, and other details that can be
> > > difficult to remember.
> > >
> > > In order to avoid these problems, users set up custom scripts
> > > that make it easy for them to run commands like "`myservice start`",
> > > instead of "`aurora job create west/markcc/prod/myserver
> > > src/main/aurora/myservice/myservice.aurora`"
> > >
> > > To reduce this, we'd like to support a way for users to set up a
> > > configuration file that defines defaults and shorthands for their
> > > everyday work. With shorthands, a user that only works with a single
> > > service could say "aurora job create", instead of needing to spell
> > > out the full jobkey and configuration file location; a user working
> with
> > > multiple datacenters could say "`aurora job create @east`" or
> > > "`aurora job create @west`" to select the correct jobkey.
> > >
> > > ## Proposal: Aurora Init Files
> > >
> > > To allow users to customize shorthands, we'll provide a
> > > builtin capability to allow users to provide a configuration
> > > file, from which their customizations will be loaded.
> > > Many applications use a simple pattern to solve similar problems.
> > > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> > > don't specifically tell the tool where to find a configuration, it
> looks
> > > in the current directory or one of its parents to find a file named
> > > "Vagrantfile".
> > >
> > > We'd like to follow a similar pattern, and create an "AuroraInit" file.
> > > The aurora init file is found by searching the following locations, in
> > > order:
> > >
> > > * the contents of the "--init-file" parameter.
> > > * if the "--init-file" parameter is unspecified, then look in the
> > > current
> > > directory for a file named "AuroraInit".
> > > * if no "AuroraInit" file exists in the current directory, then look
> > in
> > > the users
> > > home directory for an init file named ".AuroraInit".
> > >
> > >
> > > ### What goes into an init file?
> > >
> > > We should support the following kinds of things:
> > >
> > > 1. Universal defaults - user-defined default settings that will be
> > > applied to
> > > all commands. For example, if there is a default config file that
> > > should always
> > > be used if the user doesn't specify one, that would be a universal
> > > default.
> > > 2. Command specific defaults - users should be able to specify that
> > they
> > > always want
> > > to use certain parameter settings for a specific command. For
> > example,
> > > if they
> > > want to always use a default batch size of 10 for updates, but
> don't
> > > want to affect
> > > batch sizes for other commands like kill, they could use a command
> > > specific default.
> > > 3. Aliases - shorthand names for longer parameters. A user could
> > > specify shorthands
> > > "east" and "west" for full jobkeys in two different datacenters.
> > >
> > > ### Using aliases and defaults
> > >
> > > A default specifies a set of _bindings_. If a parameter is omitted
> > > from a command, and there's a binding for that parameter, it will be
> > > automatically inserted into the command as if the user had typed it.
> > >
> > > An alias is a short equivalent for a parameter. When a command line is
> > > provided by a user, aliases will be expanded inline. A user can
> > > specifically
> > > mark an alias for expansion by prefixing it with "@"; if an alias
> > > appears on the command-line surrounded by whitespace, it will be
> > > replaced even if it isn't marked with an "@".
> > >
> > >
> > > ### Defining Shorthands and Defaults
> > >
> > > The init file is, like aurora job configurations, a python source file
> > > using a Pystachio
> > > schema. The schema is loaded, and an `Init` object is retrieved from
> the
> > > top-level
> > > variable `init` in that file.
> > >
> > > The pystachio schema for init files is:
> > >
> > > class CommandDefaults(Struct)
> > > command = Required(String)
> > > defaults=Map(String, String)
> > >
> > > class Init(Struct):
> > > defaults=Map(String, String)
> > > command_defaults = Optional(CommandDefaults)
> > > aliases=Map(String, String)
> > >
> > >
> > > For example, an init file could contain the following:
> > >
> > > init=Init(
> > > defaults={
> > > "jobspec": "west/markcc/test/myservice",
> > > "config_file": "./src/aurora/myservice.aurora"
> > > },
> > > aliases={
> > > "east": "east/markcc/test/myservice",
> > > "c": "east",
> > > "me": "markchucarroll"
> > > },
> > > command_defaults(command="job update",
> > > defaults={"--batch-size": 10}
> > > ))
> > >
> > >
> > > With this configuration file, if the user ran "`aurora job create`"
> > without
> > > any parameters,
> > > it would automatically be expanded to "`aurora job create
> > > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> > >
> > > If a user ran "`aurora job create east`", the alias `east` would be
> > > expanded to "east/markcc/test/myservice", and the missing config file
> > > parameter would
> > > be instantiated using the default, to create a command-line:
> > > "`aurora job create east/markcc/test/myservice
> > > ./src/aurora/myservice.aurora`".
> > >
> > > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> > aliases
> > > would
> > > be expanded, and the omitted configuration parameter would be added:
> > > "`aurora job create east/markchucarroll/test/myservice
> > > ./src/aurora/myservice.aurora`".
> > >
> > > If a user ran "`aurora job update`", then the `jobspec` and
> `config_file`
> > > parameters would
> > > get inserted from the global defaults, and the "--batch-size=10" would
> be
> > > inserted from
> > > the command defaults for "aurora job update".
> > >
> >
>
Re: Updated proposal: Aurora Shorthands
Posted by Mark Chu-Carroll <mc...@apache.org>.
WRT to the home directory fallback: I'm not hugely committed to it, but
I've discussed this with a couple of people, and all have liked it, for one
big reason: they're using git in the twitter codebase - so the working
directory that they use contains a ton of different projects. They can't
just put an AuroraInit file in the root directory of the workspace, because
it won't work. So they want to be able to write one, put it in git in some
directory, and then put a link to it into their homedir. Then they have
defaults in place.
WRT to separate init file: I think it's very important that the two be
different. The config file is a shared entity used by multiple people, and
possibly used in different ways by different people. For example, one
member of a team may typically only work with one of a team's services,
while all are defined in the same config file. The user would to have
defaults to work with the one service that they actually work on - but
making it a default for all users of the config file would not work.
WRT @symbols: yes, we could do substitution via the shell or something
similar. But I think that the argument you made above about putting things
into the config file applies here: having all of the shorthands and
defaults that the user works with be stored in one place is better that
scattering among multiple files. I also think that you're over-estimating
the degree of memory required to use it successfully; I've certainly seen
many cases of people using stuff like that. (Think about how people use
git; they know the names of their branches, and they know the order of
things for the commands they use most. For normal operation, that's fine.
Back in my google days, we had a crazy git->perforce bridge, and it used
aliases in exactly this way. It was hugely successful among engineers. I
think that this is similar.)
Finally, about the @ symbol: I'm not in love with it, but given my argument
above, if we want that functionality, we have to use something. My first
choice would have been to use pystachio-style stashes - but the braces are
already used by bash; in fact, all of the bracket characters already are
used. For single-character prefixes, there aren't many characters that
aren't already used by the shell - we can't use !, #, $, ^, &, *, ~, or ?.
For easy to type stuff, that left very little other than @. And @ is used
as a variable prefix in perl and ruby, so it's not without precedent or
familiarity to many people.
-Mark
On Tue, May 27, 2014 at 7:19 PM, David McLaughlin <da...@dmclaughlin.com>
wrote:
> I am a huge +1 on the idea of storing my run configurations in config files
> so that I don't need to type them out every time. I also really like the
> idea of command defaults.
>
> I do however have a couple of comments about the details of the proposal.
>
> I'm a little concerned about the fallback to the user's home directory. I
> feel like these config files will be per-project and checked in next to the
> aurora config file for all to share. Given that things like update batch
> sizes probably *should* be defined per-project and not per-user, I at least
> think we should encourage this. It's also not uncommon to run commands in a
> different cwd than you intended, and having such broad fallbacks seems to
> be just asking for trouble. I think it's perfectly fine to expect
> AuroraInit to either be in the correct working directory or for the user to
> supply an explicit path to the file.
>
> I also think once you get to that stage, you can make the observation that
> most production aurora configs will have an AuroraInit file next to them.
> So I wonder if we can just simplify here and let the user roll these into
> one file? It'd be nice to have a naming convention for aurora files so they
> could be dropped from the parameters too.
>
>
> The other concern I have is the syntax for injecting variables into
> commands with the @symbol. It seems like there are already tools for
> solving this - bash, python, etc. that should we just concede defeat to. If
> I'm typing my commands manually and relying on aliases I'd make a bet that
> I'm probably not going to be remembering what I alias'd different parts of
> my jobpath to and again, what the order of the jobpath bits are.
>
>
> Thanks,
> David
>
>
>
> On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> <mc...@apache.org>wrote:
>
> > The bulk of the feedback from the first round of comments on this could
> be
> > summed up, roughly "get rid of the stupid command shortcuts you dope,
> > they're a menace!". Message received: they're gone.
> >
> > But I still think that the configurable defaults are valuable. Based on
> > some feedback, I've revised the proposal a bit. This includes trying to
> > strengthen the motivation/problem statement, and adding ways to provide
> > default parameters only for specific commands, plus many small changes.
> >
> > The updated proposal is attached below.
> >
> >
> > # Defaults and Shorthands in the Aurora Client
> >
> > ## Motivation
> >
> > Most of the time, users are doing the same thing, over and over again.
> > They're working mainly on one particular service, in one particular
> > workspace. But they need to repeat the same parameters, over and over
> > again, and they need to remember what those parameters are, what order
> > they occur in, what format they use, and other details that can be
> > difficult to remember.
> >
> > In order to avoid these problems, users set up custom scripts
> > that make it easy for them to run commands like "`myservice start`",
> > instead of "`aurora job create west/markcc/prod/myserver
> > src/main/aurora/myservice/myservice.aurora`"
> >
> > To reduce this, we'd like to support a way for users to set up a
> > configuration file that defines defaults and shorthands for their
> > everyday work. With shorthands, a user that only works with a single
> > service could say "aurora job create", instead of needing to spell
> > out the full jobkey and configuration file location; a user working with
> > multiple datacenters could say "`aurora job create @east`" or
> > "`aurora job create @west`" to select the correct jobkey.
> >
> > ## Proposal: Aurora Init Files
> >
> > To allow users to customize shorthands, we'll provide a
> > builtin capability to allow users to provide a configuration
> > file, from which their customizations will be loaded.
> > Many applications use a simple pattern to solve similar problems.
> > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> > don't specifically tell the tool where to find a configuration, it looks
> > in the current directory or one of its parents to find a file named
> > "Vagrantfile".
> >
> > We'd like to follow a similar pattern, and create an "AuroraInit" file.
> > The aurora init file is found by searching the following locations, in
> > order:
> >
> > * the contents of the "--init-file" parameter.
> > * if the "--init-file" parameter is unspecified, then look in the
> > current
> > directory for a file named "AuroraInit".
> > * if no "AuroraInit" file exists in the current directory, then look
> in
> > the users
> > home directory for an init file named ".AuroraInit".
> >
> >
> > ### What goes into an init file?
> >
> > We should support the following kinds of things:
> >
> > 1. Universal defaults - user-defined default settings that will be
> > applied to
> > all commands. For example, if there is a default config file that
> > should always
> > be used if the user doesn't specify one, that would be a universal
> > default.
> > 2. Command specific defaults - users should be able to specify that
> they
> > always want
> > to use certain parameter settings for a specific command. For
> example,
> > if they
> > want to always use a default batch size of 10 for updates, but don't
> > want to affect
> > batch sizes for other commands like kill, they could use a command
> > specific default.
> > 3. Aliases - shorthand names for longer parameters. A user could
> > specify shorthands
> > "east" and "west" for full jobkeys in two different datacenters.
> >
> > ### Using aliases and defaults
> >
> > A default specifies a set of _bindings_. If a parameter is omitted
> > from a command, and there's a binding for that parameter, it will be
> > automatically inserted into the command as if the user had typed it.
> >
> > An alias is a short equivalent for a parameter. When a command line is
> > provided by a user, aliases will be expanded inline. A user can
> > specifically
> > mark an alias for expansion by prefixing it with "@"; if an alias
> > appears on the command-line surrounded by whitespace, it will be
> > replaced even if it isn't marked with an "@".
> >
> >
> > ### Defining Shorthands and Defaults
> >
> > The init file is, like aurora job configurations, a python source file
> > using a Pystachio
> > schema. The schema is loaded, and an `Init` object is retrieved from the
> > top-level
> > variable `init` in that file.
> >
> > The pystachio schema for init files is:
> >
> > class CommandDefaults(Struct)
> > command = Required(String)
> > defaults=Map(String, String)
> >
> > class Init(Struct):
> > defaults=Map(String, String)
> > command_defaults = Optional(CommandDefaults)
> > aliases=Map(String, String)
> >
> >
> > For example, an init file could contain the following:
> >
> > init=Init(
> > defaults={
> > "jobspec": "west/markcc/test/myservice",
> > "config_file": "./src/aurora/myservice.aurora"
> > },
> > aliases={
> > "east": "east/markcc/test/myservice",
> > "c": "east",
> > "me": "markchucarroll"
> > },
> > command_defaults(command="job update",
> > defaults={"--batch-size": 10}
> > ))
> >
> >
> > With this configuration file, if the user ran "`aurora job create`"
> without
> > any parameters,
> > it would automatically be expanded to "`aurora job create
> > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job create east`", the alias `east` would be
> > expanded to "east/markcc/test/myservice", and the missing config file
> > parameter would
> > be instantiated using the default, to create a command-line:
> > "`aurora job create east/markcc/test/myservice
> > ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> aliases
> > would
> > be expanded, and the omitted configuration parameter would be added:
> > "`aurora job create east/markchucarroll/test/myservice
> > ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job update`", then the `jobspec` and `config_file`
> > parameters would
> > get inserted from the global defaults, and the "--batch-size=10" would be
> > inserted from
> > the command defaults for "aurora job update".
> >
>
Re: Updated proposal: Aurora Shorthands
Posted by David McLaughlin <da...@dmclaughlin.com>.
I am a huge +1 on the idea of storing my run configurations in config files
so that I don't need to type them out every time. I also really like the
idea of command defaults.
I do however have a couple of comments about the details of the proposal.
I'm a little concerned about the fallback to the user's home directory. I
feel like these config files will be per-project and checked in next to the
aurora config file for all to share. Given that things like update batch
sizes probably *should* be defined per-project and not per-user, I at least
think we should encourage this. It's also not uncommon to run commands in a
different cwd than you intended, and having such broad fallbacks seems to
be just asking for trouble. I think it's perfectly fine to expect
AuroraInit to either be in the correct working directory or for the user to
supply an explicit path to the file.
I also think once you get to that stage, you can make the observation that
most production aurora configs will have an AuroraInit file next to them.
So I wonder if we can just simplify here and let the user roll these into
one file? It'd be nice to have a naming convention for aurora files so they
could be dropped from the parameters too.
The other concern I have is the syntax for injecting variables into
commands with the @symbol. It seems like there are already tools for
solving this - bash, python, etc. that should we just concede defeat to. If
I'm typing my commands manually and relying on aliases I'd make a bet that
I'm probably not going to be remembering what I alias'd different parts of
my jobpath to and again, what the order of the jobpath bits are.
Thanks,
David
On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
<mc...@apache.org>wrote:
> The bulk of the feedback from the first round of comments on this could be
> summed up, roughly "get rid of the stupid command shortcuts you dope,
> they're a menace!". Message received: they're gone.
>
> But I still think that the configurable defaults are valuable. Based on
> some feedback, I've revised the proposal a bit. This includes trying to
> strengthen the motivation/problem statement, and adding ways to provide
> default parameters only for specific commands, plus many small changes.
>
> The updated proposal is attached below.
>
>
> # Defaults and Shorthands in the Aurora Client
>
> ## Motivation
>
> Most of the time, users are doing the same thing, over and over again.
> They're working mainly on one particular service, in one particular
> workspace. But they need to repeat the same parameters, over and over
> again, and they need to remember what those parameters are, what order
> they occur in, what format they use, and other details that can be
> difficult to remember.
>
> In order to avoid these problems, users set up custom scripts
> that make it easy for them to run commands like "`myservice start`",
> instead of "`aurora job create west/markcc/prod/myserver
> src/main/aurora/myservice/myservice.aurora`"
>
> To reduce this, we'd like to support a way for users to set up a
> configuration file that defines defaults and shorthands for their
> everyday work. With shorthands, a user that only works with a single
> service could say "aurora job create", instead of needing to spell
> out the full jobkey and configuration file location; a user working with
> multiple datacenters could say "`aurora job create @east`" or
> "`aurora job create @west`" to select the correct jobkey.
>
> ## Proposal: Aurora Init Files
>
> To allow users to customize shorthands, we'll provide a
> builtin capability to allow users to provide a configuration
> file, from which their customizations will be loaded.
> Many applications use a simple pattern to solve similar problems.
> Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> don't specifically tell the tool where to find a configuration, it looks
> in the current directory or one of its parents to find a file named
> "Vagrantfile".
>
> We'd like to follow a similar pattern, and create an "AuroraInit" file.
> The aurora init file is found by searching the following locations, in
> order:
>
> * the contents of the "--init-file" parameter.
> * if the "--init-file" parameter is unspecified, then look in the
> current
> directory for a file named "AuroraInit".
> * if no "AuroraInit" file exists in the current directory, then look in
> the users
> home directory for an init file named ".AuroraInit".
>
>
> ### What goes into an init file?
>
> We should support the following kinds of things:
>
> 1. Universal defaults - user-defined default settings that will be
> applied to
> all commands. For example, if there is a default config file that
> should always
> be used if the user doesn't specify one, that would be a universal
> default.
> 2. Command specific defaults - users should be able to specify that they
> always want
> to use certain parameter settings for a specific command. For example,
> if they
> want to always use a default batch size of 10 for updates, but don't
> want to affect
> batch sizes for other commands like kill, they could use a command
> specific default.
> 3. Aliases - shorthand names for longer parameters. A user could
> specify shorthands
> "east" and "west" for full jobkeys in two different datacenters.
>
> ### Using aliases and defaults
>
> A default specifies a set of _bindings_. If a parameter is omitted
> from a command, and there's a binding for that parameter, it will be
> automatically inserted into the command as if the user had typed it.
>
> An alias is a short equivalent for a parameter. When a command line is
> provided by a user, aliases will be expanded inline. A user can
> specifically
> mark an alias for expansion by prefixing it with "@"; if an alias
> appears on the command-line surrounded by whitespace, it will be
> replaced even if it isn't marked with an "@".
>
>
> ### Defining Shorthands and Defaults
>
> The init file is, like aurora job configurations, a python source file
> using a Pystachio
> schema. The schema is loaded, and an `Init` object is retrieved from the
> top-level
> variable `init` in that file.
>
> The pystachio schema for init files is:
>
> class CommandDefaults(Struct)
> command = Required(String)
> defaults=Map(String, String)
>
> class Init(Struct):
> defaults=Map(String, String)
> command_defaults = Optional(CommandDefaults)
> aliases=Map(String, String)
>
>
> For example, an init file could contain the following:
>
> init=Init(
> defaults={
> "jobspec": "west/markcc/test/myservice",
> "config_file": "./src/aurora/myservice.aurora"
> },
> aliases={
> "east": "east/markcc/test/myservice",
> "c": "east",
> "me": "markchucarroll"
> },
> command_defaults(command="job update",
> defaults={"--batch-size": 10}
> ))
>
>
> With this configuration file, if the user ran "`aurora job create`" without
> any parameters,
> it would automatically be expanded to "`aurora job create
> west/markcc/test/myservice ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job create east`", the alias `east` would be
> expanded to "east/markcc/test/myservice", and the missing config file
> parameter would
> be instantiated using the default, to create a command-line:
> "`aurora job create east/markcc/test/myservice
> ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job create @c/@me/test/myservice`", the two aliases
> would
> be expanded, and the omitted configuration parameter would be added:
> "`aurora job create east/markchucarroll/test/myservice
> ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job update`", then the `jobspec` and `config_file`
> parameters would
> get inserted from the global defaults, and the "--batch-size=10" would be
> inserted from
> the command defaults for "aurora job update".
>
Re: Updated proposal: Aurora Shorthands
Posted by Mark Chu-Carroll <mc...@apache.org>.
I think that the answer there is that we want to *reduce* scripting, not
*eliminate* it.
Scripting happens for a bunch of different reasons:
(1) Filling functional gaps, where the original application is missing some
key functionality that users really need. For example, in my last job, we
had a script to submit code for code review. The application we used for
code review used a very specific model of branching for submitting reviews,
which we thought was awkward and contrived. So we wrote scripts, so that we
didn't need to do it.
(2) Customizing an application for specific workflows. For example,
specific aurora users will have deploy processes that run specific tests,
create canaries in a particular way, etc.
(3) Patching around obscurity/complexity. Users don't like to type the same
long-winded things over and over again, or remember complex or obscure
parameters. For example, I've got shell scripts for running rbt in the
apache aurora codebase. I can't remember that the obscure "-o -g" means
"create a new review", but "-o -r" means update an existing review. So I
made some helpers so that I don't need to remember them.
Type (1) scripts represent a clear lack in the application. If people need
to write type(1) scripts for aurora, that means that we've failed to
deliver an adequate tool for what they need.
Type (2) scripts are inevitable, and we shouldn't worry about them at all.
Type (3) scripts are also a sign of failure in the application.
I don't typically wrap git in a bunch of clumsy aliases, because, while I'm
not crazy about the way it works, for most routine tasks, the parameters
are clear, and I don't have trouble remembering them.
On the other hand, for pants, I set up command aliases - not because it's
hard to remember things, but because it's damned awkward to have to type
out "pants test/python/apache/aurora/client/cli:all" instead of just
"runtests".
For aurora, we've got a system which, I think, does a decent job of
providing users with the functionality that they need. (Obviously it could
do better, and it's steadily moving in the direction of getting better at
it.) But what we don't do is make it *easy* to use. In the category 3
arena, I think we've got a massive failing. A typical aurora command
requires an easy to remember noun/verb pair - and then a 4-part jobkey
(which users have trouble remember the order of the key parts), and a long
path to a configuration file. For a typical invocation of the aurora
client, the parameters are about 70 characters long. Those 70 characters
are the same, over and over again, with minimal variation.
That's a problem that creates a need for users to write scripts, because
we're not doing a good enough job of covering their common usecases -
instead, we confront them with a need for verbose obscure parameters that
they need to remember.
The defaults/init file proposal is an attempt to address that case - making
the system easier to use for the most common use-cases, for the typical
users. They won't need to write wrapper scripts if we provide them with an
easy way of doing what they need to do.
-Mark
On Sat, May 24, 2014 at 1:52 AM, Bill Farner <wf...@apache.org> wrote:
> >
> > In order to avoid these problems, users set up custom scripts
> > that make it easy for them to run commands like "`myservice start`",
> > instead of "`aurora job create west/markcc/prod/myserver
> > src/main/aurora/myservice/myservice.aurora`"
> > To reduce this...
>
>
> The missing detail seems to be why the custom scripts need to be reduced.
> In my experience, people do this to automate a deployment workflow, which
> aurora happens to be a part of. Even with the shorthands, i'm pretty
> confident people will still have a script to roll up the "build/test", and
> "check that the service works" steps. If that holds true, it seems that
> the goal of reducing scripting is not met.
>
>
>
> -=Bill
>
>
> On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> <mc...@apache.org>wrote:
>
> > The bulk of the feedback from the first round of comments on this could
> be
> > summed up, roughly "get rid of the stupid command shortcuts you dope,
> > they're a menace!". Message received: they're gone.
> >
> > But I still think that the configurable defaults are valuable. Based on
> > some feedback, I've revised the proposal a bit. This includes trying to
> > strengthen the motivation/problem statement, and adding ways to provide
> > default parameters only for specific commands, plus many small changes.
> >
> > The updated proposal is attached below.
> >
> >
> > # Defaults and Shorthands in the Aurora Client
> >
> > ## Motivation
> >
> > Most of the time, users are doing the same thing, over and over again.
> > They're working mainly on one particular service, in one particular
> > workspace. But they need to repeat the same parameters, over and over
> > again, and they need to remember what those parameters are, what order
> > they occur in, what format they use, and other details that can be
> > difficult to remember.
> >
> > In order to avoid these problems, users set up custom scripts
> > that make it easy for them to run commands like "`myservice start`",
> > instead of "`aurora job create west/markcc/prod/myserver
> > src/main/aurora/myservice/myservice.aurora`"
> >
> > To reduce this, we'd like to support a way for users to set up a
> > configuration file that defines defaults and shorthands for their
> > everyday work. With shorthands, a user that only works with a single
> > service could say "aurora job create", instead of needing to spell
> > out the full jobkey and configuration file location; a user working with
> > multiple datacenters could say "`aurora job create @east`" or
> > "`aurora job create @west`" to select the correct jobkey.
> >
> > ## Proposal: Aurora Init Files
> >
> > To allow users to customize shorthands, we'll provide a
> > builtin capability to allow users to provide a configuration
> > file, from which their customizations will be loaded.
> > Many applications use a simple pattern to solve similar problems.
> > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> > don't specifically tell the tool where to find a configuration, it looks
> > in the current directory or one of its parents to find a file named
> > "Vagrantfile".
> >
> > We'd like to follow a similar pattern, and create an "AuroraInit" file.
> > The aurora init file is found by searching the following locations, in
> > order:
> >
> > * the contents of the "--init-file" parameter.
> > * if the "--init-file" parameter is unspecified, then look in the
> > current
> > directory for a file named "AuroraInit".
> > * if no "AuroraInit" file exists in the current directory, then look
> in
> > the users
> > home directory for an init file named ".AuroraInit".
> >
> >
> > ### What goes into an init file?
> >
> > We should support the following kinds of things:
> >
> > 1. Universal defaults - user-defined default settings that will be
> > applied to
> > all commands. For example, if there is a default config file that
> > should always
> > be used if the user doesn't specify one, that would be a universal
> > default.
> > 2. Command specific defaults - users should be able to specify that
> they
> > always want
> > to use certain parameter settings for a specific command. For
> example,
> > if they
> > want to always use a default batch size of 10 for updates, but don't
> > want to affect
> > batch sizes for other commands like kill, they could use a command
> > specific default.
> > 3. Aliases - shorthand names for longer parameters. A user could
> > specify shorthands
> > "east" and "west" for full jobkeys in two different datacenters.
> >
> > ### Using aliases and defaults
> >
> > A default specifies a set of _bindings_. If a parameter is omitted
> > from a command, and there's a binding for that parameter, it will be
> > automatically inserted into the command as if the user had typed it.
> >
> > An alias is a short equivalent for a parameter. When a command line is
> > provided by a user, aliases will be expanded inline. A user can
> > specifically
> > mark an alias for expansion by prefixing it with "@"; if an alias
> > appears on the command-line surrounded by whitespace, it will be
> > replaced even if it isn't marked with an "@".
> >
> >
> > ### Defining Shorthands and Defaults
> >
> > The init file is, like aurora job configurations, a python source file
> > using a Pystachio
> > schema. The schema is loaded, and an `Init` object is retrieved from the
> > top-level
> > variable `init` in that file.
> >
> > The pystachio schema for init files is:
> >
> > class CommandDefaults(Struct)
> > command = Required(String)
> > defaults=Map(String, String)
> >
> > class Init(Struct):
> > defaults=Map(String, String)
> > command_defaults = Optional(CommandDefaults)
> > aliases=Map(String, String)
> >
> >
> > For example, an init file could contain the following:
> >
> > init=Init(
> > defaults={
> > "jobspec": "west/markcc/test/myservice",
> > "config_file": "./src/aurora/myservice.aurora"
> > },
> > aliases={
> > "east": "east/markcc/test/myservice",
> > "c": "east",
> > "me": "markchucarroll"
> > },
> > command_defaults(command="job update",
> > defaults={"--batch-size": 10}
> > ))
> >
> >
> > With this configuration file, if the user ran "`aurora job create`"
> without
> > any parameters,
> > it would automatically be expanded to "`aurora job create
> > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job create east`", the alias `east` would be
> > expanded to "east/markcc/test/myservice", and the missing config file
> > parameter would
> > be instantiated using the default, to create a command-line:
> > "`aurora job create east/markcc/test/myservice
> > ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> aliases
> > would
> > be expanded, and the omitted configuration parameter would be added:
> > "`aurora job create east/markchucarroll/test/myservice
> > ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job update`", then the `jobspec` and `config_file`
> > parameters would
> > get inserted from the global defaults, and the "--batch-size=10" would be
> > inserted from
> > the command defaults for "aurora job update".
> >
>
Re: Updated proposal: Aurora Shorthands
Posted by Bill Farner <wf...@apache.org>.
>
> In order to avoid these problems, users set up custom scripts
> that make it easy for them to run commands like "`myservice start`",
> instead of "`aurora job create west/markcc/prod/myserver
> src/main/aurora/myservice/myservice.aurora`"
> To reduce this...
The missing detail seems to be why the custom scripts need to be reduced.
In my experience, people do this to automate a deployment workflow, which
aurora happens to be a part of. Even with the shorthands, i'm pretty
confident people will still have a script to roll up the "build/test", and
"check that the service works" steps. If that holds true, it seems that
the goal of reducing scripting is not met.
-=Bill
On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
<mc...@apache.org>wrote:
> The bulk of the feedback from the first round of comments on this could be
> summed up, roughly "get rid of the stupid command shortcuts you dope,
> they're a menace!". Message received: they're gone.
>
> But I still think that the configurable defaults are valuable. Based on
> some feedback, I've revised the proposal a bit. This includes trying to
> strengthen the motivation/problem statement, and adding ways to provide
> default parameters only for specific commands, plus many small changes.
>
> The updated proposal is attached below.
>
>
> # Defaults and Shorthands in the Aurora Client
>
> ## Motivation
>
> Most of the time, users are doing the same thing, over and over again.
> They're working mainly on one particular service, in one particular
> workspace. But they need to repeat the same parameters, over and over
> again, and they need to remember what those parameters are, what order
> they occur in, what format they use, and other details that can be
> difficult to remember.
>
> In order to avoid these problems, users set up custom scripts
> that make it easy for them to run commands like "`myservice start`",
> instead of "`aurora job create west/markcc/prod/myserver
> src/main/aurora/myservice/myservice.aurora`"
>
> To reduce this, we'd like to support a way for users to set up a
> configuration file that defines defaults and shorthands for their
> everyday work. With shorthands, a user that only works with a single
> service could say "aurora job create", instead of needing to spell
> out the full jobkey and configuration file location; a user working with
> multiple datacenters could say "`aurora job create @east`" or
> "`aurora job create @west`" to select the correct jobkey.
>
> ## Proposal: Aurora Init Files
>
> To allow users to customize shorthands, we'll provide a
> builtin capability to allow users to provide a configuration
> file, from which their customizations will be loaded.
> Many applications use a simple pattern to solve similar problems.
> Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> don't specifically tell the tool where to find a configuration, it looks
> in the current directory or one of its parents to find a file named
> "Vagrantfile".
>
> We'd like to follow a similar pattern, and create an "AuroraInit" file.
> The aurora init file is found by searching the following locations, in
> order:
>
> * the contents of the "--init-file" parameter.
> * if the "--init-file" parameter is unspecified, then look in the
> current
> directory for a file named "AuroraInit".
> * if no "AuroraInit" file exists in the current directory, then look in
> the users
> home directory for an init file named ".AuroraInit".
>
>
> ### What goes into an init file?
>
> We should support the following kinds of things:
>
> 1. Universal defaults - user-defined default settings that will be
> applied to
> all commands. For example, if there is a default config file that
> should always
> be used if the user doesn't specify one, that would be a universal
> default.
> 2. Command specific defaults - users should be able to specify that they
> always want
> to use certain parameter settings for a specific command. For example,
> if they
> want to always use a default batch size of 10 for updates, but don't
> want to affect
> batch sizes for other commands like kill, they could use a command
> specific default.
> 3. Aliases - shorthand names for longer parameters. A user could
> specify shorthands
> "east" and "west" for full jobkeys in two different datacenters.
>
> ### Using aliases and defaults
>
> A default specifies a set of _bindings_. If a parameter is omitted
> from a command, and there's a binding for that parameter, it will be
> automatically inserted into the command as if the user had typed it.
>
> An alias is a short equivalent for a parameter. When a command line is
> provided by a user, aliases will be expanded inline. A user can
> specifically
> mark an alias for expansion by prefixing it with "@"; if an alias
> appears on the command-line surrounded by whitespace, it will be
> replaced even if it isn't marked with an "@".
>
>
> ### Defining Shorthands and Defaults
>
> The init file is, like aurora job configurations, a python source file
> using a Pystachio
> schema. The schema is loaded, and an `Init` object is retrieved from the
> top-level
> variable `init` in that file.
>
> The pystachio schema for init files is:
>
> class CommandDefaults(Struct)
> command = Required(String)
> defaults=Map(String, String)
>
> class Init(Struct):
> defaults=Map(String, String)
> command_defaults = Optional(CommandDefaults)
> aliases=Map(String, String)
>
>
> For example, an init file could contain the following:
>
> init=Init(
> defaults={
> "jobspec": "west/markcc/test/myservice",
> "config_file": "./src/aurora/myservice.aurora"
> },
> aliases={
> "east": "east/markcc/test/myservice",
> "c": "east",
> "me": "markchucarroll"
> },
> command_defaults(command="job update",
> defaults={"--batch-size": 10}
> ))
>
>
> With this configuration file, if the user ran "`aurora job create`" without
> any parameters,
> it would automatically be expanded to "`aurora job create
> west/markcc/test/myservice ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job create east`", the alias `east` would be
> expanded to "east/markcc/test/myservice", and the missing config file
> parameter would
> be instantiated using the default, to create a command-line:
> "`aurora job create east/markcc/test/myservice
> ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job create @c/@me/test/myservice`", the two aliases
> would
> be expanded, and the omitted configuration parameter would be added:
> "`aurora job create east/markchucarroll/test/myservice
> ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job update`", then the `jobspec` and `config_file`
> parameters would
> get inserted from the global defaults, and the "--batch-size=10" would be
> inserted from
> the command defaults for "aurora job update".
>