You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@aurora.apache.org by wf...@apache.org on 2015/01/06 22:50:01 UTC
incubator-aurora git commit: Begin dismantling client v1.
Repository: incubator-aurora
Updated Branches:
refs/heads/master 1c53b9481 -> 9f6ec4bd9
Begin dismantling client v1.
Bugs closed: AURORA-775
Reviewed at https://reviews.apache.org/r/28971/
Project: http://git-wip-us.apache.org/repos/asf/incubator-aurora/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-aurora/commit/9f6ec4bd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-aurora/tree/9f6ec4bd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-aurora/diff/9f6ec4bd
Branch: refs/heads/master
Commit: 9f6ec4bd93c2529f34811a127f8facd6de04eb55
Parents: 1c53b94
Author: Bill Farner <wf...@apache.org>
Authored: Tue Jan 6 13:30:13 2015 -0800
Committer: Bill Farner <wf...@apache.org>
Committed: Tue Jan 6 13:30:13 2015 -0800
----------------------------------------------------------------------
docs/client-commands.md | 205 ++--------
docs/client.md | 225 ++++++++++
docs/clientv2.md | 406 -------------------
docs/developing-aurora-client.md | 36 +-
examples/vagrant/aurorabuild.sh | 12 +-
examples/vagrant/test_tutorial.sh | 8 +-
src/main/python/apache/aurora/client/BUILD | 18 +-
src/main/python/apache/aurora/client/bin/BUILD | 8 -
src/main/python/apache/aurora/client/cli/BUILD | 3 +-
.../python/apache/aurora/client/cli/__init__.py | 2 +-
.../python/apache/aurora/client/cli/client.py | 2 +-
.../python/apache/aurora/client/cli/config.py | 2 +-
.../apache/aurora/client/cli/test_help.py | 2 +-
.../sh/org/apache/aurora/e2e/test_end_to_end.sh | 46 ++-
.../org/apache/aurora/e2e/test_end_to_end_v2.sh | 125 ------
15 files changed, 324 insertions(+), 776 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/docs/client-commands.md
----------------------------------------------------------------------
diff --git a/docs/client-commands.md b/docs/client-commands.md
index 6c929c1..75e6954 100644
--- a/docs/client-commands.md
+++ b/docs/client-commands.md
@@ -128,11 +128,11 @@ the machine executing Aurora commands.
Hooks can be associated with these Aurora Client commands.
- - `cancel_update`
- - `create`
- - `kill`
- - `restart`
- - `update`
+ - `job cancel-update`
+ - `job create`
+ - `job kill`
+ - `job restart`
+ - `job update`
The process for writing and activating them is complex enough
that we explain it in a devoted document, [Hooks for Aurora Client API](hooks.md).
@@ -145,28 +145,14 @@ renaming, updating, and restarting a basic Aurora Job.
### Creating and Running a Job
- aurora create <job key> <configuration file>
+ aurora job create <job key> <configuration file>
Creates and then runs a Job with the specified job key based on a `.aurora` configuration file.
The configuration file may also contain and activate hook definitions.
-`create` can take four named parameters:
-
-- `-E NAME=VALUE` Bind a Thermos mustache variable name to a
- value. Multiple flags specify multiple values. Defaults to `[]`.
-- ` -o, --open_browser` Open a browser window to the scheduler UI Job
- page after a job changing operation happens. When `False`, the Job
- URL prints on the console and the user has to copy/paste it
- manually. Defaults to `False`. Does not work when running in Vagrant.
-- ` -j, --json` If specified, configuration argument is read as a
- string in JSON format. Defaults to False.
-- ` --wait_until=STATE` Block the client until all the Tasks have
- transitioned into the requested state. Possible values are: `PENDING`,
- `RUNNING`, `FINISHED`. Default: `PENDING`
-
### Running a Command On a Running Job
- aurora run <job_key> <cmd>
+ aurora task run CLUSTER/ROLE/ENV/NAME[/INSTANCES] <cmd>
Runs a shell command on all machines currently hosting shards of a
single Job.
@@ -175,75 +161,42 @@ single Job.
commands; i.e. anything in the `{{mesos.*}}` and `{{thermos.*}}`
namespaces.
-`run` can take three named parameters:
-
-- `-t NUM_THREADS`, `--threads=NUM_THREADS `The number of threads to
- use, defaulting to `1`.
-- `--user=SSH_USER` ssh as this user instead of the given role value.
- Defaults to None.
-- `-e, --executor_sandbox` Run the command in the executor sandbox
- instead of the Task sandbox. Defaults to False.
-
### Killing a Job
- aurora kill <job key> <configuration file>
+ aurora job killall CLUSTER/ROLE/ENV/NAME
Kills all Tasks associated with the specified Job, blocking until all
-are terminated. Defaults to killing all shards in the Job.
+are terminated. Defaults to killing all instances in the Job.
The `<configuration file>` argument for `kill` is optional. Use it only
if it contains hook definitions and activations that affect the
kill command.
-`kill` can take two named parameters:
-
-- `-o, --open_browser` Open a browser window to the scheduler UI Job
- page after a job changing operation happens. When `False`, the Job
- URL prints on the console and the user has to copy/paste it
- manually. Defaults to `False`. Does not work when running in Vagrant.
-- `--shards=SHARDS` A list of shard ids to act on. Can either be a
- comma-separated list (e.g. 0,1,2) or a range (e.g. 0-2) or any
- combination of the two (e.g. 0-2,5,7-9). Defaults to acting on all
- shards.
-
### Updating a Job
- aurora update [--shards=ids] <job key> <configuration file>
- aurora cancel_update <job key> <configuration file>
+ aurora job update CLUSTER/ROLE/ENV/NAME[/INSTANCES] <configuration file>
+ aurora job cancel-update CLUSTER/ROLE/ENV/NAME
Given a running job, does a rolling update to reflect a new
configuration version. Only updates Tasks in the Job with a changed
-configuration. You can further restrict the operated on Tasks by
-using `--shards` and specifying a comma-separated list of job shard ids.
+configuration. You can further restrict the operated on Tasks by specifying
+specific instances that should be updated.
-You may want to run `aurora diff` beforehand to validate which Tasks
+You may want to run `aurora job diff` beforehand to validate which Tasks
have different configurations.
Updating jobs are locked to be sure the update finishes without
disruption. If the update abnormally terminates, the lock may stay
around and cause failure of subsequent update attempts.
- `aurora cancel_update `unlocks the Job specified by
-its `job_key` argument. Be sure you don't issue `cancel_update` when
+ `aurora job cancel-update `unlocks the Job specified by
+its `job_key` argument. Be sure you don't issue `job cancel-update` when
another user is working with the specified Job.
-The `<configuration file>` argument for `cancel_update` is optional. Use
+The `<configuration file>` argument for `job cancel-update` is optional. Use
it only if it contains hook definitions and activations that affect the
`cancel_update` command. The `<configuration file>` argument for
`update` is required, but in addition to a new configuration it can be
-used to define and activate hooks for `update`.
-
-`update` can take four named parameters:
-
-- `--shards=SHARDS` A list of shard ids to update. Can either be a
- comma-separated list (e.g. 0,1,2) or a range (e.g. 0-2) or any
- combination of the two (e.g. 0-2,5,7-9). If not set, all shards are
- acted on. Defaults to None.
-- `-E NAME=VALUE` Binds a Thermos mustache variable name to a value.
- Use multiple flags to specify multiple values. Defaults to `[]`.
-- `-j, --json` If specified, configuration is read in JSON format.
- Defaults to `False`.
-- `--updater_health_check_interval_seconds=HEALTH_CHECK_INTERVAL_SECONDS`
- Time interval between subsequent shard status checks. Defaults to `3`.
+used to define and activate hooks for `job update`.
#### Asynchronous job updates (beta)
@@ -253,12 +206,12 @@ update progress and job update history in the browser.
There are several sub-commands to manage job updates:
- aurora2 beta-update start <job key> <configuration file
- aurora2 beta-update status <job key>
- aurora2 beta-update pause <job key>
- aurora2 beta-update resume <job key>
- aurora2 beta-update abort <job key>
- aurora2 beta-update list <cluster>
+ aurora beta-update start <job key> <configuration file>
+ aurora beta-update status <job key>
+ aurora beta-update pause <job key>
+ aurora beta-update resume <job key>
+ aurora beta-update abort <job key>
+ aurora beta-update list <cluster>
When you `start` a job update, the command will return once it has sent the
instructions to the scheduler. At that point, you may view detailed
@@ -287,12 +240,12 @@ to renaming suitable for production services is:
2. Check that only these naming components have changed
with `aurora diff`.
- aurora diff <job_key> <job_configuration>
+ aurora job diff CLUSTER/ROLE/ENV/NAME <job_configuration>
3. Create the (identical) job at the new key. You may need to request a
temporary quota increase.
- aurora create <new_job_key> <job_configuration>
+ aurora job create CLUSTER/ROLE/ENV/NEW_NAME <job_configuration>
4. Migrate all clients over to the new job key. Update all links and
dashboards. Ensure that both job keys run identical versions of the
@@ -300,7 +253,7 @@ to renaming suitable for production services is:
5. After verifying that all clients have successfully moved over, kill
the old job.
- aurora kill <old_job_key>
+ aurora job killall CLUSTER/ROLE/ENV/NAME
6. If you received a temporary quota increase, be sure to let the
powers that be know you no longer need the additional capacity.
@@ -309,52 +262,19 @@ to renaming suitable for production services is:
`restart` restarts all of a job key identified Job's shards:
- aurora restart <job_key> <configuration file>
+ aurora job restart CLUSTER/ROLE/ENV/NAME[/INSTANCES]
Restarts are controlled on the client side, so aborting
-the `restart` command halts the restart operation.
+the `job restart` command halts the restart operation.
-`restart` does a rolling restart. You almost always want to do this, but
-not if all shards of a service are misbehaving and are
-completely dysfunctional. To not do a rolling restart, use
-the `-shards` option described below.
-
-**Note**: `restart` only applies its command line arguments and does not
+**Note**: `job restart` only applies its command line arguments and does not
use or is affected by `update.config`. Restarting
does ***not*** involve a configuration change. To update the
configuration, use `update.config`.
-The `<configuration file>` argument for restart is optional. Use it only
+The `--config` argument for restart is optional. Use it only
if it contains hook definitions and activations that affect the
-`restart` command.
-
-In addition to the required job key argument, there are eight
-`restart` specific optional arguments:
-
-- `--updater_health_check_interval_seconds`: Defaults to `3`, the time
- interval between subsequent shard status checks.
-- `--shards=SHARDS`: Defaults to None, which restarts all shards.
- Otherwise, only the specified-by-id shards restart. They can be
- comma-separated `(0, 8, 9)`, a range `(3-5)` or a
- combination `(0, 3-5, 8, 9-11)`.
-- `--batch_size`: Defaults to `1`, the number of shards to be started
- in one iteration. So, for example, for value 3, it tries to restart
- the first three shards specified by `--shards` simultaneously, then
- the next three, and so on.
-- `--max_per_shard_failures=MAX_PER_SHARD_FAILURES`: Defaults to `0`,
- the maximum number of restarts per shard during restart. When
- exceeded, it increments the total failure count.
-- `--max_total_failures=MAX_TOTAL_FAILURES`: Defaults to `0`, the
- maximum total number of shard failures tolerated during restart.
-- `-o, --open_browser` Open a browser window to the scheduler UI Job
- page after a job changing operation happens. When `False`, the Job
- url prints on the console and the user has to copy/paste it
- manually. Defaults to `False`. Does not work when running in Vagrant.
-- `--restart_threshold`: Defaults to `60`, the maximum number of
- seconds before a shard must move into the `RUNNING` state before
- it's considered a failure.
-- `--watch_secs`: Defaults to `45`, the minimum number of seconds a
- shard must remain in `RUNNING` state before considered a success.
+`job restart` command.
Cron Jobs
---------
@@ -367,20 +287,12 @@ if you do.
Comparing Jobs
--------------
- aurora diff <job_key> config
+ aurora job diff CLUSTER/ROLE/ENV/NAME <job configuration>
Compares a job configuration against a running job. By default the diff
is determined using `diff`, though you may choose an alternate
diff program by specifying the `DIFF_VIEWER` environment variable.
-There are two named parameters:
-
-- `-E NAME=VALUE` Bind a Thermos mustache variable name to a
- value. Multiple flags may be used to specify multiple values.
- Defaults to `[]`.
-- `-j, --json` Read the configuration argument in JSON format.
- Defaults to `False`.
-
Viewing/Examining Jobs
----------------------
@@ -389,44 +301,20 @@ how to view and examine Jobs.
### Listing Jobs
- aurora list_jobs
- Usage: `aurora list_jobs cluster/role
+ aurora config list <job configuration>
Lists all Jobs registered with the Aurora scheduler in the named cluster for the named role.
-It has two named parameters:
-
-- `--pretty`: Displays job information in prettyprinted format.
- Defaults to `False`.
-- `-c`, `--show-cron`: Shows cron schedule for jobs. Defaults to
- `False`. Do not use, as it's not yet implemented.
-
### Inspecting a Job
- aurora inspect <job_key> config
+ aurora job inspect CLUSTER/ROLE/ENV/NAME <job configuration>
`inspect` verifies that its specified job can be parsed from a
-configuration file, and displays the parsed configuration. It has four
-named parameters:
-
-- `--local`: Inspect the configuration that the `spawn` command would
- create, defaulting to `False`.
-- `--raw`: Shows the raw configuration. Defaults to `False`.
-- `-j`, `--json`: If specified, configuration is read in JSON format.
- Defaults to `False`.
-- `-E NAME=VALUE`: Bind a Thermos Mustache variable name to a value.
- You can use multiple flags to specify multiple values. Defaults
- to `[]`
-
-### Versions
-
- aurora version
-
-Lists client build information and what Aurora API version it supports.
+configuration file, and displays the parsed configuration.
### Checking Your Quota
- aurora get_quota --cluster=CLUSTER role
+ aurora quota get CLUSTER/ROLE
Prints the production quota allocated to the role's value at the given
cluster.
@@ -451,7 +339,7 @@ production jobs and user accounts for test or development jobs.
### Getting Job Status
- aurora status <job_key>
+ aurora job status <job_key>
Returns the status of recent tasks associated with the
`job_key` specified Job in its supplied cluster. Typically this includes
@@ -464,7 +352,7 @@ Use the Job's web UI scheduler URL or the `aurora status` command to find out on
machines individual tasks are scheduled. You can open the web UI via the
`open` command line command if invoked from your machine:
- aurora open [<cluster>[/<role>[/<env>/<job_name>]]]
+ aurora job open [<cluster>[/<role>[/<env>/<job_name>]]]
If only the cluster is specified, it goes directly to that cluster's
scheduler main page. If the role is specified, it goes to the top-level
@@ -473,25 +361,16 @@ page where you can inspect individual tasks.
### SSHing to a Specific Task Machine
- aurora ssh <job_key> <shard number>
+ aurora task ssh <job_key> <shard number>
You can have the Aurora client ssh directly to the machine that has been
assigned a particular Job/shard number. This may be useful for quickly
diagnosing issues such as performance issues or abnormal behavior on a
particular machine.
-It can take three named parameters:
-
-- `-e`, `--executor_sandbox`: Run `ssh` in the executor sandbox
- instead of the task sandbox. Defaults to `False`.
-- `--user=SSH_USER`: `ssh` as the given user instead of as the role in
- the `job_key` argument. Defaults to none.
-- `-L PORT:NAME`: Add tunnel from local port `PORT` to the remote
- named port `NAME`. Defaults to `[]`.
-
### Templating Command Arguments
- aurora run [-e] [-t THREADS] <job_key> -- <<command-line>>
+ aurora task run [-e] [-t THREADS] <job_key> -- <<command-line>>
Given a job specification, run the supplied command on all hosts and
return the output. You may use the standard Mustache templating rules:
@@ -506,7 +385,7 @@ return the output. You may use the standard Mustache templating rules:
For example, the following type of pattern can be a powerful diagnostic
tool:
- aurora run -t5 cluster1/tyg/devel/seizure -- \
+ aurora task run -t5 cluster1/tyg/devel/seizure -- \
'curl -s -m1 localhost:{{thermos.ports[http]}}/vars | grep uptime'
By default, the command runs in the Task's sandbox. The `-e` option can
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/docs/client.md
----------------------------------------------------------------------
diff --git a/docs/client.md b/docs/client.md
new file mode 100644
index 0000000..3ec39b4
--- /dev/null
+++ b/docs/client.md
@@ -0,0 +1,225 @@
+Aurora Client
+=============
+
+Goals
+-------
+
+* A command line tool for interacting with Aurora that is easy for
+ users to understand.
+* A noun/verb command model.
+* A modular source-code architecture.
+* Non-disruptive transition for users.
+
+Design
+------
+
+### Interface
+
+In this section, we'll walk through the types of objects that the
+client can manipulate, and the operations that need to be provided for
+each object. These form the primary interface that engineers will use
+to interact with Aurora.
+
+In the command-line, each of the object types will have an Aurora
+subcommand. The commands to manipulate the object type will follow the
+type.
+
+### The Job Noun
+
+A job is a configured program ready to run in Aurora. A job is,
+conceptually, a task factory: when a job is submitted to the Aurora
+scheduler, it creates a collection of tasks. The job contains a
+complete description of everything it needs to create a collection of
+tasks. (Note that this subsumes "service" commands. A service is just
+a task whose configuration sets the is_service flag, so we don't have
+separate commands for working with services.) Jobs are specified using
+`cluster/role/env/name` jobkey syntax.
+
+* `aurora job create *jobkey* *config*`: submits a job to a cluster, launching
+ the task(s) specified by the job config.
+* `aurora job status *jobkey*`: query job status. Prints information about the
+ job, whether it's running, etc., to standard out. If jobkey includes globs,
+ it should list all jobs that match the glob
+* `aurora job kill *jobkey*/*instanceids*`: kill/stop some of a jobs instances.
+ This stops a job' tasks; if the job has service tasks, they'll be disabled,
+ so that they won't restart.
+* `aurora job killall *jobkey*`: kill all of the instances of a job. This
+ is distinct from the *kill* command as a safety measure: omitting the
+ instances from a kill command shouldn't result in destroying the entire job.
+* `aurora job restart *jobkey*`: conceptually, this will kill a job, and then
+ launch it again. If the job does not exist, then fail with an error
+ message. In fact, the underlying implementation does the
+ kill/relaunch on a rolling basis - so it's not an immediate kill of
+ all shards/instances, followed by a delay as all instances relaunch,
+ but rather a controlled gradual process.
+* `aurora job list *jobkey*`: list all jobs that match the jobkey spec that are
+ registered with the scheduler. This will include both jobs that are
+ currently running, and jobs that are scheduled to run at a later
+ time. The job key can be partial: if it specifies cluster, all jobs
+ on the cluster will be listed; cluster/role, all jobs running on the cluster
+ under the role will be listed, etc.
+
+The Schedule Noun (Cron)
+--------------------------
+
+Cron is a scheduler adjunct that periodically runs a job on a
+schedule. The cron commands all manipulate cron schedule entries. The
+schedules are specified as a part of the job configuration.
+
+* `aurora cron schedule jobkey config`: schedule a job to run by cron. If a cron
+ job already exists replaces its template with a new one.
+* `aurora cron deschedule jobkey`: removes a jobs entry from the cron schedule.
+* `aurora cron status jobkey`: query for a scheduled job's status.
+
+The Quota Noun
+---------------
+
+A quota is a data object maintained by the scheduler that specifies the maximum
+resources that may be consumed by jobs owned by a particular role. In the future,
+we may add new quota types. At some point, we'll also probably add an administrators
+command to set quotas.
+
+* `aurora quota get *cluster/role*`
+
+
+### Command Structure and Options Processing
+
+The implementation will follow closely on Pants goals. Pants goals use
+a static registration system to add new subcommands. In pants, each
+goal command is an implementation of a command interface, and provides
+implementations of methods to register options and parameters, and to
+actually execute the command. In this design, commands are modular and
+easy to implement, debug, and combine in different ways.
+
+For the Aurora client, we plan to use a two-level variation of the
+basic concept from pants. At the top-level we will have nouns. A noun
+will define some common command-line parameters required by all of its
+verbs, and will provide a registration hook for attaching verbs. Nouns
+will be implemented as a subclass of a basic Noun type.
+
+Each verb will, similarly, be implemented as a subclass of Verb. Verbs
+will be able to specify command-line options and parameters.
+
+Both `Noun` and `Verb` will be subclasses of a common base-class `AuroraCommand`:
+
+ class AuroraCommand(object):
+ def get_options(self):
+ """Gets the set of command-line options objects for this command.
+ The result is a list of CommandOption objects.
+ """
+ pass
+
+ @property
+ def help(self):
+ """Returns the help message for this command"""
+
+ @property
+ def usage(self):
+ """Returns a short usage description of the command"""
+
+ @property
+ def name(self):
+ """Returns the command name"""
+
+
+A command-line tool will be implemented as an instance of a `CommandLine`:
+
+ class CommandLine(object):
+ """The top-level object implementing a command-line application."""
+
+ @property
+ def name(self):
+ """Returns the name of this command-line tool"""
+
+ def print_out(self, str):
+ print(str)
+
+ def print_err(self, str):
+ print(str, file=sys.stderr)
+
+ def register_noun(self, noun):
+ """Adds a noun to the application"""
+
+ def register_plugin(self, plugin):
+ """Adds a configuration plugin to the system"""
+
+
+Nouns are registered into a command-line using the `register_noun`
+method. They are weakly coupled to the application, making it easy to
+use a single noun in several different command-line tools. Nouns allow
+the registration of verbs using the `register_verb` method.
+
+When commands execute, they're given an instance of a *context object*.
+The context object must be an instance of a subclass of `AuroraCommandContext`.
+Options, parameters, and IO are all accessed using the context object. The context
+is created dynamically by the noun object owning the verb being executed. Developers
+are strongly encouraged to implement custom contexts for their nouns, and move functionality
+shared by the noun's verbs into the context object. The context interface is:
+
+ class Context(object):
+ class Error(Exception): pass
+
+ class ArgumentException(Error): pass
+
+ class CommandError(Error):
+
+ @classmethod
+ def exit(cls, code, msg):
+ """Exit the application with an error message"""
+ raise cls.CommandError(code, msg)
+
+ def print_out(self, msg, indent=0):
+ """Prints a message to standard out, with an indent"""
+
+ def print_err(self, msg, indent=0):
+ """Prints a message to standard err, with an indent"""
+
+
+In addition to nouns and verbs, there's one more kind of registerable
+component, called a *configuration plugin*. These objects add a set of
+command-line options that can be passed to *all* of the commands
+implemented in the tool. Before the command is executed, the
+configuration plugin will be invoked, and will process its
+command-line arguments. This is useful for general configuration
+changes, like establish a secure tunnel to talk to machines in a
+datacenter. (A useful way to think of a plugin is as something like an
+aspect that can be woven in to aurora to provide environment-specific
+configuration.) A configuration plugin is implemented as an instance
+of class `ConfigurationPlugin`, and registered with the
+`register_plugin` method of the `CommandLine` object. The interface of
+a plugin is:
+
+ class ConfigurationPlugin(object):
+ """A component that can be plugged in to a command-line."""
+
+ @abstractmethod
+ def get_options(self):
+ """Return the set of options processed by this plugin"""
+
+ @abstractmethod
+ def execute(self, context):
+ """Run the context/command line initialization code for this plugin."""
+
+
+### Command Execution
+
+The options process and command execution is built as a facade over Python's
+standard argparse. All of the actual argument processing is done by the
+argparse library.
+
+Once the options are processed, the framework will start to execute the command.
+Command execution consists of:
+
+# Create a context object. The framework will use the argparse options to identify
+ which noun is being invoked, and will call that noun's `create_context` method.
+ The argparse options object will be stored in the context.
+# Execute any configuration plugins. Before any command is invoked, the framework
+ will first iterate over all of the registered configuration plugins. For each
+ plugin, it will invoke the `execute` method.
+# The noun will use the context to find out what verb is being invoked, and it will
+ then call that verb's `execute` method.
+# The command will exit. Its return code will be whatever was returned by the verb's
+ `execute` method.
+
+Commands are expected to return a code from a list of standard exit codes,
+which can be found in `src/main/python/apache/aurora/client/cli/__init__.py`.
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/docs/clientv2.md
----------------------------------------------------------------------
diff --git a/docs/clientv2.md b/docs/clientv2.md
deleted file mode 100644
index 471d80c..0000000
--- a/docs/clientv2.md
+++ /dev/null
@@ -1,406 +0,0 @@
-Aurora Client v2
-=================
-
-Overview
------------
-
-Our goal is to replace the current Aurora command-line client. The
-current client suffers from an early monolithic structure, and a long
-development history of rapid unplanned evolution.
-
-In addition to its internal problems, the current Aurora client is
-confusing for users. There are several different kinds of objects
-manipulated by the Aurora command line, and the difference between
-them is often not clear. (What's the difference between a job and a
-configuration?) For each type of object, there are different commands,
-and it's hard to remember which command should be used for which kind
-of object.
-
-Instead of continuing to let the Aurora client develop and evolve
-randomly, it's time to take a principled look at the Aurora command
-line, and figure out how to make our command line processing make
-sense. At the same time, the code needs to be cleaned up, and divided
-into small comprehensible units based on a plugin architecture.
-
-Doing this now will give us a more intuitive, consistent, and easy to
-use client, as well as a sound platform for future development.
-
-Goals
--------
-
-* A command line tool for interacting with Aurora that is easy for
- users to understand.
-* A noun/verb command model.
-* A modular source-code architecture.
-* Non-disruptive transition for users.
-
-Non-Goals
-----------
-
-* The most important non-goal is that we're not trying to redesign the
- Aurora scheduler, the Aurora executor, or any of the peripheral tools
- that the Aurora command line interacts with; we only want to create a
- better command line client.
-* We do not want to change thermos, mesos, hadoop, etc.
-* We do not want to create new objects that users will work with to
- interact with Mesos or Aurora.
-* We do not want to change Aurora job configuration files or file formats.
-* We do not want to change the Aurora API.
-* We don't want to boil the ocean: there are many things that we could
- include in the scope of this project, but we don't want to be
- distracted by re-implementing all of twitter.commons in order to
- create a perfect Aurora client.
-
-
-Background
------------
-
-Aurora is a system that's used to run and manage services and
-service-like jobs running in a datacenter. Aurora takes care of
-allocating resources in order to schedule and run jobs without
-requiring teams to manage dedicated hardware. The heart of Aurora is
-called the scheduler, and is responsible for finding and assigning
-resources to tasks.
-
-The Aurora scheduler provides a thrift API. The scheduler API is
-low-level and difficult to interact with. Users do not interact
-directly with the Aurora API; instead, they use a command-line tool,
-which provides a collection of easy-to-use commands. This command-line
-tool, in turn, talks to the scheduler API to launch and manage jobs in
-datacenter clusters. The command-line tool is called the Aurora
-client.
-
-The current implementation of the Aurora client is haphazard,
-and really needs to be cleaned up:
-
-- The code is monolithic and hard to maintain. It's implemented using
- `twitter.common.app`, which assumes that all of the command code lives
- in a single source file. To work around this, and allow some
- subdivision, it uses a hack of `twitter.common.app` to force
- registration of commands from multiple modules. It's hard to
- understand, and hard to modify.
-- The current code is very difficult to test. Because of the way it's
- built, there is no consistent way of passing key application data
- around. As a result, each unit test of client operations needs a
- difficult-to-assemble custom setup of mock objects.
-- The current code handles errors poorly, and it is difficult to
- fix. Many common errors produce unacceptable results. For example,
- issuing an unknown command generates an error message "main takes 0
- parameters but received 1"; passing an invalid parameter to other
- commands frequently produces a stack trace.
-- The current command line is confusing for users. There are several
- different kinds of objects manipulated by the Aurora command line,
- and the difference between them is often not entirely clear. (What's
- the difference between a job and a configuration?)
- For each type of object, there are different
- commands, and it's frequently not clear just which command should be
- used for which object.
-
-
-Instead of continuing to let it develop and evolve randomly, it's time
-to take a principled look at the Aurora command line, and figure out
-how to make command line processing make sense. At the same time, the
-code needs to be cleaned up, and divided into small comprehensible
-units based on a plugin architecture.
-
-Requirements
--------------
-
-Aurora is aimed at engineers who run jobs and services in a
-datacenter. As a result, the requirements for the aurora client are
-all engineering focused:
-
-* __Consistency__: commands should follow a consistent structure, so that
- users can apply knowledge and intuition gained from working with
- some aurora commands to new commands. This means that when commands
- can re-use the same options, they should; that objects should be
- referred to by consistent syntax throughout the tool.
-* __Helpfulness__: commands should be structured so that the system can
- generate helpful error messages. If a user just runs "aurora", they
- should get a basic usage message. If they try to run an invalid
- command, they should get a message that the command is invalid, not
- a stack dump or "command main() takes 0 parameters but received
- 2". Commands should not generate extraneous output that obscures the
- key facts that the user needs to know, and the default behavior of
- commands should not generate outputs that will be routinely ignored
- by users.
-* __Extensibility__: it should be easy to plug in new commands,
- including custom commands, to adapt the Aurora client to new
- environments.
-* __Script-friendly command output__: every command should at least include
- an option that generates output that's script-friendly. Scripts should be
- able to work with command-output without needing to do screen scraping.
-* __Scalability__: the tools should be usable for any foreseeable size
- of Aurora datacenters and machine clusters.
-
-Design Overview
------------------
-
-The Aurora client will be reimplemented using a noun-verb model,
-similar to the cmdlet model used by Monad/Windows Powershell. Users
-will work by providing a noun for the type of object being operated
-on, and a verb for the specific operation being performed on the
-object, followed by parameters. For example, to create a job, the user
-would execute: "`aurora job create smfd/mchucarroll/devel/jobname
-job.aurora`". The noun is `job` and the verb is `create`.
-
-The client will be implemented following that noun-verb
-convention. Each noun will be a separate component, which can be
-registered into the command-line framework. Each verb will be
-implemented by a class that registers with the appropriate noun. Nouns
-and verbs will each provide methods that add their command line
-options and parameters to the options parser, using the Python
-argparse library.
-
-Detailed Design
------------------
-
-### Interface
-
-In this section, we'll walk through the types of objects that the
-client can manipulate, and the operations that need to be provided for
-each object. These form the primary interface that engineers will use
-to interact with Aurora.
-
-In the command-line, each of the object types will have an Aurora
-subcommand. The commands to manipulate the object type will follow the
-type. For example, here are several commands in the old syntax
-contrasted against the new noun/verb syntax.
-
-* Get quota for a role:
- * Noun/Verb syntax: `aurora quota get west/www-data`
- * Old syntax: `aurora get_quota --cluster=smf1 www-data`
-* Create job:
- * Noun/Verb syntax: `aurora job create west/www-data/test/job job.aurora`
- * Old syntax: `aurora create west/www-data/test/job job.aurora`
-* Schedule a job to run at a specific interval:
- * Noun/verb: `aurora cron schedule east/www-data/test/job job.aurora`
- * Old: `aurora create east/www-data/test/job job.aurora`
-
-As you can see in these examples, the new syntax is more consistent:
-you always specify the cluster where a command executes as part of an
-identifier, where in the old syntax, it was sometimes part of the
-jobkey and sometimes specified with a "--cluster" option.
-
-The new syntax is also more clear and explicit: even without knowing
-much about Aurora, it's clear what objects each command is acting on,
-where in the old syntax, commands like "create" are unclear.
-
-### The Job Noun
-
-A job is a configured program ready to run in Aurora. A job is,
-conceptually, a task factory: when a job is submitted to the Aurora
-scheduler, it creates a collection of tasks. The job contains a
-complete description of everything it needs to create a collection of
-tasks. (Note that this subsumes "service" commands. A service is just
-a task whose configuration sets the is_service flag, so we don't have
-separate commands for working with services.) Jobs are specified using
-`cluster/role/env/name` jobkey syntax.
-
-* `aurora job create *jobkey* *config*`: submits a job to a cluster, launching the task(s) specified by the job config.
-* `aurora job status *jobkey*`: query job status. Prints information about the job,
- whether it's running, etc., to standard out. If jobkey includes
- globs, it should list all jobs that match the glob
-* `aurora job kill *jobkey*/*instanceids*`: kill/stop some of a jobs instances. This stops a job' tasks; if the job
- has service tasks, they'll be disabled, so that they won't restart.
-* `aurora job killall *jobkey*`: kill all of the instances of a job. This
- is distinct from the *kill* command as a safety measure: omitting the
- instances from a kill command shouldn't result in destroying the entire job.
-* `aurora job restart *jobkey*`: conceptually, this will kill a job, and then
- launch it again. If the job does not exist, then fail with an error
- message. In fact, the underlying implementation does the
- kill/relaunch on a rolling basis - so it's not an immediate kill of
- all shards/instances, followed by a delay as all instances relaunch,
- but rather a controlled gradual process.
-* `aurora job list *jobkey*`: list all jobs that match the jobkey spec that are
- registered with the scheduler. This will include both jobs that are
- currently running, and jobs that are scheduled to run at a later
- time. The job key can be partial: if it specifies cluster, all jobs
- on the cluster will be listed; cluster/role, all jobs running on the cluster under the role will be listed, etc.
-
-The Schedule Noun (Cron)
---------------------------
-
-Note (3/21/2014): The "cron" noun is _not_ implemented yet.
-
-Cron is a scheduler adjunct that periodically runs a job on a
-schedule. The cron commands all manipulate cron schedule entries. The
-schedules are specified as a part of the job configuration.
-
-* `aurora cron schedule jobkey config`: schedule a job to run by cron. If a cron job already exists
-replaces its template with a new one.
-* `aurora cron deschedule jobkey`: removes a jobs entry from the cron schedule.
-* `aurora cron status jobkey`: query for a scheduled job's status.
-
-The Quota Noun
----------------
-
-A quota is a data object maintained by the scheduler that specifies the maximum
-resources that may be consumed by jobs owned by a particular role. In the future,
-we may add new quota types. At some point, we'll also probably add an administrators
-command to set quotas.
-
-* `aurora quota get *cluster/role*`
-
-
-Implementation
----------------
-
-The current command line is monolithic. Every command on an Aurora
-object is a top-level command in the Aurora client. In the
-restructured command line, each of the primary object types
-manipulated by Aurora should have its own sub-command.
-
-* Advantages of this approach:
- * Easier to detangle the command-line processing. The top-level
- command-processing will be a small set of subcommand
- processors. Option processing for each subcommand can be offloaded
- to a separate module.
- * The aurora top-level help command will be much more
- comprehensible. Instead of giving a huge list of every possible
- command, it will present the list of top-level object types, and
- then users can request help on the commands for a specific type
- of object.
- * The sub-commands can be separated into distinct command-line
- tools when appropriate.
-
-### Command Structure and Options Processing
-
-The implementation will follow closely on Pants goals. Pants goals use
-a static registration system to add new subcommands. In pants, each
-goal command is an implementation of a command interface, and provides
-implementations of methods to register options and parameters, and to
-actually execute the command. In this design, commands are modular and
-easy to implement, debug, and combine in different ways.
-
-For the Aurora client, we plan to use a two-level variation of the
-basic concept from pants. At the top-level we will have nouns. A noun
-will define some common command-line parameters required by all of its
-verbs, and will provide a registration hook for attaching verbs. Nouns
-will be implemented as a subclass of a basic Noun type.
-
-Each verb will, similarly, be implemented as a subclass of Verb. Verbs
-will be able to specify command-line options and parameters.
-
-Both `Noun` and `Verb` will be subclasses of a common base-class `AuroraCommand`:
-
- class AuroraCommand(object):
- def get_options(self):
- """Gets the set of command-line options objects for this command.
- The result is a list of CommandOption objects.
- """
- pass
-
- @property
- def help(self):
- """Returns the help message for this command"""
-
- @property
- def usage(self):
- """Returns a short usage description of the command"""
-
- @property
- def name(self):
- """Returns the command name"""
-
-
-A command-line tool will be implemented as an instance of a `CommandLine`:
-
- class CommandLine(object):
- """The top-level object implementing a command-line application."""
-
- @property
- def name(self):
- """Returns the name of this command-line tool"""
-
- def print_out(self, str):
- print(str)
-
- def print_err(self, str):
- print(str, file=sys.stderr)
-
- def register_noun(self, noun):
- """Adds a noun to the application"""
-
- def register_plugin(self, plugin):
- """Adds a configuration plugin to the system"""
-
-
-Nouns are registered into a command-line using the `register_noun`
-method. They are weakly coupled to the application, making it easy to
-use a single noun in several different command-line tools. Nouns allow
-the registration of verbs using the `register_verb` method.
-
-When commands execute, they're given an instance of a *context object*.
-The context object must be an instance of a subclass of `AuroraCommandContext`.
-Options, parameters, and IO are all accessed using the context object. The context
-is created dynamically by the noun object owning the verb being executed. Developers
-are strongly encouraged to implement custom contexts for their nouns, and move functionality
-shared by the noun's verbs into the context object. The context interface is:
-
- class Context(object):
- class Error(Exception): pass
-
- class ArgumentException(Error): pass
-
- class CommandError(Error):
-
- @classmethod
- def exit(cls, code, msg):
- """Exit the application with an error message"""
- raise cls.CommandError(code, msg)
-
- def print_out(self, msg, indent=0):
- """Prints a message to standard out, with an indent"""
-
- def print_err(self, msg, indent=0):
- """Prints a message to standard err, with an indent"""
-
-
-In addition to nouns and verbs, there's one more kind of registerable
-component, called a *configuration plugin*. These objects add a set of
-command-line options that can be passed to *all* of the commands
-implemented in the tool. Before the command is executed, the
-configuration plugin will be invoked, and will process its
-command-line arguments. This is useful for general configuration
-changes, like establish a secure tunnel to talk to machines in a
-datacenter. (A useful way to think of a plugin is as something like an
-aspect that can be woven in to aurora to provide environment-specific
-configuration.) A configuration plugin is implemented as an instance
-of class `ConfigurationPlugin`, and registered with the
-`register_plugin` method of the `CommandLine` object. The interface of
-a plugin is:
-
- class ConfigurationPlugin(object):
- """A component that can be plugged in to a command-line."""
-
- @abstractmethod
- def get_options(self):
- """Return the set of options processed by this plugin"""
-
- @abstractmethod
- def execute(self, context):
- """Run the context/command line initialization code for this plugin."""
-
-
-### Command Execution
-
-The options process and command execution is built as a facade over Python's
-standard argparse. All of the actual argument processing is done by the
-argparse library.
-
-Once the options are processed, the framework will start to execute the command. Command execution consists of:
-
-# Create a context object. The framework will use the argparse options to identify
- which noun is being invoked, and will call that noun's `create_context` method.
- The argparse options object will be stored in the context.
-# Execute any configuration plugins. Before any command is invoked, the framework
- will first iterate over all of the registered configuration plugins. For each
- plugin, it will invoke the `execute` method.
-# The noun will use the context to find out what verb is being invoked, and it will
- then call that verb's `execute` method.
-# The command will exit. Its return code will be whatever was returned by the verb's
- `execute` method.
-
-Commands are expected to return a code from a list of standard exit codes,
-which can be found in `src/main/python/apache/aurora/client/cli/__init__.py`.
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/docs/developing-aurora-client.md
----------------------------------------------------------------------
diff --git a/docs/developing-aurora-client.md b/docs/developing-aurora-client.md
index bf82d05..83b843e 100644
--- a/docs/developing-aurora-client.md
+++ b/docs/developing-aurora-client.md
@@ -10,12 +10,12 @@ pants [here](http://pantsbuild.github.io/python-readme.html).
To build the client executable, run the following in a command-shell:
- $ ./pants src/main/python/apache/aurora/client/cli:aurora2
+ $ ./pants src/main/python/apache/aurora/client/cli:aurora
-This will produce a python executable _pex_ file in `dist/aurora2.pex`. Pex files
+This will produce a python executable _pex_ file in `dist/aurora.pex`. Pex files
are fully self-contained executables: just copy the pex file into your path, and you'll be able to run it. For example, for a typical installation:
- $ cp dist/aurora2.pex /usr/local/bin/aurora
+ $ cp dist/aurora.pex /usr/local/bin/aurora
To run all of the client tests:
@@ -31,31 +31,13 @@ contents of this file can be found in the
how the client locates this file can be found in the
[Client Commands](client-commands.md#cluster-configuration) documentation.
-Client Versions
-===============
-
-There are currently two versions of the aurora client, imaginatively known as v1 and v2. All new development is done entirely in v2, but we continue to support and fix bugs in v1, until we get to the point where v2 is feature-complete and tested, and aurora users have had some time at adapt and switch their processes to use v2.
-
-Both versions are built on the same underlying API code.
-
-Client v1 was implemented using twitter.common.app. The command-line processing code for v1 can be found in `src/main/python/apache/aurora/client/commands` and
-`src/main/python/apache/aurora/client/bin`.
-
-Client v2 was implemented using its own noun/verb framework. The client v2 code can be found in `src/main/python/apache/aurora/client/cli`, and the noun/verb framework can be
-found in the `__init__.py` file in that directory.
-
-
Building and Testing the Client
===============================
Building and testing the client code are both done using Pants. The relevant targets to know about are:
- * Build a client v2 executable: `./pants src/main/python/apache/aurora/client/cli:aurora2`
- * Test client v2 code: `./pants ./pants src/test/python/apache/aurora/client/cli:all`
- * Build a client v1 executable: `./pants src/main/python/apache/aurora/client/bin:aurora_client`
- * Test client v1 code: `./pants src/main/python/apache/aurora/client/commands:all`
- * Test all client code: `./pants src/main/python/apache/aurora/client:all`
-
+ * Build a client executable: `./pants src/main/python/apache/aurora/client/cli:aurora`
+ * Test client code: `./pants ./pants src/test/python/apache/aurora/client/cli:all`
Overview of the Client Architecture
===================================
@@ -79,13 +61,11 @@ The client is built on a stacked architecture:
- monitoring the new version to ensure that the update succeeded.
3. On top of the API, we have the command-line client itself. The core client, at this level,
consists of the interface to the command-line which the user will use to interact with aurora.
- The client v2 code is found in `src/python/apache/aurora/client/cli`. In the `cli` directory,
+ The client code is found in `src/python/apache/aurora/client/cli`. In the `cli` directory,
the rough structure is as follows:
- * `__init__.py` contains the noun/verb command-line processing framework used by client v2.
+ * `__init__.py` contains the noun/verb command-line processing framework used by client.
* `jobs.py` contains the implementation of the core `job` noun, and all of its operations.
- * `bridge.py` contains the implementation of a component that allows us to ship a
- combined client that runs both v1 and v2 client commands during the transition period.
- * `client.py` contains the code that binds the client v2 nouns and verbs into an executable.
+ * `client.py` contains the code that binds the client nouns and verbs into an executable.
Running/Debugging the Client
============================
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/examples/vagrant/aurorabuild.sh
----------------------------------------------------------------------
diff --git a/examples/vagrant/aurorabuild.sh b/examples/vagrant/aurorabuild.sh
index 69983d0..b7ea417 100755
--- a/examples/vagrant/aurorabuild.sh
+++ b/examples/vagrant/aurorabuild.sh
@@ -33,13 +33,8 @@ function upstart_update {
}
function build_client {
- ./pants src/main/python/apache/aurora/client/bin:aurora_client
- sudo ln -sf $DIST_DIR/aurora_client.pex /usr/local/bin/aurora
-}
-
-function build_client2 {
- ./pants src/main/python/apache/aurora/client/cli:aurora2
- sudo ln -sf $DIST_DIR/aurora2.pex /usr/local/bin/aurora2
+ ./pants src/main/python/apache/aurora/client/cli:aurora
+ sudo ln -sf $DIST_DIR/aurora.pex /usr/local/bin/aurora
}
function build_admin_client {
@@ -92,14 +87,13 @@ function build_observer {
function build_all {
build_admin_client
build_client
- build_client2
build_executor
build_observer
build_scheduler
}
function print_components {
- echo 'Please select from: admin_client, client, client2, executor, observer, scheduler or all.'
+ echo 'Please select from: admin_client, client, executor, observer, scheduler or all.'
}
if [ "$#" -eq 0 ]
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/examples/vagrant/test_tutorial.sh
----------------------------------------------------------------------
diff --git a/examples/vagrant/test_tutorial.sh b/examples/vagrant/test_tutorial.sh
old mode 100644
new mode 100755
index a5557cb..4628241
--- a/examples/vagrant/test_tutorial.sh
+++ b/examples/vagrant/test_tutorial.sh
@@ -104,7 +104,7 @@ function await_task_in_state {
do
# TODO(wfarner): This check is not that great, since it will detect any task
# in the job, rather a newly-failed task.
- result=$(vagrant ssh -c "aurora status $1 2>&1 | grep 'status: $2'" 2>&1)
+ result=$(vagrant ssh -c "aurora job status $1 2>&1 | grep 'status: $2'" 2>&1)
if [ $? -eq 0 ]
then
echo 'PASS'
@@ -128,14 +128,14 @@ cd "$(git rev-parse --show-toplevel)"
JOB_KEY=devcluster/www-data/devel/hello_world
write_test_files
-aurora_command create "create $JOB_KEY /vagrant/hello_world.aurora"
+aurora_command create "job create $JOB_KEY /vagrant/hello_world.aurora"
await_task_in_state $JOB_KEY FAILED
# Fix the bug in our test script.
perl -pi -e 's|xrang\(|xrange\(|g' hello_world.py
-aurora_command update "update $JOB_KEY /vagrant/hello_world.aurora"
+aurora_command update "job update $JOB_KEY /vagrant/hello_world.aurora"
await_task_in_state $JOB_KEY RUNNING
-aurora_command killall "killall $JOB_KEY"
+aurora_command killall "job killall $JOB_KEY"
exit 0
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/main/python/apache/aurora/client/BUILD
----------------------------------------------------------------------
diff --git a/src/main/python/apache/aurora/client/BUILD b/src/main/python/apache/aurora/client/BUILD
index 3ae2a61..1a91ff6 100644
--- a/src/main/python/apache/aurora/client/BUILD
+++ b/src/main/python/apache/aurora/client/BUILD
@@ -87,23 +87,7 @@ python_library(
name = 'apache.aurora.client',
version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(),
).with_binaries(
- aurora_admin = 'src/main/python/apache/aurora/client/bin:aurora_admin',
- aurora = 'src/main/python/apache/aurora/client/bin:aurora_client',
- )
-)
-
-python_library(
- name = 'clientv2-packaged',
- dependencies = [
- 'src/main/python/apache/aurora/common',
- 'src/main/python/apache/aurora/config:config-packaged',
- 'src/main/python/apache/thermos/common',
- ],
- provides = setup_py(
- name = 'apache.aurora.clientv2',
- version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(),
- ).with_binaries(
- aurora2 = 'src/main/python/apache/aurora/client/cli:aurora2',
+ aurora = 'src/main/python/apache/aurora/client/cli:aurora',
)
)
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/main/python/apache/aurora/client/bin/BUILD
----------------------------------------------------------------------
diff --git a/src/main/python/apache/aurora/client/bin/BUILD b/src/main/python/apache/aurora/client/bin/BUILD
index 671bd78..a69807f 100644
--- a/src/main/python/apache/aurora/client/bin/BUILD
+++ b/src/main/python/apache/aurora/client/bin/BUILD
@@ -12,14 +12,6 @@
# limitations under the License.
#
-python_binary(
- name = 'aurora_client',
- entry_point = 'apache.aurora.client.bin.aurora_client:proxy_main',
- dependencies = [
- ':aurora_client_lib'
- ]
-)
-
python_library(
name = 'aurora_client_lib',
sources = [ 'aurora_client.py' ],
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/main/python/apache/aurora/client/cli/BUILD
----------------------------------------------------------------------
diff --git a/src/main/python/apache/aurora/client/cli/BUILD b/src/main/python/apache/aurora/client/cli/BUILD
index e61cdfb..57683ae 100644
--- a/src/main/python/apache/aurora/client/cli/BUILD
+++ b/src/main/python/apache/aurora/client/cli/BUILD
@@ -13,14 +13,13 @@
#
python_binary(
- name='aurora2',
+ name='aurora',
entry_point = 'apache.aurora.client.cli.client:proxy_main',
dependencies = [
':client_lib'
],
)
-
# TODO(wfarner): Remove this along with the rest of the v1 client code.
python_library(
name = 'bridge',
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/main/python/apache/aurora/client/cli/__init__.py
----------------------------------------------------------------------
diff --git a/src/main/python/apache/aurora/client/cli/__init__.py b/src/main/python/apache/aurora/client/cli/__init__.py
index 2717210..395819f 100644
--- a/src/main/python/apache/aurora/client/cli/__init__.py
+++ b/src/main/python/apache/aurora/client/cli/__init__.py
@@ -12,7 +12,7 @@
# limitations under the License.
#
-'''Command-line tooling infrastructure for aurora client v2.
+'''Command-line tooling infrastructure for aurora client.
This module provides a framework for a noun/verb command-line application.
In this framework, an application is structured around a collection of basic objects (nouns)
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/main/python/apache/aurora/client/cli/client.py
----------------------------------------------------------------------
diff --git a/src/main/python/apache/aurora/client/cli/client.py b/src/main/python/apache/aurora/client/cli/client.py
index c8768c7..939e32b 100644
--- a/src/main/python/apache/aurora/client/cli/client.py
+++ b/src/main/python/apache/aurora/client/cli/client.py
@@ -55,7 +55,7 @@ class AuroraLogConfigurationPlugin(ConfigurationPlugin):
class AuroraCommandLine(CommandLine):
- """The CommandLine implementation for the Aurora client v2 command line."""
+ """The CommandLine implementation for the Aurora client command line."""
def __init__(self):
super(AuroraCommandLine, self).__init__()
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/main/python/apache/aurora/client/cli/config.py
----------------------------------------------------------------------
diff --git a/src/main/python/apache/aurora/client/cli/config.py b/src/main/python/apache/aurora/client/cli/config.py
index 637e7ac..73b5562 100644
--- a/src/main/python/apache/aurora/client/cli/config.py
+++ b/src/main/python/apache/aurora/client/cli/config.py
@@ -13,7 +13,7 @@
#
"""
-An implementation of a clientv2 "config" noun, for commands that
+An implementation of a "config" noun, for commands that
operate in on configuration files.
"""
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/test/python/apache/aurora/client/cli/test_help.py
----------------------------------------------------------------------
diff --git a/src/test/python/apache/aurora/client/cli/test_help.py b/src/test/python/apache/aurora/client/cli/test_help.py
index 2fef282..9fa05e6 100644
--- a/src/test/python/apache/aurora/client/cli/test_help.py
+++ b/src/test/python/apache/aurora/client/cli/test_help.py
@@ -22,7 +22,7 @@ from apache.aurora.client.cli.client import AuroraCommandLine
class TestHelp(unittest.TestCase):
- """Tests of the help command for the Aurora v2 client framework"""
+ """Tests of the help command for the Aurora client framework"""
def setUp(self):
self.cmd = AuroraCommandLine()
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
----------------------------------------------------------------------
diff --git a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
index 1458e7d..2a436b4 100755
--- a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
+++ b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
@@ -24,36 +24,62 @@ set -u -e -x
. src/test/sh/org/apache/aurora/e2e/test_common.sh
+
+function check_url_live() {
+ test $(curl -sL -w '%{http_code}' $1 -o /dev/null) == 200
+}
+
test_http_example() {
local _cluster=$1 _role=$2 _env=$3 _job=$4 _sched_ip=$5
local _base_config=$6 _updated_config=$7
jobkey="$_cluster/$_role/$_env/$_job"
+ # The vagrant/ssh command brings a trailing carriage return and newline, tr strips that.
+ joblist=$(vagrant ssh -c "aurora config list $_base_config" | tr -dc '[[:print:]]')
+ test "$joblist" = "jobs=[$jobkey]"
+
+ vagrant ssh -c "aurora job inspect $jobkey $_base_config"
+
echo '== Creating job'
- vagrant ssh -c "aurora create $jobkey $_base_config"
+ vagrant ssh -c "aurora job create $jobkey $_base_config"
- # Check that scheduler /vars being exported
+ echo "== Checking job status"
+ vagrant ssh -c "aurora job list $_cluster/$_role/$_env" | grep "$jobkey"
+ vagrant ssh -c "aurora job status $jobkey"
+ # Check that scheduler UI pages shown
base_url="http://$_sched_ip:8081"
- uptime=$(_curl -s "$base_url/vars" | grep jvm_uptime_secs | wc -l)
- test $uptime -eq 1
+ check_url_live "$base_url/scheduler"
+ check_url_live "$base_url/scheduler/$_role"
+ check_url_live "$base_url/scheduler/$_role/$_env/$_job"
+
+ echo "== Restarting test job"
+
+ vagrant ssh -c "aurora job restart $jobkey"
echo '== Updating test job'
- vagrant ssh -c "aurora update $jobkey $_updated_config"
+ vagrant ssh -c "aurora job update $jobkey $_updated_config"
echo '== Validating announce'
validate_serverset "/aurora/$_role/$_env/$_job"
- echo "== Probing job via 'aurora run'"
- # In order for `aurora run` to work, the VM needs to be forwarded a local ssh identity. To avoid
+ # In order for `aurora task run` to work, the VM needs to be forwarded a local ssh identity. To avoid
# polluting the global ssh-agent with this identity, we run this test in the context of a local
# agent. A slightly cleaner solution would be to use a here doc (ssh-agent sh <<EOF ...), but
# due to a strange confluence of issues, this required some unpalatable hacks. Simply putting
# the meat of this test in a separate file seems preferable.
- ssh-agent src/test/sh/org/apache/aurora/e2e/test_run.sh $jobkey $_sched_ip "aurora run"
+ ssh-agent src/test/sh/org/apache/aurora/e2e/test_run.sh $jobkey $_sched_ip
test $? -eq 0
- vagrant ssh -c "aurora get_quota --cluster=$_cluster $_role"
- vagrant ssh -c "aurora killall $jobkey"
+ # Run a kill without specifying instances, and verify that it gets an error, and the job
+ # isn't affected. (TODO(mchucarroll): the failed kill should return non-zero!)
+ vagrant ssh -c "aurora job kill $jobkey" 2>&1 | grep -q "The instances list cannot be omitted in a kill command"
+ check_url_live "$base_url/scheduler/$_role/$_env/$_job"
+
+ vagrant ssh -c "aurora job kill $jobkey/1"
+
+ vagrant ssh -c "aurora job killall $jobkey"
+
+ vagrant ssh -c "aurora quota get $_cluster/$_role"
}
test_admin() {
http://git-wip-us.apache.org/repos/asf/incubator-aurora/blob/9f6ec4bd/src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh
----------------------------------------------------------------------
diff --git a/src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh b/src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh
deleted file mode 100755
index d157bb1..0000000
--- a/src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh
+++ /dev/null
@@ -1,125 +0,0 @@
-#!/bin/bash
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-#
-# A simple integration test for the mesos client, intended to be run before checkin of major
-#client changes, and as a part of an integrated build process.
-#
-# This test uses the vagrant demonstration environment. It loads up a virtual cluster, and then
-# launches a job, verifies that it's running, updates it, verifies that the update succeeded,
-# and then kills the job.
-
-set -u -e -x
-
-. src/test/sh/org/apache/aurora/e2e/test_common.sh
-
-
-function check_url_live() {
- test $(curl -sL -w '%{http_code}' $1 -o /dev/null) == 200
-}
-
-test_http_example() {
- local _cluster=$1 _role=$2 _env=$3 _job=$4 _sched_ip=$5
- local _base_config=$6 _updated_config=$7
- jobkey="$_cluster/$_role/$_env/$_job"
-
- joblist=$(vagrant ssh -c "aurora2 config list $_base_config")
-# test "$joblist" = "jobs=[$jobkey]"
-
- vagrant ssh -c "aurora2 job inspect $jobkey $_base_config"
-
- echo '== Creating job'
- vagrant ssh -c "aurora2 job create $jobkey $_base_config"
-
- echo "== Checking job status"
- vagrant ssh -c "aurora2 job list $_cluster/$_role/$_env" | grep "$jobkey"
- vagrant ssh -c "aurora2 job status $jobkey"
- # Check that scheduler UI pages shown
- base_url="http://$_sched_ip:8081"
- check_url_live "$base_url/scheduler"
- check_url_live "$base_url/scheduler/$_role"
- check_url_live "$base_url/scheduler/$_role/$_env/$_job"
-
- echo "== Restarting test job"
-
- vagrant ssh -c "aurora2 job restart $jobkey"
-
- echo '== Updating test job'
- vagrant ssh -c "aurora2 job update $jobkey $_updated_config"
-
- echo '== Validating announce'
- validate_serverset "/aurora/$_role/$_env/$_job"
-
- # In order for `aurora task run` to work, the VM needs to be forwarded a local ssh identity. To avoid
- # polluting the global ssh-agent with this identity, we run this test in the context of a local
- # agent. A slightly cleaner solution would be to use a here doc (ssh-agent sh <<EOF ...), but
- # due to a strange confluence of issues, this required some unpalatable hacks. Simply putting
- # the meat of this test in a separate file seems preferable.
- ssh-agent src/test/sh/org/apache/aurora/e2e/test_run.sh $jobkey $_sched_ip
- test $? -eq 0
-
- # Run a kill without specifying instances, and verify that it gets an error, and the job
- # isn't affected. (TODO(mchucarroll): the failed kill should return non-zero!)
- vagrant ssh -c "aurora2 job kill $jobkey" 2>&1 | grep -q "The instances list cannot be omitted in a kill command"
- check_url_live "$base_url/scheduler/$_role/$_env/$_job"
-
- vagrant ssh -c "aurora2 job kill $jobkey/1"
-
- vagrant ssh -c "aurora2 job killall $jobkey"
-
- vagrant ssh -c "aurora2 quota get $_cluster/$_role"
-}
-
-test_admin() {
- local _cluster=$1 _sched_ip=$2
-
- base_url="http://$_sched_ip:8081"
-
- echo '== Testing Aurora Admin commands...'
- echo '== Getting leading scheduler'
- vagrant ssh -c "aurora_admin get_scheduler $_cluster" | grep "$base_url"
-}
-
-RETCODE=1
-# Set up shorthands for test
-export EXAMPLE_DIR=/vagrant/src/test/sh/org/apache/aurora/e2e/http
-TEST_CLUSTER=devcluster
-TEST_ROLE=vagrant
-TEST_ENV=test
-TEST_JOB=http_example
-TEST_SCHEDULER_IP=192.168.33.7
-TEST_ARGS=(
- $TEST_CLUSTER
- $TEST_ROLE
- $TEST_ENV
- $TEST_JOB
- $TEST_SCHEDULER_IP
- $EXAMPLE_DIR/http_example.aurora
- $EXAMPLE_DIR/http_example_updated.aurora
- )
-
-TEST_ADMIN_ARGS=(
- $TEST_CLUSTER
- $TEST_SCHEDULER_IP
-)
-
-trap collect_result EXIT
-vagrant up
-vagrant ssh -c "aurorabuild all"
-
-# wipe the pseudo-deploy dir, and then create it fresh, to guarantee that the
-# test runs clean.
-test_http_example "${TEST_ARGS[@]}"
-test_admin "${TEST_ADMIN_ARGS[@]}"
-RETCODE=0