Re: Aurora Config Generation

["Erb, Stephan" <St...@blue-yonder.com>]

Thanks for posting this! I'll take a closer look over the next days. Just a few questions beforehand:


* How do you share your templates? Is there an easy distribution mechanism or does one have to assume that everything is in one repository?

* In IRC you mentioned that JSON would allow you to add additional information which is not required by Aurora/Thermos itself. Did I understand this correctly? How would this work here?

* Do you have a minimal example of the commandline kungfu you are doing to work with this?

​

Best Regards,

Stephan

________________________________
From: benley@gmail.com <be...@gmail.com>
Sent: Friday, January 29, 2016 5:48 AM
To: user@aurora.apache.org
Subject: Re: Aurora Config Generation

As promised, I've put together some examples of the way I'm using jsonnet with aurora.  It's up on github: https://github.com/benley/aurora-jsonnet

Included are a set of base schema helpers for aurora, various helper functions, a couple of test cases, a minimal aurora job example, and a much more complex aurproxy service.  I've also included the generated raw JSON output from these templates for the sake of illustration.  The structure is still evolving a bit, and I suppose it would be helpful to add a medium-sized example to the set.

Hopefully the utility of this project is evident!  It has certainly been a boon to my team's ability to build nicely composable job templates, and jsonnet is proving to be a pretty nice language to work with.  Plus it's very accessible to anyone familiar with python, as it uses exactly the same syntax for things like list comprehensions and string formatting.

On Thu, Jan 14, 2016 at 1:43 AM benley@gmail.com<ma...@gmail.com> <be...@gmail.com>> wrote:
I do - just haven't had time to put it together yet.  Should have some free time tomorrow while I'm on an airplane :)

On Wed, Jan 13, 2016 at 9:01 AM Erb, Stephan <St...@blue-yonder.com>> wrote:

​Do you have a jsonnet example to share?. We are currently evaluating jsonnet for a configuration usecase. If one could integration Aurora with it, that would be interesting.


Regards,

Stephan

________________________________
From: benley@gmail.com<ma...@gmail.com> <be...@gmail.com>>
Sent: Tuesday, January 12, 2016 6:30 AM
To: user@aurora.apache.org<ma...@aurora.apache.org>
Subject: Re: Aurora Config Generation

My motivating use case was generating configs for aurproxy (https://github.com/tellapart/aurproxy), which takes all of its configuration as commandline parameters.  It would be possible to do most of it with pystachio, but I frequently feel like I'm fighting against pystachio rather than working with a helpful tool.  With jsonnet I have a purely functional templating language where I can build concise and modular interfaces that yield pretty elaborate aurora jobs and load balancer configs.  The output is guaranteed to be deterministic, as since it doesn't offer the temptation of adding terrible hacks (talking to the network, spawning subprocess, writing to disk, who knows) that Python allows.  Ultimately I'm finding that it is easier to reason about what's happening without the syntactic ambiguity of which parts of a configuration get bound when the template is loaded vs at job runtime.

My justification ends up being pretty similar to the explanation from this example in Flabbergast's examples directory: https://github.com/flabbergast-config/flabbergast/tree/master/examples/complex

I'm happy to share some of what I've been cobbling together with jsonnet if there is interest in such a thing.  The --read-json option to the aurora cli tool has come in very handy :-)

On Mon, Jan 11, 2016 at 6:02 AM Chris Bannister <c....@gmail.com>> wrote:
As part of building a tool to make it really easy to run Spark on Aurora (with the goal being to do 'spark run <name> <jar>' which launches 2 aurora jobs) I am creating the ExecutorData JSON in Go and using the Thrift interfaces to call createJob, it works, and is very verbose. I think it will work well as long as you dont want to do too much configuration. To ensure I got the correct things I wrote an aurora config then looked at the generated JobConfiguration.

It would be nice if the ExecutorData was defined in Thrift or such, but I can see why it is not.

On Mon, 11 Jan 2016 at 13:34 Erb, Stephan <St...@blue-yonder.com>> wrote:
A couple of days ago there was a rather brief discussion in IRC regarding the generation of Aurora configuration files:

> 00:16 <benley> random poll: anyone know who is something other than the normal Python DSL for building Aurora jobs?
> 00:17 <benley> I know some people are using flabbergast, and rafik was at least experimenting with Nix to generate them
> 00:17 <benley> and I've been tinkering with using Jsonnet to generate them, which ends up being quite similar to Flabbergast
> 00:19 <wfarner> benley: i don't know of any other DSLs, but i have heard of a few distillations into essentially properties files
> 00:21 <benley> that makes sense - heavily templating a job definition so it's generally applicable

I find this topic quite interesting. Does anyone has some experiences to share with the mailinglist? What was your the main motivation using an additional configuration or templating mechanism? Did it pay off?

Thanks and Best Regards,
Stephan