[jira] [Commented] (SAMZA-416) Samza Configuration DSL

["Chris Riccomini (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138157#comment-14138157 ] 

Chris Riccomini commented on SAMZA-416:
---------------------------------------

bq. The key would be to implement a standard interface for these different implementations to talk to.

IMO, the standard interface will be the API defined in ConfigStream. Whether we implement configure-job.sh, a DSL, or some builder on top, they'd all end up calling some ConfigStream.write API.

One thing to consider is how to make this language-agnostic. The ConfigStream API will be pluggable and could sit on top of multiple implementations (Kafka, HBase, etc). If we have a Python-based DSL, we'd need an easy way for Python to write to the ConfigStream.

bq. One approach would be to implement a DSL in a scripting language like Python. Implementing a Python DSL is actually very easy to do, and there are many examples of Python DSLs in non-trivial use cases, most notably, Google's build system, Blaze.

Are there any examples of DSL-based config languages? I'm a bit hesitant to use a build system as an example of how to do configuration, since I think the two use cases might be different.

Another thought on this subject: it seems to me that DSL-based config might be a bit ops-unfriendly. If an SRE has to go and edit a (potentially complicated) .py file to change a config, this might be a bit undesirable.

> Samza Configuration DSL
> -----------------------
>
>                 Key: SAMZA-416
>                 URL: https://issues.apache.org/jira/browse/SAMZA-416
>             Project: Samza
>          Issue Type: New Feature
>            Reporter: David Chen
>
> The user-facing language for Samza configurations is currently Java Properties. While this works for the time being and simple to implement, it is verbose and cumbersome for the end user.
> Since SAMZA-40 is opened to refactor the way Samza configuration is wired and SAMZA-348 is opened to allow Samza jobs to be configured through a stream, it is thus natural to consider a scripting model for configuring Samza jobs.
> One approach would be to implement a DSL in a scripting language like Python. Implementing a Python DSL is actually very easy to do, and there are many examples of Python DSLs in non-trivial use cases, most notably, [Google's build system, Blaze|http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html]. I found [a GitHub Gist containing a more detailed example of a Blaze BUILD file|https://gist.github.com/wiseman/3834928]. Blaze also inspired a number of open source clones, most notably [Twitter Pants|http://pantsbuild.github.io/].
> Another idea, by [~jonbringhurst] from SAMZA-348:
> {quote}
> Regarding the client commands to create and modify Samza jobs (such as configure-job and run-job), it may be useful to review existing commands that perform a similar role:
> My personal favorite is Slurm's set of client commands, of which sbatch is probably the most relevant (http://www.schedmd.com/slurmdocs/sbatch.html).
> To go back a bit further in history, it might be a good idea to take a look at the POSIX qsub style command from PBS/Torque (http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm). Moab's msub also follows this design.
> Regarding a possible DSL for building configuration, it may be useful to look at Slurm's lua callback for job configuration (warning, GPLv2 code) https://github.com/SchedMD/slurm/blob/master/contribs/lua/job_submit.lua
> {quote}
> We could, of course, have both a command line tool and a DSL, and I am sure once Samza takes off, there will be people implementing DSLs for other language clients as well. The key would be to implement a standard interface for these different implementations to talk to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)