Cli (or other tool?) with structured output like json for administration?

[Johan Rask <jo...@klipsk.se>]

Hi,
 we are working on automating our kafka setup (topics, users, acls etc) but
we are missing structured output (think json) from the cli tools (i.e
kafka-acls.sh, kafka-topics.sh). It only gives us human readable output
which is hard to parse.

We would like to be able to use it with common tools like bash scripts, jq
and ansible . Is there any existing solution or any ongoing initiative that
can help us with this?

Regards /Johan

Re: Cli (or other tool?) with structured output like json for administration?

[Johan Rask <jo...@klipsk.se>]

Hi again,

I am using confluent OSS, however there is nothing magical there that helps
me with this problem.
Let me explain the problem.

Setting up the cluster + zookeeper is not a problem, making changes to
existing configurations regarding security and other configs (with cli) is
a problem.

In our enterprise, we use kafka for ingestion from multiple "sources" into
multiple topics (hundreds or more) in our cluster.
Security (authentication+authorization) is very important so we use both
client certs and user+passwd.
We use groups, ACLs, quotas (+ other configs) per topic and this can be
quite complicated.

One thing that we have decided is that EVERYTHING MUST BE IN GIT and that
it must be SIMPLE. This means
that if we want to make a change on the cluster we only want to run an
ansible playbook and everything should be as we want.

To solve this we configure everything in yaml files where we have a nice
way of describing all these things.
This also works fine the first time we configure the cluster since then we
only have to add things.

Two main problems:

- The next issue is adding a single parameter, since then we have to run
the complete setup again (almost) since we
cannot perform a diff of what we have already created and what already
exist. Problem: Takes long time.

- Next thing is to delete a parameter, which means remove a line from the
config. To solve deletion we need a diff and since we cannot get structured
data, this currently does not
work (although we currently use an error prone approach that sucks...)

I am a bit surprised that more people does not have problems with this kind
of things, I have found one or two pet projects on github that solves a
part of this but not more.

Any input is appreciated

Regards /Johan


mån 13 aug. 2018 kl. 14:28 skrev M. Manna <ma...@gmail.com>:

> If you are working on automating your kafka setup, wouldn't it be better
> for you to use confluent OSS? Or, are you aiming for something very much
> home-grown?
>
> On 13 August 2018 at 13:25, Johan Rask <jo...@klipsk.se> wrote:
>
> > Hi,
> >  we are working on automating our kafka setup (topics, users, acls etc)
> but
> > we are missing structured output (think json) from the cli tools (i.e
> > kafka-acls.sh, kafka-topics.sh). It only gives us human readable output
> > which is hard to parse.
> >
> > We would like to be able to use it with common tools like bash scripts,
> jq
> > and ansible . Is there any existing solution or any ongoing initiative
> that
> > can help us with this?
> >
> > Regards /Johan
> >
>

Re: Cli (or other tool?) with structured output like json for administration?

["M. Manna" <ma...@gmail.com>]

If you are working on automating your kafka setup, wouldn't it be better
for you to use confluent OSS? Or, are you aiming for something very much
home-grown?

On 13 August 2018 at 13:25, Johan Rask <jo...@klipsk.se> wrote:

> Hi,
>  we are working on automating our kafka setup (topics, users, acls etc) but
> we are missing structured output (think json) from the cli tools (i.e
> kafka-acls.sh, kafka-topics.sh). It only gives us human readable output
> which is hard to parse.
>
> We would like to be able to use it with common tools like bash scripts, jq
> and ansible . Is there any existing solution or any ongoing initiative that
> can help us with this?
>
> Regards /Johan
>