You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Sönke Liebau <so...@opencore.com.INVALID> on 2019/05/05 07:53:41 UTC
Re: Cleaning up command line tools argument parsing a little
Hi Colin,
I totally agree! Especially the differently named bootstrap server options
have been annoying me a long time.
I'd propose a two-step approach:
1. Add new default options objects similar to CommandLineUtils and
CommandDefaultOptions (based on argparse4j) but in the clients project, as
this is referenced by all command line tools as far as I can tell
2. Refactor tools one by one to use these new helper classes (and thus
argparse) and add standardized synonyms for parameters as necessary
I think for step 1 we can get away with no KIP, as this doesn't change any
public interfaces or behavior.
Step 2 probably needs a KIP as we are adding new parameters? We can pick up
KIP-14 again for that I think. A lot of work has been done on that already.
Does that sound useful to everybody?
Best regards,
Sönke
On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cm...@apache.org> wrote:
> If we are going to standardize on one argument parsing library, it should
> certainly be argparse4j, I think.
> argparse4j is simply a better argument parsing library with support for
> more features. One example is mutually exclusive options. argparse4j
> supports this with MutuallyExclusiveGroup. jopt doesn't support this, so
> when it is needed, we have to add extra code to manually check that
> mutually exclusive options are not set.
>
> argparse4j also has subcommands. If you want something like "git add"
> with some set of flags, and "git remove" with another, you can do this with
> argparse4j, but not with jopt. This would be very helpful for clearing up
> confusion in a lot of our shell scripts which have accumulated dozens of
> arguments, most of which are only relevant to a very specific operation.
> But you just can't do it with jopt.
>
> Just to give an example, argparse4j with subcommands would allow you to
> run something like ./kafka-topics.sh list --help and get just options that
> were relevant for listing topics, not the full dozens of options that might
> relate to adding topics, removing them, etc.
>
> To be honest, though, what would help users the most is standardizing the
> option flags across tools. We should have a standard way of specifying
> bootstrap brokers, for example. (We can continue to support the old
> synonyms for a while, of course.)
>
> best,
> Colin
>
>
> On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote:
> > I took another look at the PR itself and I think it would be great to
> have
> > this cleanup too -- I cannot remember at the beginning why we gradually
> > moved to different mechanism (argparse4j) for different cmds, if there's
> no
> > rationales behind it we should just make them consistent.
> >
> > Thanks for driving this!
> >
> > Guozhang
> >
> > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> > > Sönke, I'd find this very helpful. It's annoying to keep track of which
> > > commands use which form -- I always seem to guess wrong.
> > >
> > > Though I don't think there is any reason to deprecate existing forms,
> e.g.
> > > consumer.config vs consumer-config. I think it's perfectly reasonable
> to
> > > have multiple spellings of the same arguments. I don't really see a
> > > downside to keeping the aliases around indefinitely.
> > >
> > > Ryanne
> > >
> > >
> > >
> > >
> > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau
> > > <so...@opencore.com.invalid> wrote:
> > >
> > > > Hi everybody,
> > > >
> > > > Jason and I were recently discussing command line argument parsing on
> > > > KAFKA-8131 (or rather the related pull request) [1].
> > > >
> > > > Command line tools and their arguments are somewhat diverse at the
> > > moment.
> > > > Most of the tools use joptsimple for argument parsing, some newer
> java
> > > > tools use argparse4j instead and some tools use nothing at all.
> > > > I've looked for a reason as to why there are two libraries being
> used,
> > > but
> > > > couldn't really find anything. Paolo brought up the same question on
> the
> > > > mailing list a while back [7], but got no response either.
> > > > Does anybody know why this is the case?
> > > >
> > > > This results in no central place to add universal parameters like
> help
> > > and
> > > > version, as well as the help output looking different between some
> of the
> > > > tools.
> > > > Also, there are a number of parameters that should be renamed to
> adhere
> > > to
> > > > defaults.
> > > >
> > > > There have been a few discussions and initiatives around this in the
> > > past.
> > > > Just of the top of my head (and a 5 minute jira search) there are:
> > > > - KIP-14 [2]
> > > > - KAFKA-2111 [3]
> > > > - KIP-316 [4]
> > > > - KAFKA-1292 [5]
> > > > - KAFKA-3530 [6]
> > > > - and probably many more
> > > >
> > > > Would people generally be in favor of revisiting this topic?
> > > >
> > > > What I'd propose to do is:
> > > > - comb through jira and KIPs, clean up old stuff and creae a new
> umbrella
> > > > issue to track this (maybe reuse KIP-4 as well)
> > > > - agree on one library for parsing command line arguments (don't care
> > > which
> > > > one, but two is one too many I think)
> > > > - refactor tools to use one library and default way of argument
> parsing
> > > > with central help and version parameter
> > > > - add aliases for options that should be renamed according to KIP-4
> (and
> > > > maybe others) so that both new and old work for a while, deprecate
> old
> > > > parameters for a cycle or two and then remove them
> > > >
> > > > I'll shut up now and see if people would consider this useful or
> have any
> > > > other input :)
> > > >
> > > > Best regards,
> > > > Sönke
> > > >
> > > > [1] https://github.com/apache/kafka/pull/6481#discussion_r273773003
> > > > <https://issues.apache.org/jira/browse/KAFKA-8131>
> > > > [2]
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
> > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111
> > > > [4]
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties
> > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292
> > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530
> > > > [7]
> > > >
> > > >
> > >
> https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
Re: Cleaning up command line tools argument parsing a little
Posted by Sönke Liebau <so...@opencore.com.INVALID>.
Hi Viktor,
I'll admit I've only had an extremely brief look at your code and jira, but
it sounds absolutely awesome! Plus it will give us a chance at a fresh
start without worrying about breaking existing code using the tools. Of
course...at the risk of code duplication..
I'd be absolutely in favor of creating something like you have proposed!
Best regards,
Sönke
On Wed, May 8, 2019 at 11:38 AM Viktor Somogyi-Vass <vi...@gmail.com>
wrote:
> Just to add: I wasn't too pushy by raising a KIP for this as so far I had
> the experience that the community is fine with them or at least people
> learned to live with the current tools but if there's a need I'd be happy
> to jump back working on it or helping you if you have time :)
>
> On Wed, May 8, 2019 at 11:35 AM Viktor Somogyi-Vass <
> viktorsomogyi@gmail.com>
> wrote:
>
> > Hi Sönke,
> >
> > In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately
> > didn't get much attention so far from the creators of the jira. It works
> > similarly to what Colin mentioned, like "kafka.sh topics create -n
> my-topic
> > -p 3 -r 3" or has long names like "kafka.sh topics create --name my-topic
> > --partitions 3 --replicas 3". The bootstrap server everywhere defaults to
> > :9092 or reads up the configs from a path so in many scenarios you can
> omit
> > it, also it uses the admin client of course and all in all provides a
> much
> > better experience than the current tools. It has interactive mode and
> help
> > too. Wanted to implement "code completion" too but for that I'd have to
> > exercise the code a little bit more :).
> > It is currently based on a quite old trunk but if you think it's
> > interesting I can rebase it and we can continue with raising a KIP.
> > https://github.com/viktorsomogyi/kafka/tree/kafka_shell
> >
> > Viktor
> >
> > On Tue, May 7, 2019 at 11:10 AM Sönke Liebau
> > <so...@opencore.com.invalid> wrote:
> >
> >> Hi Colin,
> >>
> >> thanks!
> >> While I personally don't like the current command line tools I
> >> realistically think we'll be stuck with them for a while yet, so a
> cleanup
> >> might make sense.
> >> So I'll start looking into that.
> >>
> >> Regarding a central entrypoint, that would indeed be brilliant and I'd
> >> love
> >> to work on that, but I currently have enough other open things to look
> at,
> >> so I won't draw that one to myself as well for now.
> >>
> >> But I'll keep it in mind for when some time frees up :)
> >>
> >> Best regards,
> >> Sönke
> >>
> >>
> >>
> >> Colin McCabe <cm...@apache.org> schrieb am Di., 7. Mai 2019, 00:56:
> >>
> >> > On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote:
> >> > > Hi Colin,
> >> > >
> >> > > it was my intention to keep the structure of the commands mostly
> >> intact
> >> > > while doing the refactoring - if that is possible, have not really
> >> > checked
> >> > > yet to be honest.
> >> > >
> >> > > But what I wanted to try and do is recreate the current parsing with
> >> > > argparse as much as possible. And in the process simply adding
> >> synonyms,
> >> > > for example make the kafka-console-producer understand a
> >> > > bootstrap-parameter in addition to broker-list.
> >> > > There is a bit of custom logic about which parameters go together
> >> etc. in
> >> > > the current classes, so output may look different here and there,
> but
> >> in
> >> > > principle I do believe that it should be possible to recreate the
> >> current
> >> > > structure.
> >> >
> >> > Sounds like a good idea. Thanks for the clarification.
> >> >
> >> > >
> >> > > If there is an appetite for a new, hadoop-like entrypoint anyway,
> then
> >> > all
> >> > > of this might be "wasted" effort, or rather effort better spent
> >> though,
> >> > you
> >> > > are right.
> >> >
> >> > I don't think anyone is working on a new entry point right now -- or
> if
> >> > they are, they haven't said anything yet :)
> >> >
> >> > I just wanted to mention it as a possible approach in case you wanted
> to
> >> > do a bigger project.
> >> >
> >> > best,
> >> > Colin
> >> >
> >> > >
> >> > > Best regards,
> >> > > Sönke
> >> > >
> >> > >
> >> > >
> >> > > On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cm...@apache.org>
> >> wrote:
> >> > >
> >> > > > Hi Sönke,
> >> > > >
> >> > > > #2 is a bit tough because people have come to rely on the way the
> >> > commands
> >> > > > are structured right now.
> >> > > >
> >> > > > If we want to make big changes, it might be easier just to create
> a
> >> > > > separate tool and deprecate the old one(s). One thing we've
> talked
> >> > about
> >> > > > doing in the past is creating a single entry point for all the
> tool
> >> > > > functionality, kind of like hadoop did with the "hadoop" command
> Or
> >> > git
> >> > > > with the "git" command, etc. Then we could deprecate the
> standalone
> >> > > > commands and remove them after enough time had passed-- kind of
> like
> >> > the
> >> > > > old consumer.
> >> > > >
> >> > > > On the other hand, a more incremental change would be
> standardizing
> >> > flags
> >> > > > a bit. So for example, at least setting it up so that there is a
> >> > standard
> >> > > > way of supplying bootstrap brokers, etc. We could keep the old
> >> flags
> >> > > > around for a while as variants to ease the transition.
> >> > > >
> >> > > > best,
> >> > > > Colin
> >> > > >
> >> > > >
> >> > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
> >> > > > > Hi Colin,
> >> > > > >
> >> > > > > I totally agree! Especially the differently named bootstrap
> server
> >> > > > options
> >> > > > > have been annoying me a long time.
> >> > > > >
> >> > > > > I'd propose a two-step approach:
> >> > > > > 1. Add new default options objects similar to CommandLineUtils
> and
> >> > > > > CommandDefaultOptions (based on argparse4j) but in the clients
> >> > project,
> >> > > > as
> >> > > > > this is referenced by all command line tools as far as I can
> tell
> >> > > > > 2. Refactor tools one by one to use these new helper classes
> (and
> >> > thus
> >> > > > > argparse) and add standardized synonyms for parameters as
> >> necessary
> >> > > > >
> >> > > > > I think for step 1 we can get away with no KIP, as this doesn't
> >> > change
> >> > > > any
> >> > > > > public interfaces or behavior.
> >> > > > > Step 2 probably needs a KIP as we are adding new parameters? We
> >> can
> >> > pick
> >> > > > up
> >> > > > > KIP-14 again for that I think. A lot of work has been done on
> that
> >> > > > already.
> >> > > > >
> >> > > > > Does that sound useful to everybody?
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Sönke
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <
> cmccabe@apache.org>
> >> > wrote:
> >> > > > >
> >> > > > > > If we are going to standardize on one argument parsing
> library,
> >> it
> >> > > > should
> >> > > > > > certainly be argparse4j, I think.
> >> > > > > > argparse4j is simply a better argument parsing library with
> >> > support
> >> > > > for
> >> > > > > > more features. One example is mutually exclusive options.
> >> > argparse4j
> >> > > > > > supports this with MutuallyExclusiveGroup. jopt doesn't
> support
> >> > this,
> >> > > > so
> >> > > > > > when it is needed, we have to add extra code to manually check
> >> that
> >> > > > > > mutually exclusive options are not set.
> >> > > > > >
> >> > > > > > argparse4j also has subcommands. If you want something like
> >> "git
> >> > add"
> >> > > > > > with some set of flags, and "git remove" with another, you can
> >> do
> >> > this
> >> > > > with
> >> > > > > > argparse4j, but not with jopt. This would be very helpful for
> >> > > > clearing up
> >> > > > > > confusion in a lot of our shell scripts which have accumulated
> >> > dozens
> >> > > > of
> >> > > > > > arguments, most of which are only relevant to a very specific
> >> > > > operation.
> >> > > > > > But you just can't do it with jopt.
> >> > > > > >
> >> > > > > > Just to give an example, argparse4j with subcommands would
> allow
> >> > you to
> >> > > > > > run something like ./kafka-topics.sh list --help and get just
> >> > options
> >> > > > that
> >> > > > > > were relevant for listing topics, not the full dozens of
> options
> >> > that
> >> > > > might
> >> > > > > > relate to adding topics, removing them, etc.
> >> > > > > >
> >> > > > > > To be honest, though, what would help users the most is
> >> > standardizing
> >> > > > the
> >> > > > > > option flags across tools. We should have a standard way of
> >> > specifying
> >> > > > > > bootstrap brokers, for example. (We can continue to support
> the
> >> > old
> >> > > > > > synonyms for a while, of course.)
> >> > > > > >
> >> > > > > > best,
> >> > > > > > Colin
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote:
> >> > > > > > > I took another look at the PR itself and I think it would be
> >> > great to
> >> > > > > > have
> >> > > > > > > this cleanup too -- I cannot remember at the beginning why
> we
> >> > > > gradually
> >> > > > > > > moved to different mechanism (argparse4j) for different
> cmds,
> >> if
> >> > > > there's
> >> > > > > > no
> >> > > > > > > rationales behind it we should just make them consistent.
> >> > > > > > >
> >> > > > > > > Thanks for driving this!
> >> > > > > > >
> >> > > > > > > Guozhang
> >> > > > > > >
> >> > > > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <
> >> > ryannedolan@gmail.com>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Sönke, I'd find this very helpful. It's annoying to keep
> >> track
> >> > of
> >> > > > which
> >> > > > > > > > commands use which form -- I always seem to guess wrong.
> >> > > > > > > >
> >> > > > > > > > Though I don't think there is any reason to deprecate
> >> existing
> >> > > > forms,
> >> > > > > > e.g.
> >> > > > > > > > consumer.config vs consumer-config. I think it's perfectly
> >> > > > reasonable
> >> > > > > > to
> >> > > > > > > > have multiple spellings of the same arguments. I don't
> >> really
> >> > see a
> >> > > > > > > > downside to keeping the aliases around indefinitely.
> >> > > > > > > >
> >> > > > > > > > Ryanne
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau
> >> > > > > > > > <so...@opencore.com.invalid> wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi everybody,
> >> > > > > > > > >
> >> > > > > > > > > Jason and I were recently discussing command line
> argument
> >> > > > parsing on
> >> > > > > > > > > KAFKA-8131 (or rather the related pull request) [1].
> >> > > > > > > > >
> >> > > > > > > > > Command line tools and their arguments are somewhat
> >> diverse
> >> > at
> >> > > > the
> >> > > > > > > > moment.
> >> > > > > > > > > Most of the tools use joptsimple for argument parsing,
> >> some
> >> > newer
> >> > > > > > java
> >> > > > > > > > > tools use argparse4j instead and some tools use nothing
> at
> >> > all.
> >> > > > > > > > > I've looked for a reason as to why there are two
> libraries
> >> > being
> >> > > > > > used,
> >> > > > > > > > but
> >> > > > > > > > > couldn't really find anything. Paolo brought up the same
> >> > > > question on
> >> > > > > > the
> >> > > > > > > > > mailing list a while back [7], but got no response
> either.
> >> > > > > > > > > Does anybody know why this is the case?
> >> > > > > > > > >
> >> > > > > > > > > This results in no central place to add universal
> >> parameters
> >> > like
> >> > > > > > help
> >> > > > > > > > and
> >> > > > > > > > > version, as well as the help output looking different
> >> between
> >> > > > some
> >> > > > > > of the
> >> > > > > > > > > tools.
> >> > > > > > > > > Also, there are a number of parameters that should be
> >> > renamed to
> >> > > > > > adhere
> >> > > > > > > > to
> >> > > > > > > > > defaults.
> >> > > > > > > > >
> >> > > > > > > > > There have been a few discussions and initiatives around
> >> > this in
> >> > > > the
> >> > > > > > > > past.
> >> > > > > > > > > Just of the top of my head (and a 5 minute jira search)
> >> there
> >> > > > are:
> >> > > > > > > > > - KIP-14 [2]
> >> > > > > > > > > - KAFKA-2111 [3]
> >> > > > > > > > > - KIP-316 [4]
> >> > > > > > > > > - KAFKA-1292 [5]
> >> > > > > > > > > - KAFKA-3530 [6]
> >> > > > > > > > > - and probably many more
> >> > > > > > > > >
> >> > > > > > > > > Would people generally be in favor of revisiting this
> >> topic?
> >> > > > > > > > >
> >> > > > > > > > > What I'd propose to do is:
> >> > > > > > > > > - comb through jira and KIPs, clean up old stuff and
> >> creae a
> >> > new
> >> > > > > > umbrella
> >> > > > > > > > > issue to track this (maybe reuse KIP-4 as well)
> >> > > > > > > > > - agree on one library for parsing command line
> arguments
> >> > (don't
> >> > > > care
> >> > > > > > > > which
> >> > > > > > > > > one, but two is one too many I think)
> >> > > > > > > > > - refactor tools to use one library and default way of
> >> > argument
> >> > > > > > parsing
> >> > > > > > > > > with central help and version parameter
> >> > > > > > > > > - add aliases for options that should be renamed
> >> according to
> >> > > > KIP-4
> >> > > > > > (and
> >> > > > > > > > > maybe others) so that both new and old work for a while,
> >> > > > deprecate
> >> > > > > > old
> >> > > > > > > > > parameters for a cycle or two and then remove them
> >> > > > > > > > >
> >> > > > > > > > > I'll shut up now and see if people would consider this
> >> > useful or
> >> > > > > > have any
> >> > > > > > > > > other input :)
> >> > > > > > > > >
> >> > > > > > > > > Best regards,
> >> > > > > > > > > Sönke
> >> > > > > > > > >
> >> > > > > > > > > [1]
> >> > > > https://github.com/apache/kafka/pull/6481#discussion_r273773003
> >> > > > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131>
> >> > > > > > > > > [2]
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
> >> > > > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111
> >> > > > > > > > > [4]
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties
> >> > > > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292
> >> > > > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530
> >> > > > > > > > > [7]
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> >>
> https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > -- Guozhang
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Sönke Liebau
> >> > > > > Partner
> >> > > > > Tel. +49 179 7940878
> >> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> >> Germany
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Sönke Liebau
> >> > > Partner
> >> > > Tel. +49 179 7940878
> >> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> Germany
> >> > >
> >> >
> >>
> >
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
Re: Cleaning up command line tools argument parsing a little
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Just to add: I wasn't too pushy by raising a KIP for this as so far I had
the experience that the community is fine with them or at least people
learned to live with the current tools but if there's a need I'd be happy
to jump back working on it or helping you if you have time :)
On Wed, May 8, 2019 at 11:35 AM Viktor Somogyi-Vass <vi...@gmail.com>
wrote:
> Hi Sönke,
>
> In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately
> didn't get much attention so far from the creators of the jira. It works
> similarly to what Colin mentioned, like "kafka.sh topics create -n my-topic
> -p 3 -r 3" or has long names like "kafka.sh topics create --name my-topic
> --partitions 3 --replicas 3". The bootstrap server everywhere defaults to
> :9092 or reads up the configs from a path so in many scenarios you can omit
> it, also it uses the admin client of course and all in all provides a much
> better experience than the current tools. It has interactive mode and help
> too. Wanted to implement "code completion" too but for that I'd have to
> exercise the code a little bit more :).
> It is currently based on a quite old trunk but if you think it's
> interesting I can rebase it and we can continue with raising a KIP.
> https://github.com/viktorsomogyi/kafka/tree/kafka_shell
>
> Viktor
>
> On Tue, May 7, 2019 at 11:10 AM Sönke Liebau
> <so...@opencore.com.invalid> wrote:
>
>> Hi Colin,
>>
>> thanks!
>> While I personally don't like the current command line tools I
>> realistically think we'll be stuck with them for a while yet, so a cleanup
>> might make sense.
>> So I'll start looking into that.
>>
>> Regarding a central entrypoint, that would indeed be brilliant and I'd
>> love
>> to work on that, but I currently have enough other open things to look at,
>> so I won't draw that one to myself as well for now.
>>
>> But I'll keep it in mind for when some time frees up :)
>>
>> Best regards,
>> Sönke
>>
>>
>>
>> Colin McCabe <cm...@apache.org> schrieb am Di., 7. Mai 2019, 00:56:
>>
>> > On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote:
>> > > Hi Colin,
>> > >
>> > > it was my intention to keep the structure of the commands mostly
>> intact
>> > > while doing the refactoring - if that is possible, have not really
>> > checked
>> > > yet to be honest.
>> > >
>> > > But what I wanted to try and do is recreate the current parsing with
>> > > argparse as much as possible. And in the process simply adding
>> synonyms,
>> > > for example make the kafka-console-producer understand a
>> > > bootstrap-parameter in addition to broker-list.
>> > > There is a bit of custom logic about which parameters go together
>> etc. in
>> > > the current classes, so output may look different here and there, but
>> in
>> > > principle I do believe that it should be possible to recreate the
>> current
>> > > structure.
>> >
>> > Sounds like a good idea. Thanks for the clarification.
>> >
>> > >
>> > > If there is an appetite for a new, hadoop-like entrypoint anyway, then
>> > all
>> > > of this might be "wasted" effort, or rather effort better spent
>> though,
>> > you
>> > > are right.
>> >
>> > I don't think anyone is working on a new entry point right now -- or if
>> > they are, they haven't said anything yet :)
>> >
>> > I just wanted to mention it as a possible approach in case you wanted to
>> > do a bigger project.
>> >
>> > best,
>> > Colin
>> >
>> > >
>> > > Best regards,
>> > > Sönke
>> > >
>> > >
>> > >
>> > > On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cm...@apache.org>
>> wrote:
>> > >
>> > > > Hi Sönke,
>> > > >
>> > > > #2 is a bit tough because people have come to rely on the way the
>> > commands
>> > > > are structured right now.
>> > > >
>> > > > If we want to make big changes, it might be easier just to create a
>> > > > separate tool and deprecate the old one(s). One thing we've talked
>> > about
>> > > > doing in the past is creating a single entry point for all the tool
>> > > > functionality, kind of like hadoop did with the "hadoop" command Or
>> > git
>> > > > with the "git" command, etc. Then we could deprecate the standalone
>> > > > commands and remove them after enough time had passed-- kind of like
>> > the
>> > > > old consumer.
>> > > >
>> > > > On the other hand, a more incremental change would be standardizing
>> > flags
>> > > > a bit. So for example, at least setting it up so that there is a
>> > standard
>> > > > way of supplying bootstrap brokers, etc. We could keep the old
>> flags
>> > > > around for a while as variants to ease the transition.
>> > > >
>> > > > best,
>> > > > Colin
>> > > >
>> > > >
>> > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
>> > > > > Hi Colin,
>> > > > >
>> > > > > I totally agree! Especially the differently named bootstrap server
>> > > > options
>> > > > > have been annoying me a long time.
>> > > > >
>> > > > > I'd propose a two-step approach:
>> > > > > 1. Add new default options objects similar to CommandLineUtils and
>> > > > > CommandDefaultOptions (based on argparse4j) but in the clients
>> > project,
>> > > > as
>> > > > > this is referenced by all command line tools as far as I can tell
>> > > > > 2. Refactor tools one by one to use these new helper classes (and
>> > thus
>> > > > > argparse) and add standardized synonyms for parameters as
>> necessary
>> > > > >
>> > > > > I think for step 1 we can get away with no KIP, as this doesn't
>> > change
>> > > > any
>> > > > > public interfaces or behavior.
>> > > > > Step 2 probably needs a KIP as we are adding new parameters? We
>> can
>> > pick
>> > > > up
>> > > > > KIP-14 again for that I think. A lot of work has been done on that
>> > > > already.
>> > > > >
>> > > > > Does that sound useful to everybody?
>> > > > >
>> > > > > Best regards,
>> > > > > Sönke
>> > > > >
>> > > > >
>> > > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cm...@apache.org>
>> > wrote:
>> > > > >
>> > > > > > If we are going to standardize on one argument parsing library,
>> it
>> > > > should
>> > > > > > certainly be argparse4j, I think.
>> > > > > > argparse4j is simply a better argument parsing library with
>> > support
>> > > > for
>> > > > > > more features. One example is mutually exclusive options.
>> > argparse4j
>> > > > > > supports this with MutuallyExclusiveGroup. jopt doesn't support
>> > this,
>> > > > so
>> > > > > > when it is needed, we have to add extra code to manually check
>> that
>> > > > > > mutually exclusive options are not set.
>> > > > > >
>> > > > > > argparse4j also has subcommands. If you want something like
>> "git
>> > add"
>> > > > > > with some set of flags, and "git remove" with another, you can
>> do
>> > this
>> > > > with
>> > > > > > argparse4j, but not with jopt. This would be very helpful for
>> > > > clearing up
>> > > > > > confusion in a lot of our shell scripts which have accumulated
>> > dozens
>> > > > of
>> > > > > > arguments, most of which are only relevant to a very specific
>> > > > operation.
>> > > > > > But you just can't do it with jopt.
>> > > > > >
>> > > > > > Just to give an example, argparse4j with subcommands would allow
>> > you to
>> > > > > > run something like ./kafka-topics.sh list --help and get just
>> > options
>> > > > that
>> > > > > > were relevant for listing topics, not the full dozens of options
>> > that
>> > > > might
>> > > > > > relate to adding topics, removing them, etc.
>> > > > > >
>> > > > > > To be honest, though, what would help users the most is
>> > standardizing
>> > > > the
>> > > > > > option flags across tools. We should have a standard way of
>> > specifying
>> > > > > > bootstrap brokers, for example. (We can continue to support the
>> > old
>> > > > > > synonyms for a while, of course.)
>> > > > > >
>> > > > > > best,
>> > > > > > Colin
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote:
>> > > > > > > I took another look at the PR itself and I think it would be
>> > great to
>> > > > > > have
>> > > > > > > this cleanup too -- I cannot remember at the beginning why we
>> > > > gradually
>> > > > > > > moved to different mechanism (argparse4j) for different cmds,
>> if
>> > > > there's
>> > > > > > no
>> > > > > > > rationales behind it we should just make them consistent.
>> > > > > > >
>> > > > > > > Thanks for driving this!
>> > > > > > >
>> > > > > > > Guozhang
>> > > > > > >
>> > > > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <
>> > ryannedolan@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Sönke, I'd find this very helpful. It's annoying to keep
>> track
>> > of
>> > > > which
>> > > > > > > > commands use which form -- I always seem to guess wrong.
>> > > > > > > >
>> > > > > > > > Though I don't think there is any reason to deprecate
>> existing
>> > > > forms,
>> > > > > > e.g.
>> > > > > > > > consumer.config vs consumer-config. I think it's perfectly
>> > > > reasonable
>> > > > > > to
>> > > > > > > > have multiple spellings of the same arguments. I don't
>> really
>> > see a
>> > > > > > > > downside to keeping the aliases around indefinitely.
>> > > > > > > >
>> > > > > > > > Ryanne
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau
>> > > > > > > > <so...@opencore.com.invalid> wrote:
>> > > > > > > >
>> > > > > > > > > Hi everybody,
>> > > > > > > > >
>> > > > > > > > > Jason and I were recently discussing command line argument
>> > > > parsing on
>> > > > > > > > > KAFKA-8131 (or rather the related pull request) [1].
>> > > > > > > > >
>> > > > > > > > > Command line tools and their arguments are somewhat
>> diverse
>> > at
>> > > > the
>> > > > > > > > moment.
>> > > > > > > > > Most of the tools use joptsimple for argument parsing,
>> some
>> > newer
>> > > > > > java
>> > > > > > > > > tools use argparse4j instead and some tools use nothing at
>> > all.
>> > > > > > > > > I've looked for a reason as to why there are two libraries
>> > being
>> > > > > > used,
>> > > > > > > > but
>> > > > > > > > > couldn't really find anything. Paolo brought up the same
>> > > > question on
>> > > > > > the
>> > > > > > > > > mailing list a while back [7], but got no response either.
>> > > > > > > > > Does anybody know why this is the case?
>> > > > > > > > >
>> > > > > > > > > This results in no central place to add universal
>> parameters
>> > like
>> > > > > > help
>> > > > > > > > and
>> > > > > > > > > version, as well as the help output looking different
>> between
>> > > > some
>> > > > > > of the
>> > > > > > > > > tools.
>> > > > > > > > > Also, there are a number of parameters that should be
>> > renamed to
>> > > > > > adhere
>> > > > > > > > to
>> > > > > > > > > defaults.
>> > > > > > > > >
>> > > > > > > > > There have been a few discussions and initiatives around
>> > this in
>> > > > the
>> > > > > > > > past.
>> > > > > > > > > Just of the top of my head (and a 5 minute jira search)
>> there
>> > > > are:
>> > > > > > > > > - KIP-14 [2]
>> > > > > > > > > - KAFKA-2111 [3]
>> > > > > > > > > - KIP-316 [4]
>> > > > > > > > > - KAFKA-1292 [5]
>> > > > > > > > > - KAFKA-3530 [6]
>> > > > > > > > > - and probably many more
>> > > > > > > > >
>> > > > > > > > > Would people generally be in favor of revisiting this
>> topic?
>> > > > > > > > >
>> > > > > > > > > What I'd propose to do is:
>> > > > > > > > > - comb through jira and KIPs, clean up old stuff and
>> creae a
>> > new
>> > > > > > umbrella
>> > > > > > > > > issue to track this (maybe reuse KIP-4 as well)
>> > > > > > > > > - agree on one library for parsing command line arguments
>> > (don't
>> > > > care
>> > > > > > > > which
>> > > > > > > > > one, but two is one too many I think)
>> > > > > > > > > - refactor tools to use one library and default way of
>> > argument
>> > > > > > parsing
>> > > > > > > > > with central help and version parameter
>> > > > > > > > > - add aliases for options that should be renamed
>> according to
>> > > > KIP-4
>> > > > > > (and
>> > > > > > > > > maybe others) so that both new and old work for a while,
>> > > > deprecate
>> > > > > > old
>> > > > > > > > > parameters for a cycle or two and then remove them
>> > > > > > > > >
>> > > > > > > > > I'll shut up now and see if people would consider this
>> > useful or
>> > > > > > have any
>> > > > > > > > > other input :)
>> > > > > > > > >
>> > > > > > > > > Best regards,
>> > > > > > > > > Sönke
>> > > > > > > > >
>> > > > > > > > > [1]
>> > > > https://github.com/apache/kafka/pull/6481#discussion_r273773003
>> > > > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131>
>> > > > > > > > > [2]
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
>> > > > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111
>> > > > > > > > > [4]
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties
>> > > > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292
>> > > > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530
>> > > > > > > > > [7]
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > >
>> >
>> https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > -- Guozhang
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Sönke Liebau
>> > > > > Partner
>> > > > > Tel. +49 179 7940878
>> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
>> Germany
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Sönke Liebau
>> > > Partner
>> > > Tel. +49 179 7940878
>> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>> > >
>> >
>>
>
Re: Cleaning up command line tools argument parsing a little
Posted by Viktor Somogyi-Vass <vi...@gmail.com>.
Hi Sönke,
In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately
didn't get much attention so far from the creators of the jira. It works
similarly to what Colin mentioned, like "kafka.sh topics create -n my-topic
-p 3 -r 3" or has long names like "kafka.sh topics create --name my-topic
--partitions 3 --replicas 3". The bootstrap server everywhere defaults to
:9092 or reads up the configs from a path so in many scenarios you can omit
it, also it uses the admin client of course and all in all provides a much
better experience than the current tools. It has interactive mode and help
too. Wanted to implement "code completion" too but for that I'd have to
exercise the code a little bit more :).
It is currently based on a quite old trunk but if you think it's
interesting I can rebase it and we can continue with raising a KIP.
https://github.com/viktorsomogyi/kafka/tree/kafka_shell
Viktor
On Tue, May 7, 2019 at 11:10 AM Sönke Liebau
<so...@opencore.com.invalid> wrote:
> Hi Colin,
>
> thanks!
> While I personally don't like the current command line tools I
> realistically think we'll be stuck with them for a while yet, so a cleanup
> might make sense.
> So I'll start looking into that.
>
> Regarding a central entrypoint, that would indeed be brilliant and I'd love
> to work on that, but I currently have enough other open things to look at,
> so I won't draw that one to myself as well for now.
>
> But I'll keep it in mind for when some time frees up :)
>
> Best regards,
> Sönke
>
>
>
> Colin McCabe <cm...@apache.org> schrieb am Di., 7. Mai 2019, 00:56:
>
> > On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote:
> > > Hi Colin,
> > >
> > > it was my intention to keep the structure of the commands mostly intact
> > > while doing the refactoring - if that is possible, have not really
> > checked
> > > yet to be honest.
> > >
> > > But what I wanted to try and do is recreate the current parsing with
> > > argparse as much as possible. And in the process simply adding
> synonyms,
> > > for example make the kafka-console-producer understand a
> > > bootstrap-parameter in addition to broker-list.
> > > There is a bit of custom logic about which parameters go together etc.
> in
> > > the current classes, so output may look different here and there, but
> in
> > > principle I do believe that it should be possible to recreate the
> current
> > > structure.
> >
> > Sounds like a good idea. Thanks for the clarification.
> >
> > >
> > > If there is an appetite for a new, hadoop-like entrypoint anyway, then
> > all
> > > of this might be "wasted" effort, or rather effort better spent though,
> > you
> > > are right.
> >
> > I don't think anyone is working on a new entry point right now -- or if
> > they are, they haven't said anything yet :)
> >
> > I just wanted to mention it as a possible approach in case you wanted to
> > do a bigger project.
> >
> > best,
> > Colin
> >
> > >
> > > Best regards,
> > > Sönke
> > >
> > >
> > >
> > > On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > > Hi Sönke,
> > > >
> > > > #2 is a bit tough because people have come to rely on the way the
> > commands
> > > > are structured right now.
> > > >
> > > > If we want to make big changes, it might be easier just to create a
> > > > separate tool and deprecate the old one(s). One thing we've talked
> > about
> > > > doing in the past is creating a single entry point for all the tool
> > > > functionality, kind of like hadoop did with the "hadoop" command Or
> > git
> > > > with the "git" command, etc. Then we could deprecate the standalone
> > > > commands and remove them after enough time had passed-- kind of like
> > the
> > > > old consumer.
> > > >
> > > > On the other hand, a more incremental change would be standardizing
> > flags
> > > > a bit. So for example, at least setting it up so that there is a
> > standard
> > > > way of supplying bootstrap brokers, etc. We could keep the old flags
> > > > around for a while as variants to ease the transition.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
> > > > > Hi Colin,
> > > > >
> > > > > I totally agree! Especially the differently named bootstrap server
> > > > options
> > > > > have been annoying me a long time.
> > > > >
> > > > > I'd propose a two-step approach:
> > > > > 1. Add new default options objects similar to CommandLineUtils and
> > > > > CommandDefaultOptions (based on argparse4j) but in the clients
> > project,
> > > > as
> > > > > this is referenced by all command line tools as far as I can tell
> > > > > 2. Refactor tools one by one to use these new helper classes (and
> > thus
> > > > > argparse) and add standardized synonyms for parameters as necessary
> > > > >
> > > > > I think for step 1 we can get away with no KIP, as this doesn't
> > change
> > > > any
> > > > > public interfaces or behavior.
> > > > > Step 2 probably needs a KIP as we are adding new parameters? We can
> > pick
> > > > up
> > > > > KIP-14 again for that I think. A lot of work has been done on that
> > > > already.
> > > > >
> > > > > Does that sound useful to everybody?
> > > > >
> > > > > Best regards,
> > > > > Sönke
> > > > >
> > > > >
> > > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > > If we are going to standardize on one argument parsing library,
> it
> > > > should
> > > > > > certainly be argparse4j, I think.
> > > > > > argparse4j is simply a better argument parsing library with
> > support
> > > > for
> > > > > > more features. One example is mutually exclusive options.
> > argparse4j
> > > > > > supports this with MutuallyExclusiveGroup. jopt doesn't support
> > this,
> > > > so
> > > > > > when it is needed, we have to add extra code to manually check
> that
> > > > > > mutually exclusive options are not set.
> > > > > >
> > > > > > argparse4j also has subcommands. If you want something like "git
> > add"
> > > > > > with some set of flags, and "git remove" with another, you can do
> > this
> > > > with
> > > > > > argparse4j, but not with jopt. This would be very helpful for
> > > > clearing up
> > > > > > confusion in a lot of our shell scripts which have accumulated
> > dozens
> > > > of
> > > > > > arguments, most of which are only relevant to a very specific
> > > > operation.
> > > > > > But you just can't do it with jopt.
> > > > > >
> > > > > > Just to give an example, argparse4j with subcommands would allow
> > you to
> > > > > > run something like ./kafka-topics.sh list --help and get just
> > options
> > > > that
> > > > > > were relevant for listing topics, not the full dozens of options
> > that
> > > > might
> > > > > > relate to adding topics, removing them, etc.
> > > > > >
> > > > > > To be honest, though, what would help users the most is
> > standardizing
> > > > the
> > > > > > option flags across tools. We should have a standard way of
> > specifying
> > > > > > bootstrap brokers, for example. (We can continue to support the
> > old
> > > > > > synonyms for a while, of course.)
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote:
> > > > > > > I took another look at the PR itself and I think it would be
> > great to
> > > > > > have
> > > > > > > this cleanup too -- I cannot remember at the beginning why we
> > > > gradually
> > > > > > > moved to different mechanism (argparse4j) for different cmds,
> if
> > > > there's
> > > > > > no
> > > > > > > rationales behind it we should just make them consistent.
> > > > > > >
> > > > > > > Thanks for driving this!
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <
> > ryannedolan@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Sönke, I'd find this very helpful. It's annoying to keep
> track
> > of
> > > > which
> > > > > > > > commands use which form -- I always seem to guess wrong.
> > > > > > > >
> > > > > > > > Though I don't think there is any reason to deprecate
> existing
> > > > forms,
> > > > > > e.g.
> > > > > > > > consumer.config vs consumer-config. I think it's perfectly
> > > > reasonable
> > > > > > to
> > > > > > > > have multiple spellings of the same arguments. I don't really
> > see a
> > > > > > > > downside to keeping the aliases around indefinitely.
> > > > > > > >
> > > > > > > > Ryanne
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau
> > > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > Hi everybody,
> > > > > > > > >
> > > > > > > > > Jason and I were recently discussing command line argument
> > > > parsing on
> > > > > > > > > KAFKA-8131 (or rather the related pull request) [1].
> > > > > > > > >
> > > > > > > > > Command line tools and their arguments are somewhat diverse
> > at
> > > > the
> > > > > > > > moment.
> > > > > > > > > Most of the tools use joptsimple for argument parsing, some
> > newer
> > > > > > java
> > > > > > > > > tools use argparse4j instead and some tools use nothing at
> > all.
> > > > > > > > > I've looked for a reason as to why there are two libraries
> > being
> > > > > > used,
> > > > > > > > but
> > > > > > > > > couldn't really find anything. Paolo brought up the same
> > > > question on
> > > > > > the
> > > > > > > > > mailing list a while back [7], but got no response either.
> > > > > > > > > Does anybody know why this is the case?
> > > > > > > > >
> > > > > > > > > This results in no central place to add universal
> parameters
> > like
> > > > > > help
> > > > > > > > and
> > > > > > > > > version, as well as the help output looking different
> between
> > > > some
> > > > > > of the
> > > > > > > > > tools.
> > > > > > > > > Also, there are a number of parameters that should be
> > renamed to
> > > > > > adhere
> > > > > > > > to
> > > > > > > > > defaults.
> > > > > > > > >
> > > > > > > > > There have been a few discussions and initiatives around
> > this in
> > > > the
> > > > > > > > past.
> > > > > > > > > Just of the top of my head (and a 5 minute jira search)
> there
> > > > are:
> > > > > > > > > - KIP-14 [2]
> > > > > > > > > - KAFKA-2111 [3]
> > > > > > > > > - KIP-316 [4]
> > > > > > > > > - KAFKA-1292 [5]
> > > > > > > > > - KAFKA-3530 [6]
> > > > > > > > > - and probably many more
> > > > > > > > >
> > > > > > > > > Would people generally be in favor of revisiting this
> topic?
> > > > > > > > >
> > > > > > > > > What I'd propose to do is:
> > > > > > > > > - comb through jira and KIPs, clean up old stuff and creae
> a
> > new
> > > > > > umbrella
> > > > > > > > > issue to track this (maybe reuse KIP-4 as well)
> > > > > > > > > - agree on one library for parsing command line arguments
> > (don't
> > > > care
> > > > > > > > which
> > > > > > > > > one, but two is one too many I think)
> > > > > > > > > - refactor tools to use one library and default way of
> > argument
> > > > > > parsing
> > > > > > > > > with central help and version parameter
> > > > > > > > > - add aliases for options that should be renamed according
> to
> > > > KIP-4
> > > > > > (and
> > > > > > > > > maybe others) so that both new and old work for a while,
> > > > deprecate
> > > > > > old
> > > > > > > > > parameters for a cycle or two and then remove them
> > > > > > > > >
> > > > > > > > > I'll shut up now and see if people would consider this
> > useful or
> > > > > > have any
> > > > > > > > > other input :)
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Sönke
> > > > > > > > >
> > > > > > > > > [1]
> > > > https://github.com/apache/kafka/pull/6481#discussion_r273773003
> > > > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131>
> > > > > > > > > [2]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
> > > > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111
> > > > > > > > > [4]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties
> > > > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292
> > > > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530
> > > > > > > > > [7]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
> https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sönke Liebau
> > > > > Partner
> > > > > Tel. +49 179 7940878
> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> Germany
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sönke Liebau
> > > Partner
> > > Tel. +49 179 7940878
> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > >
> >
>
Re: Cleaning up command line tools argument parsing a little
Posted by Sönke Liebau <so...@opencore.com.INVALID>.
Hi Colin,
thanks!
While I personally don't like the current command line tools I
realistically think we'll be stuck with them for a while yet, so a cleanup
might make sense.
So I'll start looking into that.
Regarding a central entrypoint, that would indeed be brilliant and I'd love
to work on that, but I currently have enough other open things to look at,
so I won't draw that one to myself as well for now.
But I'll keep it in mind for when some time frees up :)
Best regards,
Sönke
Colin McCabe <cm...@apache.org> schrieb am Di., 7. Mai 2019, 00:56:
> On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote:
> > Hi Colin,
> >
> > it was my intention to keep the structure of the commands mostly intact
> > while doing the refactoring - if that is possible, have not really
> checked
> > yet to be honest.
> >
> > But what I wanted to try and do is recreate the current parsing with
> > argparse as much as possible. And in the process simply adding synonyms,
> > for example make the kafka-console-producer understand a
> > bootstrap-parameter in addition to broker-list.
> > There is a bit of custom logic about which parameters go together etc. in
> > the current classes, so output may look different here and there, but in
> > principle I do believe that it should be possible to recreate the current
> > structure.
>
> Sounds like a good idea. Thanks for the clarification.
>
> >
> > If there is an appetite for a new, hadoop-like entrypoint anyway, then
> all
> > of this might be "wasted" effort, or rather effort better spent though,
> you
> > are right.
>
> I don't think anyone is working on a new entry point right now -- or if
> they are, they haven't said anything yet :)
>
> I just wanted to mention it as a possible approach in case you wanted to
> do a bigger project.
>
> best,
> Colin
>
> >
> > Best regards,
> > Sönke
> >
> >
> >
> > On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi Sönke,
> > >
> > > #2 is a bit tough because people have come to rely on the way the
> commands
> > > are structured right now.
> > >
> > > If we want to make big changes, it might be easier just to create a
> > > separate tool and deprecate the old one(s). One thing we've talked
> about
> > > doing in the past is creating a single entry point for all the tool
> > > functionality, kind of like hadoop did with the "hadoop" command Or
> git
> > > with the "git" command, etc. Then we could deprecate the standalone
> > > commands and remove them after enough time had passed-- kind of like
> the
> > > old consumer.
> > >
> > > On the other hand, a more incremental change would be standardizing
> flags
> > > a bit. So for example, at least setting it up so that there is a
> standard
> > > way of supplying bootstrap brokers, etc. We could keep the old flags
> > > around for a while as variants to ease the transition.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
> > > > Hi Colin,
> > > >
> > > > I totally agree! Especially the differently named bootstrap server
> > > options
> > > > have been annoying me a long time.
> > > >
> > > > I'd propose a two-step approach:
> > > > 1. Add new default options objects similar to CommandLineUtils and
> > > > CommandDefaultOptions (based on argparse4j) but in the clients
> project,
> > > as
> > > > this is referenced by all command line tools as far as I can tell
> > > > 2. Refactor tools one by one to use these new helper classes (and
> thus
> > > > argparse) and add standardized synonyms for parameters as necessary
> > > >
> > > > I think for step 1 we can get away with no KIP, as this doesn't
> change
> > > any
> > > > public interfaces or behavior.
> > > > Step 2 probably needs a KIP as we are adding new parameters? We can
> pick
> > > up
> > > > KIP-14 again for that I think. A lot of work has been done on that
> > > already.
> > > >
> > > > Does that sound useful to everybody?
> > > >
> > > > Best regards,
> > > > Sönke
> > > >
> > > >
> > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > > If we are going to standardize on one argument parsing library, it
> > > should
> > > > > certainly be argparse4j, I think.
> > > > > argparse4j is simply a better argument parsing library with
> support
> > > for
> > > > > more features. One example is mutually exclusive options.
> argparse4j
> > > > > supports this with MutuallyExclusiveGroup. jopt doesn't support
> this,
> > > so
> > > > > when it is needed, we have to add extra code to manually check that
> > > > > mutually exclusive options are not set.
> > > > >
> > > > > argparse4j also has subcommands. If you want something like "git
> add"
> > > > > with some set of flags, and "git remove" with another, you can do
> this
> > > with
> > > > > argparse4j, but not with jopt. This would be very helpful for
> > > clearing up
> > > > > confusion in a lot of our shell scripts which have accumulated
> dozens
> > > of
> > > > > arguments, most of which are only relevant to a very specific
> > > operation.
> > > > > But you just can't do it with jopt.
> > > > >
> > > > > Just to give an example, argparse4j with subcommands would allow
> you to
> > > > > run something like ./kafka-topics.sh list --help and get just
> options
> > > that
> > > > > were relevant for listing topics, not the full dozens of options
> that
> > > might
> > > > > relate to adding topics, removing them, etc.
> > > > >
> > > > > To be honest, though, what would help users the most is
> standardizing
> > > the
> > > > > option flags across tools. We should have a standard way of
> specifying
> > > > > bootstrap brokers, for example. (We can continue to support the
> old
> > > > > synonyms for a while, of course.)
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote:
> > > > > > I took another look at the PR itself and I think it would be
> great to
> > > > > have
> > > > > > this cleanup too -- I cannot remember at the beginning why we
> > > gradually
> > > > > > moved to different mechanism (argparse4j) for different cmds, if
> > > there's
> > > > > no
> > > > > > rationales behind it we should just make them consistent.
> > > > > >
> > > > > > Thanks for driving this!
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <
> ryannedolan@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Sönke, I'd find this very helpful. It's annoying to keep track
> of
> > > which
> > > > > > > commands use which form -- I always seem to guess wrong.
> > > > > > >
> > > > > > > Though I don't think there is any reason to deprecate existing
> > > forms,
> > > > > e.g.
> > > > > > > consumer.config vs consumer-config. I think it's perfectly
> > > reasonable
> > > > > to
> > > > > > > have multiple spellings of the same arguments. I don't really
> see a
> > > > > > > downside to keeping the aliases around indefinitely.
> > > > > > >
> > > > > > > Ryanne
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau
> > > > > > > <so...@opencore.com.invalid> wrote:
> > > > > > >
> > > > > > > > Hi everybody,
> > > > > > > >
> > > > > > > > Jason and I were recently discussing command line argument
> > > parsing on
> > > > > > > > KAFKA-8131 (or rather the related pull request) [1].
> > > > > > > >
> > > > > > > > Command line tools and their arguments are somewhat diverse
> at
> > > the
> > > > > > > moment.
> > > > > > > > Most of the tools use joptsimple for argument parsing, some
> newer
> > > > > java
> > > > > > > > tools use argparse4j instead and some tools use nothing at
> all.
> > > > > > > > I've looked for a reason as to why there are two libraries
> being
> > > > > used,
> > > > > > > but
> > > > > > > > couldn't really find anything. Paolo brought up the same
> > > question on
> > > > > the
> > > > > > > > mailing list a while back [7], but got no response either.
> > > > > > > > Does anybody know why this is the case?
> > > > > > > >
> > > > > > > > This results in no central place to add universal parameters
> like
> > > > > help
> > > > > > > and
> > > > > > > > version, as well as the help output looking different between
> > > some
> > > > > of the
> > > > > > > > tools.
> > > > > > > > Also, there are a number of parameters that should be
> renamed to
> > > > > adhere
> > > > > > > to
> > > > > > > > defaults.
> > > > > > > >
> > > > > > > > There have been a few discussions and initiatives around
> this in
> > > the
> > > > > > > past.
> > > > > > > > Just of the top of my head (and a 5 minute jira search) there
> > > are:
> > > > > > > > - KIP-14 [2]
> > > > > > > > - KAFKA-2111 [3]
> > > > > > > > - KIP-316 [4]
> > > > > > > > - KAFKA-1292 [5]
> > > > > > > > - KAFKA-3530 [6]
> > > > > > > > - and probably many more
> > > > > > > >
> > > > > > > > Would people generally be in favor of revisiting this topic?
> > > > > > > >
> > > > > > > > What I'd propose to do is:
> > > > > > > > - comb through jira and KIPs, clean up old stuff and creae a
> new
> > > > > umbrella
> > > > > > > > issue to track this (maybe reuse KIP-4 as well)
> > > > > > > > - agree on one library for parsing command line arguments
> (don't
> > > care
> > > > > > > which
> > > > > > > > one, but two is one too many I think)
> > > > > > > > - refactor tools to use one library and default way of
> argument
> > > > > parsing
> > > > > > > > with central help and version parameter
> > > > > > > > - add aliases for options that should be renamed according to
> > > KIP-4
> > > > > (and
> > > > > > > > maybe others) so that both new and old work for a while,
> > > deprecate
> > > > > old
> > > > > > > > parameters for a cycle or two and then remove them
> > > > > > > >
> > > > > > > > I'll shut up now and see if people would consider this
> useful or
> > > > > have any
> > > > > > > > other input :)
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Sönke
> > > > > > > >
> > > > > > > > [1]
> > > https://github.com/apache/kafka/pull/6481#discussion_r273773003
> > > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131>
> > > > > > > > [2]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
> > > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111
> > > > > > > > [4]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties
> > > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292
> > > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530
> > > > > > > > [7]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Sönke Liebau
> > > > Partner
> > > > Tel. +49 179 7940878
> > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > > >
> > >
> >
> >
> > --
> > Sönke Liebau
> > Partner
> > Tel. +49 179 7940878
> > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
>
Re: Cleaning up command line tools argument parsing a little
Posted by Colin McCabe <cm...@apache.org>.
On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote:
> Hi Colin,
>
> it was my intention to keep the structure of the commands mostly intact
> while doing the refactoring - if that is possible, have not really checked
> yet to be honest.
>
> But what I wanted to try and do is recreate the current parsing with
> argparse as much as possible. And in the process simply adding synonyms,
> for example make the kafka-console-producer understand a
> bootstrap-parameter in addition to broker-list.
> There is a bit of custom logic about which parameters go together etc. in
> the current classes, so output may look different here and there, but in
> principle I do believe that it should be possible to recreate the current
> structure.
Sounds like a good idea. Thanks for the clarification.
>
> If there is an appetite for a new, hadoop-like entrypoint anyway, then all
> of this might be "wasted" effort, or rather effort better spent though, you
> are right.
I don't think anyone is working on a new entry point right now -- or if they are, they haven't said anything yet :)
I just wanted to mention it as a possible approach in case you wanted to do a bigger project.
best,
Colin
>
> Best regards,
> Sönke
>
>
>
> On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Sönke,
> >
> > #2 is a bit tough because people have come to rely on the way the commands
> > are structured right now.
> >
> > If we want to make big changes, it might be easier just to create a
> > separate tool and deprecate the old one(s). One thing we've talked about
> > doing in the past is creating a single entry point for all the tool
> > functionality, kind of like hadoop did with the "hadoop" command Or git
> > with the "git" command, etc. Then we could deprecate the standalone
> > commands and remove them after enough time had passed-- kind of like the
> > old consumer.
> >
> > On the other hand, a more incremental change would be standardizing flags
> > a bit. So for example, at least setting it up so that there is a standard
> > way of supplying bootstrap brokers, etc. We could keep the old flags
> > around for a while as variants to ease the transition.
> >
> > best,
> > Colin
> >
> >
> > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
> > > Hi Colin,
> > >
> > > I totally agree! Especially the differently named bootstrap server
> > options
> > > have been annoying me a long time.
> > >
> > > I'd propose a two-step approach:
> > > 1. Add new default options objects similar to CommandLineUtils and
> > > CommandDefaultOptions (based on argparse4j) but in the clients project,
> > as
> > > this is referenced by all command line tools as far as I can tell
> > > 2. Refactor tools one by one to use these new helper classes (and thus
> > > argparse) and add standardized synonyms for parameters as necessary
> > >
> > > I think for step 1 we can get away with no KIP, as this doesn't change
> > any
> > > public interfaces or behavior.
> > > Step 2 probably needs a KIP as we are adding new parameters? We can pick
> > up
> > > KIP-14 again for that I think. A lot of work has been done on that
> > already.
> > >
> > > Does that sound useful to everybody?
> > >
> > > Best regards,
> > > Sönke
> > >
> > >
> > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > If we are going to standardize on one argument parsing library, it
> > should
> > > > certainly be argparse4j, I think.
> > > > argparse4j is simply a better argument parsing library with support
> > for
> > > > more features. One example is mutually exclusive options. argparse4j
> > > > supports this with MutuallyExclusiveGroup. jopt doesn't support this,
> > so
> > > > when it is needed, we have to add extra code to manually check that
> > > > mutually exclusive options are not set.
> > > >
> > > > argparse4j also has subcommands. If you want something like "git add"
> > > > with some set of flags, and "git remove" with another, you can do this
> > with
> > > > argparse4j, but not with jopt. This would be very helpful for
> > clearing up
> > > > confusion in a lot of our shell scripts which have accumulated dozens
> > of
> > > > arguments, most of which are only relevant to a very specific
> > operation.
> > > > But you just can't do it with jopt.
> > > >
> > > > Just to give an example, argparse4j with subcommands would allow you to
> > > > run something like ./kafka-topics.sh list --help and get just options
> > that
> > > > were relevant for listing topics, not the full dozens of options that
> > might
> > > > relate to adding topics, removing them, etc.
> > > >
> > > > To be honest, though, what would help users the most is standardizing
> > the
> > > > option flags across tools. We should have a standard way of specifying
> > > > bootstrap brokers, for example. (We can continue to support the old
> > > > synonyms for a while, of course.)
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote:
> > > > > I took another look at the PR itself and I think it would be great to
> > > > have
> > > > > this cleanup too -- I cannot remember at the beginning why we
> > gradually
> > > > > moved to different mechanism (argparse4j) for different cmds, if
> > there's
> > > > no
> > > > > rationales behind it we should just make them consistent.
> > > > >
> > > > > Thanks for driving this!
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <ry...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Sönke, I'd find this very helpful. It's annoying to keep track of
> > which
> > > > > > commands use which form -- I always seem to guess wrong.
> > > > > >
> > > > > > Though I don't think there is any reason to deprecate existing
> > forms,
> > > > e.g.
> > > > > > consumer.config vs consumer-config. I think it's perfectly
> > reasonable
> > > > to
> > > > > > have multiple spellings of the same arguments. I don't really see a
> > > > > > downside to keeping the aliases around indefinitely.
> > > > > >
> > > > > > Ryanne
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau
> > > > > > <so...@opencore.com.invalid> wrote:
> > > > > >
> > > > > > > Hi everybody,
> > > > > > >
> > > > > > > Jason and I were recently discussing command line argument
> > parsing on
> > > > > > > KAFKA-8131 (or rather the related pull request) [1].
> > > > > > >
> > > > > > > Command line tools and their arguments are somewhat diverse at
> > the
> > > > > > moment.
> > > > > > > Most of the tools use joptsimple for argument parsing, some newer
> > > > java
> > > > > > > tools use argparse4j instead and some tools use nothing at all.
> > > > > > > I've looked for a reason as to why there are two libraries being
> > > > used,
> > > > > > but
> > > > > > > couldn't really find anything. Paolo brought up the same
> > question on
> > > > the
> > > > > > > mailing list a while back [7], but got no response either.
> > > > > > > Does anybody know why this is the case?
> > > > > > >
> > > > > > > This results in no central place to add universal parameters like
> > > > help
> > > > > > and
> > > > > > > version, as well as the help output looking different between
> > some
> > > > of the
> > > > > > > tools.
> > > > > > > Also, there are a number of parameters that should be renamed to
> > > > adhere
> > > > > > to
> > > > > > > defaults.
> > > > > > >
> > > > > > > There have been a few discussions and initiatives around this in
> > the
> > > > > > past.
> > > > > > > Just of the top of my head (and a 5 minute jira search) there
> > are:
> > > > > > > - KIP-14 [2]
> > > > > > > - KAFKA-2111 [3]
> > > > > > > - KIP-316 [4]
> > > > > > > - KAFKA-1292 [5]
> > > > > > > - KAFKA-3530 [6]
> > > > > > > - and probably many more
> > > > > > >
> > > > > > > Would people generally be in favor of revisiting this topic?
> > > > > > >
> > > > > > > What I'd propose to do is:
> > > > > > > - comb through jira and KIPs, clean up old stuff and creae a new
> > > > umbrella
> > > > > > > issue to track this (maybe reuse KIP-4 as well)
> > > > > > > - agree on one library for parsing command line arguments (don't
> > care
> > > > > > which
> > > > > > > one, but two is one too many I think)
> > > > > > > - refactor tools to use one library and default way of argument
> > > > parsing
> > > > > > > with central help and version parameter
> > > > > > > - add aliases for options that should be renamed according to
> > KIP-4
> > > > (and
> > > > > > > maybe others) so that both new and old work for a while,
> > deprecate
> > > > old
> > > > > > > parameters for a cycle or two and then remove them
> > > > > > >
> > > > > > > I'll shut up now and see if people would consider this useful or
> > > > have any
> > > > > > > other input :)
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Sönke
> > > > > > >
> > > > > > > [1]
> > https://github.com/apache/kafka/pull/6481#discussion_r273773003
> > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131>
> > > > > > > [2]
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
> > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111
> > > > > > > [4]
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties
> > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292
> > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530
> > > > > > > [7]
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sönke Liebau
> > > Partner
> > > Tel. +49 179 7940878
> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > >
> >
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>
Re: Cleaning up command line tools argument parsing a little
Posted by Sönke Liebau <so...@opencore.com.INVALID>.
Hi Colin,
it was my intention to keep the structure of the commands mostly intact
while doing the refactoring - if that is possible, have not really checked
yet to be honest.
But what I wanted to try and do is recreate the current parsing with
argparse as much as possible. And in the process simply adding synonyms,
for example make the kafka-console-producer understand a
bootstrap-parameter in addition to broker-list.
There is a bit of custom logic about which parameters go together etc. in
the current classes, so output may look different here and there, but in
principle I do believe that it should be possible to recreate the current
structure.
If there is an appetite for a new, hadoop-like entrypoint anyway, then all
of this might be "wasted" effort, or rather effort better spent though, you
are right.
Best regards,
Sönke
On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cm...@apache.org> wrote:
> Hi Sönke,
>
> #2 is a bit tough because people have come to rely on the way the commands
> are structured right now.
>
> If we want to make big changes, it might be easier just to create a
> separate tool and deprecate the old one(s). One thing we've talked about
> doing in the past is creating a single entry point for all the tool
> functionality, kind of like hadoop did with the "hadoop" command Or git
> with the "git" command, etc. Then we could deprecate the standalone
> commands and remove them after enough time had passed-- kind of like the
> old consumer.
>
> On the other hand, a more incremental change would be standardizing flags
> a bit. So for example, at least setting it up so that there is a standard
> way of supplying bootstrap brokers, etc. We could keep the old flags
> around for a while as variants to ease the transition.
>
> best,
> Colin
>
>
> On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
> > Hi Colin,
> >
> > I totally agree! Especially the differently named bootstrap server
> options
> > have been annoying me a long time.
> >
> > I'd propose a two-step approach:
> > 1. Add new default options objects similar to CommandLineUtils and
> > CommandDefaultOptions (based on argparse4j) but in the clients project,
> as
> > this is referenced by all command line tools as far as I can tell
> > 2. Refactor tools one by one to use these new helper classes (and thus
> > argparse) and add standardized synonyms for parameters as necessary
> >
> > I think for step 1 we can get away with no KIP, as this doesn't change
> any
> > public interfaces or behavior.
> > Step 2 probably needs a KIP as we are adding new parameters? We can pick
> up
> > KIP-14 again for that I think. A lot of work has been done on that
> already.
> >
> > Does that sound useful to everybody?
> >
> > Best regards,
> > Sönke
> >
> >
> > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > If we are going to standardize on one argument parsing library, it
> should
> > > certainly be argparse4j, I think.
> > > argparse4j is simply a better argument parsing library with support
> for
> > > more features. One example is mutually exclusive options. argparse4j
> > > supports this with MutuallyExclusiveGroup. jopt doesn't support this,
> so
> > > when it is needed, we have to add extra code to manually check that
> > > mutually exclusive options are not set.
> > >
> > > argparse4j also has subcommands. If you want something like "git add"
> > > with some set of flags, and "git remove" with another, you can do this
> with
> > > argparse4j, but not with jopt. This would be very helpful for
> clearing up
> > > confusion in a lot of our shell scripts which have accumulated dozens
> of
> > > arguments, most of which are only relevant to a very specific
> operation.
> > > But you just can't do it with jopt.
> > >
> > > Just to give an example, argparse4j with subcommands would allow you to
> > > run something like ./kafka-topics.sh list --help and get just options
> that
> > > were relevant for listing topics, not the full dozens of options that
> might
> > > relate to adding topics, removing them, etc.
> > >
> > > To be honest, though, what would help users the most is standardizing
> the
> > > option flags across tools. We should have a standard way of specifying
> > > bootstrap brokers, for example. (We can continue to support the old
> > > synonyms for a while, of course.)
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote:
> > > > I took another look at the PR itself and I think it would be great to
> > > have
> > > > this cleanup too -- I cannot remember at the beginning why we
> gradually
> > > > moved to different mechanism (argparse4j) for different cmds, if
> there's
> > > no
> > > > rationales behind it we should just make them consistent.
> > > >
> > > > Thanks for driving this!
> > > >
> > > > Guozhang
> > > >
> > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <ry...@gmail.com>
> > > wrote:
> > > >
> > > > > Sönke, I'd find this very helpful. It's annoying to keep track of
> which
> > > > > commands use which form -- I always seem to guess wrong.
> > > > >
> > > > > Though I don't think there is any reason to deprecate existing
> forms,
> > > e.g.
> > > > > consumer.config vs consumer-config. I think it's perfectly
> reasonable
> > > to
> > > > > have multiple spellings of the same arguments. I don't really see a
> > > > > downside to keeping the aliases around indefinitely.
> > > > >
> > > > > Ryanne
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau
> > > > > <so...@opencore.com.invalid> wrote:
> > > > >
> > > > > > Hi everybody,
> > > > > >
> > > > > > Jason and I were recently discussing command line argument
> parsing on
> > > > > > KAFKA-8131 (or rather the related pull request) [1].
> > > > > >
> > > > > > Command line tools and their arguments are somewhat diverse at
> the
> > > > > moment.
> > > > > > Most of the tools use joptsimple for argument parsing, some newer
> > > java
> > > > > > tools use argparse4j instead and some tools use nothing at all.
> > > > > > I've looked for a reason as to why there are two libraries being
> > > used,
> > > > > but
> > > > > > couldn't really find anything. Paolo brought up the same
> question on
> > > the
> > > > > > mailing list a while back [7], but got no response either.
> > > > > > Does anybody know why this is the case?
> > > > > >
> > > > > > This results in no central place to add universal parameters like
> > > help
> > > > > and
> > > > > > version, as well as the help output looking different between
> some
> > > of the
> > > > > > tools.
> > > > > > Also, there are a number of parameters that should be renamed to
> > > adhere
> > > > > to
> > > > > > defaults.
> > > > > >
> > > > > > There have been a few discussions and initiatives around this in
> the
> > > > > past.
> > > > > > Just of the top of my head (and a 5 minute jira search) there
> are:
> > > > > > - KIP-14 [2]
> > > > > > - KAFKA-2111 [3]
> > > > > > - KIP-316 [4]
> > > > > > - KAFKA-1292 [5]
> > > > > > - KAFKA-3530 [6]
> > > > > > - and probably many more
> > > > > >
> > > > > > Would people generally be in favor of revisiting this topic?
> > > > > >
> > > > > > What I'd propose to do is:
> > > > > > - comb through jira and KIPs, clean up old stuff and creae a new
> > > umbrella
> > > > > > issue to track this (maybe reuse KIP-4 as well)
> > > > > > - agree on one library for parsing command line arguments (don't
> care
> > > > > which
> > > > > > one, but two is one too many I think)
> > > > > > - refactor tools to use one library and default way of argument
> > > parsing
> > > > > > with central help and version parameter
> > > > > > - add aliases for options that should be renamed according to
> KIP-4
> > > (and
> > > > > > maybe others) so that both new and old work for a while,
> deprecate
> > > old
> > > > > > parameters for a cycle or two and then remove them
> > > > > >
> > > > > > I'll shut up now and see if people would consider this useful or
> > > have any
> > > > > > other input :)
> > > > > >
> > > > > > Best regards,
> > > > > > Sönke
> > > > > >
> > > > > > [1]
> https://github.com/apache/kafka/pull/6481#discussion_r273773003
> > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131>
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
> > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111
> > > > > > [4]
> > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties
> > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292
> > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530
> > > > > > [7]
> > > > > >
> > > > > >
> > > > >
> > >
> https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> > --
> > Sönke Liebau
> > Partner
> > Tel. +49 179 7940878
> > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
Re: Cleaning up command line tools argument parsing a little
Posted by Colin McCabe <cm...@apache.org>.
Hi Sönke,
#2 is a bit tough because people have come to rely on the way the commands are structured right now.
If we want to make big changes, it might be easier just to create a separate tool and deprecate the old one(s). One thing we've talked about doing in the past is creating a single entry point for all the tool functionality, kind of like hadoop did with the "hadoop" command Or git with the "git" command, etc. Then we could deprecate the standalone commands and remove them after enough time had passed-- kind of like the old consumer.
On the other hand, a more incremental change would be standardizing flags a bit. So for example, at least setting it up so that there is a standard way of supplying bootstrap brokers, etc. We could keep the old flags around for a while as variants to ease the transition.
best,
Colin
On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
> Hi Colin,
>
> I totally agree! Especially the differently named bootstrap server options
> have been annoying me a long time.
>
> I'd propose a two-step approach:
> 1. Add new default options objects similar to CommandLineUtils and
> CommandDefaultOptions (based on argparse4j) but in the clients project, as
> this is referenced by all command line tools as far as I can tell
> 2. Refactor tools one by one to use these new helper classes (and thus
> argparse) and add standardized synonyms for parameters as necessary
>
> I think for step 1 we can get away with no KIP, as this doesn't change any
> public interfaces or behavior.
> Step 2 probably needs a KIP as we are adding new parameters? We can pick up
> KIP-14 again for that I think. A lot of work has been done on that already.
>
> Does that sound useful to everybody?
>
> Best regards,
> Sönke
>
>
> On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cm...@apache.org> wrote:
>
> > If we are going to standardize on one argument parsing library, it should
> > certainly be argparse4j, I think.
> > argparse4j is simply a better argument parsing library with support for
> > more features. One example is mutually exclusive options. argparse4j
> > supports this with MutuallyExclusiveGroup. jopt doesn't support this, so
> > when it is needed, we have to add extra code to manually check that
> > mutually exclusive options are not set.
> >
> > argparse4j also has subcommands. If you want something like "git add"
> > with some set of flags, and "git remove" with another, you can do this with
> > argparse4j, but not with jopt. This would be very helpful for clearing up
> > confusion in a lot of our shell scripts which have accumulated dozens of
> > arguments, most of which are only relevant to a very specific operation.
> > But you just can't do it with jopt.
> >
> > Just to give an example, argparse4j with subcommands would allow you to
> > run something like ./kafka-topics.sh list --help and get just options that
> > were relevant for listing topics, not the full dozens of options that might
> > relate to adding topics, removing them, etc.
> >
> > To be honest, though, what would help users the most is standardizing the
> > option flags across tools. We should have a standard way of specifying
> > bootstrap brokers, for example. (We can continue to support the old
> > synonyms for a while, of course.)
> >
> > best,
> > Colin
> >
> >
> > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote:
> > > I took another look at the PR itself and I think it would be great to
> > have
> > > this cleanup too -- I cannot remember at the beginning why we gradually
> > > moved to different mechanism (argparse4j) for different cmds, if there's
> > no
> > > rationales behind it we should just make them consistent.
> > >
> > > Thanks for driving this!
> > >
> > > Guozhang
> > >
> > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan <ry...@gmail.com>
> > wrote:
> > >
> > > > Sönke, I'd find this very helpful. It's annoying to keep track of which
> > > > commands use which form -- I always seem to guess wrong.
> > > >
> > > > Though I don't think there is any reason to deprecate existing forms,
> > e.g.
> > > > consumer.config vs consumer-config. I think it's perfectly reasonable
> > to
> > > > have multiple spellings of the same arguments. I don't really see a
> > > > downside to keeping the aliases around indefinitely.
> > > >
> > > > Ryanne
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau
> > > > <so...@opencore.com.invalid> wrote:
> > > >
> > > > > Hi everybody,
> > > > >
> > > > > Jason and I were recently discussing command line argument parsing on
> > > > > KAFKA-8131 (or rather the related pull request) [1].
> > > > >
> > > > > Command line tools and their arguments are somewhat diverse at the
> > > > moment.
> > > > > Most of the tools use joptsimple for argument parsing, some newer
> > java
> > > > > tools use argparse4j instead and some tools use nothing at all.
> > > > > I've looked for a reason as to why there are two libraries being
> > used,
> > > > but
> > > > > couldn't really find anything. Paolo brought up the same question on
> > the
> > > > > mailing list a while back [7], but got no response either.
> > > > > Does anybody know why this is the case?
> > > > >
> > > > > This results in no central place to add universal parameters like
> > help
> > > > and
> > > > > version, as well as the help output looking different between some
> > of the
> > > > > tools.
> > > > > Also, there are a number of parameters that should be renamed to
> > adhere
> > > > to
> > > > > defaults.
> > > > >
> > > > > There have been a few discussions and initiatives around this in the
> > > > past.
> > > > > Just of the top of my head (and a 5 minute jira search) there are:
> > > > > - KIP-14 [2]
> > > > > - KAFKA-2111 [3]
> > > > > - KIP-316 [4]
> > > > > - KAFKA-1292 [5]
> > > > > - KAFKA-3530 [6]
> > > > > - and probably many more
> > > > >
> > > > > Would people generally be in favor of revisiting this topic?
> > > > >
> > > > > What I'd propose to do is:
> > > > > - comb through jira and KIPs, clean up old stuff and creae a new
> > umbrella
> > > > > issue to track this (maybe reuse KIP-4 as well)
> > > > > - agree on one library for parsing command line arguments (don't care
> > > > which
> > > > > one, but two is one too many I think)
> > > > > - refactor tools to use one library and default way of argument
> > parsing
> > > > > with central help and version parameter
> > > > > - add aliases for options that should be renamed according to KIP-4
> > (and
> > > > > maybe others) so that both new and old work for a while, deprecate
> > old
> > > > > parameters for a cycle or two and then remove them
> > > > >
> > > > > I'll shut up now and see if people would consider this useful or
> > have any
> > > > > other input :)
> > > > >
> > > > > Best regards,
> > > > > Sönke
> > > > >
> > > > > [1] https://github.com/apache/kafka/pull/6481#discussion_r273773003
> > > > > <https://issues.apache.org/jira/browse/KAFKA-8131>
> > > > > [2]
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
> > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111
> > > > > [4]
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties
> > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292
> > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530
> > > > > [7]
> > > > >
> > > > >
> > > >
> > https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>