You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Paolo Patierno <pp...@live.com> on 2017/08/01 14:30:21 UTC
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Hi Tom,
thanks for your reply. I think that you are right and what you have proposed should be the way to go.
Until today I have been working on refactoring the TopicCommand tool in Java using the AdminClient getting rid of the Zookeeper usage in only "one step" and maybe it's wrong.
I'd like to have some input from committers as well to be sure that the way is good about how handling such use cases.
Thanks,
Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor
Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>
________________________________
From: Tom Bentley <t....@gmail.com>
Sent: Friday, July 28, 2017 10:36 AM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API
Hi Paolo,
Replies in line...
On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
> Hi committers,
>
> in my understanding there is the common idea to move all tools from Scala
> to Java and then using the new Admin Client API instead of using the
> Zookeeper connection.
>
> Regarding this subject I started to work on refactoring the TopicCommand
> tool but with two steps at same time :
>
>
> * re-writing it in Java
> * removing --zookeeper option (so no Zookeeper connection) and adding
> --broker-list option (so using the Admin Client API)
>
> Of course, such option substitution is a breaking change for the tool (and
> the users who are using it).
> In such a scenario, the two steps path should be needed : first
> deprecation, then removal (for the --zookeeper option)
>
A change to tools (and their options) requires a KIP. There's no
fundamental reason why both couldn't be supported during a transition
period. So I doubt a KIP that didn't propose a transition period would get
passed.
It seems that in the "deprecation" phase we have two possible solutions :
>
>
> 1. Adding Admin Client API to the Scala tools and so the --broker-list
> option and a warning on --zookeeper for deprecation
> 2. Rewriting the tool in Java using the Admin Client API (so
> --broker-list) but at same time providing --zookeeper as well (so Zookeeper
> connection)
>
> With the solution 2) we have the advantage to have the tool already in
> Java when the --zookeeper option will be removed in a next step. In any
> case we have to write the part related to use the Admin Client API so it
> make more sense to me option 2) porting the Zookeeper needed code from
> Scala to Java (then removing it in the next phase).
>
> Is my understanding right on how we have to handle deprecation and removal
> for something that is a breaking change ?
> Or this case is something "special" and we can live with a new Java based
> tool without zookeeper but with Admin Client API at same time ?
>
> Of course having both Admin Client API and Zookeeper utils working at the
> same time in the tools means more complexity in the code but it's something
> that could be factorized.
>
I think the right thing to do would be:
1. deprecate the old option, adding support for the replacement option
(using the AdminClient). Keep the code in scala. All this is in one KIP.
2. Remove the old option (needs a KIP)
3. Rewrite the tool in Java.
Parts 2 and 3 could be done at the same time. I don't believe part 3 needs
a KIP if it were a drop-in replacement.
The reason I think this is the right way to proceed is:
* It gives users a transition period to learn about the new option, and
replace any tooling of their own.
* By keeping the tool in scala you get to release your new AdminClient API
and get to iron out all the creases while users still have the --zookeeper
option as a fallback.
* Then when you know the AdminClient API works in the field you have a
straight porting job to do, and you have less code to port because you
don't have to port the code to support --zookeeper.
But I'm fairly new here, so maybe a committer will chime in an correct me
where I'm wrong!
Cheers,
Tom
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Paolo Patierno <pp...@live.com>.
Hi Ismael,
in this case I embrace your way :-)
To be more specific on what I was doing, It means that I can continue to develop the TopicCommand tool in Java using AdminClient API and then adding the deprecation for --zookeeper and the "tool switch" in the shell script. It doesn't need a KIP, does it ?
For the future --zookeeper removal, a KIP would be needed and I'll write that. Because it's something common to more tools, does it make sense to have only one KIP or writing a KIP for each effected tool ? I guess the second option because each KIP will have effect on a specific version (tools migration will be done over more releases). Is that right ?
Thanks,
Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor
Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>
________________________________
From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <is...@juma.me.uk>
Sent: Tuesday, August 1, 2017 4:46 PM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API
Hi Paolo,
If the old tool is deprecated, then it's OK to add functionality to the new
tool without adding it to the old tool as long as the old tool still works.
So, there is some maintenance overhead to keep it working (but not to add
new features). In practice, the code used by the old tool is shared with
the broker so the maintenance overhead should be small.
Ismael
On Tue, Aug 1, 2017 at 5:12 PM, Paolo Patierno <pp...@live.com> wrote:
> Hi Ismael,
>
>
> your third option is really appealing for me :-)
>
>
> Just one thought about that ...
>
> If we decide to go in this way we could have an 1.0 release with both
> Scala (zookeeper) and Java (bootstrap-server) implementations but then the
> shell script to chose one of them with the --zookeeper deprecation.
>
> The final Scala (zookeeper) removal will happen on version 2.0 (following
> the policies about "major.minor.patch" release) but it could be months and
> months later.
>
> In the meantime, let's suppose that we want to add a new --foo option to
> the tool; we should do that on both implementations having them in sync and
> it could be a problem until on 2.0 release when we get rid of the Scala
> implementation.
>
> I really like your way but it could be better if we agree that new command
> options will be added only to the Java (bootstrap-server) implementation.
> It could mean pushing users to move from old to new tool ... that is good
> for us.
>
>
> What do you think ?
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <
> ismael@juma.me.uk>
> Sent: Tuesday, August 1, 2017 3:00 PM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Another option is to write the new tool in Java without support for
> `--zookeeper` and include some logic in the shell script to pick the
> implementation based on the presence of `--bootstrap-server` or
> `--zookeeper`. This would mean that we can deprecate the Scala tool while
> still supporting it for existing users.
>
> I think this would involve the least amount of rewriting once we drop
> support for `--zookeeper`.
>
> Thoughts?
>
> Ismael
>
> On Tue, Aug 1, 2017 at 3:30 PM, Paolo Patierno <pp...@live.com> wrote:
>
> > Hi Tom,
> >
> >
> > thanks for your reply. I think that you are right and what you have
> > proposed should be the way to go.
> >
> >
> > Until today I have been working on refactoring the TopicCommand tool in
> > Java using the AdminClient getting rid of the Zookeeper usage in only
> "one
> > step" and maybe it's wrong.
> >
> > I'd like to have some input from committers as well to be sure that the
> > way is good about how handling such use cases.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > ________________________________
> > From: Tom Bentley <t....@gmail.com>
> > Sent: Friday, July 28, 2017 10:36 AM
> > To: dev@kafka.apache.org
> > Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> > Admin Client API
> >
> > Hi Paolo,
> >
> > Replies in line...
> >
> > On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
> >
> > > Hi committers,
> > >
> > > in my understanding there is the common idea to move all tools from
> Scala
> > > to Java and then using the new Admin Client API instead of using the
> > > Zookeeper connection.
> > >
> > > Regarding this subject I started to work on refactoring the
> TopicCommand
> > > tool but with two steps at same time :
> > >
> > >
> > > * re-writing it in Java
> > > * removing --zookeeper option (so no Zookeeper connection) and
> adding
> > > --broker-list option (so using the Admin Client API)
> > >
> > > Of course, such option substitution is a breaking change for the tool
> > (and
> > > the users who are using it).
> > > In such a scenario, the two steps path should be needed : first
> > > deprecation, then removal (for the --zookeeper option)
> > >
> >
> > A change to tools (and their options) requires a KIP. There's no
> > fundamental reason why both couldn't be supported during a transition
> > period. So I doubt a KIP that didn't propose a transition period would
> get
> > passed.
> >
> >
> > It seems that in the "deprecation" phase we have two possible solutions :
> > >
> > >
> > > 1. Adding Admin Client API to the Scala tools and so the
> --broker-list
> > > option and a warning on --zookeeper for deprecation
> > > 2. Rewriting the tool in Java using the Admin Client API (so
> > > --broker-list) but at same time providing --zookeeper as well (so
> > Zookeeper
> > > connection)
> > >
> > > With the solution 2) we have the advantage to have the tool already in
> > > Java when the --zookeeper option will be removed in a next step. In any
> > > case we have to write the part related to use the Admin Client API so
> it
> > > make more sense to me option 2) porting the Zookeeper needed code from
> > > Scala to Java (then removing it in the next phase).
> > >
> > > Is my understanding right on how we have to handle deprecation and
> > removal
> > > for something that is a breaking change ?
> > > Or this case is something "special" and we can live with a new Java
> based
> > > tool without zookeeper but with Admin Client API at same time ?
> > >
> > > Of course having both Admin Client API and Zookeeper utils working at
> the
> > > same time in the tools means more complexity in the code but it's
> > something
> > > that could be factorized.
> > >
> >
> > I think the right thing to do would be:
> >
> > 1. deprecate the old option, adding support for the replacement option
> > (using the AdminClient). Keep the code in scala. All this is in one KIP.
> > 2. Remove the old option (needs a KIP)
> > 3. Rewrite the tool in Java.
> >
> > Parts 2 and 3 could be done at the same time. I don't believe part 3
> needs
> > a KIP if it were a drop-in replacement.
> >
> > The reason I think this is the right way to proceed is:
> >
> > * It gives users a transition period to learn about the new option, and
> > replace any tooling of their own.
> > * By keeping the tool in scala you get to release your new AdminClient
> API
> > and get to iron out all the creases while users still have the
> --zookeeper
> > option as a fallback.
> > * Then when you know the AdminClient API works in the field you have a
> > straight porting job to do, and you have less code to port because you
> > don't have to port the code to support --zookeeper.
> >
> > But I'm fairly new here, so maybe a committer will chime in an correct me
> > where I'm wrong!
> >
> > Cheers,
> >
> > Tom
> >
>
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Paolo,
If the old tool is deprecated, then it's OK to add functionality to the new
tool without adding it to the old tool as long as the old tool still works.
So, there is some maintenance overhead to keep it working (but not to add
new features). In practice, the code used by the old tool is shared with
the broker so the maintenance overhead should be small.
Ismael
On Tue, Aug 1, 2017 at 5:12 PM, Paolo Patierno <pp...@live.com> wrote:
> Hi Ismael,
>
>
> your third option is really appealing for me :-)
>
>
> Just one thought about that ...
>
> If we decide to go in this way we could have an 1.0 release with both
> Scala (zookeeper) and Java (bootstrap-server) implementations but then the
> shell script to chose one of them with the --zookeeper deprecation.
>
> The final Scala (zookeeper) removal will happen on version 2.0 (following
> the policies about "major.minor.patch" release) but it could be months and
> months later.
>
> In the meantime, let's suppose that we want to add a new --foo option to
> the tool; we should do that on both implementations having them in sync and
> it could be a problem until on 2.0 release when we get rid of the Scala
> implementation.
>
> I really like your way but it could be better if we agree that new command
> options will be added only to the Java (bootstrap-server) implementation.
> It could mean pushing users to move from old to new tool ... that is good
> for us.
>
>
> What do you think ?
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <
> ismael@juma.me.uk>
> Sent: Tuesday, August 1, 2017 3:00 PM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Another option is to write the new tool in Java without support for
> `--zookeeper` and include some logic in the shell script to pick the
> implementation based on the presence of `--bootstrap-server` or
> `--zookeeper`. This would mean that we can deprecate the Scala tool while
> still supporting it for existing users.
>
> I think this would involve the least amount of rewriting once we drop
> support for `--zookeeper`.
>
> Thoughts?
>
> Ismael
>
> On Tue, Aug 1, 2017 at 3:30 PM, Paolo Patierno <pp...@live.com> wrote:
>
> > Hi Tom,
> >
> >
> > thanks for your reply. I think that you are right and what you have
> > proposed should be the way to go.
> >
> >
> > Until today I have been working on refactoring the TopicCommand tool in
> > Java using the AdminClient getting rid of the Zookeeper usage in only
> "one
> > step" and maybe it's wrong.
> >
> > I'd like to have some input from committers as well to be sure that the
> > way is good about how handling such use cases.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > ________________________________
> > From: Tom Bentley <t....@gmail.com>
> > Sent: Friday, July 28, 2017 10:36 AM
> > To: dev@kafka.apache.org
> > Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> > Admin Client API
> >
> > Hi Paolo,
> >
> > Replies in line...
> >
> > On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
> >
> > > Hi committers,
> > >
> > > in my understanding there is the common idea to move all tools from
> Scala
> > > to Java and then using the new Admin Client API instead of using the
> > > Zookeeper connection.
> > >
> > > Regarding this subject I started to work on refactoring the
> TopicCommand
> > > tool but with two steps at same time :
> > >
> > >
> > > * re-writing it in Java
> > > * removing --zookeeper option (so no Zookeeper connection) and
> adding
> > > --broker-list option (so using the Admin Client API)
> > >
> > > Of course, such option substitution is a breaking change for the tool
> > (and
> > > the users who are using it).
> > > In such a scenario, the two steps path should be needed : first
> > > deprecation, then removal (for the --zookeeper option)
> > >
> >
> > A change to tools (and their options) requires a KIP. There's no
> > fundamental reason why both couldn't be supported during a transition
> > period. So I doubt a KIP that didn't propose a transition period would
> get
> > passed.
> >
> >
> > It seems that in the "deprecation" phase we have two possible solutions :
> > >
> > >
> > > 1. Adding Admin Client API to the Scala tools and so the
> --broker-list
> > > option and a warning on --zookeeper for deprecation
> > > 2. Rewriting the tool in Java using the Admin Client API (so
> > > --broker-list) but at same time providing --zookeeper as well (so
> > Zookeeper
> > > connection)
> > >
> > > With the solution 2) we have the advantage to have the tool already in
> > > Java when the --zookeeper option will be removed in a next step. In any
> > > case we have to write the part related to use the Admin Client API so
> it
> > > make more sense to me option 2) porting the Zookeeper needed code from
> > > Scala to Java (then removing it in the next phase).
> > >
> > > Is my understanding right on how we have to handle deprecation and
> > removal
> > > for something that is a breaking change ?
> > > Or this case is something "special" and we can live with a new Java
> based
> > > tool without zookeeper but with Admin Client API at same time ?
> > >
> > > Of course having both Admin Client API and Zookeeper utils working at
> the
> > > same time in the tools means more complexity in the code but it's
> > something
> > > that could be factorized.
> > >
> >
> > I think the right thing to do would be:
> >
> > 1. deprecate the old option, adding support for the replacement option
> > (using the AdminClient). Keep the code in scala. All this is in one KIP.
> > 2. Remove the old option (needs a KIP)
> > 3. Rewrite the tool in Java.
> >
> > Parts 2 and 3 could be done at the same time. I don't believe part 3
> needs
> > a KIP if it were a drop-in replacement.
> >
> > The reason I think this is the right way to proceed is:
> >
> > * It gives users a transition period to learn about the new option, and
> > replace any tooling of their own.
> > * By keeping the tool in scala you get to release your new AdminClient
> API
> > and get to iron out all the creases while users still have the
> --zookeeper
> > option as a fallback.
> > * Then when you know the AdminClient API works in the field you have a
> > straight porting job to do, and you have less code to port because you
> > don't have to port the code to support --zookeeper.
> >
> > But I'm fairly new here, so maybe a committer will chime in an correct me
> > where I'm wrong!
> >
> > Cheers,
> >
> > Tom
> >
>
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Paolo Patierno <pp...@live.com>.
Hi Ismael,
your third option is really appealing for me :-)
Just one thought about that ...
If we decide to go in this way we could have an 1.0 release with both Scala (zookeeper) and Java (bootstrap-server) implementations but then the shell script to chose one of them with the --zookeeper deprecation.
The final Scala (zookeeper) removal will happen on version 2.0 (following the policies about "major.minor.patch" release) but it could be months and months later.
In the meantime, let's suppose that we want to add a new --foo option to the tool; we should do that on both implementations having them in sync and it could be a problem until on 2.0 release when we get rid of the Scala implementation.
I really like your way but it could be better if we agree that new command options will be added only to the Java (bootstrap-server) implementation. It could mean pushing users to move from old to new tool ... that is good for us.
What do you think ?
Thanks,
Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor
Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>
________________________________
From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <is...@juma.me.uk>
Sent: Tuesday, August 1, 2017 3:00 PM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API
Hi Paolo,
Another option is to write the new tool in Java without support for
`--zookeeper` and include some logic in the shell script to pick the
implementation based on the presence of `--bootstrap-server` or
`--zookeeper`. This would mean that we can deprecate the Scala tool while
still supporting it for existing users.
I think this would involve the least amount of rewriting once we drop
support for `--zookeeper`.
Thoughts?
Ismael
On Tue, Aug 1, 2017 at 3:30 PM, Paolo Patierno <pp...@live.com> wrote:
> Hi Tom,
>
>
> thanks for your reply. I think that you are right and what you have
> proposed should be the way to go.
>
>
> Until today I have been working on refactoring the TopicCommand tool in
> Java using the AdminClient getting rid of the Zookeeper usage in only "one
> step" and maybe it's wrong.
>
> I'd like to have some input from committers as well to be sure that the
> way is good about how handling such use cases.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: Tom Bentley <t....@gmail.com>
> Sent: Friday, July 28, 2017 10:36 AM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Replies in line...
>
> On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
>
> > Hi committers,
> >
> > in my understanding there is the common idea to move all tools from Scala
> > to Java and then using the new Admin Client API instead of using the
> > Zookeeper connection.
> >
> > Regarding this subject I started to work on refactoring the TopicCommand
> > tool but with two steps at same time :
> >
> >
> > * re-writing it in Java
> > * removing --zookeeper option (so no Zookeeper connection) and adding
> > --broker-list option (so using the Admin Client API)
> >
> > Of course, such option substitution is a breaking change for the tool
> (and
> > the users who are using it).
> > In such a scenario, the two steps path should be needed : first
> > deprecation, then removal (for the --zookeeper option)
> >
>
> A change to tools (and their options) requires a KIP. There's no
> fundamental reason why both couldn't be supported during a transition
> period. So I doubt a KIP that didn't propose a transition period would get
> passed.
>
>
> It seems that in the "deprecation" phase we have two possible solutions :
> >
> >
> > 1. Adding Admin Client API to the Scala tools and so the --broker-list
> > option and a warning on --zookeeper for deprecation
> > 2. Rewriting the tool in Java using the Admin Client API (so
> > --broker-list) but at same time providing --zookeeper as well (so
> Zookeeper
> > connection)
> >
> > With the solution 2) we have the advantage to have the tool already in
> > Java when the --zookeeper option will be removed in a next step. In any
> > case we have to write the part related to use the Admin Client API so it
> > make more sense to me option 2) porting the Zookeeper needed code from
> > Scala to Java (then removing it in the next phase).
> >
> > Is my understanding right on how we have to handle deprecation and
> removal
> > for something that is a breaking change ?
> > Or this case is something "special" and we can live with a new Java based
> > tool without zookeeper but with Admin Client API at same time ?
> >
> > Of course having both Admin Client API and Zookeeper utils working at the
> > same time in the tools means more complexity in the code but it's
> something
> > that could be factorized.
> >
>
> I think the right thing to do would be:
>
> 1. deprecate the old option, adding support for the replacement option
> (using the AdminClient). Keep the code in scala. All this is in one KIP.
> 2. Remove the old option (needs a KIP)
> 3. Rewrite the tool in Java.
>
> Parts 2 and 3 could be done at the same time. I don't believe part 3 needs
> a KIP if it were a drop-in replacement.
>
> The reason I think this is the right way to proceed is:
>
> * It gives users a transition period to learn about the new option, and
> replace any tooling of their own.
> * By keeping the tool in scala you get to release your new AdminClient API
> and get to iron out all the creases while users still have the --zookeeper
> option as a fallback.
> * Then when you know the AdminClient API works in the field you have a
> straight porting job to do, and you have less code to port because you
> don't have to port the code to support --zookeeper.
>
> But I'm fairly new here, so maybe a committer will chime in an correct me
> where I'm wrong!
>
> Cheers,
>
> Tom
>
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Paolo,
Another option is to write the new tool in Java without support for
`--zookeeper` and include some logic in the shell script to pick the
implementation based on the presence of `--bootstrap-server` or
`--zookeeper`. This would mean that we can deprecate the Scala tool while
still supporting it for existing users.
I think this would involve the least amount of rewriting once we drop
support for `--zookeeper`.
Thoughts?
Ismael
On Tue, Aug 1, 2017 at 3:30 PM, Paolo Patierno <pp...@live.com> wrote:
> Hi Tom,
>
>
> thanks for your reply. I think that you are right and what you have
> proposed should be the way to go.
>
>
> Until today I have been working on refactoring the TopicCommand tool in
> Java using the AdminClient getting rid of the Zookeeper usage in only "one
> step" and maybe it's wrong.
>
> I'd like to have some input from committers as well to be sure that the
> way is good about how handling such use cases.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: Tom Bentley <t....@gmail.com>
> Sent: Friday, July 28, 2017 10:36 AM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Replies in line...
>
> On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
>
> > Hi committers,
> >
> > in my understanding there is the common idea to move all tools from Scala
> > to Java and then using the new Admin Client API instead of using the
> > Zookeeper connection.
> >
> > Regarding this subject I started to work on refactoring the TopicCommand
> > tool but with two steps at same time :
> >
> >
> > * re-writing it in Java
> > * removing --zookeeper option (so no Zookeeper connection) and adding
> > --broker-list option (so using the Admin Client API)
> >
> > Of course, such option substitution is a breaking change for the tool
> (and
> > the users who are using it).
> > In such a scenario, the two steps path should be needed : first
> > deprecation, then removal (for the --zookeeper option)
> >
>
> A change to tools (and their options) requires a KIP. There's no
> fundamental reason why both couldn't be supported during a transition
> period. So I doubt a KIP that didn't propose a transition period would get
> passed.
>
>
> It seems that in the "deprecation" phase we have two possible solutions :
> >
> >
> > 1. Adding Admin Client API to the Scala tools and so the --broker-list
> > option and a warning on --zookeeper for deprecation
> > 2. Rewriting the tool in Java using the Admin Client API (so
> > --broker-list) but at same time providing --zookeeper as well (so
> Zookeeper
> > connection)
> >
> > With the solution 2) we have the advantage to have the tool already in
> > Java when the --zookeeper option will be removed in a next step. In any
> > case we have to write the part related to use the Admin Client API so it
> > make more sense to me option 2) porting the Zookeeper needed code from
> > Scala to Java (then removing it in the next phase).
> >
> > Is my understanding right on how we have to handle deprecation and
> removal
> > for something that is a breaking change ?
> > Or this case is something "special" and we can live with a new Java based
> > tool without zookeeper but with Admin Client API at same time ?
> >
> > Of course having both Admin Client API and Zookeeper utils working at the
> > same time in the tools means more complexity in the code but it's
> something
> > that could be factorized.
> >
>
> I think the right thing to do would be:
>
> 1. deprecate the old option, adding support for the replacement option
> (using the AdminClient). Keep the code in scala. All this is in one KIP.
> 2. Remove the old option (needs a KIP)
> 3. Rewrite the tool in Java.
>
> Parts 2 and 3 could be done at the same time. I don't believe part 3 needs
> a KIP if it were a drop-in replacement.
>
> The reason I think this is the right way to proceed is:
>
> * It gives users a transition period to learn about the new option, and
> replace any tooling of their own.
> * By keeping the tool in scala you get to release your new AdminClient API
> and get to iron out all the creases while users still have the --zookeeper
> option as a fallback.
> * Then when you know the AdminClient API works in the field you have a
> straight porting job to do, and you have less code to port because you
> don't have to port the code to support --zookeeper.
>
> But I'm fairly new here, so maybe a committer will chime in an correct me
> where I'm wrong!
>
> Cheers,
>
> Tom
>
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Ismael Juma <is...@juma.me.uk>.
It needs a KIP if it introduces new public APIs and/or there are
compatibility implications. Otherwise, no KIP is required.
Ismael
On Wed, Aug 16, 2017 at 4:38 PM, Paolo Patierno <pp...@live.com> wrote:
> Hi Ismael,
>
>
> after your first review on my PR and the conversation here in the mailing
> list ... I have a doubt ...
>
> We are going to remove the --zookeeper option but adding the --broker-list
> but according to this discussion we agree on your solution having the
> choice at shell script level.
>
> Does the Java re-implementation of the TopicCommand tool need a KIP ?
>
>
> Thanks,
>
> Paolo
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <
> ismael@juma.me.uk>
> Sent: Wednesday, August 2, 2017 8:51 AM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Arseniy,
>
> At some point (before I joined), it was decided that clients would be
> written in Java for a few reasons:
>
> 1. It's easier to maintain binary compatibility
> 2. Calling Java from Scala is easier than calling Scala from Java (without
> extra effort)
> 3. A single artifact instead of one artifact per Scala version supported
> 4. There are a larger number of Java users than Scala users
>
> Since the plan is for most tools to be a thin wrapper over the Java
> clients, it seems sensible for such tools not to bring unnecessary
> dependencies.
>
> Ismael
>
>
> On Wed, Aug 2, 2017 at 9:28 AM, Arseniy Tashoyan <ta...@gmail.com>
> wrote:
>
> > Hi Paolo,
> >
> > I doubt that rewriting in Java makes value by itself. What is the reason
> of
> > redoing the things that are already done? The switching to new API can be
> > done without the switching to another language. Less new code - less new
> > bugs.
> > In some cases, Scala may be more convenient than Java. For example, one
> can
> > find Scala collections more handy than Java collections. Just select a
> > language that gives necessary tools.
> >
> > Arseniy
> >
> > 2017-08-01 17:30 GMT+03:00 Paolo Patierno <pp...@live.com>:
> >
> > > Hi Tom,
> > >
> > >
> > > thanks for your reply. I think that you are right and what you have
> > > proposed should be the way to go.
> > >
> > >
> > > Until today I have been working on refactoring the TopicCommand tool in
> > > Java using the AdminClient getting rid of the Zookeeper usage in only
> > "one
> > > step" and maybe it's wrong.
> > >
> > > I'd like to have some input from committers as well to be sure that the
> > > way is good about how handling such use cases.
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Windows Embedded & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> > >
> > >
> > > ________________________________
> > > From: Tom Bentley <t....@gmail.com>
> > > Sent: Friday, July 28, 2017 10:36 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: Command tools : from Scala to Java, from Zookeeper utils
> to
> > > Admin Client API
> > >
> > > Hi Paolo,
> > >
> > > Replies in line...
> > >
> > > On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
> > >
> > > > Hi committers,
> > > >
> > > > in my understanding there is the common idea to move all tools from
> > Scala
> > > > to Java and then using the new Admin Client API instead of using the
> > > > Zookeeper connection.
> > > >
> > > > Regarding this subject I started to work on refactoring the
> > TopicCommand
> > > > tool but with two steps at same time :
> > > >
> > > >
> > > > * re-writing it in Java
> > > > * removing --zookeeper option (so no Zookeeper connection) and
> > adding
> > > > --broker-list option (so using the Admin Client API)
> > > >
> > > > Of course, such option substitution is a breaking change for the tool
> > > (and
> > > > the users who are using it).
> > > > In such a scenario, the two steps path should be needed : first
> > > > deprecation, then removal (for the --zookeeper option)
> > > >
> > >
> > > A change to tools (and their options) requires a KIP. There's no
> > > fundamental reason why both couldn't be supported during a transition
> > > period. So I doubt a KIP that didn't propose a transition period would
> > get
> > > passed.
> > >
> > >
> > > It seems that in the "deprecation" phase we have two possible
> solutions :
> > > >
> > > >
> > > > 1. Adding Admin Client API to the Scala tools and so the
> > --broker-list
> > > > option and a warning on --zookeeper for deprecation
> > > > 2. Rewriting the tool in Java using the Admin Client API (so
> > > > --broker-list) but at same time providing --zookeeper as well (so
> > > Zookeeper
> > > > connection)
> > > >
> > > > With the solution 2) we have the advantage to have the tool already
> in
> > > > Java when the --zookeeper option will be removed in a next step. In
> any
> > > > case we have to write the part related to use the Admin Client API so
> > it
> > > > make more sense to me option 2) porting the Zookeeper needed code
> from
> > > > Scala to Java (then removing it in the next phase).
> > > >
> > > > Is my understanding right on how we have to handle deprecation and
> > > removal
> > > > for something that is a breaking change ?
> > > > Or this case is something "special" and we can live with a new Java
> > based
> > > > tool without zookeeper but with Admin Client API at same time ?
> > > >
> > > > Of course having both Admin Client API and Zookeeper utils working at
> > the
> > > > same time in the tools means more complexity in the code but it's
> > > something
> > > > that could be factorized.
> > > >
> > >
> > > I think the right thing to do would be:
> > >
> > > 1. deprecate the old option, adding support for the replacement option
> > > (using the AdminClient). Keep the code in scala. All this is in one
> KIP.
> > > 2. Remove the old option (needs a KIP)
> > > 3. Rewrite the tool in Java.
> > >
> > > Parts 2 and 3 could be done at the same time. I don't believe part 3
> > needs
> > > a KIP if it were a drop-in replacement.
> > >
> > > The reason I think this is the right way to proceed is:
> > >
> > > * It gives users a transition period to learn about the new option, and
> > > replace any tooling of their own.
> > > * By keeping the tool in scala you get to release your new AdminClient
> > API
> > > and get to iron out all the creases while users still have the
> > --zookeeper
> > > option as a fallback.
> > > * Then when you know the AdminClient API works in the field you have a
> > > straight porting job to do, and you have less code to port because you
> > > don't have to port the code to support --zookeeper.
> > >
> > > But I'm fairly new here, so maybe a committer will chime in an correct
> me
> > > where I'm wrong!
> > >
> > > Cheers,
> > >
> > > Tom
> > >
> >
>
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Paolo Patierno <pp...@live.com>.
Hi Ismael,
after your first review on my PR and the conversation here in the mailing list ... I have a doubt ...
We are going to remove the --zookeeper option but adding the --broker-list but according to this discussion we agree on your solution having the choice at shell script level.
Does the Java re-implementation of the TopicCommand tool need a KIP ?
Thanks,
Paolo
Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor
Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>
________________________________
From: ismaelj@gmail.com <is...@gmail.com> on behalf of Ismael Juma <is...@juma.me.uk>
Sent: Wednesday, August 2, 2017 8:51 AM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API
Hi Arseniy,
At some point (before I joined), it was decided that clients would be
written in Java for a few reasons:
1. It's easier to maintain binary compatibility
2. Calling Java from Scala is easier than calling Scala from Java (without
extra effort)
3. A single artifact instead of one artifact per Scala version supported
4. There are a larger number of Java users than Scala users
Since the plan is for most tools to be a thin wrapper over the Java
clients, it seems sensible for such tools not to bring unnecessary
dependencies.
Ismael
On Wed, Aug 2, 2017 at 9:28 AM, Arseniy Tashoyan <ta...@gmail.com> wrote:
> Hi Paolo,
>
> I doubt that rewriting in Java makes value by itself. What is the reason of
> redoing the things that are already done? The switching to new API can be
> done without the switching to another language. Less new code - less new
> bugs.
> In some cases, Scala may be more convenient than Java. For example, one can
> find Scala collections more handy than Java collections. Just select a
> language that gives necessary tools.
>
> Arseniy
>
> 2017-08-01 17:30 GMT+03:00 Paolo Patierno <pp...@live.com>:
>
> > Hi Tom,
> >
> >
> > thanks for your reply. I think that you are right and what you have
> > proposed should be the way to go.
> >
> >
> > Until today I have been working on refactoring the TopicCommand tool in
> > Java using the AdminClient getting rid of the Zookeeper usage in only
> "one
> > step" and maybe it's wrong.
> >
> > I'd like to have some input from committers as well to be sure that the
> > way is good about how handling such use cases.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > ________________________________
> > From: Tom Bentley <t....@gmail.com>
> > Sent: Friday, July 28, 2017 10:36 AM
> > To: dev@kafka.apache.org
> > Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> > Admin Client API
> >
> > Hi Paolo,
> >
> > Replies in line...
> >
> > On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
> >
> > > Hi committers,
> > >
> > > in my understanding there is the common idea to move all tools from
> Scala
> > > to Java and then using the new Admin Client API instead of using the
> > > Zookeeper connection.
> > >
> > > Regarding this subject I started to work on refactoring the
> TopicCommand
> > > tool but with two steps at same time :
> > >
> > >
> > > * re-writing it in Java
> > > * removing --zookeeper option (so no Zookeeper connection) and
> adding
> > > --broker-list option (so using the Admin Client API)
> > >
> > > Of course, such option substitution is a breaking change for the tool
> > (and
> > > the users who are using it).
> > > In such a scenario, the two steps path should be needed : first
> > > deprecation, then removal (for the --zookeeper option)
> > >
> >
> > A change to tools (and their options) requires a KIP. There's no
> > fundamental reason why both couldn't be supported during a transition
> > period. So I doubt a KIP that didn't propose a transition period would
> get
> > passed.
> >
> >
> > It seems that in the "deprecation" phase we have two possible solutions :
> > >
> > >
> > > 1. Adding Admin Client API to the Scala tools and so the
> --broker-list
> > > option and a warning on --zookeeper for deprecation
> > > 2. Rewriting the tool in Java using the Admin Client API (so
> > > --broker-list) but at same time providing --zookeeper as well (so
> > Zookeeper
> > > connection)
> > >
> > > With the solution 2) we have the advantage to have the tool already in
> > > Java when the --zookeeper option will be removed in a next step. In any
> > > case we have to write the part related to use the Admin Client API so
> it
> > > make more sense to me option 2) porting the Zookeeper needed code from
> > > Scala to Java (then removing it in the next phase).
> > >
> > > Is my understanding right on how we have to handle deprecation and
> > removal
> > > for something that is a breaking change ?
> > > Or this case is something "special" and we can live with a new Java
> based
> > > tool without zookeeper but with Admin Client API at same time ?
> > >
> > > Of course having both Admin Client API and Zookeeper utils working at
> the
> > > same time in the tools means more complexity in the code but it's
> > something
> > > that could be factorized.
> > >
> >
> > I think the right thing to do would be:
> >
> > 1. deprecate the old option, adding support for the replacement option
> > (using the AdminClient). Keep the code in scala. All this is in one KIP.
> > 2. Remove the old option (needs a KIP)
> > 3. Rewrite the tool in Java.
> >
> > Parts 2 and 3 could be done at the same time. I don't believe part 3
> needs
> > a KIP if it were a drop-in replacement.
> >
> > The reason I think this is the right way to proceed is:
> >
> > * It gives users a transition period to learn about the new option, and
> > replace any tooling of their own.
> > * By keeping the tool in scala you get to release your new AdminClient
> API
> > and get to iron out all the creases while users still have the
> --zookeeper
> > option as a fallback.
> > * Then when you know the AdminClient API works in the field you have a
> > straight porting job to do, and you have less code to port because you
> > don't have to port the code to support --zookeeper.
> >
> > But I'm fairly new here, so maybe a committer will chime in an correct me
> > where I'm wrong!
> >
> > Cheers,
> >
> > Tom
> >
>
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Arseniy,
At some point (before I joined), it was decided that clients would be
written in Java for a few reasons:
1. It's easier to maintain binary compatibility
2. Calling Java from Scala is easier than calling Scala from Java (without
extra effort)
3. A single artifact instead of one artifact per Scala version supported
4. There are a larger number of Java users than Scala users
Since the plan is for most tools to be a thin wrapper over the Java
clients, it seems sensible for such tools not to bring unnecessary
dependencies.
Ismael
On Wed, Aug 2, 2017 at 9:28 AM, Arseniy Tashoyan <ta...@gmail.com> wrote:
> Hi Paolo,
>
> I doubt that rewriting in Java makes value by itself. What is the reason of
> redoing the things that are already done? The switching to new API can be
> done without the switching to another language. Less new code - less new
> bugs.
> In some cases, Scala may be more convenient than Java. For example, one can
> find Scala collections more handy than Java collections. Just select a
> language that gives necessary tools.
>
> Arseniy
>
> 2017-08-01 17:30 GMT+03:00 Paolo Patierno <pp...@live.com>:
>
> > Hi Tom,
> >
> >
> > thanks for your reply. I think that you are right and what you have
> > proposed should be the way to go.
> >
> >
> > Until today I have been working on refactoring the TopicCommand tool in
> > Java using the AdminClient getting rid of the Zookeeper usage in only
> "one
> > step" and maybe it's wrong.
> >
> > I'd like to have some input from committers as well to be sure that the
> > way is good about how handling such use cases.
> >
> >
> > Thanks,
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > ________________________________
> > From: Tom Bentley <t....@gmail.com>
> > Sent: Friday, July 28, 2017 10:36 AM
> > To: dev@kafka.apache.org
> > Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> > Admin Client API
> >
> > Hi Paolo,
> >
> > Replies in line...
> >
> > On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
> >
> > > Hi committers,
> > >
> > > in my understanding there is the common idea to move all tools from
> Scala
> > > to Java and then using the new Admin Client API instead of using the
> > > Zookeeper connection.
> > >
> > > Regarding this subject I started to work on refactoring the
> TopicCommand
> > > tool but with two steps at same time :
> > >
> > >
> > > * re-writing it in Java
> > > * removing --zookeeper option (so no Zookeeper connection) and
> adding
> > > --broker-list option (so using the Admin Client API)
> > >
> > > Of course, such option substitution is a breaking change for the tool
> > (and
> > > the users who are using it).
> > > In such a scenario, the two steps path should be needed : first
> > > deprecation, then removal (for the --zookeeper option)
> > >
> >
> > A change to tools (and their options) requires a KIP. There's no
> > fundamental reason why both couldn't be supported during a transition
> > period. So I doubt a KIP that didn't propose a transition period would
> get
> > passed.
> >
> >
> > It seems that in the "deprecation" phase we have two possible solutions :
> > >
> > >
> > > 1. Adding Admin Client API to the Scala tools and so the
> --broker-list
> > > option and a warning on --zookeeper for deprecation
> > > 2. Rewriting the tool in Java using the Admin Client API (so
> > > --broker-list) but at same time providing --zookeeper as well (so
> > Zookeeper
> > > connection)
> > >
> > > With the solution 2) we have the advantage to have the tool already in
> > > Java when the --zookeeper option will be removed in a next step. In any
> > > case we have to write the part related to use the Admin Client API so
> it
> > > make more sense to me option 2) porting the Zookeeper needed code from
> > > Scala to Java (then removing it in the next phase).
> > >
> > > Is my understanding right on how we have to handle deprecation and
> > removal
> > > for something that is a breaking change ?
> > > Or this case is something "special" and we can live with a new Java
> based
> > > tool without zookeeper but with Admin Client API at same time ?
> > >
> > > Of course having both Admin Client API and Zookeeper utils working at
> the
> > > same time in the tools means more complexity in the code but it's
> > something
> > > that could be factorized.
> > >
> >
> > I think the right thing to do would be:
> >
> > 1. deprecate the old option, adding support for the replacement option
> > (using the AdminClient). Keep the code in scala. All this is in one KIP.
> > 2. Remove the old option (needs a KIP)
> > 3. Rewrite the tool in Java.
> >
> > Parts 2 and 3 could be done at the same time. I don't believe part 3
> needs
> > a KIP if it were a drop-in replacement.
> >
> > The reason I think this is the right way to proceed is:
> >
> > * It gives users a transition period to learn about the new option, and
> > replace any tooling of their own.
> > * By keeping the tool in scala you get to release your new AdminClient
> API
> > and get to iron out all the creases while users still have the
> --zookeeper
> > option as a fallback.
> > * Then when you know the AdminClient API works in the field you have a
> > straight porting job to do, and you have less code to port because you
> > don't have to port the code to support --zookeeper.
> >
> > But I'm fairly new here, so maybe a committer will chime in an correct me
> > where I'm wrong!
> >
> > Cheers,
> >
> > Tom
> >
>
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Paolo Patierno <pp...@live.com>.
Hi Arseniy,
I opened this JIRA about one month ago (https://issues.apache.org/jira/browse/KAFKA-5536) asking why we have some tools in Java and other in Scala.
As Ismael pointed out the end state should be having tools written in Java (as already happened to clients from Scala to Java) living in the external "tools" module (where there are already some tools) so outside of the "core" module in Kafka. Another key point is to use the Admin Client API provided by the "clients" module in all the tools where we don't need Zookeeper connection anymore.
My guess is that, even if Kafka is developed in Scala, the committers have the idea to have all the external modules written in Java (so clients and tools for now).
As you know it's the same as for the other two bigger projects around Kafka (core) like Kafka Connect and Kafka Streams ... entirely developed in Java.
Thanks,
Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor
Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>
________________________________
From: Arseniy Tashoyan <ta...@gmail.com>
Sent: Wednesday, August 2, 2017 8:28 AM
To: dev@kafka.apache.org
Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API
Hi Paolo,
I doubt that rewriting in Java makes value by itself. What is the reason of
redoing the things that are already done? The switching to new API can be
done without the switching to another language. Less new code - less new
bugs.
In some cases, Scala may be more convenient than Java. For example, one can
find Scala collections more handy than Java collections. Just select a
language that gives necessary tools.
Arseniy
2017-08-01 17:30 GMT+03:00 Paolo Patierno <pp...@live.com>:
> Hi Tom,
>
>
> thanks for your reply. I think that you are right and what you have
> proposed should be the way to go.
>
>
> Until today I have been working on refactoring the TopicCommand tool in
> Java using the AdminClient getting rid of the Zookeeper usage in only "one
> step" and maybe it's wrong.
>
> I'd like to have some input from committers as well to be sure that the
> way is good about how handling such use cases.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: Tom Bentley <t....@gmail.com>
> Sent: Friday, July 28, 2017 10:36 AM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Replies in line...
>
> On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
>
> > Hi committers,
> >
> > in my understanding there is the common idea to move all tools from Scala
> > to Java and then using the new Admin Client API instead of using the
> > Zookeeper connection.
> >
> > Regarding this subject I started to work on refactoring the TopicCommand
> > tool but with two steps at same time :
> >
> >
> > * re-writing it in Java
> > * removing --zookeeper option (so no Zookeeper connection) and adding
> > --broker-list option (so using the Admin Client API)
> >
> > Of course, such option substitution is a breaking change for the tool
> (and
> > the users who are using it).
> > In such a scenario, the two steps path should be needed : first
> > deprecation, then removal (for the --zookeeper option)
> >
>
> A change to tools (and their options) requires a KIP. There's no
> fundamental reason why both couldn't be supported during a transition
> period. So I doubt a KIP that didn't propose a transition period would get
> passed.
>
>
> It seems that in the "deprecation" phase we have two possible solutions :
> >
> >
> > 1. Adding Admin Client API to the Scala tools and so the --broker-list
> > option and a warning on --zookeeper for deprecation
> > 2. Rewriting the tool in Java using the Admin Client API (so
> > --broker-list) but at same time providing --zookeeper as well (so
> Zookeeper
> > connection)
> >
> > With the solution 2) we have the advantage to have the tool already in
> > Java when the --zookeeper option will be removed in a next step. In any
> > case we have to write the part related to use the Admin Client API so it
> > make more sense to me option 2) porting the Zookeeper needed code from
> > Scala to Java (then removing it in the next phase).
> >
> > Is my understanding right on how we have to handle deprecation and
> removal
> > for something that is a breaking change ?
> > Or this case is something "special" and we can live with a new Java based
> > tool without zookeeper but with Admin Client API at same time ?
> >
> > Of course having both Admin Client API and Zookeeper utils working at the
> > same time in the tools means more complexity in the code but it's
> something
> > that could be factorized.
> >
>
> I think the right thing to do would be:
>
> 1. deprecate the old option, adding support for the replacement option
> (using the AdminClient). Keep the code in scala. All this is in one KIP.
> 2. Remove the old option (needs a KIP)
> 3. Rewrite the tool in Java.
>
> Parts 2 and 3 could be done at the same time. I don't believe part 3 needs
> a KIP if it were a drop-in replacement.
>
> The reason I think this is the right way to proceed is:
>
> * It gives users a transition period to learn about the new option, and
> replace any tooling of their own.
> * By keeping the tool in scala you get to release your new AdminClient API
> and get to iron out all the creases while users still have the --zookeeper
> option as a fallback.
> * Then when you know the AdminClient API works in the field you have a
> straight porting job to do, and you have less code to port because you
> don't have to port the code to support --zookeeper.
>
> But I'm fairly new here, so maybe a committer will chime in an correct me
> where I'm wrong!
>
> Cheers,
>
> Tom
>
Re: Command tools : from Scala to Java, from Zookeeper utils to Admin
Client API
Posted by Arseniy Tashoyan <ta...@gmail.com>.
Hi Paolo,
I doubt that rewriting in Java makes value by itself. What is the reason of
redoing the things that are already done? The switching to new API can be
done without the switching to another language. Less new code - less new
bugs.
In some cases, Scala may be more convenient than Java. For example, one can
find Scala collections more handy than Java collections. Just select a
language that gives necessary tools.
Arseniy
2017-08-01 17:30 GMT+03:00 Paolo Patierno <pp...@live.com>:
> Hi Tom,
>
>
> thanks for your reply. I think that you are right and what you have
> proposed should be the way to go.
>
>
> Until today I have been working on refactoring the TopicCommand tool in
> Java using the AdminClient getting rid of the Zookeeper usage in only "one
> step" and maybe it's wrong.
>
> I'd like to have some input from committers as well to be sure that the
> way is good about how handling such use cases.
>
>
> Thanks,
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: Tom Bentley <t....@gmail.com>
> Sent: Friday, July 28, 2017 10:36 AM
> To: dev@kafka.apache.org
> Subject: Re: Command tools : from Scala to Java, from Zookeeper utils to
> Admin Client API
>
> Hi Paolo,
>
> Replies in line...
>
> On 28 July 2017 at 11:14, Paolo Patierno <pp...@live.com> wrote:
>
> > Hi committers,
> >
> > in my understanding there is the common idea to move all tools from Scala
> > to Java and then using the new Admin Client API instead of using the
> > Zookeeper connection.
> >
> > Regarding this subject I started to work on refactoring the TopicCommand
> > tool but with two steps at same time :
> >
> >
> > * re-writing it in Java
> > * removing --zookeeper option (so no Zookeeper connection) and adding
> > --broker-list option (so using the Admin Client API)
> >
> > Of course, such option substitution is a breaking change for the tool
> (and
> > the users who are using it).
> > In such a scenario, the two steps path should be needed : first
> > deprecation, then removal (for the --zookeeper option)
> >
>
> A change to tools (and their options) requires a KIP. There's no
> fundamental reason why both couldn't be supported during a transition
> period. So I doubt a KIP that didn't propose a transition period would get
> passed.
>
>
> It seems that in the "deprecation" phase we have two possible solutions :
> >
> >
> > 1. Adding Admin Client API to the Scala tools and so the --broker-list
> > option and a warning on --zookeeper for deprecation
> > 2. Rewriting the tool in Java using the Admin Client API (so
> > --broker-list) but at same time providing --zookeeper as well (so
> Zookeeper
> > connection)
> >
> > With the solution 2) we have the advantage to have the tool already in
> > Java when the --zookeeper option will be removed in a next step. In any
> > case we have to write the part related to use the Admin Client API so it
> > make more sense to me option 2) porting the Zookeeper needed code from
> > Scala to Java (then removing it in the next phase).
> >
> > Is my understanding right on how we have to handle deprecation and
> removal
> > for something that is a breaking change ?
> > Or this case is something "special" and we can live with a new Java based
> > tool without zookeeper but with Admin Client API at same time ?
> >
> > Of course having both Admin Client API and Zookeeper utils working at the
> > same time in the tools means more complexity in the code but it's
> something
> > that could be factorized.
> >
>
> I think the right thing to do would be:
>
> 1. deprecate the old option, adding support for the replacement option
> (using the AdminClient). Keep the code in scala. All this is in one KIP.
> 2. Remove the old option (needs a KIP)
> 3. Rewrite the tool in Java.
>
> Parts 2 and 3 could be done at the same time. I don't believe part 3 needs
> a KIP if it were a drop-in replacement.
>
> The reason I think this is the right way to proceed is:
>
> * It gives users a transition period to learn about the new option, and
> replace any tooling of their own.
> * By keeping the tool in scala you get to release your new AdminClient API
> and get to iron out all the creases while users still have the --zookeeper
> option as a fallback.
> * Then when you know the AdminClient API works in the field you have a
> straight porting job to do, and you have less code to port because you
> don't have to port the code to support --zookeeper.
>
> But I'm fairly new here, so maybe a committer will chime in an correct me
> where I'm wrong!
>
> Cheers,
>
> Tom
>