You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Thomas Koch <th...@koch.ro> on 2011/09/02 15:15:38 UTC
ZooKeeper cleanup / refactoring / scala migration
Hi,
my university labs work has started yesterday. In the next two months I'll
work on ZooKeeper. This work has three major goals
- improve the maintainability of the code base
- migrate ZooKeeper to scala and use actors for reliable concurrency
- find other developers to collaborate on the scala version
I plan to work in these steps:
1. use static analysis tools to cleanup the java codebase
( checkstyle, pmd, findbugs, eclipse compiler warnings )
2. decrease dependencies in the ZK java packages, break cyclic dependencies
3. migrate components one-by-one to scala, starting with the leafs of the
dependency graph
I'd be happy if as much as possible of my work in steps 1. and 2. could also
be useful for you. What would be a good strategy to go? I assume you're
concentrating now on getting 3.4 out and don't have time for other things. Can
I help on 3.4 blockers?
I've set up a gerrit[1] (git based code review) and a jenkins[2] server for my
project. Jenkins is armed with checkstyle, findbugs, pmd, copy-paste-detection
and jdepends. I've carefully selected the checks run by checkstyle and pmd and
would suggest that the remaining warnings should really be eliminated.
Gerrit already hosts two changesets which eliminate nearly all eclipse
compiler warnings. I'd be happy if you'd like to create an account at the
gerrit instance (openid needed) and play around with it. I can also give you
reviewer status which lets you push changes. Every change push will trigger a
jenkins build.
I've already proposed to infra@a.o that I'd help in setting up a gerrit server
for the ASF if any project would be interested.
[1] http://koch.ro:8081
[2] http://koch.ro:8080
Best regards,
Thomas Koch, http://www.koch.ro
Re: Reviewboard, Gerrit was: ZooKeeper cleanup / refactoring / scala migration
Posted by Patrick Hunt <ph...@apache.org>.
Thomas, checkout post-review from https://github.com/reviewboard/rbtools
I haven't used it against Apache rb, however I use it to great effect
inside Cloudera. It will allow you to cut the number of steps
significantly (I have small bash scripts wrapping post-review for the
various projects I contribute to).
Enjoy,
Patrick
On Fri, Sep 2, 2011 at 12:48 PM, Thomas Koch <th...@koch.ro> wrote:
> Fournier, Camille F.:
>> We already have a reviewboard set up for zookeeper which you should plan to
>
> Hi Fournier,
>
> I'll use reviewboard for patches targeted for inclusion. However just for
> reference and to make you curious, I compare the steps necessary to create a
> review in reviewboard and in gerrit:
>
> reviewboard:
> - create a patch file
> - open web interface, log in
> - select project
> - select patch file
> - select reviewer group
> - select jira bug (lookup bug number in jira...)
> - copy summary from jira issue (or write from scratch)
> - enter "." for description
> - publish
>
> gerrit:
> - include issue number in commit message
> - git push gerrit
> ... That's it? That's it!
>
> Additionally gerrit also triggers jenkins, merges the change in the master
> branch once the review is finished and hosts your GIT repos with access
> control. There's also inclusion in mylyn to let you do reviews from eclipse...
> :-)
>
> So if you like GIT you're invited to just play around with the gerrit
> instance.
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>
Reviewboard, Gerrit was: ZooKeeper cleanup / refactoring / scala migration
Posted by Thomas Koch <th...@koch.ro>.
Fournier, Camille F.:
> We already have a reviewboard set up for zookeeper which you should plan to
Hi Fournier,
I'll use reviewboard for patches targeted for inclusion. However just for
reference and to make you curious, I compare the steps necessary to create a
review in reviewboard and in gerrit:
reviewboard:
- create a patch file
- open web interface, log in
- select project
- select patch file
- select reviewer group
- select jira bug (lookup bug number in jira...)
- copy summary from jira issue (or write from scratch)
- enter "." for description
- publish
gerrit:
- include issue number in commit message
- git push gerrit
... That's it? That's it!
Additionally gerrit also triggers jenkins, merges the change in the master
branch once the review is finished and hosts your GIT repos with access
control. There's also inclusion in mylyn to let you do reviews from eclipse...
:-)
So if you like GIT you're invited to just play around with the gerrit
instance.
Best regards,
Thomas Koch, http://www.koch.ro
Re: ZooKeeper cleanup / refactoring / scala migration
Posted by Patrick Hunt <ph...@apache.org>.
Hey Thomas! I've raised a scala port a number of times previously,
most recently at the post-summit meetup:
http://markmail.org/message/t32x22hmifo3urxk We discussed this shortly
both at the meetup and subsequently on list. Unfortunately there was
no consensus around building ZK on scala (or any other language other
than java -- Ted asked to port to clojure instead), in particular the
point was raised that we, the currently zk community, are a community
of ZK Java developers with no experience with scala (or clojure/c#(?)
for that matter).
What I think would be successful is for you to bring this to the
incubator - "ZooKeeper-Scala: re-implement ZooKeeper in Scala".
A community of ZK scala developers could be grown there, see the
proposal guide, in particular we (ZK tlp) could sponsor this effort,
with the intent that upon successful graduation from the incubator we
pull in zk-scala as either a subproject or just part of the TLP but as
a separate release artifact from zk java server. see this link for
details: http://incubator.apache.org/guides/proposal.html#template-sponsoring-entity
I'd be willing to be the champion and a mentor for this project in the
incubator.
Regards,
Patrick
On Fri, Sep 2, 2011 at 7:12 AM, Fournier, Camille F.
<Ca...@gs.com> wrote:
> Hi Thomas,
>
> Here's my feedback:
>
> 1. For any useful fixes you find here, please follow the normal procedures of raising a ticket and attaching a patch. In my experience, static analysis tools often carry with them a lot of irrelevant noise, but as long as the changes you propose are clean and don't break backwards compatibility, I would be happy to review and accept such changes.
>
> What I can't promise is that you will get much traction on trying to push a patch with a huge number of fixes at once. Every line of code we have to review increases the complexity of our job as reviewers. It would be great if you could break these up into small patches and would definitely increase the odds of the changes being accepted.
>
> We already have a reviewboard set up for zookeeper which you should plan to use, and of course the hadoop build farm. If you believe additional analysis tools would be useful in our build, please work with our build.xml and the infra team to get the necessary tools installed. It doesn't do us much good to fix things caught by your build server and then have them break again because we don't have the tooling available.
>
> 2. I doubt you will get much traction here on pushing changes back. I'm a big fan of refactoring but at some point refactoring for refactoring's sake does nothing but muddle the change history of a code base. Any refactoring that needs to be done would be best done in conjunction with a fix or feature that it helps to enable.
>
> 3. Sounds like a fun academic exercise for you. Maybe you could start with a Scala client that we could support. Not sure there's any benefit since Scala can run Java code, but it could be interesting and maybe something we could take back as a contribution if you wanted.
>
> C
>
> -----Original Message-----
> From: Thomas Koch [mailto:thomas@koch.ro]
> Sent: Friday, September 02, 2011 9:16 AM
> To: dev@zookeeper.apache.org
> Subject: ZooKeeper cleanup / refactoring / scala migration
>
> Hi,
>
> my university labs work has started yesterday. In the next two months I'll
> work on ZooKeeper. This work has three major goals
>
> - improve the maintainability of the code base
> - migrate ZooKeeper to scala and use actors for reliable concurrency
> - find other developers to collaborate on the scala version
>
> I plan to work in these steps:
>
> 1. use static analysis tools to cleanup the java codebase
> ( checkstyle, pmd, findbugs, eclipse compiler warnings )
>
> 2. decrease dependencies in the ZK java packages, break cyclic dependencies
>
> 3. migrate components one-by-one to scala, starting with the leafs of the
> dependency graph
>
> I'd be happy if as much as possible of my work in steps 1. and 2. could also
> be useful for you. What would be a good strategy to go? I assume you're
> concentrating now on getting 3.4 out and don't have time for other things. Can
> I help on 3.4 blockers?
>
> I've set up a gerrit[1] (git based code review) and a jenkins[2] server for my
> project. Jenkins is armed with checkstyle, findbugs, pmd, copy-paste-detection
> and jdepends. I've carefully selected the checks run by checkstyle and pmd and
> would suggest that the remaining warnings should really be eliminated.
>
> Gerrit already hosts two changesets which eliminate nearly all eclipse
> compiler warnings. I'd be happy if you'd like to create an account at the
> gerrit instance (openid needed) and play around with it. I can also give you
> reviewer status which lets you push changes. Every change push will trigger a
> jenkins build.
>
> I've already proposed to infra@a.o that I'd help in setting up a gerrit server
> for the ASF if any project would be interested.
>
> [1] http://koch.ro:8081
> [2] http://koch.ro:8080
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>
Re: ZooKeeper cleanup / refactoring / scala migration
Posted by Patrick Hunt <ph...@apache.org>.
+1, having sonar would be nice although I've found that having tools
that no one pays attention to is not useful. The best way to combat
that is to integrate it into the build (e.g. hadoop QA bot).
On Mon, Sep 5, 2011 at 1:27 AM, Thomas Koch <th...@koch.ro> wrote:
> Hi Camille,
>> We already have a reviewboard set up for zookeeper which you should plan to
>> use, and of course the hadoop build farm. If you believe additional
>> analysis tools would be useful in our build, please work with our
>> build.xml and the infra team to get the necessary tools installed. It
>> doesn't do us much good to fix things caught by your build server and then
>> have them break again because we don't have the tooling available.
>
> I understand that ZooKeeper is moving to maven as soon as possible. Once the
> move has been done, I could help getting ZK in the ASF sonar instance.[1]
> After that we can fine-tune the analysis run on ZK.
>
> [1] https://analysis.apache.org
>
> Regards,
>
> Thomas Koch, http://www.koch.ro
>
Re: ZooKeeper cleanup / refactoring / scala migration
Posted by Thomas Koch <th...@koch.ro>.
Hi Camille,
> We already have a reviewboard set up for zookeeper which you should plan to
> use, and of course the hadoop build farm. If you believe additional
> analysis tools would be useful in our build, please work with our
> build.xml and the infra team to get the necessary tools installed. It
> doesn't do us much good to fix things caught by your build server and then
> have them break again because we don't have the tooling available.
I understand that ZooKeeper is moving to maven as soon as possible. Once the
move has been done, I could help getting ZK in the ASF sonar instance.[1]
After that we can fine-tune the analysis run on ZK.
[1] https://analysis.apache.org
Regards,
Thomas Koch, http://www.koch.ro
RE: ZooKeeper cleanup / refactoring / scala migration
Posted by "Fournier, Camille F." <Ca...@gs.com>.
Hi Thomas,
Here's my feedback:
1. For any useful fixes you find here, please follow the normal procedures of raising a ticket and attaching a patch. In my experience, static analysis tools often carry with them a lot of irrelevant noise, but as long as the changes you propose are clean and don't break backwards compatibility, I would be happy to review and accept such changes.
What I can't promise is that you will get much traction on trying to push a patch with a huge number of fixes at once. Every line of code we have to review increases the complexity of our job as reviewers. It would be great if you could break these up into small patches and would definitely increase the odds of the changes being accepted.
We already have a reviewboard set up for zookeeper which you should plan to use, and of course the hadoop build farm. If you believe additional analysis tools would be useful in our build, please work with our build.xml and the infra team to get the necessary tools installed. It doesn't do us much good to fix things caught by your build server and then have them break again because we don't have the tooling available.
2. I doubt you will get much traction here on pushing changes back. I'm a big fan of refactoring but at some point refactoring for refactoring's sake does nothing but muddle the change history of a code base. Any refactoring that needs to be done would be best done in conjunction with a fix or feature that it helps to enable.
3. Sounds like a fun academic exercise for you. Maybe you could start with a Scala client that we could support. Not sure there's any benefit since Scala can run Java code, but it could be interesting and maybe something we could take back as a contribution if you wanted.
C
-----Original Message-----
From: Thomas Koch [mailto:thomas@koch.ro]
Sent: Friday, September 02, 2011 9:16 AM
To: dev@zookeeper.apache.org
Subject: ZooKeeper cleanup / refactoring / scala migration
Hi,
my university labs work has started yesterday. In the next two months I'll
work on ZooKeeper. This work has three major goals
- improve the maintainability of the code base
- migrate ZooKeeper to scala and use actors for reliable concurrency
- find other developers to collaborate on the scala version
I plan to work in these steps:
1. use static analysis tools to cleanup the java codebase
( checkstyle, pmd, findbugs, eclipse compiler warnings )
2. decrease dependencies in the ZK java packages, break cyclic dependencies
3. migrate components one-by-one to scala, starting with the leafs of the
dependency graph
I'd be happy if as much as possible of my work in steps 1. and 2. could also
be useful for you. What would be a good strategy to go? I assume you're
concentrating now on getting 3.4 out and don't have time for other things. Can
I help on 3.4 blockers?
I've set up a gerrit[1] (git based code review) and a jenkins[2] server for my
project. Jenkins is armed with checkstyle, findbugs, pmd, copy-paste-detection
and jdepends. I've carefully selected the checks run by checkstyle and pmd and
would suggest that the remaining warnings should really be eliminated.
Gerrit already hosts two changesets which eliminate nearly all eclipse
compiler warnings. I'd be happy if you'd like to create an account at the
gerrit instance (openid needed) and play around with it. I can also give you
reviewer status which lets you push changes. Every change push will trigger a
jenkins build.
I've already proposed to infra@a.o that I'd help in setting up a gerrit server
for the ASF if any project would be interested.
[1] http://koch.ro:8081
[2] http://koch.ro:8080
Best regards,
Thomas Koch, http://www.koch.ro
Re: ZooKeeper cleanup / refactoring / scala migration
Posted by Thomas Koch <th...@koch.ro>.
Andrew Finnell:
> Thomas,
>
> Upfront warning. I am not in charge of ZooKeeper or affiliated with them
> except for submitting patches.
>
> While I understand the power of Scala, what is the major benefit of
> changing all of the Java code over to Scala? Will this outweigh the
> decrease in skill sets that a language such as Java brings? Another
> technology called Kafka is written in Scala. It doesn't have anywhere near
> the support of ZooKeeper. I attribute most of that to it being written in
> Scala, which reduces the pool of developers that can or are willing to
> work on it.
>
> If your university will truly enhance ZooKeeper I would ask, in an
> unofficial capacity, to ensure there is a strong line of separation
> between items 1 and 2&3. Also keep in mind that I have run FindBugs on ZK
> myself. Some of the fixes will break backwards compatibility. Just be
> aware of that fact.
>
> It is my opinion (could it not be my opinion?) that a C# version that is
> capable of running on Mono would benefit the community more than a Scala
> version.
>
> My 2 cents,
>
> Andrew
Hi Andrew,
thank you for your immediate response. My university has absolutely nothing to
do with this work. It's just that this work will most probably be accepted for
my studies.
There are many factors which could lead to a different community size around
kafka and zookeeper. One factor could be that zookeeper is a more basic tool
then kafka. Zookeeper is also older.
About the smaller number of scala developers, there may be some wisdom hidden
in a quote from Linus Torvalds: "even if the choice of C were to do *nothing*
but keep the C++ programmers out, that in itself would be a huge reason to use
C".[1]
[1] Linus prefers strong language... http://lwn.net/Articles/249460/
Well, C#... Let's just say that I've an opinion on proprietary technology. Did
you know that Scala can compile to .NET? (I've not tested it and there may be
problems, but proper .NET support is on the roadmap of Scala.)
Best regards,
Thomas Koch, http://www.koch.ro
Re: ZooKeeper cleanup / refactoring / scala migration
Posted by Andrew Finnell <an...@gmail.com>.
Thomas,
Upfront warning. I am not in charge of ZooKeeper or affiliated with them except for submitting patches.
While I understand the power of Scala, what is the major benefit of changing all of the Java code over to Scala? Will this outweigh the decrease in skill sets that a language such as Java brings? Another technology called Kafka is written in Scala. It doesn't have anywhere near the support of ZooKeeper. I attribute most of that to it being written in Scala, which reduces the pool of developers that can or are willing to work on it.
If your university will truly enhance ZooKeeper I would ask, in an unofficial capacity, to ensure there is a strong line of separation between items 1 and 2&3. Also keep in mind that I have run FindBugs on ZK myself. Some of the fixes will break backwards compatibility. Just be aware of that fact.
It is my opinion (could it not be my opinion?) that a C# version that is capable of running on Mono would benefit the community more than a Scala version.
My 2 cents,
Andrew
On Sep 2, 2011, at 9:15 AM, Thomas Koch wrote:
> Hi,
>
> my university labs work has started yesterday. In the next two months I'll
> work on ZooKeeper. This work has three major goals
>
> - improve the maintainability of the code base
> - migrate ZooKeeper to scala and use actors for reliable concurrency
> - find other developers to collaborate on the scala version
>
> I plan to work in these steps:
>
> 1. use static analysis tools to cleanup the java codebase
> ( checkstyle, pmd, findbugs, eclipse compiler warnings )
>
> 2. decrease dependencies in the ZK java packages, break cyclic dependencies
>
> 3. migrate components one-by-one to scala, starting with the leafs of the
> dependency graph
>
> I'd be happy if as much as possible of my work in steps 1. and 2. could also
> be useful for you. What would be a good strategy to go? I assume you're
> concentrating now on getting 3.4 out and don't have time for other things. Can
> I help on 3.4 blockers?
>
> I've set up a gerrit[1] (git based code review) and a jenkins[2] server for my
> project. Jenkins is armed with checkstyle, findbugs, pmd, copy-paste-detection
> and jdepends. I've carefully selected the checks run by checkstyle and pmd and
> would suggest that the remaining warnings should really be eliminated.
>
> Gerrit already hosts two changesets which eliminate nearly all eclipse
> compiler warnings. I'd be happy if you'd like to create an account at the
> gerrit instance (openid needed) and play around with it. I can also give you
> reviewer status which lets you push changes. Every change push will trigger a
> jenkins build.
>
> I've already proposed to infra@a.o that I'd help in setting up a gerrit server
> for the ASF if any project would be interested.
>
> [1] http://koch.ro:8081
> [2] http://koch.ro:8080
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro