You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Peter Firmstone <pe...@zeus.net.au> on 2014/04/23 01:47:16 UTC
Decision process for a Modular build tool
I started qa-refactor with the intent of fixing latent bugs, an unintentional benefit is significantly reduced processing times, contention and increased scalability.
Changes in timing exposed more bugs.
Up until recently an occassional build failure would be experienced due to classdep only partially writing a dep file, resulting in ClassNotFoundException during testing. Knowing that RFC3986URLClassLoader is much faster resolving classes than URLClassLoader, I thought, I'd try using it in ClassDep.
Guess what the result was? That's right lot's more ClassNotFoundException's
Qa-refactor is stable, but only when the build process succeeds without error.
I now must choose, do I waste time fixing a legacy build tool, or is it time for a modular build to reduce long term maintenance and simplify, lowering the bar for new users?
To me, going modular is a no brainer, but what tool should we use and why?
* Maven
* Gradle
* Ivy & ant
Please keep responses to the point, with any pro's and cons you have, eg:
Maven:
+widely adopted
+well structured dependency relationships
-dependency downloads.
Gradle:
-adoption
+Groovy build scripts
Ivy:
+uses ant, our existing build tool
-adoption
When complete, I suggest, we all can rank the relevance or importance of each point raised (on a scale of -10 to 10) then add up for the final score to decide the tool.
Regards,
Peter.
Re: Decision process for a Modular build tool
Posted by Peter <ji...@zeus.net.au>.
Thanks for the tip.
Cheers,
Peter.
----- Original message -----
>
> On Apr 23, 2014, at 8:01 AM, Peter <ji...@zeus.net.au> wrote:
>
> > ----- Original message -----
> > >
> > > On Apr 22, 2014, at 7:47 PM, Peter Firmstone
> > > <pe...@zeus.net.au> wrote:
> > >
> > > >
> > > >
> > > > From: Peter Firmstone <pe...@zeus.net.au>
> > > > Subject: Decision process for a Modular build tool
> > > > Date: April 22, 2014 at 7:40:29 PM EDT
> > > > To: dev@apache.river.org
> > > > Reply-To: Peter Firmstone <pe...@zeus.net.au>
> > > >
> > > >
> > > > I started qa-refactor with the intent of fixing latent bugs, an
> > > > unintentional benefit is significantly reduced processing times,
> > > > contention and increased scalability.
> > > >
> > > > Changes in timing exposed more bugs.
> > > >
> > > > Up until recently an occassional build failure would be experienced
> > > > due to classdep only partially writing a dep file, resulting in
> > > > ClassNotFoundException during testing. Knowing that
> > > > RFC3986URLClassLoader is much faster resolving classes than
> > > > URLClassLoader, I thought, I'd try using it in ClassDep.
> > > >
> > > > Guess what the result was? That's right lot's more
> > > > ClassNotFoundException's
> > > >
> > > >
> > >
> > > That seems kind of odd. Since ClassDep is single-threaded (it’s
> > > basically a command line utility after all), how would faster class
> > > path resolution have any impact on the output file?
> >
> > Ok, fair call, ClassDep has a bug, I'm not sure of the exact cause.
>
> I would suggest just adding a call to System.out.flush() at the end of
> main() and even System.out.close() just because you may be using a
> broken library that is not successfully flushing and closing. Look at
> the size of the files that are output. Are they multiples of some power
> of 2 that would be like a block write size? That would indicate that
> blocks are being written as a blocks worth of output is created.
>
> Gregg Wonderly
>
Re: Decision process for a Modular build tool
Posted by Gregg Wonderly <gr...@wonderly.org>.
On Apr 23, 2014, at 8:01 AM, Peter <ji...@zeus.net.au> wrote:
> ----- Original message -----
>>
>> On Apr 22, 2014, at 7:47 PM, Peter Firmstone
>> <pe...@zeus.net.au> wrote:
>>
>>>
>>>
>>> From: Peter Firmstone <pe...@zeus.net.au>
>>> Subject: Decision process for a Modular build tool
>>> Date: April 22, 2014 at 7:40:29 PM EDT
>>> To: dev@apache.river.org
>>> Reply-To: Peter Firmstone <pe...@zeus.net.au>
>>>
>>>
>>> I started qa-refactor with the intent of fixing latent bugs, an
>>> unintentional benefit is significantly reduced processing times,
>>> contention and increased scalability.
>>>
>>> Changes in timing exposed more bugs.
>>>
>>> Up until recently an occassional build failure would be experienced
>>> due to classdep only partially writing a dep file, resulting in
>>> ClassNotFoundException during testing. Knowing that
>>> RFC3986URLClassLoader is much faster resolving classes than
>>> URLClassLoader, I thought, I'd try using it in ClassDep.
>>>
>>> Guess what the result was? That's right lot's more
>>> ClassNotFoundException's
>>>
>>>
>>
>> That seems kind of odd. Since ClassDep is single-threaded (it’s
>> basically a command line utility after all), how would faster class path
>> resolution have any impact on the output file?
>
> Ok, fair call, ClassDep has a bug, I'm not sure of the exact cause.
I would suggest just adding a call to System.out.flush() at the end of main() and even System.out.close() just because you may be using a broken library that is not successfully flushing and closing. Look at the size of the files that are output. Are they multiples of some power of 2 that would be like a block write size? That would indicate that blocks are being written as a blocks worth of output is created.
Gregg Wonderly
Re: Decision process for a Modular build tool
Posted by Peter <ji...@zeus.net.au>.
----- Original message -----
>
> On Apr 22, 2014, at 7:47 PM, Peter Firmstone
> <pe...@zeus.net.au> wrote:
>
> >
> >
> > From: Peter Firmstone <pe...@zeus.net.au>
> > Subject: Decision process for a Modular build tool
> > Date: April 22, 2014 at 7:40:29 PM EDT
> > To: dev@apache.river.org
> > Reply-To: Peter Firmstone <pe...@zeus.net.au>
> >
> >
> > I started qa-refactor with the intent of fixing latent bugs, an
> > unintentional benefit is significantly reduced processing times,
> > contention and increased scalability.
> >
> > Changes in timing exposed more bugs.
> >
> > Up until recently an occassional build failure would be experienced
> > due to classdep only partially writing a dep file, resulting in
> > ClassNotFoundException during testing. Knowing that
> > RFC3986URLClassLoader is much faster resolving classes than
> > URLClassLoader, I thought, I'd try using it in ClassDep.
> >
> > Guess what the result was? That's right lot's more
> > ClassNotFoundException's
> >
> >
>
> That seems kind of odd. Since ClassDep is single-threaded (it’s
> basically a command line utility after all), how would faster class path
> resolution have any impact on the output file?
Ok, fair call, ClassDep has a bug, I'm not sure of the exact cause.
What exactly do you mean
> by “only partially writing a dep file”? Does it not get closed off?
> Are characters omitted?
Yes, the dep list is incomplete and appears to be cut short.
Do you have examples of the incomplete “del”
> files? The once or twice I’ve looked at it in the past, the classes in
> question were actually in the jar files on the class path, which
> wouldn’t seem to point to ClassDep.
Have a look at the latest jdk7 build, while it still exists, the jar files for reggie's smart proxy are missing class files, reggie-dl.dep has been truncated and is incomplete.
When I have previously thought a class was present on the classpath, I was looking in the wrong jar file. The class files were compiled, just not included in the right jar.
Is it possible that there’s a
> problem with the new ClassLoader (RFC3986URLClassLoader)?
The bug is still present with URLClassLoader, but occurs less often.
The good news is failures are more prevalent on Jenkins, I seldom see it elsewhere.
> > Qa-refactor is stable, but only when the build process succeeds
> > without error.
> >
> > I now must choose, do I waste time fixing a legacy build tool, or is
> > it time for a modular build to reduce long term maintenance and
> > simplify, lowering the bar for new users?
> >
> >
> The question of modular builds is orthogonal to problems with either
> ClassDep or the new ClassLoader you’re talking about. It makes the most
> sense to get a solid diagnosis (and probably fix) for the problem as it
> has presented itself, rather than diving into a new source code layout.
> Not to mention that there may actually be users of ClassDepAndJar, apart
> from the River build.
>
ClassDep presents a significant maintenance burden for the project, OSGi offers a similar tool for dependency analysis.
I upgraded ClassDep to dependency analyse Java 5 language features, that bought time and allowed people to continue using it. I don't have time to maintain it and enable support of Java 8 language features.
The use of other tools for River's build process won't prevent others using or maintaining ClassDep or its continued use in the 2.2 maintenance branch.
ClassDep is blocking reliable test runs and progress on the qa-refactor branch.
If you feel strongly about class dep, add it to the list with pro's and cons and we can score it against modular build options.
> Cheers,
>
> Greg Trasuk.
> > To me, going modular is a no brainer, but what tool should we use and
> > why?
> >
> > * Maven
> > * Gradle
> > * Ivy & ant
> >
> > Please keep responses to the point, with any pro's and cons you have,
> > eg:
> >
> > Maven:
> > +widely adopted
> > +well structured dependency relationships
> > -dependency downloads.
> >
> > Gradle:
> > -adoption
> > +Groovy build scripts
> >
> > Ivy:
> > +uses ant, our existing build tool
> > -adoption
> >
> > When complete, I suggest, we all can rank the relevance or importance
> > of each point raised (on a scale of -10 to 10) then add up for the
> > final score to decide the tool.
> >
> > Regards,
> >
> > Peter.
> >
> >
> >
>
Re: Decision process for a Modular build tool
Posted by Greg Trasuk <tr...@stratuscom.com>.
On Apr 22, 2014, at 7:47 PM, Peter Firmstone <pe...@zeus.net.au> wrote:
>
>
> From: Peter Firmstone <pe...@zeus.net.au>
> Subject: Decision process for a Modular build tool
> Date: April 22, 2014 at 7:40:29 PM EDT
> To: dev@apache.river.org
> Reply-To: Peter Firmstone <pe...@zeus.net.au>
>
>
> I started qa-refactor with the intent of fixing latent bugs, an unintentional benefit is significantly reduced processing times, contention and increased scalability.
>
> Changes in timing exposed more bugs.
>
> Up until recently an occassional build failure would be experienced due to classdep only partially writing a dep file, resulting in ClassNotFoundException during testing. Knowing that RFC3986URLClassLoader is much faster resolving classes than URLClassLoader, I thought, I'd try using it in ClassDep.
>
> Guess what the result was? That's right lot's more ClassNotFoundException's
>
>
That seems kind of odd. Since ClassDep is single-threaded (it’s basically a command line utility after all), how would faster class path resolution have any impact on the output file? What exactly do you mean by “only partially writing a dep file”? Does it not get closed off? Are characters omitted? Do you have examples of the incomplete “del” files? The once or twice I’ve looked at it in the past, the classes in question were actually in the jar files on the class path, which wouldn’t seem to point to ClassDep. Is it possible that there’s a problem with the new ClassLoader (RFC3986URLClassLoader)?
> Qa-refactor is stable, but only when the build process succeeds without error.
>
> I now must choose, do I waste time fixing a legacy build tool, or is it time for a modular build to reduce long term maintenance and simplify, lowering the bar for new users?
>
>
The question of modular builds is orthogonal to problems with either ClassDep or the new ClassLoader you’re talking about. It makes the most sense to get a solid diagnosis (and probably fix) for the problem as it has presented itself, rather than diving into a new source code layout. Not to mention that there may actually be users of ClassDepAndJar, apart from the River build.
Cheers,
Greg Trasuk.
> To me, going modular is a no brainer, but what tool should we use and why?
>
> * Maven
> * Gradle
> * Ivy & ant
>
> Please keep responses to the point, with any pro's and cons you have, eg:
>
> Maven:
> +widely adopted
> +well structured dependency relationships
> -dependency downloads.
>
> Gradle:
> -adoption
> +Groovy build scripts
>
> Ivy:
> +uses ant, our existing build tool
> -adoption
>
> When complete, I suggest, we all can rank the relevance or importance of each point raised (on a scale of -10 to 10) then add up for the final score to decide the tool.
>
> Regards,
>
> Peter.
>
>
>
Re: Decision process for a Modular build tool
Posted by Zsolt Kúti <la...@gmail.com>.
On Wed, Apr 23, 2014 at 1:47 AM, Peter Firmstone
<pe...@zeus.net.au> wrote:
> To me, going modular is a no brainer, but what tool should we use and why?
>
> * Maven
> * Gradle
> * Ivy & ant
>
> Please keep responses to the point, with any pro's and cons you have, eg:
While deciding on a possible build system is certainly the business of
who participate in development, I would add some thoughts to gradle:
- how to judge adoption
- gradle dependency handling is outstanding
- transiting from ant can first be done by using what we already have,
ant is a first class citizen
- gradle folks are certainly willing to help in switching a project of
this caliber
- for more thoughts please check this out:
http://www.gradleware.com/resources/tech/java/maven
- mainstream IDEs support is not going to be a shortcoming:
http://www.gradle.org/tooling (I know only IDEA where it works nice)
Zsolt