Re: [AXIS2] Re: SVN Head - Current Build Time

[Eran Chinthaka <ch...@opensource.lk>]

Tests will take some time to run anyway. The advantage of running tests
is far more better than the cost we pay.
The best option is not to run tests when you build using sources. But
when you run the tests you must be prepared to pay the penalty.

Hope no one will suggest to move this to another goal from jar goal, so
that this won't be included in the test that will run when we type maven
clean jar. I'm 0- on that idea in advance, if some one proposes that (a
precautionary step :) )

-- Chinthaka

Ajith Ranabahu wrote:
> Hmm....
> I did add a test to codegen and compile the famous enterprise.wsdl. I
> admit it's quite bulky and seems it is the culprit for the slower
> build  (and I've been told that in computers with lesser memory,
> things are REALLY bad)
> However the enterprise WSDL seems to be a required part in the test
> harness and I'm not in favor of removing it. Any other ideas ?
> 
> Ajith
> 
> On 4/21/06, Jens Schumann <ml...@void.fm> wrote:
>> Should have been with AXIS2 prefix.
>>
>>> Well - yes I know: I will never get used to the unbelievable large build
>>> times caused by maven. And: I should have gotten a new PowerBook earlier;)
>>>
>>> But since yesterday build times (creating a war from scratch) take almost
>>> 3 1/2 times longer than before. Most of the time is spend in
>>> module/integration. Is there any thing which can be improved in that
>>> module to speed up things?
>>>
>>> Jens
>>>
>>>
>>>
>>
> 
> 
> --
> Ajith Ranabahu
> 

Re: [AXIS2] Re: SVN Head - Current Build Time

[Dennis Sosnoski <dm...@sosnoski.com>]

Ruchith Fernando wrote:

>>The fact that it checks repeatedly smells like a bug to me. Consider
>>what happens if the jar changes in the middle of your build - you have
>>some code built using the old version of the jar, and some with the new.
>>That's dangerous at best, since it means you could end up shipping a
>>broken distribution.
>>    
>>
>
>IMHO this problem will only occur IFF we have SNAPSHOT dependencies,
>since a jar of a released version cannot change. We are trying stick
>to a released version of all the projects that axis2 is dependant on
>before a particular release ... then this problem will not be there.
>  
>
This still appears to be a maven bug, if there's no way of turning off 
the repeated checks. If you *are* using SNAPSHOT dependencies it creates 
the opportunity for an apparently successful built that results in a 
broken distribution (because a SNAPSHOT changed in the middle of your 
build). If you're not using SNAPSHOT dependencies it means that maven is 
checking the same file repeatedly, which is a waste of time - in fact, 
why should it check at all in this case, since the file will always be 
the same.

  - Dennis

Re: [AXIS2] Re: SVN Head - Current Build Time

[Ruchith Fernando <ru...@gmail.com>]

>
> The fact that it checks repeatedly smells like a bug to me. Consider
> what happens if the jar changes in the middle of your build - you have
> some code built using the old version of the jar, and some with the new.
> That's dangerous at best, since it means you could end up shipping a
> broken distribution.

IMHO this problem will only occur IFF we have SNAPSHOT dependencies,
since a jar of a released version cannot change. We are trying stick
to a released version of all the projects that axis2 is dependant on
before a particular release ... then this problem will not be there.

Thanks,
Ruchith

Re: [AXIS2] Re: SVN Head - Current Build Time

[Dennis Sosnoski <dm...@sosnoski.com>]

Bill Nagy wrote:

>In my experience (and maybe others have different problems) the longest
>delays are caused by the repositories not responding/timing out.  The -o
>option is the only way to get around this, as otherwise even the hash
>comparison will cause the timeouts.  Unfortunately if you have an out-
>of-date dependency, as often happens, you're out of luck.  You can use a
>caching proxy to help out, but you still have to worry about keeping the
>cache current.
>  
>
Timeouts are a serious problem, but I find maven annoying even when the 
repositories respond - there's a noticeable delay for each check, and 
the same exact jars are repeatedly checked (and often repeatedly 
downloaded, it seems, though I don't know why). Isn't there a way to 
tell maven it should only try updating each particular jar once within a 
build?

The fact that it checks repeatedly smells like a bug to me. Consider 
what happens if the jar changes in the middle of your build - you have 
some code built using the old version of the jar, and some with the new. 
That's dangerous at best, since it means you could end up shipping a 
broken distribution.

Hopefully someone with a better knowledge of maven can comment on these 
issues.

  - Dennis

Re: [AXIS2] Re: SVN Head - Current Build Time

[Bill Nagy <na...@watson.ibm.com>]

On Sat, 2006-04-22 at 16:03 +0600, Sanjiva Weerawarana wrote:

> The only thing I can think of is to jar up someone's maven repo and put
> it up for download somewhere and say "if u want a quick start, drop this
> in your home directory before you say 'maven'".
> 

The problem is (at least at the moment) that the dependencies change so
often that it would be hard to keep a zip/jar current.

In my experience (and maybe others have different problems) the longest
delays are caused by the repositories not responding/timing out.  The -o
option is the only way to get around this, as otherwise even the hash
comparison will cause the timeouts.  Unfortunately if you have an out-
of-date dependency, as often happens, you're out of luck.  You can use a
caching proxy to help out, but you still have to worry about keeping the
cache current.

-Bill
 

Re: [AXIS2] Re: SVN Head - Current Build Time

[Eran Chinthaka <ch...@opensource.lk>]

Sanjiva Weerawarana wrote:
> On Sat, 2006-04-22 at 11:21 +0200, Jens Schumann wrote:
>> > 
> The only thing I can think of is to jar up someone's maven repo and put
> it up for download somewhere and say "if u want a quick start, drop this
> in your home directory before you say 'maven'".

Remember some one needs to maintain this, every time we change a jar
version or add a new jar. But seems a good compromise.

-- Chinthaka

> 
> Sanjiva.
> 
> 

Re: [AXIS2] Re: SVN Head - Current Build Time

[Sanjiva Weerawarana <sa...@opensource.lk>]

On Sat, 2006-04-22 at 11:21 +0200, Jens Schumann wrote:
> 
> I should have noted that I am up and running now;). What I was referring to
> was just the process on get going: Just assume a user realizes there might
> be a bug or issue in Axis2 and want to help out. Shouldn't it take 1
> 1/2hours before the build spits out the first jar? Luckily maven did create
> invalid idea project files too, so it took even longer on my side.

I agree the first time developer experience is quite a pain right now.
Unfortunately that's become the nature of the beast: maven makes it easy
to depend on other stuff and for that to "magically" come down .. and if
its only just one more from some stage, its not bad. When its so many
new little pieces its a PITA. 

The only thing I can think of is to jar up someone's maven repo and put
it up for download somewhere and say "if u want a quick start, drop this
in your home directory before you say 'maven'".

Sanjiva.

Re: [AXIS2] Re: SVN Head - Current Build Time

[Jens Schumann <ml...@void.fm>]

Eran,


On 4/21/06 8:10 AM "Eran Chinthaka" <ch...@opensource.lk> wrote:

>> To sum up things: To get going with axis2 source is just disappointing. What
>> do I need to do to ensure that all resources are available?
> 
> 1.
... 
> 5. There is a group-id for each jar found project.xml files. Create a
> folder from the name of the group-id. Create a folder called "jars"
> inside it and put the jar there.
> 
> Your local repo is setup.
> 
> From here onwards you can run all maven goals with -o options, and fly
> happily :).

I should have noted that I am up and running now;). What I was referring to
was just the process on get going: Just assume a user realizes there might
be a bug or issue in Axis2 and want to help out. Shouldn't it take 1
1/2hours before the build spits out the first jar? Luckily maven did create
invalid idea project files too, so it took even longer on my side.

Enough complained ...;).

Jens

Re: [AXIS2] Re: SVN Head - Current Build Time

[Eran Chinthaka <ch...@opensource.lk>]

Jens Schumann wrote:
> On 4/21/06 9:56 PM "Glen Daniels" <gl...@thoughtcraft.com> wrote:
> 
> 
> Let me add something here: Just a few days ago I started to play around with
> axis2 sources. Maybe it's just my personal averseness to maven. But this is
> what I did:
> 
> - svn checkout
> - run maven - it took ages, seriously (it's like 70 minutes on my PowerBook
> 1.5Ghz including Apache download timeouts)
> - went offline
> - run maven war, got a new dependency which wasn't downloaded
> - war create just failed.
> 
> To sum up things: To get going with axis2 source is just disappointing. What
> do I need to do to ensure that all resources are available? 

1. Goto each and every module we have in Axis2 and open project.xml
2. All the jars that a particular module dependent upon is listed under
the dependencies section of that project.xml. And some time the versions
of the jars are listed in etc/project.properties
3. The locations all the jars can be downloaded are listed in
"Dependency Repositories" in etc/project.properties.

or you can download all the jars by maven at once. Call "maven
create-lib" and it will get you all the dependent jars to target/lib.

4. create a folder $USER_HOME/.maven/repository
5. There is a group-id for each jar found project.xml files. Create a
folder from the name of the group-id. Create a folder called "jars"
inside it and put the jar there.

Your local repo is setup.

From here onwards you can run all maven goals with -o options, and fly
happily :).

> How can I speed
> up rebuilding my sources? Where do I get sources for axiom/ws policy etc
> from?
> 

Axiom -
https://svn.apache.org/repos/asf/webservices/commons/trunk/modules/axiom

Policy -
https://svn.apache.org/repos/asf/webservices/commons/trunk/modules/policy

-- Chinthaka

Re: [AXIS2] Re: SVN Head - Current Build Time

[Jens Schumann <ml...@void.fm>]

On 4/21/06 9:56 PM "Glen Daniels" <gl...@thoughtcraft.com> wrote:

> As we've discussed before, we should think about having various levels
> of testing included in our available targets.  This makes it feasible to
> have interoperability testing included in our codebase, but not executed
> when a user is offline, or making only small changes.
>
...
> Dev - just run unit tests, no serious integration tests.  This should
> essentially be a "sanity check" for quick development builds.  Should
> not include deep integration tests with complex extensions or big WSDLs.
>   This build should be kept, I'd like to hope, under 5 minutes on a
> reasonable machine.
>
...
> Thoughts?

Let me add something here: Just a few days ago I started to play around with
axis2 sources. Maybe it's just my personal averseness to maven. But this is
what I did:

- svn checkout
- run maven - it took ages, seriously (it's like 70 minutes on my PowerBook
1.5Ghz including Apache download timeouts)
- went offline
- run maven war, got a new dependency which wasn't downloaded
- war create just failed.

To sum up things: To get going with axis2 source is just disappointing. What
do I need to do to ensure that all resources are available? How can I speed
up rebuilding my sources? Where do I get sources for axiom/ws policy etc
from?

Also 5 minutes is just too long for someone looking into it for the first
time. Whenever I create a build management system I tend to keep a rebuild
after minor changes below 15-20 seconds. The lower, the better. (this is
probably because I pay the people who sit there and wait for the build...)
The first build can take longer, but the next one without clean should be
just fast;)

Jens

Re: [AXIS2] Re: SVN Head - Current Build Time

[Glen Daniels <gl...@thoughtcraft.com>]

Hi folks:

Thilina sez:
> I tried to do a small commit and the built took forever to run. Atlast
> Deepal came to my help and commited it for me.. But now I'll get
> second thoughts whenever i need to commit something to Axis2 :(.
> 
> IMO These long and hard build times will cause a negative effect on
> the contributions.. Specially from somebody like me, who don't have a
> super druper machine..

+1

As we've discussed before, we should think about having various levels 
of testing included in our available targets.  This makes it feasible to 
have interoperability testing included in our codebase, but not executed 
when a user is offline, or making only small changes.

IMHO, we should have at least these levels:

Dev - just run unit tests, no serious integration tests.  This should 
essentially be a "sanity check" for quick development builds.  Should 
not include deep integration tests with complex extensions or big WSDLs. 
  This build should be kept, I'd like to hope, under 5 minutes on a 
reasonable machine.

Normal - run most tests, including integration tests, but still no tests 
which require a network connection for interop testing.  This should be 
the default, I think.

Interop - run everything including remote interop suite with public 
endpoints from other vendors.

So while you're working you'd do "maven dev" to build jars and unit 
test, then before checkin you'd make sure "maven jar" passed (the normal 
tests), and every once in a while run "maven interop" (certainly before 
dists, and hopefully on gump once a day).

Thoughts?

--Glen

Re: [AXIS2] Re: SVN Head - Current Build Time

[Jens Schumann <ml...@void.fm>]

> IMO These long and hard build times will cause a negative effect on
> the contributions.. Specially from somebody like me, who don't have a
> super druper machine..

+1.

Keep in mind that you might have people (just like me) who travel a lot and
all they have is a portable. I was working on the admin web app where you
alter two lines and deploy them quickly. In the end I used a few cp and jar
uf instead of maven to speed up things.

Jens

Re: [AXIS2] Re: SVN Head - Current Build Time

[Thilina Gunarathne <cs...@gmail.com>]

Hmmm....
I tried to do a small commit and the built took forever to run. Atlast
Deepal came to my help and commited it for me.. But now I'll get
second thoughts whenever i need to commit something to Axis2 :(.

IMO These long and hard build times will cause a negative effect on
the contributions.. Specially from somebody like me, who don't have a
super druper machine..

~Thilina

On 4/21/06, Eran Chinthaka <ch...@opensource.lk> wrote:
> Tests will take some time to run anyway. The advantage of running tests
> is far more better than the cost we pay.
> The best option is not to run tests when you build using sources. But
> when you run the tests you must be prepared to pay the penalty.
>
> Hope no one will suggest to move this to another goal from jar goal, so
> that this won't be included in the test that will run when we type maven
> clean jar. I'm 0- on that idea in advance, if some one proposes that (a
> precautionary step :) )
>
> -- Chinthaka
>
> Ajith Ranabahu wrote:
> > Hmm....
> > I did add a test to codegen and compile the famous enterprise.wsdl. I
> > admit it's quite bulky and seems it is the culprit for the slower
> > build  (and I've been told that in computers with lesser memory,
> > things are REALLY bad)
> > However the enterprise WSDL seems to be a required part in the test
> > harness and I'm not in favor of removing it. Any other ideas ?
> >
> > Ajith
> >
> > On 4/21/06, Jens Schumann <ml...@void.fm> wrote:
> >> Should have been with AXIS2 prefix.
> >>
> >>> Well - yes I know: I will never get used to the unbelievable large build
> >>> times caused by maven. And: I should have gotten a new PowerBook earlier;)
> >>>
> >>> But since yesterday build times (creating a war from scratch) take almost
> >>> 3 1/2 times longer than before. Most of the time is spend in
> >>> module/integration. Is there any thing which can be improved in that
> >>> module to speed up things?
> >>>
> >>> Jens
> >>>
> >>>
> >>>
> >>
> >
> >
> > --
> > Ajith Ranabahu
> >
>
>
>
>
>


--
"May the SourcE be with u"
http://webservices.apache.org/~thilina/
http://thilinag.blogspot.com/                
http://www.bloglines.com/blog/Thilina