You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Mark Struberg <st...@yahoo.de> on 2011/07/31 17:43:22 UTC
Random errors in parallely executed tests
Hi!
While browsing our CI results I saw randomly failing tests. Most of them are caused by starting jetty on a specific port which already is in use by another test.
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
Clearly if 2 unit tests fire up a jetty instance on the same port, then this cannot run in parallel
Proposal:
What if we add a property in the maven-surefire-plugin to name a semaphore resource?
<resource>port1080</resource>
This could be a comma separated list. Surefire will then check if any resource is blocked before scheduling the next test.
The problematic part of course is that surefire must communicate this over multiple parallel builds.
WDYT?
LieGrue,
strub
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Random errors in parallely executed tests
Posted by Hervé BOUTEMY <he...@free.fr>.
ah, I knew you did something like this but could not find it :)
question: ok, random port, but random and guaranteed to success? ie it makes
the effort to find an unused port?
see r1152587: I chose an arbitrary port
we should add this to a FAQ, since this is a common use case
Regards,
Hervé
Le dimanche 31 juillet 2011, Olivier Lamy a écrit :
> Hello
> Jetty is normally able to start on a random port.
> I have fixed that recently in archetype.
> Imho better solution.
> Which tests fail for this reason ?
>
> --
> Olivier
> send from a mobile
>
> Le 31 juil. 2011 17:43, "Mark Struberg" <st...@yahoo.de> a écrit :
> > Hi!
> >
> > While browsing our CI results I saw randomly failing tests. Most of them
>
> are caused by starting jetty on a specific port which already is in use by
> another test.
>
> > java.net.BindException: Address already in use
> > at sun.nio.ch.Net.bind(Native Method)
> >
> > Clearly if 2 unit tests fire up a jetty instance on the same port, then
>
> this cannot run in parallel
>
> > Proposal:
> >
> > What if we add a property in the maven-surefire-plugin to name a
> > semaphore
>
> resource?
>
> > <resource>port1080</resource>
> >
> > This could be a comma separated list. Surefire will then check if any
>
> resource is blocked before scheduling the next test.
>
> > The problematic part of course is that surefire must communicate this
> > over
>
> multiple parallel builds.
>
> > WDYT?
> >
> > LieGrue,
> > strub
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Random errors in parallely executed tests
Posted by Rex Hoffman <re...@e-hoffman.org>.
That's correct, but you'll then have to query the content handler object for
it's currently running port.
I'm going to keep pimping out this solution I built for some clients and
open sourced a while back.
In one project I have an api for starting up a jetty application server
either off of the eclipse compiled classpath or based on a war file found in
your target directory (you can implement an interface to construct new
methods of application look up). I have some fluent API code I will be
pushing in to this soon.
Personally I don't believe we should ever be starting jetty from maven pom
files, rather that task should be done as part of test start-up,
and preferable reused across all tests hitting that app. Which would allow
tests to be easily run from with in eclipse (starting the webapp when
needed, and allowing hot code replacement when debugging). This just ends
up making devs more productive.
I've got that capability, as well as retaining complex, reusable testing
components through the full test execution here:
https://github.com/rexhoffman/test-extensions
For the eclipse based mechanism to work, simply add a webapps.properties
file to your src/main/resource dir, and the util will pull web-content from
src/main/webapp/ or WebContent/
I'll be adding more doc to these in the next couple weeks, but the tests
should be good enough examples to get this going if anyone is interested. I
will also answer questions.
I really, really want to see maven plugins like jetty/tomcat/cassandra/etc
die off. A good util package added in to tests to start applications
locally prior to testing is more usable, more easily debuggable, and just
plain cleaner. A good api on them wouldn't let the developer set a port,
but instead only query the randomly selected port. Preventing threading
issues like this. File system access should also be carefully managed by a
java api used by developers, that prevents garbage being left in target dir
or some other resource path that might interfere with other tests.
My long winded 2 cents.
Oh and I do have a ticket open to push this to central, and these are apache
licensed. Feel free to copy.
Rex
On Sun, Jul 31, 2011 at 1:08 PM, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:
> afaik since always. standard socket.open(port) behaviour when port is 0.
>
> - Stephen
>
> ---
> Sent from my Android phone, so random spelling mistakes, random nonsense
> words and other nonsense are a direct result of using swype to type on the
> screen
> On 31 Jul 2011 20:00, "Mark Struberg" <st...@yahoo.de> wrote:
> > yea, that is good news. Will try to tweak this for all wagon tests too.
> > Do you know since when this works? Is this available in jetty-6.1.26?
> >
> > Btw, when going through our Tests I found lots of tests which do not
> cleanup the target/local-repo (or similar) before they run. Some of them
> only work because they work on left-overs from previous tests. This is
> pretty dangerous and should get fixed asap.
> >
> > I didn't touch them yet because I first like to get the IT on our CI box
> working again.
> >
> > LieGrue,
> > strub
> >
> >
> > --- On Sun, 7/31/11, Stephen Connolly <st...@gmail.com>
> wrote:
> >
> >> From: Stephen Connolly <st...@gmail.com>
> >> Subject: Re: Random errors in parallely executed tests
> >> To: "Maven Developers List" <de...@maven.apache.org>
> >> Date: Sunday, July 31, 2011, 6:28 PM
> >> the buildhelper goal is for when you
> >> are launching jetty/etc from the pom.
> >> or when you need to filter the port into on disk
> >> resources.
> >>
> >> if launching within the test case, just assign it to port
> >> 0. a request for
> >> port 0 is a request for an unused port in the top 16k (or
> >> depending on the
> >> os, anywhere above 1024). then you ask the connector what
> >> port it bound to.
> >>
> >> - Stephen
> >>
> >> ---
> >> Sent from my Android phone, so random spelling mistakes,
> >> random nonsense
> >> words and other nonsense are a direct result of using swype
> >> to type on the
> >> screen
> >> On 31 Jul 2011 18:40, "Benson Margulies" <bi...@gmail.com>
> >> wrote:
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
>
Re: Random errors in parallely executed tests
Posted by Stephen Connolly <st...@gmail.com>.
afaik since always. standard socket.open(port) behaviour when port is 0.
- Stephen
---
Sent from my Android phone, so random spelling mistakes, random nonsense
words and other nonsense are a direct result of using swype to type on the
screen
On 31 Jul 2011 20:00, "Mark Struberg" <st...@yahoo.de> wrote:
> yea, that is good news. Will try to tweak this for all wagon tests too.
> Do you know since when this works? Is this available in jetty-6.1.26?
>
> Btw, when going through our Tests I found lots of tests which do not
cleanup the target/local-repo (or similar) before they run. Some of them
only work because they work on left-overs from previous tests. This is
pretty dangerous and should get fixed asap.
>
> I didn't touch them yet because I first like to get the IT on our CI box
working again.
>
> LieGrue,
> strub
>
>
> --- On Sun, 7/31/11, Stephen Connolly <st...@gmail.com>
wrote:
>
>> From: Stephen Connolly <st...@gmail.com>
>> Subject: Re: Random errors in parallely executed tests
>> To: "Maven Developers List" <de...@maven.apache.org>
>> Date: Sunday, July 31, 2011, 6:28 PM
>> the buildhelper goal is for when you
>> are launching jetty/etc from the pom.
>> or when you need to filter the port into on disk
>> resources.
>>
>> if launching within the test case, just assign it to port
>> 0. a request for
>> port 0 is a request for an unused port in the top 16k (or
>> depending on the
>> os, anywhere above 1024). then you ask the connector what
>> port it bound to.
>>
>> - Stephen
>>
>> ---
>> Sent from my Android phone, so random spelling mistakes,
>> random nonsense
>> words and other nonsense are a direct result of using swype
>> to type on the
>> screen
>> On 31 Jul 2011 18:40, "Benson Margulies" <bi...@gmail.com>
>> wrote:
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Re: Random errors in parallely executed tests
Posted by Mark Struberg <st...@yahoo.de>.
yea, that is good news. Will try to tweak this for all wagon tests too.
Do you know since when this works? Is this available in jetty-6.1.26?
Btw, when going through our Tests I found lots of tests which do not cleanup the target/local-repo (or similar) before they run. Some of them only work because they work on left-overs from previous tests. This is pretty dangerous and should get fixed asap.
I didn't touch them yet because I first like to get the IT on our CI box working again.
LieGrue,
strub
--- On Sun, 7/31/11, Stephen Connolly <st...@gmail.com> wrote:
> From: Stephen Connolly <st...@gmail.com>
> Subject: Re: Random errors in parallely executed tests
> To: "Maven Developers List" <de...@maven.apache.org>
> Date: Sunday, July 31, 2011, 6:28 PM
> the buildhelper goal is for when you
> are launching jetty/etc from the pom.
> or when you need to filter the port into on disk
> resources.
>
> if launching within the test case, just assign it to port
> 0. a request for
> port 0 is a request for an unused port in the top 16k (or
> depending on the
> os, anywhere above 1024). then you ask the connector what
> port it bound to.
>
> - Stephen
>
> ---
> Sent from my Android phone, so random spelling mistakes,
> random nonsense
> words and other nonsense are a direct result of using swype
> to type on the
> screen
> On 31 Jul 2011 18:40, "Benson Margulies" <bi...@gmail.com>
> wrote:
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Random errors in parallely executed tests
Posted by Stephen Connolly <st...@gmail.com>.
the buildhelper goal is for when you are launching jetty/etc from the pom.
or when you need to filter the port into on disk resources.
if launching within the test case, just assign it to port 0. a request for
port 0 is a request for an unused port in the top 16k (or depending on the
os, anywhere above 1024). then you ask the connector what port it bound to.
- Stephen
---
Sent from my Android phone, so random spelling mistakes, random nonsense
words and other nonsense are a direct result of using swype to type on the
screen
On 31 Jul 2011 18:40, "Benson Margulies" <bi...@gmail.com> wrote:
Re: Random errors in parallely executed tests
Posted by Benson Margulies <bi...@gmail.com>.
the build helper has a goal for this.
On Jul 31, 2011, at 12:09 PM, Mark Struberg <st...@yahoo.de> wrote:
> Hi Olivier!
>
> Do you have a sample how to incorporate this into our unit tests? I only knew that jetty is able to assign a random port for the stop port, but didn't know that the same exists for the serving port.
>
> LieGrue,
> strub
>
> --- On Sun, 7/31/11, Olivier Lamy <ol...@apache.org> wrote:
>
>> From: Olivier Lamy <ol...@apache.org>
>> Subject: Re: Random errors in parallely executed tests
>> To: "Maven Developers List" <de...@maven.apache.org>
>> Date: Sunday, July 31, 2011, 3:55 PM
>> Hello
>> Jetty is normally able to start on a random port.
>> I have fixed that recently in archetype.
>> Imho better solution.
>> Which tests fail for this reason ?
>>
>> --
>> Olivier
>> send from a mobile
>> Le 31 juil. 2011 17:43, "Mark Struberg" <st...@yahoo.de>
>> a écrit :
>>> Hi!
>>>
>>> While browsing our CI results I saw randomly failing
>> tests. Most of them
>> are caused by starting jetty on a specific port which
>> already is in use by
>> another test.
>>>
>>> java.net.BindException: Address already in use
>>> at sun.nio.ch.Net.bind(Native Method)
>>>
>>> Clearly if 2 unit tests fire up a jetty instance on
>> the same port, then
>> this cannot run in parallel
>>>
>>>
>>> Proposal:
>>>
>>> What if we add a property in the maven-surefire-plugin
>> to name a semaphore
>> resource?
>>>
>>> <resource>port1080</resource>
>>>
>>> This could be a comma separated list. Surefire will
>> then check if any
>> resource is blocked before scheduling the next test.
>>> The problematic part of course is that surefire must
>> communicate this over
>> multiple parallel builds.
>>>
>>> WDYT?
>>>
>>> LieGrue,
>>> strub
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Random errors in parallely executed tests
Posted by Mark Struberg <st...@yahoo.de>.
Hi Olivier!
Do you have a sample how to incorporate this into our unit tests? I only knew that jetty is able to assign a random port for the stop port, but didn't know that the same exists for the serving port.
LieGrue,
strub
--- On Sun, 7/31/11, Olivier Lamy <ol...@apache.org> wrote:
> From: Olivier Lamy <ol...@apache.org>
> Subject: Re: Random errors in parallely executed tests
> To: "Maven Developers List" <de...@maven.apache.org>
> Date: Sunday, July 31, 2011, 3:55 PM
> Hello
> Jetty is normally able to start on a random port.
> I have fixed that recently in archetype.
> Imho better solution.
> Which tests fail for this reason ?
>
> --
> Olivier
> send from a mobile
> Le 31 juil. 2011 17:43, "Mark Struberg" <st...@yahoo.de>
> a écrit :
> > Hi!
> >
> > While browsing our CI results I saw randomly failing
> tests. Most of them
> are caused by starting jetty on a specific port which
> already is in use by
> another test.
> >
> > java.net.BindException: Address already in use
> > at sun.nio.ch.Net.bind(Native Method)
> >
> > Clearly if 2 unit tests fire up a jetty instance on
> the same port, then
> this cannot run in parallel
> >
> >
> > Proposal:
> >
> > What if we add a property in the maven-surefire-plugin
> to name a semaphore
> resource?
> >
> > <resource>port1080</resource>
> >
> > This could be a comma separated list. Surefire will
> then check if any
> resource is blocked before scheduling the next test.
> > The problematic part of course is that surefire must
> communicate this over
> multiple parallel builds.
> >
> > WDYT?
> >
> > LieGrue,
> > strub
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Random errors in parallely executed tests
Posted by Olivier Lamy <ol...@apache.org>.
Hello
Jetty is normally able to start on a random port.
I have fixed that recently in archetype.
Imho better solution.
Which tests fail for this reason ?
--
Olivier
send from a mobile
Le 31 juil. 2011 17:43, "Mark Struberg" <st...@yahoo.de> a écrit :
> Hi!
>
> While browsing our CI results I saw randomly failing tests. Most of them
are caused by starting jetty on a specific port which already is in use by
another test.
>
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind(Native Method)
>
> Clearly if 2 unit tests fire up a jetty instance on the same port, then
this cannot run in parallel
>
>
> Proposal:
>
> What if we add a property in the maven-surefire-plugin to name a semaphore
resource?
>
> <resource>port1080</resource>
>
> This could be a comma separated list. Surefire will then check if any
resource is blocked before scheduling the next test.
> The problematic part of course is that surefire must communicate this over
multiple parallel builds.
>
> WDYT?
>
> LieGrue,
> strub
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>