You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Francesco Chicchiriccò <il...@apache.org> on 2012/05/02 17:19:15 UTC

Errors in https://builds.apache.org/job/Syncope-windows/

Hi,
we are experiencing repeated failures on Windows probably due to few 
zombie processes belonging to some former build.

As you can see from [1], in fact, ports 8082 and 9080 are already in use.

I personally don't understand how this could happen: every build, even 
with failures should always exit cleanly, but the situation needs some 
manual intervention in order to kill such zombie processes.

Thanks for your support.
Regards.

[1] 
https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log 


-- 
Francesco Chicchiriccò

Apache Cocoon PMC and Apache Syncope PPMC Member
http://people.apache.org/~ilgrosso/

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Wed, May 2, 2012 at 7:45 PM, Francesco Chicchiriccò
<il...@apache.org> wrote:
> Because of this, I would think that fixed port numbers will help avoiding
> major problems - as reported by Emmanuel above.
> If you still think it would be better to have random port numbers, we could
> explore build-helper-maven-plugin:reserve-network-port in order to get all
> needed network ports for running integration tests; as reported above, in
> fact, it's not just Tomcat.

Since you're running in a shared environment, it's always a bad thing
to assume that a certain port is free. Both ports in the case are
frequently used by JVM developers so I would expect conflicts. That
does not only sucks in cases such as shared CI environments but also
on some random developers machine who might not be able to run a your
build due to her having something using one of those ports.

Personally, I always use port=0 in all my unit tests for these exact reasons.

/niklas

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by ma...@apache.org.
"Francesco Chicchiriccò" <il...@apache.org> wrote:

>On 03/05/2012 14:49, Fabio Martelli wrote:
>> [...]
>> Hi guys,
>> I've just committed some changes to close an issue but the build on
>windows has been failed again.
>>
>> The reason is the following
>>
>> Caused by: org.codehaus.cargo.container.ContainerException: Failed to
>stop the Tomcat 7.x container. Check the
>[F:\hudson\hudson-slave\workspace\Syncope-windows\core\target/log/cargo-output.log]
>file containing the container logs for more details.
>> 	at
>org.codehaus.cargo.container.spi.AbstractLocalContainer.stop(AbstractLocalContainer.java:220)
>> 	at
>org.codehaus.cargo.maven2.ContainerStopMojo.doExecute(ContainerStopMojo.java:49)
>> 	at
>org.codehaus.cargo.maven2.AbstractCargoMojo.execute(AbstractCargoMojo.java:311)
>> 	at
>org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>> 	... 28 more
>> Caused by: org.codehaus.cargo.container.ContainerException: Server
>port 9080 did not shutdown within the timeout period [120000]
>> 	at
>org.codehaus.cargo.container.spi.AbstractLocalContainer.waitForPortShutdown(AbstractLocalContainer.java:390)
>>
>>
>> Could you kill zombie processes again?
>> Probably we have found a temporary workaround to avoid this
>situation.
>
>Hi all,
>it seems (looking at the end of [1]) that the root cause for not being 
>able to stop Tomcat is
>
>Error occurred during initialization of VM
>Could not reserve enough space for object heap
>
>The strange thing is that this error is given AFTER the (successful) 
>execution of all integration tests.

That sounds like you have configured the heap via JAVA_OPTS rather than CATALINA_OPTS.

Mark

>
>Any hint?
>Thanks.
>
>[1] 
>https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log
>
>
>
>-- 
>Francesco Chicchiriccò
>
>Apache Cocoon PMC and Apache Syncope PPMC Member
>http://people.apache.org/~ilgrosso/

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Olivier Lamy <ol...@apache.org>.
Using cargo plugin means you start an other java process to start a
Tomcat instance to run your integration tests (the windauze box can be
resources limited if other jobs are running)
What I personally do to run such it tests (selenium by example) is
using the tomcat maven plugin to start an embeded tomcat instance and
run tests. As it you don't start an other java process in the machine
and you can control memory settings using MAVEN_OPTS.
Have a look at this pom [1] to see how that works.

HTH,
-- 
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy

[1] http://svn.apache.org/repos/asf/archiva/trunk/archiva-modules/archiva-web/archiva-webapp-test-js/pom.xml

2012/5/4 Francesco Chicchiriccò <il...@apache.org>:
> On 03/05/2012 14:49, Fabio Martelli wrote:
>
> [...]
>
> Hi guys,
> I've just committed some changes to close an issue but the build on windows
> has been failed again.
>
> The reason is the following
>
> Caused by: org.codehaus.cargo.container.ContainerException: Failed to stop
> the Tomcat 7.x container. Check the
> [F:\hudson\hudson-slave\workspace\Syncope-windows\core\target/log/cargo-output.log]
> file containing the container logs for more details.
> 	at
> org.codehaus.cargo.container.spi.AbstractLocalContainer.stop(AbstractLocalContainer.java:220)
> 	at
> org.codehaus.cargo.maven2.ContainerStopMojo.doExecute(ContainerStopMojo.java:49)
> 	at
> org.codehaus.cargo.maven2.AbstractCargoMojo.execute(AbstractCargoMojo.java:311)
> 	at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> 	... 28 more
> Caused by: org.codehaus.cargo.container.ContainerException: Server port 9080
> did not shutdown within the timeout period [120000]
> 	at
> org.codehaus.cargo.container.spi.AbstractLocalContainer.waitForPortShutdown(AbstractLocalContainer.java:390)
>
>
> Could you kill zombie processes again?
> Probably we have found a temporary workaround to avoid this situation.
>
>
> Hi all,
> it seems (looking at the end of [1]) that the root cause for not being able
> to stop Tomcat is
>
> Error occurred during initialization of VM
> Could not reserve enough space for object heap
>
> The strange thing is that this error is given AFTER the (successful)
> execution of all integration tests.
>
> Any hint?
> Thanks.
>
>
> [1]
> https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log
>
> --
> Francesco Chicchiriccò
>
> Apache Cocoon PMC and Apache Syncope PPMC Member
> http://people.apache.org/~ilgrosso/

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Francesco Chicchiriccò <il...@apache.org>.
On 03/05/2012 14:49, Fabio Martelli wrote:
> [...]
> Hi guys,
> I've just committed some changes to close an issue but the build on windows has been failed again.
>
> The reason is the following
>
> Caused by: org.codehaus.cargo.container.ContainerException: Failed to stop the Tomcat 7.x container. Check the [F:\hudson\hudson-slave\workspace\Syncope-windows\core\target/log/cargo-output.log] file containing the container logs for more details.
> 	at org.codehaus.cargo.container.spi.AbstractLocalContainer.stop(AbstractLocalContainer.java:220)
> 	at org.codehaus.cargo.maven2.ContainerStopMojo.doExecute(ContainerStopMojo.java:49)
> 	at org.codehaus.cargo.maven2.AbstractCargoMojo.execute(AbstractCargoMojo.java:311)
> 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> 	... 28 more
> Caused by: org.codehaus.cargo.container.ContainerException: Server port 9080 did not shutdown within the timeout period [120000]
> 	at org.codehaus.cargo.container.spi.AbstractLocalContainer.waitForPortShutdown(AbstractLocalContainer.java:390)
>
>
> Could you kill zombie processes again?
> Probably we have found a temporary workaround to avoid this situation.

Hi all,
it seems (looking at the end of [1]) that the root cause for not being 
able to stop Tomcat is

Error occurred during initialization of VM
Could not reserve enough space for object heap

The strange thing is that this error is given AFTER the (successful) 
execution of all integration tests.

Any hint?
Thanks.

[1] 
https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log 


-- 
Francesco Chicchiriccò

Apache Cocoon PMC and Apache Syncope PPMC Member
http://people.apache.org/~ilgrosso/

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Fabio Martelli <fa...@gmail.com>.
Il giorno 02/mag/2012, alle ore 19.45, Francesco Chicchiriccò ha scritto:

> On 02/05/2012 18:39, Emmanuel Lécharny wrote:
>> Le 5/2/12 5:33 PM, Niklas Gustavsson a écrit :
>>> On Wed, May 2, 2012 at 5:19 PM, Francesco Chicchiriccò
>>> <il...@apache.org>  wrote:
>>>> we are experiencing repeated failures on Windows probably due to few zombie
>>>> processes belonging to some former build.
>>>> 
>>>> As you can see from [1], in fact, ports 8082 and 9080 are already in use.
>>>> 
>>>> I personally don't understand how this could happen: every build, even with
>>>> failures should always exit cleanly, but the situation needs some manual
>>>> intervention in order to kill such zombie processes.
>>>> 
>>>> Thanks for your support.
>>>> Regards.
>>>> 
>>>> [1]
>>>> https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log 
>>> Should be fixed now. But, I would argue that it's never a good idea to
>>> run tests on fixed ports. You're running in a shared environment (or
>>> on some random developer's box) where you can never know that these
>>> ports are available. Instead, if at all possible, let the OS pick a
>>> free port for you.
>>> 
>>> /niklas
>> Hi Niklas,
>> 
>> I totally agree with that. One thing to consider though : using port 0 to get Tomcat to pick a random port might lead to many ports remaining opened, if the cleanup is not done correctly...
> 
> I believe that the right cleanup is currently performed; there are some known conditions though (for example OutOfMemoryError: PermGen space) leading sometimes Tomcat processes to hang: the only effective cleanup I know in such cases is a good old "kill -9".
> 
> Because of this, I would think that fixed port numbers will help avoiding major problems - as reported by Emmanuel above.
> If you still think it would be better to have random port numbers, we could explore build-helper-maven-plugin:reserve-network-port in order to get all needed network ports for running integration tests; as reported above, in fact, it's not just Tomcat.

Hi guys,
I've just committed some changes to close an issue but the build on windows has been failed again.

The reason is the following

Caused by: org.codehaus.cargo.container.ContainerException: Failed to stop the Tomcat 7.x container. Check the [F:\hudson\hudson-slave\workspace\Syncope-windows\core\target/log/cargo-output.log] file containing the container logs for more details.
	at org.codehaus.cargo.container.spi.AbstractLocalContainer.stop(AbstractLocalContainer.java:220)
	at org.codehaus.cargo.maven2.ContainerStopMojo.doExecute(ContainerStopMojo.java:49)
	at org.codehaus.cargo.maven2.AbstractCargoMojo.execute(AbstractCargoMojo.java:311)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
	... 28 more
Caused by: org.codehaus.cargo.container.ContainerException: Server port 9080 did not shutdown within the timeout period [120000]
	at org.codehaus.cargo.container.spi.AbstractLocalContainer.waitForPortShutdown(AbstractLocalContainer.java:390)


Could you kill zombie processes again?
Probably we have found a temporary workaround to avoid this situation.

Thank you in advance.

Regards,
F.

> Regards.
> 
> -- 
> Francesco Chicchiriccò
> 
> Apache Cocoon Committer and PMC Member
> http://people.apache.org/~ilgrosso/
>

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 7:45 PM, Francesco Chicchiriccò a écrit :
> On 02/05/2012 18:39, Emmanuel Lécharny wrote:
>> Le 5/2/12 5:33 PM, Niklas Gustavsson a écrit :
>>> On Wed, May 2, 2012 at 5:19 PM, Francesco Chicchiriccò
>>> <il...@apache.org>  wrote:
>>>> we are experiencing repeated failures on Windows probably due to 
>>>> few zombie
>>>> processes belonging to some former build.
>>>>
>>>> As you can see from [1], in fact, ports 8082 and 9080 are already 
>>>> in use.
>>>>
>>>> I personally don't understand how this could happen: every build, 
>>>> even with
>>>> failures should always exit cleanly, but the situation needs some 
>>>> manual
>>>> intervention in order to kill such zombie processes.
>>>>
>>>> Thanks for your support.
>>>> Regards.
>>>>
>>>> [1]
>>>> https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log 
>>>>
>>> Should be fixed now. But, I would argue that it's never a good idea to
>>> run tests on fixed ports. You're running in a shared environment (or
>>> on some random developer's box) where you can never know that these
>>> ports are available. Instead, if at all possible, let the OS pick a
>>> free port for you.
>>>
>>> /niklas
>> Hi Niklas,
>>
>> I totally agree with that. One thing to consider though : using port 
>> 0 to get Tomcat to pick a random port might lead to many ports 
>> remaining opened, if the cleanup is not done correctly...
>
> I believe that the right cleanup is currently performed; there are 
> some known conditions though (for example     ) leading sometimes 
> Tomcat processes to hang: the only effective cleanup I know in such 
> cases is a good old "kill -9".
You certainly have ways to get rid of such exceptions 
(http://stackoverflow.com/questions/88235/how-to-deal-with-java-lang-outofmemoryerror-permgen-space-error)
>
> Because of this, I would think that fixed port numbers will help 
> avoiding major problems - as reported by Emmanuel above.
> If you still think it would be better to have random port numbers, we 
> could explore build-helper-maven-plugin:reserve-network-port in order 
> to get all needed network ports for running integration tests; as 
> reported above, in fact, it's not just Tomcat.
Those ports problem is certainly not a Tomcat specific issue, sadly. At 
MINA, until we discovered that we can open a port 0 (and let the system 
select an available port), we had a dirty hack that was trying to do the 
same. But MINA is certainly a different case.

Note that this is not really a big problem. We are discussing about a 
potential solution to an annoyance (Jenkins mails), so no need to rush 
and change the whole build right now :)

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Francesco Chicchiriccò <il...@apache.org>.
On 02/05/2012 18:39, Emmanuel Lécharny wrote:
> Le 5/2/12 5:33 PM, Niklas Gustavsson a écrit :
>> On Wed, May 2, 2012 at 5:19 PM, Francesco Chicchiriccò
>> <il...@apache.org>  wrote:
>>> we are experiencing repeated failures on Windows probably due to few 
>>> zombie
>>> processes belonging to some former build.
>>>
>>> As you can see from [1], in fact, ports 8082 and 9080 are already in 
>>> use.
>>>
>>> I personally don't understand how this could happen: every build, 
>>> even with
>>> failures should always exit cleanly, but the situation needs some 
>>> manual
>>> intervention in order to kill such zombie processes.
>>>
>>> Thanks for your support.
>>> Regards.
>>>
>>> [1]
>>> https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log 
>>>
>> Should be fixed now. But, I would argue that it's never a good idea to
>> run tests on fixed ports. You're running in a shared environment (or
>> on some random developer's box) where you can never know that these
>> ports are available. Instead, if at all possible, let the OS pick a
>> free port for you.
>>
>> /niklas
> Hi Niklas,
>
> I totally agree with that. One thing to consider though : using port 0 
> to get Tomcat to pick a random port might lead to many ports remaining 
> opened, if the cleanup is not done correctly...

I believe that the right cleanup is currently performed; there are some 
known conditions though (for example OutOfMemoryError: PermGen space) 
leading sometimes Tomcat processes to hang: the only effective cleanup I 
know in such cases is a good old "kill -9".

Because of this, I would think that fixed port numbers will help 
avoiding major problems - as reported by Emmanuel above.
If you still think it would be better to have random port numbers, we 
could explore build-helper-maven-plugin:reserve-network-port in order to 
get all needed network ports for running integration tests; as reported 
above, in fact, it's not just Tomcat.

Regards.

-- 
Francesco Chicchiriccò

Apache Cocoon Committer and PMC Member
http://people.apache.org/~ilgrosso/

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 8:46 PM, Niklas Gustavsson a écrit :
> On Wed, May 2, 2012 at 6:39 PM, Emmanuel Lécharny<el...@gmail.com>  wrote:
>> I totally agree with that. One thing to consider though : using port 0 to
>> get Tomcat to pick a random port might lead to many ports remaining opened,
>> if the cleanup is not done correctly...
> We got plenty of ports :) Also, we usually don't see a lot of zombie
> jobs so I'm not that worried.
No, me neither :) As I said in another reply, I'm just trying to get rid 
of those annoying Jenkins mails ;)


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Wed, May 2, 2012 at 6:39 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
> I totally agree with that. One thing to consider though : using port 0 to
> get Tomcat to pick a random port might lead to many ports remaining opened,
> if the cleanup is not done correctly...

We got plenty of ports :) Also, we usually don't see a lot of zombie
jobs so I'm not that worried.

/niklas

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 5:33 PM, Niklas Gustavsson a écrit :
> On Wed, May 2, 2012 at 5:19 PM, Francesco Chicchiriccò
> <il...@apache.org>  wrote:
>> we are experiencing repeated failures on Windows probably due to few zombie
>> processes belonging to some former build.
>>
>> As you can see from [1], in fact, ports 8082 and 9080 are already in use.
>>
>> I personally don't understand how this could happen: every build, even with
>> failures should always exit cleanly, but the situation needs some manual
>> intervention in order to kill such zombie processes.
>>
>> Thanks for your support.
>> Regards.
>>
>> [1]
>> https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log
> Should be fixed now. But, I would argue that it's never a good idea to
> run tests on fixed ports. You're running in a shared environment (or
> on some random developer's box) where you can never know that these
> ports are available. Instead, if at all possible, let the OS pick a
> free port for you.
>
> /niklas
Hi Niklas,

I totally agree with that. One thing to consider though : using port 0 
to get Tomcat to pick a random port might lead to many ports remaining 
opened, if the cleanup is not done correctly...

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Wed, May 2, 2012 at 5:19 PM, Francesco Chicchiriccò
<il...@apache.org> wrote:
> we are experiencing repeated failures on Windows probably due to few zombie
> processes belonging to some former build.
>
> As you can see from [1], in fact, ports 8082 and 9080 are already in use.
>
> I personally don't understand how this could happen: every build, even with
> failures should always exit cleanly, but the situation needs some manual
> intervention in order to kill such zombie processes.
>
> Thanks for your support.
> Regards.
>
> [1]
> https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log

Should be fixed now. But, I would argue that it's never a good idea to
run tests on fixed ports. You're running in a shared environment (or
on some random developer's box) where you can never know that these
ports are available. Instead, if at all possible, let the OS pick a
free port for you.

/niklas

Re: Errors in https://builds.apache.org/job/Syncope-windows/

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Wed, May 2, 2012 at 5:19 PM, Francesco Chicchiriccò
<il...@apache.org> wrote:
> we are experiencing repeated failures on Windows probably due to few zombie
> processes belonging to some former build.
>
> As you can see from [1], in fact, ports 8082 and 9080 are already in use.
>
> I personally don't understand how this could happen: every build, even with
> failures should always exit cleanly, but the situation needs some manual
> intervention in order to kill such zombie processes.
>
> Thanks for your support.
> Regards.
>
> [1]
> https://builds.apache.org/job/Syncope-windows/ws/core/target/log/cargo-output.log

Should be fixed now. But, I would argue that it's never a good idea to
run tests on fixed ports. You're running in a shared environment (or
on some random developer's box) where you can never know that these
ports are available. Instead, if at all possible, let the OS pick a
free port for you.

/niklas