You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by James Law <JL...@generic-software.com> on 2008/05/22 18:55:31 UTC
Broken pipe errors
Can anyone help with the below error? We have just changed from a FC8
box with the default install (Test Box) to a FC8 box with a more
stripped down install (both the same machines)
With the stripped down install we now get the following error, the only
difference's between the two was the test box was setup to use the AJP
connector attached to a Apache server. And the slimmed down is stand
alone, both using Sun's JDK6 with Tomcat 6.0.16.
Thanks!
org.apache.cxf.interceptor.Fault: Could not send Message.
at
org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingI
nterceptor.handleMessage(MessageSenderInterceptor.java:64)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorC
hain.java:208)
at
org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage
(AbstractFaultChainInitiatorObserver.java:96)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorC
hain.java:242)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiati
onObserver.java:77)
at
org.apache.cxf.transport.servlet.ServletDestination.doMessage(ServletDes
tination.java:79)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(Ser
vletController.java:264)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletControl
ler.java:160)
at
org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFSe
rvlet.java:170)
at
org.apache.cxf.transport.servlet.AbstractCXFServlet.doPost(AbstractCXFSe
rvlet.java:148)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2
86)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84
4)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(
Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Caused by: ClientAbortException: java.net.SocketException: Broken pipe
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:319
)
at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288)
at
org.apache.catalina.connector.Response.flushBuffer(Response.java:548)
at
org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.
java:279)
at
org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStrea
m.close(AbstractHTTPDestination.java:427)
at
org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
at
org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingI
nterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 23 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOut
putBuffer.java:737)
at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at
org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer
.java:299)
at
org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:963
)
at org.apache.coyote.Response.action(Response.java:183)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:314
)
... 29 more
May 22, 2008 12:30:37 AM org.apache.cxf.phase.PhaseInterceptorChain
doIntercept
****************************************************************************
Search all of our current vacancies at www.generic-software.com
****************************************************************************
The information contained within this message is intended for the addressee
only and may contain confidential and/or privileged information. If you are
not the addressee, please delete this message and notify the sender - you
should not copy, distribute or disclose its contents to other parties.
Any images, documents, views or opinions expressed in this message are
those of the sender and do not necessarily represent those of Generic
Software Consultants Ltd or any of its affiliates. No reliance may be
placed on this message without written confirmation from an authorised
company representative, and no actions should be taken based on its
contents.
Generic Software Consultants Ltd Registered in England No. 2830109 @
St. Andrews House, Caldecotte Lake Drive, Caldecotte Business Park,
Milton Keynes. MK7 8LE
Tel: 01908 278450
VAT Registered No: 608 6625 28
****************************************************************************
RE: Broken pipe errors
Posted by James Law <JL...@generic-software.com>.
> That's good. Many of the reports I've seen on this list have been people
> moving up to Tomcat 6 and experiencing /more/ memory problems --
> indicating that TC6 uses a bit more memory. I'm glad you are having the
> opposite experience.
I haven't used 5.x but 6.0.14 does seem (I'm still testing) more stable than 6.0.16, though that’s probably not a good thing for the progress of Tomcat.
> Ouch. The default session timeout is 30 minutes, so you could
> potentially have 1/2 MiB of data /per session/ laying around for 30
> minutes or so. This could certainly bust your heap.
> That depends on a few factors...
> Is there any reason for a session at all? Maybe if you explain a little
> more about what the service does, we could help you optimize it a bit.
> Do you handle all the processing during the request, or do you store the
> document for batch processing and immediately return a response to the
> client?
I'd love to be able to tell you more about it, all I really know is it’s a GATE application that's dealing with semantics, this is the problem we have, no one has ever used Gate to do what were doing and subsequently this service is a first for Tomcat so it’s a big case of trial and error at the moment and just to throw more spanners the gate developers are using JRocket.
> If you do everything during the request, I suspect that sessions are not
> necessary at all. Eliminating sessions entirely can improve the
> performance of your application and significantly improve your memory
> footprint.
All the processing is done in the request and returned straight away, so your right :) Sessions are probably not necessary so that’s another line of testing.
Thanks
****************************************************************************
Search all of our current vacancies at www.generic-software.com
****************************************************************************
The information contained within this message is intended for the addressee
only and may contain confidential and/or privileged information. If you are
not the addressee, please delete this message and notify the sender - you
should not copy, distribute or disclose its contents to other parties.
Any images, documents, views or opinions expressed in this message are
those of the sender and do not necessarily represent those of Generic
Software Consultants Ltd or any of its affiliates. No reliance may be
placed on this message without written confirmation from an authorised
company representative, and no actions should be taken based on its
contents.
Generic Software Consultants Ltd Registered in England No. 2830109 @
St. Andrews House, Caldecotte Lake Drive, Caldecotte Business Park,
Milton Keynes. MK7 8LE
Tel: 01908 278450
VAT Registered No: 608 6625 28
****************************************************************************
Re: Broken pipe errors
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
James,
James Law wrote:
| Thanks Andrew & Chris Currently testing in 6.0.14 and the memory
| usage seems a lot more stable (probably a placebo affect). In 6.0.16
| both the young/old generation heap usage would increase to 99.xx% and
| stay there, now both the eden and old heaps are increasing and
| decreasing.
That's good. Many of the reports I've seen on this list have been people
moving up to Tomcat 6 and experiencing /more/ memory problems --
indicating that TC6 uses a bit more memory. I'm glad you are having the
opposite experience.
| The sessions would contain a document of anything from 50k to 500k
| with the default timeout.
Ouch. The default session timeout is 30 minutes, so you could
potentially have 1/2 MiB of data /per session/ laying around for 30
minutes or so. This could certainly bust your heap.
| Would it be worth reducing the session timeout?
That depends on a few factors...
| Our app just annotates and returns documents that another
| server has sent to it. Which rarely takes more than a few seconds
| each and no more than 2 at a time.
Is there any reason for a session at all? Maybe if you explain a little
more about what the service does, we could help you optimize it a bit.
Do you handle all the processing during the request, or do you store the
document for batch processing and immediately return a response to the
client?
If you do everything during the request, I suspect that sessions are not
necessary at all. Eliminating sessions entirely can improve the
performance of your application and significantly improve your memory
footprint.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkg+xOEACgkQ9CaO5/Lv0PCjtwCfa/4RamyXFlGyjWg3yQsULNwJ
oHQAn3hkQPy+YGx6kTJt909Ay6ohzlve
=iraR
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Broken pipe errors
Posted by James Law <JL...@generic-software.com>.
Thanks Andrew & Chris
Currently testing in 6.0.14 and the memory usage seems a lot more stable (probably a placebo affect). In 6.0.16 both the young/old generation heap usage would increase to 99.xx% and stay there, now both the eden and old heaps are increasing and decreasing.
Though the java console lost its connection and refuses to reconnect, but Tomcat is still serving requests and jmap returns values. I'll put that down to a bug in either jconsole or running it on the local machine (never a good idea I know!).
> Do you use sessions? How long is your session timeout? How much stuff do you put into the session? How about any other caches in your application? >ThreadLocals?
>I doubt that the BrokenPipeException is the cause of the memory problems, but it's possible that it's a symptom of a server that is struggling to serve >requests with low memory. I've seen JVMs thrash a bit when under low memory conditions, but they usually suffer from OOMEs shortly after that.
> -chris
The sessions would contain a document of anything from 50k to 500k with the default timeout.
Would it be worth reducing the session timeout? Our app just annotates and returns documents that another server has sent to it. Which rarely takes more than a few seconds each and no more than 2 at a time.
Cheers
James
-----Original Message-----
From: savs@andrewsavory.com [mailto:savs@andrewsavory.com] On Behalf Of Andrew Savory
Sent: 28 May 2008 10:56
To: Tomcat Users List
Subject: Re: Broken pipe errors
Hi James,
2008/5/22 James Law <JL...@generic-software.com>:
> Can anyone help with the below error? We have just changed from a FC8
> box with the default install (Test Box) to a FC8 box with a more
> stripped down install (both the same machines)
>
> With the stripped down install we now get the following error, the only
> difference's between the two was the test box was setup to use the AJP
> connector attached to a Apache server. And the slimmed down is stand
> alone, both using Sun's JDK6 with Tomcat 6.0.16.
Try using Tomcat 6.0.14 and see if the problem persists. If you're
doing POSTs you may be falling foul of the following issue:
https://issues.apache.org/bugzilla/show_bug.cgi?id=44494
Andrew.
--
asavory@apache.org / contact@andrewsavory.com
http://www.andrewsavory.com/
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
****************************************************************************
Search all of our current vacancies at www.generic-software.com
****************************************************************************
The information contained within this message is intended for the addressee
only and may contain confidential and/or privileged information. If you are
not the addressee, please delete this message and notify the sender - you
should not copy, distribute or disclose its contents to other parties.
Any images, documents, views or opinions expressed in this message are
those of the sender and do not necessarily represent those of Generic
Software Consultants Ltd or any of its affiliates. No reliance may be
placed on this message without written confirmation from an authorised
company representative, and no actions should be taken based on its
contents.
Generic Software Consultants Ltd Registered in England No. 2830109 @
St. Andrews House, Caldecotte Lake Drive, Caldecotte Business Park,
Milton Keynes. MK7 8LE
Tel: 01908 278450
VAT Registered No: 608 6625 28
****************************************************************************
Re: Broken pipe errors
Posted by Andrew Savory <as...@apache.org>.
Hi James,
2008/5/22 James Law <JL...@generic-software.com>:
> Can anyone help with the below error? We have just changed from a FC8
> box with the default install (Test Box) to a FC8 box with a more
> stripped down install (both the same machines)
>
> With the stripped down install we now get the following error, the only
> difference's between the two was the test box was setup to use the AJP
> connector attached to a Apache server. And the slimmed down is stand
> alone, both using Sun's JDK6 with Tomcat 6.0.16.
Try using Tomcat 6.0.14 and see if the problem persists. If you're
doing POSTs you may be falling foul of the following issue:
https://issues.apache.org/bugzilla/show_bug.cgi?id=44494
Andrew.
--
asavory@apache.org / contact@andrewsavory.com
http://www.andrewsavory.com/
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Broken pipe errors
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
James,
James Law wrote:
| The Java VM is sat with 94-95% of the
| available RAM (out of 8gb) and the Swap grew to 64% (out of 2gb) over
| the time tomcat was running.
400MB should be plenty for your OS to run.
| Forcing a GC has none/very little affect
| on memory usage. I've got the inkling the issue maybe caused by the
| JVM growing to big? The xmx is set to 7808m which should leave plenty
| of head room?
That depends on the memory requirements of your web app. When your JVM
starts acting like this, try to use jmap or another tool to take a look
at the heap to see what's taking up so much room.
| it only uses 50mb RAM 0mb Swap without tomcat running.
| There is only one client that sends 3 documents at any one time to
| the service that then annotates it and sends it back Anyone got any
| ideas? Im coming to the end of my tether here? Thanks
Do you use sessions? How long is your session timeout? How much stuff do
you put into the session? How about any other caches in your
application? ThreadLocals?
I doubt that the BrokenPipeException is the cause of the memory
problems, but it's possible that it's a symptom of a server that is
struggling to serve requests with low memory. I've seen JVMs thrash a
bit when under low memory conditions, but they usually suffer from OOMEs
shortly after that.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkg20d8ACgkQ9CaO5/Lv0PAv6QCfeT3zlgcL4r+aBONuD//405mz
d4QAn254MoB/pVoePotUV+M18bv/85yy
=EwFa
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Broken pipe errors
Posted by James Law <JL...@generic-software.com>.
That’s what we thought but the firewall has been disabled along with SElinux to test.
Unfortunately it does stop working after a while, though I'm not to sure if it’s that particular error or even Tomcat itself (Hence me asking on here)
In fact really weirdly the service was working all last night fine, until I came in and checked the server, at which point the machine froze for a few seconds then connections started timing out. This happened yesterday but put it down to a coincidence.
The Java VM is sat with 94-95% of the available RAM (out of 8gb) and the Swap grew to 64% (out of 2gb) over the time tomcat was running.
Forcing a GC has none/very little affect on memory usage. I've got the inkling the issue maybe caused by the JVM growing to big? The xmx is set to 7808m which should leave plenty of head room? it only uses 50mb RAM 0mb Swap without tomcat running.
There is only one client that sends 3 documents at any one time to the service that then annotates it and sends it back
Anyone got any ideas? Im coming to the end of my tether here?
Thanks
-----Original Message-----
From: David Smith [mailto:dns4@cornell.edu]
Sent: 22 May 2008 19:47
To: Tomcat Users List
Subject: Re: Broken pipe errors
Broken pipe can also happen when faulty network hardware or sensitive
firewalls drop the connection.
--David
Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> James,
>
> James Law wrote:
> | Can anyone help with the below error? We have just changed from a FC8
> | box with the default install (Test Box) to a FC8 box with a more
> | stripped down install (both the same machines)
>
> Is anything actually not working? Broken pipe messages are occasionally
> generated when users hit STOP on their browsers before the entire
> response has been written to the output stream.
>
> AFAIK, unless you find that something is actually not working, I think
> you can safely ignore these messages.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkg1vQcACgkQ9CaO5/Lv0PCPSgCgpP2XWeyaVbbElgVJ4UUnbTA6
> uV4AoMQkfNkymmXex8XzElfGJkvjLXkp
> =L/dz
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
****************************************************************************
Search all of our current vacancies at www.generic-software.com
****************************************************************************
The information contained within this message is intended for the addressee
only and may contain confidential and/or privileged information. If you are
not the addressee, please delete this message and notify the sender - you
should not copy, distribute or disclose its contents to other parties.
Any images, documents, views or opinions expressed in this message are
those of the sender and do not necessarily represent those of Generic
Software Consultants Ltd or any of its affiliates. No reliance may be
placed on this message without written confirmation from an authorised
company representative, and no actions should be taken based on its
contents.
Generic Software Consultants Ltd Registered in England No. 2830109 @
St. Andrews House, Caldecotte Lake Drive, Caldecotte Business Park,
Milton Keynes. MK7 8LE
Tel: 01908 278450
VAT Registered No: 608 6625 28
****************************************************************************
Re: Broken pipe errors
Posted by David Smith <dn...@cornell.edu>.
Broken pipe can also happen when faulty network hardware or sensitive
firewalls drop the connection.
--David
Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> James,
>
> James Law wrote:
> | Can anyone help with the below error? We have just changed from a FC8
> | box with the default install (Test Box) to a FC8 box with a more
> | stripped down install (both the same machines)
>
> Is anything actually not working? Broken pipe messages are occasionally
> generated when users hit STOP on their browsers before the entire
> response has been written to the output stream.
>
> AFAIK, unless you find that something is actually not working, I think
> you can safely ignore these messages.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkg1vQcACgkQ9CaO5/Lv0PCPSgCgpP2XWeyaVbbElgVJ4UUnbTA6
> uV4AoMQkfNkymmXex8XzElfGJkvjLXkp
> =L/dz
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Broken pipe errors
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
James,
James Law wrote:
| Can anyone help with the below error? We have just changed from a FC8
| box with the default install (Test Box) to a FC8 box with a more
| stripped down install (both the same machines)
Is anything actually not working? Broken pipe messages are occasionally
generated when users hit STOP on their browsers before the entire
response has been written to the output stream.
AFAIK, unless you find that something is actually not working, I think
you can safely ignore these messages.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkg1vQcACgkQ9CaO5/Lv0PCPSgCgpP2XWeyaVbbElgVJ4UUnbTA6
uV4AoMQkfNkymmXex8XzElfGJkvjLXkp
=L/dz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org