You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2014/09/19 02:21:59 UTC
[Bug 56995] New: Errors from servlet writer flush() causing threads
in BLOCK state
https://issues.apache.org/bugzilla/show_bug.cgi?id=56995
Bug ID: 56995
Summary: Errors from servlet writer flush() causing threads in
BLOCK state
Product: Tomcat 7
Version: 7.0.55
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: dev@tomcat.apache.org
Reporter: charlesk40@yahoo.com
We are using tomcat 7.0.55 with bio connector.
We are seeing threads go into bad BLOCKED state when client's throw IOException
when trying to flush and the process tries to call setErrorState which then
calls processSocketAsync. But processSocketAsync waits to hold a lock on the
socket which previously held by the processor.
Below is part of the stacktrace where it shows this.
"http-bio-0.0.0.0-8042-exec-35" daemon prio=10 tid=0x0000000003647000
nid=0x3605 waiting on condition [0x00007f0e02d8c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000073d3d20b0> (a
com.google.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at
com.yahoo.yql.plus.writer.ResultEnvelope.processResults(ResultEnvelope.java:140)
at
com.yahoo.yql.plus.writer.ResultEnvelope.write(ResultEnvelope.java:64)
at
com.yahoo.yql.plus.writer.ResultEnvelope.write(ResultEnvelope.java:57)
at com.yahoo.yql.plus.ProgramEngine.write(ProgramEngine.java:247)
at com.yahoo.yql.plus.ProgramEngine.execute(ProgramEngine.java:221)
at com.yahoo.yql.plus.ProgramEngine.execute(ProgramEngine.java:263)
at
com.yahoo.yql.core.Container.executeProgramRequest(Container.java:340)
at
com.yahoo.yql.container.servlet.filter.DispatchFilter.doFilter(DispatchFilter.java:154)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:69)
at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:53)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:135)
at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:111)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
at yjava.servlet.filter.SSLCrimeFilter.doFilter(SSLCrimeFilter.java:82)
at yjava.servlet.filter.SSLCrimeFilter.doFilter(SSLCrimeFilter.java:47)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
at
yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:110)
at
yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:50)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:104)
at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:65)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
at
yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:244)
at
yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:151)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
yjava.servlet.FilterChainInvoker$ServletFilterChainInvoker.invoke(FilterChainInvoker.java:49)
at yjava.servlet.filter.StatsFilter.doFilter(StatsFilter.java:90)
at yjava.servlet.filter.StatsFilter.doFilter(StatsFilter.java:65)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
com.yahoo.yjava.AccessLogValveCopy.invoke(AccessLogValveCopy.java:580)
at
yjava.tomcat.valves.YahooConnectionValve.invoke(YahooConnectionValve.java:141)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
- locked <0x000000073cff7db0> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:724)
This is a thread that throws IOException and waiting to get hold of lock on the
socket which is privioulsy held by above http-bio thread.
"work-7907" daemon prio=10 tid=0x0000000004b26800 nid=0x3a16 waiting for
monitor entry [0x00007f0d8116b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.tomcat.util.net.JIoEndpoint.processSocketAsync(JIoEndpoint.java:560)
- waiting to lock <0x000000073cff7db0> (a
org.apache.tomcat.util.net.SocketWrapper)
at
org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:84)
at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:802)
at org.apache.coyote.Response.action(Response.java:172)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:363)
at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331)
at
org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
at java.io.PrintStream.flush(PrintStream.java:338)
- locked <0x000000073cff7e10> (a java.io.PrintStream)
at
com.yahoo.media.yql.lib.listener.ProtoSerializer.writeMessagePart(ProtoSerializer.java:77)
at
com.yahoo.media.yql.lib.listener.ProtoSerializer.onResultSuccess(ProtoSerializer.java:59)
at
com.yahoo.yql.plus.writer.ResultEnvelope$YQLResultCallback.onSuccess(ResultEnvelope.java:192)
- locked <0x000000073cff7e50> (a
com.yahoo.yql.plus.ProgramResponseImpl)
at
com.yahoo.yql.plus.writer.ResultEnvelope$YQLResultCallback.onSuccess(ResultEnvelope.java:146)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1169)
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
at
com.google.common.util.concurrent.Futures$ChainingListenableFuture$1.run(Futures.java:860)
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at
com.google.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:100)
at
com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:856)
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at
com.yahoo.yqlplus.engine.internal.scope.ExecutionScoper$1.execute(ExecutionScoper.java:99)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
at
com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
at
com.yahoo.yqlplus.engine.internal.java.runtime.TimeoutHandler$3.onSuccess(TimeoutHandler.java:100)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1169)
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
at
com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
at
com.yahoo.yqlplus.engine.internal.programs.Program62219060_b47e_42cb_b726_b87eaabec462.task_109(Program62219060_b47e_42cb_b726_b87eaabec462.java)
at
java.lang.invoke.LambdaForm$DMH/33003170.invokeSpecial_L_V(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/1542947881.reinvoke(LambdaForm$BMH)
at
java.lang.invoke.LambdaForm$MH/1589590023.exactInvoker(LambdaForm$MH)
at
java.lang.invoke.LambdaForm$MH/1744671065.exactInvoker(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/1579626491.collect(LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/284451544.invoke_MT(LambdaForm$MH)
at
com.yahoo.yqlplus.engine.internal.java.functions.MethodHandleRunnable.run(MethodHandleRunnable.java:19)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
com.yahoo.yqlplus.engine.internal.scope.ExecutionScoper$2.run(ExecutionScoper.java:113)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 56995] Errors from servlet writer flush() causing threads in
BLOCK state
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56995
--- Comment #2 from charlie <ch...@yahoo.com> ---
Please let me know if you need any other information.
Thank you.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 56995] Errors from servlet writer flush() causing threads in
BLOCK state
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56995
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |INVALID
--- Comment #3 from Mark Thomas <ma...@apache.org> ---
You have two threads trying to write to the socket at the same time. That is
not permitted.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 56995] Errors from servlet writer flush() causing threads in
BLOCK state
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56995
--- Comment #1 from charlie <ch...@yahoo.com> ---
locks the socket here.
http://svn.apache.org/repos/asf/tomcat/tc7.0.x/tags/TOMCAT_7_0_55/java/org/apache/tomcat/util/net/JIoEndpoint.java
@Override
public void run() {
boolean launch = false;
synchronized (socket) {
try {
SocketState state = SocketState.OPEN;
client tries to flush but throws IOException
http://svn.apache.org/repos/asf/tomcat/tc7.0.x/tags/TOMCAT_7_0_55/java/org/apache/coyote/http11/AbstractHttp11Processor.java
case CLIENT_FLUSH: {
try {
getOutputBuffer().flush();
} catch (IOException e) {
setErrorState(ErrorState.CLOSE_NOW, e);
response.setErrorException(e);
}
break;
}
Calls setErrorState and which then calls processSocketAsync
/**
* Update the current error state to the new error state if the new error
* state is more severe than the current error state.
*/
protected void setErrorState(ErrorState errorState, Throwable t) {
boolean blockIo = this.errorState.isIoAllowed() &&
!errorState.isIoAllowed();
this.errorState = this.errorState.getMostSevere(errorState);
if (blockIo && !ContainerThreadMarker.isContainerThread()) {
// The error occurred on a non-container thread which means not all
// of the necessary clean-up will have been completed. Dispatch to
// a container thread to do the clean-up. Need to do it this way to
// ensure that all the necessary clean-up is performed.
if (response.getStatus() < 400) {
response.setStatus(500);
}
getLog().info(sm.getString("abstractProcessor.nonContainerThreadError"), t);
getEndpoint().processSocketAsync(socketWrapper,
SocketStatus.CLOSE_NOW);
}
}
Tries to hold the same lock here causing threads to be in bad BLOCKED states
which eventually make all available threads to hang.
http://svn.apache.org/repos/asf/tomcat/tc7.0.x/tags/TOMCAT_7_0_55/java/org/apache/tomcat/util/net/JIoEndpoint.java
@Override
public void processSocketAsync(SocketWrapper<Socket> socket,
SocketStatus status) {
try {
synchronized (socket) {
if (waitingRequests.remove(socket)) {
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 56995] Errors from servlet writer flush() causing threads in
BLOCK state
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56995
--- Comment #4 from charlie <ch...@yahoo.com> ---
Okay. I'm trying to understand this better. So is the synchronization on the
socket only implemented for handling exceptions?
Application should not use separate thread to do any of response write/flush
using BIO connector? Tomcat doesn't seem to prevent that and it works when
there are no exceptions.
Thanks.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org