You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Kevin Ratnasekera <dj...@gmail.com> on 2017/02/10 07:28:07 UTC
Thread spinning at ChunkEncoder.write method
Hi Dev,
We are recently experiencing a thread spinning issue related to HTTP Core
dependency when migrating from HTTP Core version 4.2.3 to 4.3.3. Is there
any known issue reported related to this? Please find the related partial
thread dump taken at thread spinning. [1]
[1]
"HTTPS-Listener I/O dispatcher-35" #595 prio=5 os_prio=0
tid=0x00007f39904f4000 nid=0xaaee runnable [0x00007f37649c8000]
java.lang.Thread.State: RUNNABLE
at org.apache.http.impl.nio.codecs.ChunkEncoder.write(ChunkEncoder.java:134)
at org.apache.synapse.transport.passthru.Pipe.consume(Pipe.java:135)
at
org.apache.synapse.transport.passthru.SourceResponse.write(SourceResponse.java:248)
at
org.apache.synapse.transport.passthru.SourceHandler.outputReady(SourceHandler.java:260)
at
org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(DefaultNHttpServerConnection.java:312)
at
org.apache.synapse.transport.http.conn.LoggingNHttpServerConnection.produceOutput(LoggingNHttpServerConnection.java:125)
at
org.apache.synapse.transport.passthru.ServerIODispatch.onOutputReady(ServerIODispatch.java:87)
at
org.apache.synapse.transport.passthru.ServerIODispatch.onOutputReady(ServerIODispatch.java:39)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:180)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006e4180218> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
Regards
Kevin
Re: Thread spinning at ChunkEncoder.write method
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2017-03-19 at 14:15 +0530, Shafreen Anfar wrote:
> Hi Dev,
>
> After replacing the while loop in the ChunkEncorder [1] with a if
> statement, we were able to solve the thread spinning issue. However,
> we
> couldn't figure out the root cause that makes the while loop spin
> forever.
> While doing a code inspection, we could see the below code segment.
>
> if (avail > 0) {
> ����if (avail < chunk) {
> ��������// write no more than 'avail' bytes
> ��������chunk = avail;
> ��������this.lineBuffer.clear();
> ��������this.lineBuffer.append(Integer.toHexString(chunk));
> ��������this.buffer.writeLine(this.lineBuffer);
> ��������final int oldlimit = src.limit();
> ��������src.limit(src.position() + chunk);
> ��������this.buffer.write(src);
> ��������src.limit(oldlimit);
> ����} else {
> ��������// write all
> ��������this.lineBuffer.clear();
> ��������this.lineBuffer.append(Integer.toHexString(chunk));
> ��������this.buffer.writeLine(this.lineBuffer);
> ��������this.buffer.write(src);
> ����}
> ����this.lineBuffer.clear();
> ����this.buffer.writeLine(this.lineBuffer);
> ����total += chunk;
> }
> if (this.buffer.length() >= this.fragHint || src.hasRemaining()) {
> ����final int bytesWritten = flushToChannel();
> ����if (bytesWritten == 0) {
> ��������break;
> ����}
> }
>
> In the above code segment if *avail* is less that 0, we execute the
> below
> code segment.
>
> if (this.buffer.length() >= this.fragHint || src.hasRemaining()) {
> ����final int bytesWritten = flushToChannel();
> ����if (bytesWritten == 0) {
> ��������break;
> ����}
> }
>
> Now, suppose *src.hasRemainning() *returns true, then it tries to
> flush the
> data to the channel.
>
> But, Do we need to do that ?
>
Yes, we do.
> The reason being if *avail *is less than 0, that means buffer doesn't
> have
> new data to write to the channel. But above code segment still tries
> to
> write data even if *avail *is less than 0.
>
The avail value defines available capacity of the buffer.
ChunkEncorder flushes the content of the buffer to the underlying
session precisely to free up space in the buffer.
Oleg
> [1]
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.3.3/h
> ttpcore-
> nio/src/main/java/org/apache/http/impl/nio/codecs/ChunkEncoder.java
>
>
> On Fri, Feb 10, 2017 at 3:44 PM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
>
> > On Fri, 2017-02-10 at 12:58 +0530, Kevin Ratnasekera wrote:
> > > Hi Dev,
> > >
> > > We are recently experiencing a thread spinning issue related to
> > > HTTP
> > > Core
> > > dependency when migrating from HTTP Core version��4.2.3 to 4.3.3.
> > > Is
> > > there
> > > any known issue reported related to this? Please find the related
> > > partial
> > > thread dump taken at thread spinning. [1]
> > >
> >
> > HttpCore 4.3.3 release is almost 2.5 year old. 6 more releases have
> > been made since then with a number of bug fixes.
> >
> > Oleg
> >
> >
> > > [1]
> > > "HTTPS-Listener I/O dispatcher-35" #595 prio=5 os_prio=0
> > > tid=0x00007f39904f4000 nid=0xaaee runnable [0x00007f37649c8000]
> > > ���java.lang.Thread.State: RUNNABLE
> > > at
> > > org.apache.http.impl.nio.codecs.ChunkEncoder.write(ChunkEncoder.j
> > > ava:
> > > 134)
> > > at
> > > org.apache.synapse.transport.passthru.Pipe.consume(Pipe.java:135)
> > > at
> > > org.apache.synapse.transport.passthru.SourceResponse.write(Source
> > > Resp
> > > onse.java:248)
> > > at
> > > org.apache.synapse.transport.passthru.SourceHandler.outputReady(S
> > > ourc
> > > eHandler.java:260)
> > > at
> > > org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutp
> > > ut(D
> > > efaultNHttpServerConnection.java:312)
> > > at
> > > org.apache.synapse.transport.http.conn.LoggingNHttpServerConnecti
> > > on.p
> > > roduceOutput(LoggingNHttpServerConnection.java:125)
> > > at
> > > org.apache.synapse.transport.passthru.ServerIODispatch.onOutputRe
> > > ady(
> > > ServerIODispatch.java:87)
> > > at
> > > org.apache.synapse.transport.passthru.ServerIODispatch.onOutputRe
> > > ady(
> > > ServerIODispatch.java:39)
> > > at
> > > org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(A
> > > bstr
> > > actIODispatch.java:150)
> > > at
> > > org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIORea
> > > ctor
> > > .java:180)
> > > at
> > > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(A
> > > bstr
> > > actIOReactor.java:342)
> > > at
> > > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(
> > > Abst
> > > ractIOReactor.java:316)
> > > at
> > > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(Abstra
> > > ctIO
> > > Reactor.java:277)
> > > at
> > > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReac
> > > tor.
> > > java:105)
> > > at
> > > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Wor
> > > ker.
> > > run(AbstractMultiworkerIOReactor.java:586)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > ���Locked ownable synchronizers:
> > > - <0x00000006e4180218> (a
> > > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > >
> > > Regards
> > > Kevin
> >
> > -----------------------------------------------------------------
> > ----
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Thread spinning at ChunkEncoder.write method
Posted by Shafreen Anfar <sh...@wso2.com>.
Hi Dev,
After replacing the while loop in the ChunkEncorder [1] with a if
statement, we were able to solve the thread spinning issue. However, we
couldn't figure out the root cause that makes the while loop spin forever.
While doing a code inspection, we could see the below code segment.
if (avail > 0) {
if (avail < chunk) {
// write no more than 'avail' bytes
chunk = avail;
this.lineBuffer.clear();
this.lineBuffer.append(Integer.toHexString(chunk));
this.buffer.writeLine(this.lineBuffer);
final int oldlimit = src.limit();
src.limit(src.position() + chunk);
this.buffer.write(src);
src.limit(oldlimit);
} else {
// write all
this.lineBuffer.clear();
this.lineBuffer.append(Integer.toHexString(chunk));
this.buffer.writeLine(this.lineBuffer);
this.buffer.write(src);
}
this.lineBuffer.clear();
this.buffer.writeLine(this.lineBuffer);
total += chunk;
}
if (this.buffer.length() >= this.fragHint || src.hasRemaining()) {
final int bytesWritten = flushToChannel();
if (bytesWritten == 0) {
break;
}
}
In the above code segment if *avail* is less that 0, we execute the below
code segment.
if (this.buffer.length() >= this.fragHint || src.hasRemaining()) {
final int bytesWritten = flushToChannel();
if (bytesWritten == 0) {
break;
}
}
Now, suppose *src.hasRemainning() *returns true, then it tries to flush the
data to the channel.
But, Do we need to do that ?
The reason being if *avail *is less than 0, that means buffer doesn't have
new data to write to the channel. But above code segment still tries to
write data even if *avail *is less than 0.
[1]
https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.3.3/httpcore-nio/src/main/java/org/apache/http/impl/nio/codecs/ChunkEncoder.java
On Fri, Feb 10, 2017 at 3:44 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Fri, 2017-02-10 at 12:58 +0530, Kevin Ratnasekera wrote:
> > Hi Dev,
> >
> > We are recently experiencing a thread spinning issue related to HTTP
> > Core
> > dependency when migrating from HTTP Core version 4.2.3 to 4.3.3. Is
> > there
> > any known issue reported related to this? Please find the related
> > partial
> > thread dump taken at thread spinning. [1]
> >
>
> HttpCore 4.3.3 release is almost 2.5 year old. 6 more releases have
> been made since then with a number of bug fixes.
>
> Oleg
>
>
> > [1]
> > "HTTPS-Listener I/O dispatcher-35" #595 prio=5 os_prio=0
> > tid=0x00007f39904f4000 nid=0xaaee runnable [0x00007f37649c8000]
> > java.lang.Thread.State: RUNNABLE
> > at
> > org.apache.http.impl.nio.codecs.ChunkEncoder.write(ChunkEncoder.java:
> > 134)
> > at org.apache.synapse.transport.passthru.Pipe.consume(Pipe.java:135)
> > at
> > org.apache.synapse.transport.passthru.SourceResponse.write(SourceResp
> > onse.java:248)
> > at
> > org.apache.synapse.transport.passthru.SourceHandler.outputReady(Sourc
> > eHandler.java:260)
> > at
> > org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(D
> > efaultNHttpServerConnection.java:312)
> > at
> > org.apache.synapse.transport.http.conn.LoggingNHttpServerConnection.p
> > roduceOutput(LoggingNHttpServerConnection.java:125)
> > at
> > org.apache.synapse.transport.passthru.ServerIODispatch.onOutputReady(
> > ServerIODispatch.java:87)
> > at
> > org.apache.synapse.transport.passthru.ServerIODispatch.onOutputReady(
> > ServerIODispatch.java:39)
> > at
> > org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(Abstr
> > actIODispatch.java:150)
> > at
> > org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor
> > .java:180)
> > at
> > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(Abstr
> > actIOReactor.java:342)
> > at
> > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(Abst
> > ractIOReactor.java:316)
> > at
> > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIO
> > Reactor.java:277)
> > at
> > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.
> > java:105)
> > at
> > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.
> > run(AbstractMultiworkerIOReactor.java:586)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Locked ownable synchronizers:
> > - <0x00000006e4180218> (a
> > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> >
> > Regards
> > Kevin
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Regards,
*Shafreen*
Software Engineer
WSO2 Inc
Mobile : 077-556-395-1
Re: Thread spinning at ChunkEncoder.write method
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2017-02-10 at 12:58 +0530, Kevin Ratnasekera wrote:
> Hi Dev,
>
> We are recently experiencing a thread spinning issue related to HTTP
> Core
> dependency when migrating from HTTP Core version��4.2.3 to 4.3.3. Is
> there
> any known issue reported related to this? Please find the related
> partial
> thread dump taken at thread spinning. [1]
>
HttpCore 4.3.3 release is almost 2.5 year old. 6 more releases have
been made since then with a number of bug fixes.
Oleg
> [1]
> "HTTPS-Listener I/O dispatcher-35" #595 prio=5 os_prio=0
> tid=0x00007f39904f4000 nid=0xaaee runnable [0x00007f37649c8000]
> ���java.lang.Thread.State: RUNNABLE
> at
> org.apache.http.impl.nio.codecs.ChunkEncoder.write(ChunkEncoder.java:
> 134)
> at org.apache.synapse.transport.passthru.Pipe.consume(Pipe.java:135)
> at
> org.apache.synapse.transport.passthru.SourceResponse.write(SourceResp
> onse.java:248)
> at
> org.apache.synapse.transport.passthru.SourceHandler.outputReady(Sourc
> eHandler.java:260)
> at
> org.apache.http.impl.nio.DefaultNHttpServerConnection.produceOutput(D
> efaultNHttpServerConnection.java:312)
> at
> org.apache.synapse.transport.http.conn.LoggingNHttpServerConnection.p
> roduceOutput(LoggingNHttpServerConnection.java:125)
> at
> org.apache.synapse.transport.passthru.ServerIODispatch.onOutputReady(
> ServerIODispatch.java:87)
> at
> org.apache.synapse.transport.passthru.ServerIODispatch.onOutputReady(
> ServerIODispatch.java:39)
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(Abstr
> actIODispatch.java:150)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor
> .java:180)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(Abstr
> actIOReactor.java:342)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(Abst
> ractIOReactor.java:316)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIO
> Reactor.java:277)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.
> java:105)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.
> run(AbstractMultiworkerIOReactor.java:586)
> at java.lang.Thread.run(Thread.java:745)
>
> ���Locked ownable synchronizers:
> - <0x00000006e4180218> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> Regards
> Kevin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org