You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Daniel Andres Pelaez Lopez <es...@gmail.com> on 2023/07/06 19:09:01 UTC
Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Hi community,
I am working on a Spring Boot + Tomcat embedded application using
Virtual Threads (https://openjdk.org/jeps/425), and everything looks
great until we activate HTTP2. We started to see the following logs:
Thread[#72,ForkJoinPool-1-worker-9,5,CarrierThreads]
org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:86)
<== monitors:1
Thread[#70,ForkJoinPool-1-worker-7,5,CarrierThreads]
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1609)
<== monitors:1
Thread[#41,ForkJoinPool-1-worker-4,5,CarrierThreads]
org.apache.coyote.http2.Http2AsyncUpgradeHandler.writeHeaders(Http2AsyncUpgradeHandler.java:199)
<== monitors:1
org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:86)
<== monitors:1
Thread[#40,ForkJoinPool-1-worker-3,5,CarrierThreads]
org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1000)
<== monitors:1
org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940)
<== monitors:1
Thread[#61,ForkJoinPool-1-worker-6,5,CarrierThreads]
Thread[#72,ForkJoinPool-1-worker-9,5,CarrierThreads]
org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:86)
<== monitors:1
Thread[#70,ForkJoinPool-1-worker-7,5,CarrierThreads]
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1609)
<== monitors:1
Thread[#41,ForkJoinPool-1-worker-4,5,CarrierThreads]
org.apache.coyote.http2.Http2AsyncUpgradeHandler.writeHeaders(Http2AsyncUpgradeHandler.java:199)
<== monitors:1
org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:86)
<== monitors:1
Thread[#40,ForkJoinPool-1-worker-3,5,CarrierThreads]
org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1000)
<== monitors:1
org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940)
<== monitors:1
Thread[#61,ForkJoinPool-1-worker-6,5,CarrierThreads]
Inspecting the org.apache.coyote.http2.Stream class, we found several
synchronized keywords
(https://github.com/apache/tomcat/blob/main/java/org/apache/coyote/http2/Stream.java#L230).
From the JEP, this keyword will pin platform threads and might reduce
the general throughput of the application.
I am aware Tomcat community did a great effort to move Tomat to
Virtual Threads friendly, but I am not sure why HTTP2 was not part of
that effort?
--
Daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Posted by Mark Thomas <ma...@apache.org>.
On 28/07/2023 13:50, Rémy Maucherat wrote:
> On Thu, Jul 27, 2023 at 5:04 PM Mark Thomas <ma...@apache.org> wrote:
>>
>> I've refactored things to the point where the unit tests run without
>> generating any warnings for pinning. I suspect further issues will be
>> identified over time and we can address those as they are found.
>
> Ok, so the changes for Loom are: sync -> equivalent lock. That's ok
> but it's a bit riskier (syncs are hard to avoid releasing).
True, but each lock I added should follow the
lock.lock();
try {
// Do stuff
} finally {
lock.unlock();
}
pattern. This is also one of things that SpotBugs checks for so we
should catch if any new locks get added incorrectly or if any of these
get changed.
Mark
>
>> Mark
>>
>>
>> On 25/07/2023 10:21, Mark Thomas wrote:
>>> Never mind. Pretty much as soon as I hit send I managed to trigger the
>>> issue.
>>>
>>> Mark
>>>
>>>
>>> On 25/07/2023 10:19, Mark Thomas wrote:
>>>> Daniel,
>>>>
>>>> How did you trigger the pinning? I'm running some basic tests with
>>>> -Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads
>>>> reported.
>>>>
>>>> Mark
>>>>
>>>>
>>>> On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote:
>>>>> Mark,
>>>>>
>>>>> Thanks for letting me know. I will wait for the August release to test.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>> El jue, 6 jul 2023 a las 15:13, Mark Thomas (<ma...@apache.org>)
>>>>> escribió:
>>>>>>
>>>>>>
>>>>>> 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez <es...@gmail.com>:
>>>>>>
>>>>>>> I am aware Tomcat community did a great effort to move Tomat to
>>>>>>> Virtual Threads friendly, but I am not sure why HTTP2 was not part of
>>>>>>> that effort?
>>>>>>
>>>>>> The plan was always to see where the bottlenecks were as folks start to
>>>>>> experiment with Loom support and fix issues as they arose. It helps
>>>>>> focus
>>>>>> effort on where it is really needed.
>>>>>>
>>>>>> These fixes look fairly simple. We should be able to get them done for
>>>>>> the August releases (the July releases have already been tagged). We
>>>>>> can
>>>>>> make -dev builds available earlier or you can build Tomcat from
>>>>>> source to
>>>>>> test the changes if you are interested.
>>>>>>
>>>>>> As it happens I've spent most of today looking at ThreadLocal vs
>>>>>> SynchronizedStack vs new Object() in various places in the Tomcat code
>>>>>> base without reaching a clear conclusion. Which kind of proves the
>>>>>> point
>>>>>> that if we guess where bottlenecks might be we'll probably be wrong.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Posted by Rémy Maucherat <re...@apache.org>.
On Thu, Jul 27, 2023 at 5:04 PM Mark Thomas <ma...@apache.org> wrote:
>
> I've refactored things to the point where the unit tests run without
> generating any warnings for pinning. I suspect further issues will be
> identified over time and we can address those as they are found.
Ok, so the changes for Loom are: sync -> equivalent lock. That's ok
but it's a bit riskier (syncs are hard to avoid releasing).
> Mark
>
>
> On 25/07/2023 10:21, Mark Thomas wrote:
> > Never mind. Pretty much as soon as I hit send I managed to trigger the
> > issue.
> >
> > Mark
> >
> >
> > On 25/07/2023 10:19, Mark Thomas wrote:
> >> Daniel,
> >>
> >> How did you trigger the pinning? I'm running some basic tests with
> >> -Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads
> >> reported.
> >>
> >> Mark
> >>
> >>
> >> On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote:
> >>> Mark,
> >>>
> >>> Thanks for letting me know. I will wait for the August release to test.
> >>>
> >>> Regards.
> >>>
> >>>
> >>> El jue, 6 jul 2023 a las 15:13, Mark Thomas (<ma...@apache.org>)
> >>> escribió:
> >>>>
> >>>>
> >>>> 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez <es...@gmail.com>:
> >>>>
> >>>>> I am aware Tomcat community did a great effort to move Tomat to
> >>>>> Virtual Threads friendly, but I am not sure why HTTP2 was not part of
> >>>>> that effort?
> >>>>
> >>>> The plan was always to see where the bottlenecks were as folks start to
> >>>> experiment with Loom support and fix issues as they arose. It helps
> >>>> focus
> >>>> effort on where it is really needed.
> >>>>
> >>>> These fixes look fairly simple. We should be able to get them done for
> >>>> the August releases (the July releases have already been tagged). We
> >>>> can
> >>>> make -dev builds available earlier or you can build Tomcat from
> >>>> source to
> >>>> test the changes if you are interested.
> >>>>
> >>>> As it happens I've spent most of today looking at ThreadLocal vs
> >>>> SynchronizedStack vs new Object() in various places in the Tomcat code
> >>>> base without reaching a clear conclusion. Which kind of proves the
> >>>> point
> >>>> that if we guess where bottlenecks might be we'll probably be wrong.
> >>>>
> >>>> Mark
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>
> >>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Posted by Mark Thomas <ma...@apache.org>.
I've refactored things to the point where the unit tests run without
generating any warnings for pinning. I suspect further issues will be
identified over time and we can address those as they are found.
Mark
On 25/07/2023 10:21, Mark Thomas wrote:
> Never mind. Pretty much as soon as I hit send I managed to trigger the
> issue.
>
> Mark
>
>
> On 25/07/2023 10:19, Mark Thomas wrote:
>> Daniel,
>>
>> How did you trigger the pinning? I'm running some basic tests with
>> -Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads
>> reported.
>>
>> Mark
>>
>>
>> On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote:
>>> Mark,
>>>
>>> Thanks for letting me know. I will wait for the August release to test.
>>>
>>> Regards.
>>>
>>>
>>> El jue, 6 jul 2023 a las 15:13, Mark Thomas (<ma...@apache.org>)
>>> escribió:
>>>>
>>>>
>>>> 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez <es...@gmail.com>:
>>>>
>>>>> I am aware Tomcat community did a great effort to move Tomat to
>>>>> Virtual Threads friendly, but I am not sure why HTTP2 was not part of
>>>>> that effort?
>>>>
>>>> The plan was always to see where the bottlenecks were as folks start to
>>>> experiment with Loom support and fix issues as they arose. It helps
>>>> focus
>>>> effort on where it is really needed.
>>>>
>>>> These fixes look fairly simple. We should be able to get them done for
>>>> the August releases (the July releases have already been tagged). We
>>>> can
>>>> make -dev builds available earlier or you can build Tomcat from
>>>> source to
>>>> test the changes if you are interested.
>>>>
>>>> As it happens I've spent most of today looking at ThreadLocal vs
>>>> SynchronizedStack vs new Object() in various places in the Tomcat code
>>>> base without reaching a clear conclusion. Which kind of proves the
>>>> point
>>>> that if we guess where bottlenecks might be we'll probably be wrong.
>>>>
>>>> Mark
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Posted by Mark Thomas <ma...@apache.org>.
Never mind. Pretty much as soon as I hit send I managed to trigger the
issue.
Mark
On 25/07/2023 10:19, Mark Thomas wrote:
> Daniel,
>
> How did you trigger the pinning? I'm running some basic tests with
> -Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads
> reported.
>
> Mark
>
>
> On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote:
>> Mark,
>>
>> Thanks for letting me know. I will wait for the August release to test.
>>
>> Regards.
>>
>>
>> El jue, 6 jul 2023 a las 15:13, Mark Thomas (<ma...@apache.org>)
>> escribió:
>>>
>>>
>>> 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez <es...@gmail.com>:
>>>
>>>> I am aware Tomcat community did a great effort to move Tomat to
>>>> Virtual Threads friendly, but I am not sure why HTTP2 was not part of
>>>> that effort?
>>>
>>> The plan was always to see where the bottlenecks were as folks start to
>>> experiment with Loom support and fix issues as they arose. It helps
>>> focus
>>> effort on where it is really needed.
>>>
>>> These fixes look fairly simple. We should be able to get them done for
>>> the August releases (the July releases have already been tagged). We can
>>> make -dev builds available earlier or you can build Tomcat from
>>> source to
>>> test the changes if you are interested.
>>>
>>> As it happens I've spent most of today looking at ThreadLocal vs
>>> SynchronizedStack vs new Object() in various places in the Tomcat code
>>> base without reaching a clear conclusion. Which kind of proves the point
>>> that if we guess where bottlenecks might be we'll probably be wrong.
>>>
>>> Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Posted by Mark Thomas <ma...@apache.org>.
Daniel,
How did you trigger the pinning? I'm running some basic tests with
-Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads
reported.
Mark
On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote:
> Mark,
>
> Thanks for letting me know. I will wait for the August release to test.
>
> Regards.
>
>
> El jue, 6 jul 2023 a las 15:13, Mark Thomas (<ma...@apache.org>) escribió:
>>
>>
>> 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez <es...@gmail.com>:
>>
>>> I am aware Tomcat community did a great effort to move Tomat to
>>> Virtual Threads friendly, but I am not sure why HTTP2 was not part of
>>> that effort?
>>
>> The plan was always to see where the bottlenecks were as folks start to
>> experiment with Loom support and fix issues as they arose. It helps focus
>> effort on where it is really needed.
>>
>> These fixes look fairly simple. We should be able to get them done for
>> the August releases (the July releases have already been tagged). We can
>> make -dev builds available earlier or you can build Tomcat from source to
>> test the changes if you are interested.
>>
>> As it happens I've spent most of today looking at ThreadLocal vs
>> SynchronizedStack vs new Object() in various places in the Tomcat code
>> base without reaching a clear conclusion. Which kind of proves the point
>> that if we guess where bottlenecks might be we'll probably be wrong.
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Posted by Daniel Andres Pelaez Lopez <es...@gmail.com>.
Mark,
Thanks for letting me know. I will wait for the August release to test.
Regards.
El jue, 6 jul 2023 a las 15:13, Mark Thomas (<ma...@apache.org>) escribió:
>
>
> 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez <es...@gmail.com>:
>
> > I am aware Tomcat community did a great effort to move Tomat to
> > Virtual Threads friendly, but I am not sure why HTTP2 was not part of
> > that effort?
>
> The plan was always to see where the bottlenecks were as folks start to
> experiment with Loom support and fix issues as they arose. It helps focus
> effort on where it is really needed.
>
> These fixes look fairly simple. We should be able to get them done for
> the August releases (the July releases have already been tagged). We can
> make -dev builds available earlier or you can build Tomcat from source to
> test the changes if you are interested.
>
> As it happens I've spent most of today looking at ThreadLocal vs
> SynchronizedStack vs new Object() in various places in the Tomcat code
> base without reaching a clear conclusion. Which kind of proves the point
> that if we guess where bottlenecks might be we'll probably be wrong.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Daniel Andrés Pelaez López
Master’s Degree in IT Architectures, Universidad de los Andes.
Software Construction Specialist, Universidad de los Andes.
Bachelor's Degree in Computer Sciences, Universidad del Quindio.
e. estigma88@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Posted by Mark Thomas <ma...@apache.org>.
6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez <es...@gmail.com>:
> I am aware Tomcat community did a great effort to move Tomat to
> Virtual Threads friendly, but I am not sure why HTTP2 was not part of
> that effort?
The plan was always to see where the bottlenecks were as folks start to
experiment with Loom support and fix issues as they arose. It helps focus
effort on where it is really needed.
These fixes look fairly simple. We should be able to get them done for
the August releases (the July releases have already been tagged). We can
make -dev builds available earlier or you can build Tomcat from source to
test the changes if you are interested.
As it happens I've spent most of today looking at ThreadLocal vs
SynchronizedStack vs new Object() in various places in the Tomcat code
base without reaching a clear conclusion. Which kind of proves the point
that if we guess where bottlenecks might be we'll probably be wrong.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org