You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Rainer Jung <ra...@kippdata.de> on 2024/03/20 11:22:11 UTC
Observability of virtual threads
Hi hi,
I wanted to share an observation and I hope the things are correct how I
am describing them. Maybe things have already improved and I am not
aware of it, hints welcome.
Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
handle observability of virtual threads from inside the JVM and tooling.
The final outcome is, that virtual threads are not included in the
typical JVM APIs which one can use to observe threads, like enumerating
them or accessing the stacks of individual threads. As a consequence,
also jstack and "kill -QUIT" do not show virtual threads at all, not
even when they are attached to a native thread and executing code.
There is one single method to help with observability of virtual threads
https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
which dumps a full thread dump to a file. Of course that is by no means
appropriate, if you want to do a fine grained observation. At least you
can choose to write a json structure instead of a hard to parse text
format, but that's it.
For instance I am often using a tool, that inspects our
RequestProcessors to find long running requests and then retrieves the
list of Java threads as ThreadInfo objects to find the one executing the
request and finally retrieves the stack of that request from the JVM.
Such an approach is no longer possible. Almost all of JMX does not show
any info about virtual threads.
It also seems Tomcat no longer uses a pool when a connector is
configured to use virtual threads. So there are no metrics like
currentThreadCount or currentThreadsBusy.
I guess this also limits the use of some manager features and probably
also the StuckThreadsDetectionValve when combined with virtual threads.
Of course Tomcat has solved most of the problems, that virtual threads
want to solve, by doing the hard work of using the NIO and NIO2 APIs. So
virtual threads are probably not that important for us. But we should be
aware of the observability deficiencies whenever we would discuss
whether we could switch from NIO or NIO2 to virtual threads to simplify
connector code in some distant future.
Best regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Observability of virtual threads
Posted by Rainer Jung <ra...@kippdata.de>.
Thanks Romain,
I am well aware of the limits oof this attempt concerning async
processing. Still lots of applications can still be diagnosed with this
methodology and I am a bit frustrated, that project loom simply dropped
support for thread inspection via JMX and as a replacement provided a
full dump to a file (!).
But of course you are right: for complex async processing this approach
does not work. Interestingly although Tomcat NIO and NIO2 processors use
async APIs, they can still be inspected with this simple approach.
Best regards,
Rainer
Am 20.03.24 um 13:58 schrieb Romain Manni-Bucau:
> Hi Rainer,
>
> Think this is not related to virtual threads I think and way older (2010?).
> Since some years we use AsyncContext and therefore thread dumps don't match
> pending requests but just active computation (you can get 8K requests with
> 8 threads for ex).
> Most of the sync instrumentation (assumptions) are broken since Servlet 3.0
> and must not be used to conclude anything but generally other metrics and
> data enable to monitor the server properly (request counter for example is
> decoralated from threads, active connections for database - generic term -
> etc).
> Thread stacks stay useful for performance related work but not for hanging
> investigation (for good since it enables a better scalability in general)
> so guess it is mainly a habit switch in terms of tools, no?
>
> Hope it makes sense.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le mer. 20 mars 2024 à 13:11, Rainer Jung <ra...@kippdata.de> a
> écrit :
>
>> Hi hi,
>>
>> I wanted to share an observation and I hope the things are correct how I
>> am describing them. Maybe things have already improved and I am not
>> aware of it, hints welcome.
>>
>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
>> handle observability of virtual threads from inside the JVM and tooling.
>>
>> The final outcome is, that virtual threads are not included in the
>> typical JVM APIs which one can use to observe threads, like enumerating
>> them or accessing the stacks of individual threads. As a consequence,
>> also jstack and "kill -QUIT" do not show virtual threads at all, not
>> even when they are attached to a native thread and executing code.
>>
>> There is one single method to help with observability of virtual threads
>>
>>
>> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>>
>> which dumps a full thread dump to a file. Of course that is by no means
>> appropriate, if you want to do a fine grained observation. At least you
>> can choose to write a json structure instead of a hard to parse text
>> format, but that's it.
>>
>> For instance I am often using a tool, that inspects our
>> RequestProcessors to find long running requests and then retrieves the
>> list of Java threads as ThreadInfo objects to find the one executing the
>> request and finally retrieves the stack of that request from the JVM.
>> Such an approach is no longer possible. Almost all of JMX does not show
>> any info about virtual threads.
>>
>> It also seems Tomcat no longer uses a pool when a connector is
>> configured to use virtual threads. So there are no metrics like
>> currentThreadCount or currentThreadsBusy.
>>
>> I guess this also limits the use of some manager features and probably
>> also the StuckThreadsDetectionValve when combined with virtual threads.
>>
>> Of course Tomcat has solved most of the problems, that virtual threads
>> want to solve, by doing the hard work of using the NIO and NIO2 APIs. So
>> virtual threads are probably not that important for us. But we should be
>> aware of the observability deficiencies whenever we would discuss
>> whether we could switch from NIO or NIO2 to virtual threads to simplify
>> connector code in some distant future.
>>
>> Best regards,
>>
>> Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Observability of virtual threads
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi Rainer,
Think this is not related to virtual threads I think and way older (2010?).
Since some years we use AsyncContext and therefore thread dumps don't match
pending requests but just active computation (you can get 8K requests with
8 threads for ex).
Most of the sync instrumentation (assumptions) are broken since Servlet 3.0
and must not be used to conclude anything but generally other metrics and
data enable to monitor the server properly (request counter for example is
decoralated from threads, active connections for database - generic term -
etc).
Thread stacks stay useful for performance related work but not for hanging
investigation (for good since it enables a better scalability in general)
so guess it is mainly a habit switch in terms of tools, no?
Hope it makes sense.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le mer. 20 mars 2024 à 13:11, Rainer Jung <ra...@kippdata.de> a
écrit :
> Hi hi,
>
> I wanted to share an observation and I hope the things are correct how I
> am describing them. Maybe things have already improved and I am not
> aware of it, hints welcome.
>
> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
> handle observability of virtual threads from inside the JVM and tooling.
>
> The final outcome is, that virtual threads are not included in the
> typical JVM APIs which one can use to observe threads, like enumerating
> them or accessing the stacks of individual threads. As a consequence,
> also jstack and "kill -QUIT" do not show virtual threads at all, not
> even when they are attached to a native thread and executing code.
>
> There is one single method to help with observability of virtual threads
>
>
> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>
> which dumps a full thread dump to a file. Of course that is by no means
> appropriate, if you want to do a fine grained observation. At least you
> can choose to write a json structure instead of a hard to parse text
> format, but that's it.
>
> For instance I am often using a tool, that inspects our
> RequestProcessors to find long running requests and then retrieves the
> list of Java threads as ThreadInfo objects to find the one executing the
> request and finally retrieves the stack of that request from the JVM.
> Such an approach is no longer possible. Almost all of JMX does not show
> any info about virtual threads.
>
> It also seems Tomcat no longer uses a pool when a connector is
> configured to use virtual threads. So there are no metrics like
> currentThreadCount or currentThreadsBusy.
>
> I guess this also limits the use of some manager features and probably
> also the StuckThreadsDetectionValve when combined with virtual threads.
>
> Of course Tomcat has solved most of the problems, that virtual threads
> want to solve, by doing the hard work of using the NIO and NIO2 APIs. So
> virtual threads are probably not that important for us. But we should be
> aware of the observability deficiencies whenever we would discuss
> whether we could switch from NIO or NIO2 to virtual threads to simplify
> connector code in some distant future.
>
> Best regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: Observability of virtual threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Romain,
On 3/20/24 16:25, Romain Manni-Bucau wrote:
> Chris, added some comments inline.
>
> Le mer. 20 mars 2024 à 19:41, Christopher Schultz <
> chris@christopherschultz.net> a écrit :
>
>> Romain,
>>
>> On 3/20/24 13:34, Romain Manni-Bucau wrote:
>>> Thread dumps being dump of threads - literally os threads - and virtual
>>> threads not being threads at all - they are runnables in a dedicated
>> thread
>>> pool - it is quite fair to not make them the same and have their
>> scheduler
>>> - pool - only in the thread dump but not themselves no?
>>
>> Maybe.
>>
>> If you take a thread dump today (with a "real" thread), you only get a
>> Java stack trace, you get no "native stack trace" or anything like that.
>> So from that perspective, the "thread" is really the instance of
>> java.lang.Thread which could just as easily be a Virtual Thread.
>>
>> You also get no scheduling information, other than what the thread's
>> "priority" is... but you can't get any real-time data about where it
>> sits in the scheduling queue, etc.
>>
>
> Well, this does not work like that, as mentionned it is a ForkJoinPool so
> not a plain queue - except the design which is not a 1 task = 1 position,
> it has multiple queues - so the position in "the queue" does not give you
> the information you are mentionned there.
I was using "scheduling queue" as an abstract idea, there. I know it's a
work-stealing queue which is not straightforward to describe.
> If you really care about monitoring this pool you can just
> instrument java.lang.VirtualThread#DEFAULT_SCHEDULER (plain reflection
> works good enough if you open this part but agents too).
This is Rainer's complaint (which I agree with): there are no standard
-- available -- APIs for this kind of thing. Use of reflection for
instrumentation is an anti-pattern IMHO.
The JRE team over the years has learned that instrumentation is a vital
part of application monitoring and has done a really great job of
exposing JRE internals through standard APIs. This one is a sore spot
that really needs to be fixed.
>> I'm much less interested in what the "native thread" is doing _below_
>> the Java part. Presumably, it's always running
>> Thread.cpp::runTheJavaCode and that's not useful information to anybody.
>>
>
> Depends how you instrument/query it, while at code level it does not change
> much things and you still get the thread stack context.
... unless you can't get a reference to the threads in the first place,
which is what Rainer is saying.
> For raw thread dumps you indeed need the new jcmd command
> (Thread.dump_to_file) and I agree there is no real point to not let it go
> in the plain jstack with a new toggle you can enable at need (but hopefully
> not by default since we'll end up with undownloadable/too big dumps).
I don't know why this is any different or worse than taking a heap dump.
Heap dumps have been available for decades and nobody says "OMG what
will we do with a 16GiB heap dump?!" You only take a heap-dump when you
NEED it, and the same is true for a thread dump. If you want a thread
dump and your system can't log it all because you have 100M threads,
well, then you have made some bad choices.
If there existed an API to query (virtual) threads, your application or
some tool could examine the threads and only log those you care about.
That would solve BOTH problems.
> Ultimately using reflection and opening jdk.internal.vm package you can
> also just use ThreadContainers.root() and iterate over children()/threads()
> but not sure how portable it will be.
>
> There is a negative effect there, before we were often decorating executors
> (the Runnable to execute) to add before/after context and potentially track
> threads there but it does not work anymore since threads are totally
> unbounded.
>
>
>>
>> If the Virtual Thread is not mounted on an OS thread, then it's
>> "suspended" or "blocked" or whatever-it-is. If it's on an OS thread, it
>> had better be running: that's the whole point of the scheme. (I suppose
>> it could be BLOCKED-yet-on-an-OS-thread -- one of the current problems
>> which hopefully will be less of a problem in the future, but I'd like to
>> ignore that for now).
>>
>> I don't know what's wrong with having millions of threads, and still
>> being able to walk through them using existing APIs. Nobody walks
>> through threads for no good reason... if you want to see the threads,
>> you should be able to see the threads.
>>
>> If I wanted to walk-through every instance of java.lang.String in a JVM
>> (¡millions of them!), then I should be able to do it even if (a) it's a
>> weird thing to do and (b) it will take a while. Well, I want to do it,
>> so let me do it(!).
>>
>> If you have only one "real" Thread (e.g. the main thread) and everything
>> else is Virtual... when you ask for a thread dump or walk all the
>> threads, do you only see "main" and not the mounted-on-OS-threads
>> Virtual Threads? If the JVM were willing to consider Virtual Threads
>> "visible" and therefore dumpable, etc. through existing interfaces
>> _while they were mounted on an OS thread_ I could almost agree that
>> makes some sense. But if I have a unit of work not making any progress,
>> I'm gonna want to be able to see that (Virtual) Thread in a thread dump
>> or any other kind of analysis tool, including its stack trace.
>>
>>> Similarly to why threadlocal are not recommended for virtual thread this
>>> would probably make an useless pressure on the JVM IMHO - why there is an
>>> option to see it but it is mainly for debug purposes.
>>> See virtual threads as continuations (suspendable/resumable "Runnable")
>> in
>>> a dedicated and not programmatically configurable nor
>> selectable/proviable
>>> thread pool, they are not in thread dumps and this doesnt bother you ;) -
>>> ultimately if you want it you want all java objects to be monitored and
>> in
>>> the thread dump which would be weird - but I agree the semantic is
>>> misleading.
>>
>> I'm not sure why ThreadLocals are not recommended for Virtual Threads.
>> Honestly, the presence ThreadLocal is a gigantic hack for badly-written
>> APIs but if you accept that fact, there doesn't seem to be anything
>> really bad about it. It "doesn't make sense" for Virtual Threads because
>> it's wasteful: you probably use something once or twice and don't get
>> the benefit of it when the thread dies -- because you aren't supposed to
>> use thread pools anymore. But it's no less wasteful and dumb than it was
>> in the first place.
>>
>
> Guess the issue is they are bound to low level threads and for virtual
> threads this consumes too much mem and since they are shared between
> standard threads and virtual threads you can't optimize them by backing
> them by scopes like that, so let's blame the "history" maybe?
I honestly haven't looked at how ThreadLocal is managed. with Virtual
Threads. Does anybody use multi-megabyte ThreadLocals? If not, it
doesn't matter. The whole point of VTs is that yu fire-off a unit of
work and forget about it forever. The thread eventually dies and you get
your memory back.
If you use too much memory that way, VT isn't the solution to the
problem you are trying to solve.
>> IIRC, Tomcat uses ThreadLocal to store date-formatting objects for
>> access logs. That's a great use for ThreadLocal. If the threads are no
>> longer pooled, it's "wasteful" in that we create and discard a
>> SimpleDateFormat for every log, but what's the alternative? Use a shared
>> SimpleDateFormat and synchronize across threads? Yuck. So it's up to
>> Tomcat to decide if ThreadLocal makes sense in a Virtual Thread world.
>>
>
> DateTimeFormatter solved it some versions ago, a singleton is the best
> solution for such a thing.
DateTimeFormatter wasn't available until Java 1.8. In order to make
Tomcat's code maintainable across all supported versions, we have to go
back to Java 7, so we haven't moved to java.time. I suspect we will do
that relatively soon now that 8.5 is reaching EOL.
> Anything else - when thread safety can't be reached - will end up in
> "pools" - a queue being a cheap pool - which should be sized as the backing
> forkjoin pool of virtual threads probably.
Yes, we use pools for other things where each thread doesn't really
needs its own copy. Sometimes performance is more important than memory
reduction. This is why I mention SimpleDateFormat: it's a very very
small class, and one-per-thread is not expensive at all, even if you
decide that 100M threads is a good idea.
>> (Honestly, we need to start using java.time. Now that Tomcat 8.5 no
>> longer needs to be supported, we should look at our options for moving-on.)
>>
>>> From my tests virtual threads do their job but they stay slower than a
>>> proper reactive/async impl mainly due to the overhead they add
>> *everywhere*
>>> compare to reactive programming plus this single thread pool issue which
>>> adds a lot of contention when all the app/lot of tasks is/are done using
>> it
>>> - vs bulkhead pattern for ex.
>>> But if you come from a plain sync application it can be very interesting
>> if
>>> it stays compatible and you don't need to add throttling to control the
>>> memory - often in the old style you throttle with the number of threads,
>>> now you need a semaphore or alike.
>>> Will not be a free lunch ;).
>>
>> Agreed, free lunches never exist.
>>
>> But.
>>
>> If an application exists that never bothered to convert to using
>> non-blocking I/O but uses many threads, it could be very easily changed
>> (slightly) to use Virtual Threads and get a performance boost out of the
>> deal. Or at least be able to schedule many more tasks without running
>> out of stack space.
>>
>> The price you pay for that is CPU usage. And most of us have CPU to spare.
>>
>
> Hmm, I have more than doubt. I currently have a case for a customer, we
> scaled to the desired perf level tuning threads to a high number -
> everything is old style and synchronous - but if we move to VT it will just
> blow up and explode in mem in ~5s of load - tested a close approach.
Hmm. What is using all that memory? Is it Java-heap, or is it native? Or
are they launching 100M threads and that's why memory is exhausted?
without a bounded thread pool, it's easy to just allow infinite work to
be scheduled. Infinite scheduled-work requires infinite memory.
> So I suspect most of blocking apps also rely in their prod deployment on
> the implicit thread throttling so VT can just make them blow up so "easy"
> in terms of code, hard in terms of tuning since them the actual throttling
> was not planned anywhere else in the code I think.
That would make sense.
-chris
>>>> On 3/20/24 07:22, Rainer Jung wrote:
>>>>> I wanted to share an observation and I hope the things are correct how
>> I
>>>>> am describing them. Maybe things have already improved and I am not
>>>>> aware of it, hints welcome.
>>>>>
>>>>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
>>>>> handle observability of virtual threads from inside the JVM and
>> tooling.
>>>>>
>>>>> The final outcome is, that virtual threads are not included in the
>>>>> typical JVM APIs which one can use to observe threads, like enumerating
>>>>> them or accessing the stacks of individual threads. As a consequence,
>>>>> also jstack and "kill -QUIT" do not show virtual threads at all, not
>>>>> even when they are attached to a native thread and executing code.
>>>>
>>>> O_O
>>>>
>>>>> There is one single method to help with observability of virtual
>> threads
>>>>>
>>>>>
>>>>
>> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>>>>>
>>>>> which dumps a full thread dump to a file. Of course that is by no means
>>>>> appropriate, if you want to do a fine grained observation. At least you
>>>>> can choose to write a json structure instead of a hard to parse text
>>>>> format, but that's it.
>>>>
>>>> Boy, that's a real miss from the JVM team. I'm surprised that they have
>>>> overlooked this. I don't see a real reason that a Virtual Thread would
>>>> be treated any differently than a regular thread.
>>>>
>>>>> For instance I am often using a tool, that inspects our
>>>>> RequestProcessors to find long running requests and then retrieves the
>>>>> list of Java threads as ThreadInfo objects to find the one executing
>> the
>>>>> request and finally retrieves the stack of that request from the JVM.
>>>>> Such an approach is no longer possible. Almost all of JMX does not show
>>>>> any info about virtual threads.
>>>>>
>>>>> It also seems Tomcat no longer uses a pool when a connector is
>>>>> configured to use virtual threads. So there are no metrics like
>>>>> currentThreadCount or currentThreadsBusy.
>>>>
>>>> When using virtual threads, the number of requests is the same as the
>>>> number of threads, since each request -> new Virtual Thread.... though I
>>>> think the request-count is only incremented when the request has
>>>> completed, so maybe there are threads running that can't be counted
>> (yet).
>>>>
>>>> But I understand that you were hoping to get some information about
>>>> these threads and though maybe Tomcat's metrics could help. I guess not
>>>> really.
>>>>
>>>> But thread-busy count is the same as in-flight-request count. And
>>>> current thread count is the same as in-flight-request count as well.
>>>>
>>>>> I guess this also limits the use of some manager features and probably
>>>>> also the StuckThreadsDetectionValve when combined with virtual threads.
>>>>
>>>> Most likely. I'm fairly sure that uses the "usual Thread-walker things"
>>>> which you are reporting do not work any more with VTs.
>>>>
>>>> What JVM are you using for your investigations?
>>>>
>>>>> Of course Tomcat has solved most of the problems, that virtual threads
>>>>> want to solve, by doing the hard work of using the NIO and NIO2 APIs.
>> So
>>>>> virtual threads are probably not that important for us. But we should
>> be
>>>>> aware of the observability deficiencies whenever we would discuss
>>>>> whether we could switch from NIO or NIO2 to virtual threads to simplify
>>>>> connector code in some distant future.
>>>>
>>>> +1
>>>>
>>>> I've been enthusiastically talking with markt about dropping all that
>>>> nasty NIO stuff and "going back to BIO with virtual threads" which, at
>>>> this point, is mostly a threat and not a promise. But if VTs really
>>>> deliver everything they are claiming to deliver, then it may be possible
>>>> to go back to BIO as long as servlet-async is retired. And I'm not
>>>> holding my breath on that one.
>>>>
>>>> -chris
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Observability of virtual threads
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Chris, added some comments inline.
Le mer. 20 mars 2024 à 19:41, Christopher Schultz <
chris@christopherschultz.net> a écrit :
> Romain,
>
> On 3/20/24 13:34, Romain Manni-Bucau wrote:
> > Thread dumps being dump of threads - literally os threads - and virtual
> > threads not being threads at all - they are runnables in a dedicated
> thread
> > pool - it is quite fair to not make them the same and have their
> scheduler
> > - pool - only in the thread dump but not themselves no?
>
> Maybe.
>
> If you take a thread dump today (with a "real" thread), you only get a
> Java stack trace, you get no "native stack trace" or anything like that.
> So from that perspective, the "thread" is really the instance of
> java.lang.Thread which could just as easily be a Virtual Thread.
>
> You also get no scheduling information, other than what the thread's
> "priority" is... but you can't get any real-time data about where it
> sits in the scheduling queue, etc.
>
Well, this does not work like that, as mentionned it is a ForkJoinPool so
not a plain queue - except the design which is not a 1 task = 1 position,
it has multiple queues - so the position in "the queue" does not give you
the information you are mentionned there.
If you really care about monitoring this pool you can just
instrument java.lang.VirtualThread#DEFAULT_SCHEDULER (plain reflection
works good enough if you open this part but agents too).
>
> I'm much less interested in what the "native thread" is doing _below_
> the Java part. Presumably, it's always running
> Thread.cpp::runTheJavaCode and that's not useful information to anybody.
>
Depends how you instrument/query it, while at code level it does not change
much things and you still get the thread stack context.
For raw thread dumps you indeed need the new jcmd command
(Thread.dump_to_file) and I agree there is no real point to not let it go
in the plain jstack with a new toggle you can enable at need (but hopefully
not by default since we'll end up with undownloadable/too big dumps).
Ultimately using reflection and opening jdk.internal.vm package you can
also just use ThreadContainers.root() and iterate over children()/threads()
but not sure how portable it will be.
There is a negative effect there, before we were often decorating executors
(the Runnable to execute) to add before/after context and potentially track
threads there but it does not work anymore since threads are totally
unbounded.
>
> If the Virtual Thread is not mounted on an OS thread, then it's
> "suspended" or "blocked" or whatever-it-is. If it's on an OS thread, it
> had better be running: that's the whole point of the scheme. (I suppose
> it could be BLOCKED-yet-on-an-OS-thread -- one of the current problems
> which hopefully will be less of a problem in the future, but I'd like to
> ignore that for now).
>
> I don't know what's wrong with having millions of threads, and still
> being able to walk through them using existing APIs. Nobody walks
> through threads for no good reason... if you want to see the threads,
> you should be able to see the threads.
>
> If I wanted to walk-through every instance of java.lang.String in a JVM
> (¡millions of them!), then I should be able to do it even if (a) it's a
> weird thing to do and (b) it will take a while. Well, I want to do it,
> so let me do it(!).
>
> If you have only one "real" Thread (e.g. the main thread) and everything
> else is Virtual... when you ask for a thread dump or walk all the
> threads, do you only see "main" and not the mounted-on-OS-threads
> Virtual Threads? If the JVM were willing to consider Virtual Threads
> "visible" and therefore dumpable, etc. through existing interfaces
> _while they were mounted on an OS thread_ I could almost agree that
> makes some sense. But if I have a unit of work not making any progress,
> I'm gonna want to be able to see that (Virtual) Thread in a thread dump
> or any other kind of analysis tool, including its stack trace.
>
> > Similarly to why threadlocal are not recommended for virtual thread this
> > would probably make an useless pressure on the JVM IMHO - why there is an
> > option to see it but it is mainly for debug purposes.
> > See virtual threads as continuations (suspendable/resumable "Runnable")
> in
> > a dedicated and not programmatically configurable nor
> selectable/proviable
> > thread pool, they are not in thread dumps and this doesnt bother you ;) -
> > ultimately if you want it you want all java objects to be monitored and
> in
> > the thread dump which would be weird - but I agree the semantic is
> > misleading.
>
> I'm not sure why ThreadLocals are not recommended for Virtual Threads.
> Honestly, the presence ThreadLocal is a gigantic hack for badly-written
> APIs but if you accept that fact, there doesn't seem to be anything
> really bad about it. It "doesn't make sense" for Virtual Threads because
> it's wasteful: you probably use something once or twice and don't get
> the benefit of it when the thread dies -- because you aren't supposed to
> use thread pools anymore. But it's no less wasteful and dumb than it was
> in the first place.
>
Guess the issue is they are bound to low level threads and for virtual
threads this consumes too much mem and since they are shared between
standard threads and virtual threads you can't optimize them by backing
them by scopes like that, so let's blame the "history" maybe?
>
> IIRC, Tomcat uses ThreadLocal to store date-formatting objects for
> access logs. That's a great use for ThreadLocal. If the threads are no
> longer pooled, it's "wasteful" in that we create and discard a
> SimpleDateFormat for every log, but what's the alternative? Use a shared
> SimpleDateFormat and synchronize across threads? Yuck. So it's up to
> Tomcat to decide if ThreadLocal makes sense in a Virtual Thread world.
>
DateTimeFormatter solved it some versions ago, a singleton is the best
solution for such a thing.
Anything else - when thread safety can't be reached - will end up in
"pools" - a queue being a cheap pool - which should be sized as the backing
forkjoin pool of virtual threads probably.
>
> (Honestly, we need to start using java.time. Now that Tomcat 8.5 no
> longer needs to be supported, we should look at our options for moving-on.)
>
> > From my tests virtual threads do their job but they stay slower than a
> > proper reactive/async impl mainly due to the overhead they add
> *everywhere*
> > compare to reactive programming plus this single thread pool issue which
> > adds a lot of contention when all the app/lot of tasks is/are done using
> it
> > - vs bulkhead pattern for ex.
> > But if you come from a plain sync application it can be very interesting
> if
> > it stays compatible and you don't need to add throttling to control the
> > memory - often in the old style you throttle with the number of threads,
> > now you need a semaphore or alike.
> > Will not be a free lunch ;).
>
> Agreed, free lunches never exist.
>
> But.
>
> If an application exists that never bothered to convert to using
> non-blocking I/O but uses many threads, it could be very easily changed
> (slightly) to use Virtual Threads and get a performance boost out of the
> deal. Or at least be able to schedule many more tasks without running
> out of stack space.
>
> The price you pay for that is CPU usage. And most of us have CPU to spare.
>
Hmm, I have more than doubt. I currently have a case for a customer, we
scaled to the desired perf level tuning threads to a high number -
everything is old style and synchronous - but if we move to VT it will just
blow up and explode in mem in ~5s of load - tested a close approach.
So I suspect most of blocking apps also rely in their prod deployment on
the implicit thread throttling so VT can just make them blow up so "easy"
in terms of code, hard in terms of tuning since them the actual throttling
was not planned anywhere else in the code I think.
>
> -chris
>
> >> Rainer,
> >>
> >> Thanks for writing this up.
> >>
> >> On 3/20/24 07:22, Rainer Jung wrote:
> >>> I wanted to share an observation and I hope the things are correct how
> I
> >>> am describing them. Maybe things have already improved and I am not
> >>> aware of it, hints welcome.
> >>>
> >>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
> >>> handle observability of virtual threads from inside the JVM and
> tooling.
> >>>
> >>> The final outcome is, that virtual threads are not included in the
> >>> typical JVM APIs which one can use to observe threads, like enumerating
> >>> them or accessing the stacks of individual threads. As a consequence,
> >>> also jstack and "kill -QUIT" do not show virtual threads at all, not
> >>> even when they are attached to a native thread and executing code.
> >>
> >> O_O
> >>
> >>> There is one single method to help with observability of virtual
> threads
> >>>
> >>>
> >>
> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
> >>>
> >>> which dumps a full thread dump to a file. Of course that is by no means
> >>> appropriate, if you want to do a fine grained observation. At least you
> >>> can choose to write a json structure instead of a hard to parse text
> >>> format, but that's it.
> >>
> >> Boy, that's a real miss from the JVM team. I'm surprised that they have
> >> overlooked this. I don't see a real reason that a Virtual Thread would
> >> be treated any differently than a regular thread.
> >>
> >>> For instance I am often using a tool, that inspects our
> >>> RequestProcessors to find long running requests and then retrieves the
> >>> list of Java threads as ThreadInfo objects to find the one executing
> the
> >>> request and finally retrieves the stack of that request from the JVM.
> >>> Such an approach is no longer possible. Almost all of JMX does not show
> >>> any info about virtual threads.
> >>>
> >>> It also seems Tomcat no longer uses a pool when a connector is
> >>> configured to use virtual threads. So there are no metrics like
> >>> currentThreadCount or currentThreadsBusy.
> >>
> >> When using virtual threads, the number of requests is the same as the
> >> number of threads, since each request -> new Virtual Thread.... though I
> >> think the request-count is only incremented when the request has
> >> completed, so maybe there are threads running that can't be counted
> (yet).
> >>
> >> But I understand that you were hoping to get some information about
> >> these threads and though maybe Tomcat's metrics could help. I guess not
> >> really.
> >>
> >> But thread-busy count is the same as in-flight-request count. And
> >> current thread count is the same as in-flight-request count as well.
> >>
> >>> I guess this also limits the use of some manager features and probably
> >>> also the StuckThreadsDetectionValve when combined with virtual threads.
> >>
> >> Most likely. I'm fairly sure that uses the "usual Thread-walker things"
> >> which you are reporting do not work any more with VTs.
> >>
> >> What JVM are you using for your investigations?
> >>
> >>> Of course Tomcat has solved most of the problems, that virtual threads
> >>> want to solve, by doing the hard work of using the NIO and NIO2 APIs.
> So
> >>> virtual threads are probably not that important for us. But we should
> be
> >>> aware of the observability deficiencies whenever we would discuss
> >>> whether we could switch from NIO or NIO2 to virtual threads to simplify
> >>> connector code in some distant future.
> >>
> >> +1
> >>
> >> I've been enthusiastically talking with markt about dropping all that
> >> nasty NIO stuff and "going back to BIO with virtual threads" which, at
> >> this point, is mostly a threat and not a promise. But if VTs really
> >> deliver everything they are claiming to deliver, then it may be possible
> >> to go back to BIO as long as servlet-async is retired. And I'm not
> >> holding my breath on that one.
> >>
> >> -chris
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: dev-help@tomcat.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: Observability of virtual threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Romain,
On 3/20/24 13:34, Romain Manni-Bucau wrote:
> Thread dumps being dump of threads - literally os threads - and virtual
> threads not being threads at all - they are runnables in a dedicated thread
> pool - it is quite fair to not make them the same and have their scheduler
> - pool - only in the thread dump but not themselves no?
Maybe.
If you take a thread dump today (with a "real" thread), you only get a
Java stack trace, you get no "native stack trace" or anything like that.
So from that perspective, the "thread" is really the instance of
java.lang.Thread which could just as easily be a Virtual Thread.
You also get no scheduling information, other than what the thread's
"priority" is... but you can't get any real-time data about where it
sits in the scheduling queue, etc.
I'm much less interested in what the "native thread" is doing _below_
the Java part. Presumably, it's always running
Thread.cpp::runTheJavaCode and that's not useful information to anybody.
If the Virtual Thread is not mounted on an OS thread, then it's
"suspended" or "blocked" or whatever-it-is. If it's on an OS thread, it
had better be running: that's the whole point of the scheme. (I suppose
it could be BLOCKED-yet-on-an-OS-thread -- one of the current problems
which hopefully will be less of a problem in the future, but I'd like to
ignore that for now).
I don't know what's wrong with having millions of threads, and still
being able to walk through them using existing APIs. Nobody walks
through threads for no good reason... if you want to see the threads,
you should be able to see the threads.
If I wanted to walk-through every instance of java.lang.String in a JVM
(¡millions of them!), then I should be able to do it even if (a) it's a
weird thing to do and (b) it will take a while. Well, I want to do it,
so let me do it(!).
If you have only one "real" Thread (e.g. the main thread) and everything
else is Virtual... when you ask for a thread dump or walk all the
threads, do you only see "main" and not the mounted-on-OS-threads
Virtual Threads? If the JVM were willing to consider Virtual Threads
"visible" and therefore dumpable, etc. through existing interfaces
_while they were mounted on an OS thread_ I could almost agree that
makes some sense. But if I have a unit of work not making any progress,
I'm gonna want to be able to see that (Virtual) Thread in a thread dump
or any other kind of analysis tool, including its stack trace.
> Similarly to why threadlocal are not recommended for virtual thread this
> would probably make an useless pressure on the JVM IMHO - why there is an
> option to see it but it is mainly for debug purposes.
> See virtual threads as continuations (suspendable/resumable "Runnable") in
> a dedicated and not programmatically configurable nor selectable/proviable
> thread pool, they are not in thread dumps and this doesnt bother you ;) -
> ultimately if you want it you want all java objects to be monitored and in
> the thread dump which would be weird - but I agree the semantic is
> misleading.
I'm not sure why ThreadLocals are not recommended for Virtual Threads.
Honestly, the presence ThreadLocal is a gigantic hack for badly-written
APIs but if you accept that fact, there doesn't seem to be anything
really bad about it. It "doesn't make sense" for Virtual Threads because
it's wasteful: you probably use something once or twice and don't get
the benefit of it when the thread dies -- because you aren't supposed to
use thread pools anymore. But it's no less wasteful and dumb than it was
in the first place.
IIRC, Tomcat uses ThreadLocal to store date-formatting objects for
access logs. That's a great use for ThreadLocal. If the threads are no
longer pooled, it's "wasteful" in that we create and discard a
SimpleDateFormat for every log, but what's the alternative? Use a shared
SimpleDateFormat and synchronize across threads? Yuck. So it's up to
Tomcat to decide if ThreadLocal makes sense in a Virtual Thread world.
(Honestly, we need to start using java.time. Now that Tomcat 8.5 no
longer needs to be supported, we should look at our options for moving-on.)
> From my tests virtual threads do their job but they stay slower than a
> proper reactive/async impl mainly due to the overhead they add *everywhere*
> compare to reactive programming plus this single thread pool issue which
> adds a lot of contention when all the app/lot of tasks is/are done using it
> - vs bulkhead pattern for ex.
> But if you come from a plain sync application it can be very interesting if
> it stays compatible and you don't need to add throttling to control the
> memory - often in the old style you throttle with the number of threads,
> now you need a semaphore or alike.
> Will not be a free lunch ;).
Agreed, free lunches never exist.
But.
If an application exists that never bothered to convert to using
non-blocking I/O but uses many threads, it could be very easily changed
(slightly) to use Virtual Threads and get a performance boost out of the
deal. Or at least be able to schedule many more tasks without running
out of stack space.
The price you pay for that is CPU usage. And most of us have CPU to spare.
-chris
>> Rainer,
>>
>> Thanks for writing this up.
>>
>> On 3/20/24 07:22, Rainer Jung wrote:
>>> I wanted to share an observation and I hope the things are correct how I
>>> am describing them. Maybe things have already improved and I am not
>>> aware of it, hints welcome.
>>>
>>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
>>> handle observability of virtual threads from inside the JVM and tooling.
>>>
>>> The final outcome is, that virtual threads are not included in the
>>> typical JVM APIs which one can use to observe threads, like enumerating
>>> them or accessing the stacks of individual threads. As a consequence,
>>> also jstack and "kill -QUIT" do not show virtual threads at all, not
>>> even when they are attached to a native thread and executing code.
>>
>> O_O
>>
>>> There is one single method to help with observability of virtual threads
>>>
>>>
>> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>>>
>>> which dumps a full thread dump to a file. Of course that is by no means
>>> appropriate, if you want to do a fine grained observation. At least you
>>> can choose to write a json structure instead of a hard to parse text
>>> format, but that's it.
>>
>> Boy, that's a real miss from the JVM team. I'm surprised that they have
>> overlooked this. I don't see a real reason that a Virtual Thread would
>> be treated any differently than a regular thread.
>>
>>> For instance I am often using a tool, that inspects our
>>> RequestProcessors to find long running requests and then retrieves the
>>> list of Java threads as ThreadInfo objects to find the one executing the
>>> request and finally retrieves the stack of that request from the JVM.
>>> Such an approach is no longer possible. Almost all of JMX does not show
>>> any info about virtual threads.
>>>
>>> It also seems Tomcat no longer uses a pool when a connector is
>>> configured to use virtual threads. So there are no metrics like
>>> currentThreadCount or currentThreadsBusy.
>>
>> When using virtual threads, the number of requests is the same as the
>> number of threads, since each request -> new Virtual Thread.... though I
>> think the request-count is only incremented when the request has
>> completed, so maybe there are threads running that can't be counted (yet).
>>
>> But I understand that you were hoping to get some information about
>> these threads and though maybe Tomcat's metrics could help. I guess not
>> really.
>>
>> But thread-busy count is the same as in-flight-request count. And
>> current thread count is the same as in-flight-request count as well.
>>
>>> I guess this also limits the use of some manager features and probably
>>> also the StuckThreadsDetectionValve when combined with virtual threads.
>>
>> Most likely. I'm fairly sure that uses the "usual Thread-walker things"
>> which you are reporting do not work any more with VTs.
>>
>> What JVM are you using for your investigations?
>>
>>> Of course Tomcat has solved most of the problems, that virtual threads
>>> want to solve, by doing the hard work of using the NIO and NIO2 APIs. So
>>> virtual threads are probably not that important for us. But we should be
>>> aware of the observability deficiencies whenever we would discuss
>>> whether we could switch from NIO or NIO2 to virtual threads to simplify
>>> connector code in some distant future.
>>
>> +1
>>
>> I've been enthusiastically talking with markt about dropping all that
>> nasty NIO stuff and "going back to BIO with virtual threads" which, at
>> this point, is mostly a threat and not a promise. But if VTs really
>> deliver everything they are claiming to deliver, then it may be possible
>> to go back to BIO as long as servlet-async is retired. And I'm not
>> holding my breath on that one.
>>
>> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Observability of virtual threads
Posted by Rainer Jung <ra...@kippdata.de>.
Am 20.03.24 um 18:50 schrieb Romain Manni-Bucau:
> Le mer. 20 mars 2024 à 18:43, Rainer Jung <ra...@kippdata.de> a
> écrit :
>
>> Am 20.03.24 um 18:34 schrieb Romain Manni-Bucau:
>>> Hi Chris,
>>>
>>> Thread dumps being dump of threads - literally os threads - and virtual
>>> threads not being threads at all - they are runnables in a dedicated
>> thread
>>> pool - it is quite fair to not make them the same and have their
>> scheduler
>>> - pool - only in the thread dump but not themselves no?
>>> Similarly to why threadlocal are not recommended for virtual thread this
>>> would probably make an useless pressure on the JVM IMHO - why there is an
>>> option to see it but it is mainly for debug purposes.
>>> See virtual threads as continuations (suspendable/resumable "Runnable")
>> in
>>> a dedicated and not programmatically configurable nor
>> selectable/proviable
>>> thread pool, they are not in thread dumps and this doesnt bother you ;) -
>>> ultimately if you want it you want all java objects to be monitored and
>> in
>>> the thread dump which would be weird - but I agree the semantic is
>>> misleading.
>>
>> Nevertheless it would be helpful to knw, which virtual threads currently
>> have an associated OS thread or not, and for those having one, what
>> their stack is. And this not just as a dump into a file, but instead
>> available via a proper API. It might be worth to ask the JDK devs about
>> chanced to get such an API.
>>
>
> Hmm, I must miss something but virtual threads with an associated os thread
> are in the thread dump since they run code (execution is in a normal
> forkjoinpool).
> So if you lock - synchronized/mutex for ex issue - then you see it as
> before.
Did you actually try? It's my whole point: these threads are not
contained in the usual thread dump outputs. Even when that virtual
thread is on CPU. You can write a simple long running CPU hog and run it
in a request. Every time you do a jstack or "kill -QUIT" this thread is
not visible, ie. not shown in the dump. Every time you run a "jcmd PID
Thread.dump_to_file FILE" the thread is part of the dump but without
infos about locks held.
Example of the new format:
#32 "http-nio-8880-virt-0" virtual
java.base/java.lang.StrictMath.atan(StrictMath.java:227)
java.base/java.lang.Math.atan(Math.java:279)
org.apache.jsp.cpuhog_jsp._jspService(cpuhog_jsp.java:154)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
jakarta.servlet.http.HttpServlet.service(HttpServlet.java:716)
...
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
> That said for that they added a pinned system prop enabling to validate
> your app is VT friendly or not which is very helpful.
>
>
>>> From my tests virtual threads do their job but they stay slower than a
>>> proper reactive/async impl mainly due to the overhead they add
>> *everywhere*
>>> compare to reactive programming plus this single thread pool issue which
>>> adds a lot of contention when all the app/lot of tasks is/are done using
>> it
>>> - vs bulkhead pattern for ex.
>>> But if you come from a plain sync application it can be very interesting
>> if
>>> it stays compatible and you don't need to add throttling to control the
>>> memory - often in the old style you throttle with the number of threads,
>>> now you need a semaphore or alike.
>>> Will not be a free lunch ;).
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github <
>> https://github.com/rmannibucau> |
>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>>> <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>
>>>
>>>
>>> Le mer. 20 mars 2024 à 18:20, Christopher Schultz <
>>> chris@christopherschultz.net> a écrit :
>>>
>>>> Rainer,
>>>>
>>>> Thanks for writing this up.
>>>>
>>>> On 3/20/24 07:22, Rainer Jung wrote:
>>>>> I wanted to share an observation and I hope the things are correct how
>> I
>>>>> am describing them. Maybe things have already improved and I am not
>>>>> aware of it, hints welcome.
>>>>>
>>>>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
>>>>> handle observability of virtual threads from inside the JVM and
>> tooling.
>>>>>
>>>>> The final outcome is, that virtual threads are not included in the
>>>>> typical JVM APIs which one can use to observe threads, like enumerating
>>>>> them or accessing the stacks of individual threads. As a consequence,
>>>>> also jstack and "kill -QUIT" do not show virtual threads at all, not
>>>>> even when they are attached to a native thread and executing code.
>>>>
>>>> O_O
>>>>
>>>>> There is one single method to help with observability of virtual
>> threads
>>>>>
>>>>>
>>>>
>> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>>>>>
>>>>> which dumps a full thread dump to a file. Of course that is by no means
>>>>> appropriate, if you want to do a fine grained observation. At least you
>>>>> can choose to write a json structure instead of a hard to parse text
>>>>> format, but that's it.
>>>>
>>>> Boy, that's a real miss from the JVM team. I'm surprised that they have
>>>> overlooked this. I don't see a real reason that a Virtual Thread would
>>>> be treated any differently than a regular thread.
>>>>
>>>>> For instance I am often using a tool, that inspects our
>>>>> RequestProcessors to find long running requests and then retrieves the
>>>>> list of Java threads as ThreadInfo objects to find the one executing
>> the
>>>>> request and finally retrieves the stack of that request from the JVM.
>>>>> Such an approach is no longer possible. Almost all of JMX does not show
>>>>> any info about virtual threads.
>>>>>
>>>>> It also seems Tomcat no longer uses a pool when a connector is
>>>>> configured to use virtual threads. So there are no metrics like
>>>>> currentThreadCount or currentThreadsBusy.
>>>>
>>>> When using virtual threads, the number of requests is the same as the
>>>> number of threads, since each request -> new Virtual Thread.... though I
>>>> think the request-count is only incremented when the request has
>>>> completed, so maybe there are threads running that can't be counted
>> (yet).
>>>>
>>>> But I understand that you were hoping to get some information about
>>>> these threads and though maybe Tomcat's metrics could help. I guess not
>>>> really.
>>>>
>>>> But thread-busy count is the same as in-flight-request count. And
>>>> current thread count is the same as in-flight-request count as well.
>>>>
>>>>> I guess this also limits the use of some manager features and probably
>>>>> also the StuckThreadsDetectionValve when combined with virtual threads.
>>>>
>>>> Most likely. I'm fairly sure that uses the "usual Thread-walker things"
>>>> which you are reporting do not work any more with VTs.
>>>>
>>>> What JVM are you using for your investigations?
>>>>
>>>>> Of course Tomcat has solved most of the problems, that virtual threads
>>>>> want to solve, by doing the hard work of using the NIO and NIO2 APIs.
>> So
>>>>> virtual threads are probably not that important for us. But we should
>> be
>>>>> aware of the observability deficiencies whenever we would discuss
>>>>> whether we could switch from NIO or NIO2 to virtual threads to simplify
>>>>> connector code in some distant future.
>>>>
>>>> +1
>>>>
>>>> I've been enthusiastically talking with markt about dropping all that
>>>> nasty NIO stuff and "going back to BIO with virtual threads" which, at
>>>> this point, is mostly a threat and not a promise. But if VTs really
>>>> deliver everything they are claiming to deliver, then it may be possible
>>>> to go back to BIO as long as servlet-async is retired. And I'm not
>>>> holding my breath on that one.
>>>>
>>>> -chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Observability of virtual threads
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Le mer. 20 mars 2024 à 18:43, Rainer Jung <ra...@kippdata.de> a
écrit :
> Am 20.03.24 um 18:34 schrieb Romain Manni-Bucau:
> > Hi Chris,
> >
> > Thread dumps being dump of threads - literally os threads - and virtual
> > threads not being threads at all - they are runnables in a dedicated
> thread
> > pool - it is quite fair to not make them the same and have their
> scheduler
> > - pool - only in the thread dump but not themselves no?
> > Similarly to why threadlocal are not recommended for virtual thread this
> > would probably make an useless pressure on the JVM IMHO - why there is an
> > option to see it but it is mainly for debug purposes.
> > See virtual threads as continuations (suspendable/resumable "Runnable")
> in
> > a dedicated and not programmatically configurable nor
> selectable/proviable
> > thread pool, they are not in thread dumps and this doesnt bother you ;) -
> > ultimately if you want it you want all java objects to be monitored and
> in
> > the thread dump which would be weird - but I agree the semantic is
> > misleading.
>
> Nevertheless it would be helpful to knw, which virtual threads currently
> have an associated OS thread or not, and for those having one, what
> their stack is. And this not just as a dump into a file, but instead
> available via a proper API. It might be worth to ask the JDK devs about
> chanced to get such an API.
>
Hmm, I must miss something but virtual threads with an associated os thread
are in the thread dump since they run code (execution is in a normal
forkjoinpool).
So if you lock - synchronized/mutex for ex issue - then you see it as
before.
That said for that they added a pinned system prop enabling to validate
your app is VT friendly or not which is very helpful.
> > From my tests virtual threads do their job but they stay slower than a
> > proper reactive/async impl mainly due to the overhead they add
> *everywhere*
> > compare to reactive programming plus this single thread pool issue which
> > adds a lot of contention when all the app/lot of tasks is/are done using
> it
> > - vs bulkhead pattern for ex.
> > But if you come from a plain sync application it can be very interesting
> if
> > it stays compatible and you don't need to add throttling to control the
> > memory - often in the old style you throttle with the number of threads,
> > now you need a semaphore or alike.
> > Will not be a free lunch ;).
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >
> >
> > Le mer. 20 mars 2024 à 18:20, Christopher Schultz <
> > chris@christopherschultz.net> a écrit :
> >
> >> Rainer,
> >>
> >> Thanks for writing this up.
> >>
> >> On 3/20/24 07:22, Rainer Jung wrote:
> >>> I wanted to share an observation and I hope the things are correct how
> I
> >>> am describing them. Maybe things have already improved and I am not
> >>> aware of it, hints welcome.
> >>>
> >>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
> >>> handle observability of virtual threads from inside the JVM and
> tooling.
> >>>
> >>> The final outcome is, that virtual threads are not included in the
> >>> typical JVM APIs which one can use to observe threads, like enumerating
> >>> them or accessing the stacks of individual threads. As a consequence,
> >>> also jstack and "kill -QUIT" do not show virtual threads at all, not
> >>> even when they are attached to a native thread and executing code.
> >>
> >> O_O
> >>
> >>> There is one single method to help with observability of virtual
> threads
> >>>
> >>>
> >>
> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
> >>>
> >>> which dumps a full thread dump to a file. Of course that is by no means
> >>> appropriate, if you want to do a fine grained observation. At least you
> >>> can choose to write a json structure instead of a hard to parse text
> >>> format, but that's it.
> >>
> >> Boy, that's a real miss from the JVM team. I'm surprised that they have
> >> overlooked this. I don't see a real reason that a Virtual Thread would
> >> be treated any differently than a regular thread.
> >>
> >>> For instance I am often using a tool, that inspects our
> >>> RequestProcessors to find long running requests and then retrieves the
> >>> list of Java threads as ThreadInfo objects to find the one executing
> the
> >>> request and finally retrieves the stack of that request from the JVM.
> >>> Such an approach is no longer possible. Almost all of JMX does not show
> >>> any info about virtual threads.
> >>>
> >>> It also seems Tomcat no longer uses a pool when a connector is
> >>> configured to use virtual threads. So there are no metrics like
> >>> currentThreadCount or currentThreadsBusy.
> >>
> >> When using virtual threads, the number of requests is the same as the
> >> number of threads, since each request -> new Virtual Thread.... though I
> >> think the request-count is only incremented when the request has
> >> completed, so maybe there are threads running that can't be counted
> (yet).
> >>
> >> But I understand that you were hoping to get some information about
> >> these threads and though maybe Tomcat's metrics could help. I guess not
> >> really.
> >>
> >> But thread-busy count is the same as in-flight-request count. And
> >> current thread count is the same as in-flight-request count as well.
> >>
> >>> I guess this also limits the use of some manager features and probably
> >>> also the StuckThreadsDetectionValve when combined with virtual threads.
> >>
> >> Most likely. I'm fairly sure that uses the "usual Thread-walker things"
> >> which you are reporting do not work any more with VTs.
> >>
> >> What JVM are you using for your investigations?
> >>
> >>> Of course Tomcat has solved most of the problems, that virtual threads
> >>> want to solve, by doing the hard work of using the NIO and NIO2 APIs.
> So
> >>> virtual threads are probably not that important for us. But we should
> be
> >>> aware of the observability deficiencies whenever we would discuss
> >>> whether we could switch from NIO or NIO2 to virtual threads to simplify
> >>> connector code in some distant future.
> >>
> >> +1
> >>
> >> I've been enthusiastically talking with markt about dropping all that
> >> nasty NIO stuff and "going back to BIO with virtual threads" which, at
> >> this point, is mostly a threat and not a promise. But if VTs really
> >> deliver everything they are claiming to deliver, then it may be possible
> >> to go back to BIO as long as servlet-async is retired. And I'm not
> >> holding my breath on that one.
> >>
> >> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: Observability of virtual threads
Posted by Rainer Jung <ra...@kippdata.de>.
Am 20.03.24 um 18:34 schrieb Romain Manni-Bucau:
> Hi Chris,
>
> Thread dumps being dump of threads - literally os threads - and virtual
> threads not being threads at all - they are runnables in a dedicated thread
> pool - it is quite fair to not make them the same and have their scheduler
> - pool - only in the thread dump but not themselves no?
> Similarly to why threadlocal are not recommended for virtual thread this
> would probably make an useless pressure on the JVM IMHO - why there is an
> option to see it but it is mainly for debug purposes.
> See virtual threads as continuations (suspendable/resumable "Runnable") in
> a dedicated and not programmatically configurable nor selectable/proviable
> thread pool, they are not in thread dumps and this doesnt bother you ;) -
> ultimately if you want it you want all java objects to be monitored and in
> the thread dump which would be weird - but I agree the semantic is
> misleading.
Nevertheless it would be helpful to knw, which virtual threads currently
have an associated OS thread or not, and for those having one, what
their stack is. And this not just as a dump into a file, but instead
available via a proper API. It might be worth to ask the JDK devs about
chanced to get such an API.
> From my tests virtual threads do their job but they stay slower than a
> proper reactive/async impl mainly due to the overhead they add *everywhere*
> compare to reactive programming plus this single thread pool issue which
> adds a lot of contention when all the app/lot of tasks is/are done using it
> - vs bulkhead pattern for ex.
> But if you come from a plain sync application it can be very interesting if
> it stays compatible and you don't need to add throttling to control the
> memory - often in the old style you throttle with the number of threads,
> now you need a semaphore or alike.
> Will not be a free lunch ;).
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le mer. 20 mars 2024 à 18:20, Christopher Schultz <
> chris@christopherschultz.net> a écrit :
>
>> Rainer,
>>
>> Thanks for writing this up.
>>
>> On 3/20/24 07:22, Rainer Jung wrote:
>>> I wanted to share an observation and I hope the things are correct how I
>>> am describing them. Maybe things have already improved and I am not
>>> aware of it, hints welcome.
>>>
>>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
>>> handle observability of virtual threads from inside the JVM and tooling.
>>>
>>> The final outcome is, that virtual threads are not included in the
>>> typical JVM APIs which one can use to observe threads, like enumerating
>>> them or accessing the stacks of individual threads. As a consequence,
>>> also jstack and "kill -QUIT" do not show virtual threads at all, not
>>> even when they are attached to a native thread and executing code.
>>
>> O_O
>>
>>> There is one single method to help with observability of virtual threads
>>>
>>>
>> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>>>
>>> which dumps a full thread dump to a file. Of course that is by no means
>>> appropriate, if you want to do a fine grained observation. At least you
>>> can choose to write a json structure instead of a hard to parse text
>>> format, but that's it.
>>
>> Boy, that's a real miss from the JVM team. I'm surprised that they have
>> overlooked this. I don't see a real reason that a Virtual Thread would
>> be treated any differently than a regular thread.
>>
>>> For instance I am often using a tool, that inspects our
>>> RequestProcessors to find long running requests and then retrieves the
>>> list of Java threads as ThreadInfo objects to find the one executing the
>>> request and finally retrieves the stack of that request from the JVM.
>>> Such an approach is no longer possible. Almost all of JMX does not show
>>> any info about virtual threads.
>>>
>>> It also seems Tomcat no longer uses a pool when a connector is
>>> configured to use virtual threads. So there are no metrics like
>>> currentThreadCount or currentThreadsBusy.
>>
>> When using virtual threads, the number of requests is the same as the
>> number of threads, since each request -> new Virtual Thread.... though I
>> think the request-count is only incremented when the request has
>> completed, so maybe there are threads running that can't be counted (yet).
>>
>> But I understand that you were hoping to get some information about
>> these threads and though maybe Tomcat's metrics could help. I guess not
>> really.
>>
>> But thread-busy count is the same as in-flight-request count. And
>> current thread count is the same as in-flight-request count as well.
>>
>>> I guess this also limits the use of some manager features and probably
>>> also the StuckThreadsDetectionValve when combined with virtual threads.
>>
>> Most likely. I'm fairly sure that uses the "usual Thread-walker things"
>> which you are reporting do not work any more with VTs.
>>
>> What JVM are you using for your investigations?
>>
>>> Of course Tomcat has solved most of the problems, that virtual threads
>>> want to solve, by doing the hard work of using the NIO and NIO2 APIs. So
>>> virtual threads are probably not that important for us. But we should be
>>> aware of the observability deficiencies whenever we would discuss
>>> whether we could switch from NIO or NIO2 to virtual threads to simplify
>>> connector code in some distant future.
>>
>> +1
>>
>> I've been enthusiastically talking with markt about dropping all that
>> nasty NIO stuff and "going back to BIO with virtual threads" which, at
>> this point, is mostly a threat and not a promise. But if VTs really
>> deliver everything they are claiming to deliver, then it may be possible
>> to go back to BIO as long as servlet-async is retired. And I'm not
>> holding my breath on that one.
>>
>> -chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Observability of virtual threads
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi Chris,
Thread dumps being dump of threads - literally os threads - and virtual
threads not being threads at all - they are runnables in a dedicated thread
pool - it is quite fair to not make them the same and have their scheduler
- pool - only in the thread dump but not themselves no?
Similarly to why threadlocal are not recommended for virtual thread this
would probably make an useless pressure on the JVM IMHO - why there is an
option to see it but it is mainly for debug purposes.
See virtual threads as continuations (suspendable/resumable "Runnable") in
a dedicated and not programmatically configurable nor selectable/proviable
thread pool, they are not in thread dumps and this doesnt bother you ;) -
ultimately if you want it you want all java objects to be monitored and in
the thread dump which would be weird - but I agree the semantic is
misleading.
From my tests virtual threads do their job but they stay slower than a
proper reactive/async impl mainly due to the overhead they add *everywhere*
compare to reactive programming plus this single thread pool issue which
adds a lot of contention when all the app/lot of tasks is/are done using it
- vs bulkhead pattern for ex.
But if you come from a plain sync application it can be very interesting if
it stays compatible and you don't need to add throttling to control the
memory - often in the old style you throttle with the number of threads,
now you need a semaphore or alike.
Will not be a free lunch ;).
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le mer. 20 mars 2024 à 18:20, Christopher Schultz <
chris@christopherschultz.net> a écrit :
> Rainer,
>
> Thanks for writing this up.
>
> On 3/20/24 07:22, Rainer Jung wrote:
> > I wanted to share an observation and I hope the things are correct how I
> > am describing them. Maybe things have already improved and I am not
> > aware of it, hints welcome.
> >
> > Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
> > handle observability of virtual threads from inside the JVM and tooling.
> >
> > The final outcome is, that virtual threads are not included in the
> > typical JVM APIs which one can use to observe threads, like enumerating
> > them or accessing the stacks of individual threads. As a consequence,
> > also jstack and "kill -QUIT" do not show virtual threads at all, not
> > even when they are attached to a native thread and executing code.
>
> O_O
>
> > There is one single method to help with observability of virtual threads
> >
> >
> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
> >
> > which dumps a full thread dump to a file. Of course that is by no means
> > appropriate, if you want to do a fine grained observation. At least you
> > can choose to write a json structure instead of a hard to parse text
> > format, but that's it.
>
> Boy, that's a real miss from the JVM team. I'm surprised that they have
> overlooked this. I don't see a real reason that a Virtual Thread would
> be treated any differently than a regular thread.
>
> > For instance I am often using a tool, that inspects our
> > RequestProcessors to find long running requests and then retrieves the
> > list of Java threads as ThreadInfo objects to find the one executing the
> > request and finally retrieves the stack of that request from the JVM.
> > Such an approach is no longer possible. Almost all of JMX does not show
> > any info about virtual threads.
> >
> > It also seems Tomcat no longer uses a pool when a connector is
> > configured to use virtual threads. So there are no metrics like
> > currentThreadCount or currentThreadsBusy.
>
> When using virtual threads, the number of requests is the same as the
> number of threads, since each request -> new Virtual Thread.... though I
> think the request-count is only incremented when the request has
> completed, so maybe there are threads running that can't be counted (yet).
>
> But I understand that you were hoping to get some information about
> these threads and though maybe Tomcat's metrics could help. I guess not
> really.
>
> But thread-busy count is the same as in-flight-request count. And
> current thread count is the same as in-flight-request count as well.
>
> > I guess this also limits the use of some manager features and probably
> > also the StuckThreadsDetectionValve when combined with virtual threads.
>
> Most likely. I'm fairly sure that uses the "usual Thread-walker things"
> which you are reporting do not work any more with VTs.
>
> What JVM are you using for your investigations?
>
> > Of course Tomcat has solved most of the problems, that virtual threads
> > want to solve, by doing the hard work of using the NIO and NIO2 APIs. So
> > virtual threads are probably not that important for us. But we should be
> > aware of the observability deficiencies whenever we would discuss
> > whether we could switch from NIO or NIO2 to virtual threads to simplify
> > connector code in some distant future.
>
> +1
>
> I've been enthusiastically talking with markt about dropping all that
> nasty NIO stuff and "going back to BIO with virtual threads" which, at
> this point, is mostly a threat and not a promise. But if VTs really
> deliver everything they are claiming to deliver, then it may be possible
> to go back to BIO as long as servlet-async is retired. And I'm not
> holding my breath on that one.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: Observability of virtual threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Rainer,
On 3/20/24 13:33, Rainer Jung wrote:
> Am 20.03.24 um 18:17 schrieb Christopher Schultz:
>> Rainer,
>>
>> Thanks for writing this up.
>>
>> On 3/20/24 07:22, Rainer Jung wrote:
>>> I wanted to share an observation and I hope the things are correct
>>> how I am describing them. Maybe things have already improved and I am
>>> not aware of it, hints welcome.
>>>
>>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
>>> handle observability of virtual threads from inside the JVM and tooling.
>>>
>>> The final outcome is, that virtual threads are not included in the
>>> typical JVM APIs which one can use to observe threads, like
>>> enumerating them or accessing the stacks of individual threads. As a
>>> consequence, also jstack and "kill -QUIT" do not show virtual threads
>>> at all, not even when they are attached to a native thread and
>>> executing code.
>>
>> O_O
>>
>>> There is one single method to help with observability of virtual threads
>>>
>>> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>>>
>>> which dumps a full thread dump to a file. Of course that is by no
>>> means appropriate, if you want to do a fine grained observation. At
>>> least you can choose to write a json structure instead of a hard to
>>> parse text format, but that's it.
>>
>> Boy, that's a real miss from the JVM team. I'm surprised that they
>> have overlooked this. I don't see a real reason that a Virtual Thread
>> would be treated any differently than a regular thread.
>
> The JEP hints at "since thre could be millions of virtual threads it
> makes no sense to support them in the existing JMX APIs". I don't by
> this extremely simplistic reasoning.
:/
> Furthermore, the "dump to file" method they provided does not contain
> any locking information you usually expect in a thread dump.
+1
>>> For instance I am often using a tool, that inspects our
>>> RequestProcessors to find long running requests and then retrieves
>>> the list of Java threads as ThreadInfo objects to find the one
>>> executing the request and finally retrieves the stack of that request
>>> from the JVM. Such an approach is no longer possible. Almost all of
>>> JMX does not show any info about virtual threads.
>>>
>>> It also seems Tomcat no longer uses a pool when a connector is
>>> configured to use virtual threads. So there are no metrics like
>>> currentThreadCount or currentThreadsBusy.
>>
>> When using virtual threads, the number of requests is the same as the
>> number of threads, since each request -> new Virtual Thread.... though
>> I think the request-count is only incremented when the request has
>> completed, so maybe there are threads running that can't be counted
>> (yet).
>>
>> But I understand that you were hoping to get some information about
>> these threads and though maybe Tomcat's metrics could help. I guess
>> not really.
>>
>> But thread-busy count is the same as in-flight-request count. And
>> current thread count is the same as in-flight-request count as well.
>>
>>> I guess this also limits the use of some manager features and
>>> probably also the StuckThreadsDetectionValve when combined with
>>> virtual threads.
>>
>> Most likely. I'm fairly sure that uses the "usual Thread-walker
>> things" which you are reporting do not work any more with VTs.
>>
>> What JVM are you using for your investigations?
>
> Zulu 21, but I expect the other OpenJDK variants to behave the same way.
> It mostly comes out of the JEP.
>
>>> Of course Tomcat has solved most of the problems, that virtual
>>> threads want to solve, by doing the hard work of using the NIO and
>>> NIO2 APIs. So virtual threads are probably not that important for us.
>>> But we should be aware of the observability deficiencies whenever we
>>> would discuss whether we could switch from NIO or NIO2 to virtual
>>> threads to simplify connector code in some distant future.
>>
>> +1
>>
>> I've been enthusiastically talking with markt about dropping all that
>> nasty NIO stuff and "going back to BIO with virtual threads" which, at
>> this point, is mostly a threat and not a promise. But if VTs really
>> deliver everything they are claiming to deliver, then it may be
>> possible to go back to BIO as long as servlet-async is retired. And
>> I'm not holding my breath on that one.
>
> There was an interesting talk at FOSDEM about the current limitations of
> the virtual thread implementation. They have several situations where
> they switch over to blocking. I think for instance file I/O and also
> some types of locks they can not yet handle well.
Yes, these are known issues. Specifically, synchronized blocks and
virtual threads don't play well together, which is why Tomcat recently
converted many synchronized { ... } to use ReentrantLocks, which DO work
with (today's) VTs.
> They have some improvements in the pipeline, but I think one would at
> least wait for those. And hopefully they improve the observability in
> the meantime.
+100
> I wish I had known this type of problem at FOSDEM so I could have
> directly talked to the OpenJDK devs there.
There's always next year :)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Observability of virtual threads
Posted by Rainer Jung <ra...@kippdata.de>.
Am 20.03.24 um 18:17 schrieb Christopher Schultz:
> Rainer,
>
> Thanks for writing this up.
>
> On 3/20/24 07:22, Rainer Jung wrote:
>> I wanted to share an observation and I hope the things are correct how
>> I am describing them. Maybe things have already improved and I am not
>> aware of it, hints welcome.
>>
>> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
>> handle observability of virtual threads from inside the JVM and tooling.
>>
>> The final outcome is, that virtual threads are not included in the
>> typical JVM APIs which one can use to observe threads, like
>> enumerating them or accessing the stacks of individual threads. As a
>> consequence, also jstack and "kill -QUIT" do not show virtual threads
>> at all, not even when they are attached to a native thread and
>> executing code.
>
> O_O
>
>> There is one single method to help with observability of virtual threads
>>
>> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>>
>> which dumps a full thread dump to a file. Of course that is by no
>> means appropriate, if you want to do a fine grained observation. At
>> least you can choose to write a json structure instead of a hard to
>> parse text format, but that's it.
>
> Boy, that's a real miss from the JVM team. I'm surprised that they have
> overlooked this. I don't see a real reason that a Virtual Thread would
> be treated any differently than a regular thread.
The JEP hints at "since thre could be millions of virtual threads it
makes no sense to support them in the existing JMX APIs". I don't by
this extremely simplistic reasoning.
Furthermore, the "dump to file" method they provided does not contain
any locking information you usually expect in a thread dump.
>> For instance I am often using a tool, that inspects our
>> RequestProcessors to find long running requests and then retrieves the
>> list of Java threads as ThreadInfo objects to find the one executing
>> the request and finally retrieves the stack of that request from the
>> JVM. Such an approach is no longer possible. Almost all of JMX does
>> not show any info about virtual threads.
>>
>> It also seems Tomcat no longer uses a pool when a connector is
>> configured to use virtual threads. So there are no metrics like
>> currentThreadCount or currentThreadsBusy.
>
> When using virtual threads, the number of requests is the same as the
> number of threads, since each request -> new Virtual Thread.... though I
> think the request-count is only incremented when the request has
> completed, so maybe there are threads running that can't be counted (yet).
>
> But I understand that you were hoping to get some information about
> these threads and though maybe Tomcat's metrics could help. I guess not
> really.
>
> But thread-busy count is the same as in-flight-request count. And
> current thread count is the same as in-flight-request count as well.
>
>> I guess this also limits the use of some manager features and probably
>> also the StuckThreadsDetectionValve when combined with virtual threads.
>
> Most likely. I'm fairly sure that uses the "usual Thread-walker things"
> which you are reporting do not work any more with VTs.
>
> What JVM are you using for your investigations?
Zulu 21, but I expect the other OpenJDK variants to behave the same way.
It mostly comes out of the JEP.
>> Of course Tomcat has solved most of the problems, that virtual threads
>> want to solve, by doing the hard work of using the NIO and NIO2 APIs.
>> So virtual threads are probably not that important for us. But we
>> should be aware of the observability deficiencies whenever we would
>> discuss whether we could switch from NIO or NIO2 to virtual threads to
>> simplify connector code in some distant future.
>
> +1
>
> I've been enthusiastically talking with markt about dropping all that
> nasty NIO stuff and "going back to BIO with virtual threads" which, at
> this point, is mostly a threat and not a promise. But if VTs really
> deliver everything they are claiming to deliver, then it may be possible
> to go back to BIO as long as servlet-async is retired. And I'm not
> holding my breath on that one.
There was an interesting talk at FOSDEM about the current limitations of
the virtual thread implementation. They have several situations where
they switch over to blocking. I think for instance file I/O and also
some types of locks they can not yet handle well.
They have some improvements in the pipeline, but I think one would at
least wait for those. And hopefully they improve the observability in
the meantime.
I wish I had known this type of problem at FOSDEM so I could have
directly talked to the OpenJDK devs there.
Thanks and regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Observability of virtual threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Rainer,
Thanks for writing this up.
On 3/20/24 07:22, Rainer Jung wrote:
> I wanted to share an observation and I hope the things are correct how I
> am describing them. Maybe things have already improved and I am not
> aware of it, hints welcome.
>
> Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
> handle observability of virtual threads from inside the JVM and tooling.
>
> The final outcome is, that virtual threads are not included in the
> typical JVM APIs which one can use to observe threads, like enumerating
> them or accessing the stacks of individual threads. As a consequence,
> also jstack and "kill -QUIT" do not show virtual threads at all, not
> even when they are attached to a native thread and executing code.
O_O
> There is one single method to help with observability of virtual threads
>
> https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
>
> which dumps a full thread dump to a file. Of course that is by no means
> appropriate, if you want to do a fine grained observation. At least you
> can choose to write a json structure instead of a hard to parse text
> format, but that's it.
Boy, that's a real miss from the JVM team. I'm surprised that they have
overlooked this. I don't see a real reason that a Virtual Thread would
be treated any differently than a regular thread.
> For instance I am often using a tool, that inspects our
> RequestProcessors to find long running requests and then retrieves the
> list of Java threads as ThreadInfo objects to find the one executing the
> request and finally retrieves the stack of that request from the JVM.
> Such an approach is no longer possible. Almost all of JMX does not show
> any info about virtual threads.
>
> It also seems Tomcat no longer uses a pool when a connector is
> configured to use virtual threads. So there are no metrics like
> currentThreadCount or currentThreadsBusy.
When using virtual threads, the number of requests is the same as the
number of threads, since each request -> new Virtual Thread.... though I
think the request-count is only incremented when the request has
completed, so maybe there are threads running that can't be counted (yet).
But I understand that you were hoping to get some information about
these threads and though maybe Tomcat's metrics could help. I guess not
really.
But thread-busy count is the same as in-flight-request count. And
current thread count is the same as in-flight-request count as well.
> I guess this also limits the use of some manager features and probably
> also the StuckThreadsDetectionValve when combined with virtual threads.
Most likely. I'm fairly sure that uses the "usual Thread-walker things"
which you are reporting do not work any more with VTs.
What JVM are you using for your investigations?
> Of course Tomcat has solved most of the problems, that virtual threads
> want to solve, by doing the hard work of using the NIO and NIO2 APIs. So
> virtual threads are probably not that important for us. But we should be
> aware of the observability deficiencies whenever we would discuss
> whether we could switch from NIO or NIO2 to virtual threads to simplify
> connector code in some distant future.
+1
I've been enthusiastically talking with markt about dropping all that
nasty NIO stuff and "going back to BIO with virtual threads" which, at
this point, is mostly a threat and not a promise. But if VTs really
deliver everything they are claiming to deliver, then it may be possible
to go back to BIO as long as servlet-async is retired. And I'm not
holding my breath on that one.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org