[jira] [Comment Edited] (HDDS-16121) Use Jetty VirtualThreadPool for S3G

["Ivan Andika (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103576#comment-18103576 ] 

Ivan Andika edited comment on HDDS-16121 at 8/11/26 5:49 AM:
-------------------------------------------------------------

[~weichiu] Thanks for checking this.

> IMO, Jetty 12 + virtual threads is a long term roadmap, not immediately actionable

> Also note: Jetty 12 requires JDK17. But Ozone is built with JDK8 now. We need community consensus to move up to JDK17.

There is already a plan to make JDK17 for server component (HDDS-14439) including S3G, we can then upgrade to Jetty to 12+ version (HDDS-8280) cc: [~adoroszlai] [~smeng] 

> Servlet 3.1 Asynchronous IO is fine, but we don't have expertise in this framework and not as popular as Jetty

Agreed, that's why Servlet 3.1 Asynchronous IO is not the way to go since I don't think the community has any experience + asynchronous programming is hard. We can start with virtual threads first and evaluate asynchronous IO if it's still not enough. 

I think virtual threads let us keep our blocking implementation but enjoying the asynchronous IO beneftis, that's the reason I favored it. The Servlet 3.1 Asynchronous IO seems to only make sense when virtual threads is not supported.

> Let's say we upgrade Jetty 9 to Jetty12. Does using Virtual Threads require building Ozone with JDK21+? Or does it only require JDK21+ runtime? (assuming Jetty uses Virtual Threads and no direct invocation of Virtual Threads API from Ozone)

The hope is whether we can detect it in runtime (e.g. using classloader or reflection), similar to how Netty detect whether a particular runtime can use Epoll or not. So JDK17 runtime will fallback to the non-virtual threads path. If we can instantiate VirtualThreadPool then we use that (provided virtual thread config is enabled), otherwise we use the previous QueuedThreadPool.

You can refer to [https://jetty.org/docs/jetty/12.1/programming-guide/arch/threads.html] for the Jetty Virtual Threads support.

Anyway, we might do this upgrade internally first. This issue has been causing issues in our cluster for a long time.


was (Author: JIRAUSER298977):
[~weichiu] Thanks for checking this.

> IMO, Jetty 12 + virtual threads is a long term roadmap, not immediately actionable

> Also note: Jetty 12 requires JDK17. But Ozone is built with JDK8 now. We need community consensus to move up to JDK17.

There is already a plan to make JDK17 for server component (HDDS-14439) including S3G, we can then upgrade to Jetty to 12+ version (HDDS-8280) cc: [~adoroszlai] [~smeng] 

> Servlet 3.1 Asynchronous IO is fine, but we don't have expertise in this framework and not as popular as Jetty

Agreed, that's why Servlet 3.1 Asynchronous IO is not the way to go since I don't think the community has any experience. We can start with virtual threads first and evaluate asynchronous IO if it's still not enough. 

I think virtual threads let us keep our blocking implementation but enjoying the asynchronous IO beneftis, that's the reason I favored it. The Servlet 3.1 Asynchronous IO seems to only make sense when virtual threads is not supported.

> Let's say we upgrade Jetty 9 to Jetty12. Does using Virtual Threads require building Ozone with JDK21+? Or does it only require JDK21+ runtime? (assuming Jetty uses Virtual Threads and no direct invocation of Virtual Threads API from Ozone)

The hope is whether we can detect it in runtime (e.g. using classloader or reflection), similar to how Netty detect whether a particular runtime can use Epoll or not. So JDK17 runtime will fallback to the non-virtual threads path. If we can instantiate VirtualThreadPool then we use that (provided virtual thread config is enabled), otherwise we use the previous QueuedThreadPool.

You can refer to [https://jetty.org/docs/jetty/12.1/programming-guide/arch/threads.html] for the Jetty Virtual Threads support.

Anyway, we might do this upgrade internally first. This issue has been causing issues in our cluster for a long time.

> Use Jetty VirtualThreadPool for S3G
> -----------------------------------
>
>                 Key: HDDS-16121
>                 URL: https://issues.apache.org/jira/browse/HDDS-16121
>             Project: Apache Ozone
>          Issue Type: Improvement
>            Reporter: Ivan Andika
>            Assignee: Ivan Andika
>            Priority: Major
>
> Currently, S3G use the blocking thread-per-request threading model. This has the advantage of being simple to implement (as opposed to the asynchronous IO event loop model). 
> However, we encountered performance issues where S3G HTTP threads are blocked which causes the threads to hit hadoop.http.max.threads and therefore causes the subsequent requests to be waiting in the waitingTaskCount which increases the latenncy. This issue can happen even if the downstream services are fine (i.e. Ozone cluster is not slow or stuck).
> For example, if an HTTP client stalled while downloading a key from S3G (e.g. client machine is slow, etc), the S3G thread will be blocked serving this read request until hadoop.http.idle_timeout.ms which triggers the "Idle timeout expired". Check the Jetty IdleTimeout.java for the logic for idle, the idea is if no data is written in the socket for a while (e.g. notIdle is not invoked), a scheduled task will fail this request to prevent request hogging the thread for too long. 
> {code:java}
> SEVERE: An I/O error has occurred while writing a response message entity to the container output stream.
> org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 60000/60000 ms
> 	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
> 	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
> 	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
> 	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:649)
> 	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
> 	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370) {code}
> Meaning that a single request can block a single thread although there are still a lot of CPU capacity left.
> One idea is to migrate the Jetty threading model to Servlet 3.1 Asynchronous IO which implements WriteListener and ReadListener, this would prevent a single request holding the thread for too long. However, this requires changes in the S3G implementation which is tedious and risky.
> A better solution is to use virtual threads since we already support JDK21+ version. Since virtual threads are not real threads, a single OS thread can be mapped to multiple virtual threads and blocked virtual threads should not cause OS thread to be blocked since OS thread will pick another virtual thread which can do useful work. Additionally, since the threading model of virtual thread is similar to OS thread, we don't need to change threading model so the risk is lower. Of course, there are some risks in virtual threads (i.e. thread pinning, overloading the downstream services, etc), so we should still evaluate it or we can make a flag to whether enable or disable virtual threads and do canary testing.
> This requires Jetty upgrade, but Jetty upgrade is long overdue (we are still using a very old 9.x Jetty version) and we should do it asap. See [https://jetty.org/docs/jetty/12.1/programming-guide/arch/threads.html] for the Jetty virtual thread support.
> Note: I would like to hear from people who have worked with virtual threads before.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org