You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Dominic Garguilo <Do...@apache.org> on 2026/07/10 18:00:06 UTC

[DISCUSS] Exploring Java 21 virtual threads for BatchScanner

I was looking for places where Java 21 virtual threads might fit, since
they are intended for blocking / I/O-heavy work. BatchScanner seemed like a
candidate because it can issue many concurrent RPC-backed lookup tasks and
currently uses a platform-thread pool.

I tried a small local prototype that switches the BatchScanner query
executor to virtual threads:

if (Boolean.getBoolean("accumulo.batch.scanner.virtualThreads.enabled")) {
   queryThreadPool =
context.threadPools().createVirtualThreadExecutor(poolName);
} else {
   queryThreadPool
= context.threadPools().getPoolBuilder(poolName).numCoreThreads(numQueryThreads).build();
}

The prototype keeps the existing numQueryThreads behavior with a semaphore
around query task execution.

I tested this with an Accumulo JShell script against a local cluster with 4
tservers. These were immediate BatchScanner scans. The script creates a
100k-row table with 100 splits, builds 10k ranges, and runs 16 concurrent
BatchScanners with 128 scan threads. To reduce order effects, it runs full
blocks in this order: virtual, platform, virtual, platform.

Latest local summary:

platform: avg wall ~404 ms, avg throughput ~396k entries/sec, max peak
threads 1664
virtual:  avg wall ~374 ms, avg throughput ~526k entries/sec, max peak
threads 297

So the wall-time improvement in this more controlled run was modest
(~1.08x), but peak platform thread usage dropped substantially. Earlier
less-controlled runs showed larger wall-time wins, so I think the
performance result is workload/order sensitive.

One caveat: Accumulo currently requires JDK 21 to build, but still compiles
with Java 17 source/API compatibility, so my prototype uses reflection for
the virtual-thread APIs. If/when the source/API level moves to 21, that
code could be much simpler.

I’m curious what others think:
- Is BatchScanner worth exploring further as a virtual-thread use case?
- Is reducing client platform-thread pressure enough to justify continued
investigation?
- Are there other Accumulo code paths that would be good candidates for
virtual threads?

I can work on sprucing up the demo and push it to a branch if anyone is
interested in taking a look at the demo code.

Re: [DISCUSS] Exploring Java 21 virtual threads for BatchScanner

Posted by Christopher <ct...@apache.org>.
Other Java 21 features are improved pattern matching for switch
statements and records, to improve readability, and the unnamed
variable (_) to get rid of some warnings about intentionally unused
variables. These aren't really critical, but I like them and would use
them. Sequenced interfaces might make their way into our code as well.

I'm also kind of interested in the preview string templates feature
and seeing how well the ZGC works, and to see if the built-in logging
API might allow us to get rid of slf4j as the logging facade entirely.

Of course, the most important new Java 21 feature is that we can now
do: Character.isEmoji('🙃');

On Fri, Jul 10, 2026 at 3:55 PM Dave Marion <dm...@gmail.com> wrote:
>
> Regarding virtual threads, my understanding is that there are still
> significant issues in Java 21 and a good number of them are fixed in Java
> 25.
>
> Regarding changing main to use Java 21, I'd like to keep it at 17 for now
> for some testing. Ultimately, it will likely happen.
>
> I would be curious to know what other Java 21 features we can take
> advantage of, besides virtual threads, but I haven't looked at that
> specifically.
> The main reason for the Java 21 PR is that main depends on Hadoop 3.5.0,
> and the 3.5.0 release notes say that Java 21 is supported on the client
> side.
>
> On Fri, Jul 10, 2026 at 3:37 PM Christopher <ct...@apache.org> wrote:
>
> > I don't know much about virtual threads yet, but I know that there's
> > an issue if there's a blocking IO operation in a synchronized block.
> > There are workarounds using some of Java's concurrent classes
> > (ReentrantLocks), but I guess Java made some improvements in Java 24
> > that make this problem largely go away.
> >
> > I would be in favor of moving to Java 21 for the main (4.0) branch for
> > a runtime requirement. Dave has an open PR to do that. I've discussed
> > it with Dave. He can speak for himself, but my understanding is that
> > he wanted to hold off until we could do more testing. I'm okay with
> > holding off for a little while for testing, but I would like to move
> > to Java 21.
> >
> > Once we're on 21, I think the answer to all 3 question is: yes, but we
> > have to be careful to avoid the issues seen with synchronized blocks.
> > I don't know if there are other Java 21 problems with virtual threads
> > aside from this one to look out for also.
> >
> > On Fri, Jul 10, 2026 at 2:00 PM Dominic Garguilo <Do...@apache.org>
> > wrote:
> > >
> > > I was looking for places where Java 21 virtual threads might fit, since
> > > they are intended for blocking / I/O-heavy work. BatchScanner seemed
> > like a
> > > candidate because it can issue many concurrent RPC-backed lookup tasks
> > and
> > > currently uses a platform-thread pool.
> > >
> > > I tried a small local prototype that switches the BatchScanner query
> > > executor to virtual threads:
> > >
> > > if (Boolean.getBoolean("accumulo.batch.scanner.virtualThreads.enabled"))
> > {
> > >    queryThreadPool =
> > > context.threadPools().createVirtualThreadExecutor(poolName);
> > > } else {
> > >    queryThreadPool
> > > =
> > context.threadPools().getPoolBuilder(poolName).numCoreThreads(numQueryThreads).build();
> > > }
> > >
> > > The prototype keeps the existing numQueryThreads behavior with a
> > semaphore
> > > around query task execution.
> > >
> > > I tested this with an Accumulo JShell script against a local cluster
> > with 4
> > > tservers. These were immediate BatchScanner scans. The script creates a
> > > 100k-row table with 100 splits, builds 10k ranges, and runs 16 concurrent
> > > BatchScanners with 128 scan threads. To reduce order effects, it runs
> > full
> > > blocks in this order: virtual, platform, virtual, platform.
> > >
> > > Latest local summary:
> > >
> > > platform: avg wall ~404 ms, avg throughput ~396k entries/sec, max peak
> > > threads 1664
> > > virtual:  avg wall ~374 ms, avg throughput ~526k entries/sec, max peak
> > > threads 297
> > >
> > > So the wall-time improvement in this more controlled run was modest
> > > (~1.08x), but peak platform thread usage dropped substantially. Earlier
> > > less-controlled runs showed larger wall-time wins, so I think the
> > > performance result is workload/order sensitive.
> > >
> > > One caveat: Accumulo currently requires JDK 21 to build, but still
> > compiles
> > > with Java 17 source/API compatibility, so my prototype uses reflection
> > for
> > > the virtual-thread APIs. If/when the source/API level moves to 21, that
> > > code could be much simpler.
> > >
> > > I’m curious what others think:
> > > - Is BatchScanner worth exploring further as a virtual-thread use case?
> > > - Is reducing client platform-thread pressure enough to justify continued
> > > investigation?
> > > - Are there other Accumulo code paths that would be good candidates for
> > > virtual threads?
> > >
> > > I can work on sprucing up the demo and push it to a branch if anyone is
> > > interested in taking a look at the demo code.
> >

Re: [DISCUSS] Exploring Java 21 virtual threads for BatchScanner

Posted by Dave Marion <dm...@gmail.com>.
Regarding virtual threads, my understanding is that there are still
significant issues in Java 21 and a good number of them are fixed in Java
25.

Regarding changing main to use Java 21, I'd like to keep it at 17 for now
for some testing. Ultimately, it will likely happen.

I would be curious to know what other Java 21 features we can take
advantage of, besides virtual threads, but I haven't looked at that
specifically.
The main reason for the Java 21 PR is that main depends on Hadoop 3.5.0,
and the 3.5.0 release notes say that Java 21 is supported on the client
side.

On Fri, Jul 10, 2026 at 3:37 PM Christopher <ct...@apache.org> wrote:

> I don't know much about virtual threads yet, but I know that there's
> an issue if there's a blocking IO operation in a synchronized block.
> There are workarounds using some of Java's concurrent classes
> (ReentrantLocks), but I guess Java made some improvements in Java 24
> that make this problem largely go away.
>
> I would be in favor of moving to Java 21 for the main (4.0) branch for
> a runtime requirement. Dave has an open PR to do that. I've discussed
> it with Dave. He can speak for himself, but my understanding is that
> he wanted to hold off until we could do more testing. I'm okay with
> holding off for a little while for testing, but I would like to move
> to Java 21.
>
> Once we're on 21, I think the answer to all 3 question is: yes, but we
> have to be careful to avoid the issues seen with synchronized blocks.
> I don't know if there are other Java 21 problems with virtual threads
> aside from this one to look out for also.
>
> On Fri, Jul 10, 2026 at 2:00 PM Dominic Garguilo <Do...@apache.org>
> wrote:
> >
> > I was looking for places where Java 21 virtual threads might fit, since
> > they are intended for blocking / I/O-heavy work. BatchScanner seemed
> like a
> > candidate because it can issue many concurrent RPC-backed lookup tasks
> and
> > currently uses a platform-thread pool.
> >
> > I tried a small local prototype that switches the BatchScanner query
> > executor to virtual threads:
> >
> > if (Boolean.getBoolean("accumulo.batch.scanner.virtualThreads.enabled"))
> {
> >    queryThreadPool =
> > context.threadPools().createVirtualThreadExecutor(poolName);
> > } else {
> >    queryThreadPool
> > =
> context.threadPools().getPoolBuilder(poolName).numCoreThreads(numQueryThreads).build();
> > }
> >
> > The prototype keeps the existing numQueryThreads behavior with a
> semaphore
> > around query task execution.
> >
> > I tested this with an Accumulo JShell script against a local cluster
> with 4
> > tservers. These were immediate BatchScanner scans. The script creates a
> > 100k-row table with 100 splits, builds 10k ranges, and runs 16 concurrent
> > BatchScanners with 128 scan threads. To reduce order effects, it runs
> full
> > blocks in this order: virtual, platform, virtual, platform.
> >
> > Latest local summary:
> >
> > platform: avg wall ~404 ms, avg throughput ~396k entries/sec, max peak
> > threads 1664
> > virtual:  avg wall ~374 ms, avg throughput ~526k entries/sec, max peak
> > threads 297
> >
> > So the wall-time improvement in this more controlled run was modest
> > (~1.08x), but peak platform thread usage dropped substantially. Earlier
> > less-controlled runs showed larger wall-time wins, so I think the
> > performance result is workload/order sensitive.
> >
> > One caveat: Accumulo currently requires JDK 21 to build, but still
> compiles
> > with Java 17 source/API compatibility, so my prototype uses reflection
> for
> > the virtual-thread APIs. If/when the source/API level moves to 21, that
> > code could be much simpler.
> >
> > I’m curious what others think:
> > - Is BatchScanner worth exploring further as a virtual-thread use case?
> > - Is reducing client platform-thread pressure enough to justify continued
> > investigation?
> > - Are there other Accumulo code paths that would be good candidates for
> > virtual threads?
> >
> > I can work on sprucing up the demo and push it to a branch if anyone is
> > interested in taking a look at the demo code.
>

Re: [DISCUSS] Exploring Java 21 virtual threads for BatchScanner

Posted by Christopher <ct...@apache.org>.
I don't know much about virtual threads yet, but I know that there's
an issue if there's a blocking IO operation in a synchronized block.
There are workarounds using some of Java's concurrent classes
(ReentrantLocks), but I guess Java made some improvements in Java 24
that make this problem largely go away.

I would be in favor of moving to Java 21 for the main (4.0) branch for
a runtime requirement. Dave has an open PR to do that. I've discussed
it with Dave. He can speak for himself, but my understanding is that
he wanted to hold off until we could do more testing. I'm okay with
holding off for a little while for testing, but I would like to move
to Java 21.

Once we're on 21, I think the answer to all 3 question is: yes, but we
have to be careful to avoid the issues seen with synchronized blocks.
I don't know if there are other Java 21 problems with virtual threads
aside from this one to look out for also.

On Fri, Jul 10, 2026 at 2:00 PM Dominic Garguilo <Do...@apache.org> wrote:
>
> I was looking for places where Java 21 virtual threads might fit, since
> they are intended for blocking / I/O-heavy work. BatchScanner seemed like a
> candidate because it can issue many concurrent RPC-backed lookup tasks and
> currently uses a platform-thread pool.
>
> I tried a small local prototype that switches the BatchScanner query
> executor to virtual threads:
>
> if (Boolean.getBoolean("accumulo.batch.scanner.virtualThreads.enabled")) {
>    queryThreadPool =
> context.threadPools().createVirtualThreadExecutor(poolName);
> } else {
>    queryThreadPool
> = context.threadPools().getPoolBuilder(poolName).numCoreThreads(numQueryThreads).build();
> }
>
> The prototype keeps the existing numQueryThreads behavior with a semaphore
> around query task execution.
>
> I tested this with an Accumulo JShell script against a local cluster with 4
> tservers. These were immediate BatchScanner scans. The script creates a
> 100k-row table with 100 splits, builds 10k ranges, and runs 16 concurrent
> BatchScanners with 128 scan threads. To reduce order effects, it runs full
> blocks in this order: virtual, platform, virtual, platform.
>
> Latest local summary:
>
> platform: avg wall ~404 ms, avg throughput ~396k entries/sec, max peak
> threads 1664
> virtual:  avg wall ~374 ms, avg throughput ~526k entries/sec, max peak
> threads 297
>
> So the wall-time improvement in this more controlled run was modest
> (~1.08x), but peak platform thread usage dropped substantially. Earlier
> less-controlled runs showed larger wall-time wins, so I think the
> performance result is workload/order sensitive.
>
> One caveat: Accumulo currently requires JDK 21 to build, but still compiles
> with Java 17 source/API compatibility, so my prototype uses reflection for
> the virtual-thread APIs. If/when the source/API level moves to 21, that
> code could be much simpler.
>
> I’m curious what others think:
> - Is BatchScanner worth exploring further as a virtual-thread use case?
> - Is reducing client platform-thread pressure enough to justify continued
> investigation?
> - Are there other Accumulo code paths that would be good candidates for
> virtual threads?
>
> I can work on sprucing up the demo and push it to a branch if anyone is
> interested in taking a look at the demo code.