You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@resilientdb.apache.org by Shaokang Xie <sh...@apache.org> on 2026/07/30 01:34:33 UTC

Weekly Community Meeting Notes – Wednesday, July 29 at 4 PM PDT

Hi everyone,

Here’s a brief summary of our weekly *Apache ResilientDB (Incubating)*
community meeting held on *July 29, 2026* for those who couldn’t attend:

Quick recap
The meeting focused on debugging and scaling the Raft implementation in
ResilientDB, especially the client-side bottlenecks affecting throughput
and latency measurements. Josh described experiments scaling the maximum
in-flight transactions up to a million, but observed that throughput and
latency plateaued around 460,000 transactions per second and 0.1–0.2 ms,
suggesting the client was not generating enough load. He discussed
confusion about the run performance scripts, the role of the control
machine, and the KVServiceTools component, which Shaokang could not fully
explain. Josh detailed his attempts to increase producer and consumer
threads in the Performance Manager, which sometimes increased in-flight
transactions but also reduced throughput and introduced latency spikes.
Shaokang suggested using Linux profiling tools, adding logging, and
possibly controlling client speed more directly, rather than relying only
on the max in-flight parameter. They reviewed graphs showing throughput and
latency, noted some outlier results, and discussed whether the current
setup had truly reached Raft’s saturation point. The conversation also
touched on hardware specs, the number of clients and replicas, and the need
to understand both client and server bottlenecks better.

Next steps

   - Investigate and clarify the role of KVServiceTools in the run
   performance. Specifically, what it does and why it generates network
   traffic.
   - Experiment with increasing the number of producer and consumer threads
   in the Performance Manager to scale up the client throughput.
   - Investigate changing the client batch number and TCP batch number to
   see if it improves throughput.
   - Double-check the configuration files to ensure the correct
   max-in-flight-transactions parameter is being used.
   - Add a speed limiter to the client to control the transaction
   generation rate.
   - Perform profiling (using Linux tools like `perf`) on both the client
   and Raft replicas to identify bottlenecks.
   - Add logging to the client to measure and output the transaction
   generation speed.



Summary
Raft Implementation Performance Testing
The team discussed the Raft implementation and performance testing, with
Josh reporting on his efforts to scale the maximum number of in-flight
transactions. Josh clarified questions about the performance testing
scripts and discovered that increasing the number of consumer threads
reduced throughput from 460,000 to 70,000 transactions per second. The
discussion included some confusion about the role of the control machine in
the testing process, which Shaokang agreed to investigate further offline.

Performance Testing and Bottleneck Analysis
Josh discussed his throughput and latency testing, where he observed a peak
throughput of 460,000 transactions after 2,000 transactions. Still, when he
increased the in-flight limit to 1 million, the throughput decreased to
70,000 while latency increased. The team discussed whether the bottleneck
might be at the client level rather than the protocol itself. Josh agreed
to investigate further by running additional scaling experiments and
examining graphs of the behavior to identify any patterns or trends.

Client Performance and Raft Issues
The team discussed a client issue where Josh was experiencing performance
problems when trying to add more clients to powerful machines. Josh
explained that increasing thread counts and producers didn't resolve the
issue, as he was hitting queue limitations and encountering lock contention
in the Raft protocol operations. Mohammad suggested Josh conduct profiling
on both the client and raft replicas to identify bottlenecks, and Shaokang
recommended using Linux tools like perf for profiling and adding logs for
debugging.

System Performance Optimization Discussion
Josh and Shaokang discussed performance issues with transaction throughput
and latency in their system. They analyzed why throughput and latency did
not increase as expected when scaling up the maximum number of in-flight
transactions. Shaokang suggested that client speed might be the limiting
factor rather than the Raft implementation, and recommended adjusting
client speed to test the system's saturation point better. They also
discussed potential solutions, including increasing producer and consumer
threads, adjusting the TCP batch number, and adding a client speed
controller. Josh agreed to explore these options further and double-check
the configuration parameters to address the performance issues.

Best,
Shaokang Xie
PPMC Apache ResilientDB (incubating)