You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "cwang9208 (via GitHub)" <gi...@apache.org> on 2023/03/07 06:23:19 UTC
[GitHub] [arrow] cwang9208 opened a new issue, #34478: Flight RPC throughput drops when transfer large messages
cwang9208 opened a new issue, #34478:
URL: https://github.com/apache/arrow/issues/34478
### Describe the usage question you have. Please include as many useful details as possible.
I'm benchmarking the performance of flight RPC using the built-in C++ benchmark server/client. Both the client and server run on the same machine.
Below is the result for records_per_batch = 1000000.
```
$ ./arrow-flight-benchmark -records_per_stream 1000000000 -records_per_batch 1000000 -num_perf_runs 2 -num_streams 10 -num_threads 10
Testing method: DoGet
Using spawned TCP server
Server running with pid 3207456
Server host: localhost
Server port: 31337
Number of perf runs: 2
Number of concurrent gets/puts: 10
Batch size: 32000000
Batches read: 20000
Bytes read: 640000000000
Nanos: 55198332155
Speed: 11057.4 MB/s
Throughput: 362.33 batches/s
Latency mean: 26931 us
Latency quantile=0.5: 26270 us
Latency quantile=0.95: 41160 us
Latency quantile=0.99: 51025 us
Latency max: 159917 us
```
And the result for records_per_batch = 5000000.
```
$ ./arrow-flight-benchmark -records_per_stream 1000000000 -records_per_batch 5000000 -num_perf_runs 2 -num_streams 10 -num_threads 10
Testing method: DoGet
Using spawned TCP server
Server running with pid 3206446
Server host: localhost
Server port: 31337
Number of perf runs: 2
Number of concurrent gets/puts: 10
Batch size: 160000000
Batches read: 4000
Bytes read: 640000000000
Nanos: 79217518482
Speed: 7704.75 MB/s
Throughput: 50.4939 batches/s
Latency mean: 184366 us
Latency quantile=0.5: 167930 us
Latency quantile=0.95: 309036 us
Latency quantile=0.99: 487428 us
Latency max: 744113 us
```
I'm wondering why the Speed drops when the records_per_batch grows bigger?
Can anyone help explain the reason behind this?
### Component(s)
Benchmarking, C++, FlightRPC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cwang9208 commented on issue #34478: [C++][FlightRPC] Flight RPC throughput drops when transfer large messages
Posted by "cwang9208 (via GitHub)" <gi...@apache.org>.
cwang9208 commented on issue #34478:
URL: https://github.com/apache/arrow/issues/34478#issuecomment-1457702117
If I increase the records_per_stream to make the example run long enough, the Speed of records_per_batch=5000000 will increase to Speed 10264.2 MB/s. Closing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cwang9208 closed issue #34478: [C++][FlightRPC] Flight RPC throughput drops when transfer large messages
Posted by "cwang9208 (via GitHub)" <gi...@apache.org>.
cwang9208 closed issue #34478: [C++][FlightRPC] Flight RPC throughput drops when transfer large messages
URL: https://github.com/apache/arrow/issues/34478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org