You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "eyalbe12 (via GitHub)" <gi...@apache.org> on 2023/06/05 10:39:54 UTC
[GitHub] [arrow] eyalbe12 opened a new issue, #35910: arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
eyalbe12 opened a new issue, #35910:
URL: https://github.com/apache/arrow/issues/35910
### Describe the enhancement requested
Hi,
Not sure how to categorize this, as this is not really a bug, so I put it under the enhancement category.
When I moved from using ArrowFlight v8 to ArrowFlight v12 I noticed a degradation in performance. In order to verify it, I tried the arrow-flight-benchmark in localhost mode. I ran the following on Ubuntu 20.04 (8 cpu's and 32GB ram):
./release/arrow-flight-perf-server -port 31337 -transport grpc &
./release/arrow-flight-benchmark --server_host=localhost --server_port=31337 --records_per_stream=163333333 --num_streams=32 --num_threads=32 --records_per_batch=16384 --num_perf_runs=5
The V12 benchmark provided speed result which is ~10-15% less compared to V8 speed.
I traced the degradation directly to the following commit:
commit d89c0b4aadecd09dbeb8051688f347c9d0af89ad (HEAD) ARROW-16729: [C++] Bump Abseil/gRPC dependency versions (#13315)
And more specifically, the upgrade of gRPC from v1.35 to v1.46.3 in arrow/cpp/thirdparty/versions.txt .
-ARROW_GRPC_BUILD_VERSION=v1.35.0
-ARROW_GRPC_BUILD_SHA256_CHECKSUM=27dd2fc5c9809ddcde8eb6fa1fa278a3486566dfc28335fca13eb8df8bd3b958
+ARROW_GRPC_BUILD_VERSION=v1.46.3
+ARROW_GRPC_BUILD_SHA256_CHECKSUM=d6cbf22cb5007af71b61c6be316a79397469c58c82a942552a62e708bce60964
Not sure how important this is to you, but I thought you should be aware of this. Do you have any clue why gRPC v1.46.3 causes such speed degradation in ArrowFlight?
Thanks for your hard work.
### Component(s)
Benchmarking, C++, FlightRPC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #35910: arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #35910:
URL: https://github.com/apache/arrow/issues/35910#issuecomment-1576674369
Possibly, 1.46 had an unlisted changelog item that removed what had been the default polling engine, [epollex](https://github.com/grpc/grpc/commit/766cd6b5736200a2d76abb5f0b3cf2357733d752), which [has been observed to impact other benchmarks](https://github.com/grpc/grpc/pull/29160). (It's not quite clear when epoll1 became default; it appears to have been [1.45](https://github.com/grpc/grpc/pull/29239) or 1.46.)
We're not really in control here; pinning to an old gRPC version isn't desirable either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on issue #35910: [C++][FlightRPC] arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #35910:
URL: https://github.com/apache/arrow/issues/35910#issuecomment-1669954513
The reason for the degraded performance is due to grpc update and there isn't really anything we can do unless that is fiex up stream (which seems unlikely looking at the PR), so I am going to close the issue. Feel free to reopen if there is anything I have missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #35910: [C++][FlightRPC] arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #35910:
URL: https://github.com/apache/arrow/issues/35910#issuecomment-1577605725
At the same time, 10% in the synthetic benchmark may or may not translate to real-world differences?
For two processes on the same machine, my hope is eventually the UCX backend can become production-ready (it's already much faster than gRPC for this case when using shared memory)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #35910: [C++][FlightRPC] arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #35910:
URL: https://github.com/apache/arrow/issues/35910#issuecomment-1577493639
Hmm, at least for https://github.com/apache/arrow/pull/35679 I observed that FlightDataDeserialize was still being called. (If it's not called, Flight will break entirely as there isn't really a fallback path.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35910: [C++][FlightRPC] arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35910:
URL: https://github.com/apache/arrow/issues/35910#issuecomment-1577466288
Our customization code https://github.com/apache/arrow/blob/main/cpp/src/arrow/flight/transport/grpc/customize_grpc.h for zero-copy may not be used with gRPC 1.46.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35910: [C++][FlightRPC] arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35910:
URL: https://github.com/apache/arrow/issues/35910#issuecomment-1577515030
Then my assumption is wrong. Sorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser closed issue #35910: [C++][FlightRPC] arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser closed issue #35910: [C++][FlightRPC] arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35
URL: https://github.com/apache/arrow/issues/35910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org