You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Carlo Mazzaferro (Jira)" <ji...@apache.org> on 2020/11/02 16:43:00 UTC
[jira] [Comment Edited] (ARROW-10460) [FlightRPC][Python] FlightRPC
authentication mechanism changed and is undocumented, breaking current
working code
[ https://issues.apache.org/jira/browse/ARROW-10460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224794#comment-17224794 ]
Carlo Mazzaferro edited comment on ARROW-10460 at 11/2/20, 4:42 PM:
--------------------------------------------------------------------
It's a python server running behind a Traefik ingress controller (using h2c server scheme), server runs on 5005. For some obscure reason, with version 2.0.0 I am able to fix the issue by adding port `443` to the location, which wasn't needed with version 1.0.1. FWIW, here's the trace log without the port added, v2.0.0
{{D1102 17:41:42.069247000 4477541824 ev_posix.cc:174] Using polling engine: poll}}
{{D1102 17:41:42.069323000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "grpclb"}}
{{D1102 17:41:42.069333000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "cds_experimental"}}
{{D1102 17:41:42.069338000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "eds_experimental"}}
{{D1102 17:41:42.069344000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "lrs_experimental"}}
{{D1102 17:41:42.069349000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "priority_experimental"}}
{{D1102 17:41:42.069353000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "weighted_target_experimental"}}
{{D1102 17:41:42.069360000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "xds_routing_experimental"}}
{{D1102 17:41:42.069366000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "pick_first"}}
{{D1102 17:41:42.069370000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "round_robin"}}
{{D1102 17:41:42.069470000 4477541824 dns_resolver_ares.cc:503] Using ares dns resolver}}
{{I1102 17:41:42.069753000 4477541824 call.cc:626] OP[client-channel:0x7fd4bd0e69c0]: SEND_INITIAL_METADATA{key=3a 70 61 74 68 ':path' value=2f 61 72 72 6f 77 2e 66 6c 69 67 68 74 2e 70 72 6f 74 6f 63 6f 6c 2e 46 6c 69 67 68 74 53 65 72 76 69 63 65 2f 48 61 6e 64 73 68 61 6b 65 '/arrow.flight.protocol.FlightService/Handshake'}}}
{{I1102 17:41:42.077093000 4477541824 subchannel.cc:1003] Connect failed: {"created":"@1604335302.077058000","description":"Can't assign requested address","errno":49,"file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/lib/iomgr/tcp_client_posix.cc","file_line":307,"os_error":"Can't assign requested address","syscall":"connect"}}}
{{I1102 17:41:42.077219000 4477541824 subchannel.cc:1003] Connect failed: {"created":"@1604335302.077197000","description":"Can't assign requested address","errno":49,"file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/lib/iomgr/tcp_client_posix.cc","file_line":307,"os_error":"Can't assign requested address","syscall":"connect"}}}
{{I1102 17:41:42.077320000 4477541824 subchannel.cc:1003] Connect failed: {"created":"@1604335302.077298000","description":"Can't assign requested address","errno":49,"file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/lib/iomgr/tcp_client_posix.cc","file_line":307,"os_error":"Can't assign requested address","syscall":"connect"}}}
{{I1102 17:41:42.077354000 4477541824 subchannel.cc:942] Subchannel 0x7fd4c2a108e0: Retry in 100 milliseconds}}
{{I1102 17:41:42.077434000 4477541824 call.cc:626] OP[client-channel:0x7fd4bd0e69c0]: CANCEL:{"created":"@1604335302.077349000","description":"Failed to pick subchannel","file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/ext/filters/client_channel/client_channel.cc","file_line":3962,"referenced_errors":[
{"created":"@1604335302.077347000","description":"failed to connect to all addresses","file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}
]}}}
{{I1102 17:41:42.077811000 4477541824 call.cc:626] OP[client-channel:0x7fd4bd0e69c0]: SEND_MESSAGE:flags=0x00000000:len=38}}
{{I1102 17:41:42.077839000 4477541824 call.cc:626] OP[client-channel:0x7fd4bd0e69c0]: RECV_INITIAL_METADATA RECV_TRAILING_METADATA}}
{{Traceback (most recent call last):}}
\{{ File "scratch.py", line 32, in <module>}}
\{{ ApiKeyClientAuthHandler(api_key="xxxxxxxxxxxxxxxx"),}}
\{{ File "pyarrow/_flight.pyx", line 1149, in pyarrow._flight.FlightClient.authenticate}}
\{{ File "pyarrow/_flight.pyx", line 70, in pyarrow._flight.check_flight_status}}
{{pyarrow._flight.FlightUnavailableError: gRPC returned unavailable error, with message: failed to connect to all addresses}}
was (Author: carlomazzaferro):
It's a python server running behind a Traefik ingress controller (using h2c server scheme), server runs on 5005. For some obscure reason, with version 2.0.0 I am able to fix the issue by adding port `443` to the location, which wasn't needed with version 1.0.1. FWIW, here's the trace log without the port added, v2.0.0
{{D1102 17:41:42.069247000 4477541824 ev_posix.cc:174] Using polling engine: poll}}
{{D1102 17:41:42.069323000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "grpclb"}}
{{D1102 17:41:42.069333000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "cds_experimental"}}
{{D1102 17:41:42.069338000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "eds_experimental"}}
{{D1102 17:41:42.069344000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "lrs_experimental"}}
{{D1102 17:41:42.069349000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "priority_experimental"}}
{{D1102 17:41:42.069353000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "weighted_target_experimental"}}
{{D1102 17:41:42.069360000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "xds_routing_experimental"}}
{{D1102 17:41:42.069366000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "pick_first"}}
{{D1102 17:41:42.069370000 4477541824 lb_policy_registry.cc:38] registering LB policy factory for "round_robin"}}
{{D1102 17:41:42.069470000 4477541824 dns_resolver_ares.cc:503] Using ares dns resolver}}
{{I1102 17:41:42.069753000 4477541824 call.cc:626] OP[client-channel:0x7fd4bd0e69c0]: SEND_INITIAL_METADATA\{key=3a 70 61 74 68 ':path' value=2f 61 72 72 6f 77 2e 66 6c 69 67 68 74 2e 70 72 6f 74 6f 63 6f 6c 2e 46 6c 69 67 68 74 53 65 72 76 69 63 65 2f 48 61 6e 64 73 68 61 6b 65 '/arrow.flight.protocol.FlightService/Handshake'}}}
{{I1102 17:41:42.077093000 4477541824 subchannel.cc:1003] Connect failed: \{"created":"@1604335302.077058000","description":"Can't assign requested address","errno":49,"file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/lib/iomgr/tcp_client_posix.cc","file_line":307,"os_error":"Can't assign requested address","syscall":"connect"}}}
{{I1102 17:41:42.077219000 4477541824 subchannel.cc:1003] Connect failed: \{"created":"@1604335302.077197000","description":"Can't assign requested address","errno":49,"file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/lib/iomgr/tcp_client_posix.cc","file_line":307,"os_error":"Can't assign requested address","syscall":"connect"}}}
{{I1102 17:41:42.077320000 4477541824 subchannel.cc:1003] Connect failed: \{"created":"@1604335302.077298000","description":"Can't assign requested address","errno":49,"file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/lib/iomgr/tcp_client_posix.cc","file_line":307,"os_error":"Can't assign requested address","syscall":"connect"}}}
{{I1102 17:41:42.077354000 4477541824 subchannel.cc:942] Subchannel 0x7fd4c2a108e0: Retry in 100 milliseconds}}
{{I1102 17:41:42.077434000 4477541824 call.cc:626] OP[client-channel:0x7fd4bd0e69c0]: CANCEL:\{"created":"@1604335302.077349000","description":"Failed to pick subchannel","file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/ext/filters/client_channel/client_channel.cc","file_line":3962,"referenced_errors":[{"created":"@1604335302.077347000","description":"failed to connect to all addresses","file":"/Users/travis/build/ursa-labs/crossbow/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}}}
{{I1102 17:41:42.077811000 4477541824 call.cc:626] OP[client-channel:0x7fd4bd0e69c0]: SEND_MESSAGE:flags=0x00000000:len=38}}
{{I1102 17:41:42.077839000 4477541824 call.cc:626] OP[client-channel:0x7fd4bd0e69c0]: RECV_INITIAL_METADATA RECV_TRAILING_METADATA}}
{{Traceback (most recent call last):}}
{{ File "scratch.py", line 32, in <module>}}
{{ ApiKeyClientAuthHandler(api_key="c8669431-8172-4c24-8b17-300330e10f1b"),}}
{{ File "pyarrow/_flight.pyx", line 1149, in pyarrow._flight.FlightClient.authenticate}}
{{ File "pyarrow/_flight.pyx", line 70, in pyarrow._flight.check_flight_status}}
{{pyarrow._flight.FlightUnavailableError: gRPC returned unavailable error, with message: failed to connect to all addresses}}
> [FlightRPC][Python] FlightRPC authentication mechanism changed and is undocumented, breaking current working code
> -----------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-10460
> URL: https://issues.apache.org/jira/browse/ARROW-10460
> Project: Apache Arrow
> Issue Type: Bug
> Components: FlightRPC, Python
> Affects Versions: 2.0.0
> Environment: MacOS Catalina, python 3.7.4
> Reporter: Carlo Mazzaferro
> Priority: Major
> Fix For: 1.0.1
>
>
> Authentication with FlightClient changed from version 1.0.1 to 2.0.0 and is undocumented.
>
> {{>>> from pyarrow import ___version___}}
> {{>> print(}}_{{_version}}_{{_)}}
> {{'1.0.1'}}
> {{>>> from handlers import ApiKeyClientAuthHandler # custom handler}}}}
> {{>>> from pyarrow.flight import FlightClient}}
> {{>>> client = FlightClient(location="grpc+tcp://xxxxxxxxxx.com")}}
> {{>>> client.authenticate(ApiKeyClientAuthHandler(api_key="xxxxx"))}}
> {{>>>}}
>
> And the exact same code in 2.0.0 breaks
>
> {{>>> from pyarrow import ___version___}}
> {{>> print(}}_{{_version}}_{{_)}}
> {{'2.0.0'}}
> {{>>> from handlers import ApiKeyClientAuthHandler # custom handler}}}}
> {{>>> from pyarrow.flight import FlightClient}}
> {{>>> client = FlightClient(location="grpc+tcp://xxxxxxxxxx.com")}}
> {{>>> client.authenticate(ApiKeyClientAuthHandler(api_key="xxxxx"))}}
> {{Traceback (most recent call last):}}
> {{ File "scratch.py", line 15, in <module>}}
> {{ client.authenticate(ApiKeyClientAuthHandler(api_key="xxxxxxxxxx"))}}
> {{ File "pyarrow/_flight.pyx", line 1149, in pyarrow._flight.FlightClient.authenticate}}
> {{ File "pyarrow/_flight.pyx", line 70, in pyarrow._flight.check_flight_status}}
> {{pyarrow._flight.FlightUnavailableError: gRPC returned unavailable error, with message: failed to connect to all addresses}}
>
> What happened?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)