You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <so...@pitrou.net> on 2019/07/04 14:25:28 UTC
Re: Flight authentication interoperability
On Thu, 4 Jul 2019 09:04:34 -0500
Wes McKinney <we...@gmail.com> wrote:
>
> That being said, with Ryan's issue, he is using a feature
> (cross-language auth in Flight) that isn't being tested. The Flight
> integration tests do not use authentication AFAIK so I'm not surprised
> to hear that there may be an issue with it.
OTOH, it's a bit unlikely. Flight authentication is implemented is:
- the Arrow codebase simply passes opaque tokens around
- interpretation of tokens is handled by application code
- marshalling of tokens is handled by Protocol Buffers
So unless something silly is going on (such as "passing an empty string
instead of the actual token") there's not much potential for
auth interoperability issues in the core Flight codebase.
Regards
Antoine.
Re: Flight authentication interoperability
Posted by David Li <li...@gmail.com>.
I've filed/found the following:
ARROW-4575 [1] for adding Python to the Flight integration tests
ARROW-5875 [2] for adding RPC features (like auth) to integration tests
ARROW-1009 [3] for async APIs
ARROW-5876 [4] for implementing basic auth across all languages
ARROW-5877 [5] for documenting the caveats around auth
[1]: https://issues.apache.org/jira/browse/ARROW-4575
[2]: https://issues.apache.org/jira/browse/ARROW-5875
[3]: https://issues.apache.org/jira/browse/ARROW-1009
[4]: https://issues.apache.org/jira/browse/ARROW-5876
[5]: https://issues.apache.org/jira/browse/ARROW-5877
Best,
David
On 7/6/19, Wes McKinney <we...@gmail.com> wrote:
> Are there some action items (JIRA issues) to follow up here? At
> minimum having documentation about this for the Python client side
> would seem to be in order.
>
> On Thu, Jul 4, 2019 at 2:20 PM Ryan Murray <ry...@dremio.com> wrote:
>>
>> Hey David,
>>
>> I was actually testing test_flight.test_http_basic_auth(). But I think
>> the
>> same applies. The Java default implementation expects a handshake. More
>> to
>> the point it expects a BasicAuth protobuf which I believe is not exposed
>> at
>> all in python. Always returning true in
>> BasicServerAuthHandler.authenticate() allows for the test implementations
>> of Java and Python authentication to speak to each other.
>>
>> Thanks for the below link, that really clarified things for me. I would
>> add
>> to the list that we should normalise the use of BasicAuth protobuf
>> message
>> between java and cpp.
>>
>> Apologies for the urgency yesterday, I am glad it was sorted and more my
>> fault than Arrow's.
>>
>> Best,
>> Ryan
>>
>>
>> On Thu, Jul 4, 2019 at 11:29 AM David Li <li...@gmail.com> wrote:
>>
>> > Hmm, interesting. I assume you mean test_flight.test_token_auth() as
>> > the client? The tests weren't written to be explicitly compatible, but
>> > there's no reason you should get an indefinite stall.
>> >
>> > We don't use Handshake/ServerAuthHandler#authenticate, so that would
>> > explain why we don't see issues. I commented on this in the initial
>> > implementation:
>> > https://github.com/apache/arrow/pull/4125#discussion_r273935691
>> >
>> > > there is not a 1-1 mapping between connected clients and connected
>> > peers, and so you can
>> > > only know the identity of the peer at the moment it makes a call.
>> > > Just
>> > doing a handshake
>> > > (Authenticate) isn't enough to identify who is on the other end of a
>> > particular connection.
>> >
>> > the reason being that a layer 7 load balancer (e.g. Envoy) means one
>> > gRPC connection can represent multiple clients. Conversely,
>> > client-side load balancing (built into gRPC) means one client-side
>> > "connection" can actually represent multiple servers. Of course, you
>> > have to consciously deploy in this manner, so Handshake is still
>> > useful if you know you won't ever need these features.
>> >
>> > As I see it, this means there's a few things to work on:
>> > - Flight RPC feature compatibility needs to be tested, not just format
>> > compatibility.
>> > - We should start thinking about async APIs and/or timeouts in any
>> > sort of API that makes a network call (there's already a JIRA:
>> > https://issues.apache.org/jira/browse/ARROW-1009), since "never
>> > returns" is a terrible failure mode
>> >
>> > Best,
>> > David
>> >
>> > On 7/4/19, Ryan Murray <ry...@dremio.com> wrote:
>> > > Hey David,
>> > >
>> > > I am curious to see what you are doing different from me. I am
>> > > running
>> > the
>> > > Java ExampleFlightServer.java against the python auth flight tests
>> > > and
>> > they
>> > > are not passing. The particular issue is that incoming.next() never
>> > returns
>> > > in BasicServerAuthHandler.java:56
>> > >
>> > > It doesn't appear to be anything wrong w/ the auth piece specifically
>> > > rather the server appears to not be getting the auth info to verify. I
>> > > am
>> > > still investigating my issue but I am glad that someone else has
>> > > gotten
>> > > this to work.
>> > >
>> > > Best,
>> > > Ryan
>> > >
>> > > On Thu, Jul 4, 2019 at 9:13 AM Antoine Pitrou <an...@python.org>
>> > wrote:
>> > >
>> > >>
>> > >> It may be worth opening a JIRA for the flaky tests if not already
>> > >> done.
>> > >>
>> > >> Regards
>> > >>
>> > >> Antoine.
>> > >>
>> > >>
>> > >> Le 04/07/2019 à 18:11, David Li a écrit :
>> > >> > I'm also curious as to what the issue was, as we've been doing
>> > >> > Python-client-Java-server auth with development builds without
>> > >> > trouble.
>> > >> >
>> > >> > Regardless - this does point out a need for more cross-language
>> > >> > Flight
>> > >> > testing (perhaps a Flight-specific integration suite?), and to get
>> > >> > existing tests running more consistently in CI (Flight/Java in
>> > >> > particular has a lot of flaky tests, though the auth tests are
>> > >> > enabled
>> > >> > in Travis).
>> > >> >
>> > >> > Best,
>> > >> > David
>> > >> >
>> > >> > On 7/4/19, Jacques Nadeau <ja...@apache.org> wrote:
>> > >> >> Which is exactly why I was withholding a vote until there was
>> > >> >> more
>> > >> >> information.
>> > >> >>
>> > >> >> On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <so...@pitrou.net>
>> > >> wrote:
>> > >> >>
>> > >> >>> On Thu, 4 Jul 2019 09:04:34 -0500
>> > >> >>> Wes McKinney <we...@gmail.com> wrote:
>> > >> >>>>
>> > >> >>>> That being said, with Ryan's issue, he is using a feature
>> > >> >>>> (cross-language auth in Flight) that isn't being tested. The
>> > >> >>>> Flight
>> > >> >>>> integration tests do not use authentication AFAIK so I'm not
>> > >> >>>> surprised
>> > >> >>>> to hear that there may be an issue with it.
>> > >> >>>
>> > >> >>> OTOH, it's a bit unlikely. Flight authentication is implemented
>> > >> >>> is:
>> > >> >>> - the Arrow codebase simply passes opaque tokens around
>> > >> >>> - interpretation of tokens is handled by application code
>> > >> >>> - marshalling of tokens is handled by Protocol Buffers
>> > >> >>>
>> > >> >>> So unless something silly is going on (such as "passing an empty
>> > >> >>> string
>> > >> >>> instead of the actual token") there's not much potential for
>> > >> >>> auth interoperability issues in the core Flight codebase.
>> > >> >>>
>> > >> >>> Regards
>> > >> >>>
>> > >> >>> Antoine.
>> > >> >>>
>> > >> >>>
>> > >> >>>
>> > >> >>
>> > >>
>> > >
>> > >
>> > > --
>> > >
>> > > Ryan Murray | Principal Consulting Engineer
>> > >
>> > > +447540852009 | rymurr@dremio.com
>> > >
>> > > <https://www.dremio.com/>
>> > > Check out our GitHub <https://www.github.com/dremio>, join our
>> > > community
>> > > site <https://community.dremio.com/> & Download Dremio
>> > > <https://www.dremio.com/download>
>> > >
>> >
>>
>>
>> --
>>
>> Ryan Murray | Principal Consulting Engineer
>>
>> +447540852009 | rymurr@dremio.com
>>
>> <https://www.dremio.com/>
>> Check out our GitHub <https://www.github.com/dremio>, join our community
>> site <https://community.dremio.com/> & Download Dremio
>> <https://www.dremio.com/download>
>> I
>
Re: Flight authentication interoperability
Posted by Wes McKinney <we...@gmail.com>.
Are there some action items (JIRA issues) to follow up here? At
minimum having documentation about this for the Python client side
would seem to be in order.
On Thu, Jul 4, 2019 at 2:20 PM Ryan Murray <ry...@dremio.com> wrote:
>
> Hey David,
>
> I was actually testing test_flight.test_http_basic_auth(). But I think the
> same applies. The Java default implementation expects a handshake. More to
> the point it expects a BasicAuth protobuf which I believe is not exposed at
> all in python. Always returning true in
> BasicServerAuthHandler.authenticate() allows for the test implementations
> of Java and Python authentication to speak to each other.
>
> Thanks for the below link, that really clarified things for me. I would add
> to the list that we should normalise the use of BasicAuth protobuf message
> between java and cpp.
>
> Apologies for the urgency yesterday, I am glad it was sorted and more my
> fault than Arrow's.
>
> Best,
> Ryan
>
>
> On Thu, Jul 4, 2019 at 11:29 AM David Li <li...@gmail.com> wrote:
>
> > Hmm, interesting. I assume you mean test_flight.test_token_auth() as
> > the client? The tests weren't written to be explicitly compatible, but
> > there's no reason you should get an indefinite stall.
> >
> > We don't use Handshake/ServerAuthHandler#authenticate, so that would
> > explain why we don't see issues. I commented on this in the initial
> > implementation:
> > https://github.com/apache/arrow/pull/4125#discussion_r273935691
> >
> > > there is not a 1-1 mapping between connected clients and connected
> > peers, and so you can
> > > only know the identity of the peer at the moment it makes a call. Just
> > doing a handshake
> > > (Authenticate) isn't enough to identify who is on the other end of a
> > particular connection.
> >
> > the reason being that a layer 7 load balancer (e.g. Envoy) means one
> > gRPC connection can represent multiple clients. Conversely,
> > client-side load balancing (built into gRPC) means one client-side
> > "connection" can actually represent multiple servers. Of course, you
> > have to consciously deploy in this manner, so Handshake is still
> > useful if you know you won't ever need these features.
> >
> > As I see it, this means there's a few things to work on:
> > - Flight RPC feature compatibility needs to be tested, not just format
> > compatibility.
> > - We should start thinking about async APIs and/or timeouts in any
> > sort of API that makes a network call (there's already a JIRA:
> > https://issues.apache.org/jira/browse/ARROW-1009), since "never
> > returns" is a terrible failure mode
> >
> > Best,
> > David
> >
> > On 7/4/19, Ryan Murray <ry...@dremio.com> wrote:
> > > Hey David,
> > >
> > > I am curious to see what you are doing different from me. I am running
> > the
> > > Java ExampleFlightServer.java against the python auth flight tests and
> > they
> > > are not passing. The particular issue is that incoming.next() never
> > returns
> > > in BasicServerAuthHandler.java:56
> > >
> > > It doesn't appear to be anything wrong w/ the auth piece specifically
> > > rather the server appears to not be getting the auth info to verify. I am
> > > still investigating my issue but I am glad that someone else has gotten
> > > this to work.
> > >
> > > Best,
> > > Ryan
> > >
> > > On Thu, Jul 4, 2019 at 9:13 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > >>
> > >> It may be worth opening a JIRA for the flaky tests if not already done.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >>
> > >> Le 04/07/2019 à 18:11, David Li a écrit :
> > >> > I'm also curious as to what the issue was, as we've been doing
> > >> > Python-client-Java-server auth with development builds without
> > >> > trouble.
> > >> >
> > >> > Regardless - this does point out a need for more cross-language Flight
> > >> > testing (perhaps a Flight-specific integration suite?), and to get
> > >> > existing tests running more consistently in CI (Flight/Java in
> > >> > particular has a lot of flaky tests, though the auth tests are enabled
> > >> > in Travis).
> > >> >
> > >> > Best,
> > >> > David
> > >> >
> > >> > On 7/4/19, Jacques Nadeau <ja...@apache.org> wrote:
> > >> >> Which is exactly why I was withholding a vote until there was more
> > >> >> information.
> > >> >>
> > >> >> On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <so...@pitrou.net>
> > >> wrote:
> > >> >>
> > >> >>> On Thu, 4 Jul 2019 09:04:34 -0500
> > >> >>> Wes McKinney <we...@gmail.com> wrote:
> > >> >>>>
> > >> >>>> That being said, with Ryan's issue, he is using a feature
> > >> >>>> (cross-language auth in Flight) that isn't being tested. The Flight
> > >> >>>> integration tests do not use authentication AFAIK so I'm not
> > >> >>>> surprised
> > >> >>>> to hear that there may be an issue with it.
> > >> >>>
> > >> >>> OTOH, it's a bit unlikely. Flight authentication is implemented is:
> > >> >>> - the Arrow codebase simply passes opaque tokens around
> > >> >>> - interpretation of tokens is handled by application code
> > >> >>> - marshalling of tokens is handled by Protocol Buffers
> > >> >>>
> > >> >>> So unless something silly is going on (such as "passing an empty
> > >> >>> string
> > >> >>> instead of the actual token") there's not much potential for
> > >> >>> auth interoperability issues in the core Flight codebase.
> > >> >>>
> > >> >>> Regards
> > >> >>>
> > >> >>> Antoine.
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>
> > >>
> > >
> > >
> > > --
> > >
> > > Ryan Murray | Principal Consulting Engineer
> > >
> > > +447540852009 | rymurr@dremio.com
> > >
> > > <https://www.dremio.com/>
> > > Check out our GitHub <https://www.github.com/dremio>, join our community
> > > site <https://community.dremio.com/> & Download Dremio
> > > <https://www.dremio.com/download>
> > >
> >
>
>
> --
>
> Ryan Murray | Principal Consulting Engineer
>
> +447540852009 | rymurr@dremio.com
>
> <https://www.dremio.com/>
> Check out our GitHub <https://www.github.com/dremio>, join our community
> site <https://community.dremio.com/> & Download Dremio
> <https://www.dremio.com/download>
> I
Re: Flight authentication interoperability
Posted by Ryan Murray <ry...@dremio.com>.
Hey David,
I was actually testing test_flight.test_http_basic_auth(). But I think the
same applies. The Java default implementation expects a handshake. More to
the point it expects a BasicAuth protobuf which I believe is not exposed at
all in python. Always returning true in
BasicServerAuthHandler.authenticate() allows for the test implementations
of Java and Python authentication to speak to each other.
Thanks for the below link, that really clarified things for me. I would add
to the list that we should normalise the use of BasicAuth protobuf message
between java and cpp.
Apologies for the urgency yesterday, I am glad it was sorted and more my
fault than Arrow's.
Best,
Ryan
On Thu, Jul 4, 2019 at 11:29 AM David Li <li...@gmail.com> wrote:
> Hmm, interesting. I assume you mean test_flight.test_token_auth() as
> the client? The tests weren't written to be explicitly compatible, but
> there's no reason you should get an indefinite stall.
>
> We don't use Handshake/ServerAuthHandler#authenticate, so that would
> explain why we don't see issues. I commented on this in the initial
> implementation:
> https://github.com/apache/arrow/pull/4125#discussion_r273935691
>
> > there is not a 1-1 mapping between connected clients and connected
> peers, and so you can
> > only know the identity of the peer at the moment it makes a call. Just
> doing a handshake
> > (Authenticate) isn't enough to identify who is on the other end of a
> particular connection.
>
> the reason being that a layer 7 load balancer (e.g. Envoy) means one
> gRPC connection can represent multiple clients. Conversely,
> client-side load balancing (built into gRPC) means one client-side
> "connection" can actually represent multiple servers. Of course, you
> have to consciously deploy in this manner, so Handshake is still
> useful if you know you won't ever need these features.
>
> As I see it, this means there's a few things to work on:
> - Flight RPC feature compatibility needs to be tested, not just format
> compatibility.
> - We should start thinking about async APIs and/or timeouts in any
> sort of API that makes a network call (there's already a JIRA:
> https://issues.apache.org/jira/browse/ARROW-1009), since "never
> returns" is a terrible failure mode
>
> Best,
> David
>
> On 7/4/19, Ryan Murray <ry...@dremio.com> wrote:
> > Hey David,
> >
> > I am curious to see what you are doing different from me. I am running
> the
> > Java ExampleFlightServer.java against the python auth flight tests and
> they
> > are not passing. The particular issue is that incoming.next() never
> returns
> > in BasicServerAuthHandler.java:56
> >
> > It doesn't appear to be anything wrong w/ the auth piece specifically
> > rather the server appears to not be getting the auth info to verify. I am
> > still investigating my issue but I am glad that someone else has gotten
> > this to work.
> >
> > Best,
> > Ryan
> >
> > On Thu, Jul 4, 2019 at 9:13 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> It may be worth opening a JIRA for the flaky tests if not already done.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >> Le 04/07/2019 à 18:11, David Li a écrit :
> >> > I'm also curious as to what the issue was, as we've been doing
> >> > Python-client-Java-server auth with development builds without
> >> > trouble.
> >> >
> >> > Regardless - this does point out a need for more cross-language Flight
> >> > testing (perhaps a Flight-specific integration suite?), and to get
> >> > existing tests running more consistently in CI (Flight/Java in
> >> > particular has a lot of flaky tests, though the auth tests are enabled
> >> > in Travis).
> >> >
> >> > Best,
> >> > David
> >> >
> >> > On 7/4/19, Jacques Nadeau <ja...@apache.org> wrote:
> >> >> Which is exactly why I was withholding a vote until there was more
> >> >> information.
> >> >>
> >> >> On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <so...@pitrou.net>
> >> wrote:
> >> >>
> >> >>> On Thu, 4 Jul 2019 09:04:34 -0500
> >> >>> Wes McKinney <we...@gmail.com> wrote:
> >> >>>>
> >> >>>> That being said, with Ryan's issue, he is using a feature
> >> >>>> (cross-language auth in Flight) that isn't being tested. The Flight
> >> >>>> integration tests do not use authentication AFAIK so I'm not
> >> >>>> surprised
> >> >>>> to hear that there may be an issue with it.
> >> >>>
> >> >>> OTOH, it's a bit unlikely. Flight authentication is implemented is:
> >> >>> - the Arrow codebase simply passes opaque tokens around
> >> >>> - interpretation of tokens is handled by application code
> >> >>> - marshalling of tokens is handled by Protocol Buffers
> >> >>>
> >> >>> So unless something silly is going on (such as "passing an empty
> >> >>> string
> >> >>> instead of the actual token") there's not much potential for
> >> >>> auth interoperability issues in the core Flight codebase.
> >> >>>
> >> >>> Regards
> >> >>>
> >> >>> Antoine.
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >>
> >
> >
> > --
> >
> > Ryan Murray | Principal Consulting Engineer
> >
> > +447540852009 | rymurr@dremio.com
> >
> > <https://www.dremio.com/>
> > Check out our GitHub <https://www.github.com/dremio>, join our community
> > site <https://community.dremio.com/> & Download Dremio
> > <https://www.dremio.com/download>
> >
>
--
Ryan Murray | Principal Consulting Engineer
+447540852009 | rymurr@dremio.com
<https://www.dremio.com/>
Check out our GitHub <https://www.github.com/dremio>, join our community
site <https://community.dremio.com/> & Download Dremio
<https://www.dremio.com/download>
I
Re: Flight authentication interoperability
Posted by David Li <li...@gmail.com>.
Hmm, interesting. I assume you mean test_flight.test_token_auth() as
the client? The tests weren't written to be explicitly compatible, but
there's no reason you should get an indefinite stall.
We don't use Handshake/ServerAuthHandler#authenticate, so that would
explain why we don't see issues. I commented on this in the initial
implementation:
https://github.com/apache/arrow/pull/4125#discussion_r273935691
> there is not a 1-1 mapping between connected clients and connected peers, and so you can
> only know the identity of the peer at the moment it makes a call. Just doing a handshake
> (Authenticate) isn't enough to identify who is on the other end of a particular connection.
the reason being that a layer 7 load balancer (e.g. Envoy) means one
gRPC connection can represent multiple clients. Conversely,
client-side load balancing (built into gRPC) means one client-side
"connection" can actually represent multiple servers. Of course, you
have to consciously deploy in this manner, so Handshake is still
useful if you know you won't ever need these features.
As I see it, this means there's a few things to work on:
- Flight RPC feature compatibility needs to be tested, not just format
compatibility.
- We should start thinking about async APIs and/or timeouts in any
sort of API that makes a network call (there's already a JIRA:
https://issues.apache.org/jira/browse/ARROW-1009), since "never
returns" is a terrible failure mode
Best,
David
On 7/4/19, Ryan Murray <ry...@dremio.com> wrote:
> Hey David,
>
> I am curious to see what you are doing different from me. I am running the
> Java ExampleFlightServer.java against the python auth flight tests and they
> are not passing. The particular issue is that incoming.next() never returns
> in BasicServerAuthHandler.java:56
>
> It doesn't appear to be anything wrong w/ the auth piece specifically
> rather the server appears to not be getting the auth info to verify. I am
> still investigating my issue but I am glad that someone else has gotten
> this to work.
>
> Best,
> Ryan
>
> On Thu, Jul 4, 2019 at 9:13 AM Antoine Pitrou <an...@python.org> wrote:
>
>>
>> It may be worth opening a JIRA for the flaky tests if not already done.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 04/07/2019 à 18:11, David Li a écrit :
>> > I'm also curious as to what the issue was, as we've been doing
>> > Python-client-Java-server auth with development builds without
>> > trouble.
>> >
>> > Regardless - this does point out a need for more cross-language Flight
>> > testing (perhaps a Flight-specific integration suite?), and to get
>> > existing tests running more consistently in CI (Flight/Java in
>> > particular has a lot of flaky tests, though the auth tests are enabled
>> > in Travis).
>> >
>> > Best,
>> > David
>> >
>> > On 7/4/19, Jacques Nadeau <ja...@apache.org> wrote:
>> >> Which is exactly why I was withholding a vote until there was more
>> >> information.
>> >>
>> >> On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <so...@pitrou.net>
>> wrote:
>> >>
>> >>> On Thu, 4 Jul 2019 09:04:34 -0500
>> >>> Wes McKinney <we...@gmail.com> wrote:
>> >>>>
>> >>>> That being said, with Ryan's issue, he is using a feature
>> >>>> (cross-language auth in Flight) that isn't being tested. The Flight
>> >>>> integration tests do not use authentication AFAIK so I'm not
>> >>>> surprised
>> >>>> to hear that there may be an issue with it.
>> >>>
>> >>> OTOH, it's a bit unlikely. Flight authentication is implemented is:
>> >>> - the Arrow codebase simply passes opaque tokens around
>> >>> - interpretation of tokens is handled by application code
>> >>> - marshalling of tokens is handled by Protocol Buffers
>> >>>
>> >>> So unless something silly is going on (such as "passing an empty
>> >>> string
>> >>> instead of the actual token") there's not much potential for
>> >>> auth interoperability issues in the core Flight codebase.
>> >>>
>> >>> Regards
>> >>>
>> >>> Antoine.
>> >>>
>> >>>
>> >>>
>> >>
>>
>
>
> --
>
> Ryan Murray | Principal Consulting Engineer
>
> +447540852009 | rymurr@dremio.com
>
> <https://www.dremio.com/>
> Check out our GitHub <https://www.github.com/dremio>, join our community
> site <https://community.dremio.com/> & Download Dremio
> <https://www.dremio.com/download>
>
Re: Flight authentication interoperability
Posted by Ryan Murray <ry...@dremio.com>.
Hey David,
I am curious to see what you are doing different from me. I am running the
Java ExampleFlightServer.java against the python auth flight tests and they
are not passing. The particular issue is that incoming.next() never returns
in BasicServerAuthHandler.java:56
It doesn't appear to be anything wrong w/ the auth piece specifically
rather the server appears to not be getting the auth info to verify. I am
still investigating my issue but I am glad that someone else has gotten
this to work.
Best,
Ryan
On Thu, Jul 4, 2019 at 9:13 AM Antoine Pitrou <an...@python.org> wrote:
>
> It may be worth opening a JIRA for the flaky tests if not already done.
>
> Regards
>
> Antoine.
>
>
> Le 04/07/2019 à 18:11, David Li a écrit :
> > I'm also curious as to what the issue was, as we've been doing
> > Python-client-Java-server auth with development builds without
> > trouble.
> >
> > Regardless - this does point out a need for more cross-language Flight
> > testing (perhaps a Flight-specific integration suite?), and to get
> > existing tests running more consistently in CI (Flight/Java in
> > particular has a lot of flaky tests, though the auth tests are enabled
> > in Travis).
> >
> > Best,
> > David
> >
> > On 7/4/19, Jacques Nadeau <ja...@apache.org> wrote:
> >> Which is exactly why I was withholding a vote until there was more
> >> information.
> >>
> >> On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <so...@pitrou.net>
> wrote:
> >>
> >>> On Thu, 4 Jul 2019 09:04:34 -0500
> >>> Wes McKinney <we...@gmail.com> wrote:
> >>>>
> >>>> That being said, with Ryan's issue, he is using a feature
> >>>> (cross-language auth in Flight) that isn't being tested. The Flight
> >>>> integration tests do not use authentication AFAIK so I'm not surprised
> >>>> to hear that there may be an issue with it.
> >>>
> >>> OTOH, it's a bit unlikely. Flight authentication is implemented is:
> >>> - the Arrow codebase simply passes opaque tokens around
> >>> - interpretation of tokens is handled by application code
> >>> - marshalling of tokens is handled by Protocol Buffers
> >>>
> >>> So unless something silly is going on (such as "passing an empty string
> >>> instead of the actual token") there's not much potential for
> >>> auth interoperability issues in the core Flight codebase.
> >>>
> >>> Regards
> >>>
> >>> Antoine.
> >>>
> >>>
> >>>
> >>
>
--
Ryan Murray | Principal Consulting Engineer
+447540852009 | rymurr@dremio.com
<https://www.dremio.com/>
Check out our GitHub <https://www.github.com/dremio>, join our community
site <https://community.dremio.com/> & Download Dremio
<https://www.dremio.com/download>
Re: Flight authentication interoperability
Posted by Antoine Pitrou <an...@python.org>.
It may be worth opening a JIRA for the flaky tests if not already done.
Regards
Antoine.
Le 04/07/2019 à 18:11, David Li a écrit :
> I'm also curious as to what the issue was, as we've been doing
> Python-client-Java-server auth with development builds without
> trouble.
>
> Regardless - this does point out a need for more cross-language Flight
> testing (perhaps a Flight-specific integration suite?), and to get
> existing tests running more consistently in CI (Flight/Java in
> particular has a lot of flaky tests, though the auth tests are enabled
> in Travis).
>
> Best,
> David
>
> On 7/4/19, Jacques Nadeau <ja...@apache.org> wrote:
>> Which is exactly why I was withholding a vote until there was more
>> information.
>>
>> On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <so...@pitrou.net> wrote:
>>
>>> On Thu, 4 Jul 2019 09:04:34 -0500
>>> Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>> That being said, with Ryan's issue, he is using a feature
>>>> (cross-language auth in Flight) that isn't being tested. The Flight
>>>> integration tests do not use authentication AFAIK so I'm not surprised
>>>> to hear that there may be an issue with it.
>>>
>>> OTOH, it's a bit unlikely. Flight authentication is implemented is:
>>> - the Arrow codebase simply passes opaque tokens around
>>> - interpretation of tokens is handled by application code
>>> - marshalling of tokens is handled by Protocol Buffers
>>>
>>> So unless something silly is going on (such as "passing an empty string
>>> instead of the actual token") there's not much potential for
>>> auth interoperability issues in the core Flight codebase.
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>>
>>
Re: Flight authentication interoperability
Posted by David Li <li...@gmail.com>.
I'm also curious as to what the issue was, as we've been doing
Python-client-Java-server auth with development builds without
trouble.
Regardless - this does point out a need for more cross-language Flight
testing (perhaps a Flight-specific integration suite?), and to get
existing tests running more consistently in CI (Flight/Java in
particular has a lot of flaky tests, though the auth tests are enabled
in Travis).
Best,
David
On 7/4/19, Jacques Nadeau <ja...@apache.org> wrote:
> Which is exactly why I was withholding a vote until there was more
> information.
>
> On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <so...@pitrou.net> wrote:
>
>> On Thu, 4 Jul 2019 09:04:34 -0500
>> Wes McKinney <we...@gmail.com> wrote:
>> >
>> > That being said, with Ryan's issue, he is using a feature
>> > (cross-language auth in Flight) that isn't being tested. The Flight
>> > integration tests do not use authentication AFAIK so I'm not surprised
>> > to hear that there may be an issue with it.
>>
>> OTOH, it's a bit unlikely. Flight authentication is implemented is:
>> - the Arrow codebase simply passes opaque tokens around
>> - interpretation of tokens is handled by application code
>> - marshalling of tokens is handled by Protocol Buffers
>>
>> So unless something silly is going on (such as "passing an empty string
>> instead of the actual token") there's not much potential for
>> auth interoperability issues in the core Flight codebase.
>>
>> Regards
>>
>> Antoine.
>>
>>
>>
>
Re: Flight authentication interoperability
Posted by Jacques Nadeau <ja...@apache.org>.
Which is exactly why I was withholding a vote until there was more
information.
On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <so...@pitrou.net> wrote:
> On Thu, 4 Jul 2019 09:04:34 -0500
> Wes McKinney <we...@gmail.com> wrote:
> >
> > That being said, with Ryan's issue, he is using a feature
> > (cross-language auth in Flight) that isn't being tested. The Flight
> > integration tests do not use authentication AFAIK so I'm not surprised
> > to hear that there may be an issue with it.
>
> OTOH, it's a bit unlikely. Flight authentication is implemented is:
> - the Arrow codebase simply passes opaque tokens around
> - interpretation of tokens is handled by application code
> - marshalling of tokens is handled by Protocol Buffers
>
> So unless something silly is going on (such as "passing an empty string
> instead of the actual token") there's not much potential for
> auth interoperability issues in the core Flight codebase.
>
> Regards
>
> Antoine.
>
>
>