You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Rohit Gupta <ro...@dremio.com> on 2019/10/04 18:04:36 UTC
[Proposal]: Expose Flight gRPC for Dremio use case (Java)
Hi,
At dremio we are using gRPC for JobsService. One of the api's relies on
Arrow Flight. We want access to the Flight service so we can bind it to the
same managed channel as the rest of JobsService (& not have a completely
separate server).
The approach would be to create a new module within the same package
(org.apache.arrow.flight) and have 2 classes FlightGrpcServer &
FlightGrpcClient that expose the client & server, and also make
FlightClient ctor package-private.
Please let us know if you have questions or concerns.
Best,
Rohit
Re: [Proposal]: Expose Flight gRPC for Dremio use case (Java)
Posted by Rohit Gupta <ro...@dremio.com>.
I think for now we just want to expose the gRPC impl under a different
namespace
- FlightGrpcServer would expose FlightBindingService
- FlightGrpcClient would expose FlightClient
On Fri, Oct 4, 2019 at 11:48 AM David Li <li...@gmail.com> wrote:
> Hi Rohit,
>
> This sounds interesting, and I think we've voiced support for
> something similar before :)
>
> Given that Flight does want to abstract over the exact backends,
> though, how should we approach this? Is the proposal to also refactor
> Flight/Java such that the core classes are just interfaces (or
> delegate to interfaces) that anyone can implement, and have the gRPC
> implementation as the reference one? Or is this just proposing to
> expose the gRPC implementation under a separate namespace, and leave
> that question for later?
>
> Best,
> David
>
> On 10/4/19, Rohit Gupta <ro...@dremio.com> wrote:
> > Hi,
> >
> > At dremio we are using gRPC for JobsService. One of the api's relies on
> > Arrow Flight. We want access to the Flight service so we can bind it to
> the
> > same managed channel as the rest of JobsService (& not have a completely
> > separate server).
> >
> > The approach would be to create a new module within the same package
> > (org.apache.arrow.flight) and have 2 classes FlightGrpcServer &
> > FlightGrpcClient that expose the client & server, and also make
> > FlightClient ctor package-private.
> >
> > Please let us know if you have questions or concerns.
> >
> > Best,
> > Rohit
> >
>
Re: [Proposal]: Expose Flight gRPC for Dremio use case (Java)
Posted by Rohit Gupta <ro...@dremio.com>.
Corresponding JIRA issue: https://issues.apache.org/jira/browse/ARROW-6807
Wes,
It's possible for a single gRPC server to expose multiple services.The unit test in my patch
highlights that functionality.
PR: https://github.com/apache/arrow/pull/5597
On 2019/10/04 19:36:52, Wes McKinney <we...@gmail.com> wrote:
> Is it possible for a single gRPC server to expose multiple services
> through the same port (it sounds like it is)? It would be a good idea
> to do similar refactoring in C++ so that Flight RPC endpoints can be
> provided alongside some other non-Flight endpoints in the same gRPC
> server
>
> On Fri, Oct 4, 2019 at 1:49 PM David Li <li...@gmail.com> wrote:
> >
> > Hi Rohit,
> >
> > This sounds interesting, and I think we've voiced support for
> > something similar before :)
> >
> > Given that Flight does want to abstract over the exact backends,
> > though, how should we approach this? Is the proposal to also refactor
> > Flight/Java such that the core classes are just interfaces (or
> > delegate to interfaces) that anyone can implement, and have the gRPC
> > implementation as the reference one? Or is this just proposing to
> > expose the gRPC implementation under a separate namespace, and leave
> > that question for later?
> >
> > Best,
> > David
> >
> > On 10/4/19, Rohit Gupta <ro...@dremio.com> wrote:
> > > Hi,
> > >
> > > At dremio we are using gRPC for JobsService. One of the api's relies on
> > > Arrow Flight. We want access to the Flight service so we can bind it to the
> > > same managed channel as the rest of JobsService (& not have a completely
> > > separate server).
> > >
> > > The approach would be to create a new module within the same package
> > > (org.apache.arrow.flight) and have 2 classes FlightGrpcServer &
> > > FlightGrpcClient that expose the client & server, and also make
> > > FlightClient ctor package-private.
> > >
> > > Please let us know if you have questions or concerns.
> > >
> > > Best,
> > > Rohit
> > >
>
Re: [Proposal]: Expose Flight gRPC for Dremio use case (Java)
Posted by Jacques Nadeau <ja...@apache.org>.
>
> Is it possible for a single gRPC server to expose multiple services
> through the same port (it sounds like it is)? It would be a good idea
> to do similar refactoring in C++ so that Flight RPC endpoints can be
> provided alongside some other non-Flight endpoints in the same gRPC
> server
>
It definitely is in Java land as I understand it. That's exactly the goal
here. The thinking is to expose this for people who want to use it as a
separate namespace. It--by definition--has to leak grpc which means we
should keep separate from the main flight dependency which completely hides
grpc.
I think for now we just want to expose the gRPC impl under a different
> namespace
> - FlightGrpcServer would expose FlightBindingService
> - FlightGrpcClient would expose FlightClient
Rohit, can you post an initial patch for people to see what this would
actually look like?
Re: [Proposal]: Expose Flight gRPC for Dremio use case (Java)
Posted by Wes McKinney <we...@gmail.com>.
Is it possible for a single gRPC server to expose multiple services
through the same port (it sounds like it is)? It would be a good idea
to do similar refactoring in C++ so that Flight RPC endpoints can be
provided alongside some other non-Flight endpoints in the same gRPC
server
On Fri, Oct 4, 2019 at 1:49 PM David Li <li...@gmail.com> wrote:
>
> Hi Rohit,
>
> This sounds interesting, and I think we've voiced support for
> something similar before :)
>
> Given that Flight does want to abstract over the exact backends,
> though, how should we approach this? Is the proposal to also refactor
> Flight/Java such that the core classes are just interfaces (or
> delegate to interfaces) that anyone can implement, and have the gRPC
> implementation as the reference one? Or is this just proposing to
> expose the gRPC implementation under a separate namespace, and leave
> that question for later?
>
> Best,
> David
>
> On 10/4/19, Rohit Gupta <ro...@dremio.com> wrote:
> > Hi,
> >
> > At dremio we are using gRPC for JobsService. One of the api's relies on
> > Arrow Flight. We want access to the Flight service so we can bind it to the
> > same managed channel as the rest of JobsService (& not have a completely
> > separate server).
> >
> > The approach would be to create a new module within the same package
> > (org.apache.arrow.flight) and have 2 classes FlightGrpcServer &
> > FlightGrpcClient that expose the client & server, and also make
> > FlightClient ctor package-private.
> >
> > Please let us know if you have questions or concerns.
> >
> > Best,
> > Rohit
> >
Re: [Proposal]: Expose Flight gRPC for Dremio use case (Java)
Posted by David Li <li...@gmail.com>.
Hi Rohit,
This sounds interesting, and I think we've voiced support for
something similar before :)
Given that Flight does want to abstract over the exact backends,
though, how should we approach this? Is the proposal to also refactor
Flight/Java such that the core classes are just interfaces (or
delegate to interfaces) that anyone can implement, and have the gRPC
implementation as the reference one? Or is this just proposing to
expose the gRPC implementation under a separate namespace, and leave
that question for later?
Best,
David
On 10/4/19, Rohit Gupta <ro...@dremio.com> wrote:
> Hi,
>
> At dremio we are using gRPC for JobsService. One of the api's relies on
> Arrow Flight. We want access to the Flight service so we can bind it to the
> same managed channel as the rest of JobsService (& not have a completely
> separate server).
>
> The approach would be to create a new module within the same package
> (org.apache.arrow.flight) and have 2 classes FlightGrpcServer &
> FlightGrpcClient that expose the client & server, and also make
> FlightClient ctor package-private.
>
> Please let us know if you have questions or concerns.
>
> Best,
> Rohit
>