You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ryan Murray <ry...@dremio.com> on 2019/08/01 12:10:51 UTC
[DISCUSS] Add GetFlightSchema to Flight RPC
Hi All,
Please see the attached document for a proposed addition to the Flight
RPC[1]. This is the result of a previous mailing list discussion[2].
I have created the Pull Request[3] to make the proposal a little more
concrete.
<https://www.dremio.com/download>
Please let me know if you have any questions or concerns.
Best,
Ryan
[1]:
https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
[2]:
https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
[3]: https://github.com/apache/arrow/pull/4980
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by Ryan Murray <ry...@dremio.com>.
Thanks both :-)
On Fri, Aug 16, 2019 at 6:22 AM Micah Kornfield <em...@gmail.com>
wrote:
> I'll start one shortly.
>
> On Thu, Aug 15, 2019 at 4:31 PM Wes McKinney <we...@gmail.com> wrote:
>
> > Yes, I think having a vote as a procedural matter would be a good thing.
> >
> > I have run dozens of public and private votes in my role as a PMC
> > member. I would appreciate if another PMC would assist with this vote.
> >
> > Thanks
> >
> > On Wed, Aug 14, 2019 at 5:37 PM Ryan Murray <ry...@dremio.com> wrote:
> > >
> > > Hi All,
> > >
> > > Does this require a vote? If yes what is the process for initiating
> one &
> > > if no I hope this is enough time for feedback and I would like to
> remove
> > > the draft designation from the PR
> > >
> > > Best,
> > > Ryan
> > >
> > > On Wed, Aug 7, 2019 at 9:31 AM Ryan Murray <ry...@dremio.com> wrote:
> > >
> > > > As per everyone's feedback I have renamed GetFlightSchema ->
> GetSchema
> > and
> > > > have removed the descriptor on the rpc result message. The doc has
> been
> > > > updated as has the draft PR
> > > >
> > > > On Thu, Aug 1, 2019 at 6:32 PM Bryan Cutler <cu...@gmail.com>
> wrote:
> > > >
> > > >> Sounds good to me, I would just echo what others have said.
> > > >>
> > > >> On Thu, Aug 1, 2019 at 8:17 AM Ryan Murray <ry...@dremio.com>
> wrote:
> > > >>
> > > >> > Thanks Wes,
> > > >> >
> > > >> > The descriptor is only there to maintain a bit of symmetry with
> > > >> > GetFlightInfo. Happy to remove it, I don't think its necessary and
> > > >> already
> > > >> > a few people agree. Similar with the method name, I am neutral to
> > the
> > > >> > naming and can call it whatever the community is happy with.
> > > >> >
> > > >> > Best,
> > > >> > Ryan
> > > >> >
> > > >> > On Thu, Aug 1, 2019 at 3:56 PM Wes McKinney <we...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > I'm generally supporting of adding the new RPC endpoint.
> > > >> > >
> > > >> > > To make a couple points from the document
> > > >> > >
> > > >> > > * I'm not sure what the purpose of returning the
> FlightDescriptor
> > is,
> > > >> > > but I haven't thought too hard about it
> > > >> > > * The Schema consists of a single IPC message -- dictionaries
> will
> > > >> > > appear in the actual DoGet stream. To motivate why this is --
> > > >> > > different endpoints might have different dictionaries
> > corresponding to
> > > >> > > fields in the schema, to have static/constant dictionaries in a
> > > >> > > distributed Flight setting is likely to be impractical. I
> > summarize
> > > >> > > the issue as "dictionaries are data, not metadata".
> > > >> > > * I would be OK calling this GetSchema instead of
> GetFlightSchema
> > but
> > > >> > > either is okay
> > > >> > >
> > > >> > > - Wes
> > > >> > >
> > > >> > > On Thu, Aug 1, 2019 at 8:08 AM David Li <li...@gmail.com>
> > > >> wrote:
> > > >> > > >
> > > >> > > > Hi Ryan,
> > > >> > > >
> > > >> > > > Thanks for writing this up! I made a couple of minor comments
> > in the
> > > >> > > > doc/implementation, but overall I'm in favor of having this
> RPC
> > > >> > > > method.
> > > >> > > >
> > > >> > > > Best,
> > > >> > > > David
> > > >> > > >
> > > >> > > > On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
> > > >> > > > > Hi All,
> > > >> > > > >
> > > >> > > > > Please see the attached document for a proposed addition to
> > the
> > > >> > Flight
> > > >> > > > > RPC[1]. This is the result of a previous mailing list
> > > >> discussion[2].
> > > >> > > > >
> > > >> > > > > I have created the Pull Request[3] to make the proposal a
> > little
> > > >> more
> > > >> > > > > concrete.
> > > >> > > > > <https://www.dremio.com/download>
> > > >> > > > > Please let me know if you have any questions or concerns.
> > > >> > > > >
> > > >> > > > > Best,
> > > >> > > > > Ryan
> > > >> > > > >
> > > >> > > > > [1]:
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
> > > >> > > > > [2]:
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
> > > >> > > > > [3]: https://github.com/apache/arrow/pull/4980
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > Ryan Murray | Principal Consulting Engineer
> > > >> >
> > > >> > +447540852009 | rymurr@dremio.com
> > > >> >
> > > >> > <https://www.dremio.com/>
> > > >> > Check out our GitHub <https://www.github.com/dremio>, join our
> > > >> community
> > > >> > site <https://community.dremio.com/> & Download Dremio
> > > >> > <https://www.dremio.com/download>
> > > >> >
> > > >>
> > > >
> > > >
> > > > --
> > > >
> > > > Ryan Murray | Principal Consulting Engineer
> > > >
> > > > +447540852009 | rymurr@dremio.com
> > > >
> > > > <https://www.dremio.com/>
> > > > Check out our GitHub <https://www.github.com/dremio>, join our
> > community
> > > > site <https://community.dremio.com/> & Download Dremio
> > > > <https://www.dremio.com/download>
> > > >
> > >
> > >
> > > --
> > >
> > > Ryan Murray | Principal Consulting Engineer
> > >
> > > +447540852009 | rymurr@dremio.com
> > >
> > > <https://www.dremio.com/>
> > > Check out our GitHub <https://www.github.com/dremio>, join our
> community
> > > site <https://community.dremio.com/> & Download Dremio
> > > <https://www.dremio.com/download>
> >
>
--
Ryan Murray | Principal Consulting Engineer
+447540852009 | rymurr@dremio.com
<https://www.dremio.com/>
Check out our GitHub <https://www.github.com/dremio>, join our community
site <https://community.dremio.com/> & Download Dremio
<https://www.dremio.com/download>
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by Micah Kornfield <em...@gmail.com>.
I'll start one shortly.
On Thu, Aug 15, 2019 at 4:31 PM Wes McKinney <we...@gmail.com> wrote:
> Yes, I think having a vote as a procedural matter would be a good thing.
>
> I have run dozens of public and private votes in my role as a PMC
> member. I would appreciate if another PMC would assist with this vote.
>
> Thanks
>
> On Wed, Aug 14, 2019 at 5:37 PM Ryan Murray <ry...@dremio.com> wrote:
> >
> > Hi All,
> >
> > Does this require a vote? If yes what is the process for initiating one &
> > if no I hope this is enough time for feedback and I would like to remove
> > the draft designation from the PR
> >
> > Best,
> > Ryan
> >
> > On Wed, Aug 7, 2019 at 9:31 AM Ryan Murray <ry...@dremio.com> wrote:
> >
> > > As per everyone's feedback I have renamed GetFlightSchema -> GetSchema
> and
> > > have removed the descriptor on the rpc result message. The doc has been
> > > updated as has the draft PR
> > >
> > > On Thu, Aug 1, 2019 at 6:32 PM Bryan Cutler <cu...@gmail.com> wrote:
> > >
> > >> Sounds good to me, I would just echo what others have said.
> > >>
> > >> On Thu, Aug 1, 2019 at 8:17 AM Ryan Murray <ry...@dremio.com> wrote:
> > >>
> > >> > Thanks Wes,
> > >> >
> > >> > The descriptor is only there to maintain a bit of symmetry with
> > >> > GetFlightInfo. Happy to remove it, I don't think its necessary and
> > >> already
> > >> > a few people agree. Similar with the method name, I am neutral to
> the
> > >> > naming and can call it whatever the community is happy with.
> > >> >
> > >> > Best,
> > >> > Ryan
> > >> >
> > >> > On Thu, Aug 1, 2019 at 3:56 PM Wes McKinney <we...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > I'm generally supporting of adding the new RPC endpoint.
> > >> > >
> > >> > > To make a couple points from the document
> > >> > >
> > >> > > * I'm not sure what the purpose of returning the FlightDescriptor
> is,
> > >> > > but I haven't thought too hard about it
> > >> > > * The Schema consists of a single IPC message -- dictionaries will
> > >> > > appear in the actual DoGet stream. To motivate why this is --
> > >> > > different endpoints might have different dictionaries
> corresponding to
> > >> > > fields in the schema, to have static/constant dictionaries in a
> > >> > > distributed Flight setting is likely to be impractical. I
> summarize
> > >> > > the issue as "dictionaries are data, not metadata".
> > >> > > * I would be OK calling this GetSchema instead of GetFlightSchema
> but
> > >> > > either is okay
> > >> > >
> > >> > > - Wes
> > >> > >
> > >> > > On Thu, Aug 1, 2019 at 8:08 AM David Li <li...@gmail.com>
> > >> wrote:
> > >> > > >
> > >> > > > Hi Ryan,
> > >> > > >
> > >> > > > Thanks for writing this up! I made a couple of minor comments
> in the
> > >> > > > doc/implementation, but overall I'm in favor of having this RPC
> > >> > > > method.
> > >> > > >
> > >> > > > Best,
> > >> > > > David
> > >> > > >
> > >> > > > On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
> > >> > > > > Hi All,
> > >> > > > >
> > >> > > > > Please see the attached document for a proposed addition to
> the
> > >> > Flight
> > >> > > > > RPC[1]. This is the result of a previous mailing list
> > >> discussion[2].
> > >> > > > >
> > >> > > > > I have created the Pull Request[3] to make the proposal a
> little
> > >> more
> > >> > > > > concrete.
> > >> > > > > <https://www.dremio.com/download>
> > >> > > > > Please let me know if you have any questions or concerns.
> > >> > > > >
> > >> > > > > Best,
> > >> > > > > Ryan
> > >> > > > >
> > >> > > > > [1]:
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
> > >> > > > > [2]:
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
> > >> > > > > [3]: https://github.com/apache/arrow/pull/4980
> > >> > > > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Ryan Murray | Principal Consulting Engineer
> > >> >
> > >> > +447540852009 | rymurr@dremio.com
> > >> >
> > >> > <https://www.dremio.com/>
> > >> > Check out our GitHub <https://www.github.com/dremio>, join our
> > >> community
> > >> > site <https://community.dremio.com/> & Download Dremio
> > >> > <https://www.dremio.com/download>
> > >> >
> > >>
> > >
> > >
> > > --
> > >
> > > Ryan Murray | Principal Consulting Engineer
> > >
> > > +447540852009 | rymurr@dremio.com
> > >
> > > <https://www.dremio.com/>
> > > Check out our GitHub <https://www.github.com/dremio>, join our
> community
> > > site <https://community.dremio.com/> & Download Dremio
> > > <https://www.dremio.com/download>
> > >
> >
> >
> > --
> >
> > Ryan Murray | Principal Consulting Engineer
> >
> > +447540852009 | rymurr@dremio.com
> >
> > <https://www.dremio.com/>
> > Check out our GitHub <https://www.github.com/dremio>, join our community
> > site <https://community.dremio.com/> & Download Dremio
> > <https://www.dremio.com/download>
>
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by Wes McKinney <we...@gmail.com>.
Yes, I think having a vote as a procedural matter would be a good thing.
I have run dozens of public and private votes in my role as a PMC
member. I would appreciate if another PMC would assist with this vote.
Thanks
On Wed, Aug 14, 2019 at 5:37 PM Ryan Murray <ry...@dremio.com> wrote:
>
> Hi All,
>
> Does this require a vote? If yes what is the process for initiating one &
> if no I hope this is enough time for feedback and I would like to remove
> the draft designation from the PR
>
> Best,
> Ryan
>
> On Wed, Aug 7, 2019 at 9:31 AM Ryan Murray <ry...@dremio.com> wrote:
>
> > As per everyone's feedback I have renamed GetFlightSchema -> GetSchema and
> > have removed the descriptor on the rpc result message. The doc has been
> > updated as has the draft PR
> >
> > On Thu, Aug 1, 2019 at 6:32 PM Bryan Cutler <cu...@gmail.com> wrote:
> >
> >> Sounds good to me, I would just echo what others have said.
> >>
> >> On Thu, Aug 1, 2019 at 8:17 AM Ryan Murray <ry...@dremio.com> wrote:
> >>
> >> > Thanks Wes,
> >> >
> >> > The descriptor is only there to maintain a bit of symmetry with
> >> > GetFlightInfo. Happy to remove it, I don't think its necessary and
> >> already
> >> > a few people agree. Similar with the method name, I am neutral to the
> >> > naming and can call it whatever the community is happy with.
> >> >
> >> > Best,
> >> > Ryan
> >> >
> >> > On Thu, Aug 1, 2019 at 3:56 PM Wes McKinney <we...@gmail.com>
> >> wrote:
> >> >
> >> > > I'm generally supporting of adding the new RPC endpoint.
> >> > >
> >> > > To make a couple points from the document
> >> > >
> >> > > * I'm not sure what the purpose of returning the FlightDescriptor is,
> >> > > but I haven't thought too hard about it
> >> > > * The Schema consists of a single IPC message -- dictionaries will
> >> > > appear in the actual DoGet stream. To motivate why this is --
> >> > > different endpoints might have different dictionaries corresponding to
> >> > > fields in the schema, to have static/constant dictionaries in a
> >> > > distributed Flight setting is likely to be impractical. I summarize
> >> > > the issue as "dictionaries are data, not metadata".
> >> > > * I would be OK calling this GetSchema instead of GetFlightSchema but
> >> > > either is okay
> >> > >
> >> > > - Wes
> >> > >
> >> > > On Thu, Aug 1, 2019 at 8:08 AM David Li <li...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > Hi Ryan,
> >> > > >
> >> > > > Thanks for writing this up! I made a couple of minor comments in the
> >> > > > doc/implementation, but overall I'm in favor of having this RPC
> >> > > > method.
> >> > > >
> >> > > > Best,
> >> > > > David
> >> > > >
> >> > > > On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
> >> > > > > Hi All,
> >> > > > >
> >> > > > > Please see the attached document for a proposed addition to the
> >> > Flight
> >> > > > > RPC[1]. This is the result of a previous mailing list
> >> discussion[2].
> >> > > > >
> >> > > > > I have created the Pull Request[3] to make the proposal a little
> >> more
> >> > > > > concrete.
> >> > > > > <https://www.dremio.com/download>
> >> > > > > Please let me know if you have any questions or concerns.
> >> > > > >
> >> > > > > Best,
> >> > > > > Ryan
> >> > > > >
> >> > > > > [1]:
> >> > > > >
> >> > >
> >> >
> >> https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
> >> > > > > [2]:
> >> > > > >
> >> > >
> >> >
> >> https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
> >> > > > > [3]: https://github.com/apache/arrow/pull/4980
> >> > > > >
> >> > >
> >> >
> >> >
> >> > --
> >> >
> >> > Ryan Murray | Principal Consulting Engineer
> >> >
> >> > +447540852009 | rymurr@dremio.com
> >> >
> >> > <https://www.dremio.com/>
> >> > Check out our GitHub <https://www.github.com/dremio>, join our
> >> community
> >> > site <https://community.dremio.com/> & Download Dremio
> >> > <https://www.dremio.com/download>
> >> >
> >>
> >
> >
> > --
> >
> > Ryan Murray | Principal Consulting Engineer
> >
> > +447540852009 | rymurr@dremio.com
> >
> > <https://www.dremio.com/>
> > Check out our GitHub <https://www.github.com/dremio>, join our community
> > site <https://community.dremio.com/> & Download Dremio
> > <https://www.dremio.com/download>
> >
>
>
> --
>
> Ryan Murray | Principal Consulting Engineer
>
> +447540852009 | rymurr@dremio.com
>
> <https://www.dremio.com/>
> Check out our GitHub <https://www.github.com/dremio>, join our community
> site <https://community.dremio.com/> & Download Dremio
> <https://www.dremio.com/download>
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by Ryan Murray <ry...@dremio.com>.
Hi All,
Does this require a vote? If yes what is the process for initiating one &
if no I hope this is enough time for feedback and I would like to remove
the draft designation from the PR
Best,
Ryan
On Wed, Aug 7, 2019 at 9:31 AM Ryan Murray <ry...@dremio.com> wrote:
> As per everyone's feedback I have renamed GetFlightSchema -> GetSchema and
> have removed the descriptor on the rpc result message. The doc has been
> updated as has the draft PR
>
> On Thu, Aug 1, 2019 at 6:32 PM Bryan Cutler <cu...@gmail.com> wrote:
>
>> Sounds good to me, I would just echo what others have said.
>>
>> On Thu, Aug 1, 2019 at 8:17 AM Ryan Murray <ry...@dremio.com> wrote:
>>
>> > Thanks Wes,
>> >
>> > The descriptor is only there to maintain a bit of symmetry with
>> > GetFlightInfo. Happy to remove it, I don't think its necessary and
>> already
>> > a few people agree. Similar with the method name, I am neutral to the
>> > naming and can call it whatever the community is happy with.
>> >
>> > Best,
>> > Ryan
>> >
>> > On Thu, Aug 1, 2019 at 3:56 PM Wes McKinney <we...@gmail.com>
>> wrote:
>> >
>> > > I'm generally supporting of adding the new RPC endpoint.
>> > >
>> > > To make a couple points from the document
>> > >
>> > > * I'm not sure what the purpose of returning the FlightDescriptor is,
>> > > but I haven't thought too hard about it
>> > > * The Schema consists of a single IPC message -- dictionaries will
>> > > appear in the actual DoGet stream. To motivate why this is --
>> > > different endpoints might have different dictionaries corresponding to
>> > > fields in the schema, to have static/constant dictionaries in a
>> > > distributed Flight setting is likely to be impractical. I summarize
>> > > the issue as "dictionaries are data, not metadata".
>> > > * I would be OK calling this GetSchema instead of GetFlightSchema but
>> > > either is okay
>> > >
>> > > - Wes
>> > >
>> > > On Thu, Aug 1, 2019 at 8:08 AM David Li <li...@gmail.com>
>> wrote:
>> > > >
>> > > > Hi Ryan,
>> > > >
>> > > > Thanks for writing this up! I made a couple of minor comments in the
>> > > > doc/implementation, but overall I'm in favor of having this RPC
>> > > > method.
>> > > >
>> > > > Best,
>> > > > David
>> > > >
>> > > > On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
>> > > > > Hi All,
>> > > > >
>> > > > > Please see the attached document for a proposed addition to the
>> > Flight
>> > > > > RPC[1]. This is the result of a previous mailing list
>> discussion[2].
>> > > > >
>> > > > > I have created the Pull Request[3] to make the proposal a little
>> more
>> > > > > concrete.
>> > > > > <https://www.dremio.com/download>
>> > > > > Please let me know if you have any questions or concerns.
>> > > > >
>> > > > > Best,
>> > > > > Ryan
>> > > > >
>> > > > > [1]:
>> > > > >
>> > >
>> >
>> https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
>> > > > > [2]:
>> > > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
>> > > > > [3]: https://github.com/apache/arrow/pull/4980
>> > > > >
>> > >
>> >
>> >
>> > --
>> >
>> > Ryan Murray | Principal Consulting Engineer
>> >
>> > +447540852009 | rymurr@dremio.com
>> >
>> > <https://www.dremio.com/>
>> > Check out our GitHub <https://www.github.com/dremio>, join our
>> community
>> > site <https://community.dremio.com/> & Download Dremio
>> > <https://www.dremio.com/download>
>> >
>>
>
>
> --
>
> Ryan Murray | Principal Consulting Engineer
>
> +447540852009 | rymurr@dremio.com
>
> <https://www.dremio.com/>
> Check out our GitHub <https://www.github.com/dremio>, join our community
> site <https://community.dremio.com/> & Download Dremio
> <https://www.dremio.com/download>
>
--
Ryan Murray | Principal Consulting Engineer
+447540852009 | rymurr@dremio.com
<https://www.dremio.com/>
Check out our GitHub <https://www.github.com/dremio>, join our community
site <https://community.dremio.com/> & Download Dremio
<https://www.dremio.com/download>
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by Ryan Murray <ry...@dremio.com>.
As per everyone's feedback I have renamed GetFlightSchema -> GetSchema and
have removed the descriptor on the rpc result message. The doc has been
updated as has the draft PR
On Thu, Aug 1, 2019 at 6:32 PM Bryan Cutler <cu...@gmail.com> wrote:
> Sounds good to me, I would just echo what others have said.
>
> On Thu, Aug 1, 2019 at 8:17 AM Ryan Murray <ry...@dremio.com> wrote:
>
> > Thanks Wes,
> >
> > The descriptor is only there to maintain a bit of symmetry with
> > GetFlightInfo. Happy to remove it, I don't think its necessary and
> already
> > a few people agree. Similar with the method name, I am neutral to the
> > naming and can call it whatever the community is happy with.
> >
> > Best,
> > Ryan
> >
> > On Thu, Aug 1, 2019 at 3:56 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > I'm generally supporting of adding the new RPC endpoint.
> > >
> > > To make a couple points from the document
> > >
> > > * I'm not sure what the purpose of returning the FlightDescriptor is,
> > > but I haven't thought too hard about it
> > > * The Schema consists of a single IPC message -- dictionaries will
> > > appear in the actual DoGet stream. To motivate why this is --
> > > different endpoints might have different dictionaries corresponding to
> > > fields in the schema, to have static/constant dictionaries in a
> > > distributed Flight setting is likely to be impractical. I summarize
> > > the issue as "dictionaries are data, not metadata".
> > > * I would be OK calling this GetSchema instead of GetFlightSchema but
> > > either is okay
> > >
> > > - Wes
> > >
> > > On Thu, Aug 1, 2019 at 8:08 AM David Li <li...@gmail.com> wrote:
> > > >
> > > > Hi Ryan,
> > > >
> > > > Thanks for writing this up! I made a couple of minor comments in the
> > > > doc/implementation, but overall I'm in favor of having this RPC
> > > > method.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
> > > > > Hi All,
> > > > >
> > > > > Please see the attached document for a proposed addition to the
> > Flight
> > > > > RPC[1]. This is the result of a previous mailing list
> discussion[2].
> > > > >
> > > > > I have created the Pull Request[3] to make the proposal a little
> more
> > > > > concrete.
> > > > > <https://www.dremio.com/download>
> > > > > Please let me know if you have any questions or concerns.
> > > > >
> > > > > Best,
> > > > > Ryan
> > > > >
> > > > > [1]:
> > > > >
> > >
> >
> https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
> > > > > [2]:
> > > > >
> > >
> >
> https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
> > > > > [3]: https://github.com/apache/arrow/pull/4980
> > > > >
> > >
> >
> >
> > --
> >
> > Ryan Murray | Principal Consulting Engineer
> >
> > +447540852009 | rymurr@dremio.com
> >
> > <https://www.dremio.com/>
> > Check out our GitHub <https://www.github.com/dremio>, join our community
> > site <https://community.dremio.com/> & Download Dremio
> > <https://www.dremio.com/download>
> >
>
--
Ryan Murray | Principal Consulting Engineer
+447540852009 | rymurr@dremio.com
<https://www.dremio.com/>
Check out our GitHub <https://www.github.com/dremio>, join our community
site <https://community.dremio.com/> & Download Dremio
<https://www.dremio.com/download>
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by Bryan Cutler <cu...@gmail.com>.
Sounds good to me, I would just echo what others have said.
On Thu, Aug 1, 2019 at 8:17 AM Ryan Murray <ry...@dremio.com> wrote:
> Thanks Wes,
>
> The descriptor is only there to maintain a bit of symmetry with
> GetFlightInfo. Happy to remove it, I don't think its necessary and already
> a few people agree. Similar with the method name, I am neutral to the
> naming and can call it whatever the community is happy with.
>
> Best,
> Ryan
>
> On Thu, Aug 1, 2019 at 3:56 PM Wes McKinney <we...@gmail.com> wrote:
>
> > I'm generally supporting of adding the new RPC endpoint.
> >
> > To make a couple points from the document
> >
> > * I'm not sure what the purpose of returning the FlightDescriptor is,
> > but I haven't thought too hard about it
> > * The Schema consists of a single IPC message -- dictionaries will
> > appear in the actual DoGet stream. To motivate why this is --
> > different endpoints might have different dictionaries corresponding to
> > fields in the schema, to have static/constant dictionaries in a
> > distributed Flight setting is likely to be impractical. I summarize
> > the issue as "dictionaries are data, not metadata".
> > * I would be OK calling this GetSchema instead of GetFlightSchema but
> > either is okay
> >
> > - Wes
> >
> > On Thu, Aug 1, 2019 at 8:08 AM David Li <li...@gmail.com> wrote:
> > >
> > > Hi Ryan,
> > >
> > > Thanks for writing this up! I made a couple of minor comments in the
> > > doc/implementation, but overall I'm in favor of having this RPC
> > > method.
> > >
> > > Best,
> > > David
> > >
> > > On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
> > > > Hi All,
> > > >
> > > > Please see the attached document for a proposed addition to the
> Flight
> > > > RPC[1]. This is the result of a previous mailing list discussion[2].
> > > >
> > > > I have created the Pull Request[3] to make the proposal a little more
> > > > concrete.
> > > > <https://www.dremio.com/download>
> > > > Please let me know if you have any questions or concerns.
> > > >
> > > > Best,
> > > > Ryan
> > > >
> > > > [1]:
> > > >
> >
> https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
> > > > [2]:
> > > >
> >
> https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
> > > > [3]: https://github.com/apache/arrow/pull/4980
> > > >
> >
>
>
> --
>
> Ryan Murray | Principal Consulting Engineer
>
> +447540852009 | rymurr@dremio.com
>
> <https://www.dremio.com/>
> Check out our GitHub <https://www.github.com/dremio>, join our community
> site <https://community.dremio.com/> & Download Dremio
> <https://www.dremio.com/download>
>
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by Ryan Murray <ry...@dremio.com>.
Thanks Wes,
The descriptor is only there to maintain a bit of symmetry with
GetFlightInfo. Happy to remove it, I don't think its necessary and already
a few people agree. Similar with the method name, I am neutral to the
naming and can call it whatever the community is happy with.
Best,
Ryan
On Thu, Aug 1, 2019 at 3:56 PM Wes McKinney <we...@gmail.com> wrote:
> I'm generally supporting of adding the new RPC endpoint.
>
> To make a couple points from the document
>
> * I'm not sure what the purpose of returning the FlightDescriptor is,
> but I haven't thought too hard about it
> * The Schema consists of a single IPC message -- dictionaries will
> appear in the actual DoGet stream. To motivate why this is --
> different endpoints might have different dictionaries corresponding to
> fields in the schema, to have static/constant dictionaries in a
> distributed Flight setting is likely to be impractical. I summarize
> the issue as "dictionaries are data, not metadata".
> * I would be OK calling this GetSchema instead of GetFlightSchema but
> either is okay
>
> - Wes
>
> On Thu, Aug 1, 2019 at 8:08 AM David Li <li...@gmail.com> wrote:
> >
> > Hi Ryan,
> >
> > Thanks for writing this up! I made a couple of minor comments in the
> > doc/implementation, but overall I'm in favor of having this RPC
> > method.
> >
> > Best,
> > David
> >
> > On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
> > > Hi All,
> > >
> > > Please see the attached document for a proposed addition to the Flight
> > > RPC[1]. This is the result of a previous mailing list discussion[2].
> > >
> > > I have created the Pull Request[3] to make the proposal a little more
> > > concrete.
> > > <https://www.dremio.com/download>
> > > Please let me know if you have any questions or concerns.
> > >
> > > Best,
> > > Ryan
> > >
> > > [1]:
> > >
> https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
> > > [2]:
> > >
> https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
> > > [3]: https://github.com/apache/arrow/pull/4980
> > >
>
--
Ryan Murray | Principal Consulting Engineer
+447540852009 | rymurr@dremio.com
<https://www.dremio.com/>
Check out our GitHub <https://www.github.com/dremio>, join our community
site <https://community.dremio.com/> & Download Dremio
<https://www.dremio.com/download>
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by Wes McKinney <we...@gmail.com>.
I'm generally supporting of adding the new RPC endpoint.
To make a couple points from the document
* I'm not sure what the purpose of returning the FlightDescriptor is,
but I haven't thought too hard about it
* The Schema consists of a single IPC message -- dictionaries will
appear in the actual DoGet stream. To motivate why this is --
different endpoints might have different dictionaries corresponding to
fields in the schema, to have static/constant dictionaries in a
distributed Flight setting is likely to be impractical. I summarize
the issue as "dictionaries are data, not metadata".
* I would be OK calling this GetSchema instead of GetFlightSchema but
either is okay
- Wes
On Thu, Aug 1, 2019 at 8:08 AM David Li <li...@gmail.com> wrote:
>
> Hi Ryan,
>
> Thanks for writing this up! I made a couple of minor comments in the
> doc/implementation, but overall I'm in favor of having this RPC
> method.
>
> Best,
> David
>
> On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
> > Hi All,
> >
> > Please see the attached document for a proposed addition to the Flight
> > RPC[1]. This is the result of a previous mailing list discussion[2].
> >
> > I have created the Pull Request[3] to make the proposal a little more
> > concrete.
> > <https://www.dremio.com/download>
> > Please let me know if you have any questions or concerns.
> >
> > Best,
> > Ryan
> >
> > [1]:
> > https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
> > [2]:
> > https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
> > [3]: https://github.com/apache/arrow/pull/4980
> >
Re: [DISCUSS] Add GetFlightSchema to Flight RPC
Posted by David Li <li...@gmail.com>.
Hi Ryan,
Thanks for writing this up! I made a couple of minor comments in the
doc/implementation, but overall I'm in favor of having this RPC
method.
Best,
David
On 8/1/19, Ryan Murray <ry...@dremio.com> wrote:
> Hi All,
>
> Please see the attached document for a proposed addition to the Flight
> RPC[1]. This is the result of a previous mailing list discussion[2].
>
> I have created the Pull Request[3] to make the proposal a little more
> concrete.
> <https://www.dremio.com/download>
> Please let me know if you have any questions or concerns.
>
> Best,
> Ryan
>
> [1]:
> https://docs.google.com/document/d/1zLdFYikk3owbKpHvJrARLMlmYpi-Ef6OJy7H90MqViA/edit?usp=sharing
> [2]:
> https://lists.apache.org/thread.html/3539984493cf3d4d439bef25c150fa9e09e0b43ce0afb6be378d41df@%3Cdev.arrow.apache.org%3E
> [3]: https://github.com/apache/arrow/pull/4980
>