You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ryan Nicholson <rn...@gmail.com> on 2020/08/31 21:21:25 UTC
[FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Hello everyone,
I would like to propose the following specification to ease adoption of SQL
based clients and backends while leveraging data streams for transporting
data. This specification entails a series of protobuf messages to be used
in the opaque messaging framework payloads to enable browsing of database
metadata and statement style operations with scaffolding available to
help implementers provide logic for their backend.
Please take a look at the following material:
- ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825> has more
details around the goals of the proposal.
- Flight SQL Proposal
<https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing>
is
a more detailed document describing new messages and how they would be used.
- https://github.com/ryannicholson/arrow/pull/2 on my Arrow fork shows a
sample implementation with a catalog browsing action and a prepared
statement query workflow. There are details in the proposal linking to
specific areas of the PR.
Thank you for your time,
Ryan Nicholson
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by David Li <li...@apache.org>.
Hi Tiffany,
I was just curious about the status of the FlightSQL proposal. Given
that the 4.0.0 development cycle will be concluding soon, it doesn't
seem FlightSQL will make it, but is there interest in trying to get it
into 5.0.0?
Similarly, there was the related effort to expand Flight's auth
capabilities which also seems to have been inactive lately - is there
anything we can do to help keep that moving forward as well?
Best,
David
On 2021/02/09 01:52:45, Hoi Ling Tiffany Lam <ti...@gmail.com> wrote:
> HI David,
>
> I will be taking over the work for Flight SQL. Last week, I created a draft
> PR to Arrow Main <https://github.com/apache/arrow/pull/9368> that addresses
> the original comments on this PR
> <https://github.com/ryannicholson/arrow/pull/2>. It would be amazing if you
> could take a look at the draft PR. The work on Flight SQL is temporarily
> paused on my end, I will be resuming the work next week. I am also going to
> send out an email to the dev mailing list once the work on FLight SQL
> resumes next week. Hopefully I will be able to concentrate on Flight SQL
> from next week onwards. Exciting times ahead!
>
> Cheers,
> Tiffany Lam (She/Her)
>
> On Mon, Feb 8, 2021 at 10:16 AM David Li <li...@apache.org> wrote:
>
> > Hi Ryan et. al.,
> >
> > Following up here- I saw Dremio gave a presentation about this project
> > recently, and also, over the past several months, the authentication work
> > has been proceeding. Given that, what does the status of the Flight SQL
> > project itself look like now? I see there are some unanswered comments in
> > the doc and in the draft PR, would another round of comments be useful if
> > those were addressed? And the talk mentioned a release in the next few
> > months, are there any details you can share there?
> >
> > Thanks,
> > David
> >
> > On Thu, Oct 8, 2020, at 13:28, Ryan Nicholson wrote:
> > > Thanks Wes,
> > >
> > > Apologies for the delay. The next steps which we were going to pursue
> > once
> > > the authentication redesign direction was solid included a session
> > concept.
> > > The initial proposal we would put forth would likely include these
> > concepts
> > > via use of headers. I agree that with sessions and databases in mind, a
> > > close operation would be beneficial. In thinking through this earlier it
> > > wasn't clear to me if it would make sense to implement "close" as an
> > Action
> > > but it also didn't seem to fit as a new first class Flight RPC
> > altogether.
> > >
> > > Ryan
> > >
> > >
> > >
> > > On Thu, Oct 1, 2020 at 3:44 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > > I just reviewed the FlightSQL.proto on
> > > > https://github.com/ryannicholson/arrow/pull/2. It might be a good idea
> > > > to document somewhere how some of the various SQL concepts (like
> > > > sessions) are intended to be handled in Flight (which doesn't have
> > > > these concepts built in). Does it make sense to have
> > > > session-lifetime-related RPCs (start/close session, check session
> > > > status, include a session bytes token with requests, etc.)?
> > > >
> > > > On Tue, Sep 29, 2020 at 6:53 PM Ryan Nicholson <rn...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hello Andy and Wes,
> > > > >
> > > > > Thanks for reaching out.
> > > > >
> > > > > Later this week I have time freeing up to start on Flight-SQL. Would
> > > > either
> > > > > of you mind examining the approach in the POC [1] as described in the
> > > > > "Proof of Concept" section of the proposal document [2]?
> > > > >
> > > > > I would certainly be happy to chat about this with anyone interested
> > in
> > > > > participating too.
> > > > >
> > > > > Thank you,
> > > > > Ryan
> > > > >
> > > > > [1] https://github.com/ryannicholson/arrow/pull/2
> > > > > [2]
> > > > >
> > > >
> > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI
> > > > >
> > > > >
> > > > > On Tue, Sep 29, 2020 at 8:52 AM Andy Grove <an...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I am also interested in helping with this effort.
> > > > > >
> > > > > > Andy.
> > > > > >
> > > > > > On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > hi Ryan,
> > > > > > >
> > > > > > > I'm interested in enabling this effort to move forward -- is
> > there
> > > > any
> > > > > > > part where feedback or other help could be useful?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Wes
> > > > > > >
> > > > > > > On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <wesmckinn@gmail.com
> > >
> > > > wrote:
> > > > > > > >
> > > > > > > > I haven't been able to scrutinize the details greatly, but I am
> > > > > > > > supportive of this effort, otherwise we may end up with many
> > > > database
> > > > > > > > systems and vendors implementing Flight in different ways. It
> > > > would be
> > > > > > > > nice if there was a built-in SQL client layer that we ship in
> > C++
> > > > (and
> > > > > > > > its bindings), Java, and others.
> > > > > > > >
> > > > > > > > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <
> > > > rnicholson@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > Here is a link to the slides
> > > > > > > > > <
> > > > > > >
> > > > > >
> > > >
> > https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> > > > > > > >
> > > > > > > > > I would like to discuss this morning.
> > > > > > > > >
> > > > > > > > > Take care,
> > > > > > > > > Ryan Nicholson
> > > > > > > > >
> > > > > > > > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <
> > > > rnicholson@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you to all who have viewed and given feedback in the
> > > > posted
> > > > > > > material
> > > > > > > > > > so far. I am working on responses as well as preparing some
> > > > > > material
> > > > > > > which
> > > > > > > > > > I would like to share for discussion in the sync call
> > > > tomorrow. I
> > > > > > > will post
> > > > > > > > > > the material here ahead of time.
> > > > > > > > > >
> > > > > > > > > > Take care,
> > > > > > > > > > Ryan Nicholson
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <
> > > > > > rnicholson@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hello everyone,
> > > > > > > > > >>
> > > > > > > > > >> I would like to propose the following specification to
> > ease
> > > > > > > adoption of
> > > > > > > > > >> SQL based clients and backends while leveraging data
> > streams
> > > > for
> > > > > > > > > >> transporting data. This specification entails a series of
> > > > protobuf
> > > > > > > messages
> > > > > > > > > >> to be used in the opaque messaging framework payloads to
> > > > enable
> > > > > > > browsing of
> > > > > > > > > >> database metadata and statement style operations with
> > > > scaffolding
> > > > > > > available
> > > > > > > > > >> to help implementers provide logic for their backend.
> > > > > > > > > >>
> > > > > > > > > >> Please take a look at the following material:
> > > > > > > > > >>
> > > > > > > > > >> - ARROW-9825 <
> > > > https://issues.apache.org/jira/browse/ARROW-9825
> > > > > > >
> > > > > > > has
> > > > > > > > > >> more details around the goals of the proposal.
> > > > > > > > > >> - Flight SQL Proposal
> > > > > > > > > >> <
> > > > > > >
> > > > > >
> > > >
> > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing
> > > > > > >
> > > > > > > is
> > > > > > > > > >> a more detailed document describing new messages and
> > how
> > > > they
> > > > > > > would be used.
> > > > > > > > > >> - https://github.com/ryannicholson/arrow/pull/2 on my
> > > > Arrow
> > > > > > fork
> > > > > > > > > >> shows a sample implementation with a catalog browsing
> > > > action
> > > > > > and
> > > > > > > a prepared
> > > > > > > > > >> statement query workflow. There are details in the
> > proposal
> > > > > > > linking to
> > > > > > > > > >> specific areas of the PR.
> > > > > > > > > >>
> > > > > > > > > >> Thank you for your time,
> > > > > > > > > >> Ryan Nicholson
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > >
>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by David Li <li...@apache.org>.
Hi Tiffany,
Great! Thanks for the update, I'll take another look today.
Best,
David
On Mon, Feb 8, 2021, at 20:52, Hoi Ling Tiffany Lam wrote:
> HI David,
>
> I will be taking over the work for Flight SQL. Last week, I created a draft
> PR to Arrow Main <https://github.com/apache/arrow/pull/9368> that addresses
> the original comments on this PR
> <https://github.com/ryannicholson/arrow/pull/2>. It would be amazing if you
> could take a look at the draft PR. The work on Flight SQL is temporarily
> paused on my end, I will be resuming the work next week. I am also going to
> send out an email to the dev mailing list once the work on FLight SQL
> resumes next week. Hopefully I will be able to concentrate on Flight SQL
> from next week onwards. Exciting times ahead!
>
> Cheers,
> Tiffany Lam (She/Her)
>
> On Mon, Feb 8, 2021 at 10:16 AM David Li <li...@apache.org> wrote:
>
> > Hi Ryan et. al.,
> >
> > Following up here- I saw Dremio gave a presentation about this project
> > recently, and also, over the past several months, the authentication work
> > has been proceeding. Given that, what does the status of the Flight SQL
> > project itself look like now? I see there are some unanswered comments in
> > the doc and in the draft PR, would another round of comments be useful if
> > those were addressed? And the talk mentioned a release in the next few
> > months, are there any details you can share there?
> >
> > Thanks,
> > David
> >
> > On Thu, Oct 8, 2020, at 13:28, Ryan Nicholson wrote:
> > > Thanks Wes,
> > >
> > > Apologies for the delay. The next steps which we were going to pursue
> > once
> > > the authentication redesign direction was solid included a session
> > concept.
> > > The initial proposal we would put forth would likely include these
> > concepts
> > > via use of headers. I agree that with sessions and databases in mind, a
> > > close operation would be beneficial. In thinking through this earlier it
> > > wasn't clear to me if it would make sense to implement "close" as an
> > Action
> > > but it also didn't seem to fit as a new first class Flight RPC
> > altogether.
> > >
> > > Ryan
> > >
> > >
> > >
> > > On Thu, Oct 1, 2020 at 3:44 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > > I just reviewed the FlightSQL.proto on
> > > > https://github.com/ryannicholson/arrow/pull/2. It might be a good idea
> > > > to document somewhere how some of the various SQL concepts (like
> > > > sessions) are intended to be handled in Flight (which doesn't have
> > > > these concepts built in). Does it make sense to have
> > > > session-lifetime-related RPCs (start/close session, check session
> > > > status, include a session bytes token with requests, etc.)?
> > > >
> > > > On Tue, Sep 29, 2020 at 6:53 PM Ryan Nicholson <rn...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hello Andy and Wes,
> > > > >
> > > > > Thanks for reaching out.
> > > > >
> > > > > Later this week I have time freeing up to start on Flight-SQL. Would
> > > > either
> > > > > of you mind examining the approach in the POC [1] as described in the
> > > > > "Proof of Concept" section of the proposal document [2]?
> > > > >
> > > > > I would certainly be happy to chat about this with anyone interested
> > in
> > > > > participating too.
> > > > >
> > > > > Thank you,
> > > > > Ryan
> > > > >
> > > > > [1] https://github.com/ryannicholson/arrow/pull/2
> > > > > [2]
> > > > >
> > > >
> > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI
> > > > >
> > > > >
> > > > > On Tue, Sep 29, 2020 at 8:52 AM Andy Grove <an...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I am also interested in helping with this effort.
> > > > > >
> > > > > > Andy.
> > > > > >
> > > > > > On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > hi Ryan,
> > > > > > >
> > > > > > > I'm interested in enabling this effort to move forward -- is
> > there
> > > > any
> > > > > > > part where feedback or other help could be useful?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Wes
> > > > > > >
> > > > > > > On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <wesmckinn@gmail.com
> > >
> > > > wrote:
> > > > > > > >
> > > > > > > > I haven't been able to scrutinize the details greatly, but I am
> > > > > > > > supportive of this effort, otherwise we may end up with many
> > > > database
> > > > > > > > systems and vendors implementing Flight in different ways. It
> > > > would be
> > > > > > > > nice if there was a built-in SQL client layer that we ship in
> > C++
> > > > (and
> > > > > > > > its bindings), Java, and others.
> > > > > > > >
> > > > > > > > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <
> > > > rnicholson@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > Here is a link to the slides
> > > > > > > > > <
> > > > > > >
> > > > > >
> > > >
> > https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> > > > > > > >
> > > > > > > > > I would like to discuss this morning.
> > > > > > > > >
> > > > > > > > > Take care,
> > > > > > > > > Ryan Nicholson
> > > > > > > > >
> > > > > > > > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <
> > > > rnicholson@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you to all who have viewed and given feedback in the
> > > > posted
> > > > > > > material
> > > > > > > > > > so far. I am working on responses as well as preparing some
> > > > > > material
> > > > > > > which
> > > > > > > > > > I would like to share for discussion in the sync call
> > > > tomorrow. I
> > > > > > > will post
> > > > > > > > > > the material here ahead of time.
> > > > > > > > > >
> > > > > > > > > > Take care,
> > > > > > > > > > Ryan Nicholson
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <
> > > > > > rnicholson@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hello everyone,
> > > > > > > > > >>
> > > > > > > > > >> I would like to propose the following specification to
> > ease
> > > > > > > adoption of
> > > > > > > > > >> SQL based clients and backends while leveraging data
> > streams
> > > > for
> > > > > > > > > >> transporting data. This specification entails a series of
> > > > protobuf
> > > > > > > messages
> > > > > > > > > >> to be used in the opaque messaging framework payloads to
> > > > enable
> > > > > > > browsing of
> > > > > > > > > >> database metadata and statement style operations with
> > > > scaffolding
> > > > > > > available
> > > > > > > > > >> to help implementers provide logic for their backend.
> > > > > > > > > >>
> > > > > > > > > >> Please take a look at the following material:
> > > > > > > > > >>
> > > > > > > > > >> - ARROW-9825 <
> > > > https://issues.apache.org/jira/browse/ARROW-9825
> > > > > > >
> > > > > > > has
> > > > > > > > > >> more details around the goals of the proposal.
> > > > > > > > > >> - Flight SQL Proposal
> > > > > > > > > >> <
> > > > > > >
> > > > > >
> > > >
> > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing
> > > > > > >
> > > > > > > is
> > > > > > > > > >> a more detailed document describing new messages and
> > how
> > > > they
> > > > > > > would be used.
> > > > > > > > > >> - https://github.com/ryannicholson/arrow/pull/2 on my
> > > > Arrow
> > > > > > fork
> > > > > > > > > >> shows a sample implementation with a catalog browsing
> > > > action
> > > > > > and
> > > > > > > a prepared
> > > > > > > > > >> statement query workflow. There are details in the
> > proposal
> > > > > > > linking to
> > > > > > > > > >> specific areas of the PR.
> > > > > > > > > >>
> > > > > > > > > >> Thank you for your time,
> > > > > > > > > >> Ryan Nicholson
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > >
>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Hoi Ling Tiffany Lam <ti...@gmail.com>.
HI David,
I will be taking over the work for Flight SQL. Last week, I created a draft
PR to Arrow Main <https://github.com/apache/arrow/pull/9368> that addresses
the original comments on this PR
<https://github.com/ryannicholson/arrow/pull/2>. It would be amazing if you
could take a look at the draft PR. The work on Flight SQL is temporarily
paused on my end, I will be resuming the work next week. I am also going to
send out an email to the dev mailing list once the work on FLight SQL
resumes next week. Hopefully I will be able to concentrate on Flight SQL
from next week onwards. Exciting times ahead!
Cheers,
Tiffany Lam (She/Her)
On Mon, Feb 8, 2021 at 10:16 AM David Li <li...@apache.org> wrote:
> Hi Ryan et. al.,
>
> Following up here- I saw Dremio gave a presentation about this project
> recently, and also, over the past several months, the authentication work
> has been proceeding. Given that, what does the status of the Flight SQL
> project itself look like now? I see there are some unanswered comments in
> the doc and in the draft PR, would another round of comments be useful if
> those were addressed? And the talk mentioned a release in the next few
> months, are there any details you can share there?
>
> Thanks,
> David
>
> On Thu, Oct 8, 2020, at 13:28, Ryan Nicholson wrote:
> > Thanks Wes,
> >
> > Apologies for the delay. The next steps which we were going to pursue
> once
> > the authentication redesign direction was solid included a session
> concept.
> > The initial proposal we would put forth would likely include these
> concepts
> > via use of headers. I agree that with sessions and databases in mind, a
> > close operation would be beneficial. In thinking through this earlier it
> > wasn't clear to me if it would make sense to implement "close" as an
> Action
> > but it also didn't seem to fit as a new first class Flight RPC
> altogether.
> >
> > Ryan
> >
> >
> >
> > On Thu, Oct 1, 2020 at 3:44 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > I just reviewed the FlightSQL.proto on
> > > https://github.com/ryannicholson/arrow/pull/2. It might be a good idea
> > > to document somewhere how some of the various SQL concepts (like
> > > sessions) are intended to be handled in Flight (which doesn't have
> > > these concepts built in). Does it make sense to have
> > > session-lifetime-related RPCs (start/close session, check session
> > > status, include a session bytes token with requests, etc.)?
> > >
> > > On Tue, Sep 29, 2020 at 6:53 PM Ryan Nicholson <rn...@gmail.com>
> > > wrote:
> > > >
> > > > Hello Andy and Wes,
> > > >
> > > > Thanks for reaching out.
> > > >
> > > > Later this week I have time freeing up to start on Flight-SQL. Would
> > > either
> > > > of you mind examining the approach in the POC [1] as described in the
> > > > "Proof of Concept" section of the proposal document [2]?
> > > >
> > > > I would certainly be happy to chat about this with anyone interested
> in
> > > > participating too.
> > > >
> > > > Thank you,
> > > > Ryan
> > > >
> > > > [1] https://github.com/ryannicholson/arrow/pull/2
> > > > [2]
> > > >
> > >
> https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI
> > > >
> > > >
> > > > On Tue, Sep 29, 2020 at 8:52 AM Andy Grove <an...@gmail.com>
> > > wrote:
> > > >
> > > > > I am also interested in helping with this effort.
> > > > >
> > > > > Andy.
> > > > >
> > > > > On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > > > hi Ryan,
> > > > > >
> > > > > > I'm interested in enabling this effort to move forward -- is
> there
> > > any
> > > > > > part where feedback or other help could be useful?
> > > > > >
> > > > > > Thanks,
> > > > > > Wes
> > > > > >
> > > > > > On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <wesmckinn@gmail.com
> >
> > > wrote:
> > > > > > >
> > > > > > > I haven't been able to scrutinize the details greatly, but I am
> > > > > > > supportive of this effort, otherwise we may end up with many
> > > database
> > > > > > > systems and vendors implementing Flight in different ways. It
> > > would be
> > > > > > > nice if there was a built-in SQL client layer that we ship in
> C++
> > > (and
> > > > > > > its bindings), Java, and others.
> > > > > > >
> > > > > > > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <
> > > rnicholson@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Here is a link to the slides
> > > > > > > > <
> > > > > >
> > > > >
> > >
> https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> > > > > > >
> > > > > > > > I would like to discuss this morning.
> > > > > > > >
> > > > > > > > Take care,
> > > > > > > > Ryan Nicholson
> > > > > > > >
> > > > > > > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <
> > > rnicholson@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thank you to all who have viewed and given feedback in the
> > > posted
> > > > > > material
> > > > > > > > > so far. I am working on responses as well as preparing some
> > > > > material
> > > > > > which
> > > > > > > > > I would like to share for discussion in the sync call
> > > tomorrow. I
> > > > > > will post
> > > > > > > > > the material here ahead of time.
> > > > > > > > >
> > > > > > > > > Take care,
> > > > > > > > > Ryan Nicholson
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <
> > > > > rnicholson@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hello everyone,
> > > > > > > > >>
> > > > > > > > >> I would like to propose the following specification to
> ease
> > > > > > adoption of
> > > > > > > > >> SQL based clients and backends while leveraging data
> streams
> > > for
> > > > > > > > >> transporting data. This specification entails a series of
> > > protobuf
> > > > > > messages
> > > > > > > > >> to be used in the opaque messaging framework payloads to
> > > enable
> > > > > > browsing of
> > > > > > > > >> database metadata and statement style operations with
> > > scaffolding
> > > > > > available
> > > > > > > > >> to help implementers provide logic for their backend.
> > > > > > > > >>
> > > > > > > > >> Please take a look at the following material:
> > > > > > > > >>
> > > > > > > > >> - ARROW-9825 <
> > > https://issues.apache.org/jira/browse/ARROW-9825
> > > > > >
> > > > > > has
> > > > > > > > >> more details around the goals of the proposal.
> > > > > > > > >> - Flight SQL Proposal
> > > > > > > > >> <
> > > > > >
> > > > >
> > >
> https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing
> > > > > >
> > > > > > is
> > > > > > > > >> a more detailed document describing new messages and
> how
> > > they
> > > > > > would be used.
> > > > > > > > >> - https://github.com/ryannicholson/arrow/pull/2 on my
> > > Arrow
> > > > > fork
> > > > > > > > >> shows a sample implementation with a catalog browsing
> > > action
> > > > > and
> > > > > > a prepared
> > > > > > > > >> statement query workflow. There are details in the
> proposal
> > > > > > linking to
> > > > > > > > >> specific areas of the PR.
> > > > > > > > >>
> > > > > > > > >> Thank you for your time,
> > > > > > > > >> Ryan Nicholson
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > >
> > > > >
> > >
> >
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by David Li <li...@apache.org>.
Hi Ryan et. al.,
Following up here- I saw Dremio gave a presentation about this project recently, and also, over the past several months, the authentication work has been proceeding. Given that, what does the status of the Flight SQL project itself look like now? I see there are some unanswered comments in the doc and in the draft PR, would another round of comments be useful if those were addressed? And the talk mentioned a release in the next few months, are there any details you can share there?
Thanks,
David
On Thu, Oct 8, 2020, at 13:28, Ryan Nicholson wrote:
> Thanks Wes,
>
> Apologies for the delay. The next steps which we were going to pursue once
> the authentication redesign direction was solid included a session concept.
> The initial proposal we would put forth would likely include these concepts
> via use of headers. I agree that with sessions and databases in mind, a
> close operation would be beneficial. In thinking through this earlier it
> wasn't clear to me if it would make sense to implement "close" as an Action
> but it also didn't seem to fit as a new first class Flight RPC altogether.
>
> Ryan
>
>
>
> On Thu, Oct 1, 2020 at 3:44 PM Wes McKinney <we...@gmail.com> wrote:
>
> > I just reviewed the FlightSQL.proto on
> > https://github.com/ryannicholson/arrow/pull/2. It might be a good idea
> > to document somewhere how some of the various SQL concepts (like
> > sessions) are intended to be handled in Flight (which doesn't have
> > these concepts built in). Does it make sense to have
> > session-lifetime-related RPCs (start/close session, check session
> > status, include a session bytes token with requests, etc.)?
> >
> > On Tue, Sep 29, 2020 at 6:53 PM Ryan Nicholson <rn...@gmail.com>
> > wrote:
> > >
> > > Hello Andy and Wes,
> > >
> > > Thanks for reaching out.
> > >
> > > Later this week I have time freeing up to start on Flight-SQL. Would
> > either
> > > of you mind examining the approach in the POC [1] as described in the
> > > "Proof of Concept" section of the proposal document [2]?
> > >
> > > I would certainly be happy to chat about this with anyone interested in
> > > participating too.
> > >
> > > Thank you,
> > > Ryan
> > >
> > > [1] https://github.com/ryannicholson/arrow/pull/2
> > > [2]
> > >
> > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI
> > >
> > >
> > > On Tue, Sep 29, 2020 at 8:52 AM Andy Grove <an...@gmail.com>
> > wrote:
> > >
> > > > I am also interested in helping with this effort.
> > > >
> > > > Andy.
> > > >
> > > > On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > > hi Ryan,
> > > > >
> > > > > I'm interested in enabling this effort to move forward -- is there
> > any
> > > > > part where feedback or other help could be useful?
> > > > >
> > > > > Thanks,
> > > > > Wes
> > > > >
> > > > > On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > > > >
> > > > > > I haven't been able to scrutinize the details greatly, but I am
> > > > > > supportive of this effort, otherwise we may end up with many
> > database
> > > > > > systems and vendors implementing Flight in different ways. It
> > would be
> > > > > > nice if there was a built-in SQL client layer that we ship in C++
> > (and
> > > > > > its bindings), Java, and others.
> > > > > >
> > > > > > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <
> > rnicholson@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > Here is a link to the slides
> > > > > > > <
> > > > >
> > > >
> > https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> > > > > >
> > > > > > > I would like to discuss this morning.
> > > > > > >
> > > > > > > Take care,
> > > > > > > Ryan Nicholson
> > > > > > >
> > > > > > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <
> > rnicholson@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Thank you to all who have viewed and given feedback in the
> > posted
> > > > > material
> > > > > > > > so far. I am working on responses as well as preparing some
> > > > material
> > > > > which
> > > > > > > > I would like to share for discussion in the sync call
> > tomorrow. I
> > > > > will post
> > > > > > > > the material here ahead of time.
> > > > > > > >
> > > > > > > > Take care,
> > > > > > > > Ryan Nicholson
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <
> > > > rnicholson@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hello everyone,
> > > > > > > >>
> > > > > > > >> I would like to propose the following specification to ease
> > > > > adoption of
> > > > > > > >> SQL based clients and backends while leveraging data streams
> > for
> > > > > > > >> transporting data. This specification entails a series of
> > protobuf
> > > > > messages
> > > > > > > >> to be used in the opaque messaging framework payloads to
> > enable
> > > > > browsing of
> > > > > > > >> database metadata and statement style operations with
> > scaffolding
> > > > > available
> > > > > > > >> to help implementers provide logic for their backend.
> > > > > > > >>
> > > > > > > >> Please take a look at the following material:
> > > > > > > >>
> > > > > > > >> - ARROW-9825 <
> > https://issues.apache.org/jira/browse/ARROW-9825
> > > > >
> > > > > has
> > > > > > > >> more details around the goals of the proposal.
> > > > > > > >> - Flight SQL Proposal
> > > > > > > >> <
> > > > >
> > > >
> > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing
> > > > >
> > > > > is
> > > > > > > >> a more detailed document describing new messages and how
> > they
> > > > > would be used.
> > > > > > > >> - https://github.com/ryannicholson/arrow/pull/2 on my
> > Arrow
> > > > fork
> > > > > > > >> shows a sample implementation with a catalog browsing
> > action
> > > > and
> > > > > a prepared
> > > > > > > >> statement query workflow. There are details in the proposal
> > > > > linking to
> > > > > > > >> specific areas of the PR.
> > > > > > > >>
> > > > > > > >> Thank you for your time,
> > > > > > > >> Ryan Nicholson
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > >
> > > >
> >
>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Ryan Nicholson <rn...@gmail.com>.
Thanks Wes,
Apologies for the delay. The next steps which we were going to pursue once
the authentication redesign direction was solid included a session concept.
The initial proposal we would put forth would likely include these concepts
via use of headers. I agree that with sessions and databases in mind, a
close operation would be beneficial. In thinking through this earlier it
wasn't clear to me if it would make sense to implement "close" as an Action
but it also didn't seem to fit as a new first class Flight RPC altogether.
Ryan
On Thu, Oct 1, 2020 at 3:44 PM Wes McKinney <we...@gmail.com> wrote:
> I just reviewed the FlightSQL.proto on
> https://github.com/ryannicholson/arrow/pull/2. It might be a good idea
> to document somewhere how some of the various SQL concepts (like
> sessions) are intended to be handled in Flight (which doesn't have
> these concepts built in). Does it make sense to have
> session-lifetime-related RPCs (start/close session, check session
> status, include a session bytes token with requests, etc.)?
>
> On Tue, Sep 29, 2020 at 6:53 PM Ryan Nicholson <rn...@gmail.com>
> wrote:
> >
> > Hello Andy and Wes,
> >
> > Thanks for reaching out.
> >
> > Later this week I have time freeing up to start on Flight-SQL. Would
> either
> > of you mind examining the approach in the POC [1] as described in the
> > "Proof of Concept" section of the proposal document [2]?
> >
> > I would certainly be happy to chat about this with anyone interested in
> > participating too.
> >
> > Thank you,
> > Ryan
> >
> > [1] https://github.com/ryannicholson/arrow/pull/2
> > [2]
> >
> https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI
> >
> >
> > On Tue, Sep 29, 2020 at 8:52 AM Andy Grove <an...@gmail.com>
> wrote:
> >
> > > I am also interested in helping with this effort.
> > >
> > > Andy.
> > >
> > > On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > hi Ryan,
> > > >
> > > > I'm interested in enabling this effort to move forward -- is there
> any
> > > > part where feedback or other help could be useful?
> > > >
> > > > Thanks,
> > > > Wes
> > > >
> > > > On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <we...@gmail.com>
> wrote:
> > > > >
> > > > > I haven't been able to scrutinize the details greatly, but I am
> > > > > supportive of this effort, otherwise we may end up with many
> database
> > > > > systems and vendors implementing Flight in different ways. It
> would be
> > > > > nice if there was a built-in SQL client layer that we ship in C++
> (and
> > > > > its bindings), Java, and others.
> > > > >
> > > > > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <
> rnicholson@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > Here is a link to the slides
> > > > > > <
> > > >
> > >
> https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> > > > >
> > > > > > I would like to discuss this morning.
> > > > > >
> > > > > > Take care,
> > > > > > Ryan Nicholson
> > > > > >
> > > > > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <
> rnicholson@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Thank you to all who have viewed and given feedback in the
> posted
> > > > material
> > > > > > > so far. I am working on responses as well as preparing some
> > > material
> > > > which
> > > > > > > I would like to share for discussion in the sync call
> tomorrow. I
> > > > will post
> > > > > > > the material here ahead of time.
> > > > > > >
> > > > > > > Take care,
> > > > > > > Ryan Nicholson
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <
> > > rnicholson@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hello everyone,
> > > > > > >>
> > > > > > >> I would like to propose the following specification to ease
> > > > adoption of
> > > > > > >> SQL based clients and backends while leveraging data streams
> for
> > > > > > >> transporting data. This specification entails a series of
> protobuf
> > > > messages
> > > > > > >> to be used in the opaque messaging framework payloads to
> enable
> > > > browsing of
> > > > > > >> database metadata and statement style operations with
> scaffolding
> > > > available
> > > > > > >> to help implementers provide logic for their backend.
> > > > > > >>
> > > > > > >> Please take a look at the following material:
> > > > > > >>
> > > > > > >> - ARROW-9825 <
> https://issues.apache.org/jira/browse/ARROW-9825
> > > >
> > > > has
> > > > > > >> more details around the goals of the proposal.
> > > > > > >> - Flight SQL Proposal
> > > > > > >> <
> > > >
> > >
> https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing
> > > >
> > > > is
> > > > > > >> a more detailed document describing new messages and how
> they
> > > > would be used.
> > > > > > >> - https://github.com/ryannicholson/arrow/pull/2 on my
> Arrow
> > > fork
> > > > > > >> shows a sample implementation with a catalog browsing
> action
> > > and
> > > > a prepared
> > > > > > >> statement query workflow. There are details in the proposal
> > > > linking to
> > > > > > >> specific areas of the PR.
> > > > > > >>
> > > > > > >> Thank you for your time,
> > > > > > >> Ryan Nicholson
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > >
> > >
>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Wes McKinney <we...@gmail.com>.
I just reviewed the FlightSQL.proto on
https://github.com/ryannicholson/arrow/pull/2. It might be a good idea
to document somewhere how some of the various SQL concepts (like
sessions) are intended to be handled in Flight (which doesn't have
these concepts built in). Does it make sense to have
session-lifetime-related RPCs (start/close session, check session
status, include a session bytes token with requests, etc.)?
On Tue, Sep 29, 2020 at 6:53 PM Ryan Nicholson <rn...@gmail.com> wrote:
>
> Hello Andy and Wes,
>
> Thanks for reaching out.
>
> Later this week I have time freeing up to start on Flight-SQL. Would either
> of you mind examining the approach in the POC [1] as described in the
> "Proof of Concept" section of the proposal document [2]?
>
> I would certainly be happy to chat about this with anyone interested in
> participating too.
>
> Thank you,
> Ryan
>
> [1] https://github.com/ryannicholson/arrow/pull/2
> [2]
> https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI
>
>
> On Tue, Sep 29, 2020 at 8:52 AM Andy Grove <an...@gmail.com> wrote:
>
> > I am also interested in helping with this effort.
> >
> > Andy.
> >
> > On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > > hi Ryan,
> > >
> > > I'm interested in enabling this effort to move forward -- is there any
> > > part where feedback or other help could be useful?
> > >
> > > Thanks,
> > > Wes
> > >
> > > On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <we...@gmail.com> wrote:
> > > >
> > > > I haven't been able to scrutinize the details greatly, but I am
> > > > supportive of this effort, otherwise we may end up with many database
> > > > systems and vendors implementing Flight in different ways. It would be
> > > > nice if there was a built-in SQL client layer that we ship in C++ (and
> > > > its bindings), Java, and others.
> > > >
> > > > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <rn...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hello everyone,
> > > > >
> > > > > Here is a link to the slides
> > > > > <
> > >
> > https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> > > >
> > > > > I would like to discuss this morning.
> > > > >
> > > > > Take care,
> > > > > Ryan Nicholson
> > > > >
> > > > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <rn...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thank you to all who have viewed and given feedback in the posted
> > > material
> > > > > > so far. I am working on responses as well as preparing some
> > material
> > > which
> > > > > > I would like to share for discussion in the sync call tomorrow. I
> > > will post
> > > > > > the material here ahead of time.
> > > > > >
> > > > > > Take care,
> > > > > > Ryan Nicholson
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <
> > rnicholson@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hello everyone,
> > > > > >>
> > > > > >> I would like to propose the following specification to ease
> > > adoption of
> > > > > >> SQL based clients and backends while leveraging data streams for
> > > > > >> transporting data. This specification entails a series of protobuf
> > > messages
> > > > > >> to be used in the opaque messaging framework payloads to enable
> > > browsing of
> > > > > >> database metadata and statement style operations with scaffolding
> > > available
> > > > > >> to help implementers provide logic for their backend.
> > > > > >>
> > > > > >> Please take a look at the following material:
> > > > > >>
> > > > > >> - ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825
> > >
> > > has
> > > > > >> more details around the goals of the proposal.
> > > > > >> - Flight SQL Proposal
> > > > > >> <
> > >
> > https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing
> > >
> > > is
> > > > > >> a more detailed document describing new messages and how they
> > > would be used.
> > > > > >> - https://github.com/ryannicholson/arrow/pull/2 on my Arrow
> > fork
> > > > > >> shows a sample implementation with a catalog browsing action
> > and
> > > a prepared
> > > > > >> statement query workflow. There are details in the proposal
> > > linking to
> > > > > >> specific areas of the PR.
> > > > > >>
> > > > > >> Thank you for your time,
> > > > > >> Ryan Nicholson
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > >
> >
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Andy Grove <an...@nvidia.com>.
Sure, I can look at this in the next day or two and will provide feedback.
Thanks,
Andy.
On 9/29/20, 5:54 PM, "Ryan Nicholson" <rn...@gmail.com> wrote:
External email: Use caution opening links or attachments
Hello Andy and Wes,
Thanks for reaching out.
Later this week I have time freeing up to start on Flight-SQL. Would either
of you mind examining the approach in the POC [1] as described in the
"Proof of Concept" section of the proposal document [2]?
I would certainly be happy to chat about this with anyone interested in
participating too.
Thank you,
Ryan
[1] https://github.com/ryannicholson/arrow/pull/2
[2]
https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI
On Tue, Sep 29, 2020 at 8:52 AM Andy Grove <an...@gmail.com> wrote:
> I am also interested in helping with this effort.
>
> Andy.
>
> On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi Ryan,
> >
> > I'm interested in enabling this effort to move forward -- is there any
> > part where feedback or other help could be useful?
> >
> > Thanks,
> > Wes
> >
> > On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > I haven't been able to scrutinize the details greatly, but I am
> > > supportive of this effort, otherwise we may end up with many database
> > > systems and vendors implementing Flight in different ways. It would be
> > > nice if there was a built-in SQL client layer that we ship in C++ (and
> > > its bindings), Java, and others.
> > >
> > > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <rn...@gmail.com>
> > wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > Here is a link to the slides
> > > > <
> >
> https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> > >
> > > > I would like to discuss this morning.
> > > >
> > > > Take care,
> > > > Ryan Nicholson
> > > >
> > > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <rn...@gmail.com>
> > wrote:
> > > >
> > > > > Thank you to all who have viewed and given feedback in the posted
> > material
> > > > > so far. I am working on responses as well as preparing some
> material
> > which
> > > > > I would like to share for discussion in the sync call tomorrow. I
> > will post
> > > > > the material here ahead of time.
> > > > >
> > > > > Take care,
> > > > > Ryan Nicholson
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <
> rnicholson@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Hello everyone,
> > > > >>
> > > > >> I would like to propose the following specification to ease
> > adoption of
> > > > >> SQL based clients and backends while leveraging data streams for
> > > > >> transporting data. This specification entails a series of protobuf
> > messages
> > > > >> to be used in the opaque messaging framework payloads to enable
> > browsing of
> > > > >> database metadata and statement style operations with scaffolding
> > available
> > > > >> to help implementers provide logic for their backend.
> > > > >>
> > > > >> Please take a look at the following material:
> > > > >>
> > > > >> - ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825
> >
> > has
> > > > >> more details around the goals of the proposal.
> > > > >> - Flight SQL Proposal
> > > > >> <
> >
> https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing
> >
> > is
> > > > >> a more detailed document describing new messages and how they
> > would be used.
> > > > >> - https://github.com/ryannicholson/arrow/pull/2 on my Arrow
> fork
> > > > >> shows a sample implementation with a catalog browsing action
> and
> > a prepared
> > > > >> statement query workflow. There are details in the proposal
> > linking to
> > > > >> specific areas of the PR.
> > > > >>
> > > > >> Thank you for your time,
> > > > >> Ryan Nicholson
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> >
>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Ryan Nicholson <rn...@gmail.com>.
Hello Andy and Wes,
Thanks for reaching out.
Later this week I have time freeing up to start on Flight-SQL. Would either
of you mind examining the approach in the POC [1] as described in the
"Proof of Concept" section of the proposal document [2]?
I would certainly be happy to chat about this with anyone interested in
participating too.
Thank you,
Ryan
[1] https://github.com/ryannicholson/arrow/pull/2
[2]
https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI
On Tue, Sep 29, 2020 at 8:52 AM Andy Grove <an...@gmail.com> wrote:
> I am also interested in helping with this effort.
>
> Andy.
>
> On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi Ryan,
> >
> > I'm interested in enabling this effort to move forward -- is there any
> > part where feedback or other help could be useful?
> >
> > Thanks,
> > Wes
> >
> > On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > I haven't been able to scrutinize the details greatly, but I am
> > > supportive of this effort, otherwise we may end up with many database
> > > systems and vendors implementing Flight in different ways. It would be
> > > nice if there was a built-in SQL client layer that we ship in C++ (and
> > > its bindings), Java, and others.
> > >
> > > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <rn...@gmail.com>
> > wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > Here is a link to the slides
> > > > <
> >
> https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> > >
> > > > I would like to discuss this morning.
> > > >
> > > > Take care,
> > > > Ryan Nicholson
> > > >
> > > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <rn...@gmail.com>
> > wrote:
> > > >
> > > > > Thank you to all who have viewed and given feedback in the posted
> > material
> > > > > so far. I am working on responses as well as preparing some
> material
> > which
> > > > > I would like to share for discussion in the sync call tomorrow. I
> > will post
> > > > > the material here ahead of time.
> > > > >
> > > > > Take care,
> > > > > Ryan Nicholson
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <
> rnicholson@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Hello everyone,
> > > > >>
> > > > >> I would like to propose the following specification to ease
> > adoption of
> > > > >> SQL based clients and backends while leveraging data streams for
> > > > >> transporting data. This specification entails a series of protobuf
> > messages
> > > > >> to be used in the opaque messaging framework payloads to enable
> > browsing of
> > > > >> database metadata and statement style operations with scaffolding
> > available
> > > > >> to help implementers provide logic for their backend.
> > > > >>
> > > > >> Please take a look at the following material:
> > > > >>
> > > > >> - ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825
> >
> > has
> > > > >> more details around the goals of the proposal.
> > > > >> - Flight SQL Proposal
> > > > >> <
> >
> https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing
> >
> > is
> > > > >> a more detailed document describing new messages and how they
> > would be used.
> > > > >> - https://github.com/ryannicholson/arrow/pull/2 on my Arrow
> fork
> > > > >> shows a sample implementation with a catalog browsing action
> and
> > a prepared
> > > > >> statement query workflow. There are details in the proposal
> > linking to
> > > > >> specific areas of the PR.
> > > > >>
> > > > >> Thank you for your time,
> > > > >> Ryan Nicholson
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> >
>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Andy Grove <an...@gmail.com>.
I am also interested in helping with this effort.
Andy.
On Mon, Sep 28, 2020, 1:58 PM Wes McKinney <we...@gmail.com> wrote:
> hi Ryan,
>
> I'm interested in enabling this effort to move forward -- is there any
> part where feedback or other help could be useful?
>
> Thanks,
> Wes
>
> On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > I haven't been able to scrutinize the details greatly, but I am
> > supportive of this effort, otherwise we may end up with many database
> > systems and vendors implementing Flight in different ways. It would be
> > nice if there was a built-in SQL client layer that we ship in C++ (and
> > its bindings), Java, and others.
> >
> > On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <rn...@gmail.com>
> wrote:
> > >
> > > Hello everyone,
> > >
> > > Here is a link to the slides
> > > <
> https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing
> >
> > > I would like to discuss this morning.
> > >
> > > Take care,
> > > Ryan Nicholson
> > >
> > > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <rn...@gmail.com>
> wrote:
> > >
> > > > Thank you to all who have viewed and given feedback in the posted
> material
> > > > so far. I am working on responses as well as preparing some material
> which
> > > > I would like to share for discussion in the sync call tomorrow. I
> will post
> > > > the material here ahead of time.
> > > >
> > > > Take care,
> > > > Ryan Nicholson
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <rnicholson@gmail.com
> >
> > > > wrote:
> > > >
> > > >> Hello everyone,
> > > >>
> > > >> I would like to propose the following specification to ease
> adoption of
> > > >> SQL based clients and backends while leveraging data streams for
> > > >> transporting data. This specification entails a series of protobuf
> messages
> > > >> to be used in the opaque messaging framework payloads to enable
> browsing of
> > > >> database metadata and statement style operations with scaffolding
> available
> > > >> to help implementers provide logic for their backend.
> > > >>
> > > >> Please take a look at the following material:
> > > >>
> > > >> - ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825>
> has
> > > >> more details around the goals of the proposal.
> > > >> - Flight SQL Proposal
> > > >> <
> https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing>
> is
> > > >> a more detailed document describing new messages and how they
> would be used.
> > > >> - https://github.com/ryannicholson/arrow/pull/2 on my Arrow fork
> > > >> shows a sample implementation with a catalog browsing action and
> a prepared
> > > >> statement query workflow. There are details in the proposal
> linking to
> > > >> specific areas of the PR.
> > > >>
> > > >> Thank you for your time,
> > > >> Ryan Nicholson
> > > >>
> > > >>
> > > >>
> > > >>
>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Wes McKinney <we...@gmail.com>.
hi Ryan,
I'm interested in enabling this effort to move forward -- is there any
part where feedback or other help could be useful?
Thanks,
Wes
On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney <we...@gmail.com> wrote:
>
> I haven't been able to scrutinize the details greatly, but I am
> supportive of this effort, otherwise we may end up with many database
> systems and vendors implementing Flight in different ways. It would be
> nice if there was a built-in SQL client layer that we ship in C++ (and
> its bindings), Java, and others.
>
> On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <rn...@gmail.com> wrote:
> >
> > Hello everyone,
> >
> > Here is a link to the slides
> > <https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing>
> > I would like to discuss this morning.
> >
> > Take care,
> > Ryan Nicholson
> >
> > On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <rn...@gmail.com> wrote:
> >
> > > Thank you to all who have viewed and given feedback in the posted material
> > > so far. I am working on responses as well as preparing some material which
> > > I would like to share for discussion in the sync call tomorrow. I will post
> > > the material here ahead of time.
> > >
> > > Take care,
> > > Ryan Nicholson
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <rn...@gmail.com>
> > > wrote:
> > >
> > >> Hello everyone,
> > >>
> > >> I would like to propose the following specification to ease adoption of
> > >> SQL based clients and backends while leveraging data streams for
> > >> transporting data. This specification entails a series of protobuf messages
> > >> to be used in the opaque messaging framework payloads to enable browsing of
> > >> database metadata and statement style operations with scaffolding available
> > >> to help implementers provide logic for their backend.
> > >>
> > >> Please take a look at the following material:
> > >>
> > >> - ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825> has
> > >> more details around the goals of the proposal.
> > >> - Flight SQL Proposal
> > >> <https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing> is
> > >> a more detailed document describing new messages and how they would be used.
> > >> - https://github.com/ryannicholson/arrow/pull/2 on my Arrow fork
> > >> shows a sample implementation with a catalog browsing action and a prepared
> > >> statement query workflow. There are details in the proposal linking to
> > >> specific areas of the PR.
> > >>
> > >> Thank you for your time,
> > >> Ryan Nicholson
> > >>
> > >>
> > >>
> > >>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Wes McKinney <we...@gmail.com>.
I haven't been able to scrutinize the details greatly, but I am
supportive of this effort, otherwise we may end up with many database
systems and vendors implementing Flight in different ways. It would be
nice if there was a built-in SQL client layer that we ship in C++ (and
its bindings), Java, and others.
On Wed, Sep 2, 2020 at 10:24 AM Ryan Nicholson <rn...@gmail.com> wrote:
>
> Hello everyone,
>
> Here is a link to the slides
> <https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing>
> I would like to discuss this morning.
>
> Take care,
> Ryan Nicholson
>
> On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <rn...@gmail.com> wrote:
>
> > Thank you to all who have viewed and given feedback in the posted material
> > so far. I am working on responses as well as preparing some material which
> > I would like to share for discussion in the sync call tomorrow. I will post
> > the material here ahead of time.
> >
> > Take care,
> > Ryan Nicholson
> >
> >
> >
> >
> >
> >
> > On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <rn...@gmail.com>
> > wrote:
> >
> >> Hello everyone,
> >>
> >> I would like to propose the following specification to ease adoption of
> >> SQL based clients and backends while leveraging data streams for
> >> transporting data. This specification entails a series of protobuf messages
> >> to be used in the opaque messaging framework payloads to enable browsing of
> >> database metadata and statement style operations with scaffolding available
> >> to help implementers provide logic for their backend.
> >>
> >> Please take a look at the following material:
> >>
> >> - ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825> has
> >> more details around the goals of the proposal.
> >> - Flight SQL Proposal
> >> <https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing> is
> >> a more detailed document describing new messages and how they would be used.
> >> - https://github.com/ryannicholson/arrow/pull/2 on my Arrow fork
> >> shows a sample implementation with a catalog browsing action and a prepared
> >> statement query workflow. There are details in the proposal linking to
> >> specific areas of the PR.
> >>
> >> Thank you for your time,
> >> Ryan Nicholson
> >>
> >>
> >>
> >>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Ryan Nicholson <rn...@gmail.com>.
Hello everyone,
Here is a link to the slides
<https://docs.google.com/presentation/d/1JcOKA1yKYTNodQl5Z1BtK3dhJLx-Vu_RSAqWF_wiJH8/edit?usp=sharing>
I would like to discuss this morning.
Take care,
Ryan Nicholson
On Tue, Sep 1, 2020 at 4:02 PM Ryan Nicholson <rn...@gmail.com> wrote:
> Thank you to all who have viewed and given feedback in the posted material
> so far. I am working on responses as well as preparing some material which
> I would like to share for discussion in the sync call tomorrow. I will post
> the material here ahead of time.
>
> Take care,
> Ryan Nicholson
>
>
>
>
>
>
> On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <rn...@gmail.com>
> wrote:
>
>> Hello everyone,
>>
>> I would like to propose the following specification to ease adoption of
>> SQL based clients and backends while leveraging data streams for
>> transporting data. This specification entails a series of protobuf messages
>> to be used in the opaque messaging framework payloads to enable browsing of
>> database metadata and statement style operations with scaffolding available
>> to help implementers provide logic for their backend.
>>
>> Please take a look at the following material:
>>
>> - ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825> has
>> more details around the goals of the proposal.
>> - Flight SQL Proposal
>> <https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing> is
>> a more detailed document describing new messages and how they would be used.
>> - https://github.com/ryannicholson/arrow/pull/2 on my Arrow fork
>> shows a sample implementation with a catalog browsing action and a prepared
>> statement query workflow. There are details in the proposal linking to
>> specific areas of the PR.
>>
>> Thank you for your time,
>> Ryan Nicholson
>>
>>
>>
>>
Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC
Posted by Ryan Nicholson <rn...@gmail.com>.
Thank you to all who have viewed and given feedback in the posted material
so far. I am working on responses as well as preparing some material which
I would like to share for discussion in the sync call tomorrow. I will post
the material here ahead of time.
Take care,
Ryan Nicholson
On Mon, Aug 31, 2020 at 2:21 PM Ryan Nicholson <rn...@gmail.com> wrote:
> Hello everyone,
>
> I would like to propose the following specification to ease adoption of
> SQL based clients and backends while leveraging data streams for
> transporting data. This specification entails a series of protobuf messages
> to be used in the opaque messaging framework payloads to enable browsing of
> database metadata and statement style operations with scaffolding available
> to help implementers provide logic for their backend.
>
> Please take a look at the following material:
>
> - ARROW-9825 <https://issues.apache.org/jira/browse/ARROW-9825> has
> more details around the goals of the proposal.
> - Flight SQL Proposal
> <https://docs.google.com/document/d/1WQz32bDF06GgMdEYyzhakqUigBZkALFwDF2y1x3DTAI/edit?usp=sharing> is
> a more detailed document describing new messages and how they would be used.
> - https://github.com/ryannicholson/arrow/pull/2 on my Arrow fork shows
> a sample implementation with a catalog browsing action and a prepared
> statement query workflow. There are details in the proposal linking to
> specific areas of the PR.
>
> Thank you for your time,
> Ryan Nicholson
>
>
>
>