You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Andy Grove <an...@gmail.com> on 2018/09/04 03:08:52 UTC
Re: Rust IPC and Integration Testing
Flatbuffers now has support for Rust - the PR was just merged in the last
couple days:
https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
I hope to find some time in the next week or two to start experimenting
with this for Arrow IPC.
Andy.
On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <an...@gmail.com> wrote:
> The author of the Rust version of FlatBuffers is now working on a public
> fork and seems to be making good progress. I just wanted to send a quick
> update on this since I am now back onto working on Arrow IPC in Rust (I'm
> sure this is going to take quite a while though so don't want to get
> anyones hopes up too much). For anyone that is interested, here is the
> FlatBuffers branch: https://github.com/rw/flatbuffers/tree/2018-02--rust
>
> Andy.
>
>
> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <we...@gmail.com> wrote:
>
>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and
>> Rust. One possibility is to build an interface to the Flatbuffers data
>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if these
>> bindings are header-only like C++ and if that makes things any easier
>> for you.
>>
>> If you defined an API in Rust for reading and writing the Arrow
>> metadata that does not expose any Flatbuffers-specific details, then
>> once there is a production-grade Rust implementation of Flatbuffers,
>> the implementation details could be swapped out without disruption.
>>
>> > my recent attempts at contacting the author have been unsuccessful
>>
>> Have you e-mailed the author directly or only pings on GitHub? Pings
>> may not be making it to their inbox.
>>
>> Appreciate your efforts on this; I think we'll see a lot more work in
>> data processing in Rust in the coming years.
>>
>> - Wes
>>
>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <an...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > Now that the refactor I've been working on has been merged, the next
>> > priority for me personally with the Rust implementation is getting IPC
>> and
>> > integration testing working.
>> >
>> > Unfortunately the official Flatbuffers Rust version is not available yet
>> > and my recent attempts at contacting the author have been unsuccessful
>> so I
>> > have started working with this fork of Flatbuffers which has Rust
>> support:
>> > https://github.com/josephDunne/flatbuffers.
>> >
>> > I was able to generate code from Schema.fbs but it doesn't compile and
>> I've
>> > started filing issues and debugging this.
>> >
>> > Working with this fork isn't ideal but I don't see what other choice I
>> > have, other than just waiting for the official project to support Rust.
>> >
>> > I'm interested to hear if anyone has any alternate suggestions. I know
>> it
>> > would be possible to wrap C code but I'd like to keep the Rust
>> > implementation as a pure Rust project if possible.
>> >
>> > Thanks,
>> >
>> > Andy.
>>
>
Re: Rust IPC and Integration Testing
Posted by Chao Sun <su...@apache.org>.
Yes it is possible - we have done similar things in parquet-rs to generate
Thrift files, using build.rs (see here
<https://github.com/sunchao/parquet-rs/blob/f2f0993098bdeaa62a0790be18faac2f5bf2fe3b/build.rs>).
Once the manual edits are not required we can explore that path.
On Tue, Nov 20, 2018 at 1:20 PM Krisztián Szűcs <sz...@gmail.com>
wrote:
> Auto generation sounds like a good idea, similarly like rust-bindgen
> does it: https://rust-lang-nursery.github.io/rust-bindgen/tutorial-3.html
> Of course it depends on the complexity of manual edits.
>
> We can still separate the crates later if that's desired.
>
> On Tue, Nov 20, 2018 at 6:39 PM Chao Sun <su...@uber.com.invalid> wrote:
>
> > I think I suggested this. The idea is to isolate the auto-generated code
> > from the other source files, and also to isolate future changes due to
> > updates on either format or flatbuffers versions. IMO this makes the repo
> > cleaner and more organized. Of course, ideally we should auto-generate
> > these files when compiling.
> >
> > On Mon, Nov 19, 2018 at 11:55 AM Wes McKinney <we...@gmail.com>
> wrote:
> >
> > > Could you explain why having a separate crate would be a good idea?
> > > On Mon, Nov 19, 2018 at 2:40 PM Andy Grove <an...@gmail.com>
> > wrote:
> > > >
> > > > A question has been raised on this PR as to whether we should
> publish a
> > > > separate crate for the format/ipc generated code. I think there might
> > be
> > > > some merit in this and wanted to raise it here to see what everyone
> > > thinks.
> > > >
> > > > This would mean having two directories under the rust directory ...
> one
> > > for
> > > > arrow-format (or maybe the name should be arrow-ipc?) and one for the
> > > main
> > > > arrow crate.
> > > >
> > > > It is possible we might want other crates in the future e.g.
> > > arrow-parquet
> > > > or whatever.
> > > >
> > > > Andy.
> > > >
> > > >
> > > > On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <an...@gmail.com>
> > > wrote:
> > > >
> > > > > I have created a PR this morning that adds the generated
> Flatbuffers
> > > code
> > > > > to the Rust implementation (
> > https://github.com/apache/arrow/pull/2986
> > > ).
> > > > >
> > > > > Just throwing these files into the repo doesn't add a ton of value
> > > until
> > > > > we use them for something, but it took some effort to generate them
> > > (and
> > > > > then fix a couple things by hand) so it seems useful to contribute
> > > this.
> > > > > The PR includes a brief README with the instructions for generating
> > the
> > > > > files.
> > > > >
> > > > > I have taken this week off for Thanksgiving so I actually have some
> > > > > dedicated time to contribute to Arrow. I suppose my next step on
> this
> > > is to
> > > > > write a test that can serialize some data and then de-serialize it
> > > again.
> > > > >
> > > > > Andy.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >
> > > > >> hi Andy,
> > > > >>
> > > > >> AFAIK in the other implementations, the Flatbuffers serialization
> is
> > > > >> an implementation detail of offering support for the columnar IPC
> > > > >> protocol (which in turn depends on the columnar data structures,
> > > > >> memory management, etc.). I'm not sure what a crate offering the
> > > > >> Flatbuffers bindings alone would offer, outside of helping people
> > > > >> create new Rust Arrow implementations, I guess. In any case, if
> you
> > > > >> wanted to do that, we would need to make a release first.
> > > > >>
> > > > >> - Wes
> > > > >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <an...@gmail.com>
> > > wrote:
> > > > >> >
> > > > >> > Brief update on this. I have now been able to use flatbuffers to
> > > > >> generate
> > > > >> > Rust code from the Arrow schema, and it compiles. In theory,
> this
> > > can be
> > > > >> > published as a standalone "arrow-format" crate but I'm wondering
> > > what
> > > > >> the
> > > > >> > plan should be here.
> > > > >> >
> > > > >> > I will start reviewing how this is handled in the other
> > > implementations
> > > > >> but
> > > > >> > if anyone has suggestions, please let me know.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Andy.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <
> andygrove73@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Flatbuffers now has support for Rust - the PR was just merged
> in
> > > the
> > > > >> last
> > > > >> > > couple days:
> > > > >> > >
> > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
> > > > >> > >
> > > > >> > > I hope to find some time in the next week or two to start
> > > > >> experimenting
> > > > >> > > with this for Arrow IPC.
> > > > >> > >
> > > > >> > > Andy.
> > > > >> > >
> > > > >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <
> > andygrove73@gmail.com
> > > >
> > > > >> wrote:
> > > > >> > >
> > > > >> > >> The author of the Rust version of FlatBuffers is now working
> > on a
> > > > >> public
> > > > >> > >> fork and seems to be making good progress. I just wanted to
> > send
> > > a
> > > > >> quick
> > > > >> > >> update on this since I am now back onto working on Arrow IPC
> in
> > > Rust
> > > > >> (I'm
> > > > >> > >> sure this is going to take quite a while though so don't want
> > to
> > > get
> > > > >> > >> anyones hopes up too much). For anyone that is interested,
> here
> > > is
> > > > >> the
> > > > >> > >> FlatBuffers branch:
> > > > >> https://github.com/rw/flatbuffers/tree/2018-02--rust
> > > > >> > >>
> > > > >> > >> Andy.
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > >> wrote:
> > > > >> > >>
> > > > >> > >>> Sorry to hear, it's a bit of a rough situation with
> > Flatbuffers
> > > and
> > > > >> > >>> Rust. One possibility is to build an interface to the
> > > Flatbuffers
> > > > >> data
> > > > >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I
> wonder
> > if
> > > > >> these
> > > > >> > >>> bindings are header-only like C++ and if that makes things
> any
> > > > >> easier
> > > > >> > >>> for you.
> > > > >> > >>>
> > > > >> > >>> If you defined an API in Rust for reading and writing the
> > Arrow
> > > > >> > >>> metadata that does not expose any Flatbuffers-specific
> > details,
> > > then
> > > > >> > >>> once there is a production-grade Rust implementation of
> > > Flatbuffers,
> > > > >> > >>> the implementation details could be swapped out without
> > > disruption.
> > > > >> > >>>
> > > > >> > >>> > my recent attempts at contacting the author have been
> > > unsuccessful
> > > > >> > >>>
> > > > >> > >>> Have you e-mailed the author directly or only pings on
> GitHub?
> > > Pings
> > > > >> > >>> may not be making it to their inbox.
> > > > >> > >>>
> > > > >> > >>> Appreciate your efforts on this; I think we'll see a lot
> more
> > > work
> > > > >> in
> > > > >> > >>> data processing in Rust in the coming years.
> > > > >> > >>>
> > > > >> > >>> - Wes
> > > > >> > >>>
> > > > >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <
> > > andygrove73@gmail.com>
> > > > >> > >>> wrote:
> > > > >> > >>> > Hi,
> > > > >> > >>> >
> > > > >> > >>> > Now that the refactor I've been working on has been
> merged,
> > > the
> > > > >> next
> > > > >> > >>> > priority for me personally with the Rust implementation is
> > > > >> getting IPC
> > > > >> > >>> and
> > > > >> > >>> > integration testing working.
> > > > >> > >>> >
> > > > >> > >>> > Unfortunately the official Flatbuffers Rust version is not
> > > > >> available
> > > > >> > >>> yet
> > > > >> > >>> > and my recent attempts at contacting the author have been
> > > > >> unsuccessful
> > > > >> > >>> so I
> > > > >> > >>> > have started working with this fork of Flatbuffers which
> has
> > > Rust
> > > > >> > >>> support:
> > > > >> > >>> > https://github.com/josephDunne/flatbuffers.
> > > > >> > >>> >
> > > > >> > >>> > I was able to generate code from Schema.fbs but it doesn't
> > > > >> compile and
> > > > >> > >>> I've
> > > > >> > >>> > started filing issues and debugging this.
> > > > >> > >>> >
> > > > >> > >>> > Working with this fork isn't ideal but I don't see what
> > other
> > > > >> choice I
> > > > >> > >>> > have, other than just waiting for the official project to
> > > support
> > > > >> Rust.
> > > > >> > >>> >
> > > > >> > >>> > I'm interested to hear if anyone has any alternate
> > > suggestions. I
> > > > >> know
> > > > >> > >>> it
> > > > >> > >>> > would be possible to wrap C code but I'd like to keep the
> > Rust
> > > > >> > >>> > implementation as a pure Rust project if possible.
> > > > >> > >>> >
> > > > >> > >>> > Thanks,
> > > > >> > >>> >
> > > > >> > >>> > Andy.
> > > > >> > >>>
> > > > >> > >>
> > > > >>
> > > > >
> > >
> >
>
Re: Rust IPC and Integration Testing
Posted by Krisztián Szűcs <sz...@gmail.com>.
Auto generation sounds like a good idea, similarly like rust-bindgen
does it: https://rust-lang-nursery.github.io/rust-bindgen/tutorial-3.html
Of course it depends on the complexity of manual edits.
We can still separate the crates later if that's desired.
On Tue, Nov 20, 2018 at 6:39 PM Chao Sun <su...@uber.com.invalid> wrote:
> I think I suggested this. The idea is to isolate the auto-generated code
> from the other source files, and also to isolate future changes due to
> updates on either format or flatbuffers versions. IMO this makes the repo
> cleaner and more organized. Of course, ideally we should auto-generate
> these files when compiling.
>
> On Mon, Nov 19, 2018 at 11:55 AM Wes McKinney <we...@gmail.com> wrote:
>
> > Could you explain why having a separate crate would be a good idea?
> > On Mon, Nov 19, 2018 at 2:40 PM Andy Grove <an...@gmail.com>
> wrote:
> > >
> > > A question has been raised on this PR as to whether we should publish a
> > > separate crate for the format/ipc generated code. I think there might
> be
> > > some merit in this and wanted to raise it here to see what everyone
> > thinks.
> > >
> > > This would mean having two directories under the rust directory ... one
> > for
> > > arrow-format (or maybe the name should be arrow-ipc?) and one for the
> > main
> > > arrow crate.
> > >
> > > It is possible we might want other crates in the future e.g.
> > arrow-parquet
> > > or whatever.
> > >
> > > Andy.
> > >
> > >
> > > On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <an...@gmail.com>
> > wrote:
> > >
> > > > I have created a PR this morning that adds the generated Flatbuffers
> > code
> > > > to the Rust implementation (
> https://github.com/apache/arrow/pull/2986
> > ).
> > > >
> > > > Just throwing these files into the repo doesn't add a ton of value
> > until
> > > > we use them for something, but it took some effort to generate them
> > (and
> > > > then fix a couple things by hand) so it seems useful to contribute
> > this.
> > > > The PR includes a brief README with the instructions for generating
> the
> > > > files.
> > > >
> > > > I have taken this week off for Thanksgiving so I actually have some
> > > > dedicated time to contribute to Arrow. I suppose my next step on this
> > is to
> > > > write a test that can serialize some data and then de-serialize it
> > again.
> > > >
> > > > Andy.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > >> hi Andy,
> > > >>
> > > >> AFAIK in the other implementations, the Flatbuffers serialization is
> > > >> an implementation detail of offering support for the columnar IPC
> > > >> protocol (which in turn depends on the columnar data structures,
> > > >> memory management, etc.). I'm not sure what a crate offering the
> > > >> Flatbuffers bindings alone would offer, outside of helping people
> > > >> create new Rust Arrow implementations, I guess. In any case, if you
> > > >> wanted to do that, we would need to make a release first.
> > > >>
> > > >> - Wes
> > > >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <an...@gmail.com>
> > wrote:
> > > >> >
> > > >> > Brief update on this. I have now been able to use flatbuffers to
> > > >> generate
> > > >> > Rust code from the Arrow schema, and it compiles. In theory, this
> > can be
> > > >> > published as a standalone "arrow-format" crate but I'm wondering
> > what
> > > >> the
> > > >> > plan should be here.
> > > >> >
> > > >> > I will start reviewing how this is handled in the other
> > implementations
> > > >> but
> > > >> > if anyone has suggestions, please let me know.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Andy.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <an...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > Flatbuffers now has support for Rust - the PR was just merged in
> > the
> > > >> last
> > > >> > > couple days:
> > > >> > >
> > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
> > > >> > >
> > > >> > > I hope to find some time in the next week or two to start
> > > >> experimenting
> > > >> > > with this for Arrow IPC.
> > > >> > >
> > > >> > > Andy.
> > > >> > >
> > > >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <
> andygrove73@gmail.com
> > >
> > > >> wrote:
> > > >> > >
> > > >> > >> The author of the Rust version of FlatBuffers is now working
> on a
> > > >> public
> > > >> > >> fork and seems to be making good progress. I just wanted to
> send
> > a
> > > >> quick
> > > >> > >> update on this since I am now back onto working on Arrow IPC in
> > Rust
> > > >> (I'm
> > > >> > >> sure this is going to take quite a while though so don't want
> to
> > get
> > > >> > >> anyones hopes up too much). For anyone that is interested, here
> > is
> > > >> the
> > > >> > >> FlatBuffers branch:
> > > >> https://github.com/rw/flatbuffers/tree/2018-02--rust
> > > >> > >>
> > > >> > >> Andy.
> > > >> > >>
> > > >> > >>
> > > >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <
> > wesmckinn@gmail.com>
> > > >> wrote:
> > > >> > >>
> > > >> > >>> Sorry to hear, it's a bit of a rough situation with
> Flatbuffers
> > and
> > > >> > >>> Rust. One possibility is to build an interface to the
> > Flatbuffers
> > > >> data
> > > >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder
> if
> > > >> these
> > > >> > >>> bindings are header-only like C++ and if that makes things any
> > > >> easier
> > > >> > >>> for you.
> > > >> > >>>
> > > >> > >>> If you defined an API in Rust for reading and writing the
> Arrow
> > > >> > >>> metadata that does not expose any Flatbuffers-specific
> details,
> > then
> > > >> > >>> once there is a production-grade Rust implementation of
> > Flatbuffers,
> > > >> > >>> the implementation details could be swapped out without
> > disruption.
> > > >> > >>>
> > > >> > >>> > my recent attempts at contacting the author have been
> > unsuccessful
> > > >> > >>>
> > > >> > >>> Have you e-mailed the author directly or only pings on GitHub?
> > Pings
> > > >> > >>> may not be making it to their inbox.
> > > >> > >>>
> > > >> > >>> Appreciate your efforts on this; I think we'll see a lot more
> > work
> > > >> in
> > > >> > >>> data processing in Rust in the coming years.
> > > >> > >>>
> > > >> > >>> - Wes
> > > >> > >>>
> > > >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <
> > andygrove73@gmail.com>
> > > >> > >>> wrote:
> > > >> > >>> > Hi,
> > > >> > >>> >
> > > >> > >>> > Now that the refactor I've been working on has been merged,
> > the
> > > >> next
> > > >> > >>> > priority for me personally with the Rust implementation is
> > > >> getting IPC
> > > >> > >>> and
> > > >> > >>> > integration testing working.
> > > >> > >>> >
> > > >> > >>> > Unfortunately the official Flatbuffers Rust version is not
> > > >> available
> > > >> > >>> yet
> > > >> > >>> > and my recent attempts at contacting the author have been
> > > >> unsuccessful
> > > >> > >>> so I
> > > >> > >>> > have started working with this fork of Flatbuffers which has
> > Rust
> > > >> > >>> support:
> > > >> > >>> > https://github.com/josephDunne/flatbuffers.
> > > >> > >>> >
> > > >> > >>> > I was able to generate code from Schema.fbs but it doesn't
> > > >> compile and
> > > >> > >>> I've
> > > >> > >>> > started filing issues and debugging this.
> > > >> > >>> >
> > > >> > >>> > Working with this fork isn't ideal but I don't see what
> other
> > > >> choice I
> > > >> > >>> > have, other than just waiting for the official project to
> > support
> > > >> Rust.
> > > >> > >>> >
> > > >> > >>> > I'm interested to hear if anyone has any alternate
> > suggestions. I
> > > >> know
> > > >> > >>> it
> > > >> > >>> > would be possible to wrap C code but I'd like to keep the
> Rust
> > > >> > >>> > implementation as a pure Rust project if possible.
> > > >> > >>> >
> > > >> > >>> > Thanks,
> > > >> > >>> >
> > > >> > >>> > Andy.
> > > >> > >>>
> > > >> > >>
> > > >>
> > > >
> >
>
Re: Rust IPC and Integration Testing
Posted by Chao Sun <su...@uber.com.INVALID>.
I think I suggested this. The idea is to isolate the auto-generated code
from the other source files, and also to isolate future changes due to
updates on either format or flatbuffers versions. IMO this makes the repo
cleaner and more organized. Of course, ideally we should auto-generate
these files when compiling.
On Mon, Nov 19, 2018 at 11:55 AM Wes McKinney <we...@gmail.com> wrote:
> Could you explain why having a separate crate would be a good idea?
> On Mon, Nov 19, 2018 at 2:40 PM Andy Grove <an...@gmail.com> wrote:
> >
> > A question has been raised on this PR as to whether we should publish a
> > separate crate for the format/ipc generated code. I think there might be
> > some merit in this and wanted to raise it here to see what everyone
> thinks.
> >
> > This would mean having two directories under the rust directory ... one
> for
> > arrow-format (or maybe the name should be arrow-ipc?) and one for the
> main
> > arrow crate.
> >
> > It is possible we might want other crates in the future e.g.
> arrow-parquet
> > or whatever.
> >
> > Andy.
> >
> >
> > On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <an...@gmail.com>
> wrote:
> >
> > > I have created a PR this morning that adds the generated Flatbuffers
> code
> > > to the Rust implementation (https://github.com/apache/arrow/pull/2986
> ).
> > >
> > > Just throwing these files into the repo doesn't add a ton of value
> until
> > > we use them for something, but it took some effort to generate them
> (and
> > > then fix a couple things by hand) so it seems useful to contribute
> this.
> > > The PR includes a brief README with the instructions for generating the
> > > files.
> > >
> > > I have taken this week off for Thanksgiving so I actually have some
> > > dedicated time to contribute to Arrow. I suppose my next step on this
> is to
> > > write a test that can serialize some data and then de-serialize it
> again.
> > >
> > > Andy.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > >> hi Andy,
> > >>
> > >> AFAIK in the other implementations, the Flatbuffers serialization is
> > >> an implementation detail of offering support for the columnar IPC
> > >> protocol (which in turn depends on the columnar data structures,
> > >> memory management, etc.). I'm not sure what a crate offering the
> > >> Flatbuffers bindings alone would offer, outside of helping people
> > >> create new Rust Arrow implementations, I guess. In any case, if you
> > >> wanted to do that, we would need to make a release first.
> > >>
> > >> - Wes
> > >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <an...@gmail.com>
> wrote:
> > >> >
> > >> > Brief update on this. I have now been able to use flatbuffers to
> > >> generate
> > >> > Rust code from the Arrow schema, and it compiles. In theory, this
> can be
> > >> > published as a standalone "arrow-format" crate but I'm wondering
> what
> > >> the
> > >> > plan should be here.
> > >> >
> > >> > I will start reviewing how this is handled in the other
> implementations
> > >> but
> > >> > if anyone has suggestions, please let me know.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Andy.
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <an...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Flatbuffers now has support for Rust - the PR was just merged in
> the
> > >> last
> > >> > > couple days:
> > >> > >
> https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
> > >> > >
> > >> > > I hope to find some time in the next week or two to start
> > >> experimenting
> > >> > > with this for Arrow IPC.
> > >> > >
> > >> > > Andy.
> > >> > >
> > >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <andygrove73@gmail.com
> >
> > >> wrote:
> > >> > >
> > >> > >> The author of the Rust version of FlatBuffers is now working on a
> > >> public
> > >> > >> fork and seems to be making good progress. I just wanted to send
> a
> > >> quick
> > >> > >> update on this since I am now back onto working on Arrow IPC in
> Rust
> > >> (I'm
> > >> > >> sure this is going to take quite a while though so don't want to
> get
> > >> > >> anyones hopes up too much). For anyone that is interested, here
> is
> > >> the
> > >> > >> FlatBuffers branch:
> > >> https://github.com/rw/flatbuffers/tree/2018-02--rust
> > >> > >>
> > >> > >> Andy.
> > >> > >>
> > >> > >>
> > >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <
> wesmckinn@gmail.com>
> > >> wrote:
> > >> > >>
> > >> > >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers
> and
> > >> > >>> Rust. One possibility is to build an interface to the
> Flatbuffers
> > >> data
> > >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if
> > >> these
> > >> > >>> bindings are header-only like C++ and if that makes things any
> > >> easier
> > >> > >>> for you.
> > >> > >>>
> > >> > >>> If you defined an API in Rust for reading and writing the Arrow
> > >> > >>> metadata that does not expose any Flatbuffers-specific details,
> then
> > >> > >>> once there is a production-grade Rust implementation of
> Flatbuffers,
> > >> > >>> the implementation details could be swapped out without
> disruption.
> > >> > >>>
> > >> > >>> > my recent attempts at contacting the author have been
> unsuccessful
> > >> > >>>
> > >> > >>> Have you e-mailed the author directly or only pings on GitHub?
> Pings
> > >> > >>> may not be making it to their inbox.
> > >> > >>>
> > >> > >>> Appreciate your efforts on this; I think we'll see a lot more
> work
> > >> in
> > >> > >>> data processing in Rust in the coming years.
> > >> > >>>
> > >> > >>> - Wes
> > >> > >>>
> > >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <
> andygrove73@gmail.com>
> > >> > >>> wrote:
> > >> > >>> > Hi,
> > >> > >>> >
> > >> > >>> > Now that the refactor I've been working on has been merged,
> the
> > >> next
> > >> > >>> > priority for me personally with the Rust implementation is
> > >> getting IPC
> > >> > >>> and
> > >> > >>> > integration testing working.
> > >> > >>> >
> > >> > >>> > Unfortunately the official Flatbuffers Rust version is not
> > >> available
> > >> > >>> yet
> > >> > >>> > and my recent attempts at contacting the author have been
> > >> unsuccessful
> > >> > >>> so I
> > >> > >>> > have started working with this fork of Flatbuffers which has
> Rust
> > >> > >>> support:
> > >> > >>> > https://github.com/josephDunne/flatbuffers.
> > >> > >>> >
> > >> > >>> > I was able to generate code from Schema.fbs but it doesn't
> > >> compile and
> > >> > >>> I've
> > >> > >>> > started filing issues and debugging this.
> > >> > >>> >
> > >> > >>> > Working with this fork isn't ideal but I don't see what other
> > >> choice I
> > >> > >>> > have, other than just waiting for the official project to
> support
> > >> Rust.
> > >> > >>> >
> > >> > >>> > I'm interested to hear if anyone has any alternate
> suggestions. I
> > >> know
> > >> > >>> it
> > >> > >>> > would be possible to wrap C code but I'd like to keep the Rust
> > >> > >>> > implementation as a pure Rust project if possible.
> > >> > >>> >
> > >> > >>> > Thanks,
> > >> > >>> >
> > >> > >>> > Andy.
> > >> > >>>
> > >> > >>
> > >>
> > >
>
Re: Rust IPC and Integration Testing
Posted by Wes McKinney <we...@gmail.com>.
Could you explain why having a separate crate would be a good idea?
On Mon, Nov 19, 2018 at 2:40 PM Andy Grove <an...@gmail.com> wrote:
>
> A question has been raised on this PR as to whether we should publish a
> separate crate for the format/ipc generated code. I think there might be
> some merit in this and wanted to raise it here to see what everyone thinks.
>
> This would mean having two directories under the rust directory ... one for
> arrow-format (or maybe the name should be arrow-ipc?) and one for the main
> arrow crate.
>
> It is possible we might want other crates in the future e.g. arrow-parquet
> or whatever.
>
> Andy.
>
>
> On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <an...@gmail.com> wrote:
>
> > I have created a PR this morning that adds the generated Flatbuffers code
> > to the Rust implementation (https://github.com/apache/arrow/pull/2986).
> >
> > Just throwing these files into the repo doesn't add a ton of value until
> > we use them for something, but it took some effort to generate them (and
> > then fix a couple things by hand) so it seems useful to contribute this.
> > The PR includes a brief README with the instructions for generating the
> > files.
> >
> > I have taken this week off for Thanksgiving so I actually have some
> > dedicated time to contribute to Arrow. I suppose my next step on this is to
> > write a test that can serialize some data and then de-serialize it again.
> >
> > Andy.
> >
> >
> >
> >
> >
> >
> >
> > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <we...@gmail.com> wrote:
> >
> >> hi Andy,
> >>
> >> AFAIK in the other implementations, the Flatbuffers serialization is
> >> an implementation detail of offering support for the columnar IPC
> >> protocol (which in turn depends on the columnar data structures,
> >> memory management, etc.). I'm not sure what a crate offering the
> >> Flatbuffers bindings alone would offer, outside of helping people
> >> create new Rust Arrow implementations, I guess. In any case, if you
> >> wanted to do that, we would need to make a release first.
> >>
> >> - Wes
> >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <an...@gmail.com> wrote:
> >> >
> >> > Brief update on this. I have now been able to use flatbuffers to
> >> generate
> >> > Rust code from the Arrow schema, and it compiles. In theory, this can be
> >> > published as a standalone "arrow-format" crate but I'm wondering what
> >> the
> >> > plan should be here.
> >> >
> >> > I will start reviewing how this is handled in the other implementations
> >> but
> >> > if anyone has suggestions, please let me know.
> >> >
> >> > Thanks,
> >> >
> >> > Andy.
> >> >
> >> >
> >> >
> >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <an...@gmail.com>
> >> wrote:
> >> >
> >> > > Flatbuffers now has support for Rust - the PR was just merged in the
> >> last
> >> > > couple days:
> >> > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
> >> > >
> >> > > I hope to find some time in the next week or two to start
> >> experimenting
> >> > > with this for Arrow IPC.
> >> > >
> >> > > Andy.
> >> > >
> >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <an...@gmail.com>
> >> wrote:
> >> > >
> >> > >> The author of the Rust version of FlatBuffers is now working on a
> >> public
> >> > >> fork and seems to be making good progress. I just wanted to send a
> >> quick
> >> > >> update on this since I am now back onto working on Arrow IPC in Rust
> >> (I'm
> >> > >> sure this is going to take quite a while though so don't want to get
> >> > >> anyones hopes up too much). For anyone that is interested, here is
> >> the
> >> > >> FlatBuffers branch:
> >> https://github.com/rw/flatbuffers/tree/2018-02--rust
> >> > >>
> >> > >> Andy.
> >> > >>
> >> > >>
> >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <we...@gmail.com>
> >> wrote:
> >> > >>
> >> > >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and
> >> > >>> Rust. One possibility is to build an interface to the Flatbuffers
> >> data
> >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if
> >> these
> >> > >>> bindings are header-only like C++ and if that makes things any
> >> easier
> >> > >>> for you.
> >> > >>>
> >> > >>> If you defined an API in Rust for reading and writing the Arrow
> >> > >>> metadata that does not expose any Flatbuffers-specific details, then
> >> > >>> once there is a production-grade Rust implementation of Flatbuffers,
> >> > >>> the implementation details could be swapped out without disruption.
> >> > >>>
> >> > >>> > my recent attempts at contacting the author have been unsuccessful
> >> > >>>
> >> > >>> Have you e-mailed the author directly or only pings on GitHub? Pings
> >> > >>> may not be making it to their inbox.
> >> > >>>
> >> > >>> Appreciate your efforts on this; I think we'll see a lot more work
> >> in
> >> > >>> data processing in Rust in the coming years.
> >> > >>>
> >> > >>> - Wes
> >> > >>>
> >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <an...@gmail.com>
> >> > >>> wrote:
> >> > >>> > Hi,
> >> > >>> >
> >> > >>> > Now that the refactor I've been working on has been merged, the
> >> next
> >> > >>> > priority for me personally with the Rust implementation is
> >> getting IPC
> >> > >>> and
> >> > >>> > integration testing working.
> >> > >>> >
> >> > >>> > Unfortunately the official Flatbuffers Rust version is not
> >> available
> >> > >>> yet
> >> > >>> > and my recent attempts at contacting the author have been
> >> unsuccessful
> >> > >>> so I
> >> > >>> > have started working with this fork of Flatbuffers which has Rust
> >> > >>> support:
> >> > >>> > https://github.com/josephDunne/flatbuffers.
> >> > >>> >
> >> > >>> > I was able to generate code from Schema.fbs but it doesn't
> >> compile and
> >> > >>> I've
> >> > >>> > started filing issues and debugging this.
> >> > >>> >
> >> > >>> > Working with this fork isn't ideal but I don't see what other
> >> choice I
> >> > >>> > have, other than just waiting for the official project to support
> >> Rust.
> >> > >>> >
> >> > >>> > I'm interested to hear if anyone has any alternate suggestions. I
> >> know
> >> > >>> it
> >> > >>> > would be possible to wrap C code but I'd like to keep the Rust
> >> > >>> > implementation as a pure Rust project if possible.
> >> > >>> >
> >> > >>> > Thanks,
> >> > >>> >
> >> > >>> > Andy.
> >> > >>>
> >> > >>
> >>
> >
Re: Rust IPC and Integration Testing
Posted by Andy Grove <an...@gmail.com>.
A question has been raised on this PR as to whether we should publish a
separate crate for the format/ipc generated code. I think there might be
some merit in this and wanted to raise it here to see what everyone thinks.
This would mean having two directories under the rust directory ... one for
arrow-format (or maybe the name should be arrow-ipc?) and one for the main
arrow crate.
It is possible we might want other crates in the future e.g. arrow-parquet
or whatever.
Andy.
On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <an...@gmail.com> wrote:
> I have created a PR this morning that adds the generated Flatbuffers code
> to the Rust implementation (https://github.com/apache/arrow/pull/2986).
>
> Just throwing these files into the repo doesn't add a ton of value until
> we use them for something, but it took some effort to generate them (and
> then fix a couple things by hand) so it seems useful to contribute this.
> The PR includes a brief README with the instructions for generating the
> files.
>
> I have taken this week off for Thanksgiving so I actually have some
> dedicated time to contribute to Arrow. I suppose my next step on this is to
> write a test that can serialize some data and then de-serialize it again.
>
> Andy.
>
>
>
>
>
>
>
> On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <we...@gmail.com> wrote:
>
>> hi Andy,
>>
>> AFAIK in the other implementations, the Flatbuffers serialization is
>> an implementation detail of offering support for the columnar IPC
>> protocol (which in turn depends on the columnar data structures,
>> memory management, etc.). I'm not sure what a crate offering the
>> Flatbuffers bindings alone would offer, outside of helping people
>> create new Rust Arrow implementations, I guess. In any case, if you
>> wanted to do that, we would need to make a release first.
>>
>> - Wes
>> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <an...@gmail.com> wrote:
>> >
>> > Brief update on this. I have now been able to use flatbuffers to
>> generate
>> > Rust code from the Arrow schema, and it compiles. In theory, this can be
>> > published as a standalone "arrow-format" crate but I'm wondering what
>> the
>> > plan should be here.
>> >
>> > I will start reviewing how this is handled in the other implementations
>> but
>> > if anyone has suggestions, please let me know.
>> >
>> > Thanks,
>> >
>> > Andy.
>> >
>> >
>> >
>> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <an...@gmail.com>
>> wrote:
>> >
>> > > Flatbuffers now has support for Rust - the PR was just merged in the
>> last
>> > > couple days:
>> > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
>> > >
>> > > I hope to find some time in the next week or two to start
>> experimenting
>> > > with this for Arrow IPC.
>> > >
>> > > Andy.
>> > >
>> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <an...@gmail.com>
>> wrote:
>> > >
>> > >> The author of the Rust version of FlatBuffers is now working on a
>> public
>> > >> fork and seems to be making good progress. I just wanted to send a
>> quick
>> > >> update on this since I am now back onto working on Arrow IPC in Rust
>> (I'm
>> > >> sure this is going to take quite a while though so don't want to get
>> > >> anyones hopes up too much). For anyone that is interested, here is
>> the
>> > >> FlatBuffers branch:
>> https://github.com/rw/flatbuffers/tree/2018-02--rust
>> > >>
>> > >> Andy.
>> > >>
>> > >>
>> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <we...@gmail.com>
>> wrote:
>> > >>
>> > >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and
>> > >>> Rust. One possibility is to build an interface to the Flatbuffers
>> data
>> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if
>> these
>> > >>> bindings are header-only like C++ and if that makes things any
>> easier
>> > >>> for you.
>> > >>>
>> > >>> If you defined an API in Rust for reading and writing the Arrow
>> > >>> metadata that does not expose any Flatbuffers-specific details, then
>> > >>> once there is a production-grade Rust implementation of Flatbuffers,
>> > >>> the implementation details could be swapped out without disruption.
>> > >>>
>> > >>> > my recent attempts at contacting the author have been unsuccessful
>> > >>>
>> > >>> Have you e-mailed the author directly or only pings on GitHub? Pings
>> > >>> may not be making it to their inbox.
>> > >>>
>> > >>> Appreciate your efforts on this; I think we'll see a lot more work
>> in
>> > >>> data processing in Rust in the coming years.
>> > >>>
>> > >>> - Wes
>> > >>>
>> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <an...@gmail.com>
>> > >>> wrote:
>> > >>> > Hi,
>> > >>> >
>> > >>> > Now that the refactor I've been working on has been merged, the
>> next
>> > >>> > priority for me personally with the Rust implementation is
>> getting IPC
>> > >>> and
>> > >>> > integration testing working.
>> > >>> >
>> > >>> > Unfortunately the official Flatbuffers Rust version is not
>> available
>> > >>> yet
>> > >>> > and my recent attempts at contacting the author have been
>> unsuccessful
>> > >>> so I
>> > >>> > have started working with this fork of Flatbuffers which has Rust
>> > >>> support:
>> > >>> > https://github.com/josephDunne/flatbuffers.
>> > >>> >
>> > >>> > I was able to generate code from Schema.fbs but it doesn't
>> compile and
>> > >>> I've
>> > >>> > started filing issues and debugging this.
>> > >>> >
>> > >>> > Working with this fork isn't ideal but I don't see what other
>> choice I
>> > >>> > have, other than just waiting for the official project to support
>> Rust.
>> > >>> >
>> > >>> > I'm interested to hear if anyone has any alternate suggestions. I
>> know
>> > >>> it
>> > >>> > would be possible to wrap C code but I'd like to keep the Rust
>> > >>> > implementation as a pure Rust project if possible.
>> > >>> >
>> > >>> > Thanks,
>> > >>> >
>> > >>> > Andy.
>> > >>>
>> > >>
>>
>
Re: Rust IPC and Integration Testing
Posted by Andy Grove <an...@gmail.com>.
I have created a PR this morning that adds the generated Flatbuffers code
to the Rust implementation (https://github.com/apache/arrow/pull/2986).
Just throwing these files into the repo doesn't add a ton of value until we
use them for something, but it took some effort to generate them (and then
fix a couple things by hand) so it seems useful to contribute this. The PR
includes a brief README with the instructions for generating the files.
I have taken this week off for Thanksgiving so I actually have some
dedicated time to contribute to Arrow. I suppose my next step on this is to
write a test that can serialize some data and then de-serialize it again.
Andy.
On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <we...@gmail.com> wrote:
> hi Andy,
>
> AFAIK in the other implementations, the Flatbuffers serialization is
> an implementation detail of offering support for the columnar IPC
> protocol (which in turn depends on the columnar data structures,
> memory management, etc.). I'm not sure what a crate offering the
> Flatbuffers bindings alone would offer, outside of helping people
> create new Rust Arrow implementations, I guess. In any case, if you
> wanted to do that, we would need to make a release first.
>
> - Wes
> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <an...@gmail.com> wrote:
> >
> > Brief update on this. I have now been able to use flatbuffers to generate
> > Rust code from the Arrow schema, and it compiles. In theory, this can be
> > published as a standalone "arrow-format" crate but I'm wondering what the
> > plan should be here.
> >
> > I will start reviewing how this is handled in the other implementations
> but
> > if anyone has suggestions, please let me know.
> >
> > Thanks,
> >
> > Andy.
> >
> >
> >
> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <an...@gmail.com> wrote:
> >
> > > Flatbuffers now has support for Rust - the PR was just merged in the
> last
> > > couple days:
> > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
> > >
> > > I hope to find some time in the next week or two to start experimenting
> > > with this for Arrow IPC.
> > >
> > > Andy.
> > >
> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <an...@gmail.com>
> wrote:
> > >
> > >> The author of the Rust version of FlatBuffers is now working on a
> public
> > >> fork and seems to be making good progress. I just wanted to send a
> quick
> > >> update on this since I am now back onto working on Arrow IPC in Rust
> (I'm
> > >> sure this is going to take quite a while though so don't want to get
> > >> anyones hopes up too much). For anyone that is interested, here is the
> > >> FlatBuffers branch:
> https://github.com/rw/flatbuffers/tree/2018-02--rust
> > >>
> > >> Andy.
> > >>
> > >>
> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >>
> > >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and
> > >>> Rust. One possibility is to build an interface to the Flatbuffers
> data
> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if
> these
> > >>> bindings are header-only like C++ and if that makes things any easier
> > >>> for you.
> > >>>
> > >>> If you defined an API in Rust for reading and writing the Arrow
> > >>> metadata that does not expose any Flatbuffers-specific details, then
> > >>> once there is a production-grade Rust implementation of Flatbuffers,
> > >>> the implementation details could be swapped out without disruption.
> > >>>
> > >>> > my recent attempts at contacting the author have been unsuccessful
> > >>>
> > >>> Have you e-mailed the author directly or only pings on GitHub? Pings
> > >>> may not be making it to their inbox.
> > >>>
> > >>> Appreciate your efforts on this; I think we'll see a lot more work in
> > >>> data processing in Rust in the coming years.
> > >>>
> > >>> - Wes
> > >>>
> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <an...@gmail.com>
> > >>> wrote:
> > >>> > Hi,
> > >>> >
> > >>> > Now that the refactor I've been working on has been merged, the
> next
> > >>> > priority for me personally with the Rust implementation is getting
> IPC
> > >>> and
> > >>> > integration testing working.
> > >>> >
> > >>> > Unfortunately the official Flatbuffers Rust version is not
> available
> > >>> yet
> > >>> > and my recent attempts at contacting the author have been
> unsuccessful
> > >>> so I
> > >>> > have started working with this fork of Flatbuffers which has Rust
> > >>> support:
> > >>> > https://github.com/josephDunne/flatbuffers.
> > >>> >
> > >>> > I was able to generate code from Schema.fbs but it doesn't compile
> and
> > >>> I've
> > >>> > started filing issues and debugging this.
> > >>> >
> > >>> > Working with this fork isn't ideal but I don't see what other
> choice I
> > >>> > have, other than just waiting for the official project to support
> Rust.
> > >>> >
> > >>> > I'm interested to hear if anyone has any alternate suggestions. I
> know
> > >>> it
> > >>> > would be possible to wrap C code but I'd like to keep the Rust
> > >>> > implementation as a pure Rust project if possible.
> > >>> >
> > >>> > Thanks,
> > >>> >
> > >>> > Andy.
> > >>>
> > >>
>
Re: Rust IPC and Integration Testing
Posted by Wes McKinney <we...@gmail.com>.
hi Andy,
AFAIK in the other implementations, the Flatbuffers serialization is
an implementation detail of offering support for the columnar IPC
protocol (which in turn depends on the columnar data structures,
memory management, etc.). I'm not sure what a crate offering the
Flatbuffers bindings alone would offer, outside of helping people
create new Rust Arrow implementations, I guess. In any case, if you
wanted to do that, we would need to make a release first.
- Wes
On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <an...@gmail.com> wrote:
>
> Brief update on this. I have now been able to use flatbuffers to generate
> Rust code from the Arrow schema, and it compiles. In theory, this can be
> published as a standalone "arrow-format" crate but I'm wondering what the
> plan should be here.
>
> I will start reviewing how this is handled in the other implementations but
> if anyone has suggestions, please let me know.
>
> Thanks,
>
> Andy.
>
>
>
> On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <an...@gmail.com> wrote:
>
> > Flatbuffers now has support for Rust - the PR was just merged in the last
> > couple days:
> > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
> >
> > I hope to find some time in the next week or two to start experimenting
> > with this for Arrow IPC.
> >
> > Andy.
> >
> > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <an...@gmail.com> wrote:
> >
> >> The author of the Rust version of FlatBuffers is now working on a public
> >> fork and seems to be making good progress. I just wanted to send a quick
> >> update on this since I am now back onto working on Arrow IPC in Rust (I'm
> >> sure this is going to take quite a while though so don't want to get
> >> anyones hopes up too much). For anyone that is interested, here is the
> >> FlatBuffers branch: https://github.com/rw/flatbuffers/tree/2018-02--rust
> >>
> >> Andy.
> >>
> >>
> >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <we...@gmail.com> wrote:
> >>
> >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and
> >>> Rust. One possibility is to build an interface to the Flatbuffers data
> >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if these
> >>> bindings are header-only like C++ and if that makes things any easier
> >>> for you.
> >>>
> >>> If you defined an API in Rust for reading and writing the Arrow
> >>> metadata that does not expose any Flatbuffers-specific details, then
> >>> once there is a production-grade Rust implementation of Flatbuffers,
> >>> the implementation details could be swapped out without disruption.
> >>>
> >>> > my recent attempts at contacting the author have been unsuccessful
> >>>
> >>> Have you e-mailed the author directly or only pings on GitHub? Pings
> >>> may not be making it to their inbox.
> >>>
> >>> Appreciate your efforts on this; I think we'll see a lot more work in
> >>> data processing in Rust in the coming years.
> >>>
> >>> - Wes
> >>>
> >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <an...@gmail.com>
> >>> wrote:
> >>> > Hi,
> >>> >
> >>> > Now that the refactor I've been working on has been merged, the next
> >>> > priority for me personally with the Rust implementation is getting IPC
> >>> and
> >>> > integration testing working.
> >>> >
> >>> > Unfortunately the official Flatbuffers Rust version is not available
> >>> yet
> >>> > and my recent attempts at contacting the author have been unsuccessful
> >>> so I
> >>> > have started working with this fork of Flatbuffers which has Rust
> >>> support:
> >>> > https://github.com/josephDunne/flatbuffers.
> >>> >
> >>> > I was able to generate code from Schema.fbs but it doesn't compile and
> >>> I've
> >>> > started filing issues and debugging this.
> >>> >
> >>> > Working with this fork isn't ideal but I don't see what other choice I
> >>> > have, other than just waiting for the official project to support Rust.
> >>> >
> >>> > I'm interested to hear if anyone has any alternate suggestions. I know
> >>> it
> >>> > would be possible to wrap C code but I'd like to keep the Rust
> >>> > implementation as a pure Rust project if possible.
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Andy.
> >>>
> >>
Re: Rust IPC and Integration Testing
Posted by Andy Grove <an...@gmail.com>.
Brief update on this. I have now been able to use flatbuffers to generate
Rust code from the Arrow schema, and it compiles. In theory, this can be
published as a standalone "arrow-format" crate but I'm wondering what the
plan should be here.
I will start reviewing how this is handled in the other implementations but
if anyone has suggestions, please let me know.
Thanks,
Andy.
On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <an...@gmail.com> wrote:
> Flatbuffers now has support for Rust - the PR was just merged in the last
> couple days:
> https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
>
> I hope to find some time in the next week or two to start experimenting
> with this for Arrow IPC.
>
> Andy.
>
> On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <an...@gmail.com> wrote:
>
>> The author of the Rust version of FlatBuffers is now working on a public
>> fork and seems to be making good progress. I just wanted to send a quick
>> update on this since I am now back onto working on Arrow IPC in Rust (I'm
>> sure this is going to take quite a while though so don't want to get
>> anyones hopes up too much). For anyone that is interested, here is the
>> FlatBuffers branch: https://github.com/rw/flatbuffers/tree/2018-02--rust
>>
>> Andy.
>>
>>
>> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <we...@gmail.com> wrote:
>>
>>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and
>>> Rust. One possibility is to build an interface to the Flatbuffers data
>>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if these
>>> bindings are header-only like C++ and if that makes things any easier
>>> for you.
>>>
>>> If you defined an API in Rust for reading and writing the Arrow
>>> metadata that does not expose any Flatbuffers-specific details, then
>>> once there is a production-grade Rust implementation of Flatbuffers,
>>> the implementation details could be swapped out without disruption.
>>>
>>> > my recent attempts at contacting the author have been unsuccessful
>>>
>>> Have you e-mailed the author directly or only pings on GitHub? Pings
>>> may not be making it to their inbox.
>>>
>>> Appreciate your efforts on this; I think we'll see a lot more work in
>>> data processing in Rust in the coming years.
>>>
>>> - Wes
>>>
>>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <an...@gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > Now that the refactor I've been working on has been merged, the next
>>> > priority for me personally with the Rust implementation is getting IPC
>>> and
>>> > integration testing working.
>>> >
>>> > Unfortunately the official Flatbuffers Rust version is not available
>>> yet
>>> > and my recent attempts at contacting the author have been unsuccessful
>>> so I
>>> > have started working with this fork of Flatbuffers which has Rust
>>> support:
>>> > https://github.com/josephDunne/flatbuffers.
>>> >
>>> > I was able to generate code from Schema.fbs but it doesn't compile and
>>> I've
>>> > started filing issues and debugging this.
>>> >
>>> > Working with this fork isn't ideal but I don't see what other choice I
>>> > have, other than just waiting for the official project to support Rust.
>>> >
>>> > I'm interested to hear if anyone has any alternate suggestions. I know
>>> it
>>> > would be possible to wrap C code but I'd like to keep the Rust
>>> > implementation as a pure Rust project if possible.
>>> >
>>> > Thanks,
>>> >
>>> > Andy.
>>>
>>