You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Renjie Liu <li...@gmail.com> on 2018/08/20 02:39:57 UTC
Fwd: [DISCUSS] Rust add adapter for parquet
cc:Sunchao and Any
---------- Forwarded message ---------
From: Uwe L. Korn <uw...@xhochy.com>
Date: Sun, Aug 19, 2018 at 5:08 PM
Subject: Re: [DISCUSS] Rust add adapter for parquet
To: <de...@arrow.apache.org>
Hello,
you might also want to raise this with the
https://github.com/sunchao/parquet-rs project. The overlap between the
developers of this project and the Arrow Rust implementation is quite large
but still it may make sense to also start a discussion there.
Uwe
On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> Hi, all:
>
> Now the rust component is approaching a stable state and rust reader for
> parquet is ready. I think it maybe a good time to start an adapter for
> parquet, just like adapter for orc in cpp. How you guys think about it?
> --
> Liu, Renjie
> Software Engineer, MVAD
--
Liu, Renjie
Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Ivan Sadikov <iv...@gmail.com>.
Hello!
That would be great! Agree with Chao and Wes, we should do it similar to
parquet-cpp, as long as it does not make it difficult for others to work
with the Arrow repository:).
Ha, Arrow data source sounds interesting. I will also catch up on the Arrow
development.
Cheers,
Ivan
On Tue, 20 Nov 2018 at 9:49 PM, Andy Grove <an...@gmail.com> wrote:
> This sounds like a great idea.
>
> With support for both CSV and Parquet in the Arrow crate, it would be nice
> to design a standard interface for Arrow data sources. Maybe this is as
> simple as implementing `Iterator<RecordBatch>`.
>
> Andy.
>
> On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
>
>> Yes, we'd be interested to move forward. I'm inclined to merge this into
>> Arrow because of the issues that you pointed out with parquet c++ merge,
>> and I do see a tight relationship between the two projects, and potential
>> sharing of common libraries. @Ivan Sadikov <iv...@gmail.com> what
>> do you think?
>>
>> Chao
>>
>> On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
>> wrote:
>>
>>> hi folks,
>>>
>>> Would you all be interested in moving forward the parquet-rs project?
>>> I have a little more bandwidth to help with the code donation in the
>>> next month or two.
>>>
>>> I know we voted on the Parquet mailing list about the donation
>>> already. One big question is whether you want to create an
>>> apache/parquet-rs repository or whether you want to co-develop
>>> parquet-rs together with Arrow in Rust, similar to what we are doing
>>> with C++. It's possible you might run into the same kinds of issues
>>> that led us to consider the monorepo arrangement.
>>>
>>> Thanks
>>> Wes
>>> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
>>> wrote:
>>> >
>>> > Hi, Chao:
>>> > I've opened an jira issue for that and planning to work on that.
>>> >
>>> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com>
>>> wrote:
>>> >
>>> > > Yes, it's a mistake, sorry for that
>>> > >
>>> > >
>>> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
>>> wrote:
>>> > >
>>> > >> (s/flink/arrow - it is a mistake?)
>>> > >>
>>> > >> Thanks Renjie for your interest. Yes, one of the next step in
>>> parquet-rs
>>> > >> is to integrate with Apache Arrow. Actually we just had a discussion
>>> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
>>> recently.
>>> > >> Feel free to share your comments on the github.
>>> > >>
>>> > >> Best,
>>> > >> Chao
>>> > >>
>>> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
>>> liurenjie2008@gmail.com>
>>> > >> wrote:
>>> > >>
>>> > >>> cc:Sunchao and Any
>>> > >>>
>>> > >>>
>>> > >>> ---------- Forwarded message ---------
>>> > >>> From: Uwe L. Korn <uw...@xhochy.com>
>>> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
>>> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
>>> > >>> To: <de...@arrow.apache.org>
>>> > >>>
>>> > >>>
>>> > >>> Hello,
>>> > >>>
>>> > >>> you might also want to raise this with the
>>> > >>> https://github.com/sunchao/parquet-rs project. The overlap
>>> between the
>>> > >>> developers of this project and the Arrow Rust implementation is
>>> quite large
>>> > >>> but still it may make sense to also start a discussion there.
>>> > >>>
>>> > >>> Uwe
>>> > >>>
>>> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
>>> > >>> > Hi, all:
>>> > >>> >
>>> > >>> > Now the rust component is approaching a stable state and rust
>>> reader
>>> > >>> for
>>> > >>> > parquet is ready. I think it maybe a good time to start an
>>> adapter for
>>> > >>> > parquet, just like adapter for orc in cpp. How you guys think
>>> about it?
>>> > >>> > --
>>> > >>> > Liu, Renjie
>>> > >>> > Software Engineer, MVAD
>>> > >>> --
>>> > >>> Liu, Renjie
>>> > >>> Software Engineer, MVAD
>>> > >>>
>>> > >>
>>> > >> --
>>> > > Liu, Renjie
>>> > > Software Engineer, MVAD
>>> > >
>>> > --
>>> > Liu, Renjie
>>> > Software Engineer, MVAD
>>>
>>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Wes McKinney <we...@gmail.com>.
Thanks. I will review today and start a vote about the code donation.
I know that we already voted to accept in Apache Parquet, but I want
to double check that the Arrow community is also on board with sharing
responsibility for this code. If the Parquet community wants to make
Rust Parquet releases, this is something that can be discussed and
handled later.
Do any corporations have IP rights in this code (to see if a Software
Grant Agreement is necessary)? Otherwise your ICLAs are sufficient.
On Thu, Nov 29, 2018 at 12:45 AM Chao Sun <su...@apache.org> wrote:
>
> I just opened a pull request <https://github.com/apache/arrow/pull/3050> for
> the donation. Let me know if this is in the correct format and whether it's
> OK to proceed to the next steps.
>
> Thanks,
> Chao
>
> On Wed, Nov 21, 2018 at 9:18 PM Wes McKinney <we...@gmail.com> wrote:
>
> > > Is the change to the nightly compiler the kind of thing that the PMC
> > should vote on?
> >
> > A vote is not necessary if there is reasonable consensus.
> > On Wed, Nov 21, 2018 at 9:11 PM paddy horan <pa...@hotmail.com>
> > wrote:
> > >
> > > I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll
> > take a closer look when I get a chance. I agree that lack of support for 32
> > bit should not hold this up.
> > >
> > > Is the change to the nightly compiler the kind of thing that the PMC
> > should vote on? This could be done in advance of the code donantion.
> > Specialization, in particular, would be really useful within the existing
> > code base.
> > >
> > > Paddy
> > >
> > > Get Outlook for iOS
> > > ________________________________
> > > From: Chao Sun <su...@apache.org>
> > > Sent: Wednesday, November 21, 2018 2:42 PM
> > > To: dev@arrow.apache.org
> > > Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev
> > > Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >
> > > > A bigger issue is windows support for parquet-rs, Chao – I don’t
> > believe
> > > that parquet-rs supports windows, right? When I test it myself I get
> > > errors regarding clang and libclang which parquet-rs must use.
> > >
> > > I think there are some issue regarding clang working with zstd on 32-bit
> > > platform. However, it was able to compile with target:
> > x86_64-pc-windows-msvc,
> > > which seems to be the only one that arrow is using for windows CI. So I
> > > think we can move forward and address the 32-bit platform issue later.
> > >
> > > On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com>
> > wrote:
> > >
> > > > I think using nightly is fine for the reasons mentioned already. We
> > > > should switch our CI to still run CI against stable (non fatal) so we
> > know
> > > > when we can make the move back to stable.
> > > >
> > > >
> > > >
> > > > A bigger issue is windows support for parquet-rs, Chao – I don’t
> > believe
> > > > that parquet-rs supports windows, right? When I test it myself I get
> > > > errors regarding clang and libclang which parquet-rs must use.
> > > >
> > > >
> > > >
> > > > We have had windows support pretty much since the beginning for Rust.
> > Is
> > > > it possible to put parquet support behind a feature gate initially and
> > only
> > > > run CI for non-windows? I would be willing to help get windows support
> > > > working after the fact, although I know very little about parquet
> > right now.
> > > >
> > > >
> > > >
> > > > Are there other strategies for dealing with this?
> > > >
> > > >
> > > >
> > > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> > > > Windows 10
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: Chao Sun <su...@apache.org>
> > > > Sent: Wednesday, November 21, 2018 12:52:32 PM
> > > > To: Wes McKinney
> > > > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> > > > Subject: Re: [DISCUSS] Rust add adapter for parquet
> > > >
> > > > > Can you remind us all why nightly is required?
> > > >
> > > > Here's a tracking issue <
> > https://github.com/sunchao/parquet-rs/issues/119>
> > > > for all the unstable features parquet-rs uses. I'm personally inclined
> > to
> > > > use nightly since some new features such as specialization makes
> > > > development much easier. Like Andy mentioned, as we are still
> > developing
> > > > arrow + parquet I think we can stay with nightly and transition back to
> > > > stable once major features are implemented and more and more people
> > start
> > > > to use it (hopefully by that time the unstable features are
> > stabilized).
> > > > Moreover, I've seen quite a few popular projects rely on nightly such
> > as
> > > > rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> > > >
> > > > > The steps from here are for you all to get the codebase into a state
> > > > that is ready for donation, including ASF license headers, etc. A pull
> > > > request into apache/arrow would be the best thing
> > > >
> > > > Sure. I'll prepare a pull request in the next few days, and then we can
> > > > proceed to the voting, ICLA, etc. Thanks.
> > > >
> > > > Chao
> > > >
> > > >
> > > > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > > The steps from here are for you all to get the codebase into a state
> > > > > that is ready for donation, including ASF license headers, etc. A
> > pull
> > > > > request into apache/arrow would be the best thing
> > > > >
> > > > > Then we have to do the following
> > > > >
> > > > > * Vote on the Arrow mailing list
> > > > > * Receive ICLAs from contributors
> > > > > * Complete IP clearance
> > > > > * Merge codebase
> > > > >
> > > > > Let me know when you are ready to move forward. From start to finish
> > > > > that can get done in approximately 6 days if the code is ready
> > > > >
> > > > > Thanks
> > > > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Renjie,
> > > > > >
> > > > > > Can you remind us all why nightly is required?
> > > > > >
> > > > > > My personal feeling is that stable is a nice-to-have, but Rust is
> > still
> > > > > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > > > > relying on nightly for now. Maybe we can have a plan to transition
> > back
> > > > to
> > > > > stable for a future release if we go with nightly now.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Andy.
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <
> > liurenjie2008@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> That sounds great. But parquet-rs currently relies on nightly
> > rust,
> > > > that
> > > > > >> would be the first problem to resolve.
> > > > > >>
> > > > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <andygrove73@gmail.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > This sounds like a great idea.
> > > > > >> >
> > > > > >> > With support for both CSV and Parquet in the Arrow crate, it
> > would
> > > > be
> > > > > nice
> > > > > >> > to design a standard interface for Arrow data sources. Maybe
> > this is
> > > > > as
> > > > > >> > simple as implementing `Iterator<RecordBatch>`.
> > > > > >> >
> > > > > >> > Andy.
> > > > > >> >
> > > > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> > > > wrote:
> > > > > >> >
> > > > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> > > > this
> > > > > into
> > > > > >> > > Arrow because of the issues that you pointed out with parquet
> > c++
> > > > > merge,
> > > > > >> > > and I do see a tight relationship between the two projects,
> > and
> > > > > potential
> > > > > >> > > sharing of common libraries. @Ivan Sadikov <
> > > > ivan.sadikov@gmail.com>
> > > > > what
> > > > > >> > > do you think?
> > > > > >> > >
> > > > > >> > > Chao
> > > > > >> > >
> > > > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> > > > wesmckinn@gmail.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> hi folks,
> > > > > >> > >>
> > > > > >> > >> Would you all be interested in moving forward the parquet-rs
> > > > > project?
> > > > > >> > >> I have a little more bandwidth to help with the code
> > donation in
> > > > > the
> > > > > >> > >> next month or two.
> > > > > >> > >>
> > > > > >> > >> I know we voted on the Parquet mailing list about the
> > donation
> > > > > >> > >> already. One big question is whether you want to create an
> > > > > >> > >> apache/parquet-rs repository or whether you want to
> > co-develop
> > > > > >> > >> parquet-rs together with Arrow in Rust, similar to what we
> > are
> > > > > doing
> > > > > >> > >> with C++. It's possible you might run into the same kinds of
> > > > issues
> > > > > >> > >> that led us to consider the monorepo arrangement.
> > > > > >> > >>
> > > > > >> > >> Thanks
> > > > > >> > >> Wes
> > > > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > > > > liurenjie2008@gmail.com>
> > > > > >> > >> wrote:
> > > > > >> > >> >
> > > > > >> > >> > Hi, Chao:
> > > > > >> > >> > I've opened an jira issue for that and planning to work on
> > > > that.
> > > > > >> > >> >
> > > > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > > > > liurenjie2008@gmail.com>
> > > > > >> > >> wrote:
> > > > > >> > >> >
> > > > > >> > >> > > Yes, it's a mistake, sorry for that
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> > > > sunchao@apache.org>
> > > > > >> > wrote:
> > > > > >> > >> > >
> > > > > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > > > > >> > >> > >>
> > > > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next
> > step
> > > > in
> > > > > >> > >> parquet-rs
> > > > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had
> > a
> > > > > >> > discussion
> > > > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140>
> > about
> > > > this
> > > > > >> > >> recently.
> > > > > >> > >> > >> Feel free to share your comments on the github.
> > > > > >> > >> > >>
> > > > > >> > >> > >> Best,
> > > > > >> > >> > >> Chao
> > > > > >> > >> > >>
> > > > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > > > > >> > liurenjie2008@gmail.com
> > > > > >> > >> >
> > > > > >> > >> > >> wrote:
> > > > > >> > >> > >>
> > > > > >> > >> > >>> cc:Sunchao and Any
> > > > > >> > >> > >>>
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> ---------- Forwarded message ---------
> > > > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > > > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > > > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > > > > >> > >> > >>> To: <de...@arrow.apache.org>
> > > > > >> > >> > >>>
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> Hello,
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> you might also want to raise this with the
> > > > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The
> > overlap
> > > > > >> > between
> > > > > >> > >> the
> > > > > >> > >> > >>> developers of this project and the Arrow Rust
> > > > implementation
> > > > > is
> > > > > >> > >> quite large
> > > > > >> > >> > >>> but still it may make sense to also start a discussion
> > > > there.
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> Uwe
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > > > > >> > >> > >>> > Hi, all:
> > > > > >> > >> > >>> >
> > > > > >> > >> > >>> > Now the rust component is approaching a stable state
> > and
> > > > > rust
> > > > > >> > >> reader
> > > > > >> > >> > >>> for
> > > > > >> > >> > >>> > parquet is ready. I think it maybe a good time to
> > start
> > > > an
> > > > > >> > >> adapter for
> > > > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you
> > guys
> > > > > think
> > > > > >> > >> about it?
> > > > > >> > >> > >>> > --
> > > > > >> > >> > >>> > Liu, Renjie
> > > > > >> > >> > >>> > Software Engineer, MVAD
> > > > > >> > >> > >>> --
> > > > > >> > >> > >>> Liu, Renjie
> > > > > >> > >> > >>> Software Engineer, MVAD
> > > > > >> > >> > >>>
> > > > > >> > >> > >>
> > > > > >> > >> > >> --
> > > > > >> > >> > > Liu, Renjie
> > > > > >> > >> > > Software Engineer, MVAD
> > > > > >> > >> > >
> > > > > >> > >> > --
> > > > > >> > >> > Liu, Renjie
> > > > > >> > >> > Software Engineer, MVAD
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >> --
> > > > > >> Renjie Liu
> > > > > >> Software Engineer, MVAD
> > > > >
> > > >
> >
Re: [DISCUSS] Rust add adapter for parquet
Posted by Wes McKinney <we...@gmail.com>.
Thanks. I will review today and start a vote about the code donation.
I know that we already voted to accept in Apache Parquet, but I want
to double check that the Arrow community is also on board with sharing
responsibility for this code. If the Parquet community wants to make
Rust Parquet releases, this is something that can be discussed and
handled later.
Do any corporations have IP rights in this code (to see if a Software
Grant Agreement is necessary)? Otherwise your ICLAs are sufficient.
On Thu, Nov 29, 2018 at 12:45 AM Chao Sun <su...@apache.org> wrote:
>
> I just opened a pull request <https://github.com/apache/arrow/pull/3050> for
> the donation. Let me know if this is in the correct format and whether it's
> OK to proceed to the next steps.
>
> Thanks,
> Chao
>
> On Wed, Nov 21, 2018 at 9:18 PM Wes McKinney <we...@gmail.com> wrote:
>
> > > Is the change to the nightly compiler the kind of thing that the PMC
> > should vote on?
> >
> > A vote is not necessary if there is reasonable consensus.
> > On Wed, Nov 21, 2018 at 9:11 PM paddy horan <pa...@hotmail.com>
> > wrote:
> > >
> > > I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll
> > take a closer look when I get a chance. I agree that lack of support for 32
> > bit should not hold this up.
> > >
> > > Is the change to the nightly compiler the kind of thing that the PMC
> > should vote on? This could be done in advance of the code donantion.
> > Specialization, in particular, would be really useful within the existing
> > code base.
> > >
> > > Paddy
> > >
> > > Get Outlook for iOS
> > > ________________________________
> > > From: Chao Sun <su...@apache.org>
> > > Sent: Wednesday, November 21, 2018 2:42 PM
> > > To: dev@arrow.apache.org
> > > Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev
> > > Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >
> > > > A bigger issue is windows support for parquet-rs, Chao – I don’t
> > believe
> > > that parquet-rs supports windows, right? When I test it myself I get
> > > errors regarding clang and libclang which parquet-rs must use.
> > >
> > > I think there are some issue regarding clang working with zstd on 32-bit
> > > platform. However, it was able to compile with target:
> > x86_64-pc-windows-msvc,
> > > which seems to be the only one that arrow is using for windows CI. So I
> > > think we can move forward and address the 32-bit platform issue later.
> > >
> > > On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com>
> > wrote:
> > >
> > > > I think using nightly is fine for the reasons mentioned already. We
> > > > should switch our CI to still run CI against stable (non fatal) so we
> > know
> > > > when we can make the move back to stable.
> > > >
> > > >
> > > >
> > > > A bigger issue is windows support for parquet-rs, Chao – I don’t
> > believe
> > > > that parquet-rs supports windows, right? When I test it myself I get
> > > > errors regarding clang and libclang which parquet-rs must use.
> > > >
> > > >
> > > >
> > > > We have had windows support pretty much since the beginning for Rust.
> > Is
> > > > it possible to put parquet support behind a feature gate initially and
> > only
> > > > run CI for non-windows? I would be willing to help get windows support
> > > > working after the fact, although I know very little about parquet
> > right now.
> > > >
> > > >
> > > >
> > > > Are there other strategies for dealing with this?
> > > >
> > > >
> > > >
> > > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> > > > Windows 10
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: Chao Sun <su...@apache.org>
> > > > Sent: Wednesday, November 21, 2018 12:52:32 PM
> > > > To: Wes McKinney
> > > > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> > > > Subject: Re: [DISCUSS] Rust add adapter for parquet
> > > >
> > > > > Can you remind us all why nightly is required?
> > > >
> > > > Here's a tracking issue <
> > https://github.com/sunchao/parquet-rs/issues/119>
> > > > for all the unstable features parquet-rs uses. I'm personally inclined
> > to
> > > > use nightly since some new features such as specialization makes
> > > > development much easier. Like Andy mentioned, as we are still
> > developing
> > > > arrow + parquet I think we can stay with nightly and transition back to
> > > > stable once major features are implemented and more and more people
> > start
> > > > to use it (hopefully by that time the unstable features are
> > stabilized).
> > > > Moreover, I've seen quite a few popular projects rely on nightly such
> > as
> > > > rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> > > >
> > > > > The steps from here are for you all to get the codebase into a state
> > > > that is ready for donation, including ASF license headers, etc. A pull
> > > > request into apache/arrow would be the best thing
> > > >
> > > > Sure. I'll prepare a pull request in the next few days, and then we can
> > > > proceed to the voting, ICLA, etc. Thanks.
> > > >
> > > > Chao
> > > >
> > > >
> > > > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >
> > > > > The steps from here are for you all to get the codebase into a state
> > > > > that is ready for donation, including ASF license headers, etc. A
> > pull
> > > > > request into apache/arrow would be the best thing
> > > > >
> > > > > Then we have to do the following
> > > > >
> > > > > * Vote on the Arrow mailing list
> > > > > * Receive ICLAs from contributors
> > > > > * Complete IP clearance
> > > > > * Merge codebase
> > > > >
> > > > > Let me know when you are ready to move forward. From start to finish
> > > > > that can get done in approximately 6 days if the code is ready
> > > > >
> > > > > Thanks
> > > > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Renjie,
> > > > > >
> > > > > > Can you remind us all why nightly is required?
> > > > > >
> > > > > > My personal feeling is that stable is a nice-to-have, but Rust is
> > still
> > > > > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > > > > relying on nightly for now. Maybe we can have a plan to transition
> > back
> > > > to
> > > > > stable for a future release if we go with nightly now.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Andy.
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <
> > liurenjie2008@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> That sounds great. But parquet-rs currently relies on nightly
> > rust,
> > > > that
> > > > > >> would be the first problem to resolve.
> > > > > >>
> > > > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <andygrove73@gmail.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > This sounds like a great idea.
> > > > > >> >
> > > > > >> > With support for both CSV and Parquet in the Arrow crate, it
> > would
> > > > be
> > > > > nice
> > > > > >> > to design a standard interface for Arrow data sources. Maybe
> > this is
> > > > > as
> > > > > >> > simple as implementing `Iterator<RecordBatch>`.
> > > > > >> >
> > > > > >> > Andy.
> > > > > >> >
> > > > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> > > > wrote:
> > > > > >> >
> > > > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> > > > this
> > > > > into
> > > > > >> > > Arrow because of the issues that you pointed out with parquet
> > c++
> > > > > merge,
> > > > > >> > > and I do see a tight relationship between the two projects,
> > and
> > > > > potential
> > > > > >> > > sharing of common libraries. @Ivan Sadikov <
> > > > ivan.sadikov@gmail.com>
> > > > > what
> > > > > >> > > do you think?
> > > > > >> > >
> > > > > >> > > Chao
> > > > > >> > >
> > > > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> > > > wesmckinn@gmail.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> hi folks,
> > > > > >> > >>
> > > > > >> > >> Would you all be interested in moving forward the parquet-rs
> > > > > project?
> > > > > >> > >> I have a little more bandwidth to help with the code
> > donation in
> > > > > the
> > > > > >> > >> next month or two.
> > > > > >> > >>
> > > > > >> > >> I know we voted on the Parquet mailing list about the
> > donation
> > > > > >> > >> already. One big question is whether you want to create an
> > > > > >> > >> apache/parquet-rs repository or whether you want to
> > co-develop
> > > > > >> > >> parquet-rs together with Arrow in Rust, similar to what we
> > are
> > > > > doing
> > > > > >> > >> with C++. It's possible you might run into the same kinds of
> > > > issues
> > > > > >> > >> that led us to consider the monorepo arrangement.
> > > > > >> > >>
> > > > > >> > >> Thanks
> > > > > >> > >> Wes
> > > > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > > > > liurenjie2008@gmail.com>
> > > > > >> > >> wrote:
> > > > > >> > >> >
> > > > > >> > >> > Hi, Chao:
> > > > > >> > >> > I've opened an jira issue for that and planning to work on
> > > > that.
> > > > > >> > >> >
> > > > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > > > > liurenjie2008@gmail.com>
> > > > > >> > >> wrote:
> > > > > >> > >> >
> > > > > >> > >> > > Yes, it's a mistake, sorry for that
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> > > > sunchao@apache.org>
> > > > > >> > wrote:
> > > > > >> > >> > >
> > > > > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > > > > >> > >> > >>
> > > > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next
> > step
> > > > in
> > > > > >> > >> parquet-rs
> > > > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had
> > a
> > > > > >> > discussion
> > > > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140>
> > about
> > > > this
> > > > > >> > >> recently.
> > > > > >> > >> > >> Feel free to share your comments on the github.
> > > > > >> > >> > >>
> > > > > >> > >> > >> Best,
> > > > > >> > >> > >> Chao
> > > > > >> > >> > >>
> > > > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > > > > >> > liurenjie2008@gmail.com
> > > > > >> > >> >
> > > > > >> > >> > >> wrote:
> > > > > >> > >> > >>
> > > > > >> > >> > >>> cc:Sunchao and Any
> > > > > >> > >> > >>>
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> ---------- Forwarded message ---------
> > > > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > > > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > > > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > > > > >> > >> > >>> To: <de...@arrow.apache.org>
> > > > > >> > >> > >>>
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> Hello,
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> you might also want to raise this with the
> > > > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The
> > overlap
> > > > > >> > between
> > > > > >> > >> the
> > > > > >> > >> > >>> developers of this project and the Arrow Rust
> > > > implementation
> > > > > is
> > > > > >> > >> quite large
> > > > > >> > >> > >>> but still it may make sense to also start a discussion
> > > > there.
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> Uwe
> > > > > >> > >> > >>>
> > > > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > > > > >> > >> > >>> > Hi, all:
> > > > > >> > >> > >>> >
> > > > > >> > >> > >>> > Now the rust component is approaching a stable state
> > and
> > > > > rust
> > > > > >> > >> reader
> > > > > >> > >> > >>> for
> > > > > >> > >> > >>> > parquet is ready. I think it maybe a good time to
> > start
> > > > an
> > > > > >> > >> adapter for
> > > > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you
> > guys
> > > > > think
> > > > > >> > >> about it?
> > > > > >> > >> > >>> > --
> > > > > >> > >> > >>> > Liu, Renjie
> > > > > >> > >> > >>> > Software Engineer, MVAD
> > > > > >> > >> > >>> --
> > > > > >> > >> > >>> Liu, Renjie
> > > > > >> > >> > >>> Software Engineer, MVAD
> > > > > >> > >> > >>>
> > > > > >> > >> > >>
> > > > > >> > >> > >> --
> > > > > >> > >> > > Liu, Renjie
> > > > > >> > >> > > Software Engineer, MVAD
> > > > > >> > >> > >
> > > > > >> > >> > --
> > > > > >> > >> > Liu, Renjie
> > > > > >> > >> > Software Engineer, MVAD
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >> --
> > > > > >> Renjie Liu
> > > > > >> Software Engineer, MVAD
> > > > >
> > > >
> >
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
I just opened a pull request <https://github.com/apache/arrow/pull/3050> for
the donation. Let me know if this is in the correct format and whether it's
OK to proceed to the next steps.
Thanks,
Chao
On Wed, Nov 21, 2018 at 9:18 PM Wes McKinney <we...@gmail.com> wrote:
> > Is the change to the nightly compiler the kind of thing that the PMC
> should vote on?
>
> A vote is not necessary if there is reasonable consensus.
> On Wed, Nov 21, 2018 at 9:11 PM paddy horan <pa...@hotmail.com>
> wrote:
> >
> > I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll
> take a closer look when I get a chance. I agree that lack of support for 32
> bit should not hold this up.
> >
> > Is the change to the nightly compiler the kind of thing that the PMC
> should vote on? This could be done in advance of the code donantion.
> Specialization, in particular, would be really useful within the existing
> code base.
> >
> > Paddy
> >
> > Get Outlook for iOS
> > ________________________________
> > From: Chao Sun <su...@apache.org>
> > Sent: Wednesday, November 21, 2018 2:42 PM
> > To: dev@arrow.apache.org
> > Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev
> > Subject: Re: [DISCUSS] Rust add adapter for parquet
> >
> > > A bigger issue is windows support for parquet-rs, Chao – I don’t
> believe
> > that parquet-rs supports windows, right? When I test it myself I get
> > errors regarding clang and libclang which parquet-rs must use.
> >
> > I think there are some issue regarding clang working with zstd on 32-bit
> > platform. However, it was able to compile with target:
> x86_64-pc-windows-msvc,
> > which seems to be the only one that arrow is using for windows CI. So I
> > think we can move forward and address the 32-bit platform issue later.
> >
> > On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com>
> wrote:
> >
> > > I think using nightly is fine for the reasons mentioned already. We
> > > should switch our CI to still run CI against stable (non fatal) so we
> know
> > > when we can make the move back to stable.
> > >
> > >
> > >
> > > A bigger issue is windows support for parquet-rs, Chao – I don’t
> believe
> > > that parquet-rs supports windows, right? When I test it myself I get
> > > errors regarding clang and libclang which parquet-rs must use.
> > >
> > >
> > >
> > > We have had windows support pretty much since the beginning for Rust.
> Is
> > > it possible to put parquet support behind a feature gate initially and
> only
> > > run CI for non-windows? I would be willing to help get windows support
> > > working after the fact, although I know very little about parquet
> right now.
> > >
> > >
> > >
> > > Are there other strategies for dealing with this?
> > >
> > >
> > >
> > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> > > Windows 10
> > >
> > >
> > >
> > > ________________________________
> > > From: Chao Sun <su...@apache.org>
> > > Sent: Wednesday, November 21, 2018 12:52:32 PM
> > > To: Wes McKinney
> > > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> > > Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >
> > > > Can you remind us all why nightly is required?
> > >
> > > Here's a tracking issue <
> https://github.com/sunchao/parquet-rs/issues/119>
> > > for all the unstable features parquet-rs uses. I'm personally inclined
> to
> > > use nightly since some new features such as specialization makes
> > > development much easier. Like Andy mentioned, as we are still
> developing
> > > arrow + parquet I think we can stay with nightly and transition back to
> > > stable once major features are implemented and more and more people
> start
> > > to use it (hopefully by that time the unstable features are
> stabilized).
> > > Moreover, I've seen quite a few popular projects rely on nightly such
> as
> > > rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> > >
> > > > The steps from here are for you all to get the codebase into a state
> > > that is ready for donation, including ASF license headers, etc. A pull
> > > request into apache/arrow would be the best thing
> > >
> > > Sure. I'll prepare a pull request in the next few days, and then we can
> > > proceed to the voting, ICLA, etc. Thanks.
> > >
> > > Chao
> > >
> > >
> > > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > The steps from here are for you all to get the codebase into a state
> > > > that is ready for donation, including ASF license headers, etc. A
> pull
> > > > request into apache/arrow would be the best thing
> > > >
> > > > Then we have to do the following
> > > >
> > > > * Vote on the Arrow mailing list
> > > > * Receive ICLAs from contributors
> > > > * Complete IP clearance
> > > > * Merge codebase
> > > >
> > > > Let me know when you are ready to move forward. From start to finish
> > > > that can get done in approximately 6 days if the code is ready
> > > >
> > > > Thanks
> > > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> > > wrote:
> > > > >
> > > > > Renjie,
> > > > >
> > > > > Can you remind us all why nightly is required?
> > > > >
> > > > > My personal feeling is that stable is a nice-to-have, but Rust is
> still
> > > > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > > > relying on nightly for now. Maybe we can have a plan to transition
> back
> > > to
> > > > stable for a future release if we go with nightly now.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Andy.
> > > > >
> > > > >
> > > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <
> liurenjie2008@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> That sounds great. But parquet-rs currently relies on nightly
> rust,
> > > that
> > > > >> would be the first problem to resolve.
> > > > >>
> > > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <andygrove73@gmail.com
> >
> > > > wrote:
> > > > >>
> > > > >> > This sounds like a great idea.
> > > > >> >
> > > > >> > With support for both CSV and Parquet in the Arrow crate, it
> would
> > > be
> > > > nice
> > > > >> > to design a standard interface for Arrow data sources. Maybe
> this is
> > > > as
> > > > >> > simple as implementing `Iterator<RecordBatch>`.
> > > > >> >
> > > > >> > Andy.
> > > > >> >
> > > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> > > this
> > > > into
> > > > >> > > Arrow because of the issues that you pointed out with parquet
> c++
> > > > merge,
> > > > >> > > and I do see a tight relationship between the two projects,
> and
> > > > potential
> > > > >> > > sharing of common libraries. @Ivan Sadikov <
> > > ivan.sadikov@gmail.com>
> > > > what
> > > > >> > > do you think?
> > > > >> > >
> > > > >> > > Chao
> > > > >> > >
> > > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > >> hi folks,
> > > > >> > >>
> > > > >> > >> Would you all be interested in moving forward the parquet-rs
> > > > project?
> > > > >> > >> I have a little more bandwidth to help with the code
> donation in
> > > > the
> > > > >> > >> next month or two.
> > > > >> > >>
> > > > >> > >> I know we voted on the Parquet mailing list about the
> donation
> > > > >> > >> already. One big question is whether you want to create an
> > > > >> > >> apache/parquet-rs repository or whether you want to
> co-develop
> > > > >> > >> parquet-rs together with Arrow in Rust, similar to what we
> are
> > > > doing
> > > > >> > >> with C++. It's possible you might run into the same kinds of
> > > issues
> > > > >> > >> that led us to consider the monorepo arrangement.
> > > > >> > >>
> > > > >> > >> Thanks
> > > > >> > >> Wes
> > > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > > > liurenjie2008@gmail.com>
> > > > >> > >> wrote:
> > > > >> > >> >
> > > > >> > >> > Hi, Chao:
> > > > >> > >> > I've opened an jira issue for that and planning to work on
> > > that.
> > > > >> > >> >
> > > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > > > liurenjie2008@gmail.com>
> > > > >> > >> wrote:
> > > > >> > >> >
> > > > >> > >> > > Yes, it's a mistake, sorry for that
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> > > sunchao@apache.org>
> > > > >> > wrote:
> > > > >> > >> > >
> > > > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > > > >> > >> > >>
> > > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next
> step
> > > in
> > > > >> > >> parquet-rs
> > > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had
> a
> > > > >> > discussion
> > > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140>
> about
> > > this
> > > > >> > >> recently.
> > > > >> > >> > >> Feel free to share your comments on the github.
> > > > >> > >> > >>
> > > > >> > >> > >> Best,
> > > > >> > >> > >> Chao
> > > > >> > >> > >>
> > > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > > > >> > liurenjie2008@gmail.com
> > > > >> > >> >
> > > > >> > >> > >> wrote:
> > > > >> > >> > >>
> > > > >> > >> > >>> cc:Sunchao and Any
> > > > >> > >> > >>>
> > > > >> > >> > >>>
> > > > >> > >> > >>> ---------- Forwarded message ---------
> > > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > > > >> > >> > >>> To: <de...@arrow.apache.org>
> > > > >> > >> > >>>
> > > > >> > >> > >>>
> > > > >> > >> > >>> Hello,
> > > > >> > >> > >>>
> > > > >> > >> > >>> you might also want to raise this with the
> > > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The
> overlap
> > > > >> > between
> > > > >> > >> the
> > > > >> > >> > >>> developers of this project and the Arrow Rust
> > > implementation
> > > > is
> > > > >> > >> quite large
> > > > >> > >> > >>> but still it may make sense to also start a discussion
> > > there.
> > > > >> > >> > >>>
> > > > >> > >> > >>> Uwe
> > > > >> > >> > >>>
> > > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > > > >> > >> > >>> > Hi, all:
> > > > >> > >> > >>> >
> > > > >> > >> > >>> > Now the rust component is approaching a stable state
> and
> > > > rust
> > > > >> > >> reader
> > > > >> > >> > >>> for
> > > > >> > >> > >>> > parquet is ready. I think it maybe a good time to
> start
> > > an
> > > > >> > >> adapter for
> > > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you
> guys
> > > > think
> > > > >> > >> about it?
> > > > >> > >> > >>> > --
> > > > >> > >> > >>> > Liu, Renjie
> > > > >> > >> > >>> > Software Engineer, MVAD
> > > > >> > >> > >>> --
> > > > >> > >> > >>> Liu, Renjie
> > > > >> > >> > >>> Software Engineer, MVAD
> > > > >> > >> > >>>
> > > > >> > >> > >>
> > > > >> > >> > >> --
> > > > >> > >> > > Liu, Renjie
> > > > >> > >> > > Software Engineer, MVAD
> > > > >> > >> > >
> > > > >> > >> > --
> > > > >> > >> > Liu, Renjie
> > > > >> > >> > Software Engineer, MVAD
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >> --
> > > > >> Renjie Liu
> > > > >> Software Engineer, MVAD
> > > >
> > >
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
I just opened a pull request <https://github.com/apache/arrow/pull/3050> for
the donation. Let me know if this is in the correct format and whether it's
OK to proceed to the next steps.
Thanks,
Chao
On Wed, Nov 21, 2018 at 9:18 PM Wes McKinney <we...@gmail.com> wrote:
> > Is the change to the nightly compiler the kind of thing that the PMC
> should vote on?
>
> A vote is not necessary if there is reasonable consensus.
> On Wed, Nov 21, 2018 at 9:11 PM paddy horan <pa...@hotmail.com>
> wrote:
> >
> > I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll
> take a closer look when I get a chance. I agree that lack of support for 32
> bit should not hold this up.
> >
> > Is the change to the nightly compiler the kind of thing that the PMC
> should vote on? This could be done in advance of the code donantion.
> Specialization, in particular, would be really useful within the existing
> code base.
> >
> > Paddy
> >
> > Get Outlook for iOS
> > ________________________________
> > From: Chao Sun <su...@apache.org>
> > Sent: Wednesday, November 21, 2018 2:42 PM
> > To: dev@arrow.apache.org
> > Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev
> > Subject: Re: [DISCUSS] Rust add adapter for parquet
> >
> > > A bigger issue is windows support for parquet-rs, Chao – I don’t
> believe
> > that parquet-rs supports windows, right? When I test it myself I get
> > errors regarding clang and libclang which parquet-rs must use.
> >
> > I think there are some issue regarding clang working with zstd on 32-bit
> > platform. However, it was able to compile with target:
> x86_64-pc-windows-msvc,
> > which seems to be the only one that arrow is using for windows CI. So I
> > think we can move forward and address the 32-bit platform issue later.
> >
> > On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com>
> wrote:
> >
> > > I think using nightly is fine for the reasons mentioned already. We
> > > should switch our CI to still run CI against stable (non fatal) so we
> know
> > > when we can make the move back to stable.
> > >
> > >
> > >
> > > A bigger issue is windows support for parquet-rs, Chao – I don’t
> believe
> > > that parquet-rs supports windows, right? When I test it myself I get
> > > errors regarding clang and libclang which parquet-rs must use.
> > >
> > >
> > >
> > > We have had windows support pretty much since the beginning for Rust.
> Is
> > > it possible to put parquet support behind a feature gate initially and
> only
> > > run CI for non-windows? I would be willing to help get windows support
> > > working after the fact, although I know very little about parquet
> right now.
> > >
> > >
> > >
> > > Are there other strategies for dealing with this?
> > >
> > >
> > >
> > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> > > Windows 10
> > >
> > >
> > >
> > > ________________________________
> > > From: Chao Sun <su...@apache.org>
> > > Sent: Wednesday, November 21, 2018 12:52:32 PM
> > > To: Wes McKinney
> > > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> > > Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >
> > > > Can you remind us all why nightly is required?
> > >
> > > Here's a tracking issue <
> https://github.com/sunchao/parquet-rs/issues/119>
> > > for all the unstable features parquet-rs uses. I'm personally inclined
> to
> > > use nightly since some new features such as specialization makes
> > > development much easier. Like Andy mentioned, as we are still
> developing
> > > arrow + parquet I think we can stay with nightly and transition back to
> > > stable once major features are implemented and more and more people
> start
> > > to use it (hopefully by that time the unstable features are
> stabilized).
> > > Moreover, I've seen quite a few popular projects rely on nightly such
> as
> > > rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> > >
> > > > The steps from here are for you all to get the codebase into a state
> > > that is ready for donation, including ASF license headers, etc. A pull
> > > request into apache/arrow would be the best thing
> > >
> > > Sure. I'll prepare a pull request in the next few days, and then we can
> > > proceed to the voting, ICLA, etc. Thanks.
> > >
> > > Chao
> > >
> > >
> > > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com>
> wrote:
> > >
> > > > The steps from here are for you all to get the codebase into a state
> > > > that is ready for donation, including ASF license headers, etc. A
> pull
> > > > request into apache/arrow would be the best thing
> > > >
> > > > Then we have to do the following
> > > >
> > > > * Vote on the Arrow mailing list
> > > > * Receive ICLAs from contributors
> > > > * Complete IP clearance
> > > > * Merge codebase
> > > >
> > > > Let me know when you are ready to move forward. From start to finish
> > > > that can get done in approximately 6 days if the code is ready
> > > >
> > > > Thanks
> > > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> > > wrote:
> > > > >
> > > > > Renjie,
> > > > >
> > > > > Can you remind us all why nightly is required?
> > > > >
> > > > > My personal feeling is that stable is a nice-to-have, but Rust is
> still
> > > > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > > > relying on nightly for now. Maybe we can have a plan to transition
> back
> > > to
> > > > stable for a future release if we go with nightly now.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Andy.
> > > > >
> > > > >
> > > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <
> liurenjie2008@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> That sounds great. But parquet-rs currently relies on nightly
> rust,
> > > that
> > > > >> would be the first problem to resolve.
> > > > >>
> > > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <andygrove73@gmail.com
> >
> > > > wrote:
> > > > >>
> > > > >> > This sounds like a great idea.
> > > > >> >
> > > > >> > With support for both CSV and Parquet in the Arrow crate, it
> would
> > > be
> > > > nice
> > > > >> > to design a standard interface for Arrow data sources. Maybe
> this is
> > > > as
> > > > >> > simple as implementing `Iterator<RecordBatch>`.
> > > > >> >
> > > > >> > Andy.
> > > > >> >
> > > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> > > wrote:
> > > > >> >
> > > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> > > this
> > > > into
> > > > >> > > Arrow because of the issues that you pointed out with parquet
> c++
> > > > merge,
> > > > >> > > and I do see a tight relationship between the two projects,
> and
> > > > potential
> > > > >> > > sharing of common libraries. @Ivan Sadikov <
> > > ivan.sadikov@gmail.com>
> > > > what
> > > > >> > > do you think?
> > > > >> > >
> > > > >> > > Chao
> > > > >> > >
> > > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> > > wesmckinn@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > >> hi folks,
> > > > >> > >>
> > > > >> > >> Would you all be interested in moving forward the parquet-rs
> > > > project?
> > > > >> > >> I have a little more bandwidth to help with the code
> donation in
> > > > the
> > > > >> > >> next month or two.
> > > > >> > >>
> > > > >> > >> I know we voted on the Parquet mailing list about the
> donation
> > > > >> > >> already. One big question is whether you want to create an
> > > > >> > >> apache/parquet-rs repository or whether you want to
> co-develop
> > > > >> > >> parquet-rs together with Arrow in Rust, similar to what we
> are
> > > > doing
> > > > >> > >> with C++. It's possible you might run into the same kinds of
> > > issues
> > > > >> > >> that led us to consider the monorepo arrangement.
> > > > >> > >>
> > > > >> > >> Thanks
> > > > >> > >> Wes
> > > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > > > liurenjie2008@gmail.com>
> > > > >> > >> wrote:
> > > > >> > >> >
> > > > >> > >> > Hi, Chao:
> > > > >> > >> > I've opened an jira issue for that and planning to work on
> > > that.
> > > > >> > >> >
> > > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > > > liurenjie2008@gmail.com>
> > > > >> > >> wrote:
> > > > >> > >> >
> > > > >> > >> > > Yes, it's a mistake, sorry for that
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> > > sunchao@apache.org>
> > > > >> > wrote:
> > > > >> > >> > >
> > > > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > > > >> > >> > >>
> > > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next
> step
> > > in
> > > > >> > >> parquet-rs
> > > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had
> a
> > > > >> > discussion
> > > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140>
> about
> > > this
> > > > >> > >> recently.
> > > > >> > >> > >> Feel free to share your comments on the github.
> > > > >> > >> > >>
> > > > >> > >> > >> Best,
> > > > >> > >> > >> Chao
> > > > >> > >> > >>
> > > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > > > >> > liurenjie2008@gmail.com
> > > > >> > >> >
> > > > >> > >> > >> wrote:
> > > > >> > >> > >>
> > > > >> > >> > >>> cc:Sunchao and Any
> > > > >> > >> > >>>
> > > > >> > >> > >>>
> > > > >> > >> > >>> ---------- Forwarded message ---------
> > > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > > > >> > >> > >>> To: <de...@arrow.apache.org>
> > > > >> > >> > >>>
> > > > >> > >> > >>>
> > > > >> > >> > >>> Hello,
> > > > >> > >> > >>>
> > > > >> > >> > >>> you might also want to raise this with the
> > > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The
> overlap
> > > > >> > between
> > > > >> > >> the
> > > > >> > >> > >>> developers of this project and the Arrow Rust
> > > implementation
> > > > is
> > > > >> > >> quite large
> > > > >> > >> > >>> but still it may make sense to also start a discussion
> > > there.
> > > > >> > >> > >>>
> > > > >> > >> > >>> Uwe
> > > > >> > >> > >>>
> > > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > > > >> > >> > >>> > Hi, all:
> > > > >> > >> > >>> >
> > > > >> > >> > >>> > Now the rust component is approaching a stable state
> and
> > > > rust
> > > > >> > >> reader
> > > > >> > >> > >>> for
> > > > >> > >> > >>> > parquet is ready. I think it maybe a good time to
> start
> > > an
> > > > >> > >> adapter for
> > > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you
> guys
> > > > think
> > > > >> > >> about it?
> > > > >> > >> > >>> > --
> > > > >> > >> > >>> > Liu, Renjie
> > > > >> > >> > >>> > Software Engineer, MVAD
> > > > >> > >> > >>> --
> > > > >> > >> > >>> Liu, Renjie
> > > > >> > >> > >>> Software Engineer, MVAD
> > > > >> > >> > >>>
> > > > >> > >> > >>
> > > > >> > >> > >> --
> > > > >> > >> > > Liu, Renjie
> > > > >> > >> > > Software Engineer, MVAD
> > > > >> > >> > >
> > > > >> > >> > --
> > > > >> > >> > Liu, Renjie
> > > > >> > >> > Software Engineer, MVAD
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >> --
> > > > >> Renjie Liu
> > > > >> Software Engineer, MVAD
> > > >
> > >
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Wes McKinney <we...@gmail.com>.
> Is the change to the nightly compiler the kind of thing that the PMC should vote on?
A vote is not necessary if there is reasonable consensus.
On Wed, Nov 21, 2018 at 9:11 PM paddy horan <pa...@hotmail.com> wrote:
>
> I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll take a closer look when I get a chance. I agree that lack of support for 32 bit should not hold this up.
>
> Is the change to the nightly compiler the kind of thing that the PMC should vote on? This could be done in advance of the code donantion. Specialization, in particular, would be really useful within the existing code base.
>
> Paddy
>
> Get Outlook for iOS
> ________________________________
> From: Chao Sun <su...@apache.org>
> Sent: Wednesday, November 21, 2018 2:42 PM
> To: dev@arrow.apache.org
> Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev
> Subject: Re: [DISCUSS] Rust add adapter for parquet
>
> > A bigger issue is windows support for parquet-rs, Chao – I don’t believe
> that parquet-rs supports windows, right? When I test it myself I get
> errors regarding clang and libclang which parquet-rs must use.
>
> I think there are some issue regarding clang working with zstd on 32-bit
> platform. However, it was able to compile with target: x86_64-pc-windows-msvc,
> which seems to be the only one that arrow is using for windows CI. So I
> think we can move forward and address the 32-bit platform issue later.
>
> On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com> wrote:
>
> > I think using nightly is fine for the reasons mentioned already. We
> > should switch our CI to still run CI against stable (non fatal) so we know
> > when we can make the move back to stable.
> >
> >
> >
> > A bigger issue is windows support for parquet-rs, Chao – I don’t believe
> > that parquet-rs supports windows, right? When I test it myself I get
> > errors regarding clang and libclang which parquet-rs must use.
> >
> >
> >
> > We have had windows support pretty much since the beginning for Rust. Is
> > it possible to put parquet support behind a feature gate initially and only
> > run CI for non-windows? I would be willing to help get windows support
> > working after the fact, although I know very little about parquet right now.
> >
> >
> >
> > Are there other strategies for dealing with this?
> >
> >
> >
> > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> > Windows 10
> >
> >
> >
> > ________________________________
> > From: Chao Sun <su...@apache.org>
> > Sent: Wednesday, November 21, 2018 12:52:32 PM
> > To: Wes McKinney
> > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> > Subject: Re: [DISCUSS] Rust add adapter for parquet
> >
> > > Can you remind us all why nightly is required?
> >
> > Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119>
> > for all the unstable features parquet-rs uses. I'm personally inclined to
> > use nightly since some new features such as specialization makes
> > development much easier. Like Andy mentioned, as we are still developing
> > arrow + parquet I think we can stay with nightly and transition back to
> > stable once major features are implemented and more and more people start
> > to use it (hopefully by that time the unstable features are stabilized).
> > Moreover, I've seen quite a few popular projects rely on nightly such as
> > rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> >
> > > The steps from here are for you all to get the codebase into a state
> > that is ready for donation, including ASF license headers, etc. A pull
> > request into apache/arrow would be the best thing
> >
> > Sure. I'll prepare a pull request in the next few days, and then we can
> > proceed to the voting, ICLA, etc. Thanks.
> >
> > Chao
> >
> >
> > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com> wrote:
> >
> > > The steps from here are for you all to get the codebase into a state
> > > that is ready for donation, including ASF license headers, etc. A pull
> > > request into apache/arrow would be the best thing
> > >
> > > Then we have to do the following
> > >
> > > * Vote on the Arrow mailing list
> > > * Receive ICLAs from contributors
> > > * Complete IP clearance
> > > * Merge codebase
> > >
> > > Let me know when you are ready to move forward. From start to finish
> > > that can get done in approximately 6 days if the code is ready
> > >
> > > Thanks
> > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> > wrote:
> > > >
> > > > Renjie,
> > > >
> > > > Can you remind us all why nightly is required?
> > > >
> > > > My personal feeling is that stable is a nice-to-have, but Rust is still
> > > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > > relying on nightly for now. Maybe we can have a plan to transition back
> > to
> > > stable for a future release if we go with nightly now.
> > > >
> > > > Thanks,
> > > >
> > > > Andy.
> > > >
> > > >
> > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com>
> > > wrote:
> > > >>
> > > >> That sounds great. But parquet-rs currently relies on nightly rust,
> > that
> > > >> would be the first problem to resolve.
> > > >>
> > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com>
> > > wrote:
> > > >>
> > > >> > This sounds like a great idea.
> > > >> >
> > > >> > With support for both CSV and Parquet in the Arrow crate, it would
> > be
> > > nice
> > > >> > to design a standard interface for Arrow data sources. Maybe this is
> > > as
> > > >> > simple as implementing `Iterator<RecordBatch>`.
> > > >> >
> > > >> > Andy.
> > > >> >
> > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> > wrote:
> > > >> >
> > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> > this
> > > into
> > > >> > > Arrow because of the issues that you pointed out with parquet c++
> > > merge,
> > > >> > > and I do see a tight relationship between the two projects, and
> > > potential
> > > >> > > sharing of common libraries. @Ivan Sadikov <
> > ivan.sadikov@gmail.com>
> > > what
> > > >> > > do you think?
> > > >> > >
> > > >> > > Chao
> > > >> > >
> > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> > wesmckinn@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > >> hi folks,
> > > >> > >>
> > > >> > >> Would you all be interested in moving forward the parquet-rs
> > > project?
> > > >> > >> I have a little more bandwidth to help with the code donation in
> > > the
> > > >> > >> next month or two.
> > > >> > >>
> > > >> > >> I know we voted on the Parquet mailing list about the donation
> > > >> > >> already. One big question is whether you want to create an
> > > >> > >> apache/parquet-rs repository or whether you want to co-develop
> > > >> > >> parquet-rs together with Arrow in Rust, similar to what we are
> > > doing
> > > >> > >> with C++. It's possible you might run into the same kinds of
> > issues
> > > >> > >> that led us to consider the monorepo arrangement.
> > > >> > >>
> > > >> > >> Thanks
> > > >> > >> Wes
> > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > > liurenjie2008@gmail.com>
> > > >> > >> wrote:
> > > >> > >> >
> > > >> > >> > Hi, Chao:
> > > >> > >> > I've opened an jira issue for that and planning to work on
> > that.
> > > >> > >> >
> > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > > liurenjie2008@gmail.com>
> > > >> > >> wrote:
> > > >> > >> >
> > > >> > >> > > Yes, it's a mistake, sorry for that
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> > sunchao@apache.org>
> > > >> > wrote:
> > > >> > >> > >
> > > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > > >> > >> > >>
> > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step
> > in
> > > >> > >> parquet-rs
> > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> > > >> > discussion
> > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about
> > this
> > > >> > >> recently.
> > > >> > >> > >> Feel free to share your comments on the github.
> > > >> > >> > >>
> > > >> > >> > >> Best,
> > > >> > >> > >> Chao
> > > >> > >> > >>
> > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > > >> > liurenjie2008@gmail.com
> > > >> > >> >
> > > >> > >> > >> wrote:
> > > >> > >> > >>
> > > >> > >> > >>> cc:Sunchao and Any
> > > >> > >> > >>>
> > > >> > >> > >>>
> > > >> > >> > >>> ---------- Forwarded message ---------
> > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > > >> > >> > >>> To: <de...@arrow.apache.org>
> > > >> > >> > >>>
> > > >> > >> > >>>
> > > >> > >> > >>> Hello,
> > > >> > >> > >>>
> > > >> > >> > >>> you might also want to raise this with the
> > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> > > >> > between
> > > >> > >> the
> > > >> > >> > >>> developers of this project and the Arrow Rust
> > implementation
> > > is
> > > >> > >> quite large
> > > >> > >> > >>> but still it may make sense to also start a discussion
> > there.
> > > >> > >> > >>>
> > > >> > >> > >>> Uwe
> > > >> > >> > >>>
> > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > > >> > >> > >>> > Hi, all:
> > > >> > >> > >>> >
> > > >> > >> > >>> > Now the rust component is approaching a stable state and
> > > rust
> > > >> > >> reader
> > > >> > >> > >>> for
> > > >> > >> > >>> > parquet is ready. I think it maybe a good time to start
> > an
> > > >> > >> adapter for
> > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys
> > > think
> > > >> > >> about it?
> > > >> > >> > >>> > --
> > > >> > >> > >>> > Liu, Renjie
> > > >> > >> > >>> > Software Engineer, MVAD
> > > >> > >> > >>> --
> > > >> > >> > >>> Liu, Renjie
> > > >> > >> > >>> Software Engineer, MVAD
> > > >> > >> > >>>
> > > >> > >> > >>
> > > >> > >> > >> --
> > > >> > >> > > Liu, Renjie
> > > >> > >> > > Software Engineer, MVAD
> > > >> > >> > >
> > > >> > >> > --
> > > >> > >> > Liu, Renjie
> > > >> > >> > Software Engineer, MVAD
> > > >> > >>
> > > >> > >
> > > >> >
> > > >> --
> > > >> Renjie Liu
> > > >> Software Engineer, MVAD
> > >
> >
Re: [DISCUSS] Rust add adapter for parquet
Posted by Wes McKinney <we...@gmail.com>.
> Is the change to the nightly compiler the kind of thing that the PMC should vote on?
A vote is not necessary if there is reasonable consensus.
On Wed, Nov 21, 2018 at 9:11 PM paddy horan <pa...@hotmail.com> wrote:
>
> I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll take a closer look when I get a chance. I agree that lack of support for 32 bit should not hold this up.
>
> Is the change to the nightly compiler the kind of thing that the PMC should vote on? This could be done in advance of the code donantion. Specialization, in particular, would be really useful within the existing code base.
>
> Paddy
>
> Get Outlook for iOS
> ________________________________
> From: Chao Sun <su...@apache.org>
> Sent: Wednesday, November 21, 2018 2:42 PM
> To: dev@arrow.apache.org
> Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev
> Subject: Re: [DISCUSS] Rust add adapter for parquet
>
> > A bigger issue is windows support for parquet-rs, Chao – I don’t believe
> that parquet-rs supports windows, right? When I test it myself I get
> errors regarding clang and libclang which parquet-rs must use.
>
> I think there are some issue regarding clang working with zstd on 32-bit
> platform. However, it was able to compile with target: x86_64-pc-windows-msvc,
> which seems to be the only one that arrow is using for windows CI. So I
> think we can move forward and address the 32-bit platform issue later.
>
> On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com> wrote:
>
> > I think using nightly is fine for the reasons mentioned already. We
> > should switch our CI to still run CI against stable (non fatal) so we know
> > when we can make the move back to stable.
> >
> >
> >
> > A bigger issue is windows support for parquet-rs, Chao – I don’t believe
> > that parquet-rs supports windows, right? When I test it myself I get
> > errors regarding clang and libclang which parquet-rs must use.
> >
> >
> >
> > We have had windows support pretty much since the beginning for Rust. Is
> > it possible to put parquet support behind a feature gate initially and only
> > run CI for non-windows? I would be willing to help get windows support
> > working after the fact, although I know very little about parquet right now.
> >
> >
> >
> > Are there other strategies for dealing with this?
> >
> >
> >
> > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> > Windows 10
> >
> >
> >
> > ________________________________
> > From: Chao Sun <su...@apache.org>
> > Sent: Wednesday, November 21, 2018 12:52:32 PM
> > To: Wes McKinney
> > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> > Subject: Re: [DISCUSS] Rust add adapter for parquet
> >
> > > Can you remind us all why nightly is required?
> >
> > Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119>
> > for all the unstable features parquet-rs uses. I'm personally inclined to
> > use nightly since some new features such as specialization makes
> > development much easier. Like Andy mentioned, as we are still developing
> > arrow + parquet I think we can stay with nightly and transition back to
> > stable once major features are implemented and more and more people start
> > to use it (hopefully by that time the unstable features are stabilized).
> > Moreover, I've seen quite a few popular projects rely on nightly such as
> > rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> >
> > > The steps from here are for you all to get the codebase into a state
> > that is ready for donation, including ASF license headers, etc. A pull
> > request into apache/arrow would be the best thing
> >
> > Sure. I'll prepare a pull request in the next few days, and then we can
> > proceed to the voting, ICLA, etc. Thanks.
> >
> > Chao
> >
> >
> > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com> wrote:
> >
> > > The steps from here are for you all to get the codebase into a state
> > > that is ready for donation, including ASF license headers, etc. A pull
> > > request into apache/arrow would be the best thing
> > >
> > > Then we have to do the following
> > >
> > > * Vote on the Arrow mailing list
> > > * Receive ICLAs from contributors
> > > * Complete IP clearance
> > > * Merge codebase
> > >
> > > Let me know when you are ready to move forward. From start to finish
> > > that can get done in approximately 6 days if the code is ready
> > >
> > > Thanks
> > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> > wrote:
> > > >
> > > > Renjie,
> > > >
> > > > Can you remind us all why nightly is required?
> > > >
> > > > My personal feeling is that stable is a nice-to-have, but Rust is still
> > > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > > relying on nightly for now. Maybe we can have a plan to transition back
> > to
> > > stable for a future release if we go with nightly now.
> > > >
> > > > Thanks,
> > > >
> > > > Andy.
> > > >
> > > >
> > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com>
> > > wrote:
> > > >>
> > > >> That sounds great. But parquet-rs currently relies on nightly rust,
> > that
> > > >> would be the first problem to resolve.
> > > >>
> > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com>
> > > wrote:
> > > >>
> > > >> > This sounds like a great idea.
> > > >> >
> > > >> > With support for both CSV and Parquet in the Arrow crate, it would
> > be
> > > nice
> > > >> > to design a standard interface for Arrow data sources. Maybe this is
> > > as
> > > >> > simple as implementing `Iterator<RecordBatch>`.
> > > >> >
> > > >> > Andy.
> > > >> >
> > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> > wrote:
> > > >> >
> > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> > this
> > > into
> > > >> > > Arrow because of the issues that you pointed out with parquet c++
> > > merge,
> > > >> > > and I do see a tight relationship between the two projects, and
> > > potential
> > > >> > > sharing of common libraries. @Ivan Sadikov <
> > ivan.sadikov@gmail.com>
> > > what
> > > >> > > do you think?
> > > >> > >
> > > >> > > Chao
> > > >> > >
> > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> > wesmckinn@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > >> hi folks,
> > > >> > >>
> > > >> > >> Would you all be interested in moving forward the parquet-rs
> > > project?
> > > >> > >> I have a little more bandwidth to help with the code donation in
> > > the
> > > >> > >> next month or two.
> > > >> > >>
> > > >> > >> I know we voted on the Parquet mailing list about the donation
> > > >> > >> already. One big question is whether you want to create an
> > > >> > >> apache/parquet-rs repository or whether you want to co-develop
> > > >> > >> parquet-rs together with Arrow in Rust, similar to what we are
> > > doing
> > > >> > >> with C++. It's possible you might run into the same kinds of
> > issues
> > > >> > >> that led us to consider the monorepo arrangement.
> > > >> > >>
> > > >> > >> Thanks
> > > >> > >> Wes
> > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > > liurenjie2008@gmail.com>
> > > >> > >> wrote:
> > > >> > >> >
> > > >> > >> > Hi, Chao:
> > > >> > >> > I've opened an jira issue for that and planning to work on
> > that.
> > > >> > >> >
> > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > > liurenjie2008@gmail.com>
> > > >> > >> wrote:
> > > >> > >> >
> > > >> > >> > > Yes, it's a mistake, sorry for that
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> > sunchao@apache.org>
> > > >> > wrote:
> > > >> > >> > >
> > > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > > >> > >> > >>
> > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step
> > in
> > > >> > >> parquet-rs
> > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> > > >> > discussion
> > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about
> > this
> > > >> > >> recently.
> > > >> > >> > >> Feel free to share your comments on the github.
> > > >> > >> > >>
> > > >> > >> > >> Best,
> > > >> > >> > >> Chao
> > > >> > >> > >>
> > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > > >> > liurenjie2008@gmail.com
> > > >> > >> >
> > > >> > >> > >> wrote:
> > > >> > >> > >>
> > > >> > >> > >>> cc:Sunchao and Any
> > > >> > >> > >>>
> > > >> > >> > >>>
> > > >> > >> > >>> ---------- Forwarded message ---------
> > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > > >> > >> > >>> To: <de...@arrow.apache.org>
> > > >> > >> > >>>
> > > >> > >> > >>>
> > > >> > >> > >>> Hello,
> > > >> > >> > >>>
> > > >> > >> > >>> you might also want to raise this with the
> > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> > > >> > between
> > > >> > >> the
> > > >> > >> > >>> developers of this project and the Arrow Rust
> > implementation
> > > is
> > > >> > >> quite large
> > > >> > >> > >>> but still it may make sense to also start a discussion
> > there.
> > > >> > >> > >>>
> > > >> > >> > >>> Uwe
> > > >> > >> > >>>
> > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > > >> > >> > >>> > Hi, all:
> > > >> > >> > >>> >
> > > >> > >> > >>> > Now the rust component is approaching a stable state and
> > > rust
> > > >> > >> reader
> > > >> > >> > >>> for
> > > >> > >> > >>> > parquet is ready. I think it maybe a good time to start
> > an
> > > >> > >> adapter for
> > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys
> > > think
> > > >> > >> about it?
> > > >> > >> > >>> > --
> > > >> > >> > >>> > Liu, Renjie
> > > >> > >> > >>> > Software Engineer, MVAD
> > > >> > >> > >>> --
> > > >> > >> > >>> Liu, Renjie
> > > >> > >> > >>> Software Engineer, MVAD
> > > >> > >> > >>>
> > > >> > >> > >>
> > > >> > >> > >> --
> > > >> > >> > > Liu, Renjie
> > > >> > >> > > Software Engineer, MVAD
> > > >> > >> > >
> > > >> > >> > --
> > > >> > >> > Liu, Renjie
> > > >> > >> > Software Engineer, MVAD
> > > >> > >>
> > > >> > >
> > > >> >
> > > >> --
> > > >> Renjie Liu
> > > >> Software Engineer, MVAD
> > >
> >
Re: [DISCUSS] Rust add adapter for parquet
Posted by paddy horan <pa...@hotmail.com>.
I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll take a closer look when I get a chance. I agree that lack of support for 32 bit should not hold this up.
Is the change to the nightly compiler the kind of thing that the PMC should vote on? This could be done in advance of the code donantion. Specialization, in particular, would be really useful within the existing code base.
Paddy
Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Chao Sun <su...@apache.org>
Sent: Wednesday, November 21, 2018 2:42 PM
To: dev@arrow.apache.org
Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev
Subject: Re: [DISCUSS] Rust add adapter for parquet
> A bigger issue is windows support for parquet-rs, Chao – I don’t believe
that parquet-rs supports windows, right? When I test it myself I get
errors regarding clang and libclang which parquet-rs must use.
I think there are some issue regarding clang working with zstd on 32-bit
platform. However, it was able to compile with target: x86_64-pc-windows-msvc,
which seems to be the only one that arrow is using for windows CI. So I
think we can move forward and address the 32-bit platform issue later.
On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com> wrote:
> I think using nightly is fine for the reasons mentioned already. We
> should switch our CI to still run CI against stable (non fatal) so we know
> when we can make the move back to stable.
>
>
>
> A bigger issue is windows support for parquet-rs, Chao – I don’t believe
> that parquet-rs supports windows, right? When I test it myself I get
> errors regarding clang and libclang which parquet-rs must use.
>
>
>
> We have had windows support pretty much since the beginning for Rust. Is
> it possible to put parquet support behind a feature gate initially and only
> run CI for non-windows? I would be willing to help get windows support
> working after the fact, although I know very little about parquet right now.
>
>
>
> Are there other strategies for dealing with this?
>
>
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> ________________________________
> From: Chao Sun <su...@apache.org>
> Sent: Wednesday, November 21, 2018 12:52:32 PM
> To: Wes McKinney
> Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> Subject: Re: [DISCUSS] Rust add adapter for parquet
>
> > Can you remind us all why nightly is required?
>
> Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119>
> for all the unstable features parquet-rs uses. I'm personally inclined to
> use nightly since some new features such as specialization makes
> development much easier. Like Andy mentioned, as we are still developing
> arrow + parquet I think we can stay with nightly and transition back to
> stable once major features are implemented and more and more people start
> to use it (hopefully by that time the unstable features are stabilized).
> Moreover, I've seen quite a few popular projects rely on nightly such as
> rocket, tikv, etc., so seems it is not uncommon in the Rust world.
>
> > The steps from here are for you all to get the codebase into a state
> that is ready for donation, including ASF license headers, etc. A pull
> request into apache/arrow would be the best thing
>
> Sure. I'll prepare a pull request in the next few days, and then we can
> proceed to the voting, ICLA, etc. Thanks.
>
> Chao
>
>
> On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com> wrote:
>
> > The steps from here are for you all to get the codebase into a state
> > that is ready for donation, including ASF license headers, etc. A pull
> > request into apache/arrow would be the best thing
> >
> > Then we have to do the following
> >
> > * Vote on the Arrow mailing list
> > * Receive ICLAs from contributors
> > * Complete IP clearance
> > * Merge codebase
> >
> > Let me know when you are ready to move forward. From start to finish
> > that can get done in approximately 6 days if the code is ready
> >
> > Thanks
> > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> wrote:
> > >
> > > Renjie,
> > >
> > > Can you remind us all why nightly is required?
> > >
> > > My personal feeling is that stable is a nice-to-have, but Rust is still
> > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > relying on nightly for now. Maybe we can have a plan to transition back
> to
> > stable for a future release if we go with nightly now.
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > >
> > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com>
> > wrote:
> > >>
> > >> That sounds great. But parquet-rs currently relies on nightly rust,
> that
> > >> would be the first problem to resolve.
> > >>
> > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com>
> > wrote:
> > >>
> > >> > This sounds like a great idea.
> > >> >
> > >> > With support for both CSV and Parquet in the Arrow crate, it would
> be
> > nice
> > >> > to design a standard interface for Arrow data sources. Maybe this is
> > as
> > >> > simple as implementing `Iterator<RecordBatch>`.
> > >> >
> > >> > Andy.
> > >> >
> > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> wrote:
> > >> >
> > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> this
> > into
> > >> > > Arrow because of the issues that you pointed out with parquet c++
> > merge,
> > >> > > and I do see a tight relationship between the two projects, and
> > potential
> > >> > > sharing of common libraries. @Ivan Sadikov <
> ivan.sadikov@gmail.com>
> > what
> > >> > > do you think?
> > >> > >
> > >> > > Chao
> > >> > >
> > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> wesmckinn@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > >> hi folks,
> > >> > >>
> > >> > >> Would you all be interested in moving forward the parquet-rs
> > project?
> > >> > >> I have a little more bandwidth to help with the code donation in
> > the
> > >> > >> next month or two.
> > >> > >>
> > >> > >> I know we voted on the Parquet mailing list about the donation
> > >> > >> already. One big question is whether you want to create an
> > >> > >> apache/parquet-rs repository or whether you want to co-develop
> > >> > >> parquet-rs together with Arrow in Rust, similar to what we are
> > doing
> > >> > >> with C++. It's possible you might run into the same kinds of
> issues
> > >> > >> that led us to consider the monorepo arrangement.
> > >> > >>
> > >> > >> Thanks
> > >> > >> Wes
> > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > liurenjie2008@gmail.com>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > Hi, Chao:
> > >> > >> > I've opened an jira issue for that and planning to work on
> that.
> > >> > >> >
> > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > liurenjie2008@gmail.com>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > > Yes, it's a mistake, sorry for that
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> sunchao@apache.org>
> > >> > wrote:
> > >> > >> > >
> > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > >> > >> > >>
> > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step
> in
> > >> > >> parquet-rs
> > >> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> > >> > discussion
> > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about
> this
> > >> > >> recently.
> > >> > >> > >> Feel free to share your comments on the github.
> > >> > >> > >>
> > >> > >> > >> Best,
> > >> > >> > >> Chao
> > >> > >> > >>
> > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > >> > liurenjie2008@gmail.com
> > >> > >> >
> > >> > >> > >> wrote:
> > >> > >> > >>
> > >> > >> > >>> cc:Sunchao and Any
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>> ---------- Forwarded message ---------
> > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >> > >> > >>> To: <de...@arrow.apache.org>
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>> Hello,
> > >> > >> > >>>
> > >> > >> > >>> you might also want to raise this with the
> > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> > >> > between
> > >> > >> the
> > >> > >> > >>> developers of this project and the Arrow Rust
> implementation
> > is
> > >> > >> quite large
> > >> > >> > >>> but still it may make sense to also start a discussion
> there.
> > >> > >> > >>>
> > >> > >> > >>> Uwe
> > >> > >> > >>>
> > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > >> > >> > >>> > Hi, all:
> > >> > >> > >>> >
> > >> > >> > >>> > Now the rust component is approaching a stable state and
> > rust
> > >> > >> reader
> > >> > >> > >>> for
> > >> > >> > >>> > parquet is ready. I think it maybe a good time to start
> an
> > >> > >> adapter for
> > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys
> > think
> > >> > >> about it?
> > >> > >> > >>> > --
> > >> > >> > >>> > Liu, Renjie
> > >> > >> > >>> > Software Engineer, MVAD
> > >> > >> > >>> --
> > >> > >> > >>> Liu, Renjie
> > >> > >> > >>> Software Engineer, MVAD
> > >> > >> > >>>
> > >> > >> > >>
> > >> > >> > >> --
> > >> > >> > > Liu, Renjie
> > >> > >> > > Software Engineer, MVAD
> > >> > >> > >
> > >> > >> > --
> > >> > >> > Liu, Renjie
> > >> > >> > Software Engineer, MVAD
> > >> > >>
> > >> > >
> > >> >
> > >> --
> > >> Renjie Liu
> > >> Software Engineer, MVAD
> >
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
> A bigger issue is windows support for parquet-rs, Chao – I don’t believe
that parquet-rs supports windows, right? When I test it myself I get
errors regarding clang and libclang which parquet-rs must use.
I think there are some issue regarding clang working with zstd on 32-bit
platform. However, it was able to compile with target: x86_64-pc-windows-msvc,
which seems to be the only one that arrow is using for windows CI. So I
think we can move forward and address the 32-bit platform issue later.
On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com> wrote:
> I think using nightly is fine for the reasons mentioned already. We
> should switch our CI to still run CI against stable (non fatal) so we know
> when we can make the move back to stable.
>
>
>
> A bigger issue is windows support for parquet-rs, Chao – I don’t believe
> that parquet-rs supports windows, right? When I test it myself I get
> errors regarding clang and libclang which parquet-rs must use.
>
>
>
> We have had windows support pretty much since the beginning for Rust. Is
> it possible to put parquet support behind a feature gate initially and only
> run CI for non-windows? I would be willing to help get windows support
> working after the fact, although I know very little about parquet right now.
>
>
>
> Are there other strategies for dealing with this?
>
>
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> ________________________________
> From: Chao Sun <su...@apache.org>
> Sent: Wednesday, November 21, 2018 12:52:32 PM
> To: Wes McKinney
> Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> Subject: Re: [DISCUSS] Rust add adapter for parquet
>
> > Can you remind us all why nightly is required?
>
> Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119>
> for all the unstable features parquet-rs uses. I'm personally inclined to
> use nightly since some new features such as specialization makes
> development much easier. Like Andy mentioned, as we are still developing
> arrow + parquet I think we can stay with nightly and transition back to
> stable once major features are implemented and more and more people start
> to use it (hopefully by that time the unstable features are stabilized).
> Moreover, I've seen quite a few popular projects rely on nightly such as
> rocket, tikv, etc., so seems it is not uncommon in the Rust world.
>
> > The steps from here are for you all to get the codebase into a state
> that is ready for donation, including ASF license headers, etc. A pull
> request into apache/arrow would be the best thing
>
> Sure. I'll prepare a pull request in the next few days, and then we can
> proceed to the voting, ICLA, etc. Thanks.
>
> Chao
>
>
> On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com> wrote:
>
> > The steps from here are for you all to get the codebase into a state
> > that is ready for donation, including ASF license headers, etc. A pull
> > request into apache/arrow would be the best thing
> >
> > Then we have to do the following
> >
> > * Vote on the Arrow mailing list
> > * Receive ICLAs from contributors
> > * Complete IP clearance
> > * Merge codebase
> >
> > Let me know when you are ready to move forward. From start to finish
> > that can get done in approximately 6 days if the code is ready
> >
> > Thanks
> > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> wrote:
> > >
> > > Renjie,
> > >
> > > Can you remind us all why nightly is required?
> > >
> > > My personal feeling is that stable is a nice-to-have, but Rust is still
> > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > relying on nightly for now. Maybe we can have a plan to transition back
> to
> > stable for a future release if we go with nightly now.
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > >
> > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com>
> > wrote:
> > >>
> > >> That sounds great. But parquet-rs currently relies on nightly rust,
> that
> > >> would be the first problem to resolve.
> > >>
> > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com>
> > wrote:
> > >>
> > >> > This sounds like a great idea.
> > >> >
> > >> > With support for both CSV and Parquet in the Arrow crate, it would
> be
> > nice
> > >> > to design a standard interface for Arrow data sources. Maybe this is
> > as
> > >> > simple as implementing `Iterator<RecordBatch>`.
> > >> >
> > >> > Andy.
> > >> >
> > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> wrote:
> > >> >
> > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> this
> > into
> > >> > > Arrow because of the issues that you pointed out with parquet c++
> > merge,
> > >> > > and I do see a tight relationship between the two projects, and
> > potential
> > >> > > sharing of common libraries. @Ivan Sadikov <
> ivan.sadikov@gmail.com>
> > what
> > >> > > do you think?
> > >> > >
> > >> > > Chao
> > >> > >
> > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> wesmckinn@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > >> hi folks,
> > >> > >>
> > >> > >> Would you all be interested in moving forward the parquet-rs
> > project?
> > >> > >> I have a little more bandwidth to help with the code donation in
> > the
> > >> > >> next month or two.
> > >> > >>
> > >> > >> I know we voted on the Parquet mailing list about the donation
> > >> > >> already. One big question is whether you want to create an
> > >> > >> apache/parquet-rs repository or whether you want to co-develop
> > >> > >> parquet-rs together with Arrow in Rust, similar to what we are
> > doing
> > >> > >> with C++. It's possible you might run into the same kinds of
> issues
> > >> > >> that led us to consider the monorepo arrangement.
> > >> > >>
> > >> > >> Thanks
> > >> > >> Wes
> > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > liurenjie2008@gmail.com>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > Hi, Chao:
> > >> > >> > I've opened an jira issue for that and planning to work on
> that.
> > >> > >> >
> > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > liurenjie2008@gmail.com>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > > Yes, it's a mistake, sorry for that
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> sunchao@apache.org>
> > >> > wrote:
> > >> > >> > >
> > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > >> > >> > >>
> > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step
> in
> > >> > >> parquet-rs
> > >> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> > >> > discussion
> > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about
> this
> > >> > >> recently.
> > >> > >> > >> Feel free to share your comments on the github.
> > >> > >> > >>
> > >> > >> > >> Best,
> > >> > >> > >> Chao
> > >> > >> > >>
> > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > >> > liurenjie2008@gmail.com
> > >> > >> >
> > >> > >> > >> wrote:
> > >> > >> > >>
> > >> > >> > >>> cc:Sunchao and Any
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>> ---------- Forwarded message ---------
> > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >> > >> > >>> To: <de...@arrow.apache.org>
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>> Hello,
> > >> > >> > >>>
> > >> > >> > >>> you might also want to raise this with the
> > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> > >> > between
> > >> > >> the
> > >> > >> > >>> developers of this project and the Arrow Rust
> implementation
> > is
> > >> > >> quite large
> > >> > >> > >>> but still it may make sense to also start a discussion
> there.
> > >> > >> > >>>
> > >> > >> > >>> Uwe
> > >> > >> > >>>
> > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > >> > >> > >>> > Hi, all:
> > >> > >> > >>> >
> > >> > >> > >>> > Now the rust component is approaching a stable state and
> > rust
> > >> > >> reader
> > >> > >> > >>> for
> > >> > >> > >>> > parquet is ready. I think it maybe a good time to start
> an
> > >> > >> adapter for
> > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys
> > think
> > >> > >> about it?
> > >> > >> > >>> > --
> > >> > >> > >>> > Liu, Renjie
> > >> > >> > >>> > Software Engineer, MVAD
> > >> > >> > >>> --
> > >> > >> > >>> Liu, Renjie
> > >> > >> > >>> Software Engineer, MVAD
> > >> > >> > >>>
> > >> > >> > >>
> > >> > >> > >> --
> > >> > >> > > Liu, Renjie
> > >> > >> > > Software Engineer, MVAD
> > >> > >> > >
> > >> > >> > --
> > >> > >> > Liu, Renjie
> > >> > >> > Software Engineer, MVAD
> > >> > >>
> > >> > >
> > >> >
> > >> --
> > >> Renjie Liu
> > >> Software Engineer, MVAD
> >
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
> A bigger issue is windows support for parquet-rs, Chao – I don’t believe
that parquet-rs supports windows, right? When I test it myself I get
errors regarding clang and libclang which parquet-rs must use.
I think there are some issue regarding clang working with zstd on 32-bit
platform. However, it was able to compile with target: x86_64-pc-windows-msvc,
which seems to be the only one that arrow is using for windows CI. So I
think we can move forward and address the 32-bit platform issue later.
On Wed, Nov 21, 2018 at 10:18 AM paddy horan <pa...@hotmail.com> wrote:
> I think using nightly is fine for the reasons mentioned already. We
> should switch our CI to still run CI against stable (non fatal) so we know
> when we can make the move back to stable.
>
>
>
> A bigger issue is windows support for parquet-rs, Chao – I don’t believe
> that parquet-rs supports windows, right? When I test it myself I get
> errors regarding clang and libclang which parquet-rs must use.
>
>
>
> We have had windows support pretty much since the beginning for Rust. Is
> it possible to put parquet support behind a feature gate initially and only
> run CI for non-windows? I would be willing to help get windows support
> working after the fact, although I know very little about parquet right now.
>
>
>
> Are there other strategies for dealing with this?
>
>
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> ________________________________
> From: Chao Sun <su...@apache.org>
> Sent: Wednesday, November 21, 2018 12:52:32 PM
> To: Wes McKinney
> Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
> Subject: Re: [DISCUSS] Rust add adapter for parquet
>
> > Can you remind us all why nightly is required?
>
> Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119>
> for all the unstable features parquet-rs uses. I'm personally inclined to
> use nightly since some new features such as specialization makes
> development much easier. Like Andy mentioned, as we are still developing
> arrow + parquet I think we can stay with nightly and transition back to
> stable once major features are implemented and more and more people start
> to use it (hopefully by that time the unstable features are stabilized).
> Moreover, I've seen quite a few popular projects rely on nightly such as
> rocket, tikv, etc., so seems it is not uncommon in the Rust world.
>
> > The steps from here are for you all to get the codebase into a state
> that is ready for donation, including ASF license headers, etc. A pull
> request into apache/arrow would be the best thing
>
> Sure. I'll prepare a pull request in the next few days, and then we can
> proceed to the voting, ICLA, etc. Thanks.
>
> Chao
>
>
> On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com> wrote:
>
> > The steps from here are for you all to get the codebase into a state
> > that is ready for donation, including ASF license headers, etc. A pull
> > request into apache/arrow would be the best thing
> >
> > Then we have to do the following
> >
> > * Vote on the Arrow mailing list
> > * Receive ICLAs from contributors
> > * Complete IP clearance
> > * Merge codebase
> >
> > Let me know when you are ready to move forward. From start to finish
> > that can get done in approximately 6 days if the code is ready
> >
> > Thanks
> > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com>
> wrote:
> > >
> > > Renjie,
> > >
> > > Can you remind us all why nightly is required?
> > >
> > > My personal feeling is that stable is a nice-to-have, but Rust is still
> > moving fast and we are on the bleeding edge here so I'm OK with Arrow
> > relying on nightly for now. Maybe we can have a plan to transition back
> to
> > stable for a future release if we go with nightly now.
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > >
> > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com>
> > wrote:
> > >>
> > >> That sounds great. But parquet-rs currently relies on nightly rust,
> that
> > >> would be the first problem to resolve.
> > >>
> > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com>
> > wrote:
> > >>
> > >> > This sounds like a great idea.
> > >> >
> > >> > With support for both CSV and Parquet in the Arrow crate, it would
> be
> > nice
> > >> > to design a standard interface for Arrow data sources. Maybe this is
> > as
> > >> > simple as implementing `Iterator<RecordBatch>`.
> > >> >
> > >> > Andy.
> > >> >
> > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org>
> wrote:
> > >> >
> > >> > > Yes, we'd be interested to move forward. I'm inclined to merge
> this
> > into
> > >> > > Arrow because of the issues that you pointed out with parquet c++
> > merge,
> > >> > > and I do see a tight relationship between the two projects, and
> > potential
> > >> > > sharing of common libraries. @Ivan Sadikov <
> ivan.sadikov@gmail.com>
> > what
> > >> > > do you think?
> > >> > >
> > >> > > Chao
> > >> > >
> > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <
> wesmckinn@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > >> hi folks,
> > >> > >>
> > >> > >> Would you all be interested in moving forward the parquet-rs
> > project?
> > >> > >> I have a little more bandwidth to help with the code donation in
> > the
> > >> > >> next month or two.
> > >> > >>
> > >> > >> I know we voted on the Parquet mailing list about the donation
> > >> > >> already. One big question is whether you want to create an
> > >> > >> apache/parquet-rs repository or whether you want to co-develop
> > >> > >> parquet-rs together with Arrow in Rust, similar to what we are
> > doing
> > >> > >> with C++. It's possible you might run into the same kinds of
> issues
> > >> > >> that led us to consider the monorepo arrangement.
> > >> > >>
> > >> > >> Thanks
> > >> > >> Wes
> > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> > liurenjie2008@gmail.com>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > Hi, Chao:
> > >> > >> > I've opened an jira issue for that and planning to work on
> that.
> > >> > >> >
> > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> > liurenjie2008@gmail.com>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > > Yes, it's a mistake, sorry for that
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <
> sunchao@apache.org>
> > >> > wrote:
> > >> > >> > >
> > >> > >> > >> (s/flink/arrow - it is a mistake?)
> > >> > >> > >>
> > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step
> in
> > >> > >> parquet-rs
> > >> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> > >> > discussion
> > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about
> this
> > >> > >> recently.
> > >> > >> > >> Feel free to share your comments on the github.
> > >> > >> > >>
> > >> > >> > >> Best,
> > >> > >> > >> Chao
> > >> > >> > >>
> > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > >> > liurenjie2008@gmail.com
> > >> > >> >
> > >> > >> > >> wrote:
> > >> > >> > >>
> > >> > >> > >>> cc:Sunchao and Any
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>> ---------- Forwarded message ---------
> > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >> > >> > >>> To: <de...@arrow.apache.org>
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>> Hello,
> > >> > >> > >>>
> > >> > >> > >>> you might also want to raise this with the
> > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> > >> > between
> > >> > >> the
> > >> > >> > >>> developers of this project and the Arrow Rust
> implementation
> > is
> > >> > >> quite large
> > >> > >> > >>> but still it may make sense to also start a discussion
> there.
> > >> > >> > >>>
> > >> > >> > >>> Uwe
> > >> > >> > >>>
> > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > >> > >> > >>> > Hi, all:
> > >> > >> > >>> >
> > >> > >> > >>> > Now the rust component is approaching a stable state and
> > rust
> > >> > >> reader
> > >> > >> > >>> for
> > >> > >> > >>> > parquet is ready. I think it maybe a good time to start
> an
> > >> > >> adapter for
> > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys
> > think
> > >> > >> about it?
> > >> > >> > >>> > --
> > >> > >> > >>> > Liu, Renjie
> > >> > >> > >>> > Software Engineer, MVAD
> > >> > >> > >>> --
> > >> > >> > >>> Liu, Renjie
> > >> > >> > >>> Software Engineer, MVAD
> > >> > >> > >>>
> > >> > >> > >>
> > >> > >> > >> --
> > >> > >> > > Liu, Renjie
> > >> > >> > > Software Engineer, MVAD
> > >> > >> > >
> > >> > >> > --
> > >> > >> > Liu, Renjie
> > >> > >> > Software Engineer, MVAD
> > >> > >>
> > >> > >
> > >> >
> > >> --
> > >> Renjie Liu
> > >> Software Engineer, MVAD
> >
>
RE: [DISCUSS] Rust add adapter for parquet
Posted by paddy horan <pa...@hotmail.com>.
I think using nightly is fine for the reasons mentioned already. We should switch our CI to still run CI against stable (non fatal) so we know when we can make the move back to stable.
A bigger issue is windows support for parquet-rs, Chao – I don’t believe that parquet-rs supports windows, right? When I test it myself I get errors regarding clang and libclang which parquet-rs must use.
We have had windows support pretty much since the beginning for Rust. Is it possible to put parquet support behind a feature gate initially and only run CI for non-windows? I would be willing to help get windows support working after the fact, although I know very little about parquet right now.
Are there other strategies for dealing with this?
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
________________________________
From: Chao Sun <su...@apache.org>
Sent: Wednesday, November 21, 2018 12:52:32 PM
To: Wes McKinney
Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev
Subject: Re: [DISCUSS] Rust add adapter for parquet
> Can you remind us all why nightly is required?
Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119>
for all the unstable features parquet-rs uses. I'm personally inclined to
use nightly since some new features such as specialization makes
development much easier. Like Andy mentioned, as we are still developing
arrow + parquet I think we can stay with nightly and transition back to
stable once major features are implemented and more and more people start
to use it (hopefully by that time the unstable features are stabilized).
Moreover, I've seen quite a few popular projects rely on nightly such as
rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> The steps from here are for you all to get the codebase into a state
that is ready for donation, including ASF license headers, etc. A pull
request into apache/arrow would be the best thing
Sure. I'll prepare a pull request in the next few days, and then we can
proceed to the voting, ICLA, etc. Thanks.
Chao
On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com> wrote:
> The steps from here are for you all to get the codebase into a state
> that is ready for donation, including ASF license headers, etc. A pull
> request into apache/arrow would be the best thing
>
> Then we have to do the following
>
> * Vote on the Arrow mailing list
> * Receive ICLAs from contributors
> * Complete IP clearance
> * Merge codebase
>
> Let me know when you are ready to move forward. From start to finish
> that can get done in approximately 6 days if the code is ready
>
> Thanks
> On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com> wrote:
> >
> > Renjie,
> >
> > Can you remind us all why nightly is required?
> >
> > My personal feeling is that stable is a nice-to-have, but Rust is still
> moving fast and we are on the bleeding edge here so I'm OK with Arrow
> relying on nightly for now. Maybe we can have a plan to transition back to
> stable for a future release if we go with nightly now.
> >
> > Thanks,
> >
> > Andy.
> >
> >
> > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com>
> wrote:
> >>
> >> That sounds great. But parquet-rs currently relies on nightly rust, that
> >> would be the first problem to resolve.
> >>
> >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com>
> wrote:
> >>
> >> > This sounds like a great idea.
> >> >
> >> > With support for both CSV and Parquet in the Arrow crate, it would be
> nice
> >> > to design a standard interface for Arrow data sources. Maybe this is
> as
> >> > simple as implementing `Iterator<RecordBatch>`.
> >> >
> >> > Andy.
> >> >
> >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
> >> >
> >> > > Yes, we'd be interested to move forward. I'm inclined to merge this
> into
> >> > > Arrow because of the issues that you pointed out with parquet c++
> merge,
> >> > > and I do see a tight relationship between the two projects, and
> potential
> >> > > sharing of common libraries. @Ivan Sadikov <iv...@gmail.com>
> what
> >> > > do you think?
> >> > >
> >> > > Chao
> >> > >
> >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> hi folks,
> >> > >>
> >> > >> Would you all be interested in moving forward the parquet-rs
> project?
> >> > >> I have a little more bandwidth to help with the code donation in
> the
> >> > >> next month or two.
> >> > >>
> >> > >> I know we voted on the Parquet mailing list about the donation
> >> > >> already. One big question is whether you want to create an
> >> > >> apache/parquet-rs repository or whether you want to co-develop
> >> > >> parquet-rs together with Arrow in Rust, similar to what we are
> doing
> >> > >> with C++. It's possible you might run into the same kinds of issues
> >> > >> that led us to consider the monorepo arrangement.
> >> > >>
> >> > >> Thanks
> >> > >> Wes
> >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> liurenjie2008@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > Hi, Chao:
> >> > >> > I've opened an jira issue for that and planning to work on that.
> >> > >> >
> >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> liurenjie2008@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Yes, it's a mistake, sorry for that
> >> > >> > >
> >> > >> > >
> >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
> >> > wrote:
> >> > >> > >
> >> > >> > >> (s/flink/arrow - it is a mistake?)
> >> > >> > >>
> >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step in
> >> > >> parquet-rs
> >> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> >> > discussion
> >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
> >> > >> recently.
> >> > >> > >> Feel free to share your comments on the github.
> >> > >> > >>
> >> > >> > >> Best,
> >> > >> > >> Chao
> >> > >> > >>
> >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> >> > liurenjie2008@gmail.com
> >> > >> >
> >> > >> > >> wrote:
> >> > >> > >>
> >> > >> > >>> cc:Sunchao and Any
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> ---------- Forwarded message ---------
> >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> >> > >> > >>> To: <de...@arrow.apache.org>
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> Hello,
> >> > >> > >>>
> >> > >> > >>> you might also want to raise this with the
> >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> >> > between
> >> > >> the
> >> > >> > >>> developers of this project and the Arrow Rust implementation
> is
> >> > >> quite large
> >> > >> > >>> but still it may make sense to also start a discussion there.
> >> > >> > >>>
> >> > >> > >>> Uwe
> >> > >> > >>>
> >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> >> > >> > >>> > Hi, all:
> >> > >> > >>> >
> >> > >> > >>> > Now the rust component is approaching a stable state and
> rust
> >> > >> reader
> >> > >> > >>> for
> >> > >> > >>> > parquet is ready. I think it maybe a good time to start an
> >> > >> adapter for
> >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys
> think
> >> > >> about it?
> >> > >> > >>> > --
> >> > >> > >>> > Liu, Renjie
> >> > >> > >>> > Software Engineer, MVAD
> >> > >> > >>> --
> >> > >> > >>> Liu, Renjie
> >> > >> > >>> Software Engineer, MVAD
> >> > >> > >>>
> >> > >> > >>
> >> > >> > >> --
> >> > >> > > Liu, Renjie
> >> > >> > > Software Engineer, MVAD
> >> > >> > >
> >> > >> > --
> >> > >> > Liu, Renjie
> >> > >> > Software Engineer, MVAD
> >> > >>
> >> > >
> >> >
> >> --
> >> Renjie Liu
> >> Software Engineer, MVAD
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
> Can you remind us all why nightly is required?
Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119>
for all the unstable features parquet-rs uses. I'm personally inclined to
use nightly since some new features such as specialization makes
development much easier. Like Andy mentioned, as we are still developing
arrow + parquet I think we can stay with nightly and transition back to
stable once major features are implemented and more and more people start
to use it (hopefully by that time the unstable features are stabilized).
Moreover, I've seen quite a few popular projects rely on nightly such as
rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> The steps from here are for you all to get the codebase into a state
that is ready for donation, including ASF license headers, etc. A pull
request into apache/arrow would be the best thing
Sure. I'll prepare a pull request in the next few days, and then we can
proceed to the voting, ICLA, etc. Thanks.
Chao
On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com> wrote:
> The steps from here are for you all to get the codebase into a state
> that is ready for donation, including ASF license headers, etc. A pull
> request into apache/arrow would be the best thing
>
> Then we have to do the following
>
> * Vote on the Arrow mailing list
> * Receive ICLAs from contributors
> * Complete IP clearance
> * Merge codebase
>
> Let me know when you are ready to move forward. From start to finish
> that can get done in approximately 6 days if the code is ready
>
> Thanks
> On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com> wrote:
> >
> > Renjie,
> >
> > Can you remind us all why nightly is required?
> >
> > My personal feeling is that stable is a nice-to-have, but Rust is still
> moving fast and we are on the bleeding edge here so I'm OK with Arrow
> relying on nightly for now. Maybe we can have a plan to transition back to
> stable for a future release if we go with nightly now.
> >
> > Thanks,
> >
> > Andy.
> >
> >
> > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com>
> wrote:
> >>
> >> That sounds great. But parquet-rs currently relies on nightly rust, that
> >> would be the first problem to resolve.
> >>
> >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com>
> wrote:
> >>
> >> > This sounds like a great idea.
> >> >
> >> > With support for both CSV and Parquet in the Arrow crate, it would be
> nice
> >> > to design a standard interface for Arrow data sources. Maybe this is
> as
> >> > simple as implementing `Iterator<RecordBatch>`.
> >> >
> >> > Andy.
> >> >
> >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
> >> >
> >> > > Yes, we'd be interested to move forward. I'm inclined to merge this
> into
> >> > > Arrow because of the issues that you pointed out with parquet c++
> merge,
> >> > > and I do see a tight relationship between the two projects, and
> potential
> >> > > sharing of common libraries. @Ivan Sadikov <iv...@gmail.com>
> what
> >> > > do you think?
> >> > >
> >> > > Chao
> >> > >
> >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> hi folks,
> >> > >>
> >> > >> Would you all be interested in moving forward the parquet-rs
> project?
> >> > >> I have a little more bandwidth to help with the code donation in
> the
> >> > >> next month or two.
> >> > >>
> >> > >> I know we voted on the Parquet mailing list about the donation
> >> > >> already. One big question is whether you want to create an
> >> > >> apache/parquet-rs repository or whether you want to co-develop
> >> > >> parquet-rs together with Arrow in Rust, similar to what we are
> doing
> >> > >> with C++. It's possible you might run into the same kinds of issues
> >> > >> that led us to consider the monorepo arrangement.
> >> > >>
> >> > >> Thanks
> >> > >> Wes
> >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> liurenjie2008@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > Hi, Chao:
> >> > >> > I've opened an jira issue for that and planning to work on that.
> >> > >> >
> >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> liurenjie2008@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Yes, it's a mistake, sorry for that
> >> > >> > >
> >> > >> > >
> >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
> >> > wrote:
> >> > >> > >
> >> > >> > >> (s/flink/arrow - it is a mistake?)
> >> > >> > >>
> >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step in
> >> > >> parquet-rs
> >> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> >> > discussion
> >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
> >> > >> recently.
> >> > >> > >> Feel free to share your comments on the github.
> >> > >> > >>
> >> > >> > >> Best,
> >> > >> > >> Chao
> >> > >> > >>
> >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> >> > liurenjie2008@gmail.com
> >> > >> >
> >> > >> > >> wrote:
> >> > >> > >>
> >> > >> > >>> cc:Sunchao and Any
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> ---------- Forwarded message ---------
> >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> >> > >> > >>> To: <de...@arrow.apache.org>
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> Hello,
> >> > >> > >>>
> >> > >> > >>> you might also want to raise this with the
> >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> >> > between
> >> > >> the
> >> > >> > >>> developers of this project and the Arrow Rust implementation
> is
> >> > >> quite large
> >> > >> > >>> but still it may make sense to also start a discussion there.
> >> > >> > >>>
> >> > >> > >>> Uwe
> >> > >> > >>>
> >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> >> > >> > >>> > Hi, all:
> >> > >> > >>> >
> >> > >> > >>> > Now the rust component is approaching a stable state and
> rust
> >> > >> reader
> >> > >> > >>> for
> >> > >> > >>> > parquet is ready. I think it maybe a good time to start an
> >> > >> adapter for
> >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys
> think
> >> > >> about it?
> >> > >> > >>> > --
> >> > >> > >>> > Liu, Renjie
> >> > >> > >>> > Software Engineer, MVAD
> >> > >> > >>> --
> >> > >> > >>> Liu, Renjie
> >> > >> > >>> Software Engineer, MVAD
> >> > >> > >>>
> >> > >> > >>
> >> > >> > >> --
> >> > >> > > Liu, Renjie
> >> > >> > > Software Engineer, MVAD
> >> > >> > >
> >> > >> > --
> >> > >> > Liu, Renjie
> >> > >> > Software Engineer, MVAD
> >> > >>
> >> > >
> >> >
> >> --
> >> Renjie Liu
> >> Software Engineer, MVAD
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
> Can you remind us all why nightly is required?
Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119>
for all the unstable features parquet-rs uses. I'm personally inclined to
use nightly since some new features such as specialization makes
development much easier. Like Andy mentioned, as we are still developing
arrow + parquet I think we can stay with nightly and transition back to
stable once major features are implemented and more and more people start
to use it (hopefully by that time the unstable features are stabilized).
Moreover, I've seen quite a few popular projects rely on nightly such as
rocket, tikv, etc., so seems it is not uncommon in the Rust world.
> The steps from here are for you all to get the codebase into a state
that is ready for donation, including ASF license headers, etc. A pull
request into apache/arrow would be the best thing
Sure. I'll prepare a pull request in the next few days, and then we can
proceed to the voting, ICLA, etc. Thanks.
Chao
On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <we...@gmail.com> wrote:
> The steps from here are for you all to get the codebase into a state
> that is ready for donation, including ASF license headers, etc. A pull
> request into apache/arrow would be the best thing
>
> Then we have to do the following
>
> * Vote on the Arrow mailing list
> * Receive ICLAs from contributors
> * Complete IP clearance
> * Merge codebase
>
> Let me know when you are ready to move forward. From start to finish
> that can get done in approximately 6 days if the code is ready
>
> Thanks
> On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com> wrote:
> >
> > Renjie,
> >
> > Can you remind us all why nightly is required?
> >
> > My personal feeling is that stable is a nice-to-have, but Rust is still
> moving fast and we are on the bleeding edge here so I'm OK with Arrow
> relying on nightly for now. Maybe we can have a plan to transition back to
> stable for a future release if we go with nightly now.
> >
> > Thanks,
> >
> > Andy.
> >
> >
> > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com>
> wrote:
> >>
> >> That sounds great. But parquet-rs currently relies on nightly rust, that
> >> would be the first problem to resolve.
> >>
> >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com>
> wrote:
> >>
> >> > This sounds like a great idea.
> >> >
> >> > With support for both CSV and Parquet in the Arrow crate, it would be
> nice
> >> > to design a standard interface for Arrow data sources. Maybe this is
> as
> >> > simple as implementing `Iterator<RecordBatch>`.
> >> >
> >> > Andy.
> >> >
> >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
> >> >
> >> > > Yes, we'd be interested to move forward. I'm inclined to merge this
> into
> >> > > Arrow because of the issues that you pointed out with parquet c++
> merge,
> >> > > and I do see a tight relationship between the two projects, and
> potential
> >> > > sharing of common libraries. @Ivan Sadikov <iv...@gmail.com>
> what
> >> > > do you think?
> >> > >
> >> > > Chao
> >> > >
> >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> hi folks,
> >> > >>
> >> > >> Would you all be interested in moving forward the parquet-rs
> project?
> >> > >> I have a little more bandwidth to help with the code donation in
> the
> >> > >> next month or two.
> >> > >>
> >> > >> I know we voted on the Parquet mailing list about the donation
> >> > >> already. One big question is whether you want to create an
> >> > >> apache/parquet-rs repository or whether you want to co-develop
> >> > >> parquet-rs together with Arrow in Rust, similar to what we are
> doing
> >> > >> with C++. It's possible you might run into the same kinds of issues
> >> > >> that led us to consider the monorepo arrangement.
> >> > >>
> >> > >> Thanks
> >> > >> Wes
> >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <
> liurenjie2008@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > Hi, Chao:
> >> > >> > I've opened an jira issue for that and planning to work on that.
> >> > >> >
> >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> liurenjie2008@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Yes, it's a mistake, sorry for that
> >> > >> > >
> >> > >> > >
> >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
> >> > wrote:
> >> > >> > >
> >> > >> > >> (s/flink/arrow - it is a mistake?)
> >> > >> > >>
> >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step in
> >> > >> parquet-rs
> >> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> >> > discussion
> >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
> >> > >> recently.
> >> > >> > >> Feel free to share your comments on the github.
> >> > >> > >>
> >> > >> > >> Best,
> >> > >> > >> Chao
> >> > >> > >>
> >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> >> > liurenjie2008@gmail.com
> >> > >> >
> >> > >> > >> wrote:
> >> > >> > >>
> >> > >> > >>> cc:Sunchao and Any
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> ---------- Forwarded message ---------
> >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> >> > >> > >>> To: <de...@arrow.apache.org>
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> Hello,
> >> > >> > >>>
> >> > >> > >>> you might also want to raise this with the
> >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> >> > between
> >> > >> the
> >> > >> > >>> developers of this project and the Arrow Rust implementation
> is
> >> > >> quite large
> >> > >> > >>> but still it may make sense to also start a discussion there.
> >> > >> > >>>
> >> > >> > >>> Uwe
> >> > >> > >>>
> >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> >> > >> > >>> > Hi, all:
> >> > >> > >>> >
> >> > >> > >>> > Now the rust component is approaching a stable state and
> rust
> >> > >> reader
> >> > >> > >>> for
> >> > >> > >>> > parquet is ready. I think it maybe a good time to start an
> >> > >> adapter for
> >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys
> think
> >> > >> about it?
> >> > >> > >>> > --
> >> > >> > >>> > Liu, Renjie
> >> > >> > >>> > Software Engineer, MVAD
> >> > >> > >>> --
> >> > >> > >>> Liu, Renjie
> >> > >> > >>> Software Engineer, MVAD
> >> > >> > >>>
> >> > >> > >>
> >> > >> > >> --
> >> > >> > > Liu, Renjie
> >> > >> > > Software Engineer, MVAD
> >> > >> > >
> >> > >> > --
> >> > >> > Liu, Renjie
> >> > >> > Software Engineer, MVAD
> >> > >>
> >> > >
> >> >
> >> --
> >> Renjie Liu
> >> Software Engineer, MVAD
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Wes McKinney <we...@gmail.com>.
The steps from here are for you all to get the codebase into a state
that is ready for donation, including ASF license headers, etc. A pull
request into apache/arrow would be the best thing
Then we have to do the following
* Vote on the Arrow mailing list
* Receive ICLAs from contributors
* Complete IP clearance
* Merge codebase
Let me know when you are ready to move forward. From start to finish
that can get done in approximately 6 days if the code is ready
Thanks
On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com> wrote:
>
> Renjie,
>
> Can you remind us all why nightly is required?
>
> My personal feeling is that stable is a nice-to-have, but Rust is still moving fast and we are on the bleeding edge here so I'm OK with Arrow relying on nightly for now. Maybe we can have a plan to transition back to stable for a future release if we go with nightly now.
>
> Thanks,
>
> Andy.
>
>
> On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com> wrote:
>>
>> That sounds great. But parquet-rs currently relies on nightly rust, that
>> would be the first problem to resolve.
>>
>> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com> wrote:
>>
>> > This sounds like a great idea.
>> >
>> > With support for both CSV and Parquet in the Arrow crate, it would be nice
>> > to design a standard interface for Arrow data sources. Maybe this is as
>> > simple as implementing `Iterator<RecordBatch>`.
>> >
>> > Andy.
>> >
>> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
>> >
>> > > Yes, we'd be interested to move forward. I'm inclined to merge this into
>> > > Arrow because of the issues that you pointed out with parquet c++ merge,
>> > > and I do see a tight relationship between the two projects, and potential
>> > > sharing of common libraries. @Ivan Sadikov <iv...@gmail.com> what
>> > > do you think?
>> > >
>> > > Chao
>> > >
>> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
>> > wrote:
>> > >
>> > >> hi folks,
>> > >>
>> > >> Would you all be interested in moving forward the parquet-rs project?
>> > >> I have a little more bandwidth to help with the code donation in the
>> > >> next month or two.
>> > >>
>> > >> I know we voted on the Parquet mailing list about the donation
>> > >> already. One big question is whether you want to create an
>> > >> apache/parquet-rs repository or whether you want to co-develop
>> > >> parquet-rs together with Arrow in Rust, similar to what we are doing
>> > >> with C++. It's possible you might run into the same kinds of issues
>> > >> that led us to consider the monorepo arrangement.
>> > >>
>> > >> Thanks
>> > >> Wes
>> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > Hi, Chao:
>> > >> > I've opened an jira issue for that and planning to work on that.
>> > >> >
>> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > > Yes, it's a mistake, sorry for that
>> > >> > >
>> > >> > >
>> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
>> > wrote:
>> > >> > >
>> > >> > >> (s/flink/arrow - it is a mistake?)
>> > >> > >>
>> > >> > >> Thanks Renjie for your interest. Yes, one of the next step in
>> > >> parquet-rs
>> > >> > >> is to integrate with Apache Arrow. Actually we just had a
>> > discussion
>> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
>> > >> recently.
>> > >> > >> Feel free to share your comments on the github.
>> > >> > >>
>> > >> > >> Best,
>> > >> > >> Chao
>> > >> > >>
>> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
>> > liurenjie2008@gmail.com
>> > >> >
>> > >> > >> wrote:
>> > >> > >>
>> > >> > >>> cc:Sunchao and Any
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> ---------- Forwarded message ---------
>> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
>> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
>> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
>> > >> > >>> To: <de...@arrow.apache.org>
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> Hello,
>> > >> > >>>
>> > >> > >>> you might also want to raise this with the
>> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
>> > between
>> > >> the
>> > >> > >>> developers of this project and the Arrow Rust implementation is
>> > >> quite large
>> > >> > >>> but still it may make sense to also start a discussion there.
>> > >> > >>>
>> > >> > >>> Uwe
>> > >> > >>>
>> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
>> > >> > >>> > Hi, all:
>> > >> > >>> >
>> > >> > >>> > Now the rust component is approaching a stable state and rust
>> > >> reader
>> > >> > >>> for
>> > >> > >>> > parquet is ready. I think it maybe a good time to start an
>> > >> adapter for
>> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys think
>> > >> about it?
>> > >> > >>> > --
>> > >> > >>> > Liu, Renjie
>> > >> > >>> > Software Engineer, MVAD
>> > >> > >>> --
>> > >> > >>> Liu, Renjie
>> > >> > >>> Software Engineer, MVAD
>> > >> > >>>
>> > >> > >>
>> > >> > >> --
>> > >> > > Liu, Renjie
>> > >> > > Software Engineer, MVAD
>> > >> > >
>> > >> > --
>> > >> > Liu, Renjie
>> > >> > Software Engineer, MVAD
>> > >>
>> > >
>> >
>> --
>> Renjie Liu
>> Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Wes McKinney <we...@gmail.com>.
The steps from here are for you all to get the codebase into a state
that is ready for donation, including ASF license headers, etc. A pull
request into apache/arrow would be the best thing
Then we have to do the following
* Vote on the Arrow mailing list
* Receive ICLAs from contributors
* Complete IP clearance
* Merge codebase
Let me know when you are ready to move forward. From start to finish
that can get done in approximately 6 days if the code is ready
Thanks
On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <an...@gmail.com> wrote:
>
> Renjie,
>
> Can you remind us all why nightly is required?
>
> My personal feeling is that stable is a nice-to-have, but Rust is still moving fast and we are on the bleeding edge here so I'm OK with Arrow relying on nightly for now. Maybe we can have a plan to transition back to stable for a future release if we go with nightly now.
>
> Thanks,
>
> Andy.
>
>
> On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com> wrote:
>>
>> That sounds great. But parquet-rs currently relies on nightly rust, that
>> would be the first problem to resolve.
>>
>> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com> wrote:
>>
>> > This sounds like a great idea.
>> >
>> > With support for both CSV and Parquet in the Arrow crate, it would be nice
>> > to design a standard interface for Arrow data sources. Maybe this is as
>> > simple as implementing `Iterator<RecordBatch>`.
>> >
>> > Andy.
>> >
>> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
>> >
>> > > Yes, we'd be interested to move forward. I'm inclined to merge this into
>> > > Arrow because of the issues that you pointed out with parquet c++ merge,
>> > > and I do see a tight relationship between the two projects, and potential
>> > > sharing of common libraries. @Ivan Sadikov <iv...@gmail.com> what
>> > > do you think?
>> > >
>> > > Chao
>> > >
>> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
>> > wrote:
>> > >
>> > >> hi folks,
>> > >>
>> > >> Would you all be interested in moving forward the parquet-rs project?
>> > >> I have a little more bandwidth to help with the code donation in the
>> > >> next month or two.
>> > >>
>> > >> I know we voted on the Parquet mailing list about the donation
>> > >> already. One big question is whether you want to create an
>> > >> apache/parquet-rs repository or whether you want to co-develop
>> > >> parquet-rs together with Arrow in Rust, similar to what we are doing
>> > >> with C++. It's possible you might run into the same kinds of issues
>> > >> that led us to consider the monorepo arrangement.
>> > >>
>> > >> Thanks
>> > >> Wes
>> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > Hi, Chao:
>> > >> > I've opened an jira issue for that and planning to work on that.
>> > >> >
>> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > > Yes, it's a mistake, sorry for that
>> > >> > >
>> > >> > >
>> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
>> > wrote:
>> > >> > >
>> > >> > >> (s/flink/arrow - it is a mistake?)
>> > >> > >>
>> > >> > >> Thanks Renjie for your interest. Yes, one of the next step in
>> > >> parquet-rs
>> > >> > >> is to integrate with Apache Arrow. Actually we just had a
>> > discussion
>> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
>> > >> recently.
>> > >> > >> Feel free to share your comments on the github.
>> > >> > >>
>> > >> > >> Best,
>> > >> > >> Chao
>> > >> > >>
>> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
>> > liurenjie2008@gmail.com
>> > >> >
>> > >> > >> wrote:
>> > >> > >>
>> > >> > >>> cc:Sunchao and Any
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> ---------- Forwarded message ---------
>> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
>> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
>> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
>> > >> > >>> To: <de...@arrow.apache.org>
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> Hello,
>> > >> > >>>
>> > >> > >>> you might also want to raise this with the
>> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
>> > between
>> > >> the
>> > >> > >>> developers of this project and the Arrow Rust implementation is
>> > >> quite large
>> > >> > >>> but still it may make sense to also start a discussion there.
>> > >> > >>>
>> > >> > >>> Uwe
>> > >> > >>>
>> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
>> > >> > >>> > Hi, all:
>> > >> > >>> >
>> > >> > >>> > Now the rust component is approaching a stable state and rust
>> > >> reader
>> > >> > >>> for
>> > >> > >>> > parquet is ready. I think it maybe a good time to start an
>> > >> adapter for
>> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys think
>> > >> about it?
>> > >> > >>> > --
>> > >> > >>> > Liu, Renjie
>> > >> > >>> > Software Engineer, MVAD
>> > >> > >>> --
>> > >> > >>> Liu, Renjie
>> > >> > >>> Software Engineer, MVAD
>> > >> > >>>
>> > >> > >>
>> > >> > >> --
>> > >> > > Liu, Renjie
>> > >> > > Software Engineer, MVAD
>> > >> > >
>> > >> > --
>> > >> > Liu, Renjie
>> > >> > Software Engineer, MVAD
>> > >>
>> > >
>> >
>> --
>> Renjie Liu
>> Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Andy Grove <an...@gmail.com>.
Renjie,
Can you remind us all why nightly is required?
My personal feeling is that stable is a nice-to-have, but Rust is still
moving fast and we are on the bleeding edge here so I'm OK with Arrow
relying on nightly for now. Maybe we can have a plan to transition back to
stable for a future release if we go with nightly now.
Thanks,
Andy.
On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com> wrote:
> That sounds great. But parquet-rs currently relies on nightly rust, that
> would be the first problem to resolve.
>
> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com> wrote:
>
> > This sounds like a great idea.
> >
> > With support for both CSV and Parquet in the Arrow crate, it would be
> nice
> > to design a standard interface for Arrow data sources. Maybe this is as
> > simple as implementing `Iterator<RecordBatch>`.
> >
> > Andy.
> >
> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
> >
> > > Yes, we'd be interested to move forward. I'm inclined to merge this
> into
> > > Arrow because of the issues that you pointed out with parquet c++
> merge,
> > > and I do see a tight relationship between the two projects, and
> potential
> > > sharing of common libraries. @Ivan Sadikov <iv...@gmail.com>
> what
> > > do you think?
> > >
> > > Chao
> > >
> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > >
> > >> hi folks,
> > >>
> > >> Would you all be interested in moving forward the parquet-rs project?
> > >> I have a little more bandwidth to help with the code donation in the
> > >> next month or two.
> > >>
> > >> I know we voted on the Parquet mailing list about the donation
> > >> already. One big question is whether you want to create an
> > >> apache/parquet-rs repository or whether you want to co-develop
> > >> parquet-rs together with Arrow in Rust, similar to what we are doing
> > >> with C++. It's possible you might run into the same kinds of issues
> > >> that led us to consider the monorepo arrangement.
> > >>
> > >> Thanks
> > >> Wes
> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
> > >> wrote:
> > >> >
> > >> > Hi, Chao:
> > >> > I've opened an jira issue for that and planning to work on that.
> > >> >
> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> liurenjie2008@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Yes, it's a mistake, sorry for that
> > >> > >
> > >> > >
> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
> > wrote:
> > >> > >
> > >> > >> (s/flink/arrow - it is a mistake?)
> > >> > >>
> > >> > >> Thanks Renjie for your interest. Yes, one of the next step in
> > >> parquet-rs
> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> > discussion
> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
> > >> recently.
> > >> > >> Feel free to share your comments on the github.
> > >> > >>
> > >> > >> Best,
> > >> > >> Chao
> > >> > >>
> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > liurenjie2008@gmail.com
> > >> >
> > >> > >> wrote:
> > >> > >>
> > >> > >>> cc:Sunchao and Any
> > >> > >>>
> > >> > >>>
> > >> > >>> ---------- Forwarded message ---------
> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >> > >>> To: <de...@arrow.apache.org>
> > >> > >>>
> > >> > >>>
> > >> > >>> Hello,
> > >> > >>>
> > >> > >>> you might also want to raise this with the
> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> > between
> > >> the
> > >> > >>> developers of this project and the Arrow Rust implementation is
> > >> quite large
> > >> > >>> but still it may make sense to also start a discussion there.
> > >> > >>>
> > >> > >>> Uwe
> > >> > >>>
> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > >> > >>> > Hi, all:
> > >> > >>> >
> > >> > >>> > Now the rust component is approaching a stable state and rust
> > >> reader
> > >> > >>> for
> > >> > >>> > parquet is ready. I think it maybe a good time to start an
> > >> adapter for
> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys think
> > >> about it?
> > >> > >>> > --
> > >> > >>> > Liu, Renjie
> > >> > >>> > Software Engineer, MVAD
> > >> > >>> --
> > >> > >>> Liu, Renjie
> > >> > >>> Software Engineer, MVAD
> > >> > >>>
> > >> > >>
> > >> > >> --
> > >> > > Liu, Renjie
> > >> > > Software Engineer, MVAD
> > >> > >
> > >> > --
> > >> > Liu, Renjie
> > >> > Software Engineer, MVAD
> > >>
> > >
> >
> --
> Renjie Liu
> Software Engineer, MVAD
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Andy Grove <an...@gmail.com>.
Renjie,
Can you remind us all why nightly is required?
My personal feeling is that stable is a nice-to-have, but Rust is still
moving fast and we are on the bleeding edge here so I'm OK with Arrow
relying on nightly for now. Maybe we can have a plan to transition back to
stable for a future release if we go with nightly now.
Thanks,
Andy.
On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <li...@gmail.com> wrote:
> That sounds great. But parquet-rs currently relies on nightly rust, that
> would be the first problem to resolve.
>
> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com> wrote:
>
> > This sounds like a great idea.
> >
> > With support for both CSV and Parquet in the Arrow crate, it would be
> nice
> > to design a standard interface for Arrow data sources. Maybe this is as
> > simple as implementing `Iterator<RecordBatch>`.
> >
> > Andy.
> >
> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
> >
> > > Yes, we'd be interested to move forward. I'm inclined to merge this
> into
> > > Arrow because of the issues that you pointed out with parquet c++
> merge,
> > > and I do see a tight relationship between the two projects, and
> potential
> > > sharing of common libraries. @Ivan Sadikov <iv...@gmail.com>
> what
> > > do you think?
> > >
> > > Chao
> > >
> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > >
> > >> hi folks,
> > >>
> > >> Would you all be interested in moving forward the parquet-rs project?
> > >> I have a little more bandwidth to help with the code donation in the
> > >> next month or two.
> > >>
> > >> I know we voted on the Parquet mailing list about the donation
> > >> already. One big question is whether you want to create an
> > >> apache/parquet-rs repository or whether you want to co-develop
> > >> parquet-rs together with Arrow in Rust, similar to what we are doing
> > >> with C++. It's possible you might run into the same kinds of issues
> > >> that led us to consider the monorepo arrangement.
> > >>
> > >> Thanks
> > >> Wes
> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
> > >> wrote:
> > >> >
> > >> > Hi, Chao:
> > >> > I've opened an jira issue for that and planning to work on that.
> > >> >
> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <
> liurenjie2008@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Yes, it's a mistake, sorry for that
> > >> > >
> > >> > >
> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
> > wrote:
> > >> > >
> > >> > >> (s/flink/arrow - it is a mistake?)
> > >> > >>
> > >> > >> Thanks Renjie for your interest. Yes, one of the next step in
> > >> parquet-rs
> > >> > >> is to integrate with Apache Arrow. Actually we just had a
> > discussion
> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
> > >> recently.
> > >> > >> Feel free to share your comments on the github.
> > >> > >>
> > >> > >> Best,
> > >> > >> Chao
> > >> > >>
> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> > liurenjie2008@gmail.com
> > >> >
> > >> > >> wrote:
> > >> > >>
> > >> > >>> cc:Sunchao and Any
> > >> > >>>
> > >> > >>>
> > >> > >>> ---------- Forwarded message ---------
> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >> > >>> To: <de...@arrow.apache.org>
> > >> > >>>
> > >> > >>>
> > >> > >>> Hello,
> > >> > >>>
> > >> > >>> you might also want to raise this with the
> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> > between
> > >> the
> > >> > >>> developers of this project and the Arrow Rust implementation is
> > >> quite large
> > >> > >>> but still it may make sense to also start a discussion there.
> > >> > >>>
> > >> > >>> Uwe
> > >> > >>>
> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > >> > >>> > Hi, all:
> > >> > >>> >
> > >> > >>> > Now the rust component is approaching a stable state and rust
> > >> reader
> > >> > >>> for
> > >> > >>> > parquet is ready. I think it maybe a good time to start an
> > >> adapter for
> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys think
> > >> about it?
> > >> > >>> > --
> > >> > >>> > Liu, Renjie
> > >> > >>> > Software Engineer, MVAD
> > >> > >>> --
> > >> > >>> Liu, Renjie
> > >> > >>> Software Engineer, MVAD
> > >> > >>>
> > >> > >>
> > >> > >> --
> > >> > > Liu, Renjie
> > >> > > Software Engineer, MVAD
> > >> > >
> > >> > --
> > >> > Liu, Renjie
> > >> > Software Engineer, MVAD
> > >>
> > >
> >
> --
> Renjie Liu
> Software Engineer, MVAD
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Renjie Liu <li...@gmail.com>.
That sounds great. But parquet-rs currently relies on nightly rust, that
would be the first problem to resolve.
On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <an...@gmail.com> wrote:
> This sounds like a great idea.
>
> With support for both CSV and Parquet in the Arrow crate, it would be nice
> to design a standard interface for Arrow data sources. Maybe this is as
> simple as implementing `Iterator<RecordBatch>`.
>
> Andy.
>
> On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
>
> > Yes, we'd be interested to move forward. I'm inclined to merge this into
> > Arrow because of the issues that you pointed out with parquet c++ merge,
> > and I do see a tight relationship between the two projects, and potential
> > sharing of common libraries. @Ivan Sadikov <iv...@gmail.com> what
> > do you think?
> >
> > Chao
> >
> > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com>
> wrote:
> >
> >> hi folks,
> >>
> >> Would you all be interested in moving forward the parquet-rs project?
> >> I have a little more bandwidth to help with the code donation in the
> >> next month or two.
> >>
> >> I know we voted on the Parquet mailing list about the donation
> >> already. One big question is whether you want to create an
> >> apache/parquet-rs repository or whether you want to co-develop
> >> parquet-rs together with Arrow in Rust, similar to what we are doing
> >> with C++. It's possible you might run into the same kinds of issues
> >> that led us to consider the monorepo arrangement.
> >>
> >> Thanks
> >> Wes
> >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
> >> wrote:
> >> >
> >> > Hi, Chao:
> >> > I've opened an jira issue for that and planning to work on that.
> >> >
> >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com>
> >> wrote:
> >> >
> >> > > Yes, it's a mistake, sorry for that
> >> > >
> >> > >
> >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org>
> wrote:
> >> > >
> >> > >> (s/flink/arrow - it is a mistake?)
> >> > >>
> >> > >> Thanks Renjie for your interest. Yes, one of the next step in
> >> parquet-rs
> >> > >> is to integrate with Apache Arrow. Actually we just had a
> discussion
> >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
> >> recently.
> >> > >> Feel free to share your comments on the github.
> >> > >>
> >> > >> Best,
> >> > >> Chao
> >> > >>
> >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <
> liurenjie2008@gmail.com
> >> >
> >> > >> wrote:
> >> > >>
> >> > >>> cc:Sunchao and Any
> >> > >>>
> >> > >>>
> >> > >>> ---------- Forwarded message ---------
> >> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> >> > >>> To: <de...@arrow.apache.org>
> >> > >>>
> >> > >>>
> >> > >>> Hello,
> >> > >>>
> >> > >>> you might also want to raise this with the
> >> > >>> https://github.com/sunchao/parquet-rs project. The overlap
> between
> >> the
> >> > >>> developers of this project and the Arrow Rust implementation is
> >> quite large
> >> > >>> but still it may make sense to also start a discussion there.
> >> > >>>
> >> > >>> Uwe
> >> > >>>
> >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> >> > >>> > Hi, all:
> >> > >>> >
> >> > >>> > Now the rust component is approaching a stable state and rust
> >> reader
> >> > >>> for
> >> > >>> > parquet is ready. I think it maybe a good time to start an
> >> adapter for
> >> > >>> > parquet, just like adapter for orc in cpp. How you guys think
> >> about it?
> >> > >>> > --
> >> > >>> > Liu, Renjie
> >> > >>> > Software Engineer, MVAD
> >> > >>> --
> >> > >>> Liu, Renjie
> >> > >>> Software Engineer, MVAD
> >> > >>>
> >> > >>
> >> > >> --
> >> > > Liu, Renjie
> >> > > Software Engineer, MVAD
> >> > >
> >> > --
> >> > Liu, Renjie
> >> > Software Engineer, MVAD
> >>
> >
>
--
Renjie Liu
Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Andy Grove <an...@gmail.com>.
This sounds like a great idea.
With support for both CSV and Parquet in the Arrow crate, it would be nice
to design a standard interface for Arrow data sources. Maybe this is as
simple as implementing `Iterator<RecordBatch>`.
Andy.
On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
> Yes, we'd be interested to move forward. I'm inclined to merge this into
> Arrow because of the issues that you pointed out with parquet c++ merge,
> and I do see a tight relationship between the two projects, and potential
> sharing of common libraries. @Ivan Sadikov <iv...@gmail.com> what
> do you think?
>
> Chao
>
> On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com> wrote:
>
>> hi folks,
>>
>> Would you all be interested in moving forward the parquet-rs project?
>> I have a little more bandwidth to help with the code donation in the
>> next month or two.
>>
>> I know we voted on the Parquet mailing list about the donation
>> already. One big question is whether you want to create an
>> apache/parquet-rs repository or whether you want to co-develop
>> parquet-rs together with Arrow in Rust, similar to what we are doing
>> with C++. It's possible you might run into the same kinds of issues
>> that led us to consider the monorepo arrangement.
>>
>> Thanks
>> Wes
>> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
>> wrote:
>> >
>> > Hi, Chao:
>> > I've opened an jira issue for that and planning to work on that.
>> >
>> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com>
>> wrote:
>> >
>> > > Yes, it's a mistake, sorry for that
>> > >
>> > >
>> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org> wrote:
>> > >
>> > >> (s/flink/arrow - it is a mistake?)
>> > >>
>> > >> Thanks Renjie for your interest. Yes, one of the next step in
>> parquet-rs
>> > >> is to integrate with Apache Arrow. Actually we just had a discussion
>> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
>> recently.
>> > >> Feel free to share your comments on the github.
>> > >>
>> > >> Best,
>> > >> Chao
>> > >>
>> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <liurenjie2008@gmail.com
>> >
>> > >> wrote:
>> > >>
>> > >>> cc:Sunchao and Any
>> > >>>
>> > >>>
>> > >>> ---------- Forwarded message ---------
>> > >>> From: Uwe L. Korn <uw...@xhochy.com>
>> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
>> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
>> > >>> To: <de...@arrow.apache.org>
>> > >>>
>> > >>>
>> > >>> Hello,
>> > >>>
>> > >>> you might also want to raise this with the
>> > >>> https://github.com/sunchao/parquet-rs project. The overlap between
>> the
>> > >>> developers of this project and the Arrow Rust implementation is
>> quite large
>> > >>> but still it may make sense to also start a discussion there.
>> > >>>
>> > >>> Uwe
>> > >>>
>> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
>> > >>> > Hi, all:
>> > >>> >
>> > >>> > Now the rust component is approaching a stable state and rust
>> reader
>> > >>> for
>> > >>> > parquet is ready. I think it maybe a good time to start an
>> adapter for
>> > >>> > parquet, just like adapter for orc in cpp. How you guys think
>> about it?
>> > >>> > --
>> > >>> > Liu, Renjie
>> > >>> > Software Engineer, MVAD
>> > >>> --
>> > >>> Liu, Renjie
>> > >>> Software Engineer, MVAD
>> > >>>
>> > >>
>> > >> --
>> > > Liu, Renjie
>> > > Software Engineer, MVAD
>> > >
>> > --
>> > Liu, Renjie
>> > Software Engineer, MVAD
>>
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Andy Grove <an...@gmail.com>.
This sounds like a great idea.
With support for both CSV and Parquet in the Arrow crate, it would be nice
to design a standard interface for Arrow data sources. Maybe this is as
simple as implementing `Iterator<RecordBatch>`.
Andy.
On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <su...@apache.org> wrote:
> Yes, we'd be interested to move forward. I'm inclined to merge this into
> Arrow because of the issues that you pointed out with parquet c++ merge,
> and I do see a tight relationship between the two projects, and potential
> sharing of common libraries. @Ivan Sadikov <iv...@gmail.com> what
> do you think?
>
> Chao
>
> On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com> wrote:
>
>> hi folks,
>>
>> Would you all be interested in moving forward the parquet-rs project?
>> I have a little more bandwidth to help with the code donation in the
>> next month or two.
>>
>> I know we voted on the Parquet mailing list about the donation
>> already. One big question is whether you want to create an
>> apache/parquet-rs repository or whether you want to co-develop
>> parquet-rs together with Arrow in Rust, similar to what we are doing
>> with C++. It's possible you might run into the same kinds of issues
>> that led us to consider the monorepo arrangement.
>>
>> Thanks
>> Wes
>> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
>> wrote:
>> >
>> > Hi, Chao:
>> > I've opened an jira issue for that and planning to work on that.
>> >
>> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com>
>> wrote:
>> >
>> > > Yes, it's a mistake, sorry for that
>> > >
>> > >
>> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org> wrote:
>> > >
>> > >> (s/flink/arrow - it is a mistake?)
>> > >>
>> > >> Thanks Renjie for your interest. Yes, one of the next step in
>> parquet-rs
>> > >> is to integrate with Apache Arrow. Actually we just had a discussion
>> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
>> recently.
>> > >> Feel free to share your comments on the github.
>> > >>
>> > >> Best,
>> > >> Chao
>> > >>
>> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <liurenjie2008@gmail.com
>> >
>> > >> wrote:
>> > >>
>> > >>> cc:Sunchao and Any
>> > >>>
>> > >>>
>> > >>> ---------- Forwarded message ---------
>> > >>> From: Uwe L. Korn <uw...@xhochy.com>
>> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
>> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
>> > >>> To: <de...@arrow.apache.org>
>> > >>>
>> > >>>
>> > >>> Hello,
>> > >>>
>> > >>> you might also want to raise this with the
>> > >>> https://github.com/sunchao/parquet-rs project. The overlap between
>> the
>> > >>> developers of this project and the Arrow Rust implementation is
>> quite large
>> > >>> but still it may make sense to also start a discussion there.
>> > >>>
>> > >>> Uwe
>> > >>>
>> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
>> > >>> > Hi, all:
>> > >>> >
>> > >>> > Now the rust component is approaching a stable state and rust
>> reader
>> > >>> for
>> > >>> > parquet is ready. I think it maybe a good time to start an
>> adapter for
>> > >>> > parquet, just like adapter for orc in cpp. How you guys think
>> about it?
>> > >>> > --
>> > >>> > Liu, Renjie
>> > >>> > Software Engineer, MVAD
>> > >>> --
>> > >>> Liu, Renjie
>> > >>> Software Engineer, MVAD
>> > >>>
>> > >>
>> > >> --
>> > > Liu, Renjie
>> > > Software Engineer, MVAD
>> > >
>> > --
>> > Liu, Renjie
>> > Software Engineer, MVAD
>>
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
Yes, we'd be interested to move forward. I'm inclined to merge this into
Arrow because of the issues that you pointed out with parquet c++ merge,
and I do see a tight relationship between the two projects, and potential
sharing of common libraries. @Ivan Sadikov <iv...@gmail.com> what do
you think?
Chao
On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com> wrote:
> hi folks,
>
> Would you all be interested in moving forward the parquet-rs project?
> I have a little more bandwidth to help with the code donation in the
> next month or two.
>
> I know we voted on the Parquet mailing list about the donation
> already. One big question is whether you want to create an
> apache/parquet-rs repository or whether you want to co-develop
> parquet-rs together with Arrow in Rust, similar to what we are doing
> with C++. It's possible you might run into the same kinds of issues
> that led us to consider the monorepo arrangement.
>
> Thanks
> Wes
> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
> wrote:
> >
> > Hi, Chao:
> > I've opened an jira issue for that and planning to work on that.
> >
> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com>
> wrote:
> >
> > > Yes, it's a mistake, sorry for that
> > >
> > >
> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org> wrote:
> > >
> > >> (s/flink/arrow - it is a mistake?)
> > >>
> > >> Thanks Renjie for your interest. Yes, one of the next step in
> parquet-rs
> > >> is to integrate with Apache Arrow. Actually we just had a discussion
> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
> recently.
> > >> Feel free to share your comments on the github.
> > >>
> > >> Best,
> > >> Chao
> > >>
> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <li...@gmail.com>
> > >> wrote:
> > >>
> > >>> cc:Sunchao and Any
> > >>>
> > >>>
> > >>> ---------- Forwarded message ---------
> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >>> To: <de...@arrow.apache.org>
> > >>>
> > >>>
> > >>> Hello,
> > >>>
> > >>> you might also want to raise this with the
> > >>> https://github.com/sunchao/parquet-rs project. The overlap between
> the
> > >>> developers of this project and the Arrow Rust implementation is
> quite large
> > >>> but still it may make sense to also start a discussion there.
> > >>>
> > >>> Uwe
> > >>>
> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > >>> > Hi, all:
> > >>> >
> > >>> > Now the rust component is approaching a stable state and rust
> reader
> > >>> for
> > >>> > parquet is ready. I think it maybe a good time to start an adapter
> for
> > >>> > parquet, just like adapter for orc in cpp. How you guys think
> about it?
> > >>> > --
> > >>> > Liu, Renjie
> > >>> > Software Engineer, MVAD
> > >>> --
> > >>> Liu, Renjie
> > >>> Software Engineer, MVAD
> > >>>
> > >>
> > >> --
> > > Liu, Renjie
> > > Software Engineer, MVAD
> > >
> > --
> > Liu, Renjie
> > Software Engineer, MVAD
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
Yes, we'd be interested to move forward. I'm inclined to merge this into
Arrow because of the issues that you pointed out with parquet c++ merge,
and I do see a tight relationship between the two projects, and potential
sharing of common libraries. @Ivan Sadikov <iv...@gmail.com> what do
you think?
Chao
On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney <we...@gmail.com> wrote:
> hi folks,
>
> Would you all be interested in moving forward the parquet-rs project?
> I have a little more bandwidth to help with the code donation in the
> next month or two.
>
> I know we voted on the Parquet mailing list about the donation
> already. One big question is whether you want to create an
> apache/parquet-rs repository or whether you want to co-develop
> parquet-rs together with Arrow in Rust, similar to what we are doing
> with C++. It's possible you might run into the same kinds of issues
> that led us to consider the monorepo arrangement.
>
> Thanks
> Wes
> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com>
> wrote:
> >
> > Hi, Chao:
> > I've opened an jira issue for that and planning to work on that.
> >
> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com>
> wrote:
> >
> > > Yes, it's a mistake, sorry for that
> > >
> > >
> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org> wrote:
> > >
> > >> (s/flink/arrow - it is a mistake?)
> > >>
> > >> Thanks Renjie for your interest. Yes, one of the next step in
> parquet-rs
> > >> is to integrate with Apache Arrow. Actually we just had a discussion
> > >> <https://github.com/sunchao/parquet-rs/issues/140> about this
> recently.
> > >> Feel free to share your comments on the github.
> > >>
> > >> Best,
> > >> Chao
> > >>
> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <li...@gmail.com>
> > >> wrote:
> > >>
> > >>> cc:Sunchao and Any
> > >>>
> > >>>
> > >>> ---------- Forwarded message ---------
> > >>> From: Uwe L. Korn <uw...@xhochy.com>
> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> > >>> To: <de...@arrow.apache.org>
> > >>>
> > >>>
> > >>> Hello,
> > >>>
> > >>> you might also want to raise this with the
> > >>> https://github.com/sunchao/parquet-rs project. The overlap between
> the
> > >>> developers of this project and the Arrow Rust implementation is
> quite large
> > >>> but still it may make sense to also start a discussion there.
> > >>>
> > >>> Uwe
> > >>>
> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > >>> > Hi, all:
> > >>> >
> > >>> > Now the rust component is approaching a stable state and rust
> reader
> > >>> for
> > >>> > parquet is ready. I think it maybe a good time to start an adapter
> for
> > >>> > parquet, just like adapter for orc in cpp. How you guys think
> about it?
> > >>> > --
> > >>> > Liu, Renjie
> > >>> > Software Engineer, MVAD
> > >>> --
> > >>> Liu, Renjie
> > >>> Software Engineer, MVAD
> > >>>
> > >>
> > >> --
> > > Liu, Renjie
> > > Software Engineer, MVAD
> > >
> > --
> > Liu, Renjie
> > Software Engineer, MVAD
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Wes McKinney <we...@gmail.com>.
hi folks,
Would you all be interested in moving forward the parquet-rs project?
I have a little more bandwidth to help with the code donation in the
next month or two.
I know we voted on the Parquet mailing list about the donation
already. One big question is whether you want to create an
apache/parquet-rs repository or whether you want to co-develop
parquet-rs together with Arrow in Rust, similar to what we are doing
with C++. It's possible you might run into the same kinds of issues
that led us to consider the monorepo arrangement.
Thanks
Wes
On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com> wrote:
>
> Hi, Chao:
> I've opened an jira issue for that and planning to work on that.
>
> On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com> wrote:
>
> > Yes, it's a mistake, sorry for that
> >
> >
> > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org> wrote:
> >
> >> (s/flink/arrow - it is a mistake?)
> >>
> >> Thanks Renjie for your interest. Yes, one of the next step in parquet-rs
> >> is to integrate with Apache Arrow. Actually we just had a discussion
> >> <https://github.com/sunchao/parquet-rs/issues/140> about this recently.
> >> Feel free to share your comments on the github.
> >>
> >> Best,
> >> Chao
> >>
> >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <li...@gmail.com>
> >> wrote:
> >>
> >>> cc:Sunchao and Any
> >>>
> >>>
> >>> ---------- Forwarded message ---------
> >>> From: Uwe L. Korn <uw...@xhochy.com>
> >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> >>> To: <de...@arrow.apache.org>
> >>>
> >>>
> >>> Hello,
> >>>
> >>> you might also want to raise this with the
> >>> https://github.com/sunchao/parquet-rs project. The overlap between the
> >>> developers of this project and the Arrow Rust implementation is quite large
> >>> but still it may make sense to also start a discussion there.
> >>>
> >>> Uwe
> >>>
> >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> >>> > Hi, all:
> >>> >
> >>> > Now the rust component is approaching a stable state and rust reader
> >>> for
> >>> > parquet is ready. I think it maybe a good time to start an adapter for
> >>> > parquet, just like adapter for orc in cpp. How you guys think about it?
> >>> > --
> >>> > Liu, Renjie
> >>> > Software Engineer, MVAD
> >>> --
> >>> Liu, Renjie
> >>> Software Engineer, MVAD
> >>>
> >>
> >> --
> > Liu, Renjie
> > Software Engineer, MVAD
> >
> --
> Liu, Renjie
> Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Wes McKinney <we...@gmail.com>.
hi folks,
Would you all be interested in moving forward the parquet-rs project?
I have a little more bandwidth to help with the code donation in the
next month or two.
I know we voted on the Parquet mailing list about the donation
already. One big question is whether you want to create an
apache/parquet-rs repository or whether you want to co-develop
parquet-rs together with Arrow in Rust, similar to what we are doing
with C++. It's possible you might run into the same kinds of issues
that led us to consider the monorepo arrangement.
Thanks
Wes
On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu <li...@gmail.com> wrote:
>
> Hi, Chao:
> I've opened an jira issue for that and planning to work on that.
>
> On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com> wrote:
>
> > Yes, it's a mistake, sorry for that
> >
> >
> > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org> wrote:
> >
> >> (s/flink/arrow - it is a mistake?)
> >>
> >> Thanks Renjie for your interest. Yes, one of the next step in parquet-rs
> >> is to integrate with Apache Arrow. Actually we just had a discussion
> >> <https://github.com/sunchao/parquet-rs/issues/140> about this recently.
> >> Feel free to share your comments on the github.
> >>
> >> Best,
> >> Chao
> >>
> >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <li...@gmail.com>
> >> wrote:
> >>
> >>> cc:Sunchao and Any
> >>>
> >>>
> >>> ---------- Forwarded message ---------
> >>> From: Uwe L. Korn <uw...@xhochy.com>
> >>> Date: Sun, Aug 19, 2018 at 5:08 PM
> >>> Subject: Re: [DISCUSS] Rust add adapter for parquet
> >>> To: <de...@arrow.apache.org>
> >>>
> >>>
> >>> Hello,
> >>>
> >>> you might also want to raise this with the
> >>> https://github.com/sunchao/parquet-rs project. The overlap between the
> >>> developers of this project and the Arrow Rust implementation is quite large
> >>> but still it may make sense to also start a discussion there.
> >>>
> >>> Uwe
> >>>
> >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> >>> > Hi, all:
> >>> >
> >>> > Now the rust component is approaching a stable state and rust reader
> >>> for
> >>> > parquet is ready. I think it maybe a good time to start an adapter for
> >>> > parquet, just like adapter for orc in cpp. How you guys think about it?
> >>> > --
> >>> > Liu, Renjie
> >>> > Software Engineer, MVAD
> >>> --
> >>> Liu, Renjie
> >>> Software Engineer, MVAD
> >>>
> >>
> >> --
> > Liu, Renjie
> > Software Engineer, MVAD
> >
> --
> Liu, Renjie
> Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Renjie Liu <li...@gmail.com>.
Hi, Chao:
I've opened an jira issue for that and planning to work on that.
On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu <li...@gmail.com> wrote:
> Yes, it's a mistake, sorry for that
>
>
> On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org> wrote:
>
>> (s/flink/arrow - it is a mistake?)
>>
>> Thanks Renjie for your interest. Yes, one of the next step in parquet-rs
>> is to integrate with Apache Arrow. Actually we just had a discussion
>> <https://github.com/sunchao/parquet-rs/issues/140> about this recently.
>> Feel free to share your comments on the github.
>>
>> Best,
>> Chao
>>
>> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <li...@gmail.com>
>> wrote:
>>
>>> cc:Sunchao and Any
>>>
>>>
>>> ---------- Forwarded message ---------
>>> From: Uwe L. Korn <uw...@xhochy.com>
>>> Date: Sun, Aug 19, 2018 at 5:08 PM
>>> Subject: Re: [DISCUSS] Rust add adapter for parquet
>>> To: <de...@arrow.apache.org>
>>>
>>>
>>> Hello,
>>>
>>> you might also want to raise this with the
>>> https://github.com/sunchao/parquet-rs project. The overlap between the
>>> developers of this project and the Arrow Rust implementation is quite large
>>> but still it may make sense to also start a discussion there.
>>>
>>> Uwe
>>>
>>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
>>> > Hi, all:
>>> >
>>> > Now the rust component is approaching a stable state and rust reader
>>> for
>>> > parquet is ready. I think it maybe a good time to start an adapter for
>>> > parquet, just like adapter for orc in cpp. How you guys think about it?
>>> > --
>>> > Liu, Renjie
>>> > Software Engineer, MVAD
>>> --
>>> Liu, Renjie
>>> Software Engineer, MVAD
>>>
>>
>> --
> Liu, Renjie
> Software Engineer, MVAD
>
--
Liu, Renjie
Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Renjie Liu <li...@gmail.com>.
Yes, it's a mistake, sorry for that
On Mon, Aug 20, 2018 at 10:57 AM Chao Sun <su...@apache.org> wrote:
> (s/flink/arrow - it is a mistake?)
>
> Thanks Renjie for your interest. Yes, one of the next step in parquet-rs
> is to integrate with Apache Arrow. Actually we just had a discussion
> <https://github.com/sunchao/parquet-rs/issues/140> about this recently.
> Feel free to share your comments on the github.
>
> Best,
> Chao
>
> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <li...@gmail.com>
> wrote:
>
>> cc:Sunchao and Any
>>
>>
>> ---------- Forwarded message ---------
>> From: Uwe L. Korn <uw...@xhochy.com>
>> Date: Sun, Aug 19, 2018 at 5:08 PM
>> Subject: Re: [DISCUSS] Rust add adapter for parquet
>> To: <de...@arrow.apache.org>
>>
>>
>> Hello,
>>
>> you might also want to raise this with the
>> https://github.com/sunchao/parquet-rs project. The overlap between the
>> developers of this project and the Arrow Rust implementation is quite large
>> but still it may make sense to also start a discussion there.
>>
>> Uwe
>>
>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
>> > Hi, all:
>> >
>> > Now the rust component is approaching a stable state and rust reader for
>> > parquet is ready. I think it maybe a good time to start an adapter for
>> > parquet, just like adapter for orc in cpp. How you guys think about it?
>> > --
>> > Liu, Renjie
>> > Software Engineer, MVAD
>> --
>> Liu, Renjie
>> Software Engineer, MVAD
>>
>
> --
Liu, Renjie
Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Chao Sun <su...@apache.org>.
(s/flink/arrow - it is a mistake?)
Thanks Renjie for your interest. Yes, one of the next step in parquet-rs is
to integrate with Apache Arrow. Actually we just had a discussion
<https://github.com/sunchao/parquet-rs/issues/140> about this recently.
Feel free to share your comments on the github.
Best,
Chao
On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu <li...@gmail.com> wrote:
> cc:Sunchao and Any
>
>
> ---------- Forwarded message ---------
> From: Uwe L. Korn <uw...@xhochy.com>
> Date: Sun, Aug 19, 2018 at 5:08 PM
> Subject: Re: [DISCUSS] Rust add adapter for parquet
> To: <de...@arrow.apache.org>
>
>
> Hello,
>
> you might also want to raise this with the https://github.com/sunchao/
> parquet-rs project. The overlap between the developers of this project
> and the Arrow Rust implementation is quite large but still it may make
> sense to also start a discussion there.
>
> Uwe
>
> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > Hi, all:
> >
> > Now the rust component is approaching a stable state and rust reader for
> > parquet is ready. I think it maybe a good time to start an adapter for
> > parquet, just like adapter for orc in cpp. How you guys think about it?
> > --
> > Liu, Renjie
> > Software Engineer, MVAD
> --
> Liu, Renjie
> Software Engineer, MVAD
>
Re: [DISCUSS] Rust add adapter for parquet
Posted by Renjie Liu <li...@gmail.com>.
This thread is for arrow, I'm sorry for sending it to flink community by
mistake.
On Mon, Aug 20, 2018 at 3:57 PM Fabian Hueske <fh...@gmail.com> wrote:
> Hi Renjie,
>
> Did you intend to send this mail to dev@arrow.a.o instead of dev@flink.a.o
> ?
>
> Best, Fabian
>
> 2018-08-20 4:39 GMT+02:00 Renjie Liu <li...@gmail.com>:
>
> > cc:Sunchao and Any
> >
> > ---------- Forwarded message ---------
> > From: Uwe L. Korn <uw...@xhochy.com>
> > Date: Sun, Aug 19, 2018 at 5:08 PM
> > Subject: Re: [DISCUSS] Rust add adapter for parquet
> > To: <de...@arrow.apache.org>
> >
> >
> > Hello,
> >
> > you might also want to raise this with the
> > https://github.com/sunchao/parquet-rs project. The overlap between the
> > developers of this project and the Arrow Rust implementation is quite
> large
> > but still it may make sense to also start a discussion there.
> >
> > Uwe
> >
> > On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > > Hi, all:
> > >
> > > Now the rust component is approaching a stable state and rust reader
> for
> > > parquet is ready. I think it maybe a good time to start an adapter for
> > > parquet, just like adapter for orc in cpp. How you guys think about it?
> > > --
> > > Liu, Renjie
> > > Software Engineer, MVAD
> > --
> > Liu, Renjie
> > Software Engineer, MVAD
> >
>
--
Liu, Renjie
Software Engineer, MVAD
Re: [DISCUSS] Rust add adapter for parquet
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Renjie,
Did you intend to send this mail to dev@arrow.a.o instead of dev@flink.a.o?
Best, Fabian
2018-08-20 4:39 GMT+02:00 Renjie Liu <li...@gmail.com>:
> cc:Sunchao and Any
>
> ---------- Forwarded message ---------
> From: Uwe L. Korn <uw...@xhochy.com>
> Date: Sun, Aug 19, 2018 at 5:08 PM
> Subject: Re: [DISCUSS] Rust add adapter for parquet
> To: <de...@arrow.apache.org>
>
>
> Hello,
>
> you might also want to raise this with the
> https://github.com/sunchao/parquet-rs project. The overlap between the
> developers of this project and the Arrow Rust implementation is quite large
> but still it may make sense to also start a discussion there.
>
> Uwe
>
> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote:
> > Hi, all:
> >
> > Now the rust component is approaching a stable state and rust reader for
> > parquet is ready. I think it maybe a good time to start an adapter for
> > parquet, just like adapter for orc in cpp. How you guys think about it?
> > --
> > Liu, Renjie
> > Software Engineer, MVAD
> --
> Liu, Renjie
> Software Engineer, MVAD
>