You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Neville Dipale <ne...@gmail.com> on 2021/04/27 15:54:36 UTC
[DISCUSS] Moving the format directory to arrow-format repository
Hi Arrow devs,
Andy noticed that we carry a copy of the format directory in arrow-rs,
which
is bound to get outdated in the future.
We would like to propose creating an arrow-format repository, similar to
parquet-format, so that arrow-rs and other future separate repositories
could
add this as a submodule.
What are your thoughts?
Regards
Neville
Re: [DISCUSS] Moving the format directory to arrow-format repository
Posted by Neville Dipale <ne...@gmail.com>.
Hi,
Thanks for the feedback, in light of what's been said, I'm also now fine
with
leaving the format as is.
Changes to the format are visible enough that we shouldn't miss them, as
there's normally be a discussion in the ML.
Regards
Neville
On Wed, 28 Apr 2021 at 17:31, Jorge Cardoso Leitão <jo...@gmail.com>
wrote:
> Hi,
>
> imo the time-scale of changes in the format is too large to justify the
> complexity. I also think that we should not force users to clone or
> submodule the repo to even compile the crate.
>
> What if we just do not have the format files there at all, and instead just
> keep the generated code? Updates to the format are manually performed (i.e.
> instead of copying the file, we run the build against latest). This is what
> we already do in practice, anyways: whenever the format changed, tonic
> would auto-generate new rust code that we would commit before any change.
>
> This avoids having copies of files between repos, which imo confuses people
> into which one is the official one (even when they are all equal, people
> will not do diffs in their heads).
>
> Best,
> Jorge
>
>
>
> On Wed, Apr 28, 2021 at 12:13 PM Andrew Lamb <al...@influxdata.com> wrote:
>
> > I also think manually copying the format .fbs files to arrow-rs is
> probably
> > ok for the time being.
> >
> > Once Arrow gets to the point where many implementations that need
> > format.fbs live in many different repos, pulling out the format files
> into
> > their own repo might be worth reconsidering.
> >
> > Andrew
> >
> > On Tue, Apr 27, 2021 at 5:45 PM Wes McKinney <we...@gmail.com>
> wrote:
> >
> > > I wouldn't be too excited about this. Here are my thoughts:
> > >
> > > 1. Having the format/ directory in apache/arrow be a submodule would be
> > > cumbersome and error-prone for developers. The only submodules we have
> > > right now are optional testing dependencies — not having these
> > initialized
> > > and updated does not result in a broken project, whereas this change
> > would.
> > > We have a copy of parquet.thrift from apache/parquet-format for similar
> > > reasons.
> > >
> > > 2. So based on #1, we would want to maintain a copy of the format files
> > in
> > > apache/arrow, even if there were a separate apache/arrow-format
> > repository.
> > > The format files are slow-moving enough that I don't think it's
> > burdensome
> > > to mirror these into satellite repositories like arrow/arrow-rs.
> > >
> > > On Tue, Apr 27, 2021 at 10:54 AM Neville Dipale <nevilledips@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Arrow devs,
> > > >
> > > > Andy noticed that we carry a copy of the format directory in
> arrow-rs,
> > > > which
> > > > is bound to get outdated in the future.
> > > >
> > > > We would like to propose creating an arrow-format repository, similar
> > to
> > > > parquet-format, so that arrow-rs and other future separate
> repositories
> > > > could
> > > > add this as a submodule.
> > > >
> > > > What are your thoughts?
> > > >
> > > > Regards
> > > > Neville
> > > >
> > >
> >
>
Re: [DISCUSS] Moving the format directory to arrow-format repository
Posted by Jorge Cardoso Leitão <jo...@gmail.com>.
Hi,
imo the time-scale of changes in the format is too large to justify the
complexity. I also think that we should not force users to clone or
submodule the repo to even compile the crate.
What if we just do not have the format files there at all, and instead just
keep the generated code? Updates to the format are manually performed (i.e.
instead of copying the file, we run the build against latest). This is what
we already do in practice, anyways: whenever the format changed, tonic
would auto-generate new rust code that we would commit before any change.
This avoids having copies of files between repos, which imo confuses people
into which one is the official one (even when they are all equal, people
will not do diffs in their heads).
Best,
Jorge
On Wed, Apr 28, 2021 at 12:13 PM Andrew Lamb <al...@influxdata.com> wrote:
> I also think manually copying the format .fbs files to arrow-rs is probably
> ok for the time being.
>
> Once Arrow gets to the point where many implementations that need
> format.fbs live in many different repos, pulling out the format files into
> their own repo might be worth reconsidering.
>
> Andrew
>
> On Tue, Apr 27, 2021 at 5:45 PM Wes McKinney <we...@gmail.com> wrote:
>
> > I wouldn't be too excited about this. Here are my thoughts:
> >
> > 1. Having the format/ directory in apache/arrow be a submodule would be
> > cumbersome and error-prone for developers. The only submodules we have
> > right now are optional testing dependencies — not having these
> initialized
> > and updated does not result in a broken project, whereas this change
> would.
> > We have a copy of parquet.thrift from apache/parquet-format for similar
> > reasons.
> >
> > 2. So based on #1, we would want to maintain a copy of the format files
> in
> > apache/arrow, even if there were a separate apache/arrow-format
> repository.
> > The format files are slow-moving enough that I don't think it's
> burdensome
> > to mirror these into satellite repositories like arrow/arrow-rs.
> >
> > On Tue, Apr 27, 2021 at 10:54 AM Neville Dipale <ne...@gmail.com>
> > wrote:
> >
> > > Hi Arrow devs,
> > >
> > > Andy noticed that we carry a copy of the format directory in arrow-rs,
> > > which
> > > is bound to get outdated in the future.
> > >
> > > We would like to propose creating an arrow-format repository, similar
> to
> > > parquet-format, so that arrow-rs and other future separate repositories
> > > could
> > > add this as a submodule.
> > >
> > > What are your thoughts?
> > >
> > > Regards
> > > Neville
> > >
> >
>
Re: [DISCUSS] Moving the format directory to arrow-format repository
Posted by Andrew Lamb <al...@influxdata.com>.
I also think manually copying the format .fbs files to arrow-rs is probably
ok for the time being.
Once Arrow gets to the point where many implementations that need
format.fbs live in many different repos, pulling out the format files into
their own repo might be worth reconsidering.
Andrew
On Tue, Apr 27, 2021 at 5:45 PM Wes McKinney <we...@gmail.com> wrote:
> I wouldn't be too excited about this. Here are my thoughts:
>
> 1. Having the format/ directory in apache/arrow be a submodule would be
> cumbersome and error-prone for developers. The only submodules we have
> right now are optional testing dependencies — not having these initialized
> and updated does not result in a broken project, whereas this change would.
> We have a copy of parquet.thrift from apache/parquet-format for similar
> reasons.
>
> 2. So based on #1, we would want to maintain a copy of the format files in
> apache/arrow, even if there were a separate apache/arrow-format repository.
> The format files are slow-moving enough that I don't think it's burdensome
> to mirror these into satellite repositories like arrow/arrow-rs.
>
> On Tue, Apr 27, 2021 at 10:54 AM Neville Dipale <ne...@gmail.com>
> wrote:
>
> > Hi Arrow devs,
> >
> > Andy noticed that we carry a copy of the format directory in arrow-rs,
> > which
> > is bound to get outdated in the future.
> >
> > We would like to propose creating an arrow-format repository, similar to
> > parquet-format, so that arrow-rs and other future separate repositories
> > could
> > add this as a submodule.
> >
> > What are your thoughts?
> >
> > Regards
> > Neville
> >
>
Re: [DISCUSS] Moving the format directory to arrow-format repository
Posted by Wes McKinney <we...@gmail.com>.
I wouldn't be too excited about this. Here are my thoughts:
1. Having the format/ directory in apache/arrow be a submodule would be
cumbersome and error-prone for developers. The only submodules we have
right now are optional testing dependencies — not having these initialized
and updated does not result in a broken project, whereas this change would.
We have a copy of parquet.thrift from apache/parquet-format for similar
reasons.
2. So based on #1, we would want to maintain a copy of the format files in
apache/arrow, even if there were a separate apache/arrow-format repository.
The format files are slow-moving enough that I don't think it's burdensome
to mirror these into satellite repositories like arrow/arrow-rs.
On Tue, Apr 27, 2021 at 10:54 AM Neville Dipale <ne...@gmail.com>
wrote:
> Hi Arrow devs,
>
> Andy noticed that we carry a copy of the format directory in arrow-rs,
> which
> is bound to get outdated in the future.
>
> We would like to propose creating an arrow-format repository, similar to
> parquet-format, so that arrow-rs and other future separate repositories
> could
> add this as a submodule.
>
> What are your thoughts?
>
> Regards
> Neville
>