You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Gene Pang <ge...@gmail.com> on 2024/09/10 04:32:41 UTC
[DISCUSS] Moving Variant to Parquet Details
Hi all,
The Spark community has agreed
<https://lists.apache.org/thread/pkybo148j6qyn2wsjnmyrhqs3crn9b89> to move
the Variant specification and implementation to the Parquet project.
However, there are several details we need to figure out with the move to
Parquet. I have started a document with some of the topics and details we
need to finalize.
https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit?usp=sharing
Please take a look at the document and leave comments, questions and
feedback to help reach a conclusion.
Thanks,
Gene
Re: [DISCUSS] Moving Variant to Parquet Details
Posted by Gene Pang <ge...@gmail.com>.
Hi all,
Thanks for the comments/suggestions and discussions about the details of
moving Variant to Parquet
<https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit?usp=sharing>
.
We have general alignment for the next steps for the plan.
- We will open a PR for the spec documents to the parquet-format repo
- The spec will include the disclaimer that this is still "WIP" or
"experimental", and we can iterate on the spec in parquet-format
- Eventually, after the necessary changes, we can vote in Parquet to remove
the "WIP" disclaimers
- The location and release process of the Java implementation of the binary
encoding is not fully decided yet, so we will continue to work towards that
effort.
Thanks,
Gene
On Mon, Sep 9, 2024 at 9:32 PM Gene Pang <ge...@gmail.com> wrote:
> Hi all,
>
> The Spark community has agreed
> <https://lists.apache.org/thread/pkybo148j6qyn2wsjnmyrhqs3crn9b89> to
> move the Variant specification and implementation to the Parquet project.
>
> However, there are several details we need to figure out with the move to
> Parquet. I have started a document with some of the topics and details we
> need to finalize.
>
>
> https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit?usp=sharing
>
> Please take a look at the document and leave comments, questions and
> feedback to help reach a conclusion.
>
> Thanks,
> Gene
>
Re: [DISCUSS] Moving Variant to Parquet Details
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
I'm thinking about some implementation issues, especially that well-known
obsession of mine: demonstrating the correctness of specifications
through machine readable formats such as TLA+, JUnit and scalatest (*)
1. the spec and at least some test suites should be closely linked, so
that all changes to the spec can trigger some regression testing. (
2. which lines up for requiring all subsequent work from including new
tests.
3. And ideally wiring up CI builds so that PRs can be fed all the way
through downstream uses (spark, iceberg, ...) so it becomes immediately
clear when something has been broken.
4. Finally, the notion of: who becomes a committer on this, or at least
promises to be active reviewers. that's organisational, not technical
steve
(*) Yes, I consider test suites to be non-normative formal specification
languages when these tests are directly derived from the specification.
What do people think the Hadoop FS API spec and tests are other than Z-lang
specs and an translation of them to Junit?)
On Tue, 10 Sept 2024 at 20:22, rdblue@gmail.com <rd...@gmail.com> wrote:
> To me, what matters the most is not really the repository, but the release
> process. Since the variant code is going to be fairly rapidly developed and
> may not have a stable API, I'd prefer to have it on a separate release
> cycle and start the versioning at 0.1.0 to avoid a misconception that the
> API is stable.
>
> Coming back to the repository decision, if we agree on separate releases
> then I think it is probably easier to use a separate repository as well.
> That way it is easier to find the code, manage dependencies (including
> thrift and protobuf), and we don't have to worry about unifying the build
> system.
>
> Ryan
>
> On Tue, Sep 10, 2024 at 8:45 AM Daniel Weeks <dw...@apache.org> wrote:
>
> > I feel like it's reasonable to put the specification in the
> > 'parquet-format' repo and reduce the confusion that would be caused by
> > having specs split across repos.
> >
> > As for the implementations, we already know there will be multiple and
> some
> > will be in languages where there is no current repo in the parquet
> > project. I agree with the proposed approach of a 'parquet-variant'
> project
> > where we keep all of the different language implementations. There are a
> > number of benefits including keeping implementations more consistent and
> > having a single place for reviewers/maintainers to focus their attention
> > while the initial donation/implementation progresses. It's easier to
> > split out an implementation if necessary than combine them and given the
> > relatively small size of this feature, it may never be an issue.
> >
> > Another thing to consider is that a lot of projects have custom
> > implementations of a parquet read/write path and requiring that they add
> a
> > dependency to parquet-java or arrow-rs to get variant support, for
> example,
> > feels like it would just cause more fragmentation across implementations
> as
> > they may choose to build their own. I feel like the fastest path to
> > general adoption is to keep the implementation separate so that we can
> rely
> > on reuse as much as possible.
> >
> > -Dan
> >
> >
> >
> > On Tue, Sep 10, 2024 at 2:29 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> > > From a Rust perspective, I think putting the spec in the parquet-format
> > > repo makes sense as it will become part of the parquet spec.
> > >
> > > In terms of what repository the rust variant implementation would live
> > in:
> > > * if there are parquet committers who plan to help implement and
> maintain
> > > it, then putting it in parquet-variant could make sense
> > > * if the idea is that the existing parquet-rs maintainers would help
> > > maintain it, putting it in the existing `arrow-rs` repo makes more
> sense
> > to
> > > me (this would likely also make initial development easier)
> > >
> > > Technically I would expect the rust implementation to be its own
> "crate"
> > > (equivalent of a library) that is released separately, that the parquet
> > > crate depended on but not the other way around.
> > >
> > > Hope that helps,
> > > Andrew
> > >
> > > On Tue, Sep 10, 2024 at 12:33 AM Gene Pang <ge...@gmail.com>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > The Spark community has agreed
> > > > <https://lists.apache.org/thread/pkybo148j6qyn2wsjnmyrhqs3crn9b89>
> to
> > > move
> > > > the Variant specification and implementation to the Parquet project.
> > > >
> > > > However, there are several details we need to figure out with the
> move
> > to
> > > > Parquet. I have started a document with some of the topics and
> details
> > we
> > > > need to finalize.
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit?usp=sharing
> > > >
> > > > Please take a look at the document and leave comments, questions and
> > > > feedback to help reach a conclusion.
> > > >
> > > > Thanks,
> > > > Gene
> > > >
> > >
> >
>
Re: [DISCUSS] Moving Variant to Parquet Details
Posted by "rdblue@gmail.com" <rd...@gmail.com>.
To me, what matters the most is not really the repository, but the release
process. Since the variant code is going to be fairly rapidly developed and
may not have a stable API, I'd prefer to have it on a separate release
cycle and start the versioning at 0.1.0 to avoid a misconception that the
API is stable.
Coming back to the repository decision, if we agree on separate releases
then I think it is probably easier to use a separate repository as well.
That way it is easier to find the code, manage dependencies (including
thrift and protobuf), and we don't have to worry about unifying the build
system.
Ryan
On Tue, Sep 10, 2024 at 8:45 AM Daniel Weeks <dw...@apache.org> wrote:
> I feel like it's reasonable to put the specification in the
> 'parquet-format' repo and reduce the confusion that would be caused by
> having specs split across repos.
>
> As for the implementations, we already know there will be multiple and some
> will be in languages where there is no current repo in the parquet
> project. I agree with the proposed approach of a 'parquet-variant' project
> where we keep all of the different language implementations. There are a
> number of benefits including keeping implementations more consistent and
> having a single place for reviewers/maintainers to focus their attention
> while the initial donation/implementation progresses. It's easier to
> split out an implementation if necessary than combine them and given the
> relatively small size of this feature, it may never be an issue.
>
> Another thing to consider is that a lot of projects have custom
> implementations of a parquet read/write path and requiring that they add a
> dependency to parquet-java or arrow-rs to get variant support, for example,
> feels like it would just cause more fragmentation across implementations as
> they may choose to build their own. I feel like the fastest path to
> general adoption is to keep the implementation separate so that we can rely
> on reuse as much as possible.
>
> -Dan
>
>
>
> On Tue, Sep 10, 2024 at 2:29 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > From a Rust perspective, I think putting the spec in the parquet-format
> > repo makes sense as it will become part of the parquet spec.
> >
> > In terms of what repository the rust variant implementation would live
> in:
> > * if there are parquet committers who plan to help implement and maintain
> > it, then putting it in parquet-variant could make sense
> > * if the idea is that the existing parquet-rs maintainers would help
> > maintain it, putting it in the existing `arrow-rs` repo makes more sense
> to
> > me (this would likely also make initial development easier)
> >
> > Technically I would expect the rust implementation to be its own "crate"
> > (equivalent of a library) that is released separately, that the parquet
> > crate depended on but not the other way around.
> >
> > Hope that helps,
> > Andrew
> >
> > On Tue, Sep 10, 2024 at 12:33 AM Gene Pang <ge...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > The Spark community has agreed
> > > <https://lists.apache.org/thread/pkybo148j6qyn2wsjnmyrhqs3crn9b89> to
> > move
> > > the Variant specification and implementation to the Parquet project.
> > >
> > > However, there are several details we need to figure out with the move
> to
> > > Parquet. I have started a document with some of the topics and details
> we
> > > need to finalize.
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit?usp=sharing
> > >
> > > Please take a look at the document and leave comments, questions and
> > > feedback to help reach a conclusion.
> > >
> > > Thanks,
> > > Gene
> > >
> >
>
Re: [DISCUSS] Moving Variant to Parquet Details
Posted by Daniel Weeks <dw...@apache.org>.
I feel like it's reasonable to put the specification in the
'parquet-format' repo and reduce the confusion that would be caused by
having specs split across repos.
As for the implementations, we already know there will be multiple and some
will be in languages where there is no current repo in the parquet
project. I agree with the proposed approach of a 'parquet-variant' project
where we keep all of the different language implementations. There are a
number of benefits including keeping implementations more consistent and
having a single place for reviewers/maintainers to focus their attention
while the initial donation/implementation progresses. It's easier to
split out an implementation if necessary than combine them and given the
relatively small size of this feature, it may never be an issue.
Another thing to consider is that a lot of projects have custom
implementations of a parquet read/write path and requiring that they add a
dependency to parquet-java or arrow-rs to get variant support, for example,
feels like it would just cause more fragmentation across implementations as
they may choose to build their own. I feel like the fastest path to
general adoption is to keep the implementation separate so that we can rely
on reuse as much as possible.
-Dan
On Tue, Sep 10, 2024 at 2:29 AM Andrew Lamb <an...@gmail.com> wrote:
> From a Rust perspective, I think putting the spec in the parquet-format
> repo makes sense as it will become part of the parquet spec.
>
> In terms of what repository the rust variant implementation would live in:
> * if there are parquet committers who plan to help implement and maintain
> it, then putting it in parquet-variant could make sense
> * if the idea is that the existing parquet-rs maintainers would help
> maintain it, putting it in the existing `arrow-rs` repo makes more sense to
> me (this would likely also make initial development easier)
>
> Technically I would expect the rust implementation to be its own "crate"
> (equivalent of a library) that is released separately, that the parquet
> crate depended on but not the other way around.
>
> Hope that helps,
> Andrew
>
> On Tue, Sep 10, 2024 at 12:33 AM Gene Pang <ge...@gmail.com> wrote:
>
> > Hi all,
> >
> > The Spark community has agreed
> > <https://lists.apache.org/thread/pkybo148j6qyn2wsjnmyrhqs3crn9b89> to
> move
> > the Variant specification and implementation to the Parquet project.
> >
> > However, there are several details we need to figure out with the move to
> > Parquet. I have started a document with some of the topics and details we
> > need to finalize.
> >
> >
> >
> https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit?usp=sharing
> >
> > Please take a look at the document and leave comments, questions and
> > feedback to help reach a conclusion.
> >
> > Thanks,
> > Gene
> >
>
Re: [DISCUSS] Moving Variant to Parquet Details
Posted by Andrew Lamb <an...@gmail.com>.
From a Rust perspective, I think putting the spec in the parquet-format
repo makes sense as it will become part of the parquet spec.
In terms of what repository the rust variant implementation would live in:
* if there are parquet committers who plan to help implement and maintain
it, then putting it in parquet-variant could make sense
* if the idea is that the existing parquet-rs maintainers would help
maintain it, putting it in the existing `arrow-rs` repo makes more sense to
me (this would likely also make initial development easier)
Technically I would expect the rust implementation to be its own "crate"
(equivalent of a library) that is released separately, that the parquet
crate depended on but not the other way around.
Hope that helps,
Andrew
On Tue, Sep 10, 2024 at 12:33 AM Gene Pang <ge...@gmail.com> wrote:
> Hi all,
>
> The Spark community has agreed
> <https://lists.apache.org/thread/pkybo148j6qyn2wsjnmyrhqs3crn9b89> to move
> the Variant specification and implementation to the Parquet project.
>
> However, there are several details we need to figure out with the move to
> Parquet. I have started a document with some of the topics and details we
> need to finalize.
>
>
> https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit?usp=sharing
>
> Please take a look at the document and leave comments, questions and
> feedback to help reach a conclusion.
>
> Thanks,
> Gene
>