You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by Gang Wu <us...@gmail.com> on 2024/09/07 16:30:13 UTC

Re: [DISCUSS] Add geometry logical type

Update:

We are working with Apache Iceberg, Apache Sedona, and GeoParquet
communities to finalize the spec addition of Geometry type [1]. Two PoC
implementations [2][3] are on the way with the help from the Apache Sedona
community.

Now the spec change has been stable and the implementations are close to
the finish line. In the meantime, Apache Iceberg is waiting for the Parquet
side before adding the geometry support to its v3 spec. I want to encourage
everyone to take a review on the spec and the PoCs. A vote thread will
be sent to finalize the process.

[1] https://github.com/apache/parquet-format/pull/240
[2] https://github.com/apache/parquet-java/pull/2971
[3] https://github.com/apache/arrow/pull/43977

Thanks,
Gang

On Sun, May 12, 2024 at 4:14 PM Gang Wu <us...@gmail.com> wrote:

> Hi,
>
> Apache Iceberg community is proposing to add geospatial support [1]. It
> would
> be good if Apache Parquet can support native geometry type to implement
> more
> efficient encoding, statistics and filtering. Therefore, I'd like to
> propose a format
> change to add a new geometry logical type: [2]. It is still in a very
> early stage and
> I'm working with Apache Iceberg community and GeoParquet community to
> polish
> the design. Hopefully this design will be a good starting point to support
> geospatial
> features from the file format level and be interoperable to more table
> formats and
> popular query engines in the future. Any feedback is welcomed!
>
> [1]
> https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI
> [2] https://github.com/apache/parquet-format/pull/240
>
> Best,
> Gang
>
>

Re: [DISCUSS] Add geometry logical type

Posted by Russell Spitzer <ru...@gmail.com>.

Got it, ok i'm 100% on board then

On Fri, Dec 6, 2024 at 2:47 PM Jia Yu <ji...@apache.org> wrote:

> Hi Russel,
>
> No, min/max wraparound will be allowed in both Geometry and Geography.
>
> The main challenging of the Geography type and why we are isolating it
> from the Geometry type is that the Geography type needs to define how
> engines interpolate curves to edges, and subsequently how to calculate the
> spatial relationship between non-linear edges.
>
> Right now, the behaviors vary among engines and no standard exists in the
> Geo community. So the Geography type discussion is going to take some time.
> And, we can get the geometry type in first.
>
> Thanks,
> Jia
>
> On 2024/12/05 16:32:52 Russell Spitzer wrote:
> > I probably need to be taught more about this, but isn't the only
> difference
> > from the Iceberg perspective whether or not to wrap min max values? I
> > thought the actual projection encoding was being saved in table
> properties
> > in either case?
> >
> > On Wed, Dec 4, 2024 at 8:18 PM Gang Wu <us...@gmail.com> wrote:
> >
> > > Update:
> > >
> > > The Iceberg community has discussed and agreed on splitting geometry
> and
> > > geography types and focusing on the geometry type [1] at this point. We
> > > have also modified the Parquet PR [2] to reflect this. Please take a
> look
> > > and let me know what you think. If there is no objection, we will
> start a
> > > formal vote to accept the new geometry logical type and release it in
> the
> > > parquet-format version 2.11.0.
> > >
> > > [1] https://github.com/apache/iceberg/pull/10981
> > > [2] https://github.com/apache/parquet-format/pull/240
> > >
> > > Best,
> > > Gang
> > >
> > > On Sun, Sep 8, 2024 at 12:30 AM Gang Wu <us...@gmail.com> wrote:
> > >
> > > > Update:
> > > >
> > > > We are working with Apache Iceberg, Apache Sedona, and GeoParquet
> > > > communities to finalize the spec addition of Geometry type [1]. Two
> PoC
> > > > implementations [2][3] are on the way with the help from the Apache
> > > Sedona
> > > > community.
> > > >
> > > > Now the spec change has been stable and the implementations are
> close to
> > > > the finish line. In the meantime, Apache Iceberg is waiting for the
> > > Parquet
> > > > side before adding the geometry support to its v3 spec. I want to
> > > encourage
> > > > everyone to take a review on the spec and the PoCs. A vote thread
> will
> > > > be sent to finalize the process.
> > > >
> > > > [1] https://github.com/apache/parquet-format/pull/240
> > > > [2] https://github.com/apache/parquet-java/pull/2971
> > > > [3] https://github.com/apache/arrow/pull/43977
> > > >
> > > > Thanks,
> > > > Gang
> > > >
> > > > On Sun, May 12, 2024 at 4:14 PM Gang Wu <us...@gmail.com> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Apache Iceberg community is proposing to add geospatial support
> [1]. It
> > > >> would
> > > >> be good if Apache Parquet can support native geometry type to
> implement
> > > >> more
> > > >> efficient encoding, statistics and filtering. Therefore, I'd like to
> > > >> propose a format
> > > >> change to add a new geometry logical type: [2]. It is still in a
> very
> > > >> early stage and
> > > >> I'm working with Apache Iceberg community and GeoParquet community
> to
> > > >> polish
> > > >> the design. Hopefully this design will be a good starting point to
> > > >> support geospatial
> > > >> features from the file format level and be interoperable to more
> table
> > > >> formats and
> > > >> popular query engines in the future. Any feedback is welcomed!
> > > >>
> > > >> [1]
> > > >>
> > >
> https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI
> > > >> [2] https://github.com/apache/parquet-format/pull/240
> > > >>
> > > >> Best,
> > > >> Gang
> > > >>
> > > >>
> > >
> >
>

Re: [DISCUSS] Add geometry logical type

Posted by Jia Yu <ji...@apache.org>.

Hi Russel,

No, min/max wraparound will be allowed in both Geometry and Geography.

The main challenging of the Geography type and why we are isolating it from the Geometry type is that the Geography type needs to define how engines interpolate curves to edges, and subsequently how to calculate the spatial relationship between non-linear edges.

Right now, the behaviors vary among engines and no standard exists in the Geo community. So the Geography type discussion is going to take some time. And, we can get the geometry type in first.

Thanks,
Jia

On 2024/12/05 16:32:52 Russell Spitzer wrote:
> I probably need to be taught more about this, but isn't the only difference
> from the Iceberg perspective whether or not to wrap min max values? I
> thought the actual projection encoding was being saved in table properties
> in either case?
> 
> On Wed, Dec 4, 2024 at 8:18 PM Gang Wu <us...@gmail.com> wrote:
> 
> > Update:
> >
> > The Iceberg community has discussed and agreed on splitting geometry and
> > geography types and focusing on the geometry type [1] at this point. We
> > have also modified the Parquet PR [2] to reflect this. Please take a look
> > and let me know what you think. If there is no objection, we will start a
> > formal vote to accept the new geometry logical type and release it in the
> > parquet-format version 2.11.0.
> >
> > [1] https://github.com/apache/iceberg/pull/10981
> > [2] https://github.com/apache/parquet-format/pull/240
> >
> > Best,
> > Gang
> >
> > On Sun, Sep 8, 2024 at 12:30 AM Gang Wu <us...@gmail.com> wrote:
> >
> > > Update:
> > >
> > > We are working with Apache Iceberg, Apache Sedona, and GeoParquet
> > > communities to finalize the spec addition of Geometry type [1]. Two PoC
> > > implementations [2][3] are on the way with the help from the Apache
> > Sedona
> > > community.
> > >
> > > Now the spec change has been stable and the implementations are close to
> > > the finish line. In the meantime, Apache Iceberg is waiting for the
> > Parquet
> > > side before adding the geometry support to its v3 spec. I want to
> > encourage
> > > everyone to take a review on the spec and the PoCs. A vote thread will
> > > be sent to finalize the process.
> > >
> > > [1] https://github.com/apache/parquet-format/pull/240
> > > [2] https://github.com/apache/parquet-java/pull/2971
> > > [3] https://github.com/apache/arrow/pull/43977
> > >
> > > Thanks,
> > > Gang
> > >
> > > On Sun, May 12, 2024 at 4:14 PM Gang Wu <us...@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> Apache Iceberg community is proposing to add geospatial support [1]. It
> > >> would
> > >> be good if Apache Parquet can support native geometry type to implement
> > >> more
> > >> efficient encoding, statistics and filtering. Therefore, I'd like to
> > >> propose a format
> > >> change to add a new geometry logical type: [2]. It is still in a very
> > >> early stage and
> > >> I'm working with Apache Iceberg community and GeoParquet community to
> > >> polish
> > >> the design. Hopefully this design will be a good starting point to
> > >> support geospatial
> > >> features from the file format level and be interoperable to more table
> > >> formats and
> > >> popular query engines in the future. Any feedback is welcomed!
> > >>
> > >> [1]
> > >>
> > https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI
> > >> [2] https://github.com/apache/parquet-format/pull/240
> > >>
> > >> Best,
> > >> Gang
> > >>
> > >>
> >
>

Re: [DISCUSS] Add geometry logical type

Posted by Russell Spitzer <ru...@gmail.com>.

I probably need to be taught more about this, but isn't the only difference
from the Iceberg perspective whether or not to wrap min max values? I
thought the actual projection encoding was being saved in table properties
in either case?

On Wed, Dec 4, 2024 at 8:18 PM Gang Wu <us...@gmail.com> wrote:

> Update:
>
> The Iceberg community has discussed and agreed on splitting geometry and
> geography types and focusing on the geometry type [1] at this point. We
> have also modified the Parquet PR [2] to reflect this. Please take a look
> and let me know what you think. If there is no objection, we will start a
> formal vote to accept the new geometry logical type and release it in the
> parquet-format version 2.11.0.
>
> [1] https://github.com/apache/iceberg/pull/10981
> [2] https://github.com/apache/parquet-format/pull/240
>
> Best,
> Gang
>
> On Sun, Sep 8, 2024 at 12:30 AM Gang Wu <us...@gmail.com> wrote:
>
> > Update:
> >
> > We are working with Apache Iceberg, Apache Sedona, and GeoParquet
> > communities to finalize the spec addition of Geometry type [1]. Two PoC
> > implementations [2][3] are on the way with the help from the Apache
> Sedona
> > community.
> >
> > Now the spec change has been stable and the implementations are close to
> > the finish line. In the meantime, Apache Iceberg is waiting for the
> Parquet
> > side before adding the geometry support to its v3 spec. I want to
> encourage
> > everyone to take a review on the spec and the PoCs. A vote thread will
> > be sent to finalize the process.
> >
> > [1] https://github.com/apache/parquet-format/pull/240
> > [2] https://github.com/apache/parquet-java/pull/2971
> > [3] https://github.com/apache/arrow/pull/43977
> >
> > Thanks,
> > Gang
> >
> > On Sun, May 12, 2024 at 4:14 PM Gang Wu <us...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> Apache Iceberg community is proposing to add geospatial support [1]. It
> >> would
> >> be good if Apache Parquet can support native geometry type to implement
> >> more
> >> efficient encoding, statistics and filtering. Therefore, I'd like to
> >> propose a format
> >> change to add a new geometry logical type: [2]. It is still in a very
> >> early stage and
> >> I'm working with Apache Iceberg community and GeoParquet community to
> >> polish
> >> the design. Hopefully this design will be a good starting point to
> >> support geospatial
> >> features from the file format level and be interoperable to more table
> >> formats and
> >> popular query engines in the future. Any feedback is welcomed!
> >>
> >> [1]
> >>
> https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI
> >> [2] https://github.com/apache/parquet-format/pull/240
> >>
> >> Best,
> >> Gang
> >>
> >>
>

Re: [DISCUSS] Add geometry logical type

Posted by Gang Wu <us...@gmail.com>.

Update:

The Iceberg community has discussed and agreed on splitting geometry and
geography types and focusing on the geometry type [1] at this point. We
have also modified the Parquet PR [2] to reflect this. Please take a look
and let me know what you think. If there is no objection, we will start a
formal vote to accept the new geometry logical type and release it in the
parquet-format version 2.11.0.

[1] https://github.com/apache/iceberg/pull/10981
[2] https://github.com/apache/parquet-format/pull/240

Best,
Gang

On Sun, Sep 8, 2024 at 12:30 AM Gang Wu <us...@gmail.com> wrote:

> Update:
>
> We are working with Apache Iceberg, Apache Sedona, and GeoParquet
> communities to finalize the spec addition of Geometry type [1]. Two PoC
> implementations [2][3] are on the way with the help from the Apache Sedona
> community.
>
> Now the spec change has been stable and the implementations are close to
> the finish line. In the meantime, Apache Iceberg is waiting for the Parquet
> side before adding the geometry support to its v3 spec. I want to encourage
> everyone to take a review on the spec and the PoCs. A vote thread will
> be sent to finalize the process.
>
> [1] https://github.com/apache/parquet-format/pull/240
> [2] https://github.com/apache/parquet-java/pull/2971
> [3] https://github.com/apache/arrow/pull/43977
>
> Thanks,
> Gang
>
> On Sun, May 12, 2024 at 4:14 PM Gang Wu <us...@gmail.com> wrote:
>
>> Hi,
>>
>> Apache Iceberg community is proposing to add geospatial support [1]. It
>> would
>> be good if Apache Parquet can support native geometry type to implement
>> more
>> efficient encoding, statistics and filtering. Therefore, I'd like to
>> propose a format
>> change to add a new geometry logical type: [2]. It is still in a very
>> early stage and
>> I'm working with Apache Iceberg community and GeoParquet community to
>> polish
>> the design. Hopefully this design will be a good starting point to
>> support geospatial
>> features from the file format level and be interoperable to more table
>> formats and
>> popular query engines in the future. Any feedback is welcomed!
>>
>> [1]
>> https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI
>> [2] https://github.com/apache/parquet-format/pull/240
>>
>> Best,
>> Gang
>>
>>