You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Matthew Powers <ma...@gmail.com> on 2025/04/23 20:30:46 UTC
Discuss: add geospatial types to Avro
Hello Avro community. Thank you for the wonderful work on this project.
Iceberg recently added geometry/geography types
<https://github.com/apache/iceberg/pull/10981> to support geospatial
workflows and I was thinking Avro users could benefit from these types as
well.
I don't have the Java or Rust coding skills to implement this myself, but I
would be happy to help with the spec and cheerlead the feature to get
users! Thank you for considering this proposal.
Re: Discuss: add geospatial types to Avro
Posted by Matthew Powers <ma...@gmail.com>.
Thanks for the response Oscar. Your understanding is correct at a high
level.
I think the Parquet geo spec can be almost directly borrowed to Avro,
here's the Parquet spec:
https://github.com/apache/parquet-format/blob/3ce0760933b875bc8a11f5be0b883cd107b95b43/Geospatial.md
I think this change would make Avro a great file format for a bunch of
geospatial data analyses, and I appreciate your consideration of the
proposal!
On Sun, Apr 27, 2025 at 3:04 AM Martin Grigorov <mg...@apache.org>
wrote:
> CC-ing Matthew
>
> On Fri, Apr 25, 2025 at 1:59 PM Oscar Westra van Holthe - Kind <
> oscar@westravanholthe.nl> wrote:
>
>> Hi,
>>
>> From what I see on the PR you mentioned, is that this would mean (in Avro
>> terms) logical types for geometry & geography (both coordinates)? Do I
>> understand this correctly?
>>
>> Additional things, such as lines, polygons and expressions, fall outside
>> our scope of data representation.
>>
>> Both the geometry and geography types are be possible using a logical type
>> on a record with 4 double fields (x, y, z & m), where the first two should
>> not have a NaN value.
>>
>> Kind regards,
>> Oscar
>>
>>
>> On wed 23 apr 2025 22:30, Matthew Powers <ma...@gmail.com>
>> wrote:
>>
>> > Hello Avro community. Thank you for the wonderful work on this project.
>> >
>> > Iceberg recently added geometry/geography types
>> > <https://github.com/apache/iceberg/pull/10981> to support geospatial
>> > workflows and I was thinking Avro users could benefit from these types
>> as
>> > well.
>> >
>> > I don't have the Java or Rust coding skills to implement this myself,
>> but I
>> > would be happy to help with the spec and cheerlead the feature to get
>> > users! Thank you for considering this proposal.
>> >
>>
>
Re: Discuss: add geospatial types to Avro
Posted by Martin Grigorov <mg...@apache.org>.
CC-ing Matthew
On Fri, Apr 25, 2025 at 1:59 PM Oscar Westra van Holthe - Kind <
oscar@westravanholthe.nl> wrote:
> Hi,
>
> From what I see on the PR you mentioned, is that this would mean (in Avro
> terms) logical types for geometry & geography (both coordinates)? Do I
> understand this correctly?
>
> Additional things, such as lines, polygons and expressions, fall outside
> our scope of data representation.
>
> Both the geometry and geography types are be possible using a logical type
> on a record with 4 double fields (x, y, z & m), where the first two should
> not have a NaN value.
>
> Kind regards,
> Oscar
>
>
> On wed 23 apr 2025 22:30, Matthew Powers <ma...@gmail.com>
> wrote:
>
> > Hello Avro community. Thank you for the wonderful work on this project.
> >
> > Iceberg recently added geometry/geography types
> > <https://github.com/apache/iceberg/pull/10981> to support geospatial
> > workflows and I was thinking Avro users could benefit from these types as
> > well.
> >
> > I don't have the Java or Rust coding skills to implement this myself,
> but I
> > would be happy to help with the spec and cheerlead the feature to get
> > users! Thank you for considering this proposal.
> >
>
Re: Discuss: add geospatial types to Avro
Posted by Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>.
Hi,
From what I see on the PR you mentioned, is that this would mean (in Avro
terms) logical types for geometry & geography (both coordinates)? Do I
understand this correctly?
Additional things, such as lines, polygons and expressions, fall outside
our scope of data representation.
Both the geometry and geography types are be possible using a logical type
on a record with 4 double fields (x, y, z & m), where the first two should
not have a NaN value.
Kind regards,
Oscar
On wed 23 apr 2025 22:30, Matthew Powers <ma...@gmail.com>
wrote:
> Hello Avro community. Thank you for the wonderful work on this project.
>
> Iceberg recently added geometry/geography types
> <https://github.com/apache/iceberg/pull/10981> to support geospatial
> workflows and I was thinking Avro users could benefit from these types as
> well.
>
> I don't have the Java or Rust coding skills to implement this myself, but I
> would be happy to help with the spec and cheerlead the feature to get
> users! Thank you for considering this proposal.
>