You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "Kontinuation (via GitHub)" <gi...@apache.org> on 2025/02/24 17:03:53 UTC
[PR] [WIP] Provide geospatial support for Iceberg Spark [sedona]
Kontinuation opened a new pull request, #1830:
URL: https://github.com/apache/sedona/pull/1830
This PR depends on https://github.com/apache/sedona/pull/1828
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/)
- No, I haven't read it.
## Is this PR related to a JIRA ticket?
- Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-XXX. The PR name follows the format `[SEDONA-XXX] my subject`.
- No:
- this is a documentation update. The PR name follows the format `[DOCS] my subject`
- this is a CI update. The PR name follows the format `[CI] my subject`
## What changes were proposed in this PR?
GEOMETRY and GEOGRAPHY types has been accepted as part of Iceberg V3 spec. We are actively working on implementing geospatial support for the apache/iceberg project. This repository demonstrates how we'll integrate Sedona with Iceberg Spark to work with geometry and geography data. The overall idea is:
1. We'll define an SPI interface `GeospatialLibraryProvider` in apache/iceberg to allow third party libraries to integrate their geospatial support into iceberg-spark.
2. Sedona implements this `GeospatialLibraryProvider` interface in sedona-spark-iceberg module.
3. Iceberg will work with Sedona when iceberg-spark-runtime, sedona-spark-shaded and sedona-spark-iceberg are all installed into the Spark environment.
This PR is still in a proof-of-concept stage. It relies on several unreleased dependencies to work:
* parquet-format: for the new GEOMETRY and GEOGRAPHY logical types
* parquet-java: for the implementation of the parquet-geo standard
* iceberg: proof-of-concept implementation of the geo spec
## How was this patch tested?
Tested locally
## Did this PR include necessary documentation updates?
- Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/99239524f17389fc4ae9548ea88756f8ea538bb9/pom.xml#L29) in `vX.Y.Z` format.
- Yes, I have updated the documentation.
- No, this PR does not affect any public API so no need to change the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Provide geospatial support for Iceberg Spark [sedona]
Posted by "freamdx (via GitHub)" <gi...@apache.org>.
freamdx commented on PR #1830:
URL: https://github.com/apache/sedona/pull/1830#issuecomment-2817531515
> This PR depends on #1828
>
> ## Did you read the Contributor Guide?
> * Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/)
> * No, I haven't read it.
>
> ## Is this PR related to a JIRA ticket?
> * Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-XXX. The PR name follows the format `[SEDONA-XXX] my subject`.
> * No:
>
> * this is a documentation update. The PR name follows the format `[DOCS] my subject`
> * this is a CI update. The PR name follows the format `[CI] my subject`
>
> ## What changes were proposed in this PR?
> GEOMETRY and GEOGRAPHY types has been accepted as part of Iceberg V3 spec. We are actively working on implementing geospatial support for the apache/iceberg project. This repository demonstrates how we'll integrate Sedona with Iceberg Spark to work with geometry and geography data. The overall idea is:
>
> 1. We'll define an SPI interface `GeospatialLibraryProvider` in apache/iceberg to allow third party libraries to integrate their geospatial support into iceberg-spark.
> 2. Sedona implements this `GeospatialLibraryProvider` interface in [sedona-spark-iceberg](https://issues.apache.org/jira/browse/SEDONA-spark-iceberg) module.
> 3. Iceberg will work with Sedona when iceberg-spark-runtime, [sedona-spark-shaded](https://issues.apache.org/jira/browse/SEDONA-spark-shaded) and [sedona-spark-iceberg](https://issues.apache.org/jira/browse/SEDONA-spark-iceberg) are all installed into the Spark environment.
>
> This PR is still in a proof-of-concept stage. It relies on several unreleased dependencies to work:
>
> * parquet-format: for the new GEOMETRY and GEOGRAPHY logical types
> * parquet-java: for the implementation of the parquet-geo standard
> * iceberg: proof-of-concept implementation of the geo spec
>
> ## How was this patch tested?
> Tested locally
>
> ## Did this PR include necessary documentation updates?
> * Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/99239524f17389fc4ae9548ea88756f8ea538bb9/pom.xml#L29) in `vX.Y.Z` format.
> * Yes, I have updated the documentation.
> * No, this PR does not affect any public API so no need to change the documentation.
great!
my simple and invasive patch，https://github.com/freamdx/iceberg/commit/929dfae730d41516c77adf6801da99a01e410810 , i hope it will help you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org