You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Menelaos Karavelas <me...@gmail.com> on 2025/03/28 17:46:15 UTC
[DISCUSS] SPIP: Add geospatial types to Spark
Dear Spark community,
I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
SPIP: https://issues.apache.org/jira/browse/SPARK-51658
Looking forward to your comments and feedback.
Best regards,
Menelaos Karavelas
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Menelaos Karavelas <me...@gmail.com>.
Hello Jia,
Wenchen summarized the intent very clearly. The scope of the proposal is primarily the type system and storage, not processing. Let’s work together on the technical details and make sure the work we propose to do in Spark works best with Apache Sedona.
Best,
Menelaos
> On Mar 29, 2025, at 5:23 AM, Wenchen Fan <cl...@gmail.com> wrote:
>
> Hi Jia,
>
> This is a good question. As the shepherd of this SPIP, I'd like to clarify the motivation here: the focus of this project is more about the storage part, not the processing. Apache Sedona is a great library for geo processing, but without native geo type support in Spark, users can't do the following things:
> - read the geo type columns from Parquet files (or other data sources) directly
> - write geo values into Parquet files (or other data sources) as native geo types.
> - push down geo predicates to the data source when reading
>
> In the SPIP JIRA, we explicitly mentioned that "This proposal is laying the foundation - building the infrastructure to handle geospatial data, but not creating a full-featured geospatial processing system. Such extension can be done later as a separate improvement." Maybe the right direction is to not do it and leave it to third-party libraries.
>
> The ultimate goal is to establish Spark as a comprehensive platform that can connect to a rich ecosystem of third-party data sources and processing libraries. For this project, we should definitely work with the Apache Sedona community closely, to figure out the best protocol (what binary/text format to use? how to represent geo values in Java? etc.)
>
> Thanks,
> Wenchen
>
> On Sat, Mar 29, 2025 at 5:28 AM Jia Yu <jiayu@apache.org <ma...@apache.org>> wrote:
>> Dear Menelaos,
>>
>> Thanks for bringing this up again. I’ve seen similar proposals come up on the mailing list before, and I’d like to offer some thoughts.
>>
>> For full transparency, I’m Jia Yu, PMC Chair of Apache Sedona (https://github.com/apache/sedona), a widely used open-source cluster computing framework for processing large-scale geospatial data on Spark, Flink, and other engines.
>>
>> From what I understand, this proposal aims to add native geospatial types and functionality directly into Spark. However, this seems to replicate much of the work already done by the Sedona project over the past 10 years.
>>
>> Sedona has a strong and active community with well-established contribution guidelines. It is already used extensively with Spark in production—on platforms like Databricks, AWS EMR, Microsoft Fabric, and Google Cloud. Users simply add the Sedona jar, flip a Spark config, and it just works—similar to other mature Spark ecosystem libraries.
>>
>> The project sees over 2 million downloads per month across PyPI, Maven, etc., and has been downloaded more than 45 million times overall. Thousands of organizations rely on Sedona in production Spark environments.
>>
>> Sedona has also actively contributed to upstream ecosystem efforts, such as geospatial support in Parquet and Iceberg formats.
>>
>> Additionally, Sedona’s core technology has been peer-reviewed and published at top academic conferences. Its performance has been evaluated and benchmarked by many independent research articles: https://sedona.apache.org/latest/community/publication/
>>
>> Given all of this, I’m genuinely unsure what gap a new Spark-native effort is aiming to fill. If there’s a specific limitation that Sedona cannot address, I’d be eager to understand it. Otherwise, duplicating this functionality risks fragmenting the ecosystem and introducing confusion for current users. I would strongly advocate for close coordination with the Sedona community to avoid disruption and ensure alignment with the broader Spark ecosystem.
>>
>> Thanks again for raising this—we’re always happy to collaborate and strengthen the ecosystem together.
>>
>>
>> Here is a quick overview of what Apache Sedona already offers on Spark:
>> • Geospatial type support:
>> • Vector: Geometry, partial Geography
>> • Raster
>> • Vector data sources:
>> • GeoParquet (read/write), GeoJSON (read/write), Shapefile, GeoPackage, OpenStreetMap PBF
>> • Raster data sources:
>> • STAC catalog, GeoTiff (read/write), NetCDF/HDF
>> • Functions:
>> • 209+ vector (ST_*) functions
>> • 100+ raster (RS_*) functions
>> • GeoStats SQL: DBSCAN, hotspot analysis, outlier detection
>> • Language support:
>> • Scala, Java, SQL, Python, R
>> • Query acceleration via R-Tree:
>> • Distributed and broadcast spatial joins
>> • KNN joins
>> • Range queries
>> • UDF support:
>> • Scala UDFs (JTS), Python UDFs (Shapely, Rasterio, NumPy), Pandas UDFs
>> • Serialization:
>> • Custom serializers for geometry types
>> • Ecosystem integrations:
>> • Jupyter, Zeppelin, Apache Arrow, GeoPandas read/write, GeoPandas-like API
>>
>> Jia Yu
>>
>>
>> On 2025/03/28 17:46:15 Menelaos Karavelas wrote:
>> > Dear Spark community,
>> >
>> > I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
>> >
>> > The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
>> >
>> > SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>> >
>> > Looking forward to your comments and feedback.
>> >
>> >
>> > Best regards,
>> >
>> > Menelaos Karavelas
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Wenchen Fan <cl...@gmail.com>.
Hi Jia,
This is a good question. As the shepherd of this SPIP, I'd like to clarify
the motivation here: the focus of this project is more about the storage
part, not the processing. Apache Sedona is a great library for geo
processing, but without native geo type support in Spark, users can't do
the following things:
- read the geo type columns from Parquet files (or other data sources)
directly
- write geo values into Parquet files (or other data sources) as native geo
types.
- push down geo predicates to the data source when reading
In the SPIP JIRA, we explicitly mentioned that "This proposal is laying the
foundation - building the infrastructure to handle geospatial data, but not
creating a full-featured geospatial processing system. Such extension can
be done later as a separate improvement." Maybe the right direction is to
not do it and leave it to third-party libraries.
The ultimate goal is to establish Spark as a comprehensive platform that
can connect to a rich ecosystem of third-party data sources and processing
libraries. For this project, we should definitely work with the Apache
Sedona community closely, to figure out the best protocol (what binary/text
format to use? how to represent geo values in Java? etc.)
Thanks,
Wenchen
On Sat, Mar 29, 2025 at 5:28 AM Jia Yu <ji...@apache.org> wrote:
> Dear Menelaos,
>
> Thanks for bringing this up again. I’ve seen similar proposals come up on
> the mailing list before, and I’d like to offer some thoughts.
>
> For full transparency, I’m Jia Yu, PMC Chair of Apache Sedona (
> https://github.com/apache/sedona), a widely used open-source cluster
> computing framework for processing large-scale geospatial data on Spark,
> Flink, and other engines.
>
> From what I understand, this proposal aims to add native geospatial types
> and functionality directly into Spark. However, this seems to replicate
> much of the work already done by the Sedona project over the past 10 years.
>
> Sedona has a strong and active community with well-established
> contribution guidelines. It is already used extensively with Spark in
> production—on platforms like Databricks, AWS EMR, Microsoft Fabric, and
> Google Cloud. Users simply add the Sedona jar, flip a Spark config, and it
> just works—similar to other mature Spark ecosystem libraries.
>
> The project sees over 2 million downloads per month across PyPI, Maven,
> etc., and has been downloaded more than 45 million times overall. Thousands
> of organizations rely on Sedona in production Spark environments.
>
> Sedona has also actively contributed to upstream ecosystem efforts, such
> as geospatial support in Parquet and Iceberg formats.
>
> Additionally, Sedona’s core technology has been peer-reviewed and
> published at top academic conferences. Its performance has been evaluated
> and benchmarked by many independent research articles:
> https://sedona.apache.org/latest/community/publication/
>
> Given all of this, I’m genuinely unsure what gap a new Spark-native effort
> is aiming to fill. If there’s a specific limitation that Sedona cannot
> address, I’d be eager to understand it. Otherwise, duplicating this
> functionality risks fragmenting the ecosystem and introducing confusion for
> current users. I would strongly advocate for close coordination with the
> Sedona community to avoid disruption and ensure alignment with the broader
> Spark ecosystem.
>
> Thanks again for raising this—we’re always happy to collaborate and
> strengthen the ecosystem together.
>
>
> Here is a quick overview of what Apache Sedona already offers on Spark:
> • Geospatial type support:
> • Vector: Geometry, partial Geography
> • Raster
> • Vector data sources:
> • GeoParquet (read/write), GeoJSON (read/write), Shapefile,
> GeoPackage, OpenStreetMap PBF
> • Raster data sources:
> • STAC catalog, GeoTiff (read/write), NetCDF/HDF
> • Functions:
> • 209+ vector (ST_*) functions
> • 100+ raster (RS_*) functions
> • GeoStats SQL: DBSCAN, hotspot analysis, outlier detection
> • Language support:
> • Scala, Java, SQL, Python, R
> • Query acceleration via R-Tree:
> • Distributed and broadcast spatial joins
> • KNN joins
> • Range queries
> • UDF support:
> • Scala UDFs (JTS), Python UDFs (Shapely, Rasterio, NumPy),
> Pandas UDFs
> • Serialization:
> • Custom serializers for geometry types
> • Ecosystem integrations:
> • Jupyter, Zeppelin, Apache Arrow, GeoPandas read/write,
> GeoPandas-like API
>
> Jia Yu
>
>
> On 2025/03/28 17:46:15 Menelaos Karavelas wrote:
> > Dear Spark community,
> >
> > I would like to propose the addition of new geospatial data types
> (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently
> added as new logical types in the Parquet specification.
> >
> > The new types should improve Spark’s ability to read the new Parquet
> logical types and perform some minimal meaningful operations on them.
> >
> > SPIP: https://issues.apache.org/jira/browse/SPARK-51658
> >
> > Looking forward to your comments and feedback.
> >
> >
> > Best regards,
> >
> > Menelaos Karavelas
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Jia Yu <ji...@apache.org>.
Dear Menelaos,
Thanks for bringing this up again. I’ve seen similar proposals come up on the mailing list before, and I’d like to offer some thoughts.
For full transparency, I’m Jia Yu, PMC Chair of Apache Sedona (https://github.com/apache/sedona), a widely used open-source cluster computing framework for processing large-scale geospatial data on Spark, Flink, and other engines.
From what I understand, this proposal aims to add native geospatial types and functionality directly into Spark. However, this seems to replicate much of the work already done by the Sedona project over the past 10 years.
Sedona has a strong and active community with well-established contribution guidelines. It is already used extensively with Spark in production—on platforms like Databricks, AWS EMR, Microsoft Fabric, and Google Cloud. Users simply add the Sedona jar, flip a Spark config, and it just works—similar to other mature Spark ecosystem libraries.
The project sees over 2 million downloads per month across PyPI, Maven, etc., and has been downloaded more than 45 million times overall. Thousands of organizations rely on Sedona in production Spark environments.
Sedona has also actively contributed to upstream ecosystem efforts, such as geospatial support in Parquet and Iceberg formats.
Additionally, Sedona’s core technology has been peer-reviewed and published at top academic conferences. Its performance has been evaluated and benchmarked by many independent research articles: https://sedona.apache.org/latest/community/publication/
Given all of this, I’m genuinely unsure what gap a new Spark-native effort is aiming to fill. If there’s a specific limitation that Sedona cannot address, I’d be eager to understand it. Otherwise, duplicating this functionality risks fragmenting the ecosystem and introducing confusion for current users. I would strongly advocate for close coordination with the Sedona community to avoid disruption and ensure alignment with the broader Spark ecosystem.
Thanks again for raising this—we’re always happy to collaborate and strengthen the ecosystem together.
Here is a quick overview of what Apache Sedona already offers on Spark:
• Geospatial type support:
• Vector: Geometry, partial Geography
• Raster
• Vector data sources:
• GeoParquet (read/write), GeoJSON (read/write), Shapefile, GeoPackage, OpenStreetMap PBF
• Raster data sources:
• STAC catalog, GeoTiff (read/write), NetCDF/HDF
• Functions:
• 209+ vector (ST_*) functions
• 100+ raster (RS_*) functions
• GeoStats SQL: DBSCAN, hotspot analysis, outlier detection
• Language support:
• Scala, Java, SQL, Python, R
• Query acceleration via R-Tree:
• Distributed and broadcast spatial joins
• KNN joins
• Range queries
• UDF support:
• Scala UDFs (JTS), Python UDFs (Shapely, Rasterio, NumPy), Pandas UDFs
• Serialization:
• Custom serializers for geometry types
• Ecosystem integrations:
• Jupyter, Zeppelin, Apache Arrow, GeoPandas read/write, GeoPandas-like API
Jia Yu
On 2025/03/28 17:46:15 Menelaos Karavelas wrote:
> Dear Spark community,
>
> I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
>
> The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
>
> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>
> Looking forward to your comments and feedback.
>
>
> Best regards,
>
> Menelaos Karavelas
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Wenchen Fan <cl...@gmail.com>.
Hi Jia,
Thanks for your detailed explanation! The existing implementation of
geospatial serialization, predicate pushdown, and other features in Apache
Sedona is indeed valuable for this project. What we’re proposing isn’t
something entirely new to the industry but rather a re-architecture: we
believe it’s important to standardize common data types in Spark and
clearly define the boundaries between different layers in the Lakehouse
ecosystem.
While it makes sense for Apache Sedona to have its own Parquet data source
for geospatial types in the absence of a standard, the long-term goal
should be for Parquet to support geospatial types natively (which is
already in progress). This would eliminate the need for other geospatial
frameworks to reinvent the same functionality.
That said, this SPIP does not directly benefit Apache Sedona users today,
but it lays a solid foundation for geospatial processing in Spark,
ultimately benefiting the entire ecosystem. Let’s collaborate closely to
define a standard protocol for geospatial processing in Spark—one that
serves all stakeholders.
Looking forward to working together!
Thanks,
Wenchen
On Mon, Mar 31, 2025 at 3:07 AM Szehon Ho <sz...@gmail.com> wrote:
> Hi Jia
>
> Yea I think the SPIP was trying to be concise and focused, but definitely
> was chatting yesterday with Menelaus on how to mention Sedona here but
> still keep it concise and focused :)
>
> It makes sense to collect feedbacks before the vote, thanks for the
> comments and collaboration here, and again for the expertise from Sedona
> side in these efforts.
>
> Thanks!
> Szehon
>
> Sent from my iPhone
>
> > On Mar 29, 2025, at 11:42 PM, Jia Yu <ji...@apache.org> wrote:
> >
> > Hi Reynold and team,
> >
> > I’m glad to see that the Spark community is recognizing the importance
> > of geospatial support. The Sedona community has long been a strong
> > advocate for Spark, and we’ve proudly supported large-scale geospatial
> > workloads on Spark for nearly a decade. We’re absolutely open to
> > collaborating and figuring out what’s best for the users together.
> >
> > I’d like to give the community a bit more time to weigh in on the
> > scope of the SPIP — especially around the proposal’s current focus on
> > native types versus simply supporting reading/writing Parquet Geo
> > data.
> >
> > While we wait, I took a quick look at the SPIP and noticed something a
> > bit surprising: as the most widely used geospatial framework in the
> > Spark ecosystem, Apache Sedona isn’t mentioned at all. Many of the
> > topics raised in the proposal — geometry types, geospatial
> > serialization, predicate pushdown, UDFs, and more — have already been
> > solved by Sedona, and are used at scale across Spark, Flink, and even
> > Snowflake.
> >
> > I believe the SPIP JIRA ticket and the associated Google Doc would
> > benefit from a clearer explanation of the current state of the art,
> > including Apache Sedona — especially for community members who may not
> > be familiar with the existing geospatial ecosystem. Providing this
> > context can help prevent unnecessary reinvention and reduce the risk
> > of introducing incompatibilities with Sedona. Again, for context,
> > Sedona sees over 2 million downloads each month, and millions of
> > Sedona-Spark sessions are created by users monthly across various
> > platforms.
> >
> > As sister projects under the Apache umbrella, Spark and Sedona should
> > aim to support and complement each other. The Sedona community is more
> > than willing to make adjustments on our side to ensure compatibility
> > and minimize disruption for users. We’re also happy to contribute
> > relevant portions of our code back to Spark where it makes sense.
> >
> > For reference, here are a few relevant components that already exist in
> Sedona:
> > 1. Geometry, geography, and raster UDTs
> >
> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/UDT
> > 2. Geospatial serializer
> >
> https://github.com/apache/sedona/tree/master/common/src/main/java/org/apache/sedona/common/geometrySerde
> > 3. Geometry and Raster function catalog
> >
> https://github.com/apache/sedona/blob/master/spark/common/src/main/scala/org/apache/sedona/sql/UDF/Catalog.scala
> > 4. Shared function implementations (used by Spark, Flink, Snowflake)
> >
> https://github.com/apache/sedona/tree/master/common/src/main/java/org/apache/sedona/common
> > 5. Catalyst expressions for Sedona Spark
> >
> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions
> >
> > Happy to chat further and hear your thoughts. Again, Sedona is an
> > Apache project — Spark is welcome to depend on Sedona and re-use any
> > of our work if helpful.
> >
> > Thanks,
> > Jia
> >
> >> On Sat, Mar 29, 2025 at 2:41 PM Reynold Xin <rx...@databricks.com>
> wrote:
> >>
> >> While I don’t think Spark should become a super specialized geospatial
> processing engine, I don’t think it makes sense to focus *only* on reading
> and writing from storage. Geospatial is a pretty common and fundamental
> capability of analytics systems and virtually every mature and popular
> analytics systems, be it open source or proprietary, storage or query, has
> some basic geospatial data type and support. Adding geospatial type and
> some basic expressions is such a no brainer.
> >>
> >>> On Sat, Mar 29, 2025 at 2:27 PM Jia Yu <ji...@apache.org> wrote:
> >>>
> >>> Hi Wenchen, Menelaos and Szehon,
> >>>
> >>> Thanks for the clarification — I’m glad to hear the primary motivation
> of this SPIP is focused on reading and writing geospatial data with Parquet
> and Iceberg. That’s an important goal, and I want to highlight that this
> problem is being solved by the Apache Sedona community.
> >>>
> >>> Since the primary motivation here is Parquet-level support, I suggest
> shifting the focus of this discussion toward enabling geo support in Spark
> Parquet DataSource rather than introducing core types.
> >>>
> >>> ** Why Spark Should Avoid Hardcoding Domain-Specific Types like geo
> types **
> >>>
> >>> 1. Domain types evolve quickly.
> >>>
> >>> In geospatial, we already have geometry, geography, raster,
> trajectory, point clouds — and the list keeps growing. In AI/ML, we’re
> seeing tensors, vectors, and multi-dimensional arrays. Spark’s strength has
> always been in its general-purpose architecture and extensibility.
> Introducing hardcoded support for fast-changing domain-specific types risks
> long-term maintenance issues and eventual incompatibility with emerging
> standards.
> >>>
> >>> 2. Geospatial in Java and Python is a dependency hell.
> >>>
> >>> There are multiple competing geometry libraries with incompatible
> APIs. No widely adopted Java library supports geography types. The most
> authoritative CRS dataset (EPSG) is not Apache-compatible. The json format
> for CRS definitions (projjson) is only fully supported in PROJ, a C++
> library without a Java equivalent and no formal OGC standard status. On the
> Python side, this might involve Shapely and GeoPandas dependencies.
> >>>
> >>> 3. Sedona already supports Geo fully in (Geo)Parquet.
> >>>
> >>> Sedona has supported reading, writing, metadata preservation, and data
> skipping for GeoParquet (predecessor of Parquet Geo) for over two years
> [2][3]. These features are production-tested and widely used.
> >>>
> >>> ** Proposed Path Forward: Geo Support via Spark Extensions **
> >>>
> >>> To enable seamless Parquet integration without burdening Spark core,
> here are two options:
> >>>
> >>> Option 1:
> >>> Sedona offers a dedicated `parquet-geo` DataSource that handles type
> encoding, metadata, and data skipping. No changes to Spark are required.
> This is already underway and will be maintained by the Sedona community to
> keep up with the evolving Geo standards.
> >>>
> >>> Option 2:
> >>> Spark provides hooks to inject:
> >>> - custom logical types / user-defined types (UDTs)
> >>> - custom statistics and filter pushdowns
> >>> Sedona can then extend the built-in `parquet` DataSource to integrate
> geo type metadata, predicate pushdown, and serialization seamlessly.
> >>>
> >>> For Iceberg, we’ve already published a proof-of-concept connector [4]
> showing Sedona, Spark, and Iceberg working together without any Spark core
> changes [5].
> >>>
> >>> ** On the Bigger Picture **
> >>>
> >>> I also agree with your long-term vision. I believe Spark is on the
> path to becoming a foundational compute engine — much like Postgres or
> Pandas — where the core remains focused and stable, while powerful
> domain-specific capabilities emerge from its ecosystem.
> >>>
> >>> To support this future, Spark could prioritize flexible extension
> hooks so that third-party libraries can thrive — just like we’ve seen with
> PostGIS, pgvector, TimescaleDB in the Postgres ecosystem, and GeoPandas in
> the Pandas ecosystem.
> >>>
> >>> Sedona is following this model by building geospatial support around
> Spark — not inside it — and we’d love to continue collaborating in this
> spirit.
> >>>
> >>> Happy to work together on providing Geo support in Parquet!
> >>>
> >>> Best,
> >>> Jia
> >>>
> >>> References
> >>>
> >>> [1] GeoParquet project:
> >>> https://github.com/opengeospatial/geoparquet
> >>>
> >>> [2] Sedona’s GeoParquet DataSource implementation:
> >>>
> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet
> >>>
> >>> [3] Sedona’s GeoParquet documentation:
> >>>
> https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
> >>>
> >>> [4] Sedona-Iceberg connector (PoC):
> >>> https://github.com/wherobots/sedona-iceberg-connector
> >>>
> >>> [5] Spark-Sedona-Iceberg working example:
> >>>
> https://github.com/wherobots/sedona-iceberg-connector/blob/main/src/test/scala/com/wherobots/sedona/TestGeospatial.scala#L53
> >>>
> >>>
> >>> On 2025/03/29 19:27:08 Menelaos Karavelas wrote:
> >>>> To continue along the line of thought of Szehon:
> >>>>
> >>>> I am really excited that the Parquet and Iceberg communities have
> adopted geospatial logical types and of course I am grateful for the work
> put in that direction.
> >>>>
> >>>> As both Wenchen and Szehon pointed out in their own way, the goal is
> to have minimal support in Spark, as a common platform, for these types.
> >>>>
> >>>> To be more specific and explicit: The proposal scope is to add
> support for reading/writing to Parquet, based on the new standard, as well
> as adding the types as built-in types in Spark to complement the storage
> support. The few ST expressions that are in the proposal are what seem to
> be the minimal set of expressions needed to support working with geospatial
> values in the Spark engine in a meaningful way.
> >>>>
> >>>> Best,
> >>>>
> >>>> Menelaos
> >>>>
> >>>>
> >>>>> On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com>
> wrote:
> >>>>>
> >>>>> Thank you Menelaos, will do!
> >>>>>
> >>>>> To give a little background, Jia and Sedona community, also
> GeoParquet community, and others really put much effort contributing to
> defining the Parquet and Iceberg geo types, which couldn't be done without
> their experience and help!
> >>>>>
> >>>>> But I do agree with Wenchen , now that the types are in most common
> data sources in ecosystem , I think Apache Spark as a common platform needs
> to have this type definition for inter-op, otherwise users of vanilla Spark
> cannot work with those data sources with stored geospatial data. (Imo a
> similar rationale in adding timestamp nano in the other ongoing SPIP.).
> >>>>>
> >>>>> And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment
> the ecosystem by implementing Sedona’s advanced geospatial analytic tech in
> Spark itself, which you may be right belongs in pluggable frameworks.
> Menelaus may explain more about the SPIP goal.
> >>>>>
> >>>>> I do hope there can be more collaboration across communities (like
> in Iceberg/Parquet collaboration) in getting Sedona community’s experience
> in making sure these type definitions are optimal , and compatible for
> Sedona.
> >>>>>
> >>>>> Thanks!
> >>>>> Szehon
> >>>>>
> >>>>>
> >>>>>> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <
> menelaos.karavelas@gmail.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>> Hello Szehon,
> >>>>>>
> >>>>>> I just created a Google doc and also linked it in the JIRA:
> >>>>>>
> >>>>>>
> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
> >>>>>>
> >>>>>> Please feel free to comment on it.
> >>>>>>
> >>>>>> Best,
> >>>>>>
> >>>>>> Menelaos
> >>>>>>
> >>>>>>
> >>>>>>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> Thanks Menelaos, this is exciting ! Is there a google doc we can
> comment, or just on the JIRA?
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> Szehon
> >>>>>>>
> >>>>>>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <
> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
> wrote:
> >>>>>>>> Sorry, I only had a quick look at the proposal, looked for WKT
> and didn't find anything.
> >>>>>>>>
> >>>>>>>> It's been years since I worked on geospatial projects and I'm not
> an expert (at all). Maybe starting with something simple but useful like
> conversion WKT<=>WKB?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <
> menelaos.karavelas@gmail.com <ma...@gmail.com>>
> escribió:
> >>>>>>>>> In the SPIP Jira the proposal is to add the expressions
> ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
> >>>>>>>>> Is there anything else that you think should be added?
> >>>>>>>>>
> >>>>>>>>> Regarding WKT, what do you think should be added?
> >>>>>>>>>
> >>>>>>>>> - Menelaos
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <
> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> What about adding support for WKT <
> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB
> <
> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary
> >?
> >>>>>>>>>>
> >>>>>>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<
> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>)
> escribió:
> >>>>>>>>>>> +1 (non-binding)
> >>>>>>>>>>>
> >>>>>>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <
> menelaos.karavelas@gmail.com <ma...@gmail.com>>
> escribió:
> >>>>>>>>>>>> Dear Spark community,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I would like to propose the addition of new geospatial data
> types (GEOMETRY and GEOGRAPHY) which represent geospatial values as
> recently added as new logical types in the Parquet specification.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The new types should improve Spark’s ability to read the new
> Parquet logical types and perform some minimal meaningful operations on
> them.
> >>>>>>>>>>>>
> >>>>>>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
> >>>>>>>>>>>>
> >>>>>>>>>>>> Looking forward to your comments and feedback.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Menelaos Karavelas
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Szehon Ho <sz...@gmail.com>.
Hi Jia
Yea I think the SPIP was trying to be concise and focused, but definitely was chatting yesterday with Menelaus on how to mention Sedona here but still keep it concise and focused :)
It makes sense to collect feedbacks before the vote, thanks for the comments and collaboration here, and again for the expertise from Sedona side in these efforts.
Thanks!
Szehon
Sent from my iPhone
> On Mar 29, 2025, at 11:42 PM, Jia Yu <ji...@apache.org> wrote:
>
> Hi Reynold and team,
>
> I’m glad to see that the Spark community is recognizing the importance
> of geospatial support. The Sedona community has long been a strong
> advocate for Spark, and we’ve proudly supported large-scale geospatial
> workloads on Spark for nearly a decade. We’re absolutely open to
> collaborating and figuring out what’s best for the users together.
>
> I’d like to give the community a bit more time to weigh in on the
> scope of the SPIP — especially around the proposal’s current focus on
> native types versus simply supporting reading/writing Parquet Geo
> data.
>
> While we wait, I took a quick look at the SPIP and noticed something a
> bit surprising: as the most widely used geospatial framework in the
> Spark ecosystem, Apache Sedona isn’t mentioned at all. Many of the
> topics raised in the proposal — geometry types, geospatial
> serialization, predicate pushdown, UDFs, and more — have already been
> solved by Sedona, and are used at scale across Spark, Flink, and even
> Snowflake.
>
> I believe the SPIP JIRA ticket and the associated Google Doc would
> benefit from a clearer explanation of the current state of the art,
> including Apache Sedona — especially for community members who may not
> be familiar with the existing geospatial ecosystem. Providing this
> context can help prevent unnecessary reinvention and reduce the risk
> of introducing incompatibilities with Sedona. Again, for context,
> Sedona sees over 2 million downloads each month, and millions of
> Sedona-Spark sessions are created by users monthly across various
> platforms.
>
> As sister projects under the Apache umbrella, Spark and Sedona should
> aim to support and complement each other. The Sedona community is more
> than willing to make adjustments on our side to ensure compatibility
> and minimize disruption for users. We’re also happy to contribute
> relevant portions of our code back to Spark where it makes sense.
>
> For reference, here are a few relevant components that already exist in Sedona:
> 1. Geometry, geography, and raster UDTs
> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/UDT
> 2. Geospatial serializer
> https://github.com/apache/sedona/tree/master/common/src/main/java/org/apache/sedona/common/geometrySerde
> 3. Geometry and Raster function catalog
> https://github.com/apache/sedona/blob/master/spark/common/src/main/scala/org/apache/sedona/sql/UDF/Catalog.scala
> 4. Shared function implementations (used by Spark, Flink, Snowflake)
> https://github.com/apache/sedona/tree/master/common/src/main/java/org/apache/sedona/common
> 5. Catalyst expressions for Sedona Spark
> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions
>
> Happy to chat further and hear your thoughts. Again, Sedona is an
> Apache project — Spark is welcome to depend on Sedona and re-use any
> of our work if helpful.
>
> Thanks,
> Jia
>
>> On Sat, Mar 29, 2025 at 2:41 PM Reynold Xin <rx...@databricks.com> wrote:
>>
>> While I don’t think Spark should become a super specialized geospatial processing engine, I don’t think it makes sense to focus *only* on reading and writing from storage. Geospatial is a pretty common and fundamental capability of analytics systems and virtually every mature and popular analytics systems, be it open source or proprietary, storage or query, has some basic geospatial data type and support. Adding geospatial type and some basic expressions is such a no brainer.
>>
>>> On Sat, Mar 29, 2025 at 2:27 PM Jia Yu <ji...@apache.org> wrote:
>>>
>>> Hi Wenchen, Menelaos and Szehon,
>>>
>>> Thanks for the clarification — I’m glad to hear the primary motivation of this SPIP is focused on reading and writing geospatial data with Parquet and Iceberg. That’s an important goal, and I want to highlight that this problem is being solved by the Apache Sedona community.
>>>
>>> Since the primary motivation here is Parquet-level support, I suggest shifting the focus of this discussion toward enabling geo support in Spark Parquet DataSource rather than introducing core types.
>>>
>>> ** Why Spark Should Avoid Hardcoding Domain-Specific Types like geo types **
>>>
>>> 1. Domain types evolve quickly.
>>>
>>> In geospatial, we already have geometry, geography, raster, trajectory, point clouds — and the list keeps growing. In AI/ML, we’re seeing tensors, vectors, and multi-dimensional arrays. Spark’s strength has always been in its general-purpose architecture and extensibility. Introducing hardcoded support for fast-changing domain-specific types risks long-term maintenance issues and eventual incompatibility with emerging standards.
>>>
>>> 2. Geospatial in Java and Python is a dependency hell.
>>>
>>> There are multiple competing geometry libraries with incompatible APIs. No widely adopted Java library supports geography types. The most authoritative CRS dataset (EPSG) is not Apache-compatible. The json format for CRS definitions (projjson) is only fully supported in PROJ, a C++ library without a Java equivalent and no formal OGC standard status. On the Python side, this might involve Shapely and GeoPandas dependencies.
>>>
>>> 3. Sedona already supports Geo fully in (Geo)Parquet.
>>>
>>> Sedona has supported reading, writing, metadata preservation, and data skipping for GeoParquet (predecessor of Parquet Geo) for over two years [2][3]. These features are production-tested and widely used.
>>>
>>> ** Proposed Path Forward: Geo Support via Spark Extensions **
>>>
>>> To enable seamless Parquet integration without burdening Spark core, here are two options:
>>>
>>> Option 1:
>>> Sedona offers a dedicated `parquet-geo` DataSource that handles type encoding, metadata, and data skipping. No changes to Spark are required. This is already underway and will be maintained by the Sedona community to keep up with the evolving Geo standards.
>>>
>>> Option 2:
>>> Spark provides hooks to inject:
>>> - custom logical types / user-defined types (UDTs)
>>> - custom statistics and filter pushdowns
>>> Sedona can then extend the built-in `parquet` DataSource to integrate geo type metadata, predicate pushdown, and serialization seamlessly.
>>>
>>> For Iceberg, we’ve already published a proof-of-concept connector [4] showing Sedona, Spark, and Iceberg working together without any Spark core changes [5].
>>>
>>> ** On the Bigger Picture **
>>>
>>> I also agree with your long-term vision. I believe Spark is on the path to becoming a foundational compute engine — much like Postgres or Pandas — where the core remains focused and stable, while powerful domain-specific capabilities emerge from its ecosystem.
>>>
>>> To support this future, Spark could prioritize flexible extension hooks so that third-party libraries can thrive — just like we’ve seen with PostGIS, pgvector, TimescaleDB in the Postgres ecosystem, and GeoPandas in the Pandas ecosystem.
>>>
>>> Sedona is following this model by building geospatial support around Spark — not inside it — and we’d love to continue collaborating in this spirit.
>>>
>>> Happy to work together on providing Geo support in Parquet!
>>>
>>> Best,
>>> Jia
>>>
>>> References
>>>
>>> [1] GeoParquet project:
>>> https://github.com/opengeospatial/geoparquet
>>>
>>> [2] Sedona’s GeoParquet DataSource implementation:
>>> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet
>>>
>>> [3] Sedona’s GeoParquet documentation:
>>> https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
>>>
>>> [4] Sedona-Iceberg connector (PoC):
>>> https://github.com/wherobots/sedona-iceberg-connector
>>>
>>> [5] Spark-Sedona-Iceberg working example:
>>> https://github.com/wherobots/sedona-iceberg-connector/blob/main/src/test/scala/com/wherobots/sedona/TestGeospatial.scala#L53
>>>
>>>
>>> On 2025/03/29 19:27:08 Menelaos Karavelas wrote:
>>>> To continue along the line of thought of Szehon:
>>>>
>>>> I am really excited that the Parquet and Iceberg communities have adopted geospatial logical types and of course I am grateful for the work put in that direction.
>>>>
>>>> As both Wenchen and Szehon pointed out in their own way, the goal is to have minimal support in Spark, as a common platform, for these types.
>>>>
>>>> To be more specific and explicit: The proposal scope is to add support for reading/writing to Parquet, based on the new standard, as well as adding the types as built-in types in Spark to complement the storage support. The few ST expressions that are in the proposal are what seem to be the minimal set of expressions needed to support working with geospatial values in the Spark engine in a meaningful way.
>>>>
>>>> Best,
>>>>
>>>> Menelaos
>>>>
>>>>
>>>>> On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com> wrote:
>>>>>
>>>>> Thank you Menelaos, will do!
>>>>>
>>>>> To give a little background, Jia and Sedona community, also GeoParquet community, and others really put much effort contributing to defining the Parquet and Iceberg geo types, which couldn't be done without their experience and help!
>>>>>
>>>>> But I do agree with Wenchen , now that the types are in most common data sources in ecosystem , I think Apache Spark as a common platform needs to have this type definition for inter-op, otherwise users of vanilla Spark cannot work with those data sources with stored geospatial data. (Imo a similar rationale in adding timestamp nano in the other ongoing SPIP.).
>>>>>
>>>>> And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment the ecosystem by implementing Sedona’s advanced geospatial analytic tech in Spark itself, which you may be right belongs in pluggable frameworks. Menelaus may explain more about the SPIP goal.
>>>>>
>>>>> I do hope there can be more collaboration across communities (like in Iceberg/Parquet collaboration) in getting Sedona community’s experience in making sure these type definitions are optimal , and compatible for Sedona.
>>>>>
>>>>> Thanks!
>>>>> Szehon
>>>>>
>>>>>
>>>>>> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <me...@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>> Hello Szehon,
>>>>>>
>>>>>> I just created a Google doc and also linked it in the JIRA:
>>>>>>
>>>>>> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
>>>>>>
>>>>>> Please feel free to comment on it.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Menelaos
>>>>>>
>>>>>>
>>>>>>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com> wrote:
>>>>>>>
>>>>>>> Thanks Menelaos, this is exciting ! Is there a google doc we can comment, or just on the JIRA?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Szehon
>>>>>>>
>>>>>>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>> Sorry, I only had a quick look at the proposal, looked for WKT and didn't find anything.
>>>>>>>>
>>>>>>>> It's been years since I worked on geospatial projects and I'm not an expert (at all). Maybe starting with something simple but useful like conversion WKT<=>WKB?
>>>>>>>>
>>>>>>>>
>>>>>>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>>>>>>>>> In the SPIP Jira the proposal is to add the expressions ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
>>>>>>>>> Is there anything else that you think should be added?
>>>>>>>>>
>>>>>>>>> Regarding WKT, what do you think should be added?
>>>>>>>>>
>>>>>>>>> - Menelaos
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>>>
>>>>>>>>>> What about adding support for WKT <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>?
>>>>>>>>>>
>>>>>>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<angel.alvarez.pascua@gmail.com <ma...@gmail.com>>) escribió:
>>>>>>>>>>> +1 (non-binding)
>>>>>>>>>>>
>>>>>>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>>>>>>>>>>>> Dear Spark community,
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
>>>>>>>>>>>>
>>>>>>>>>>>> The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
>>>>>>>>>>>>
>>>>>>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>>>>>>>>>>>
>>>>>>>>>>>> Looking forward to your comments and feedback.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Menelaos Karavelas
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Jia Yu <ji...@apache.org>.
Hi Reynold and team,
I’m glad to see that the Spark community is recognizing the importance
of geospatial support. The Sedona community has long been a strong
advocate for Spark, and we’ve proudly supported large-scale geospatial
workloads on Spark for nearly a decade. We’re absolutely open to
collaborating and figuring out what’s best for the users together.
I’d like to give the community a bit more time to weigh in on the
scope of the SPIP — especially around the proposal’s current focus on
native types versus simply supporting reading/writing Parquet Geo
data.
While we wait, I took a quick look at the SPIP and noticed something a
bit surprising: as the most widely used geospatial framework in the
Spark ecosystem, Apache Sedona isn’t mentioned at all. Many of the
topics raised in the proposal — geometry types, geospatial
serialization, predicate pushdown, UDFs, and more — have already been
solved by Sedona, and are used at scale across Spark, Flink, and even
Snowflake.
I believe the SPIP JIRA ticket and the associated Google Doc would
benefit from a clearer explanation of the current state of the art,
including Apache Sedona — especially for community members who may not
be familiar with the existing geospatial ecosystem. Providing this
context can help prevent unnecessary reinvention and reduce the risk
of introducing incompatibilities with Sedona. Again, for context,
Sedona sees over 2 million downloads each month, and millions of
Sedona-Spark sessions are created by users monthly across various
platforms.
As sister projects under the Apache umbrella, Spark and Sedona should
aim to support and complement each other. The Sedona community is more
than willing to make adjustments on our side to ensure compatibility
and minimize disruption for users. We’re also happy to contribute
relevant portions of our code back to Spark where it makes sense.
For reference, here are a few relevant components that already exist in Sedona:
1. Geometry, geography, and raster UDTs
https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/UDT
2. Geospatial serializer
https://github.com/apache/sedona/tree/master/common/src/main/java/org/apache/sedona/common/geometrySerde
3. Geometry and Raster function catalog
https://github.com/apache/sedona/blob/master/spark/common/src/main/scala/org/apache/sedona/sql/UDF/Catalog.scala
4. Shared function implementations (used by Spark, Flink, Snowflake)
https://github.com/apache/sedona/tree/master/common/src/main/java/org/apache/sedona/common
5. Catalyst expressions for Sedona Spark
https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/expressions
Happy to chat further and hear your thoughts. Again, Sedona is an
Apache project — Spark is welcome to depend on Sedona and re-use any
of our work if helpful.
Thanks,
Jia
On Sat, Mar 29, 2025 at 2:41 PM Reynold Xin <rx...@databricks.com> wrote:
>
> While I don’t think Spark should become a super specialized geospatial processing engine, I don’t think it makes sense to focus *only* on reading and writing from storage. Geospatial is a pretty common and fundamental capability of analytics systems and virtually every mature and popular analytics systems, be it open source or proprietary, storage or query, has some basic geospatial data type and support. Adding geospatial type and some basic expressions is such a no brainer.
>
> On Sat, Mar 29, 2025 at 2:27 PM Jia Yu <ji...@apache.org> wrote:
>>
>> Hi Wenchen, Menelaos and Szehon,
>>
>> Thanks for the clarification — I’m glad to hear the primary motivation of this SPIP is focused on reading and writing geospatial data with Parquet and Iceberg. That’s an important goal, and I want to highlight that this problem is being solved by the Apache Sedona community.
>>
>> Since the primary motivation here is Parquet-level support, I suggest shifting the focus of this discussion toward enabling geo support in Spark Parquet DataSource rather than introducing core types.
>>
>> ** Why Spark Should Avoid Hardcoding Domain-Specific Types like geo types **
>>
>> 1. Domain types evolve quickly.
>>
>> In geospatial, we already have geometry, geography, raster, trajectory, point clouds — and the list keeps growing. In AI/ML, we’re seeing tensors, vectors, and multi-dimensional arrays. Spark’s strength has always been in its general-purpose architecture and extensibility. Introducing hardcoded support for fast-changing domain-specific types risks long-term maintenance issues and eventual incompatibility with emerging standards.
>>
>> 2. Geospatial in Java and Python is a dependency hell.
>>
>> There are multiple competing geometry libraries with incompatible APIs. No widely adopted Java library supports geography types. The most authoritative CRS dataset (EPSG) is not Apache-compatible. The json format for CRS definitions (projjson) is only fully supported in PROJ, a C++ library without a Java equivalent and no formal OGC standard status. On the Python side, this might involve Shapely and GeoPandas dependencies.
>>
>> 3. Sedona already supports Geo fully in (Geo)Parquet.
>>
>> Sedona has supported reading, writing, metadata preservation, and data skipping for GeoParquet (predecessor of Parquet Geo) for over two years [2][3]. These features are production-tested and widely used.
>>
>> ** Proposed Path Forward: Geo Support via Spark Extensions **
>>
>> To enable seamless Parquet integration without burdening Spark core, here are two options:
>>
>> Option 1:
>> Sedona offers a dedicated `parquet-geo` DataSource that handles type encoding, metadata, and data skipping. No changes to Spark are required. This is already underway and will be maintained by the Sedona community to keep up with the evolving Geo standards.
>>
>> Option 2:
>> Spark provides hooks to inject:
>> - custom logical types / user-defined types (UDTs)
>> - custom statistics and filter pushdowns
>> Sedona can then extend the built-in `parquet` DataSource to integrate geo type metadata, predicate pushdown, and serialization seamlessly.
>>
>> For Iceberg, we’ve already published a proof-of-concept connector [4] showing Sedona, Spark, and Iceberg working together without any Spark core changes [5].
>>
>> ** On the Bigger Picture **
>>
>> I also agree with your long-term vision. I believe Spark is on the path to becoming a foundational compute engine — much like Postgres or Pandas — where the core remains focused and stable, while powerful domain-specific capabilities emerge from its ecosystem.
>>
>> To support this future, Spark could prioritize flexible extension hooks so that third-party libraries can thrive — just like we’ve seen with PostGIS, pgvector, TimescaleDB in the Postgres ecosystem, and GeoPandas in the Pandas ecosystem.
>>
>> Sedona is following this model by building geospatial support around Spark — not inside it — and we’d love to continue collaborating in this spirit.
>>
>> Happy to work together on providing Geo support in Parquet!
>>
>> Best,
>> Jia
>>
>> References
>>
>> [1] GeoParquet project:
>> https://github.com/opengeospatial/geoparquet
>>
>> [2] Sedona’s GeoParquet DataSource implementation:
>> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet
>>
>> [3] Sedona’s GeoParquet documentation:
>> https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
>>
>> [4] Sedona-Iceberg connector (PoC):
>> https://github.com/wherobots/sedona-iceberg-connector
>>
>> [5] Spark-Sedona-Iceberg working example:
>> https://github.com/wherobots/sedona-iceberg-connector/blob/main/src/test/scala/com/wherobots/sedona/TestGeospatial.scala#L53
>>
>>
>> On 2025/03/29 19:27:08 Menelaos Karavelas wrote:
>> > To continue along the line of thought of Szehon:
>> >
>> > I am really excited that the Parquet and Iceberg communities have adopted geospatial logical types and of course I am grateful for the work put in that direction.
>> >
>> > As both Wenchen and Szehon pointed out in their own way, the goal is to have minimal support in Spark, as a common platform, for these types.
>> >
>> > To be more specific and explicit: The proposal scope is to add support for reading/writing to Parquet, based on the new standard, as well as adding the types as built-in types in Spark to complement the storage support. The few ST expressions that are in the proposal are what seem to be the minimal set of expressions needed to support working with geospatial values in the Spark engine in a meaningful way.
>> >
>> > Best,
>> >
>> > Menelaos
>> >
>> >
>> > > On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com> wrote:
>> > >
>> > > Thank you Menelaos, will do!
>> > >
>> > > To give a little background, Jia and Sedona community, also GeoParquet community, and others really put much effort contributing to defining the Parquet and Iceberg geo types, which couldn't be done without their experience and help!
>> > >
>> > > But I do agree with Wenchen , now that the types are in most common data sources in ecosystem , I think Apache Spark as a common platform needs to have this type definition for inter-op, otherwise users of vanilla Spark cannot work with those data sources with stored geospatial data. (Imo a similar rationale in adding timestamp nano in the other ongoing SPIP.).
>> > >
>> > > And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment the ecosystem by implementing Sedona’s advanced geospatial analytic tech in Spark itself, which you may be right belongs in pluggable frameworks. Menelaus may explain more about the SPIP goal.
>> > >
>> > > I do hope there can be more collaboration across communities (like in Iceberg/Parquet collaboration) in getting Sedona community’s experience in making sure these type definitions are optimal , and compatible for Sedona.
>> > >
>> > > Thanks!
>> > > Szehon
>> > >
>> > >
>> > >> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <me...@gmail.com> wrote:
>> > >>
>> > >>
>> > >> Hello Szehon,
>> > >>
>> > >> I just created a Google doc and also linked it in the JIRA:
>> > >>
>> > >> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
>> > >>
>> > >> Please feel free to comment on it.
>> > >>
>> > >> Best,
>> > >>
>> > >> Menelaos
>> > >>
>> > >>
>> > >>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com> wrote:
>> > >>>
>> > >>> Thanks Menelaos, this is exciting ! Is there a google doc we can comment, or just on the JIRA?
>> > >>>
>> > >>> Thanks
>> > >>> Szehon
>> > >>>
>> > >>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
>> > >>>> Sorry, I only had a quick look at the proposal, looked for WKT and didn't find anything.
>> > >>>>
>> > >>>> It's been years since I worked on geospatial projects and I'm not an expert (at all). Maybe starting with something simple but useful like conversion WKT<=>WKB?
>> > >>>>
>> > >>>>
>> > >>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>> > >>>>> In the SPIP Jira the proposal is to add the expressions ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
>> > >>>>> Is there anything else that you think should be added?
>> > >>>>>
>> > >>>>> Regarding WKT, what do you think should be added?
>> > >>>>>
>> > >>>>> - Menelaos
>> > >>>>>
>> > >>>>>
>> > >>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
>> > >>>>>>
>> > >>>>>> What about adding support for WKT <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>?
>> > >>>>>>
>> > >>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<angel.alvarez.pascua@gmail.com <ma...@gmail.com>>) escribió:
>> > >>>>>>> +1 (non-binding)
>> > >>>>>>>
>> > >>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>> > >>>>>>>> Dear Spark community,
>> > >>>>>>>>
>> > >>>>>>>> I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
>> > >>>>>>>>
>> > >>>>>>>> The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
>> > >>>>>>>>
>> > >>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>> > >>>>>>>>
>> > >>>>>>>> Looking forward to your comments and feedback.
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> Best regards,
>> > >>>>>>>>
>> > >>>>>>>> Menelaos Karavelas
>> > >>>>>>>>
>> > >>>>>
>> > >>
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Ángel Álvarez Pascua <an...@gmail.com>.
Hi Jia,
I really appreciate your very instructive answer. I truly believe that
discussing topics with people who know far more than I do is a great way to
learn new and interesting things. Your explanations are quite logical and
make perfect sense to me. Sh**, I'm not that sure about this proposal now
... 😂
*"If you haven’t encountered this kind of ‘dependency hell’ while working
on geospatial projects with Spark, you may have been fortunate to deal with
relatively simple cases."*
Yes, that was the case for us. We loaded OpenStreetMap data from Spain,
calculated some Haversine distances between points in rasters using a
custom-made C++ library, and did a bit more. We also used GeoTrellis for
something, but I can't quite remember what
Thanks again!
El dom, 30 mar 2025 a las 9:25, Jia Yu (<ji...@apache.org>) escribió:
> Hey Angel,
>
> I am glad that you asked these questions. Please see my answers below.
>
>
> *1. Domain types evolve quickly. - It has taken years for Parquet to
> include these new types in its format... We could evolve alongside Parquet.
> Unfortunately, Spark is not known for upgrading its dependencies quickly.*
>
> Exactly — domain-specific types evolve rapidly and may head in directions
> that aren’t fully aligned with formats like Parquet, Avro, and others. In
> such cases, should Spark, as a general-purpose compute engine, really be
> tightly coupled to the specifics of a single storage format?
>
> Personally, I really appreciate Spark’s UserDefinedType mechanism and
> Apache Arrow’s ExtensionType — both offer maximum flexibility while keeping
> the core engine clean and extensible.
>
>
>
> * 2. Geospatial in Java and Python is a dependency hell.- How has Parquet
> solved that problem, then?*
>
> Exactly — this problem is not fully solved by Parquet. While the Parquet
> spec now includes a definition for geospatial types, it’s more of a vision
> than a complete, production-ready solution. Many aspects of the spec are
> not yet implemented in Spark. In fact, the spec represents a compromise
> among multiple vendors (e.g., BigQuery, Snowflake), and many design choices
> are not aligned with Spark’s architecture or ecosystem.
>
> For example:
> • The CRS property in the spec uses a PROJJSON string, which currently
> only has a C++ implementation — there is no Java implementation available.
> • The edge interpolation algorithms (e.g., for great-circle arcs)
> mentioned in the spec also only exist in C++ libraries.
> • Handling of antimeridian-crossing geometries is another complex topic
> that isn’t addressed in Spark today.
>
> The Sedona community is actively working on solutions — either building
> Java equivalents for these features or creating workarounds. These are
> deeply domain-specific efforts and often require non-trivial geospatial
> expertise.
>
> We are currently contributing a Java implementation of the Parquet Geo
> format here: https://github.com/apache/parquet-java/pull/2971
>
> In Python, geospatial manipulation depends on libraries like Shapely and
> GeoPandas, which evolve quickly and frequently introduce breaking changes.
> Sedona has invested significant effort to maintain compatibility and
> stability for Python UDFs across these ecosystems.
>
> If you haven’t encountered this kind of “dependency hell” while working on
> geospatial projects with Spark, you may have been fortunate to deal with
> relatively simple cases — e.g., only working with point data or simple
> polygons.
>
> That usually means:
> 1. All geometries are in a single CRS, typically WGS84 (SRID 4326)
> 2. No antimeridian-crossing geometries
> 3. No need for high-precision distance calculations or spherical geometry
> 4. No need to handle topology or wraparound issues
>
> If that’s the case, then Spark already works fine as-is for your use case
> — so why complicate it?
>
>
> *3. Sedona already supports Geo fully in (Geo)Parquet.- The default format
> in Spark is Parquet, and Parquet now natively supports these types. Are we
> going to force users to add Sedona?*
>
> While opinions may vary, I would encourage users to adopt a solution like
> Apache Sedona that laser focuses on geospatial. Sedona provides
> comprehensive, step-by-step tutorials on how to handle geospatial
> dependencies across major platforms — including Databricks, AWS EMR,
> Microsoft Azure, and Google Cloud. We’re also actively collaborating with
> cloud providers to bundle Sedona natively into their offerings, making it
> even easier for users to get started.
>
>
> That said, I generally share the same perspective — if the Spark community
> believes it would benefit from having basic geospatial support built in,
> the Sedona community would be happy to collaborate on this effort. We’re
> open to contributing the necessary functionality and, if appropriate,
> having Spark depend on Sedona directly to avoid reinvention.
>
> Thanks,
> JIa
>
>
>
> On Sat, Mar 29, 2025 at 11:02 PM Ángel Álvarez Pascua <
> angel.alvarez.pascua@gmail.com> wrote:
>
>>
>> * 1. Domain types evolve quickly.*
>> It has taken years for Parquet to include these new types in its
>> format... We could evolve alongside Parquet. Unfortunately, Spark is not
>> known for upgrading its dependencies quickly.
>>
>> * 2. Geospatial in Java and Python is a dependency hell.*
>> How has Parquet solved that problem, then? I don't recall experiencing
>> any "dependency hell" when working on geospatial projects with Spark, to be
>> honest. Besides, Spark already includes Parquet as a dependency, so...
>> where is the problem?
>>
>> *3. Sedona already supports Geo fully in (Geo)Parquet.*
>> The default format in Spark is Parquet, and Parquet now natively supports
>> these types. Are we going to force users to add Sedona (along with all its
>> third-party dependencies, I assume) to their projects just for reading,
>> writing, and performing basic operations with these types?
>>
>> Anyway, let's vote and see...
>>
>> El sáb, 29 mar 2025 a las 22:41, Reynold Xin (<rx...@databricks.com.invalid>)
>> escribió:
>>
>>> While I don’t think Spark should become a super specialized geospatial
>>> processing engine, I don’t think it makes sense to focus *only* on reading
>>> and writing from storage. Geospatial is a pretty common and fundamental
>>> capability of analytics systems and virtually every mature and popular
>>> analytics systems, be it open source or proprietary, storage or query, has
>>> some basic geospatial data type and support. Adding geospatial type and
>>> some basic expressions is such a no brainer.
>>>
>>> On Sat, Mar 29, 2025 at 2:27 PM Jia Yu <ji...@apache.org> wrote:
>>>
>>>> Hi Wenchen, Menelaos and Szehon,
>>>>
>>>> Thanks for the clarification — I’m glad to hear the primary motivation
>>>> of this SPIP is focused on reading and writing geospatial data with Parquet
>>>> and Iceberg. That’s an important goal, and I want to highlight that this
>>>> problem is being solved by the Apache Sedona community.
>>>>
>>>> Since the primary motivation here is Parquet-level support, I suggest
>>>> shifting the focus of this discussion toward enabling geo support in Spark
>>>> Parquet DataSource rather than introducing core types.
>>>>
>>>> ** Why Spark Should Avoid Hardcoding Domain-Specific Types like geo
>>>> types **
>>>>
>>>> 1. Domain types evolve quickly.
>>>>
>>>> In geospatial, we already have geometry, geography, raster, trajectory,
>>>> point clouds — and the list keeps growing. In AI/ML, we’re seeing tensors,
>>>> vectors, and multi-dimensional arrays. Spark’s strength has always been in
>>>> its general-purpose architecture and extensibility. Introducing hardcoded
>>>> support for fast-changing domain-specific types risks long-term maintenance
>>>> issues and eventual incompatibility with emerging standards.
>>>>
>>>> 2. Geospatial in Java and Python is a dependency hell.
>>>>
>>>> There are multiple competing geometry libraries with incompatible APIs.
>>>> No widely adopted Java library supports geography types. The most
>>>> authoritative CRS dataset (EPSG) is not Apache-compatible. The json format
>>>> for CRS definitions (projjson) is only fully supported in PROJ, a C++
>>>> library without a Java equivalent and no formal OGC standard status. On the
>>>> Python side, this might involve Shapely and GeoPandas dependencies.
>>>>
>>>> 3. Sedona already supports Geo fully in (Geo)Parquet.
>>>>
>>>> Sedona has supported reading, writing, metadata preservation, and data
>>>> skipping for GeoParquet (predecessor of Parquet Geo) for over two years
>>>> [2][3]. These features are production-tested and widely used.
>>>>
>>>> ** Proposed Path Forward: Geo Support via Spark Extensions **
>>>>
>>>> To enable seamless Parquet integration without burdening Spark core,
>>>> here are two options:
>>>>
>>>> Option 1:
>>>> Sedona offers a dedicated `parquet-geo` DataSource that handles type
>>>> encoding, metadata, and data skipping. No changes to Spark are required.
>>>> This is already underway and will be maintained by the Sedona community to
>>>> keep up with the evolving Geo standards.
>>>>
>>>> Option 2:
>>>> Spark provides hooks to inject:
>>>> - custom logical types / user-defined types (UDTs)
>>>> - custom statistics and filter pushdowns
>>>> Sedona can then extend the built-in `parquet` DataSource to integrate
>>>> geo type metadata, predicate pushdown, and serialization seamlessly.
>>>>
>>>> For Iceberg, we’ve already published a proof-of-concept connector [4]
>>>> showing Sedona, Spark, and Iceberg working together without any Spark core
>>>> changes [5].
>>>>
>>>> ** On the Bigger Picture **
>>>>
>>>> I also agree with your long-term vision. I believe Spark is on the path
>>>> to becoming a foundational compute engine — much like Postgres or Pandas —
>>>> where the core remains focused and stable, while powerful domain-specific
>>>> capabilities emerge from its ecosystem.
>>>>
>>>> To support this future, Spark could prioritize flexible extension hooks
>>>> so that third-party libraries can thrive — just like we’ve seen with
>>>> PostGIS, pgvector, TimescaleDB in the Postgres ecosystem, and GeoPandas in
>>>> the Pandas ecosystem.
>>>>
>>>> Sedona is following this model by building geospatial support around
>>>> Spark — not inside it — and we’d love to continue collaborating in this
>>>> spirit.
>>>>
>>>> Happy to work together on providing Geo support in Parquet!
>>>>
>>>> Best,
>>>> Jia
>>>>
>>>> References
>>>>
>>>> [1] GeoParquet project:
>>>> https://github.com/opengeospatial/geoparquet
>>>>
>>>> [2] Sedona’s GeoParquet DataSource implementation:
>>>>
>>>> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet
>>>>
>>>> [3] Sedona’s GeoParquet documentation:
>>>> https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
>>>>
>>>> [4] Sedona-Iceberg connector (PoC):
>>>> https://github.com/wherobots/sedona-iceberg-connector
>>>>
>>>> [5] Spark-Sedona-Iceberg working example:
>>>>
>>>> https://github.com/wherobots/sedona-iceberg-connector/blob/main/src/test/scala/com/wherobots/sedona/TestGeospatial.scala#L53
>>>>
>>>>
>>>> On 2025/03/29 19:27:08 Menelaos Karavelas wrote:
>>>> > To continue along the line of thought of Szehon:
>>>> >
>>>> > I am really excited that the Parquet and Iceberg communities have
>>>> adopted geospatial logical types and of course I am grateful for the work
>>>> put in that direction.
>>>> >
>>>> > As both Wenchen and Szehon pointed out in their own way, the goal is
>>>> to have minimal support in Spark, as a common platform, for these types.
>>>> >
>>>> > To be more specific and explicit: The proposal scope is to add
>>>> support for reading/writing to Parquet, based on the new standard, as well
>>>> as adding the types as built-in types in Spark to complement the storage
>>>> support. The few ST expressions that are in the proposal are what seem to
>>>> be the minimal set of expressions needed to support working with geospatial
>>>> values in the Spark engine in a meaningful way.
>>>> >
>>>> > Best,
>>>> >
>>>> > Menelaos
>>>> >
>>>> >
>>>> > > On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com>
>>>> wrote:
>>>> > >
>>>> > > Thank you Menelaos, will do!
>>>> > >
>>>> > > To give a little background, Jia and Sedona community, also
>>>> GeoParquet community, and others really put much effort contributing to
>>>> defining the Parquet and Iceberg geo types, which couldn't be done without
>>>> their experience and help!
>>>> > >
>>>> > > But I do agree with Wenchen , now that the types are in most common
>>>> data sources in ecosystem , I think Apache Spark as a common platform needs
>>>> to have this type definition for inter-op, otherwise users of vanilla Spark
>>>> cannot work with those data sources with stored geospatial data. (Imo a
>>>> similar rationale in adding timestamp nano in the other ongoing SPIP.).
>>>> > >
>>>> > > And like Wenchen said, the SPIP’s goal doesnt seem to be to
>>>> fragment the ecosystem by implementing Sedona’s advanced geospatial
>>>> analytic tech in Spark itself, which you may be right belongs in pluggable
>>>> frameworks. Menelaus may explain more about the SPIP goal.
>>>> > >
>>>> > > I do hope there can be more collaboration across communities (like
>>>> in Iceberg/Parquet collaboration) in getting Sedona community’s experience
>>>> in making sure these type definitions are optimal , and compatible for
>>>> Sedona.
>>>> > >
>>>> > > Thanks!
>>>> > > Szehon
>>>> > >
>>>> > >
>>>> > >> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <
>>>> menelaos.karavelas@gmail.com> wrote:
>>>> > >>
>>>> > >>
>>>> > >> Hello Szehon,
>>>> > >>
>>>> > >> I just created a Google doc and also linked it in the JIRA:
>>>> > >>
>>>> > >>
>>>> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
>>>> > >>
>>>> > >> Please feel free to comment on it.
>>>> > >>
>>>> > >> Best,
>>>> > >>
>>>> > >> Menelaos
>>>> > >>
>>>> > >>
>>>> > >>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com>
>>>> wrote:
>>>> > >>>
>>>> > >>> Thanks Menelaos, this is exciting ! Is there a google doc we can
>>>> comment, or just on the JIRA?
>>>> > >>>
>>>> > >>> Thanks
>>>> > >>> Szehon
>>>> > >>>
>>>> > >>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <
>>>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
>>>> wrote:
>>>> > >>>> Sorry, I only had a quick look at the proposal, looked for WKT
>>>> and didn't find anything.
>>>> > >>>>
>>>> > >>>> It's been years since I worked on geospatial projects and I'm
>>>> not an expert (at all). Maybe starting with something simple but useful
>>>> like conversion WKT<=>WKB?
>>>> > >>>>
>>>> > >>>>
>>>> > >>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <
>>>> menelaos.karavelas@gmail.com <ma...@gmail.com>>
>>>> escribió:
>>>> > >>>>> In the SPIP Jira the proposal is to add the expressions
>>>> ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
>>>> > >>>>> Is there anything else that you think should be added?
>>>> > >>>>>
>>>> > >>>>> Regarding WKT, what do you think should be added?
>>>> > >>>>>
>>>> > >>>>> - Menelaos
>>>> > >>>>>
>>>> > >>>>>
>>>> > >>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <
>>>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
>>>> wrote:
>>>> > >>>>>>
>>>> > >>>>>> What about adding support for WKT <
>>>> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB
>>>> <
>>>> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary
>>>> >?
>>>> > >>>>>>
>>>> > >>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<
>>>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>)
>>>> escribió:
>>>> > >>>>>>> +1 (non-binding)
>>>> > >>>>>>>
>>>> > >>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <
>>>> menelaos.karavelas@gmail.com <ma...@gmail.com>>
>>>> escribió:
>>>> > >>>>>>>> Dear Spark community,
>>>> > >>>>>>>>
>>>> > >>>>>>>> I would like to propose the addition of new geospatial data
>>>> types (GEOMETRY and GEOGRAPHY) which represent geospatial values as
>>>> recently added as new logical types in the Parquet specification.
>>>> > >>>>>>>>
>>>> > >>>>>>>> The new types should improve Spark’s ability to read the new
>>>> Parquet logical types and perform some minimal meaningful operations on
>>>> them.
>>>> > >>>>>>>>
>>>> > >>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>>> > >>>>>>>>
>>>> > >>>>>>>> Looking forward to your comments and feedback.
>>>> > >>>>>>>>
>>>> > >>>>>>>>
>>>> > >>>>>>>> Best regards,
>>>> > >>>>>>>>
>>>> > >>>>>>>> Menelaos Karavelas
>>>> > >>>>>>>>
>>>> > >>>>>
>>>> > >>
>>>> >
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Jia Yu <ji...@apache.org>.
Hey Angel,
I am glad that you asked these questions. Please see my answers below.
*1. Domain types evolve quickly. - It has taken years for Parquet to
include these new types in its format... We could evolve alongside Parquet.
Unfortunately, Spark is not known for upgrading its dependencies quickly.*
Exactly — domain-specific types evolve rapidly and may head in directions
that aren’t fully aligned with formats like Parquet, Avro, and others. In
such cases, should Spark, as a general-purpose compute engine, really be
tightly coupled to the specifics of a single storage format?
Personally, I really appreciate Spark’s UserDefinedType mechanism and
Apache Arrow’s ExtensionType — both offer maximum flexibility while keeping
the core engine clean and extensible.
* 2. Geospatial in Java and Python is a dependency hell.- How has Parquet
solved that problem, then?*
Exactly — this problem is not fully solved by Parquet. While the Parquet
spec now includes a definition for geospatial types, it’s more of a vision
than a complete, production-ready solution. Many aspects of the spec are
not yet implemented in Spark. In fact, the spec represents a compromise
among multiple vendors (e.g., BigQuery, Snowflake), and many design choices
are not aligned with Spark’s architecture or ecosystem.
For example:
• The CRS property in the spec uses a PROJJSON string, which currently only
has a C++ implementation — there is no Java implementation available.
• The edge interpolation algorithms (e.g., for great-circle arcs) mentioned
in the spec also only exist in C++ libraries.
• Handling of antimeridian-crossing geometries is another complex topic
that isn’t addressed in Spark today.
The Sedona community is actively working on solutions — either building
Java equivalents for these features or creating workarounds. These are
deeply domain-specific efforts and often require non-trivial geospatial
expertise.
We are currently contributing a Java implementation of the Parquet Geo
format here: https://github.com/apache/parquet-java/pull/2971
In Python, geospatial manipulation depends on libraries like Shapely and
GeoPandas, which evolve quickly and frequently introduce breaking changes.
Sedona has invested significant effort to maintain compatibility and
stability for Python UDFs across these ecosystems.
If you haven’t encountered this kind of “dependency hell” while working on
geospatial projects with Spark, you may have been fortunate to deal with
relatively simple cases — e.g., only working with point data or simple
polygons.
That usually means:
1. All geometries are in a single CRS, typically WGS84 (SRID 4326)
2. No antimeridian-crossing geometries
3. No need for high-precision distance calculations or spherical geometry
4. No need to handle topology or wraparound issues
If that’s the case, then Spark already works fine as-is for your use case —
so why complicate it?
*3. Sedona already supports Geo fully in (Geo)Parquet.- The default format
in Spark is Parquet, and Parquet now natively supports these types. Are we
going to force users to add Sedona?*
While opinions may vary, I would encourage users to adopt a solution like
Apache Sedona that laser focuses on geospatial. Sedona provides
comprehensive, step-by-step tutorials on how to handle geospatial
dependencies across major platforms — including Databricks, AWS EMR,
Microsoft Azure, and Google Cloud. We’re also actively collaborating with
cloud providers to bundle Sedona natively into their offerings, making it
even easier for users to get started.
That said, I generally share the same perspective — if the Spark community
believes it would benefit from having basic geospatial support built in,
the Sedona community would be happy to collaborate on this effort. We’re
open to contributing the necessary functionality and, if appropriate,
having Spark depend on Sedona directly to avoid reinvention.
Thanks,
JIa
On Sat, Mar 29, 2025 at 11:02 PM Ángel Álvarez Pascua <
angel.alvarez.pascua@gmail.com> wrote:
>
> * 1. Domain types evolve quickly.*
> It has taken years for Parquet to include these new types in its format...
> We could evolve alongside Parquet. Unfortunately, Spark is not known for
> upgrading its dependencies quickly.
>
> * 2. Geospatial in Java and Python is a dependency hell.*
> How has Parquet solved that problem, then? I don't recall experiencing any
> "dependency hell" when working on geospatial projects with Spark, to be
> honest. Besides, Spark already includes Parquet as a dependency, so...
> where is the problem?
>
> *3. Sedona already supports Geo fully in (Geo)Parquet.*
> The default format in Spark is Parquet, and Parquet now natively supports
> these types. Are we going to force users to add Sedona (along with all its
> third-party dependencies, I assume) to their projects just for reading,
> writing, and performing basic operations with these types?
>
> Anyway, let's vote and see...
>
> El sáb, 29 mar 2025 a las 22:41, Reynold Xin (<rx...@databricks.com.invalid>)
> escribió:
>
>> While I don’t think Spark should become a super specialized geospatial
>> processing engine, I don’t think it makes sense to focus *only* on reading
>> and writing from storage. Geospatial is a pretty common and fundamental
>> capability of analytics systems and virtually every mature and popular
>> analytics systems, be it open source or proprietary, storage or query, has
>> some basic geospatial data type and support. Adding geospatial type and
>> some basic expressions is such a no brainer.
>>
>> On Sat, Mar 29, 2025 at 2:27 PM Jia Yu <ji...@apache.org> wrote:
>>
>>> Hi Wenchen, Menelaos and Szehon,
>>>
>>> Thanks for the clarification — I’m glad to hear the primary motivation
>>> of this SPIP is focused on reading and writing geospatial data with Parquet
>>> and Iceberg. That’s an important goal, and I want to highlight that this
>>> problem is being solved by the Apache Sedona community.
>>>
>>> Since the primary motivation here is Parquet-level support, I suggest
>>> shifting the focus of this discussion toward enabling geo support in Spark
>>> Parquet DataSource rather than introducing core types.
>>>
>>> ** Why Spark Should Avoid Hardcoding Domain-Specific Types like geo
>>> types **
>>>
>>> 1. Domain types evolve quickly.
>>>
>>> In geospatial, we already have geometry, geography, raster, trajectory,
>>> point clouds — and the list keeps growing. In AI/ML, we’re seeing tensors,
>>> vectors, and multi-dimensional arrays. Spark’s strength has always been in
>>> its general-purpose architecture and extensibility. Introducing hardcoded
>>> support for fast-changing domain-specific types risks long-term maintenance
>>> issues and eventual incompatibility with emerging standards.
>>>
>>> 2. Geospatial in Java and Python is a dependency hell.
>>>
>>> There are multiple competing geometry libraries with incompatible APIs.
>>> No widely adopted Java library supports geography types. The most
>>> authoritative CRS dataset (EPSG) is not Apache-compatible. The json format
>>> for CRS definitions (projjson) is only fully supported in PROJ, a C++
>>> library without a Java equivalent and no formal OGC standard status. On the
>>> Python side, this might involve Shapely and GeoPandas dependencies.
>>>
>>> 3. Sedona already supports Geo fully in (Geo)Parquet.
>>>
>>> Sedona has supported reading, writing, metadata preservation, and data
>>> skipping for GeoParquet (predecessor of Parquet Geo) for over two years
>>> [2][3]. These features are production-tested and widely used.
>>>
>>> ** Proposed Path Forward: Geo Support via Spark Extensions **
>>>
>>> To enable seamless Parquet integration without burdening Spark core,
>>> here are two options:
>>>
>>> Option 1:
>>> Sedona offers a dedicated `parquet-geo` DataSource that handles type
>>> encoding, metadata, and data skipping. No changes to Spark are required.
>>> This is already underway and will be maintained by the Sedona community to
>>> keep up with the evolving Geo standards.
>>>
>>> Option 2:
>>> Spark provides hooks to inject:
>>> - custom logical types / user-defined types (UDTs)
>>> - custom statistics and filter pushdowns
>>> Sedona can then extend the built-in `parquet` DataSource to integrate
>>> geo type metadata, predicate pushdown, and serialization seamlessly.
>>>
>>> For Iceberg, we’ve already published a proof-of-concept connector [4]
>>> showing Sedona, Spark, and Iceberg working together without any Spark core
>>> changes [5].
>>>
>>> ** On the Bigger Picture **
>>>
>>> I also agree with your long-term vision. I believe Spark is on the path
>>> to becoming a foundational compute engine — much like Postgres or Pandas —
>>> where the core remains focused and stable, while powerful domain-specific
>>> capabilities emerge from its ecosystem.
>>>
>>> To support this future, Spark could prioritize flexible extension hooks
>>> so that third-party libraries can thrive — just like we’ve seen with
>>> PostGIS, pgvector, TimescaleDB in the Postgres ecosystem, and GeoPandas in
>>> the Pandas ecosystem.
>>>
>>> Sedona is following this model by building geospatial support around
>>> Spark — not inside it — and we’d love to continue collaborating in this
>>> spirit.
>>>
>>> Happy to work together on providing Geo support in Parquet!
>>>
>>> Best,
>>> Jia
>>>
>>> References
>>>
>>> [1] GeoParquet project:
>>> https://github.com/opengeospatial/geoparquet
>>>
>>> [2] Sedona’s GeoParquet DataSource implementation:
>>>
>>> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet
>>>
>>> [3] Sedona’s GeoParquet documentation:
>>> https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
>>>
>>> [4] Sedona-Iceberg connector (PoC):
>>> https://github.com/wherobots/sedona-iceberg-connector
>>>
>>> [5] Spark-Sedona-Iceberg working example:
>>>
>>> https://github.com/wherobots/sedona-iceberg-connector/blob/main/src/test/scala/com/wherobots/sedona/TestGeospatial.scala#L53
>>>
>>>
>>> On 2025/03/29 19:27:08 Menelaos Karavelas wrote:
>>> > To continue along the line of thought of Szehon:
>>> >
>>> > I am really excited that the Parquet and Iceberg communities have
>>> adopted geospatial logical types and of course I am grateful for the work
>>> put in that direction.
>>> >
>>> > As both Wenchen and Szehon pointed out in their own way, the goal is
>>> to have minimal support in Spark, as a common platform, for these types.
>>> >
>>> > To be more specific and explicit: The proposal scope is to add support
>>> for reading/writing to Parquet, based on the new standard, as well as
>>> adding the types as built-in types in Spark to complement the storage
>>> support. The few ST expressions that are in the proposal are what seem to
>>> be the minimal set of expressions needed to support working with geospatial
>>> values in the Spark engine in a meaningful way.
>>> >
>>> > Best,
>>> >
>>> > Menelaos
>>> >
>>> >
>>> > > On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com>
>>> wrote:
>>> > >
>>> > > Thank you Menelaos, will do!
>>> > >
>>> > > To give a little background, Jia and Sedona community, also
>>> GeoParquet community, and others really put much effort contributing to
>>> defining the Parquet and Iceberg geo types, which couldn't be done without
>>> their experience and help!
>>> > >
>>> > > But I do agree with Wenchen , now that the types are in most common
>>> data sources in ecosystem , I think Apache Spark as a common platform needs
>>> to have this type definition for inter-op, otherwise users of vanilla Spark
>>> cannot work with those data sources with stored geospatial data. (Imo a
>>> similar rationale in adding timestamp nano in the other ongoing SPIP.).
>>> > >
>>> > > And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment
>>> the ecosystem by implementing Sedona’s advanced geospatial analytic tech in
>>> Spark itself, which you may be right belongs in pluggable frameworks.
>>> Menelaus may explain more about the SPIP goal.
>>> > >
>>> > > I do hope there can be more collaboration across communities (like
>>> in Iceberg/Parquet collaboration) in getting Sedona community’s experience
>>> in making sure these type definitions are optimal , and compatible for
>>> Sedona.
>>> > >
>>> > > Thanks!
>>> > > Szehon
>>> > >
>>> > >
>>> > >> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <
>>> menelaos.karavelas@gmail.com> wrote:
>>> > >>
>>> > >>
>>> > >> Hello Szehon,
>>> > >>
>>> > >> I just created a Google doc and also linked it in the JIRA:
>>> > >>
>>> > >>
>>> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
>>> > >>
>>> > >> Please feel free to comment on it.
>>> > >>
>>> > >> Best,
>>> > >>
>>> > >> Menelaos
>>> > >>
>>> > >>
>>> > >>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com>
>>> wrote:
>>> > >>>
>>> > >>> Thanks Menelaos, this is exciting ! Is there a google doc we can
>>> comment, or just on the JIRA?
>>> > >>>
>>> > >>> Thanks
>>> > >>> Szehon
>>> > >>>
>>> > >>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <
>>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
>>> wrote:
>>> > >>>> Sorry, I only had a quick look at the proposal, looked for WKT
>>> and didn't find anything.
>>> > >>>>
>>> > >>>> It's been years since I worked on geospatial projects and I'm not
>>> an expert (at all). Maybe starting with something simple but useful like
>>> conversion WKT<=>WKB?
>>> > >>>>
>>> > >>>>
>>> > >>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <
>>> menelaos.karavelas@gmail.com <ma...@gmail.com>>
>>> escribió:
>>> > >>>>> In the SPIP Jira the proposal is to add the expressions
>>> ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
>>> > >>>>> Is there anything else that you think should be added?
>>> > >>>>>
>>> > >>>>> Regarding WKT, what do you think should be added?
>>> > >>>>>
>>> > >>>>> - Menelaos
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <
>>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
>>> wrote:
>>> > >>>>>>
>>> > >>>>>> What about adding support for WKT <
>>> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB
>>> <
>>> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary
>>> >?
>>> > >>>>>>
>>> > >>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<
>>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>)
>>> escribió:
>>> > >>>>>>> +1 (non-binding)
>>> > >>>>>>>
>>> > >>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <
>>> menelaos.karavelas@gmail.com <ma...@gmail.com>>
>>> escribió:
>>> > >>>>>>>> Dear Spark community,
>>> > >>>>>>>>
>>> > >>>>>>>> I would like to propose the addition of new geospatial data
>>> types (GEOMETRY and GEOGRAPHY) which represent geospatial values as
>>> recently added as new logical types in the Parquet specification.
>>> > >>>>>>>>
>>> > >>>>>>>> The new types should improve Spark’s ability to read the new
>>> Parquet logical types and perform some minimal meaningful operations on
>>> them.
>>> > >>>>>>>>
>>> > >>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>> > >>>>>>>>
>>> > >>>>>>>> Looking forward to your comments and feedback.
>>> > >>>>>>>>
>>> > >>>>>>>>
>>> > >>>>>>>> Best regards,
>>> > >>>>>>>>
>>> > >>>>>>>> Menelaos Karavelas
>>> > >>>>>>>>
>>> > >>>>>
>>> > >>
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Ángel Álvarez Pascua <an...@gmail.com>.
* 1. Domain types evolve quickly.*
It has taken years for Parquet to include these new types in its format...
We could evolve alongside Parquet. Unfortunately, Spark is not known for
upgrading its dependencies quickly.
* 2. Geospatial in Java and Python is a dependency hell.*
How has Parquet solved that problem, then? I don't recall experiencing any
"dependency hell" when working on geospatial projects with Spark, to be
honest. Besides, Spark already includes Parquet as a dependency, so...
where is the problem?
*3. Sedona already supports Geo fully in (Geo)Parquet.*
The default format in Spark is Parquet, and Parquet now natively supports
these types. Are we going to force users to add Sedona (along with all its
third-party dependencies, I assume) to their projects just for reading,
writing, and performing basic operations with these types?
Anyway, let's vote and see...
El sáb, 29 mar 2025 a las 22:41, Reynold Xin (<rx...@databricks.com.invalid>)
escribió:
> While I don’t think Spark should become a super specialized geospatial
> processing engine, I don’t think it makes sense to focus *only* on reading
> and writing from storage. Geospatial is a pretty common and fundamental
> capability of analytics systems and virtually every mature and popular
> analytics systems, be it open source or proprietary, storage or query, has
> some basic geospatial data type and support. Adding geospatial type and
> some basic expressions is such a no brainer.
>
> On Sat, Mar 29, 2025 at 2:27 PM Jia Yu <ji...@apache.org> wrote:
>
>> Hi Wenchen, Menelaos and Szehon,
>>
>> Thanks for the clarification — I’m glad to hear the primary motivation of
>> this SPIP is focused on reading and writing geospatial data with Parquet
>> and Iceberg. That’s an important goal, and I want to highlight that this
>> problem is being solved by the Apache Sedona community.
>>
>> Since the primary motivation here is Parquet-level support, I suggest
>> shifting the focus of this discussion toward enabling geo support in Spark
>> Parquet DataSource rather than introducing core types.
>>
>> ** Why Spark Should Avoid Hardcoding Domain-Specific Types like geo types
>> **
>>
>> 1. Domain types evolve quickly.
>>
>> In geospatial, we already have geometry, geography, raster, trajectory,
>> point clouds — and the list keeps growing. In AI/ML, we’re seeing tensors,
>> vectors, and multi-dimensional arrays. Spark’s strength has always been in
>> its general-purpose architecture and extensibility. Introducing hardcoded
>> support for fast-changing domain-specific types risks long-term maintenance
>> issues and eventual incompatibility with emerging standards.
>>
>> 2. Geospatial in Java and Python is a dependency hell.
>>
>> There are multiple competing geometry libraries with incompatible APIs.
>> No widely adopted Java library supports geography types. The most
>> authoritative CRS dataset (EPSG) is not Apache-compatible. The json format
>> for CRS definitions (projjson) is only fully supported in PROJ, a C++
>> library without a Java equivalent and no formal OGC standard status. On the
>> Python side, this might involve Shapely and GeoPandas dependencies.
>>
>> 3. Sedona already supports Geo fully in (Geo)Parquet.
>>
>> Sedona has supported reading, writing, metadata preservation, and data
>> skipping for GeoParquet (predecessor of Parquet Geo) for over two years
>> [2][3]. These features are production-tested and widely used.
>>
>> ** Proposed Path Forward: Geo Support via Spark Extensions **
>>
>> To enable seamless Parquet integration without burdening Spark core, here
>> are two options:
>>
>> Option 1:
>> Sedona offers a dedicated `parquet-geo` DataSource that handles type
>> encoding, metadata, and data skipping. No changes to Spark are required.
>> This is already underway and will be maintained by the Sedona community to
>> keep up with the evolving Geo standards.
>>
>> Option 2:
>> Spark provides hooks to inject:
>> - custom logical types / user-defined types (UDTs)
>> - custom statistics and filter pushdowns
>> Sedona can then extend the built-in `parquet` DataSource to integrate geo
>> type metadata, predicate pushdown, and serialization seamlessly.
>>
>> For Iceberg, we’ve already published a proof-of-concept connector [4]
>> showing Sedona, Spark, and Iceberg working together without any Spark core
>> changes [5].
>>
>> ** On the Bigger Picture **
>>
>> I also agree with your long-term vision. I believe Spark is on the path
>> to becoming a foundational compute engine — much like Postgres or Pandas —
>> where the core remains focused and stable, while powerful domain-specific
>> capabilities emerge from its ecosystem.
>>
>> To support this future, Spark could prioritize flexible extension hooks
>> so that third-party libraries can thrive — just like we’ve seen with
>> PostGIS, pgvector, TimescaleDB in the Postgres ecosystem, and GeoPandas in
>> the Pandas ecosystem.
>>
>> Sedona is following this model by building geospatial support around
>> Spark — not inside it — and we’d love to continue collaborating in this
>> spirit.
>>
>> Happy to work together on providing Geo support in Parquet!
>>
>> Best,
>> Jia
>>
>> References
>>
>> [1] GeoParquet project:
>> https://github.com/opengeospatial/geoparquet
>>
>> [2] Sedona’s GeoParquet DataSource implementation:
>>
>> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet
>>
>> [3] Sedona’s GeoParquet documentation:
>> https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
>>
>> [4] Sedona-Iceberg connector (PoC):
>> https://github.com/wherobots/sedona-iceberg-connector
>>
>> [5] Spark-Sedona-Iceberg working example:
>>
>> https://github.com/wherobots/sedona-iceberg-connector/blob/main/src/test/scala/com/wherobots/sedona/TestGeospatial.scala#L53
>>
>>
>> On 2025/03/29 19:27:08 Menelaos Karavelas wrote:
>> > To continue along the line of thought of Szehon:
>> >
>> > I am really excited that the Parquet and Iceberg communities have
>> adopted geospatial logical types and of course I am grateful for the work
>> put in that direction.
>> >
>> > As both Wenchen and Szehon pointed out in their own way, the goal is to
>> have minimal support in Spark, as a common platform, for these types.
>> >
>> > To be more specific and explicit: The proposal scope is to add support
>> for reading/writing to Parquet, based on the new standard, as well as
>> adding the types as built-in types in Spark to complement the storage
>> support. The few ST expressions that are in the proposal are what seem to
>> be the minimal set of expressions needed to support working with geospatial
>> values in the Spark engine in a meaningful way.
>> >
>> > Best,
>> >
>> > Menelaos
>> >
>> >
>> > > On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com>
>> wrote:
>> > >
>> > > Thank you Menelaos, will do!
>> > >
>> > > To give a little background, Jia and Sedona community, also
>> GeoParquet community, and others really put much effort contributing to
>> defining the Parquet and Iceberg geo types, which couldn't be done without
>> their experience and help!
>> > >
>> > > But I do agree with Wenchen , now that the types are in most common
>> data sources in ecosystem , I think Apache Spark as a common platform needs
>> to have this type definition for inter-op, otherwise users of vanilla Spark
>> cannot work with those data sources with stored geospatial data. (Imo a
>> similar rationale in adding timestamp nano in the other ongoing SPIP.).
>> > >
>> > > And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment
>> the ecosystem by implementing Sedona’s advanced geospatial analytic tech in
>> Spark itself, which you may be right belongs in pluggable frameworks.
>> Menelaus may explain more about the SPIP goal.
>> > >
>> > > I do hope there can be more collaboration across communities (like in
>> Iceberg/Parquet collaboration) in getting Sedona community’s experience in
>> making sure these type definitions are optimal , and compatible for Sedona.
>> > >
>> > > Thanks!
>> > > Szehon
>> > >
>> > >
>> > >> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <
>> menelaos.karavelas@gmail.com> wrote:
>> > >>
>> > >>
>> > >> Hello Szehon,
>> > >>
>> > >> I just created a Google doc and also linked it in the JIRA:
>> > >>
>> > >>
>> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
>> > >>
>> > >> Please feel free to comment on it.
>> > >>
>> > >> Best,
>> > >>
>> > >> Menelaos
>> > >>
>> > >>
>> > >>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com>
>> wrote:
>> > >>>
>> > >>> Thanks Menelaos, this is exciting ! Is there a google doc we can
>> comment, or just on the JIRA?
>> > >>>
>> > >>> Thanks
>> > >>> Szehon
>> > >>>
>> > >>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <
>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
>> wrote:
>> > >>>> Sorry, I only had a quick look at the proposal, looked for WKT and
>> didn't find anything.
>> > >>>>
>> > >>>> It's been years since I worked on geospatial projects and I'm not
>> an expert (at all). Maybe starting with something simple but useful like
>> conversion WKT<=>WKB?
>> > >>>>
>> > >>>>
>> > >>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <
>> menelaos.karavelas@gmail.com <ma...@gmail.com>>
>> escribió:
>> > >>>>> In the SPIP Jira the proposal is to add the expressions
>> ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
>> > >>>>> Is there anything else that you think should be added?
>> > >>>>>
>> > >>>>> Regarding WKT, what do you think should be added?
>> > >>>>>
>> > >>>>> - Menelaos
>> > >>>>>
>> > >>>>>
>> > >>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <
>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
>> wrote:
>> > >>>>>>
>> > >>>>>> What about adding support for WKT <
>> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB
>> <
>> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary
>> >?
>> > >>>>>>
>> > >>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<
>> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>)
>> escribió:
>> > >>>>>>> +1 (non-binding)
>> > >>>>>>>
>> > >>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <
>> menelaos.karavelas@gmail.com <ma...@gmail.com>>
>> escribió:
>> > >>>>>>>> Dear Spark community,
>> > >>>>>>>>
>> > >>>>>>>> I would like to propose the addition of new geospatial data
>> types (GEOMETRY and GEOGRAPHY) which represent geospatial values as
>> recently added as new logical types in the Parquet specification.
>> > >>>>>>>>
>> > >>>>>>>> The new types should improve Spark’s ability to read the new
>> Parquet logical types and perform some minimal meaningful operations on
>> them.
>> > >>>>>>>>
>> > >>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>> > >>>>>>>>
>> > >>>>>>>> Looking forward to your comments and feedback.
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> Best regards,
>> > >>>>>>>>
>> > >>>>>>>> Menelaos Karavelas
>> > >>>>>>>>
>> > >>>>>
>> > >>
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Reynold Xin <rx...@databricks.com.INVALID>.
While I don’t think Spark should become a super specialized geospatial
processing engine, I don’t think it makes sense to focus *only* on reading
and writing from storage. Geospatial is a pretty common and fundamental
capability of analytics systems and virtually every mature and popular
analytics systems, be it open source or proprietary, storage or query, has
some basic geospatial data type and support. Adding geospatial type and
some basic expressions is such a no brainer.
On Sat, Mar 29, 2025 at 2:27 PM Jia Yu <ji...@apache.org> wrote:
> Hi Wenchen, Menelaos and Szehon,
>
> Thanks for the clarification — I’m glad to hear the primary motivation of
> this SPIP is focused on reading and writing geospatial data with Parquet
> and Iceberg. That’s an important goal, and I want to highlight that this
> problem is being solved by the Apache Sedona community.
>
> Since the primary motivation here is Parquet-level support, I suggest
> shifting the focus of this discussion toward enabling geo support in Spark
> Parquet DataSource rather than introducing core types.
>
> ** Why Spark Should Avoid Hardcoding Domain-Specific Types like geo types
> **
>
> 1. Domain types evolve quickly.
>
> In geospatial, we already have geometry, geography, raster, trajectory,
> point clouds — and the list keeps growing. In AI/ML, we’re seeing tensors,
> vectors, and multi-dimensional arrays. Spark’s strength has always been in
> its general-purpose architecture and extensibility. Introducing hardcoded
> support for fast-changing domain-specific types risks long-term maintenance
> issues and eventual incompatibility with emerging standards.
>
> 2. Geospatial in Java and Python is a dependency hell.
>
> There are multiple competing geometry libraries with incompatible APIs. No
> widely adopted Java library supports geography types. The most
> authoritative CRS dataset (EPSG) is not Apache-compatible. The json format
> for CRS definitions (projjson) is only fully supported in PROJ, a C++
> library without a Java equivalent and no formal OGC standard status. On the
> Python side, this might involve Shapely and GeoPandas dependencies.
>
> 3. Sedona already supports Geo fully in (Geo)Parquet.
>
> Sedona has supported reading, writing, metadata preservation, and data
> skipping for GeoParquet (predecessor of Parquet Geo) for over two years
> [2][3]. These features are production-tested and widely used.
>
> ** Proposed Path Forward: Geo Support via Spark Extensions **
>
> To enable seamless Parquet integration without burdening Spark core, here
> are two options:
>
> Option 1:
> Sedona offers a dedicated `parquet-geo` DataSource that handles type
> encoding, metadata, and data skipping. No changes to Spark are required.
> This is already underway and will be maintained by the Sedona community to
> keep up with the evolving Geo standards.
>
> Option 2:
> Spark provides hooks to inject:
> - custom logical types / user-defined types (UDTs)
> - custom statistics and filter pushdowns
> Sedona can then extend the built-in `parquet` DataSource to integrate geo
> type metadata, predicate pushdown, and serialization seamlessly.
>
> For Iceberg, we’ve already published a proof-of-concept connector [4]
> showing Sedona, Spark, and Iceberg working together without any Spark core
> changes [5].
>
> ** On the Bigger Picture **
>
> I also agree with your long-term vision. I believe Spark is on the path to
> becoming a foundational compute engine — much like Postgres or Pandas —
> where the core remains focused and stable, while powerful domain-specific
> capabilities emerge from its ecosystem.
>
> To support this future, Spark could prioritize flexible extension hooks so
> that third-party libraries can thrive — just like we’ve seen with PostGIS,
> pgvector, TimescaleDB in the Postgres ecosystem, and GeoPandas in the
> Pandas ecosystem.
>
> Sedona is following this model by building geospatial support around Spark
> — not inside it — and we’d love to continue collaborating in this spirit.
>
> Happy to work together on providing Geo support in Parquet!
>
> Best,
> Jia
>
> References
>
> [1] GeoParquet project:
> https://github.com/opengeospatial/geoparquet
>
> [2] Sedona’s GeoParquet DataSource implementation:
>
> https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet
>
> [3] Sedona’s GeoParquet documentation:
> https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
>
> [4] Sedona-Iceberg connector (PoC):
> https://github.com/wherobots/sedona-iceberg-connector
>
> [5] Spark-Sedona-Iceberg working example:
>
> https://github.com/wherobots/sedona-iceberg-connector/blob/main/src/test/scala/com/wherobots/sedona/TestGeospatial.scala#L53
>
>
> On 2025/03/29 19:27:08 Menelaos Karavelas wrote:
> > To continue along the line of thought of Szehon:
> >
> > I am really excited that the Parquet and Iceberg communities have
> adopted geospatial logical types and of course I am grateful for the work
> put in that direction.
> >
> > As both Wenchen and Szehon pointed out in their own way, the goal is to
> have minimal support in Spark, as a common platform, for these types.
> >
> > To be more specific and explicit: The proposal scope is to add support
> for reading/writing to Parquet, based on the new standard, as well as
> adding the types as built-in types in Spark to complement the storage
> support. The few ST expressions that are in the proposal are what seem to
> be the minimal set of expressions needed to support working with geospatial
> values in the Spark engine in a meaningful way.
> >
> > Best,
> >
> > Menelaos
> >
> >
> > > On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com>
> wrote:
> > >
> > > Thank you Menelaos, will do!
> > >
> > > To give a little background, Jia and Sedona community, also GeoParquet
> community, and others really put much effort contributing to defining the
> Parquet and Iceberg geo types, which couldn't be done without their
> experience and help!
> > >
> > > But I do agree with Wenchen , now that the types are in most common
> data sources in ecosystem , I think Apache Spark as a common platform needs
> to have this type definition for inter-op, otherwise users of vanilla Spark
> cannot work with those data sources with stored geospatial data. (Imo a
> similar rationale in adding timestamp nano in the other ongoing SPIP.).
> > >
> > > And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment
> the ecosystem by implementing Sedona’s advanced geospatial analytic tech in
> Spark itself, which you may be right belongs in pluggable frameworks.
> Menelaus may explain more about the SPIP goal.
> > >
> > > I do hope there can be more collaboration across communities (like in
> Iceberg/Parquet collaboration) in getting Sedona community’s experience in
> making sure these type definitions are optimal , and compatible for Sedona.
> > >
> > > Thanks!
> > > Szehon
> > >
> > >
> > >> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <
> menelaos.karavelas@gmail.com> wrote:
> > >>
> > >>
> > >> Hello Szehon,
> > >>
> > >> I just created a Google doc and also linked it in the JIRA:
> > >>
> > >>
> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
> > >>
> > >> Please feel free to comment on it.
> > >>
> > >> Best,
> > >>
> > >> Menelaos
> > >>
> > >>
> > >>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com>
> wrote:
> > >>>
> > >>> Thanks Menelaos, this is exciting ! Is there a google doc we can
> comment, or just on the JIRA?
> > >>>
> > >>> Thanks
> > >>> Szehon
> > >>>
> > >>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <
> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
> wrote:
> > >>>> Sorry, I only had a quick look at the proposal, looked for WKT and
> didn't find anything.
> > >>>>
> > >>>> It's been years since I worked on geospatial projects and I'm not
> an expert (at all). Maybe starting with something simple but useful like
> conversion WKT<=>WKB?
> > >>>>
> > >>>>
> > >>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <
> menelaos.karavelas@gmail.com <ma...@gmail.com>>
> escribió:
> > >>>>> In the SPIP Jira the proposal is to add the expressions
> ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
> > >>>>> Is there anything else that you think should be added?
> > >>>>>
> > >>>>> Regarding WKT, what do you think should be added?
> > >>>>>
> > >>>>> - Menelaos
> > >>>>>
> > >>>>>
> > >>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <
> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>
> wrote:
> > >>>>>>
> > >>>>>> What about adding support for WKT <
> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB
> <
> https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary
> >?
> > >>>>>>
> > >>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<
> angel.alvarez.pascua@gmail.com <ma...@gmail.com>>)
> escribió:
> > >>>>>>> +1 (non-binding)
> > >>>>>>>
> > >>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <
> menelaos.karavelas@gmail.com <ma...@gmail.com>>
> escribió:
> > >>>>>>>> Dear Spark community,
> > >>>>>>>>
> > >>>>>>>> I would like to propose the addition of new geospatial data
> types (GEOMETRY and GEOGRAPHY) which represent geospatial values as
> recently added as new logical types in the Parquet specification.
> > >>>>>>>>
> > >>>>>>>> The new types should improve Spark’s ability to read the new
> Parquet logical types and perform some minimal meaningful operations on
> them.
> > >>>>>>>>
> > >>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
> > >>>>>>>>
> > >>>>>>>> Looking forward to your comments and feedback.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Best regards,
> > >>>>>>>>
> > >>>>>>>> Menelaos Karavelas
> > >>>>>>>>
> > >>>>>
> > >>
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Jia Yu <ji...@apache.org>.
Hi Wenchen, Menelaos and Szehon,
Thanks for the clarification — I’m glad to hear the primary motivation of this SPIP is focused on reading and writing geospatial data with Parquet and Iceberg. That’s an important goal, and I want to highlight that this problem is being solved by the Apache Sedona community.
Since the primary motivation here is Parquet-level support, I suggest shifting the focus of this discussion toward enabling geo support in Spark Parquet DataSource rather than introducing core types.
** Why Spark Should Avoid Hardcoding Domain-Specific Types like geo types **
1. Domain types evolve quickly.
In geospatial, we already have geometry, geography, raster, trajectory, point clouds — and the list keeps growing. In AI/ML, we’re seeing tensors, vectors, and multi-dimensional arrays. Spark’s strength has always been in its general-purpose architecture and extensibility. Introducing hardcoded support for fast-changing domain-specific types risks long-term maintenance issues and eventual incompatibility with emerging standards.
2. Geospatial in Java and Python is a dependency hell.
There are multiple competing geometry libraries with incompatible APIs. No widely adopted Java library supports geography types. The most authoritative CRS dataset (EPSG) is not Apache-compatible. The json format for CRS definitions (projjson) is only fully supported in PROJ, a C++ library without a Java equivalent and no formal OGC standard status. On the Python side, this might involve Shapely and GeoPandas dependencies.
3. Sedona already supports Geo fully in (Geo)Parquet.
Sedona has supported reading, writing, metadata preservation, and data skipping for GeoParquet (predecessor of Parquet Geo) for over two years [2][3]. These features are production-tested and widely used.
** Proposed Path Forward: Geo Support via Spark Extensions **
To enable seamless Parquet integration without burdening Spark core, here are two options:
Option 1:
Sedona offers a dedicated `parquet-geo` DataSource that handles type encoding, metadata, and data skipping. No changes to Spark are required. This is already underway and will be maintained by the Sedona community to keep up with the evolving Geo standards.
Option 2:
Spark provides hooks to inject:
- custom logical types / user-defined types (UDTs)
- custom statistics and filter pushdowns
Sedona can then extend the built-in `parquet` DataSource to integrate geo type metadata, predicate pushdown, and serialization seamlessly.
For Iceberg, we’ve already published a proof-of-concept connector [4] showing Sedona, Spark, and Iceberg working together without any Spark core changes [5].
** On the Bigger Picture **
I also agree with your long-term vision. I believe Spark is on the path to becoming a foundational compute engine — much like Postgres or Pandas — where the core remains focused and stable, while powerful domain-specific capabilities emerge from its ecosystem.
To support this future, Spark could prioritize flexible extension hooks so that third-party libraries can thrive — just like we’ve seen with PostGIS, pgvector, TimescaleDB in the Postgres ecosystem, and GeoPandas in the Pandas ecosystem.
Sedona is following this model by building geospatial support around Spark — not inside it — and we’d love to continue collaborating in this spirit.
Happy to work together on providing Geo support in Parquet!
Best,
Jia
References
[1] GeoParquet project:
https://github.com/opengeospatial/geoparquet
[2] Sedona’s GeoParquet DataSource implementation:
https://github.com/apache/sedona/tree/master/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet
[3] Sedona’s GeoParquet documentation:
https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/
[4] Sedona-Iceberg connector (PoC):
https://github.com/wherobots/sedona-iceberg-connector
[5] Spark-Sedona-Iceberg working example:
https://github.com/wherobots/sedona-iceberg-connector/blob/main/src/test/scala/com/wherobots/sedona/TestGeospatial.scala#L53
On 2025/03/29 19:27:08 Menelaos Karavelas wrote:
> To continue along the line of thought of Szehon:
>
> I am really excited that the Parquet and Iceberg communities have adopted geospatial logical types and of course I am grateful for the work put in that direction.
>
> As both Wenchen and Szehon pointed out in their own way, the goal is to have minimal support in Spark, as a common platform, for these types.
>
> To be more specific and explicit: The proposal scope is to add support for reading/writing to Parquet, based on the new standard, as well as adding the types as built-in types in Spark to complement the storage support. The few ST expressions that are in the proposal are what seem to be the minimal set of expressions needed to support working with geospatial values in the Spark engine in a meaningful way.
>
> Best,
>
> Menelaos
>
>
> > On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com> wrote:
> >
> > Thank you Menelaos, will do!
> >
> > To give a little background, Jia and Sedona community, also GeoParquet community, and others really put much effort contributing to defining the Parquet and Iceberg geo types, which couldn't be done without their experience and help!
> >
> > But I do agree with Wenchen , now that the types are in most common data sources in ecosystem , I think Apache Spark as a common platform needs to have this type definition for inter-op, otherwise users of vanilla Spark cannot work with those data sources with stored geospatial data. (Imo a similar rationale in adding timestamp nano in the other ongoing SPIP.).
> >
> > And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment the ecosystem by implementing Sedona’s advanced geospatial analytic tech in Spark itself, which you may be right belongs in pluggable frameworks. Menelaus may explain more about the SPIP goal.
> >
> > I do hope there can be more collaboration across communities (like in Iceberg/Parquet collaboration) in getting Sedona community’s experience in making sure these type definitions are optimal , and compatible for Sedona.
> >
> > Thanks!
> > Szehon
> >
> >
> >> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <me...@gmail.com> wrote:
> >>
> >>
> >> Hello Szehon,
> >>
> >> I just created a Google doc and also linked it in the JIRA:
> >>
> >> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
> >>
> >> Please feel free to comment on it.
> >>
> >> Best,
> >>
> >> Menelaos
> >>
> >>
> >>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com> wrote:
> >>>
> >>> Thanks Menelaos, this is exciting ! Is there a google doc we can comment, or just on the JIRA?
> >>>
> >>> Thanks
> >>> Szehon
> >>>
> >>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
> >>>> Sorry, I only had a quick look at the proposal, looked for WKT and didn't find anything.
> >>>>
> >>>> It's been years since I worked on geospatial projects and I'm not an expert (at all). Maybe starting with something simple but useful like conversion WKT<=>WKB?
> >>>>
> >>>>
> >>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
> >>>>> In the SPIP Jira the proposal is to add the expressions ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
> >>>>> Is there anything else that you think should be added?
> >>>>>
> >>>>> Regarding WKT, what do you think should be added?
> >>>>>
> >>>>> - Menelaos
> >>>>>
> >>>>>
> >>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
> >>>>>>
> >>>>>> What about adding support for WKT <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>?
> >>>>>>
> >>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<angel.alvarez.pascua@gmail.com <ma...@gmail.com>>) escribió:
> >>>>>>> +1 (non-binding)
> >>>>>>>
> >>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
> >>>>>>>> Dear Spark community,
> >>>>>>>>
> >>>>>>>> I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
> >>>>>>>>
> >>>>>>>> The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
> >>>>>>>>
> >>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
> >>>>>>>>
> >>>>>>>> Looking forward to your comments and feedback.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>>
> >>>>>>>> Menelaos Karavelas
> >>>>>>>>
> >>>>>
> >>
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Menelaos Karavelas <me...@gmail.com>.
To continue along the line of thought of Szehon:
I am really excited that the Parquet and Iceberg communities have adopted geospatial logical types and of course I am grateful for the work put in that direction.
As both Wenchen and Szehon pointed out in their own way, the goal is to have minimal support in Spark, as a common platform, for these types.
To be more specific and explicit: The proposal scope is to add support for reading/writing to Parquet, based on the new standard, as well as adding the types as built-in types in Spark to complement the storage support. The few ST expressions that are in the proposal are what seem to be the minimal set of expressions needed to support working with geospatial values in the Spark engine in a meaningful way.
Best,
Menelaos
> On Mar 29, 2025, at 12:06 PM, Szehon Ho <sz...@gmail.com> wrote:
>
> Thank you Menelaos, will do!
>
> To give a little background, Jia and Sedona community, also GeoParquet community, and others really put much effort contributing to defining the Parquet and Iceberg geo types, which couldn't be done without their experience and help!
>
> But I do agree with Wenchen , now that the types are in most common data sources in ecosystem , I think Apache Spark as a common platform needs to have this type definition for inter-op, otherwise users of vanilla Spark cannot work with those data sources with stored geospatial data. (Imo a similar rationale in adding timestamp nano in the other ongoing SPIP.).
>
> And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment the ecosystem by implementing Sedona’s advanced geospatial analytic tech in Spark itself, which you may be right belongs in pluggable frameworks. Menelaus may explain more about the SPIP goal.
>
> I do hope there can be more collaboration across communities (like in Iceberg/Parquet collaboration) in getting Sedona community’s experience in making sure these type definitions are optimal , and compatible for Sedona.
>
> Thanks!
> Szehon
>
>
>> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <me...@gmail.com> wrote:
>>
>>
>> Hello Szehon,
>>
>> I just created a Google doc and also linked it in the JIRA:
>>
>> https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
>>
>> Please feel free to comment on it.
>>
>> Best,
>>
>> Menelaos
>>
>>
>>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com> wrote:
>>>
>>> Thanks Menelaos, this is exciting ! Is there a google doc we can comment, or just on the JIRA?
>>>
>>> Thanks
>>> Szehon
>>>
>>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
>>>> Sorry, I only had a quick look at the proposal, looked for WKT and didn't find anything.
>>>>
>>>> It's been years since I worked on geospatial projects and I'm not an expert (at all). Maybe starting with something simple but useful like conversion WKT<=>WKB?
>>>>
>>>>
>>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>>>>> In the SPIP Jira the proposal is to add the expressions ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
>>>>> Is there anything else that you think should be added?
>>>>>
>>>>> Regarding WKT, what do you think should be added?
>>>>>
>>>>> - Menelaos
>>>>>
>>>>>
>>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
>>>>>>
>>>>>> What about adding support for WKT <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>?
>>>>>>
>>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<angel.alvarez.pascua@gmail.com <ma...@gmail.com>>) escribió:
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>>>>>>>> Dear Spark community,
>>>>>>>>
>>>>>>>> I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
>>>>>>>>
>>>>>>>> The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
>>>>>>>>
>>>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>>>>>>>
>>>>>>>> Looking forward to your comments and feedback.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Menelaos Karavelas
>>>>>>>>
>>>>>
>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Szehon Ho <sz...@gmail.com>.
Thank you Menelaos, will do!
To give a little background, Jia and Sedona community, also GeoParquet
community, and others really put much effort contributing to defining the
Parquet and Iceberg geo types, which couldn't be done without their experience
and help!
But I do agree with Wenchen , now that the types are in most common data
sources in ecosystem , I think Apache Spark as a common platform needs to have
this type definition for inter-op, otherwise users of vanilla Spark cannot
work with those data sources with stored geospatial data. (Imo a similar
rationale in adding timestamp nano in the other ongoing SPIP.).
And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment the
ecosystem by implementing Sedona’s advanced geospatial analytic tech in Spark
itself, which you may be right belongs in pluggable frameworks. Menelaus may
explain more about the SPIP goal.
I do hope there can be more collaboration across communities (like in
Iceberg/Parquet collaboration) in getting Sedona community’s experience in
making sure these type definitions are optimal , and compatible for Sedona.
Thanks!
Szehon
> On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas
> <me...@gmail.com> wrote:
>
>
>
>
> Hello Szehon,
>
>
>
>
> I just created a Google doc and also linked it in the JIRA:
>
>
>
>
>
> [<AHkbwyJs6lPtVzUxjbrfqKw757d0H_JV74f3hT4Xk8HHTCOKpV1-cFqeKUenWcjs177-yLciZ4R4iDvShZpsD0GRgLD68tgCVRF5H-oamJM9f-iPFH77SvQ=w1200-h630-p.png>
> ---
> | [SPIP: Add geospatial types in
> Spark](https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0)[docs.google.com](https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0)
> ---
>
> ](https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0)
>
>
>
>
> Please feel free to comment on it.
>
>
>
>
> Best,
>
>
>
>
> Menelaos
>
>
>
>
>
>
>
>
>> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com> wrote:
>>
>>
>
>>
>> Thanks Menelaos, this is exciting ! Is there a google doc we can comment,
or just on the JIRA?
>>
>>
>
>>
>> Thanks
>>
>> Szehon
>>
>>
>
>>
>> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua
<[angel.alvarez.pascua@gmail.com](mailto:angel.alvarez.pascua@gmail.com)>
wrote:
>
>>
>>> Sorry, I only had a quick look at the proposal, looked for WKT and didn't
find anything.
>>>
>>>
>
>>>
>>> It's been years since I worked on geospatial projects and I'm not an
expert (at all). Maybe starting with something simple but useful like
conversion WKT<=>WKB?
>>>
>>>
>
>>>
>>>
>
>>>
>>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas
<[menelaos.karavelas@gmail.com](mailto:menelaos.karavelas@gmail.com)>
escribió:
>
>>>
>>>> In the SPIP Jira the proposal is to add the expressions ST_AsBinary,
ST_GeomFromWKB, and ST_GeogFromWKB.
>>>>
>>>> Is there anything else that you think should be added?
>
>>>>
>>>>
>
>>>>
>>>> Regarding WKT, what do you think should be added?
>>>>
>>>>
>
>>>>
>>>> \- Menelaos
>>>>
>>>>
>
>>>>
>>>>
>
>>>>
>>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua
<[angel.alvarez.pascua@gmail.com](mailto:angel.alvarez.pascua@gmail.com)>
wrote:
>>>>>
>>>>>
>
>>>>>
>>>>> What about adding support for [WKT](https://en.wikipedia.org/wiki/Well-
known_text_representation_of_geometry)/[WKB](https://en.wikipedia.org/wiki/Well-
known_text_representation_of_geometry#Well-known_binary)?
>>>>>
>>>>>
>
>>>>>
>>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua
(<[angel.alvarez.pascua@gmail.com](mailto:angel.alvarez.pascua@gmail.com)>)
escribió:
>
>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>>
>
>>>>>>
>>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas
<[menelaos.karavelas@gmail.com](mailto:menelaos.karavelas@gmail.com)>
escribió:
>
>>>>>>
>>>>>>> Dear Spark community,
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> I would like to propose the addition of new geospatial data types
(GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added
as new logical types in the Parquet specification.
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> The new types should improve Spark’s ability to read the new Parquet
logical types and perform some minimal meaningful operations on them.
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> SPIP: <https://issues.apache.org/jira/browse/SPARK-51658>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> Looking forward to your comments and feedback.
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>>
>
>>>>>>>
>>>>>>> Menelaos Karavelas
>>>>>>>
>>>>>>>
>
>>>>
>>>>
>
>
>
>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Menelaos Karavelas <me...@gmail.com>.
Hello Szehon,
I just created a Google doc and also linked it in the JIRA:
https://docs.google.com/document/d/1cYSNPGh95OjnpS0k_KDHGM9Ae3j-_0Wnc_eGBZL4D3w/edit?tab=t.0
SPIP: Add geospatial types in Spark
docs.google.com
Please feel free to comment on it.
Best,
Menelaos
> On Mar 28, 2025, at 2:19 PM, Szehon Ho <sz...@gmail.com> wrote:
>
> Thanks Menelaos, this is exciting ! Is there a google doc we can comment, or just on the JIRA?
>
> Thanks
> Szehon
>
> On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
>> Sorry, I only had a quick look at the proposal, looked for WKT and didn't find anything.
>>
>> It's been years since I worked on geospatial projects and I'm not an expert (at all). Maybe starting with something simple but useful like conversion WKT<=>WKB?
>>
>>
>> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>>> In the SPIP Jira the proposal is to add the expressions ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
>>> Is there anything else that you think should be added?
>>>
>>> Regarding WKT, what do you think should be added?
>>>
>>> - Menelaos
>>>
>>>
>>>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <angel.alvarez.pascua@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> What about adding support for WKT <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>?
>>>>
>>>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<angel.alvarez.pascua@gmail.com <ma...@gmail.com>>) escribió:
>>>>> +1 (non-binding)
>>>>>
>>>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>>>>>> Dear Spark community,
>>>>>>
>>>>>> I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
>>>>>>
>>>>>> The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
>>>>>>
>>>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>>>>>
>>>>>> Looking forward to your comments and feedback.
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Menelaos Karavelas
>>>>>>
>>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Szehon Ho <sz...@gmail.com>.
Thanks Menelaos, this is exciting ! Is there a google doc we can comment,
or just on the JIRA?
Thanks
Szehon
On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <
angel.alvarez.pascua@gmail.com> wrote:
> Sorry, I only had a quick look at the proposal, looked for WKT and didn't
> find anything.
>
> It's been years since I worked on geospatial projects and I'm not an
> expert (at all). Maybe starting with something simple but useful like
> conversion WKT<=>WKB?
>
>
> El vie, 28 mar 2025, 21:27, Menelaos Karavelas <
> menelaos.karavelas@gmail.com> escribió:
>
>> In the SPIP Jira the proposal is to add the expressions ST_AsBinary,
>> ST_GeomFromWKB, and ST_GeogFromWKB.
>> Is there anything else that you think should be added?
>>
>> Regarding WKT, what do you think should be added?
>>
>> - Menelaos
>>
>>
>> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <
>> angel.alvarez.pascua@gmail.com> wrote:
>>
>> What about adding support for WKT
>> <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>
>> /WKB
>> <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>
>> ?
>>
>> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<
>> angel.alvarez.pascua@gmail.com>) escribió:
>>
>>> +1 (non-binding)
>>>
>>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <
>>> menelaos.karavelas@gmail.com> escribió:
>>>
>>>> Dear Spark community,
>>>>
>>>> I would like to propose the addition of new geospatial data types
>>>> (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently
>>>> added as new logical types in the Parquet specification.
>>>>
>>>> The new types should improve Spark’s ability to read the new Parquet
>>>> logical types and perform some minimal meaningful operations on them.
>>>>
>>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>>>
>>>> Looking forward to your comments and feedback.
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Menelaos Karavelas
>>>>
>>>>
>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Ángel Álvarez Pascua <an...@gmail.com>.
Sorry, I only had a quick look at the proposal, looked for WKT and didn't
find anything.
It's been years since I worked on geospatial projects and I'm not an expert
(at all). Maybe starting with something simple but useful like conversion
WKT<=>WKB?
El vie, 28 mar 2025, 21:27, Menelaos Karavelas <me...@gmail.com>
escribió:
> In the SPIP Jira the proposal is to add the expressions ST_AsBinary,
> ST_GeomFromWKB, and ST_GeogFromWKB.
> Is there anything else that you think should be added?
>
> Regarding WKT, what do you think should be added?
>
> - Menelaos
>
>
> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <
> angel.alvarez.pascua@gmail.com> wrote:
>
> What about adding support for WKT
> <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>
> /WKB
> <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>
> ?
>
> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<
> angel.alvarez.pascua@gmail.com>) escribió:
>
>> +1 (non-binding)
>>
>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <
>> menelaos.karavelas@gmail.com> escribió:
>>
>>> Dear Spark community,
>>>
>>> I would like to propose the addition of new geospatial data types
>>> (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently
>>> added as new logical types in the Parquet specification.
>>>
>>> The new types should improve Spark’s ability to read the new Parquet
>>> logical types and perform some minimal meaningful operations on them.
>>>
>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>>
>>> Looking forward to your comments and feedback.
>>>
>>>
>>> Best regards,
>>>
>>> Menelaos Karavelas
>>>
>>>
>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Menelaos Karavelas <me...@gmail.com>.
In the SPIP Jira the proposal is to add the expressions ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
Is there anything else that you think should be added?
Regarding WKT, what do you think should be added?
- Menelaos
> On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <an...@gmail.com> wrote:
>
> What about adding support for WKT <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/WKB <https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>?
>
> El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<angel.alvarez.pascua@gmail.com <ma...@gmail.com>>) escribió:
>> +1 (non-binding)
>>
>> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <menelaos.karavelas@gmail.com <ma...@gmail.com>> escribió:
>>> Dear Spark community,
>>>
>>> I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.
>>>
>>> The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.
>>>
>>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>>
>>> Looking forward to your comments and feedback.
>>>
>>>
>>> Best regards,
>>>
>>> Menelaos Karavelas
>>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Ángel Álvarez Pascua <an...@gmail.com>.
What about adding support for WKT
<https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>/
WKB
<https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary>
?
El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<
angel.alvarez.pascua@gmail.com>) escribió:
> +1 (non-binding)
>
> El vie, 28 mar 2025, 18:48, Menelaos Karavelas <
> menelaos.karavelas@gmail.com> escribió:
>
>> Dear Spark community,
>>
>> I would like to propose the addition of new geospatial data types
>> (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently
>> added as new logical types in the Parquet specification.
>>
>> The new types should improve Spark’s ability to read the new Parquet
>> logical types and perform some minimal meaningful operations on them.
>>
>> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>>
>> Looking forward to your comments and feedback.
>>
>>
>> Best regards,
>>
>> Menelaos Karavelas
>>
>>
Re: [DISCUSS] SPIP: Add geospatial types to Spark
Posted by Ángel Álvarez Pascua <an...@gmail.com>.
+1 (non-binding)
El vie, 28 mar 2025, 18:48, Menelaos Karavelas <me...@gmail.com>
escribió:
> Dear Spark community,
>
> I would like to propose the addition of new geospatial data types
> (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently
> added as new logical types in the Parquet specification.
>
> The new types should improve Spark’s ability to read the new Parquet
> logical types and perform some minimal meaningful operations on them.
>
> SPIP: https://issues.apache.org/jira/browse/SPARK-51658
>
> Looking forward to your comments and feedback.
>
>
> Best regards,
>
> Menelaos Karavelas
>
>