You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "uros-db (via GitHub)" <gi...@apache.org> on 2026/01/28 18:08:25 UTC
[PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
uros-db opened a new pull request, #54040:
URL: https://github.com/apache/spark/pull/54040
### What changes were proposed in this pull request?
Enable reading Geometry and Geography data from Parquet files.
### Why are the changes needed?
Allowing users to read persisted geospatial data from Parquet format.
### Does this PR introduce _any_ user-facing change?
Yes, geo data can now be read from Parquet.
### How was this patch tested?
Added tests for reading GEOMETRY and GEOGRAPHY from Parquet.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #54040:
URL: https://github.com/apache/spark/pull/54040#discussion_r2806250676
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java:
##########
@@ -100,6 +102,48 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
readValues(total, c, rowId);
}
+ @Override
+ public void readGeometry(int total, WritableColumnVector c, int rowId) {
+ assert(c.dataType() instanceof GeometryType);
+ int srid = ((GeometryType) c.dataType()).srid();
+ readGeoData(total, c, rowId, srid, WKBToGeometryConverter.INSTANCE);
+ }
+
+ @Override
+ public void readGeography(int total, WritableColumnVector c, int rowId) {
+ assert(c.dataType() instanceof GeographyType);
+ int srid = ((GeographyType) c.dataType()).srid();
+ readGeoData(total, c, rowId, srid, WKBToGeographyConverter.INSTANCE);
+ }
+
+ private void readGeoData(int total, WritableColumnVector c, int rowId, int srid,
+ WKBConverterStrategy converter) {
+ for (int i = 0; i < total; i++) {
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ ByteBuffer suffix = suffixReader.getBytes(currentRow);
+ int suffixLength = suffix.limit() - suffix.position();
+ int length = prefixLength + suffixLength;
+
+ byte[] wkb = new byte[length];
+ if (prefixLength > 0) {
+ previous.get(wkb, 0, prefixLength);
+ }
+ suffix.get(wkb, prefixLength, suffixLength);
+
+ // Converts WKB into a physical representation of geometry/geography.
Review Comment:
So here's the main idea - it doesn't (and shouldn't) matter what the format is. Format itself can change and we don't guarantee/rely on it directly, but we instead use the `WKBConverterStrategy` to interface with geo types using appropriate conversion logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on PR #54040:
URL: https://github.com/apache/spark/pull/54040#issuecomment-3890250232
> Waiting for CI.
The `pyspark.ml.tests.connect.test_parity_clustering` failure doesn't seem related, I've seen the same failure in other PRs too. Otherwise, checks are good. @cloud-fan Please review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54040:
URL: https://github.com/apache/spark/pull/54040#discussion_r2804886266
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java:
##########
@@ -100,6 +102,48 @@ public void readBinary(int total, WritableColumnVector c, int rowId) {
readValues(total, c, rowId);
}
+ @Override
+ public void readGeometry(int total, WritableColumnVector c, int rowId) {
+ assert(c.dataType() instanceof GeometryType);
+ int srid = ((GeometryType) c.dataType()).srid();
+ readGeoData(total, c, rowId, srid, WKBToGeometryConverter.INSTANCE);
+ }
+
+ @Override
+ public void readGeography(int total, WritableColumnVector c, int rowId) {
+ assert(c.dataType() instanceof GeographyType);
+ int srid = ((GeographyType) c.dataType()).srid();
+ readGeoData(total, c, rowId, srid, WKBToGeographyConverter.INSTANCE);
+ }
+
+ private void readGeoData(int total, WritableColumnVector c, int rowId, int srid,
+ WKBConverterStrategy converter) {
+ for (int i = 0; i < total; i++) {
+ int prefixLength = prefixLengthVector.getInt(currentRow);
+ ByteBuffer suffix = suffixReader.getBytes(currentRow);
+ int suffixLength = suffix.limit() - suffix.position();
+ int length = prefixLength + suffixLength;
+
+ byte[] wkb = new byte[length];
+ if (prefixLength > 0) {
+ previous.get(wkb, 0, prefixLength);
+ }
+ suffix.get(wkb, prefixLength, suffixLength);
+
+ // Converts WKB into a physical representation of geometry/geography.
Review Comment:
what's the Spark internal physical binary format for these two types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54040:
URL: https://github.com/apache/spark/pull/54040#discussion_r2804881534
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:
##########
@@ -180,6 +182,8 @@ private boolean isLazyDecodingSupported(
break;
case BINARY:
isSupported = !needsDecimalScaleRebase(sparkType);
+ boolean isGeoType = sparkType instanceof GeometryType || sparkType instanceof GeographyType;
+ isSupported = isSupported && !isGeoType;
Review Comment:
why geo type can't support lazy decoding?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54040:
URL: https://github.com/apache/spark/pull/54040#issuecomment-3812963433
## JIRA Issue Information
=== Sub-task SPARK-55261 ===
Summary: Implement Parquet read support for Geo types
Assignee: None
Status: Open
Affected: ["4.2"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #54040:
URL: https://github.com/apache/spark/pull/54040#issuecomment-3909611766
pyspark test failure is unrelated, thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #54040:
URL: https://github.com/apache/spark/pull/54040#discussion_r2806267002
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:
##########
@@ -180,6 +182,8 @@ private boolean isLazyDecodingSupported(
break;
case BINARY:
isSupported = !needsDecimalScaleRebase(sparkType);
+ boolean isGeoType = sparkType instanceof GeometryType || sparkType instanceof GeographyType;
+ isSupported = isSupported && !isGeoType;
Review Comment:
We are incompatible with the lazy decoding optimization because we do not guarantee the internal memory format for geo types - this can change in the future, and we need use `WKBConverterStrategy` for interfacing with these (please see: https://github.com/apache/spark/pull/54040#discussion_r2806250676).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #54040: [SPARK-55261][Geo][SQL] Implement Parquet read support for Geo types
URL: https://github.com/apache/spark/pull/54040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org