You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/10/22 14:27:50 UTC

[PR] [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal [spark]

dongjoon-hyun opened a new pull request, #59711:
URL: https://github.com/apache/spark/pull/59711

   ### What changes were proposed in this pull request?
   
   This PR aims to fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal when it converts an `InternalRow` to an external `Row`.
   
   `GeometryConverter.toScalaImpl(row, column)` and `GeographyConverter.toScalaImpl(row, column)` read `row.getBinaryView(0)` instead of `row.getBinaryView(column)`. Since `StructConverter` calls `toScala(row, idx)` for every field, a `GEOMETRY` or `GEOGRAPHY` field at a non-zero ordinal of a row or a struct is read from the first field. The null check in `toScala(row, column)` already uses the given ordinal, so only non-null values are affected.
   
   ### Why are the changes needed?
   
   A `GEOMETRY` or `GEOGRAPHY` field at a non-zero ordinal silently gets the value of the first field if the first field has the same type. Otherwise, the first field is read as a geospatial value, which can fail, e.g., with `IndexOutOfBoundsException`.
   
   `CatalystTypeConverters.createToScalaConverter` with a `StructType` is used for
   - the result rows of the `spark-sql` CLI (`HiveResult.hiveResultString` via `SparkPlan.executeCollectPublic`)
   - `Row` inputs of Scala and Java UDFs
   - the inputs of `UserDefinedAggregateFunction`
   - the metrics of `Dataset.observe`
   
   `Dataset.collect` is not affected because it uses an encoder.
   
   For example, Apache Spark 4.2.0 shows `POINT(1 2)` for both columns of the following query, although the second one is `POINT(3 4)`.
   
   ```
   spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
   POINT(1 2)	POINT(1 2)
   ```
   
   SPARK-54110 (#52813) introduced this bug as `row.getGeometry(0)` and `row.getGeography(0)` in Apache Spark 4.1.0 ([v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0), 2025-12-11), and SPARK-57058 (#56104) kept the ordinal `0` when it changed them to `row.getBinaryView(0)`. Apache Spark 4.1.x disables the geospatial types by default, so this bug affects users by default since Apache Spark 4.2.0 ([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11).
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. `GEOMETRY` and `GEOGRAPHY` fields at non-zero ordinals of rows and structs are converted correctly.
   
   **BEFORE (Apache Spark 4.2.0)**
   
   ```
   spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
   POINT(1 2)	POINT(1 2)
   spark-sql (default)> SELECT ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeogFromWKB(X'010100000000000000000008400000000000001040') b;
   SRID=4326;POINT(1 2)	SRID=4326;POINT(1 2)
   spark-sql (default)> SELECT 1 i, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') g;
   org.apache.spark.sql.execution.QueryExecutionException: java.lang.IndexOutOfBoundsException
   ```
   
   **AFTER**
   
   ```
   spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
   POINT(1 2)	POINT(3 4)
   spark-sql (default)> SELECT ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeogFromWKB(X'010100000000000000000008400000000000001040') b;
   SRID=4326;POINT(1 2)	SRID=4326;POINT(3 4)
   spark-sql (default)> SELECT 1 i, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') g;
   1	POINT(3 4)
   ```
   
   ### How was this patch tested?
   
   Pass the CIs with the newly added test case in `CatalystTypeConvertersSuite`. It converts rows whose two `GEOMETRY` or `GEOGRAPHY` fields have different values. Without the fix, it fails.
   
   ```
   [info] - SPARK-60470: createToScalaConverter for geospatial fields at non-zero ordinals *** FAILED *** (8 milliseconds)
   [info]   [org.apache.spark.sql.types.Geometry@6f7338ee,org.apache.spark.sql.types.Geometry@6f7338ee] did not equal [org.apache.spark.sql.types.Geometry@6f7338ee,org.apache.spark.sql.types.Geometry@96d26af9] (CatalystTypeConvertersSuite.scala:822)
   ```
   
   The related test suites also pass.
   
   ```
   $ build/sbt "catalyst/testOnly *CatalystTypeConvertersSuite"
   $ build/sbt "sql/testOnly *HiveResultSuite *GeometryDataFrameSuite *GeographyDataFrameSuite *STExpressionsSuite *STFunctionsSuite *GeometryTypeSuite *GeographyTypeSuite"
   $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z st-functions.sql"
   ```
   
   The `spark-sql` examples above were also checked manually with Apache Spark 4.2.0 and this PR.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 5.5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #59711: [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal
URL: https://github.com/apache/spark/pull/59711


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59711:
URL: https://github.com/apache/spark/pull/59711#issuecomment-6343273996

   cc @uros-b and @cloud-fan 
   
   Also, FYI, @szehon-ho because this is a correctness issue (not a blocker).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59711:
URL: https://github.com/apache/spark/pull/59711#issuecomment-6350975006

   Thank you, @HyukjinKwon !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59711:
URL: https://github.com/apache/spark/pull/59711#issuecomment-6351007037

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/6f257a187bcafe8e4a0f7dacba09d25c76748bc8
   - merged into branch-4.x https://github.com/apache/spark/commit/9cfbe3ea7bf730b4d920e761bb3c7ca4981bcb6b
   - merged into branch-4.3 https://github.com/apache/spark/commit/dd96ea6de1325572fb9e9e3ccb89f33dc0b39c05
   - merged into branch-4.2 https://github.com/apache/spark/commit/0e01336b6a4d80f182aee2c0f44eb66917dba702
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org