You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/22 20:49:05 UTC
(spark) branch branch-4.x updated: [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 9cfbe3ea7bf7 [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal
9cfbe3ea7bf7 is described below
commit 9cfbe3ea7bf730b4d920e761bb3c7ca4981bcb6b
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Oct 22 13:48:32 2026 -0700
[SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal
### What changes were proposed in this pull request?
This PR aims to fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal when it converts an `InternalRow` to an external `Row`.
`GeometryConverter.toScalaImpl(row, column)` and `GeographyConverter.toScalaImpl(row, column)` read `row.getBinaryView(0)` instead of `row.getBinaryView(column)`. Since `StructConverter` calls `toScala(row, idx)` for every field, a `GEOMETRY` or `GEOGRAPHY` field at a non-zero ordinal of a row or a struct is read from the first field. The null check in `toScala(row, column)` already uses the given ordinal, so only non-null values are affected.
### Why are the changes needed?
A `GEOMETRY` or `GEOGRAPHY` field at a non-zero ordinal silently gets the value of the first field if the first field has the same type. Otherwise, the first field is read as a geospatial value, which can fail, e.g., with `IndexOutOfBoundsException`.
`CatalystTypeConverters.createToScalaConverter` with a `StructType` is used for
- the result rows of the `spark-sql` CLI (`HiveResult.hiveResultString` via `SparkPlan.executeCollectPublic`)
- `Row` inputs of Scala and Java UDFs
- the inputs of `UserDefinedAggregateFunction`
- the metrics of `Dataset.observe`
`Dataset.collect` is not affected because it uses an encoder.
For example, Apache Spark 4.2.0 shows `POINT(1 2)` for both columns of the following query, although the second one is `POINT(3 4)`.
```
spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
POINT(1 2) POINT(1 2)
```
SPARK-54110 (#52813) introduced this bug as `row.getGeometry(0)` and `row.getGeography(0)` in Apache Spark 4.1.0 ([v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0), 2025-12-11), and SPARK-57058 (#56104) kept the ordinal `0` when it changed them to `row.getBinaryView(0)`. Apache Spark 4.1.x disables the geospatial types by default, so this bug affects users by default since Apache Spark 4.2.0 ([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11).
- https://github.com/apache/spark/pull/52813
### Does this PR introduce _any_ user-facing change?
Yes. `GEOMETRY` and `GEOGRAPHY` fields at non-zero ordinals of rows and structs are converted correctly.
**BEFORE (Apache Spark 4.2.0)**
```
spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
POINT(1 2) POINT(1 2)
spark-sql (default)> SELECT ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeogFromWKB(X'010100000000000000000008400000000000001040') b;
SRID=4326;POINT(1 2) SRID=4326;POINT(1 2)
spark-sql (default)> SELECT 1 i, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') g;
org.apache.spark.sql.execution.QueryExecutionException: java.lang.IndexOutOfBoundsException
```
**AFTER**
```
spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
POINT(1 2) POINT(3 4)
spark-sql (default)> SELECT ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeogFromWKB(X'010100000000000000000008400000000000001040') b;
SRID=4326;POINT(1 2) SRID=4326;POINT(3 4)
spark-sql (default)> SELECT 1 i, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') g;
1 POINT(3 4)
```
### How was this patch tested?
Pass the CIs with the newly added test case in `CatalystTypeConvertersSuite`. It converts rows whose two `GEOMETRY` or `GEOGRAPHY` fields have different values. Without the fix, it fails.
```
[info] - SPARK-60470: createToScalaConverter for geospatial fields at non-zero ordinals *** FAILED *** (8 milliseconds)
[info] [org.apache.spark.sql.types.Geometry6f7338ee,org.apache.spark.sql.types.Geometry6f7338ee] did not equal [org.apache.spark.sql.types.Geometry6f7338ee,org.apache.spark.sql.types.Geometry96d26af9] (CatalystTypeConvertersSuite.scala:822)
```
The related test suites also pass.
```
$ build/sbt "catalyst/testOnly *CatalystTypeConvertersSuite"
$ build/sbt "sql/testOnly *HiveResultSuite *GeometryDataFrameSuite *GeographyDataFrameSuite *STExpressionsSuite *STFunctionsSuite *GeometryTypeSuite *GeographyTypeSuite"
$ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z st-functions.sql"
```
The `spark-sql` examples above were also checked manually with Apache Spark 4.2.0 and this PR.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59711 from dongjoon-hyun/SPARK-60470.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 6f257a187bcafe8e4a0f7dacba09d25c76748bc8)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../spark/sql/catalyst/CatalystTypeConverters.scala | 4 ++--
.../sql/catalyst/CatalystTypeConvertersSuite.scala | 17 +++++++++++++++++
2 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
index cab367b4badf..1074f13c3473 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
@@ -391,7 +391,7 @@ object CatalystTypeConverters {
override def toScalaImpl(row: InternalRow, column: Int):
org.apache.spark.sql.types.Geometry = {
assertGeospatialEnabled()
- STUtils.deserializeGeom(row.getBinaryView(0), dataType)
+ STUtils.deserializeGeom(row.getBinaryView(column), dataType)
}
}
@@ -411,7 +411,7 @@ object CatalystTypeConverters {
override def toScalaImpl(row: InternalRow, column: Int):
org.apache.spark.sql.types.Geography = {
assertGeospatialEnabled()
- STUtils.deserializeGeog(row.getBinaryView(0), dataType)
+ STUtils.deserializeGeog(row.getBinaryView(column), dataType)
}
}
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala
index 988a8d4510cb..4fc47e5a03da 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala
@@ -806,6 +806,23 @@ class CatalystTypeConvertersSuite extends SparkFunSuite with SQLHelper {
assert(result === geog)
}
+ test("SPARK-60470: createToScalaConverter for geospatial fields at non-zero ordinals") {
+ // WKB bytes for POINT (3 4), distinct from `pointWkb`.
+ val otherPointWkb: Array[Byte] = "010100000000000000000008400000000000001040"
+ .grouped(2).map(Integer.parseInt(_, 16).toByte).toArray
+ Seq(
+ (GeometryType(0), Geometry.fromWKB(pointWkb, 0), Geometry.fromWKB(otherPointWkb, 0)),
+ (GeographyType(4326), Geography.fromWKB(pointWkb, 4326),
+ Geography.fromWKB(otherPointWkb, 4326))
+ ).foreach { case (dt, first, second) =>
+ val schema = new StructType().add("first", dt).add("second", dt)
+ val toCatalyst = CatalystTypeConverters.createToCatalystConverter(schema)
+ val toScala = CatalystTypeConverters.createToScalaConverter(schema)
+ val row = Row(first, second)
+ assert(toScala(toCatalyst(row)) === row)
+ }
+ }
+
test("null handling for geospatial individual values") {
assert(CatalystTypeConverters.createToScalaConverter(GeometryType(0))(null) === null)
assert(CatalystTypeConverters.createToScalaConverter(GeographyType(4326))(null) === null)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org