You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "jiayuasu (via GitHub)" <gi...@apache.org> on 2026/02/19 05:52:34 UTC

[PR] [GH-2376] Preserve CRS SRID from GeoParquet metadata when reading geometries [sedona]

jiayuasu opened a new pull request, #2661:
URL: https://github.com/apache/sedona/pull/2661

   ## Did you read the Contributor Guide?
   
   - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
   
   ## Is this PR related to a ticket?
   
   - Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #2376
   
   ## What changes were proposed in this PR?
   
   The GeoParquet reader currently ignores CRS metadata when deserializing geometries, always setting SRID=0. This PR fixes that by extracting the SRID from the PROJJSON CRS metadata and setting it on each deserialized geometry.
   
   ### Changes
   
   **`GeoParquetMetaData.scala`** - Added `extractSridFromCrs(crs: Option[JValue]): Int` that uses proj4sedona's `Proj.toAuthority()` to parse PROJJSON and extract the authority/code pair. Per the GeoParquet spec:
   - CRS omitted (None) → SRID 4326 (OGC:CRS84 default)
   - CRS explicitly null → SRID 0 (unknown)
   - PROJJSON with EPSG authority → the EPSG integer code
   - PROJJSON with OGC:CRS84 → SRID 4326
   - Anything else (non-EPSG authority, no id field, parse error) → SRID 0
   
   **`GeoParquetSchemaConverter.scala`** - Added `getSrid(columnName: String): Int` to look up the SRID for a geometry column from parsed GeoParquet metadata.
   
   **`GeoParquetRowConverter.scala`** - After `WKBReader.read()`, calls `geom.setSRID(srid)` to apply the SRID from file metadata to each deserialized geometry.
   
   **`pom.xml`** - Bumped proj4sedona dependency from 0.0.4 to 0.0.5 to use the new `Proj.toAuthority()` API.
   
   ### How it works
   
   1. When a GeoParquet file is read, `GeoParquetSchemaConverter` parses the file-level `"geo"` metadata and extracts the CRS PROJJSON for each geometry column.
   2. For each geometry column, `extractSridFromCrs()` converts the PROJJSON to a compact JSON string, passes it to `new Proj(jsonStr).toAuthority()`, and maps the result to an integer SRID.
   3. `GeoParquetRowConverter` receives the SRID and sets it on every geometry deserialized from that column.
   
   ## How was this patch tested?
   
   12 new tests added to `geoparquetIOTests.scala` (all 34 tests in the suite pass):
   
   **5 integration tests** (write GeoParquet with specific CRS, read back, verify SRID):
   - EPSG PROJJSON (NAD83(2011), EPSG:6318) → SRID 6318
   - Omitted CRS → SRID 4326
   - Null CRS → SRID 0
   - CRS without EPSG id → SRID 0
   - Per-column multi-CRS (g0=EPSG:4326, g1=EPSG:32632)
   
   **7 unit tests** for `extractSridFromCrs`:
   - None → 4326
   - JNull → 0
   - EPSG:4326 PROJJSON → 4326
   - EPSG:32632 PROJJSON → 32632
   - OGC:CRS84 → 4326
   - No id field → 0
   - Non-EPSG authority (IAU:49900) → 0
   
   ## Did this PR include necessary documentation updates?
   
   - No, this PR does not affect any public API so no need to change the documentation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-2376] Preserve CRS SRID from GeoParquet metadata when reading geometries [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #2661:
URL: https://github.com/apache/sedona/pull/2661#discussion_r2825952353


##########
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetRowConverter.scala:
##########
@@ -233,6 +235,8 @@ private[geoparquet] class GeoParquetRowConverter(
                 val wkbReader = new WKBReader()
                 val byteArray = currentArray.map(_.asInstanceOf[Byte]).toArray
                 val geom = wkbReader.read(byteArray)
+                val srid = schemaConverter.getSrid(parquetType.getName)
+                geom.setSRID(srid)
                 this.updater.set(GeometryUDT.serialize(geom))

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-2376] Preserve CRS SRID from GeoParquet metadata when reading geometries [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #2661:
URL: https://github.com/apache/sedona/pull/2661#discussion_r2825951287


##########
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetMetaData.scala:
##########
@@ -137,6 +138,57 @@ object GeoParquetMetaData {
     compactJson(serializedGeoObject)
   }
 
+  /**
+   * Default SRID for GeoParquet files where the CRS field is omitted. Per the GeoParquet spec,
+   * omitting the CRS implies OGC:CRS84, which is equivalent to EPSG:4326.
+   */
+  val DEFAULT_SRID: Int = 4326
+
+  /**
+   * Extract SRID from a GeoParquet CRS metadata value.
+   *
+   * Per the GeoParquet specification:
+   *   - If the CRS field is absent (None), the CRS is OGC:CRS84 (EPSG:4326).
+   *   - If the CRS field is explicitly null, the CRS is unknown (SRID 0).
+   *   - If the CRS field is a PROJJSON object with an "id" containing "authority" and "code", the
+   *     EPSG code is used as the SRID.
+   *
+   * @param crs
+   *   The CRS field from GeoParquet column metadata.
+   * @return
+   *   The SRID corresponding to the CRS. Returns 4326 for omitted CRS, 0 for null or unrecognized
+   *   CRS, and the EPSG code for PROJJSON with an EPSG identifier.
+   */
+  def extractSridFromCrs(crs: Option[JValue]): Int = {
+    crs match {
+      case None =>
+        // CRS omitted: default to OGC:CRS84 (EPSG:4326) per GeoParquet spec
+        DEFAULT_SRID
+      case Some(JNull) =>
+        // CRS explicitly null: unknown CRS
+        0
+      case Some(projjson) =>
+        // Use proj4sedona to extract authority and code from PROJJSON
+        try {
+          val jsonStr = compactJson(projjson)
+          val result = new Proj(jsonStr).toAuthority()
+          if (result != null && result.length == 2) {
+            result(0) match {
+              case "EPSG" =>
+                try { result(1).toInt }
+                catch { case _: NumberFormatException => 0 }
+              case "OGC" if result(1) == "CRS84" => 4326
+              case _ => 0
+            }
+          } else {
+            0
+          }
+        } catch {
+          case _: Exception => 0
+        }

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-2376] Preserve CRS SRID from GeoParquet metadata when reading geometries [sedona]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2661:
URL: https://github.com/apache/sedona/pull/2661#discussion_r2825826264


##########
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetSchemaConverter.scala:
##########
@@ -78,6 +78,21 @@ class GeoParquetToSparkSchemaConverter(
       conf.get(PortableSQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED.key).toBoolean,
     parameters = parameters)
 
+  /**
+   * Returns the SRID for a geometry column based on the CRS metadata in the GeoParquet file.
+   * @param columnName
+   *   name of the geometry column
+   * @return
+   *   the SRID extracted from the CRS metadata. Returns 4326 if the CRS is omitted, 0 if the CRS
+   *   is null or unrecognized, or the EPSG code if the PROJJSON contains an EPSG identifier.
+   */
+  def getSrid(columnName: String): Int = {
+    geoParquetMetaData.columns.get(columnName) match {
+      case Some(fieldMetadata) => GeoParquetMetaData.extractSridFromCrs(fieldMetadata.crs)
+      case None => 0
+    }
+  }

Review Comment:
   getSrid() can be called frequently during row conversion, and it currently triggers CRS parsing (Proj/PROJJSON) on every call via extractSridFromCrs(). Consider caching the computed SRID per column (e.g., build a Map[String, Int] once from geoParquetMetaData.columns) so row decoding only does a map lookup. Also, the Scaladoc describes behavior for omitted/null/unrecognized CRS but doesn’t mention that an unknown columnName returns 0; please document that case (or change the default if appropriate).



##########
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetRowConverter.scala:
##########
@@ -233,6 +235,8 @@ private[geoparquet] class GeoParquetRowConverter(
                 val wkbReader = new WKBReader()
                 val byteArray = currentArray.map(_.asInstanceOf[Byte]).toArray
                 val geom = wkbReader.read(byteArray)
+                val srid = schemaConverter.getSrid(parquetType.getName)
+                geom.setSRID(srid)
                 this.updater.set(GeometryUDT.serialize(geom))

Review Comment:
   Same as the primitive geometry path: SRID is computed inside end() for every array value, which can repeatedly parse CRS metadata. Compute the SRID once when constructing this converter and reuse it to avoid per-row CRS parsing overhead.



##########
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetMetaData.scala:
##########
@@ -137,6 +138,57 @@ object GeoParquetMetaData {
     compactJson(serializedGeoObject)
   }
 
+  /**
+   * Default SRID for GeoParquet files where the CRS field is omitted. Per the GeoParquet spec,
+   * omitting the CRS implies OGC:CRS84, which is equivalent to EPSG:4326.
+   */
+  val DEFAULT_SRID: Int = 4326
+
+  /**
+   * Extract SRID from a GeoParquet CRS metadata value.
+   *
+   * Per the GeoParquet specification:
+   *   - If the CRS field is absent (None), the CRS is OGC:CRS84 (EPSG:4326).
+   *   - If the CRS field is explicitly null, the CRS is unknown (SRID 0).
+   *   - If the CRS field is a PROJJSON object with an "id" containing "authority" and "code", the
+   *     EPSG code is used as the SRID.
+   *
+   * @param crs
+   *   The CRS field from GeoParquet column metadata.
+   * @return
+   *   The SRID corresponding to the CRS. Returns 4326 for omitted CRS, 0 for null or unrecognized
+   *   CRS, and the EPSG code for PROJJSON with an EPSG identifier.
+   */
+  def extractSridFromCrs(crs: Option[JValue]): Int = {
+    crs match {
+      case None =>
+        // CRS omitted: default to OGC:CRS84 (EPSG:4326) per GeoParquet spec
+        DEFAULT_SRID
+      case Some(JNull) =>
+        // CRS explicitly null: unknown CRS
+        0
+      case Some(projjson) =>
+        // Use proj4sedona to extract authority and code from PROJJSON
+        try {
+          val jsonStr = compactJson(projjson)
+          val result = new Proj(jsonStr).toAuthority()
+          if (result != null && result.length == 2) {
+            result(0) match {
+              case "EPSG" =>
+                try { result(1).toInt }
+                catch { case _: NumberFormatException => 0 }
+              case "OGC" if result(1) == "CRS84" => 4326
+              case _ => 0
+            }
+          } else {
+            0
+          }
+        } catch {
+          case _: Exception => 0
+        }

Review Comment:
   extractSridFromCrs catches all Exception and silently returns 0. This can accidentally swallow InterruptedException and other non-data-related failures. Prefer catching NonFatal and, if an InterruptedException occurs, restore the thread interrupt status before returning. Also consider using DEFAULT_SRID instead of the hard-coded 4326 in the OGC:CRS84 branch for consistency.



##########
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetRowConverter.scala:
##########
@@ -220,6 +220,8 @@ private[geoparquet] class GeoParquetRowConverter(
             override def addBinary(value: Binary): Unit = {
               val wkbReader = new WKBReader()
               val geom = wkbReader.read(value.getBytes)
+              val srid = schemaConverter.getSrid(parquetType.getName)
+              geom.setSRID(srid)
               this.updater.set(GeometryUDT.serialize(geom))

Review Comment:
   SRID is looked up and CRS is parsed inside addBinary for every geometry value. Since getSrid() ultimately calls extractSridFromCrs() which instantiates Proj and parses PROJJSON, this adds per-row overhead. Consider computing the SRID once when creating the converter for this field (e.g., val srid = schemaConverter.getSrid(parquetType.getName) outside addBinary) and reusing it for all rows of that column (or caching SRIDs in the schemaConverter).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-2376] Preserve CRS SRID from GeoParquet metadata when reading geometries [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #2661:
URL: https://github.com/apache/sedona/pull/2661#discussion_r2825950474


##########
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetRowConverter.scala:
##########
@@ -220,6 +220,8 @@ private[geoparquet] class GeoParquetRowConverter(
             override def addBinary(value: Binary): Unit = {
               val wkbReader = new WKBReader()
               val geom = wkbReader.read(value.getBytes)
+              val srid = schemaConverter.getSrid(parquetType.getName)
+              geom.setSRID(srid)
               this.updater.set(GeometryUDT.serialize(geom))

Review Comment:
   fixed



##########
spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/GeoParquetSchemaConverter.scala:
##########
@@ -78,6 +78,21 @@ class GeoParquetToSparkSchemaConverter(
       conf.get(PortableSQLConf.PARQUET_INFER_TIMESTAMP_NTZ_ENABLED.key).toBoolean,
     parameters = parameters)
 
+  /**
+   * Returns the SRID for a geometry column based on the CRS metadata in the GeoParquet file.
+   * @param columnName
+   *   name of the geometry column
+   * @return
+   *   the SRID extracted from the CRS metadata. Returns 4326 if the CRS is omitted, 0 if the CRS
+   *   is null or unrecognized, or the EPSG code if the PROJJSON contains an EPSG identifier.
+   */
+  def getSrid(columnName: String): Int = {
+    geoParquetMetaData.columns.get(columnName) match {
+      case Some(fieldMetadata) => GeoParquetMetaData.extractSridFromCrs(fieldMetadata.crs)
+      case None => 0
+    }
+  }

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GH-2376] Preserve CRS SRID from GeoParquet metadata when reading geometries [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #2661:
URL: https://github.com/apache/sedona/pull/2661


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org