[I] GeoParquet: honor an explicit null CRS and primary_column as the spec defines them [sedona-db]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu opened a new issue, #1404:
URL: https://github.com/apache/sedona-db/issues/1404

   SedonaDB's GeoParquet reader and writer depart from the [GeoParquet 1.1 spec](https://geoparquet.org/releases/v1.1.0/) on the CRS and the primary column. Found while adding `read_parquet`/`to_parquet` to `sedonadb-geopandas` (#1403), which documents these as known differences for now.
   
   1. **An explicit `null` CRS reads as OGC:CRS84.** The spec distinguishes a missing key from a null one: "If the `crs` key does not exist, all coordinates in the geometries MUST use longitude, latitude based on the WGS84 datum, and the default value is OGC:CRS84", while "the value of this key may be explicitly set to `null` to indicate that there is no CRS assigned to this column (CRS is undefined or unknown)." SedonaDB reads both as OGC:CRS84, so a file declaring an unknown CRS gains one.
   2. **A geometry column without a CRS can't be written** ("Can't write GeoParquet from null CRS"). The spec's way to write such a column is `"crs": null`.
   3. **`primary_column` is not used on read, and is not kept on a round trip.** The spec defines it as the geometry that "may be used by default in geospatial operations", but the reader picks the primary geometry by name (`geometry`, `geography`, `geom`, `geog`, else the first geometry column), and the writer derives `primary_column` the same way, with no way to set it. Reading and rewriting a file therefore changes its primary column.
   
   ```python
   import json
   
   import geopandas as gpd
   import pyarrow.parquet as pq
   import sedonadb
   
   sd = sedonadb.connect()
   
   
   def geo(path):
       return json.loads(pq.read_schema(path).metadata[b"geo"])
   
   
   # 1. An explicit null CRS reads as OGC:CRS84
   gpd.GeoDataFrame(geometry=gpd.points_from_xy([0], [0])).to_parquet("null_crs.parquet")
   print(geo("null_crs.parquet")["columns"]["geometry"]["crs"])  # None
   print(sd.read_parquet("null_crs.parquet").schema.field("geometry").type.crs)  # OGC:CRS84
   
   # 2. A geometry column without a CRS cannot be written (as an explicit null)
   try:
       sd.sql("SELECT ST_GeomFromWKT('POINT (0 1)') AS geometry").to_parquet("out.parquet")
   except Exception as e:
       print(str(e).splitlines()[0])  # Can't write GeoParquet from null CRS
   
   # 3. primary_column is not used on read, and changes on a round trip
   two = gpd.GeoDataFrame(
       {"a_geom": gpd.GeoSeries.from_wkt(["POINT (9 9)"], crs=4326)},
       geometry=gpd.GeoSeries.from_wkt(["POINT (1 1)"], crs=3857),
   ).rename_geometry("z_geom")
   two.to_parquet("two.parquet")
   print(geo("two.parquet")["primary_column"])  # z_geom
   df = sd.read_parquet("two.parquet")
   print(df.to_pandas().geometry.name)  # a_geom
   df.to_parquet("two_out.parquet")
   print(geo("two_out.parquet")["primary_column"])  # a_geom
   ```
   
   For 1 and 2, `GeoParquetColumnMetadata::crs` is an `Option<Value>`, so serde reads an explicit `null` the same as a missing key. Keeping them apart (for example, a `deserialize_with` that maps an explicit `null` to `Some(Value::Null)`) would let the reader leave the CRS unset and the writer emit `"crs": null` for a column without one.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] GeoParquet: honor an explicit null CRS and primary_column as the spec defines them [sedona-db]

["paleolimbot (via GitHub)" <gi...@apache.org>]

paleolimbot commented on issue #1404:
URL: https://github.com/apache/sedona-db/issues/1404#issuecomment-6001682076

   > primary_column is not used on read, and is not kept on a round trip
   
   This one is tricky and I think we should implement it as part of our statistics reporting and propagation. Another option would be field metadata.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org