[I] RS_Metadata converts the raster to a GDAL dataset just for tile size, and fails on N-D bands [sedona-db]

["james-willis (via GitHub)" <gi...@apache.org>]

james-willis opened a new issue, #1392:
URL: https://github.com/apache/sedona-db/issues/1392

   `RS_Metadata` returns twelve fields. Ten of them (upper-left corner, width, height, scale, skew, SRID, number of bands) come straight from the raster struct. The other two, `tileWidth` and `tileHeight`, are found by converting the whole raster to a GDAL dataset and reading band 1's block size (`rust/sedona-raster-gdal/src/rs_metadata.rs`):
   
   ```rust
   let dataset = provider.raster_ref_to_gdal(raster)?;
   let (block_x, block_y) = dataset.as_dataset().rasterband(1)?.block_size();
   ```
   
   That one step causes three problems.
   
   **1. It fails on rasters GDAL can't represent, losing the other ten fields too.** A Zarr chunk read with sedonadb-zarr has `[time, y, x]` bands (`outdb_format = 'zarr'`, `outdb_uri` like `file:///…/00000.zarr#array=tasmax&chunk=0,0,0`):
   
   ```
   SELECT RS_Metadata(raster) FROM zarr_table
   -- Failed to create GDAL dataset: Execution error: GDAL backend requires 2-dim bands; got dim_names=["time", "y", "x"]
   ```
   
   `raster_ref_to_gdal` also ignores `outdb_format` and hands any out-db `outdb_uri` to GDAL as a file to open, so a 2-D Zarr band would most likely fail as well (not tested).
   
   **2. It does I/O for an out-db raster.** For a raster from `RS_FromPath`, the conversion opens the source file (through the per-thread dataset cache) and builds a VRT over it, so every `RS_Metadata` call on remote data costs a file open and an HTTP round trip to read two integers. This will also undo a header-read-free raster constructor (building out-db rasters from STAC `proj:shape` / `proj:transform` without opening the files): `RS_Metadata` would still open every file.
   
   **3. For an in-db raster the value is GDAL's in-memory default, not a property of the data:**
   
   ```
   SELECT RS_Metadata(RS_MakeEmptyRaster(1, 'B', 1000, 600, 0.0, 600.0, 1.0))
   -- … tileWidth: 1000, tileHeight: 1
   ```
   
   ### Proposal
   
   Make `RS_Metadata` read only the raster struct and never go through GDAL:
   
   - Record a tile / chunk shape per out-db band in the raster struct where the producer knows it:
     - `RS_FromPath` already opens the file to read the header, so it can store the band's block size;
     - sedonadb-zarr knows the array's chunk (or subchunk, for sharded v3) shape;
     - a raster built from catalog columns can take it as an optional argument.
   - `RS_Metadata` returns that for `tileWidth` / `tileHeight`, and NULL when it isn't recorded (in-db rasters, or producers that don't know it). The other ten fields are always returned.
   
   ### Questions
   
   - **Schema:** where should the tile shape live in the raster struct? A nullable per-band list of `Int64` alongside `source_shape` seems natural; it would also give the loader a natural read granularity.
   - **Spark parity:** Sedona Spark's `RS_MetaData` always returns a tile size. Is NULL acceptable when it isn't known, or should it fall back to the raster's width and height?
   - **The dataset cache:** `RS_Metadata` is one of the remaining users of the per-thread GDAL dataset cache (with `RS_AsGeoTiff`, `RS_AsRaster` and `RS_Polygonize`). This change would drop it to three.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org