[PR] fix(zarr): read sharded arrays one row per inner chunk, not per shard [sedona-db]

["james-willis (via GitHub)" <gi...@apache.org>]

james-willis opened a new pull request, #1418:
URL: https://github.com/apache/sedona-db/pull/1418

   ## What
   
   The Zarr reader now emits one raster row per **inner chunk** of an array stored with the `sharding_indexed` codec, instead of one row per shard, and the byte loader reads an inner chunk as its own byte range out of the shard.
   
   ## Why
   
   zarrs' `chunk_grid` is the shard grid for a sharded array (its docs: "the chunk_grid and chunk indices in store/retrieve methods reference the chunks ('shards') of an array"). The reader walked that grid and the loader called `async_retrieve_chunk`, so a sharded store read as one row per shard, with the row shape following the shard shape and each `RS_EnsureLoaded` decoding a whole shard. With sharding a shard is only the storage container; the chunk shape the writer chose is the inner chunk shape, and that is what zarr-python and xarray iterate. A store written as one 256 MiB shard per array therefore read as a single row of that size.
   
   ## How
   
   - `loader.rs`: `collect_array_infos` takes the grid from `ArrayShardedExt::subchunk_grid_shape` and the row shape from `effective_subchunk_shape`. For an unsharded array both are the ordinary chunk grid and chunk shape, so nothing changes there. When zarrs cannot determine the effective inner chunk shape (an array-to-array codec ahead of the sharding codec whose decoded shape is indeterminate) its subchunk grid is the shard grid, and the shard shape is used as before.
   - `raster_loader.rs`: each opened array handle now carries an `AsyncArrayShardedReadableExtCache`, and `retrieve_chunk` goes through `async_retrieve_subchunk_opt`. The shard's index is fetched once per shard and opened array, cached for the life of the handle (same TTL and LRU as the handle), and every later read into that shard fetches only its inner chunk's byte range. For an unsharded array zarrs falls through to the plain chunk read, so there is one path.
   - The chunk anchor format is unchanged; its indices are on the inner-chunk grid for a sharded array. Docs in `source_uri.rs`, `lib.rs` and the Python README say so.
   
   ## Tests
   
   - `loader::tests::sharded_array_rows_follow_inner_chunks`: one 4×4 shard of 2×2 inner chunks reads as a 2×2 grid of 2×2 rows, the same as the unsharded array with 2×2 chunks.
   - `raster_loader::tests::zarr_loader_reads_one_inner_chunk_of_a_sharded_array`: the anchor `chunk=1,0` of a sharded array returns that 2×2 block's bytes, not the shard, and a second load reuses the handle.
   - `test_zarr.py::test_sharded_array_reads_one_row_per_inner_chunk`: a (3, 5) array with 2×2 inner chunks in 4×4 shards (ragged on both axes) reads as six 2×2 rows in row-major order over the inner-chunk grid, each holding its inner chunk's pixels with the fill value past the array edge. Reading per shard would give two 4×4 rows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(zarr): read sharded arrays one row per inner chunk, not per shard [sedona-db]

["james-willis (via GitHub)" <gi...@apache.org>]

james-willis merged PR #1418:
URL: https://github.com/apache/sedona-db/pull/1418


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org