[I] Fix geotiff.metadata defects found during the netcdf.metadata reviews [sedona]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu opened a new issue, #3128:
URL: https://github.com/apache/sedona/issues/3128

   ### Description
   
   While building the `netcdf.metadata` data source (#3115, merged in #2829), several defects were found and fixed in code patterns that `netcdf.metadata` originally copied from the `geotiff.metadata` data source (#2846). The same defects still exist in the `geotiff.metadata` package on master. The corresponding fixes are already on master in `spark/common/.../io/netcdfmetadata/`, so this is a targeted transplant.
   
   ### Defects to fix in `spark/common/.../io/geotiffmetadata/`
   
   **Correctness (silent wrong results):**
   
   1. **Wrong Hive partition values in bin-packed partitions.** `GeoTiffMetadataPartitionReaderFactory` wraps the whole multi-file partition in one `PartitionReaderWithPartitionValues` using `partition.files.head.partitionValues`, so when files from different Hive partitions (e.g. `year=2020/`, `year=2021/`) are packed into one Spark partition, every row reports the first file's partition values. Fix: per-file reader wrapping, as in `NetCdfMetadataConcatPartitionReader`.
   
   2. **Scan equality ignores pushdown state.** `GeoTiffMetadataScan` is a case class, but `FileScan` provides concrete `equals`/`hashCode`, which suppresses the synthesized ones — `pushedLimit` and `options` do not participate in equality. Since the scan alone enforces the pushed limit (`isPartiallyPushed = false` removes the Limit operator), exchange/subquery reuse can serve a limited scan's result for an unlimited one. Fix: override `equals` to include `options` and `pushedLimit`, and derive `hashCode` from the same fields, as in `NetCdfMetadataScan`.
   
   **Crash:**
   
   3. **`fallbackFileFormat` returns null.** `CREATE TABLE ... USING geotiff.metadata` throws a `NullPointerException` (Spark instantiates the V1 fallback format for catalog paths). Fix: a stub `FileFormat` that fails on instantiation with a descriptive `UnsupportedOperationException`, which also rejects `CREATE TABLE` with an explicit schema (that path skips `inferSchema`), as in `NetCdfMetadataUnsupportedFileFormat`.
   
   **Behavior:**
   
   4. **Directory handling overrides explicit options.** Single-directory loads force `recursiveFileLookup=true`, overriding an explicit `false` and silently disabling Hive-style partition discovery; multi-directory loads bypass the `.tif`/`.tiff` extension filter entirely. Fix: apply both as defaults only, for any directory root.
   
   5. **Option presence checks are case-sensitive.** Defaults are guarded with `containsKey` on a copied `HashMap` instead of the `CaseInsensitiveStringMap`, so an explicit mixed-case option (e.g. `RecursiveFileLookup=false`) can nondeterministically lose to the lowercase default.
   
   6. **A glob path combined with an explicit `pathGlobFilter` drops one constraint.** `.load("/dir/a*.tif")` with `pathGlobFilter="b*.tif"` rewrites the path to `/dir` keeping only one filter; native Spark semantics apply both. Fix: skip the glob rewrite when the user supplied a filter.
   
   ### Test plan
   
   Port the corresponding regression tests from `netcdfMetadataTest` to the GeoTIFF suite: Hive-partitioned directory with per-file partition values, limit/exchange-reuse scan identity, `CREATE TABLE` (with and without explicit schema) failing with a clear error, explicit `recursiveFileLookup=false` keeping partition discovery, mixed-case option spelling, multi-directory extension filtering, and glob + filter combination.
   
   ### Related
   
   - #2846 (`geotiff.metadata` data source)
   - #3115 / #2829 (`netcdf.metadata`, where these fixes landed)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Fix geotiff.metadata defects found during the netcdf.metadata reviews [sedona]

["jiayuasu (via GitHub)" <gi...@apache.org>]

jiayuasu closed issue #3128: Fix geotiff.metadata defects found during the netcdf.metadata reviews
URL: https://github.com/apache/sedona/issues/3128


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org