[I] Hive-style partition pruning with TimestampBasedKeyGenerator excludes every partition [hudi-rs]

["slachiewicz (via GitHub)" <gi...@apache.org>]

slachiewicz opened a new issue, #798:
URL: https://github.com/apache/hudi-rs/issues/798

   ### Is there an existing issue for this?
   
   - [x] I have searched the existing issues — #286 (closed, added the transform via #462), #580 (multi-field ordering), #549 (CustomKeyGenerator) and #779 (storage listing) are adjacent; none covers the hive-style path shape.
   
   ### Description of the bug
   
   On a hive-style table with `hoodie.table.partition.fields=ts` and `hoodie.keygen.timebased.output.dateformat=yyyy-MM-dd`, an equality predicate on `ts` returns zero file slices and zero rows although a partition holds a matching row. No warning is logged. Without the `hoodie.keygen.timebased.*` keys the same predicate returns the right row, because the pruner logs `Failed to create TimestampBasedKeyGenerator ... Filters will not be transformed` and column-stats pruning takes over. Supplying the generator's parameters turns a correct read into a wrong one.
   
   Cause, at `release-0.5.0`: [`parse_partition_fields`](https://github.com/apache/hudi-rs/blob/release-0.5.0/crates/core/src/keygen/timestamp_based.rs#L281-L302) derives one hive field name per `/`-separated token of `output.dateformat` (`yyyy` → `year`, …), and an unrecognised token becomes its own name, so a format with no `/` yields the field name `yyyy-MM-dd`. [`format_partition_path`](https://github.com/apache/hudi-rs/blob/release-0.5.0/crates/core/src/keygen/timestamp_based.rs#L425-L444) then rewrites the predicate to `_hoodie_partition_path = "yyyy-MM-dd=2024-03-02"`. The directory is `ts=2024-03-02`. Hudi's `TimestampBasedAvroKeyGenerator.getPartitionPath` prefixes the whole formatted value with `partition.fields[0] + "="` exactly once, so no hive-style timestamp-keygen table can match the rewritten path. The shape does not depend on `timestamp.type`.
   
   #724 corrects the path shape and carries Hudi 0.15 and 1.2 fixtures for it; this issue is the symptom-level entry for that PR.
   
   ### Steps To Reproduce
   
   Table written by polytable (an independent Go Hudi writer), COW, table version 6, three day partitions:
   
   ```
   ts=2024-03-01/<one base file, 1 row>
   ts=2024-03-02/<one base file, 2 rows, one with ts = 2024-03-02 12:00:00>
   ts=2024-03-03/<one base file, 3 rows>
   ```
   
   `hoodie.properties` — polytable writes the first three lines; the four `hoodie.keygen.timebased.*` lines were added by hand so the generator could be built (they are the keys Hudi 1.x persists on its own; the shipped `v9_timebasedkeygen_*` fixtures carry `timestamp.type` and `output.dateformat`):
   
   ```
   hoodie.table.partition.fields=ts
   hoodie.datasource.write.hive_style_partitioning=true
   hoodie.table.keygenerator.class=org.apache.hudi.keygen.TimestampBasedKeyGenerator
   hoodie.keygen.timebased.timestamp.type=DATE_STRING
   hoodie.keygen.timebased.input.dateformat=yyyy-MM-dd HH:mm:ss
   hoodie.keygen.timebased.output.dateformat=yyyy-MM-dd
   hoodie.keygen.timebased.timezone=UTC
   ```
   
   ```rust
   let opts = ReadOptions::new().with_filters([("ts", "=", "2024-03-02 12:00:00")])?;
   table.get_file_slices(&opts).await?   // 0 slices
   table.read(&opts).await?              // 0 rows
   ```
   
   Remove the four `hoodie.keygen.timebased.*` lines and the same call returns 1 slice and 1 row.
   
   ### Expected behavior
   
   1 file slice (`ts=2024-03-02`) and 1 row, the same as the read without the keygen keys.
   
   ### Screenshots / Logs
   
   With the keys present, `RUST_LOG=warn` prints nothing. Without them:
   
   ```
   WARN hudi_core::table::partition] Failed to create TimestampBasedKeyGenerator: Config error: Config 'Missing required configuration: hoodie.keygen.timebased.timestamp.type' not found. Filters will not be transformed.
   ```
   
   ### Software information
   
   - Operating system: macOS 27.0, arm64
   - Rust version: rustc 1.98.1
   - Project version: hudi 0.5.0 (crates.io, `default-features = false`)
   
   ### Additional context
   
   A date-only literal on the same table, `("ts", "=", "2024-03-02")`, fails separately with `Failed to parse date string '2024-03-02' with format '%Y-%m-%d %H:%M:%S': premature end of input`, because the literal is parsed with the column's `input.dateformat`. That is #746's case, not this one.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org