[I] Add more end to end tests for schema evolution and partition stats index [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #17375:
URL: https://github.com/apache/hudi/issues/17375

   ref link: [https://github.com/apache/hudi/pull/12714#discussion_r1930022912] 
   
    
   Can we add some validation for partition stats read/write in this test? Something like:
    # Start with MOR table having an int field.
    # Promote to long in an upsert.
    # Validate partition stats content.
    # Validate partition pruning using stats.
    
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-8931
   - Type: Sub-task
   - Parent: https://issues.apache.org/jira/browse/HUDI-8727
   - Fix version(s):
     - 1.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add more end to end tests for schema evolution and partition stats index [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #17375:
URL: https://github.com/apache/hudi/issues/17375#issuecomment-5351240576

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8931).
   
   **Findings: not done as far as I can tell; keeping open as a coverage task.**
   
   The ask from the review comment ([PR #12714 (comment)](https://github.com/apache/hudi/pull/12714#discussion_r1930022912)) is specific:
   
   1. Start with a MOR table having an int field
   2. Promote to long in an upsert
   3. Validate partition stats content
   4. Validate partition pruning using stats
   
   I did not find a test on `master` exercising that full sequence - int-to-long promotion followed by both partition-stats content validation and pruning validation. Coverage gaps are hard to prove by search alone, so the assignee should confirm rather than take this as definitive, but nothing indicates it landed.
   
   Two things worth folding into the same test matrix while it is being written:
   
   - #17356 (HUDI-8923) is now **fixed** - `ValueType` supports all the timestamp/time logical types plus nanos variants - so schema evolution across those types is newly worth covering.
   - #17367 (HUDI-8909) is **not** fixed, and `ColumnStatIndexTestBase.scala:69` currently types the `c7` column as `StringType` specifically to avoid it. Any new partition-stats test should avoid re-encoding that workaround.
   
   Keeping this open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org