You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:46:50 UTC

[I] HoodieTest tables enhancement [hudi]

hudi-bot opened a new issue, #14855:
URL: https://github.com/apache/hudi/issues/14855

   Design doc
   
   https://lucid.app/publicSegments/view/a9427d7e-b5cf-4c4f-b99b-dc81d595c9f6/image.png
   
    
   
    
    * Objective : test metadata table for files and timeline integrity. 
    **  Manipulate commits and transitions. empty files should do. Ability to sync to metadata table. Commit metadata is the crux here. 
    *** - Commit/DeltaCommit
    *** - Compaction
    *** - Cleaning
    *** - ReplaceCommit/Clustering
    *** - Savepoint/delete savepoint/restore savepoint
    *** - Rollback
    *** - Restore
    ** We will list using this test table and verify data integrity. 
   
    
   
   Also, enhance to support actual records.
   
   Objective: test whole of Hoodie for data integrity. records to file locations are user defined or test driven. 
    * Updates? Deletes. should we let callers pass in HoodieRecords w/ proper file location and write them directly. 
    * should work for inserts, upserts, deletes, compaction, clustering, rollback. 
    * how does cleaner plan, compaction plan would pan out?
    * can we maintain in-memory state and simulate updates, etc. anyways, its not distributed right. We are testing just functionality. 
   
    
   
   Document what do we miss testing in actual code path if we start using this test tables for testing.  
    * for eg: index. 
    * partitioner. 
    * write handles (create, append, merge). 
    * ...
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-2405
   - Type: Improvement
   - Fix version(s):
     - 1.1.0
   
   
   ---
   
   
   ## Comments
   
   09/Sep/21 17:27;shivnarayan;[~rxu]: 
   
   I checked the design proposal. Def looks good and the way we wanna go.
   
   couple of comments
   
   1. I feel apis like this (with10Records3PartitionsAsCommits() ) api is tad bit rigid. I am ok having these apis. but also, we should have apis so that users can dictate the partitions and just pass count of files. Something like testTable.insert(commitInstant, operationType, list of new partitions to add, list of partitions to insert/update, files to be added per partition. I feel this will be very useful to write tests for around certain partitions like updates, insert_overwrite, etc. 
   2. Not sure if its implicit in the attached doc. but would like to ensure we have 2 diff set of apis. 1 set of apis is just about metadata management w/ just empty files. and another set of apis to operate with actual records. ;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] HoodieTest tables enhancement [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #14855:
URL: https://github.com/apache/hudi/issues/14855#issuecomment-5066710656

   This issue was reviewed as part of the JIRA-migrated backlog triage.
   
   Findings: This is a test-infrastructure enhancement task (migrated from HUDI-2405) rather than a bug — it proposes extending the HoodieTestTable utilities to manipulate commit/deltacommit, compaction, cleaning, replace/clustering, savepoint, rollback and restore transitions, and to optionally operate on real HoodieRecords for end-to-end data-integrity testing. HoodieTestTable exists today (hudi-hadoop-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java) and has been incrementally enhanced, but the full scope in the design doc is not a single closeable deliverable. 
   
   Keeping open as an unfinished test-infrastructure improvement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org