[GitHub] [hudi] xushiyan commented on pull request #3595: [HUDI-2395] Rewrite metadata tests using HoodieTestTable

[GitBox <gi...@apache.org>]

xushiyan commented on pull request #3595:
URL: https://github.com/apache/hudi/pull/3595#issuecomment-914038407


   > In general, I want to make sure the new methods added to HoodieTestTable are in line with its design. @xushiyan any comments on that?
   
   @vinothchandar @nsivabalan At the beginning we were thinking make `HoodieTestTable` provide concise APIs doing basic table operations with empty files and `HoodieWriteableTestTable` writing actual data. `FileCreateUtils` is used by `HoodieTestTable` at lower level interfacing with the actual files. When more and more APIs are introduced, we have to re-write many logics and re-implement action-to-file-change translations. This resulted in the testutils having cumbersome code and learning hurdle, also error-prone. I felt some re-design is needed. 
   
   Some thoughts on the re-design: developers are familiar with HoodieXXXClient so we would need `HoodieTestTable` make use of some client with similar public APIs, say `HoodieTestTableClient`. 
   - HoodieTestTable owns a HoodieTestTableClient and exposes the client's APIs for dev to prep their own data
   - HoodieTestTableClient should manipulate timeline and metadata as normal
   - For UTs don't need real data, the client writes out empty log and base files.
   - For FTs, the test table and client should be aware of the EngineContext set by `XXXFunctionalTestHarness` and act like the actual clients
   - Eventually dev only learn APIs provided by `HoodieTestTableClient` to prep their tests. It can include high-level APIs like `.write100RecordsIn3Partitions()` as such.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org