[GitHub] [hudi] HarjeetSinghGoldy commented on issue #9641: Get only latest records even after already writen existing data.

["HarjeetSinghGoldy (via GitHub)" <gi...@apache.org>]

HarjeetSinghGoldy commented on issue #9641:
URL: https://github.com/apache/hudi/issues/9641#issuecomment-1711098542

   I have attempted to use "precombineField" with the value set as "updatedAt," but it's only necessary to configure this field before writing the data into the Hudi table.
   
   Here's the broader context: I'm dealing with a daily batch processing scenario involving multiple tables labelled as "t1." Each table corresponds to a specific date, and they contain data in CSV format. These batches are organized as follows:
   
   Batch 1 = 2023-10-01
   Batch 2 = 2023-10-02
   Batch 3 = 2023-10-03
   Batch 4 = 2023-10-04
   
   When I ingest these batches in chronological order (from Batch 1 to Batch 4), everything functions correctly. However, the challenge arises when I ingest these batches in a jumbled or random sequence. In such cases, Hudi tends to treat the records from the last ingested batch as the latest records, potentially causing data inaccuracies.
   
   This context provides a better understanding of the issue I'm facing with the sequencing of batch ingestion and how it impacts the behaviour of the Hudi table.
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org