You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2025/12/30 18:35:25 UTC

[I] Investigate data.copy() workaround in HoodieSparkLanceReader [hudi]

rahil-c opened a new issue, #17754:
URL: https://github.com/apache/hudi/issues/17754

   ### Task Description
   
   **What needs to be done:**
   
   Investigate the need for this .copy() workaround in HoodieSparkLanceReader
   
   When running the following `testMultipleRegularInsertsWithCommitValidation`, noticed the following data correctedness issue where the actualDF seems to have had one row copied multiple times.
   <img width="714" height="649" alt="Image" src="https://github.com/user-attachments/assets/a2b605eb-1475-4a3c-842e-34d4d2afdead" />
   
   Currently in order to workaround this issue I noticed that in the 
   
   ```
   @Override
     public ClosableIterator<HoodieRecord<InternalRow>> getRecordIterator(HoodieSchema schema) throws IOException {
       ClosableIterator<UnsafeRow> iterator = getUnsafeRowIterator(schema);
       //TODO .copy() is needed for correctness,
       return new CloseableMappingIterator<>(iterator, data -> unsafeCast(new HoodieSparkRecord(data)));
     }
   ```
   when I applied the `data.copy()` I was able to get past this issue. 
   
   <img width="690" height="650" alt="Image" src="https://github.com/user-attachments/assets/afae1dd5-81a9-4945-966d-5e523a140b09" />
   
   ### Task Type
   
   Code improvement/refactoring
   
   ### Related Issues
   
   **Parent feature issue:** (if applicable )
   **Related issues:**
   NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Investigate data.copy() workaround in HoodieSparkLanceReader [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on issue #17754:
URL: https://github.com/apache/hudi/issues/17754#issuecomment-3701327487

   Discussed with Tim on https://github.com/apache/hudi/pull/17731#discussion_r2654326983 will close out


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Investigate data.copy() workaround in HoodieSparkLanceReader [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c closed issue #17754: Investigate data.copy() workaround in HoodieSparkLanceReader
URL: https://github.com/apache/hudi/issues/17754


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org