Re: [PR] Flink-1.19: Fix the file offset mismatch when Flink reader first seek… [iceberg]

["zhongyujiang (via GitHub)" <gi...@apache.org>]

zhongyujiang commented on PR #10567:
URL: https://github.com/apache/iceberg/pull/10567#issuecomment-2295217325

   > What if we test the reader operator recovery by using OneInputStreamOperatorTestHarness? We can store the state of the operator, and later use it to restore the reader
   
    Hi @pvary sorry for the late reply.
   
   I found that I cannot directly use `OneInputStreamOperatorTestHarness` for testing. I noticed that we currently use it to test the `StreamingReaderOperator`, but the input for `StreamingReaderOperato`r is `FlinkInputSplit` (corresponding to a `CombinedScanTask`), and it processes at least one completed split each time. This means it does not perform checkpoints while consuming partial splits.
   
   I looked into whether there might be other utility classes that would allow me to write an end-to-end unit test, but I couldn't find any. Do you have any other suggestions?
   
   Additionally, I added a test `testDataIteratorWithResidualFilter` in the new commit, where I correlate the values in the records with their respective files order in the task, to validate that the `fileOffset` in `DataIterator` might be incorrect when reading records (hence, the checkpoint would also record incorrect information). I believe this also proves the error present in `DataIterator`. Could you please take a look? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org