[I] Not use magic number of next block to determine current log block [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #15081:
URL: https://github.com/apache/hudi/issues/15081

   HoodieLogFileReader use magic number of next log block to determine if current log block is corrupted. However, when next block has a corrupted magic number, we will abandon current block, which leads to data loss.
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-3694
   - Type: Bug
   - Fix version(s):
     - 1.1.0
   
   
   ---
   
   
   ## Comments
   
   30/Apr/23 18:15;shivnarayan;This might potentially be an issue only for hdfs like storage schemes where appends are feasible. ;;;
   
   ---
   
   02/May/23 11:17;codope;Unlike S3 or GCS, writes are not transactional in HDFS. So, in case of S3 and GCS we write out each log file and the atomicity is assured by the underlying filesystem i.e. either complete file is written or not written at all. While in HDFS, there can be partially written log blocks, and hence it is possible that magic header in the next log block is corrupt. 
   
   One possible solution is to throw {{CorruptedLogFileException}} from {{HoodieLogFileReader#isBlockCorrupted}} if the magic header is corrupt and {{{}StorageSchemes.isWriteTransaction{}}}.;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org