You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "GuanghuiCong (via GitHub)" <gi...@apache.org> on 2026/08/17 11:43:42 UTC

[I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

GuanghuiCong opened a new issue, #19649:
URL: https://github.com/apache/hudi/issues/19649

   ### Bug Description
   
   **Describe the problem**
   
   We encountered an issue with Hudi LSM Timeline archival when running a Flink streaming job.
   
   A transient HDFS replication failure occurred while Hudi was creating a new LSM timeline manifest file.
   
   Hudi created a temporary manifest file with the following name:
   
   manifest_114.a5e022a6-e5c9-4450-b9e5-9296262329b5
   
   
   
   The HDFS file close operation failed because the last block did not have enough replicas.
   
   After the Flink job restarted, this temporary manifest file remained under:
   
   .hoodie/timeline/history/
   
   
   On subsequent archive/clean operations, Hudi scanned this file and attempted to parse:
   
   114.a5e022a6-e5c9-4450-b9e5-9296262329b5
   
   as an integer manifest version.
   
   This caused:
   
   java.lang.NumberFormatException:
   For input string: "114.a5e022a6-e5c9-4450-b9e5-9296262329b5"
   
   The job then repeatedly failed every time timeline archival was triggered.
   
   The important point is that the original HDFS issue was transient, but the leftover temporary manifest made the Hudi table unable to recover automatically.
   
   ### Environment
   
   Hudi version:1.0.2
   
   Flink version:1.18.1
   
   Java:1.8.0_281
   
   Table type:MERGE_ON_READ
   
   Storage:HDFS
   
   Hudi bundle:hudi-flink1.18-bundle-1.0.2.jar
   
   
   ### Logs and Stack Trace
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

Posted by "GuanghuiCong (via GitHub)" <gi...@apache.org>.

GuanghuiCong commented on issue #19649:
URL: https://github.com/apache/hudi/issues/19649#issuecomment-5324666398

   @cshuo Thanks! I’ll work on the fix and submit a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 closed issue #19649: [BUG] LSM Timeline archival leaves manifest_<version>.<uuid> after HDFS write failure, causing persistent NumberFormatException
URL: https://github.com/apache/hudi/issues/19649


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #19649:
URL: https://github.com/apache/hudi/issues/19649#issuecomment-5328376736

   Here is the fix: https://github.com/apache/hudi/pull/19659


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

Posted by "GuanghuiCong (via GitHub)" <gi...@apache.org>.

GuanghuiCong commented on issue #19649:
URL: https://github.com/apache/hudi/issues/19649#issuecomment-5352165916

   @danny0405 
   Thanks for the detailed analysis and the quick fix.
   
   The root cause and the fix are clear now. I’ll verify the fix in our environment and also check whether we need to backport it to Hudi 1.0.2.
   
   Thanks again for your help!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

Posted by "GuanghuiCong (via GitHub)" <gi...@apache.org>.

GuanghuiCong commented on issue #19649:
URL: https://github.com/apache/hudi/issues/19649#issuecomment-5324241550

   Thanks for confirming the root cause and the proposed fixes.
   
   Will you submit a PR for these two changes, or would you prefer me to work on a fix and submit a PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #19649:
URL: https://github.com/apache/hudi/issues/19649#issuecomment-5327806999

   @GuanghuiCong thx for the report, the failure case is valid, the file should be ignored regularly since the pointer file(_version file) has not been updated on the read route which should point to a valid manifest file, I will check whether there is possibility the pointer file is missing, in that case, the LSM timeline reader will fallback to list all the manifest files instead which could include the invalid file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.

cshuo commented on issue #19649:
URL: https://github.com/apache/hudi/issues/19649#issuecomment-5323469331

   Thks for reporting the issue, I think we need to fix this in two places:
   
   1. In `LSMTimeline#getManifestFilePathFilter()` and `getManifestVersion()`, only files matching `^manifest_\\d+$` should be treated as valid manifests. Temporary files like `manifest_114.<uuid>` should simply be ignored.  
   
   2. In `HoodieStorage#createImmutableFileInPath()`, the temporary file should only be renamed after both write and close succeed. If write, close, or rename fails, we should try to clean it up without hiding the original exception.  
   
   For now, the leftover `manifest_114.<uuid>` can be deleted directly, as long as no timeline archival is running. That should unblock the job, but the code changes are still needed to prevent it from happening again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [BUG] LSM Timeline archival leaves manifest_. after HDFS write failure, causing persistent NumberFormatException [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.

cshuo commented on issue #19649:
URL: https://github.com/apache/hudi/issues/19649#issuecomment-5324533980

   @GuanghuiCong Feel free to work on the fix and submit a PR. Contributions are welcome.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org