You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "yungkei (via GitHub)" <gi...@apache.org> on 2025/10/27 15:58:22 UTC

[I] [Bug] There are code risks in tag expiration, which can cause data corruption [paimon]

yungkei opened a new issue, #6479:
URL: https://github.com/apache/paimon/issues/6479

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   I found this mistake in version 1.9.0, and it still exists in the master branch.
   
   ### Compute Engine
   
   flink version1.16, spark version 3.3.1
   
   ### Minimal reproduce step
   
   If the baseManifestList or deltaManifestList associated with the tag are deleted in advance, the datafile will be deleted mistakenly during tag cleaning, which can cause data corruption, especially since the datafile is associated with the earliests snapshot.
   
   step1: delete baseManifestList or deltaManifestList associated with the tag, The premise is that the tag expiration time is greater than the snapshot expiration time
   step2: execute expired tag program
   step3: query the current snapshot or the earliest snapshot data, we will find a FileNotFoundException about the orc file
   
   ### What doesn't meet your expectations?
   
   This issue will result in datafile loss, and cause paimon unavailable.
   
   ### Anything else?
   
   When a tag expires, the left neighbor tag and the nearest right neighbor tag will be collected in skipping sets to prevent the datafile from being mistakenly deleted. if baseManifestList of the nearest right neighbor tag does not exist, the relevant datafiles will be accidentally deleted. So, I suggest the skipping set can collect both the left neighbor tag and the nearest right neighbor tag, along with the earliest snapshot.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] There are code risks in tag expiration, which can cause data corruption [paimon]

Posted by "yungkei (via GitHub)" <gi...@apache.org>.
yungkei commented on issue #6479:
URL: https://github.com/apache/paimon/issues/6479#issuecomment-3477905694

   I have checked the code and it has been fixed in the new version(https://github.com/apache/paimon/pull/5237). And, I will close this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] There are code risks in tag expiration, which can cause data corruption [paimon]

Posted by "yungkei (via GitHub)" <gi...@apache.org>.
yungkei commented on issue #6479:
URL: https://github.com/apache/paimon/issues/6479#issuecomment-3452117192

   I submit a PR https://github.com/apache/paimon/pull/6480


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] There are code risks in tag expiration, which can cause data corruption [paimon]

Posted by "yungkei (via GitHub)" <gi...@apache.org>.
yungkei closed issue #6479: [Bug] There are code risks in tag expiration, which can cause data corruption
URL: https://github.com/apache/paimon/issues/6479


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org