You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "yuzelin (via GitHub)" <gi...@apache.org> on 2023/07/17 02:27:15 UTC

[GitHub] [incubator-paimon] yuzelin opened a new issue, #1571: [Feature] Introduce file clean action

yuzelin opened a new issue, #1571:
URL: https://github.com/apache/incubator-paimon/issues/1571

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
   Currently, the `FileStoreExpireImpl` delete file quietly, which means if there occurs error, the process won't throw exceptions. It's possible that some files won't be deleted when snapshot expiring, so I propose to introduce an action to clean files unused by current snapshots. 
   
   ### Solution
   
   Scan all files and filter out that still used by at least one snapshot, then delete those not used by any snapshot.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Introduce file clean action [incubator-paimon]

Posted by "john8628 (via GitHub)" <gi...@apache.org>.
john8628 commented on issue #1571:
URL: https://github.com/apache/incubator-paimon/issues/1571#issuecomment-1762893252

   Apologies, I am unable to continue working on this PR due to other commitments,and I will need to close this pull request for now as the work is incomplete.  @john8628 @JingsongLi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-paimon] yuzelin commented on issue #1571: [Feature] Introduce file clean action

Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin commented on issue #1571:
URL: https://github.com/apache/incubator-paimon/issues/1571#issuecomment-1657392217

   > can I have a try?
   
   Hi, john, thanks. Surely you can just push some commits.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Introduce file clean action [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #1571: [Feature] Introduce file clean action
URL: https://github.com/apache/incubator-paimon/issues/1571


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-paimon] JingsongLi commented on issue #1571: [Feature] Introduce file clean action

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1571:
URL: https://github.com/apache/incubator-paimon/issues/1571#issuecomment-1665143627

   @john8628 You can just create a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-paimon] john8628 commented on issue #1571: [Feature] Introduce file clean action

Posted by "john8628 (via GitHub)" <gi...@apache.org>.
john8628 commented on issue #1571:
URL: https://github.com/apache/incubator-paimon/issues/1571#issuecomment-1656999461

   can I have a try?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-paimon] john8628 commented on issue #1571: [Feature] Introduce file clean action

Posted by "john8628 (via GitHub)" <gi...@apache.org>.
john8628 commented on issue #1571:
URL: https://github.com/apache/incubator-paimon/issues/1571#issuecomment-1659422916

   > > can I have a try?
   > 
   > Hi, @john8628, thanks. Surely you can just push some commits.
   
   thanks. I will give some implements using Flink action . The steps will be:
   1. Perform an initial scan of all the data files ,  A.
   2. Obtain the files for all snapshots files  ,  B.
   3. Calculate the set difference A - B to find new files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org