You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/02/23 08:35:32 UTC

[GitHub] [hudi] cdzryou opened a new issue #4882: [SUPPORT]How to clean action files theses are not used with flink task?

cdzryou opened a new issue #4882:
URL: https://github.com/apache/hudi/issues/4882


   - how to autu clean timeline action files in flink ? these files like `*.commit`,`.*deltacommit` , `.*clean` and in hdfs path '../.hoodie/'
   
   - it may create a lot of small files in flink streaming task, so i want to know that Hudi how to manage these files, delete them or not delete?
   
   - Flink Version 1.13.5
   - Hudi Version 0.10.1
   - Hadoop Version 3.0.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4882: [SUPPORT]How to clean action files theses are not used with flink task?

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4882:
URL: https://github.com/apache/hudi/issues/4882#issuecomment-1054639622


   @leesf @danny0405 : can you loop in someone to follow up on this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on issue #4882: [SUPPORT]How to clean action files theses are not used with flink task?

Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #4882:
URL: https://github.com/apache/hudi/issues/4882#issuecomment-1055174981


   For flink option, it is `archive.min_commits` default 20 and `archive.max_commits` 30, i would suggest you keep the default value as it is to avoid weird problem. Because in hoodie, many operations depend on the active timeline ~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4882: [SUPPORT]How to clean action files theses are not used with flink task?

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4882:
URL: https://github.com/apache/hudi/issues/4882#issuecomment-1067588792


   Feel free to close out the github issue if your question has been answered. if not, do follow up w/ more questions/clarifications.
   thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] cdzryou commented on issue #4882: [SUPPORT]How to clean action files theses are not used with flink task?

Posted by GitBox <gi...@apache.org>.
cdzryou commented on issue #4882:
URL: https://github.com/apache/hudi/issues/4882#issuecomment-1055152064


   > These files should be archived then, did your job have pending compactions on the timeline ?
   
   thanks for your comment.
   have no pending compactions, i run a job for a long time, and find these files were archived, files don't grow indefinitely.
   however, can I control the number of files by setting parameters? for example, i only want to keep 5 files or less


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on issue #4882: [SUPPORT]How to clean action files theses are not used with flink task?

Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #4882:
URL: https://github.com/apache/hudi/issues/4882#issuecomment-1055114614


   These files should be archived then, did your job have pending compactions on the timeline ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org