You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 11:34:35 UTC

[I] Improve RunCompaction Procedure does not run for all pending compactions when op is scheduleAndExecute [hudi]

hudi-bot opened a new issue, #16828:
URL: https://github.com/apache/hudi/issues/16828

   The current op modes for RunCompactionProcedure are as follows:
   
    
    # schedule - schedule a new plan
    # execute - if specific instants exist, execute them, otherwise execute all pending plans
    # scheduleandexecute - schedule a new plan and then execute it, if no plan is generated during schedule, execute all pending plans
   
    
   
   While the current implementation of the code holds true to above specification, it is not very user friendly.
   
    
   
   There is no option to schedule a new compaction plan, and execute all pending compaction plans. If a previous `scheduleandexecute` fails the user's automated job retries, it will generate a new compaction plan, leaving a pending compaction plan on the table. 
   
    
   
   If there is no user intervention, some log files may grow exponentially, making compaction more computationally expensive.
   
    
   
   So, this ticket is proposing to change the `scheduleandexecute` op to the following implementation:
    # schedule a new plan and then execute {color:#de350b}*ALL*{color} pending plans, if no plan is generated during schedule, execute all pending plans
   
    
   
   This would make the RunCompactionProcedure more user-friendly under workflows that are triggered on a set frequency.
   
   If user would like to control the number of pending plans (to 1) that they would like to execute during ad-hoc runs, they can still do so by using the LIMIT keyword.
   
   Take note that LIMITING to 1 will not execute the compaction that is just scheduled, instead, it will execute the oldest pending plan.
   
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-8970
   - Type: Improvement


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Improve RunCompaction Procedure does not run for all pending compactions when op is scheduleAndExecute [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous closed issue #16828: Improve RunCompaction Procedure does not run for all pending compactions when op is scheduleAndExecute
URL: https://github.com/apache/hudi/issues/16828


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Improve RunCompaction Procedure does not run for all pending compactions when op is scheduleAndExecute [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #16828:
URL: https://github.com/apache/hudi/issues/16828#issuecomment-3630131072

   ### Linked PR(s)
   
   - https://github.com/apache/hudi/pull/12794


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org