You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:04:23 UTC
[I] Add separate control for compaction operation sync/async mode [hudi]
hudi-bot opened a new issue, #15256:
URL: https://github.com/apache/hudi/issues/15256
*Problem Review*
The compact operation sync/async in CompactionFunction is now controlled by FlinkOptions#COMPACTION_ASYNC_ENABLED
{code:java}
public CompactFunction(Configuration conf) {
this.conf = conf;
this.asyncCompaction = StreamerUtil.needsAsyncCompaction(conf);
}
{code}
While in fact it cannot be switched to sync mode because the pipeline defined by sync compaction will only include the clean but not compact operators.
{code:java}
// compaction
if (StreamerUtil.needsAsyncCompaction(conf)) {
return Pipelines.compact(conf, pipeline);
} else {
return Pipelines.clean(conf, pipeline);
}
{code}
*Improvement*
Add another separate control switch for compaction operation sync/async mode.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4329
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add separate control for compaction operation sync/async mode [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #15256:
URL: https://github.com/apache/hudi/issues/15256#issuecomment-3610145562
The watermark conflicts issue has been addressed in https://github.com/apache/hudi/pull/8379, while I still deem this PR as valid, the suggestions:
1. rename `compaction.async.enabled` to `compaction.enabled`
2. store the isBounded option in the `conf` to switch to sync compaction for bounded source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add separate control for compaction operation sync/async mode [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #15256:
URL: https://github.com/apache/hudi/issues/15256#issuecomment-3604942494
pended pr: https://github.com/apache/hudi/pull/5991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add separate control for compaction operation sync/async mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15256:
URL: https://github.com/apache/hudi/issues/15256#issuecomment-3630115566
### Linked PR(s)
- https://github.com/apache/hudi/pull/5991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org