You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bithw1 (via GitHub)" <gi...@apache.org> on 2026/01/29 01:22:28 UTC
[I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
bithw1 opened a new issue, #18032:
URL: https://github.com/apache/hudi/issues/18032
### Describe the problem you faced
Hi,
I am using Hudi 0.15.0,
I would ask how the configuration `compaction.async.enabled` really works in Flink on Hudi.
Please correct me if I am wrong.
1. `compaction.async.enabled=false`, it means that flink will do compaction after normal data write to Hudi in the same flink job,so that it may cause data write latency.
2. `compaction.async.enabled=true` means that flink write and flink compaction run in the same flink job, but different threads?
Then, how to configure so that flink write job(with compaction schedule) and flink compaction job run two different processes.
### To Reproduce
1.
2.
3.
4.
### Expected behavior
1
### Environment Description
* Hudi version:
* Spark version:
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):
### Additional context
_No response_
### Stacktrace
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 closed issue #18032: Try to understand the meaning of configuration compaction.async.enabled
URL: https://github.com/apache/hudi/issues/18032
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #18032:
URL: https://github.com/apache/hudi/issues/18032#issuecomment-3822085381
Thanks @danny0405 ， I think I have got the answer,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18032:
URL: https://github.com/apache/hudi/issues/18032#issuecomment-3815717298
> I have thought that compaction.async.enabled=false means compaction is performed in sync mode.
Not true for Flink, flink only got async compaction for the online ingestion job now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18032:
URL: https://github.com/apache/hudi/issues/18032#issuecomment-3821891206
it's a batch job separate from the streaming ingestion workload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #18032:
URL: https://github.com/apache/hudi/issues/18032#issuecomment-3815757384
Thanks @danny0405 .
I think configuration `compaction.async.enabled` is totally misused. No one could think that `compaction.async.enabled=false` means schedule compaction plan, but not execute the compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #18032:
URL: https://github.com/apache/hudi/issues/18032#issuecomment-3815677146
Thanks @danny0405 for the reply.
You mean that, with `compaction.async.enabled=false` , flink will only schedule the compaction plan, and will not perform the real compaction. Other job is needed to do the compaction exuection?
I have thought that `compaction.async.enabled=false` means compaction is performed in sync mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18032:
URL: https://github.com/apache/hudi/issues/18032#issuecomment-3815518411
just set up `compaction.async.enabled` as false, the ingesion job would still schedule the compaction plan, the offline job could be utilitied to trigger the execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Try to understand the meaning of configuration compaction.async.enabled [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #18032:
URL: https://github.com/apache/hudi/issues/18032#issuecomment-3815771990
> just set up `compaction.async.enabled` as false, the ingesion job would still schedule the compaction plan, the offline job could be utilitied to trigger the execution.
@danny0405 What does the offline job mean here? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org