You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xccui (via GitHub)" <gi...@apache.org> on 2023/06/12 13:17:42 UTC
[GitHub] [hudi] xccui opened a new issue, #8940: [SUPPORT] Too frequent compaction plan generation
xccui opened a new issue, #8940:
URL: https://github.com/apache/hudi/issues/8940
**Describe the problem you faced**
The compaction plan generation scheduler seems to be not working properly for our Flink Hudi writer job. We used the following settings and saw too frequent plan generation for a single table in JobManager logs.
Flink checkpoint settings:
```
flinkcheckpoint.interval: 5 mins
flinkcheckpoint.min.interval: 3 mins
```
Hudi compaction settings:
```
compaction.trigger.strategy: num_and_time
compaction.delta_commits: 5
compaction.delta_seconds: 1200
```
logs:
```
2023-06-08 19:56:50 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler.
2023-06-08 19:56:50 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table
...
2023-06-08 20:04:03 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler.
2023-06-08 20:04:03 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table
...
2023-06-08 20:11:35 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler.
2023-06-08 20:11:35 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table
...
2023-06-08 20:18:36 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler.
2023-06-08 20:18:36 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table
...
2023-06-08 20:25:58 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The delta commits >= 5 and elapsed_time >=1200s, trigger compaction scheduler.
2023-06-08 20:25:58 [pool-43-thread-1] INFO org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://path-to-table
```
I checked `ScheduleCompactionActionExecutor.getLatestDeltaCommitInfo()`. It checks time-escaped/delta-commits-number from the last commit. I wonder if we should replace "last commit" from "last commit request".
**Environment Description**
* Hudi version : 0.13.1
* Flink version : 0.16.1
* Hadoop version : 3.3.4
* Storage (HDFS/S3/GCS..) : S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #8940: [SUPPORT] Too frequent compaction plan generation
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui commented on issue #8940:
URL: https://github.com/apache/hudi/issues/8940#issuecomment-1592442100
Ah, this policy should work for us. Thank you, @danny0405! I'll close this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8940: [SUPPORT] Too frequent compaction plan generation
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8940:
URL: https://github.com/apache/hudi/issues/8940#issuecomment-1589005454
There is a compaction trigger strategy named: `NUM_COMMITS_AFTER_LAST_REQUEST`, is that what you need?
It is introduced in https://github.com/apache/hudi/pull/6144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] big-doudou commented on issue #8940: [SUPPORT] Too frequent compaction plan generation
Posted by "big-doudou (via GitHub)" <gi...@apache.org>.
big-doudou commented on issue #8940:
URL: https://github.com/apache/hudi/issues/8940#issuecomment-1588457063
I also encountered this situation. compaction is not performing as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui closed issue #8940: [SUPPORT] Too frequent compaction plan generation
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui closed issue #8940: [SUPPORT] Too frequent compaction plan generation
URL: https://github.com/apache/hudi/issues/8940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org