You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "eric9204 (via GitHub)" <gi...@apache.org> on 2023/06/02 02:22:53 UTC
[GitHub] [hudi] eric9204 opened a new pull request, #8871: [HUDI-6308]add num_commits_after_last_request to flink
eric9204 opened a new pull request, #8871:
URL: https://github.com/apache/hudi/pull/8871
### Change Logs
None
### Impact
None
### Risk level (write none, low medium or high below)
None
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on a diff in pull request #8871: [HUDI-6308]add num_commits_after_last_request to flink
Posted by "eric9204 (via GitHub)" <gi...@apache.org>.
eric9204 commented on code in PR #8871:
URL: https://github.com/apache/hudi/pull/8871#discussion_r1213950388
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -642,6 +642,7 @@ private FlinkOptions() {
public static final String TIME_ELAPSED = "time_elapsed";
public static final String NUM_AND_TIME = "num_and_time";
public static final String NUM_OR_TIME = "num_or_time";
+ public static final String NUM_COMMITS_AFTER_LAST_REQUEST = "num_commits_after_last_request";
@AdvancedConfig
public static final ConfigOption<String> COMPACTION_TRIGGER_STRATEGY = ConfigOptions
Review Comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 closed pull request #8871: [HUDI-6308]add num_commits_after_last_request to flink
Posted by "eric9204 (via GitHub)" <gi...@apache.org>.
eric9204 closed pull request #8871: [HUDI-6308]add num_commits_after_last_request to flink
URL: https://github.com/apache/hudi/pull/8871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on a diff in pull request #8871: [HUDI-6308]add num_commits_after_last_request to flink
Posted by "eric9204 (via GitHub)" <gi...@apache.org>.
eric9204 commented on code in PR #8871:
URL: https://github.com/apache/hudi/pull/8871#discussion_r1213939610
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -642,6 +642,7 @@ private FlinkOptions() {
public static final String TIME_ELAPSED = "time_elapsed";
public static final String NUM_AND_TIME = "num_and_time";
public static final String NUM_OR_TIME = "num_or_time";
+ public static final String NUM_COMMITS_AFTER_LAST_REQUEST = "num_commits_after_last_request";
@AdvancedConfig
public static final ConfigOption<String> COMPACTION_TRIGGER_STRATEGY = ConfigOptions
Review Comment:
@danny0405 Thank you for your reply.
By adding this parameter `'compaction.trigger.strategy'='num_commits_after_last_request'`, in my test, the job can generate a compaction plan for every fixed number of `deltacommits`.
I don't understand that "The variable may not be used anywhere". Is there a problem with this compaction trigger strategy in some scenarios?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8871: [HUDI-6308]add num_commits_after_last_request to flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8871:
URL: https://github.com/apache/hudi/pull/8871#discussion_r1213944708
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -642,6 +642,7 @@ private FlinkOptions() {
public static final String TIME_ELAPSED = "time_elapsed";
public static final String NUM_AND_TIME = "num_and_time";
public static final String NUM_OR_TIME = "num_or_time";
+ public static final String NUM_COMMITS_AFTER_LAST_REQUEST = "num_commits_after_last_request";
@AdvancedConfig
public static final ConfigOption<String> COMPACTION_TRIGGER_STRATEGY = ConfigOptions
Review Comment:
`NUM_COMMITS_AFTER_LAST_REQUEST` this variable is not used by any other code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8871: [HUDI-6308]add num_commits_after_last_request to flink
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8871:
URL: https://github.com/apache/hudi/pull/8871#issuecomment-1573073017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "88a2f49c4c02ce1ade13549ac56d6bf396411289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17570",
"triggerID" : "88a2f49c4c02ce1ade13549ac56d6bf396411289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88a2f49c4c02ce1ade13549ac56d6bf396411289 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17570)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8871: [HUDI-6308]add num_commits_after_last_request to flink
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8871:
URL: https://github.com/apache/hudi/pull/8871#discussion_r1213865107
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -642,6 +642,7 @@ private FlinkOptions() {
public static final String TIME_ELAPSED = "time_elapsed";
public static final String NUM_AND_TIME = "num_and_time";
public static final String NUM_OR_TIME = "num_or_time";
+ public static final String NUM_COMMITS_AFTER_LAST_REQUEST = "num_commits_after_last_request";
@AdvancedConfig
public static final ConfigOption<String> COMPACTION_TRIGGER_STRATEGY = ConfigOptions
Review Comment:
The varialbe may not be used anywhere, maybe we should just fix the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8871: [HUDI-6308]add num_commits_after_last_request to flink
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8871:
URL: https://github.com/apache/hudi/pull/8871#issuecomment-1573067586
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "88a2f49c4c02ce1ade13549ac56d6bf396411289",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88a2f49c4c02ce1ade13549ac56d6bf396411289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88a2f49c4c02ce1ade13549ac56d6bf396411289 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org