You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/13 03:03:48 UTC
[GitHub] [hudi] LinMingQiang opened a new pull request, #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode.
LinMingQiang opened a new pull request, #6093:
URL: https://github.com/apache/hudi/pull/6093
Signed-off-by: LinMingQiang <13...@qq.com>
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186928346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* 7b48af31ab96eee64a19193b7b75bb24253d3e83 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r928524041
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##########
@@ -95,6 +95,10 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context context) {
pipeline = Pipelines.hoodieStreamWrite(conf, parallelism, hoodieRecordDataStream);
// compaction
if (OptionsResolver.needsAsyncCompaction(conf)) {
+ // batch mode write must use syncCompaction.
+ if (context.isBounded()) {
+ conf.setBoolean(FlinkOptions.COMPACTION_ASYNC_ENABLED, false);
Review Comment:
Not exactly, because the bounded source can also be long running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r919954965
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -496,6 +496,12 @@ private FlinkOptions() {
// Compaction Options
// ------------------------------------------------------------------------
+ public static final ConfigOption<Boolean> COMPACTION_BATCH_MODE_ENABLED = ConfigOptions
+ .key("compaction.batch.mode.enabled")
+ .booleanType()
+ .defaultValue(false) // default false for MOR write
Review Comment:
You can overwrite the option in hoodie table sink, when we know that the source is bounded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r928363020
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##########
@@ -95,6 +95,10 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context context) {
pipeline = Pipelines.hoodieStreamWrite(conf, parallelism, hoodieRecordDataStream);
// compaction
if (OptionsResolver.needsAsyncCompaction(conf)) {
+ // batch mode write must use syncCompaction.
+ if (context.isBounded()) {
+ conf.setBoolean(FlinkOptions.COMPACTION_ASYNC_ENABLED, false);
Review Comment:
My idea is that when the source is bounded, we should not do compaction in checkpoint, because compaction will be done once in `endinput`. Am I right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1205046865
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
}, {
"hash" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197",
"triggerID" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"triggerType" : "PUSH"
}, {
"hash" : "1f7b6810974e2dd88810b04d56410471de55d466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10580",
"triggerID" : "1f7b6810974e2dd88810b04d56410471de55d466",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* 1f7b6810974e2dd88810b04d56410471de55d466 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10580)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1183101515
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f568684332c9f030751b8a492d13e979a6c260f3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186653728
@LinMingQiang @danny0405 can we aim to land this for 0.12?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186743165
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f568684332c9f030751b8a492d13e979a6c260f3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890)
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* 7b48af31ab96eee64a19193b7b75bb24253d3e83 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1192447036
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
}, {
"hash" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197",
"triggerID" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* dfda12d18c6f58afdf0e401d72640e560d92bb38 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186738081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f568684332c9f030751b8a492d13e979a6c260f3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890)
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186803549
👌🏻
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #6093:
URL: https://github.com/apache/hudi/pull/6093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r919638944
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -496,6 +496,12 @@ private FlinkOptions() {
// Compaction Options
// ------------------------------------------------------------------------
+ public static final ConfigOption<Boolean> COMPACTION_BATCH_MODE_ENABLED = ConfigOptions
+ .key("compaction.batch.mode.enabled")
+ .booleanType()
+ .defaultValue(false) // default false for MOR write
Review Comment:
Can we remove this option, default always true for MOR table is okey ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r919960659
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -496,6 +496,12 @@ private FlinkOptions() {
// Compaction Options
// ------------------------------------------------------------------------
+ public static final ConfigOption<Boolean> COMPACTION_BATCH_MODE_ENABLED = ConfigOptions
+ .key("compaction.batch.mode.enabled")
+ .booleanType()
+ .defaultValue(false) // default false for MOR write
Review Comment:
Our thoughts are the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186740532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f568684332c9f030751b8a492d13e979a6c260f3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890)
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* 7b48af31ab96eee64a19193b7b75bb24253d3e83 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1182733450
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6a0bcec0d17c7dd966938d86721142febd02c3d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1192432006
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
}, {
"hash" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* 7b48af31ab96eee64a19193b7b75bb24253d3e83 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020)
* dfda12d18c6f58afdf0e401d72640e560d92bb38 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1183052832
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6a0bcec0d17c7dd966938d86721142febd02c3d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880)
* f568684332c9f030751b8a492d13e979a6c260f3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1192435697
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
}, {
"hash" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197",
"triggerID" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* 7b48af31ab96eee64a19193b7b75bb24253d3e83 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020)
* dfda12d18c6f58afdf0e401d72640e560d92bb38 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1204978851
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
}, {
"hash" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197",
"triggerID" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"triggerType" : "PUSH"
}, {
"hash" : "1f7b6810974e2dd88810b04d56410471de55d466",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10580",
"triggerID" : "1f7b6810974e2dd88810b04d56410471de55d466",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* dfda12d18c6f58afdf0e401d72640e560d92bb38 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197)
* 1f7b6810974e2dd88810b04d56410471de55d466 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10580)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1204973425
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
}, {
"hash" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70f88aff79f2a3a3ddf4fbab153a5c264cb64e64",
"triggerType" : "PUSH"
}, {
"hash" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10020",
"triggerID" : "7b48af31ab96eee64a19193b7b75bb24253d3e83",
"triggerType" : "PUSH"
}, {
"hash" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197",
"triggerID" : "dfda12d18c6f58afdf0e401d72640e560d92bb38",
"triggerType" : "PUSH"
}, {
"hash" : "1f7b6810974e2dd88810b04d56410471de55d466",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f7b6810974e2dd88810b04d56410471de55d466",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70f88aff79f2a3a3ddf4fbab153a5c264cb64e64 UNKNOWN
* dfda12d18c6f58afdf0e401d72640e560d92bb38 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10197)
* 1f7b6810974e2dd88810b04d56410471de55d466 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1204778626
Thanks for the fix, i have applied a fix patch:
[4385.patch.zip](https://github.com/apache/hudi/files/9256858/4385.patch.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r927388993
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##########
@@ -95,6 +95,10 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context context) {
pipeline = Pipelines.hoodieStreamWrite(conf, parallelism, hoodieRecordDataStream);
// compaction
if (OptionsResolver.needsAsyncCompaction(conf)) {
+ // batch mode write must use syncCompaction.
+ if (context.isBounded()) {
+ conf.setBoolean(FlinkOptions.COMPACTION_ASYNC_ENABLED, false);
Review Comment:
In streaming exec mode, a bounded source would also triggers checkpoint, should we disable the async compaction for them ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1186705005
I think we can cc @LinMingQiang ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1182762470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6a0bcec0d17c7dd966938d86721142febd02c3d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r919855218
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -496,6 +496,12 @@ private FlinkOptions() {
// Compaction Options
// ------------------------------------------------------------------------
+ public static final ConfigOption<Boolean> COMPACTION_BATCH_MODE_ENABLED = ConfigOptions
+ .key("compaction.batch.mode.enabled")
+ .booleanType()
+ .defaultValue(false) // default false for MOR write
Review Comment:
But I need a parameter to tell `CompactFunction` which to use `asyncCompaction` or `syncCompaction`, Batch mode must use `syncCompaction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1182735247
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6a0bcec0d17c7dd966938d86721142febd02c3d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r919694474
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -496,6 +496,12 @@ private FlinkOptions() {
// Compaction Options
// ------------------------------------------------------------------------
+ public static final ConfigOption<Boolean> COMPACTION_BATCH_MODE_ENABLED = ConfigOptions
+ .key("compaction.batch.mode.enabled")
+ .booleanType()
+ .defaultValue(false) // default false for MOR write
Review Comment:
👌🏻
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r919638944
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -496,6 +496,12 @@ private FlinkOptions() {
// Compaction Options
// ------------------------------------------------------------------------
+ public static final ConfigOption<Boolean> COMPACTION_BATCH_MODE_ENABLED = ConfigOptions
+ .key("compaction.batch.mode.enabled")
+ .booleanType()
+ .defaultValue(false) // default false for MOR write
Review Comment:
Can we remove this option ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6093:
URL: https://github.com/apache/hudi/pull/6093#issuecomment-1183056777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880",
"triggerID" : "d6a0bcec0d17c7dd966938d86721142febd02c3d",
"triggerType" : "PUSH"
}, {
"hash" : "f568684332c9f030751b8a492d13e979a6c260f3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890",
"triggerID" : "f568684332c9f030751b8a492d13e979a6c260f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6a0bcec0d17c7dd966938d86721142febd02c3d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9880)
* f568684332c9f030751b8a492d13e979a6c260f3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9890)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r927388993
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSink.java:
##########
@@ -95,6 +95,10 @@ public SinkRuntimeProvider getSinkRuntimeProvider(Context context) {
pipeline = Pipelines.hoodieStreamWrite(conf, parallelism, hoodieRecordDataStream);
// compaction
if (OptionsResolver.needsAsyncCompaction(conf)) {
+ // batch mode write must use syncCompaction.
+ if (context.isBounded()) {
+ conf.setBoolean(FlinkOptions.COMPACTION_ASYNC_ENABLED, false);
Review Comment:
In streaming exec mode, bounded source would also trigger checkpoints, should we disable the async compaction for them ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6093: [HUDI-4385] Support to trigger the compaction in the flink batch mode write.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6093:
URL: https://github.com/apache/hudi/pull/6093#discussion_r927388386
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -428,13 +428,17 @@ private void handleEndInputEvent(WriteMetadataEvent event) {
addEventToBuffer(event);
if (allEventsReceived()) {
// start to commit the instant.
- boolean committed = commitInstant(this.instant);
- if (committed) {
- // The executor thread inherits the classloader of the #handleEventFromOperator
Review Comment:
Why removes the check ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org