You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/26 08:25:32 UTC
[GitHub] [hudi] f7753 opened a new pull request, #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
f7753 opened a new pull request, #6218:
URL: https://github.com/apache/hudi/pull/6218
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Previously, we can not support more than 128 insert tasks in flink writer, since we use Flink's `StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM`, which is 128.
## Brief change log
Add WRITE_BULK_INSERT_MAX_PARALLELISM configuration in `FlinkOptions`.
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] f7753 commented on a diff in pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
f7753 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r934517252
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -137,7 +136,7 @@ public static DataStreamSink<Object> bulkInsert(Configuration conf, RowType rowT
// use #partitionCustom instead of #keyBy to avoid duplicate sort operations,
// see BatchExecutionUtils#applyBatchExecutionSettings for details.
Partitioner<String> partitioner = (key, channels) ->
- KeyGroupRangeAssignment.assignKeyToParallelOperator(key, StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM, channels);
+ KeyGroupRangeAssignment.assignKeyToParallelOperator(key, KeyGroupRangeAssignment.UPPER_BOUND_MAX_PARALLELISM, channels);
dataStream = dataStream.partitionCustom(partitioner, rowDataKeyGen::getPartitionPath);
Review Comment:
Could I use `conf.getInteger(FlinkOptions.WRITE_TASKS)` directly?
```java
// shuffle by partition keys
// use #partitionCustom instead of #keyBy to avoid duplicate sort operations,
// see BatchExecutionUtils#applyBatchExecutionSettings for details.
Partitioner<String> partitioner = (key, channels) -> KeyGroupRangeAssignment.assignKeyToParallelOperator(key,
conf.getInteger(FlinkOptions.WRITE_TASKS), channels);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1195279900
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22d458ad79c201859adf4c444bb92316b6168ad3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353)
* 5e175012e7e80e454f7ed0e2b93b003b0f3572e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r935067948
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -137,7 +136,7 @@ public static DataStreamSink<Object> bulkInsert(Configuration conf, RowType rowT
// use #partitionCustom instead of #keyBy to avoid duplicate sort operations,
// see BatchExecutionUtils#applyBatchExecutionSettings for details.
Partitioner<String> partitioner = (key, channels) ->
- KeyGroupRangeAssignment.assignKeyToParallelOperator(key, StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM, channels);
+ KeyGroupRangeAssignment.assignKeyToParallelOperator(key, KeyGroupRangeAssignment.UPPER_BOUND_MAX_PARALLELISM, channels);
dataStream = dataStream.partitionCustom(partitioner, rowDataKeyGen::getPartitionPath);
Review Comment:
No, we should use a max parallelism here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #6218:
URL: https://github.com/apache/hudi/pull/6218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1195284892
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22d458ad79c201859adf4c444bb92316b6168ad3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353)
* 5e175012e7e80e454f7ed0e2b93b003b0f3572e2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r931970563
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -487,6 +487,13 @@ private FlinkOptions() {
.defaultValue(true) // default true for MOR write
.withDescription("Schedule the compaction plan, enabled by default for MOR");
+ public static final ConfigOption<Integer> WRITE_BULK_INSERT_MAX_PARALLELISM = ConfigOptions
+ .key("write.bulk_insert.max.parallelism")
+ .intType()
+ .defaultValue(StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM)
+ .withDescription("The maximum number of flink write tasks, since the original " +
Review Comment:
What i mean is in `Pipelines` line 139, we hard code a greater value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r934198097
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -137,7 +136,7 @@ public static DataStreamSink<Object> bulkInsert(Configuration conf, RowType rowT
// use #partitionCustom instead of #keyBy to avoid duplicate sort operations,
// see BatchExecutionUtils#applyBatchExecutionSettings for details.
Partitioner<String> partitioner = (key, channels) ->
- KeyGroupRangeAssignment.assignKeyToParallelOperator(key, StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM, channels);
+ KeyGroupRangeAssignment.assignKeyToParallelOperator(key, KeyGroupRangeAssignment.UPPER_BOUND_MAX_PARALLELISM, channels);
dataStream = dataStream.partitionCustom(partitioner, rowDataKeyGen::getPartitionPath);
Review Comment:
Does `KeyGroupRangeAssignment#computeDefaultMaxParallelism` work here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] f7753 commented on a diff in pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
f7753 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r934501311
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -137,7 +136,7 @@ public static DataStreamSink<Object> bulkInsert(Configuration conf, RowType rowT
// use #partitionCustom instead of #keyBy to avoid duplicate sort operations,
// see BatchExecutionUtils#applyBatchExecutionSettings for details.
Partitioner<String> partitioner = (key, channels) ->
- KeyGroupRangeAssignment.assignKeyToParallelOperator(key, StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM, channels);
+ KeyGroupRangeAssignment.assignKeyToParallelOperator(key, KeyGroupRangeAssignment.UPPER_BOUND_MAX_PARALLELISM, channels);
dataStream = dataStream.partitionCustom(partitioner, rowDataKeyGen::getPartitionPath);
Review Comment:
It seems `KeyGroupRangeAssignment.computeDefaultMaxParallelism(conf.getInteger(FlinkOptions.WRITE_TASKS))` would be a possible solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] f7753 commented on a diff in pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
f7753 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r936190581
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -137,7 +136,7 @@ public static DataStreamSink<Object> bulkInsert(Configuration conf, RowType rowT
// use #partitionCustom instead of #keyBy to avoid duplicate sort operations,
// see BatchExecutionUtils#applyBatchExecutionSettings for details.
Partitioner<String> partitioner = (key, channels) ->
- KeyGroupRangeAssignment.assignKeyToParallelOperator(key, StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM, channels);
+ KeyGroupRangeAssignment.assignKeyToParallelOperator(key, KeyGroupRangeAssignment.UPPER_BOUND_MAX_PARALLELISM, channels);
dataStream = dataStream.partitionCustom(partitioner, rowDataKeyGen::getPartitionPath);
Review Comment:
OK, I've pushed another:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1195406916
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e175012e7e80e454f7ed0e2b93b003b0f3572e2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1200839049
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
}, {
"hash" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483",
"triggerID" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b80129568f1e7c9676fe2b13e5c0de08576e47db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1203364438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
}, {
"hash" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483",
"triggerID" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"triggerType" : "PUSH"
}, {
"hash" : "e8f46d2b1e05de82a5476d7de639f39aa015b840",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8f46d2b1e05de82a5476d7de639f39aa015b840",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b80129568f1e7c9676fe2b13e5c0de08576e47db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483)
* e8f46d2b1e05de82a5476d7de639f39aa015b840 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] f7753 commented on a diff in pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
f7753 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r934107862
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -487,6 +487,13 @@ private FlinkOptions() {
.defaultValue(true) // default true for MOR write
.withDescription("Schedule the compaction plan, enabled by default for MOR");
+ public static final ConfigOption<Integer> WRITE_BULK_INSERT_MAX_PARALLELISM = ConfigOptions
+ .key("write.bulk_insert.max.parallelism")
+ .intType()
+ .defaultValue(StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM)
+ .withDescription("The maximum number of flink write tasks, since the original " +
Review Comment:
i see, how about `org.apache.flink.runtime.state.KeyGroupRangeAssignment.UPPER_BOUND_MAX_PARALLELISM`, which is hardcoded to 32768? i guess it will work for almost all the workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1203367044
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
}, {
"hash" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483",
"triggerID" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"triggerType" : "PUSH"
}, {
"hash" : "e8f46d2b1e05de82a5476d7de639f39aa015b840",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10531",
"triggerID" : "e8f46d2b1e05de82a5476d7de639f39aa015b840",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b80129568f1e7c9676fe2b13e5c0de08576e47db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483)
* e8f46d2b1e05de82a5476d7de639f39aa015b840 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10531)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1195203375
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22d458ad79c201859adf4c444bb92316b6168ad3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r929726962
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -487,6 +487,13 @@ private FlinkOptions() {
.defaultValue(true) // default true for MOR write
.withDescription("Schedule the compaction plan, enabled by default for MOR");
+ public static final ConfigOption<Integer> WRITE_BULK_INSERT_MAX_PARALLELISM = ConfigOptions
+ .key("write.bulk_insert.max.parallelism")
+ .intType()
+ .defaultValue(StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM)
+ .withDescription("The maximum number of flink write tasks, since the original " +
Review Comment:
How about we just increase the `StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM` so we can avoid the option ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1200728264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
}, {
"hash" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e175012e7e80e454f7ed0e2b93b003b0f3572e2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354)
* b80129568f1e7c9676fe2b13e5c0de08576e47db UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1200731170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
}, {
"hash" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483",
"triggerID" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e175012e7e80e454f7ed0e2b93b003b0f3572e2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354)
* b80129568f1e7c9676fe2b13e5c0de08576e47db Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1195208622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22d458ad79c201859adf4c444bb92316b6168ad3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] f7753 commented on a diff in pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
f7753 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r929872960
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -487,6 +487,13 @@ private FlinkOptions() {
.defaultValue(true) // default true for MOR write
.withDescription("Schedule the compaction plan, enabled by default for MOR");
+ public static final ConfigOption<Integer> WRITE_BULK_INSERT_MAX_PARALLELISM = ConfigOptions
+ .key("write.bulk_insert.max.parallelism")
+ .intType()
+ .defaultValue(StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM)
+ .withDescription("The maximum number of flink write tasks, since the original " +
Review Comment:
Thank you for your advice:)
Yep, I was thinking about it at first sight, but after referring to `StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM`, it seems that the number can not be changed since it comes from `org.apache.flink.runtime.state.KeyGroupRangeAssignment.DEFAULT_LOWER_BOUND_MAX_PARALLELISM`, which was hard coded to 128.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] f7753 commented on a diff in pull request #6218: [HUDI-4477] Add an option for bulk insert tasks to adjust parallelism
Posted by GitBox <gi...@apache.org>.
f7753 commented on code in PR #6218:
URL: https://github.com/apache/hudi/pull/6218#discussion_r929872960
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -487,6 +487,13 @@ private FlinkOptions() {
.defaultValue(true) // default true for MOR write
.withDescription("Schedule the compaction plan, enabled by default for MOR");
+ public static final ConfigOption<Integer> WRITE_BULK_INSERT_MAX_PARALLELISM = ConfigOptions
+ .key("write.bulk_insert.max.parallelism")
+ .intType()
+ .defaultValue(StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM)
+ .withDescription("The maximum number of flink write tasks, since the original " +
Review Comment:
Yep, I was thinking about it at first sight, but after referring to `StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM`, it seems that the number can not be changed since it comes from `org.apache.flink.runtime.state.KeyGroupRangeAssignment.DEFAULT_LOWER_BOUND_MAX_PARALLELISM`, which was hard coded to 128.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6218: [HUDI-4477] Adjust partition number of flink sink task
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6218:
URL: https://github.com/apache/hudi/pull/6218#issuecomment-1203433057
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10353",
"triggerID" : "22d458ad79c201859adf4c444bb92316b6168ad3",
"triggerType" : "PUSH"
}, {
"hash" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10354",
"triggerID" : "5e175012e7e80e454f7ed0e2b93b003b0f3572e2",
"triggerType" : "PUSH"
}, {
"hash" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10483",
"triggerID" : "b80129568f1e7c9676fe2b13e5c0de08576e47db",
"triggerType" : "PUSH"
}, {
"hash" : "e8f46d2b1e05de82a5476d7de639f39aa015b840",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10531",
"triggerID" : "e8f46d2b1e05de82a5476d7de639f39aa015b840",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8f46d2b1e05de82a5476d7de639f39aa015b840 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10531)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org