You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "HuangZhenQiu (via GitHub)" <gi...@apache.org> on 2026/02/04 20:19:46 UTC
[PR] feat: support flink split distribution strategy [hudi]
HuangZhenQiu opened a new pull request, #18082:
URL: https://github.com/apache/hudi/pull/18082
### Describe the issue this Pull Request addresses
Support flink split distribution strategy
### Summary and Changelog
1. Add default split assigner and bucket based assigner
2. Update the split provider to provide split according to the assigner algorithm
3. Add test cases for new classes.
### Impact
none
### Risk Level
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766500754
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/HoodieSource.java:
##########
@@ -126,16 +130,26 @@ private SplitEnumerator<HoodieSourceSplit, HoodieSplitEnumeratorState> createEnu
SplitEnumeratorContext<HoodieSourceSplit> enumContext,
@Nullable HoodieSplitEnumeratorState enumeratorState) {
HoodieSplitProvider splitProvider;
+ HoodieSplitAssigner splitAssigner;
+
+ if (OptionsResolver.isMorWithBucketIndexUpsert(scanContext.getConf())) {
+ splitAssigner = new HoodieSplitBucketAssigner(enumContext.currentParallelism());
+ } else {
+ splitAssigner = new DefaultHoodieSplitAssigner(enumContext.currentParallelism());
+ }
+
if (enumeratorState == null) {
- splitProvider = new DefaultHoodieSplitProvider(new HoodieSourceSplitComparator());
+ splitProvider = new DefaultHoodieSplitProvider(
+ new HoodieSourceSplitComparator(),
+ splitAssigner);
} else {
LOG.info(
"Hoodie source restored {} splits from state for table {}",
enumeratorState.getPendingSplitStates().size(),
metaClient.getTableConfig().getTableName());
List<HoodieSourceSplit> pendingSplits =
enumeratorState.getPendingSplitStates().stream().map(HoodieSourceSplitState::split).collect(Collectors.toList());
- splitProvider = new DefaultHoodieSplitProvider();
+ splitProvider = new DefaultHoodieSplitProvider(splitAssigner);
Review Comment:
when restore from state, comparator is not passed to this constructor, this can lead to inconsistent ordering of splits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3857910869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
}, {
"hash" : "840e7eb77889632c2a93274c73d00252b012c26f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753",
"triggerID" : "840e7eb77889632c2a93274c73d00252b012c26f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b07ee0df16a4c56bab40bfcdb47c775c36d3758 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710)
* 840e7eb77889632c2a93274c73d00252b012c26f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3858174860
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
}, {
"hash" : "840e7eb77889632c2a93274c73d00252b012c26f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753",
"triggerID" : "840e7eb77889632c2a93274c73d00252b012c26f",
"triggerType" : "PUSH"
}, {
"hash" : "c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11755",
"triggerID" : "c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b07ee0df16a4c56bab40bfcdb47c775c36d3758 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710)
* 840e7eb77889632c2a93274c73d00252b012c26f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753)
* c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11755)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3849580896
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bbd60b5060f1b0b384402e87751567e37aaffb13 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850656449
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 927e507f70906fb42135ca453c4a4414d28b8932 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3849730083
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bbd60b5060f1b0b384402e87751567e37aaffb13 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3849861075
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bbd60b5060f1b0b384402e87751567e37aaffb13 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661)
* 414cdb19c16c9b25308caf0908eea73f97780617 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850286174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 414cdb19c16c9b25308caf0908eea73f97780617 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663)
* 6e07495142a88908c64304650574c739c144132f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673)
* 927e507f70906fb42135ca453c4a4414d28b8932 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766996385
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
`maxParallelism` is a constant value: https://github.com/apache/flink/blob/e8ac5b12b4f7188b86cb3ecd9fc9ee5d9ccc3410/flink-runtime/src/main/java/org/apache/flink/streaming/api/datastream/KeyedStream.java#L138;
numberOfChannels is the source parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850270745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 414cdb19c16c9b25308caf0908eea73f97780617 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663)
* 6e07495142a88908c64304650574c739c144132f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850964884
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 927e507f70906fb42135ca453c4a4414d28b8932 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677)
* c6306dd367d2d76656cd092910f737ce6bf66743 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2772001016
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/assign/TestDefaultHoodieSplitAssigner.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Test cases for {@link DefaultHoodieSplitAssigner}.
+ */
+public class TestDefaultHoodieSplitAssigner {
+
+ @Test
+ public void testAssignWithSingleParallelism() {
+ DefaultHoodieSplitAssigner assigner = new DefaultHoodieSplitAssigner(1);
+
+ HoodieSourceSplit split1 = createTestSplit(0, "file1");
+ HoodieSourceSplit split2 = createTestSplit(1, "file2");
+ HoodieSourceSplit split3 = createTestSplit(2, "file3");
+
+ assertEquals(0, assigner.assign(split1), "All splits should be assigned to task 0");
+ assertEquals(0, assigner.assign(split2), "All splits should be assigned to task 0");
+ assertEquals(0, assigner.assign(split3), "All splits should be assigned to task 0");
+ }
+
+ @Test
+ public void testAssignWithMultipleParallelism() {
+ DefaultHoodieSplitAssigner assigner = new DefaultHoodieSplitAssigner(3);
+
+ // DefaultHoodieSplitAssigner uses KeyGroupRangeAssignment which uses hash-based assignment
+ // We test that it assigns consistently and all assignments are within range
+ HoodieSourceSplit split0 = createTestSplit(0, "file0");
+ HoodieSourceSplit split1 = createTestSplit(1, "file1");
+ HoodieSourceSplit split2 = createTestSplit(2, "file2");
+
+ // Verify assignments are within valid range [0, parallelism)
+ int task0 = assigner.assign(split0);
+ int task1 = assigner.assign(split1);
+ int task2 = assigner.assign(split2);
+
+ assert task0 >= 0 && task0 < 3 : "Task ID should be in range [0, 3)";
+ assert task1 >= 0 && task1 < 3 : "Task ID should be in range [0, 3)";
+ assert task2 >= 0 && task2 < 3 : "Task ID should be in range [0, 3)";
Review Comment:
always use junit assert in test, not java assert keyword
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766842844
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
keyBy uses an internal stream partitioner `KeyGroupStreamPartitioner`, we can reuse its distribution logic directly: `KeyGroupRangeAssignment.assignKeyToParallelOperator(...)`.
https://github.com/apache/flink/blob/e8ac5b12b4f7188b86cb3ecd9fc9ee5d9ccc3410/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java#L63
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2767685471
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/DefaultHoodieSplitAssigner.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.flink.runtime.state.KeyGroupRangeAssignment;
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Default implementation of {@link HoodieSplitAssigner} that assigns Hoodie
+ * source splits to task IDs by hashing the file ID.
+ */
+public class DefaultHoodieSplitAssigner implements HoodieSplitAssigner {
+ private final int parallelism;
+
+ /**
+ * Creates a new DefaultHoodieSplitAssigner.
+ *
+ * @param parallelism the number of parallel tasks (must be positive)
+ * @throws IllegalArgumentException if parallelism is less than or equal to 0
+ */
+ public DefaultHoodieSplitAssigner(int parallelism) {
+ if (parallelism <= 0) {
+ throw new IllegalArgumentException("Parallelism must be positive, but was: " + parallelism);
+ }
+ this.parallelism = parallelism;
+ }
+
+ @Override
+ public int assign(HoodieSourceSplit split) {
+ int maxParallelism = KeyGroupRangeAssignment.computeDefaultMaxParallelism(parallelism);
Review Comment:
`maxParallelism` can be moved as a class field, and initialized only once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766824703
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
Yes, Actually, it is what I want to double confirm with you. The keyby logic which has internal mapping logic in Flink is not easy to implement here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3857901714
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
}, {
"hash" : "840e7eb77889632c2a93274c73d00252b012c26f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "840e7eb77889632c2a93274c73d00252b012c26f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b07ee0df16a4c56bab40bfcdb47c775c36d3758 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710)
* 840e7eb77889632c2a93274c73d00252b012c26f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766763660
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/DefaultHoodieSplitAssigner.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+import java.io.Serializable;
+
+/**
+ * Default implementation of {@link HoodieSplitAssigner} that assigns Hoodie
+ * source splits to task IDs using round-robin distribution based on split number.
+ */
+public class DefaultHoodieSplitAssigner implements HoodieSplitAssigner, Serializable {
Review Comment:
Seems the split assigner doesn't need to be Serializable, it's a field of `DefaultHoodieSplitProvider`, which is not serializable.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
For legacy source, there are 3 kinds split distribution strategy:
1) distribute by fileId ;
2) distribute by bucketId;
3) distribute by split number;
Seems 1 is missed in this PR.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/DefaultHoodieSplitProvider.java:
##########
@@ -37,29 +40,45 @@ public class DefaultHoodieSplitProvider implements HoodieSplitProvider {
public static final int DEFAULT_SPLIT_QUEUE_SIZE = 20;
private Queue<HoodieSourceSplit> pendingSplits;
+ private HoodieSplitAssigner splitAssigner;
- public DefaultHoodieSplitProvider() {
+ public DefaultHoodieSplitProvider(HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new ConcurrentLinkedDeque<>();
+ this.splitAssigner = splitAssigner;
}
/**
* Creates a DefaultHoodieSplitProvider with a custom comparator for ordering splits.
*
- * @param comparator the comparator to use for ordering splits
+ * @param comparator the comparator to use for ordering splits (must not be null)
+ * @param splitAssigner the assigner that assigns each split to a task ID (must not be null)
*/
- public DefaultHoodieSplitProvider(SerializableComparator<HoodieSourceSplit> comparator) {
- ValidationUtils.checkArgument(comparator != null,
- "The hoodie source split comparator can't be null");
+ public DefaultHoodieSplitProvider(
+ SerializableComparator<HoodieSourceSplit> comparator,
+ HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new PriorityBlockingQueue<>(DEFAULT_SPLIT_QUEUE_SIZE, comparator);
+ this.splitAssigner = splitAssigner;
}
@Override
- public Option<HoodieSourceSplit> getNext(@Nullable String hostname) {
- if (!pendingSplits.isEmpty()) {
- return Option.of(pendingSplits.poll());
+ public Option<HoodieSourceSplit> getNext(int subTaskId, @Nullable String hostname) {
+ List<HoodieSourceSplit> visited = new ArrayList<>();
+ HoodieSourceSplit targetSplit = null;
+ while (!pendingSplits.isEmpty()) {
+ HoodieSourceSplit split = pendingSplits.poll();
+ int target = splitAssigner.assign(split);
+ if (subTaskId != target) {
+ visited.add(split);
+ } else {
+ targetSplit = split;
+ break;
+ }
}
- return Option.empty();
+ if (!visited.isEmpty()) {
+ pendingSplits.addAll(visited);
Review Comment:
Maybe we can use structure like `Map<Integer, Queue<HoodieSourceSplit>>` to keep mapping: subtask -> pending splits.
Splits are assigned during addition, and can be returned directly when requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3855943617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14527710cce409336ad47a7c7c0cbc80d8e15289 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686)
* 2b07ee0df16a4c56bab40bfcdb47c775c36d3758 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850510192
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6e07495142a88908c64304650574c739c144132f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673)
* 927e507f70906fb42135ca453c4a4414d28b8932 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3851238726
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8f6a404ce39753f66b7c99798f9bcc8e4f47da41 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766380446
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/DefaultHoodieSplitProvider.java:
##########
@@ -37,29 +40,48 @@ public class DefaultHoodieSplitProvider implements HoodieSplitProvider {
public static final int DEFAULT_SPLIT_QUEUE_SIZE = 20;
private Queue<HoodieSourceSplit> pendingSplits;
+ private HoodieSplitAssigner splitAssigner;
- public DefaultHoodieSplitProvider() {
+ public DefaultHoodieSplitProvider(HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new ConcurrentLinkedDeque<>();
+ this.splitAssigner = splitAssigner;
}
/**
* Creates a DefaultHoodieSplitProvider with a custom comparator for ordering splits.
*
- * @param comparator the comparator to use for ordering splits
+ * @param comparator the comparator to use for ordering splits (must not be null)
+ * @param splitAssigner the assigner that assigns each split to a task ID (must not be null)
+ * @throws IllegalArgumentException if comparator is null
*/
- public DefaultHoodieSplitProvider(SerializableComparator<HoodieSourceSplit> comparator) {
+ public DefaultHoodieSplitProvider(
+ SerializableComparator<HoodieSourceSplit> comparator,
+ HoodieSplitAssigner splitAssigner) {
ValidationUtils.checkArgument(comparator != null,
"The hoodie source split comparator can't be null");
Review Comment:
I think we can just skip the null check, by convention, args should not be null, and make `nullable` when needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850262707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 414cdb19c16c9b25308caf0908eea73f97780617 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663)
* 6e07495142a88908c64304650574c739c144132f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3851442501
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14527710cce409336ad47a7c7c0cbc80d8e15289 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2767401024
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
@danny0405 the basic distribution logic for `datastream.keyBy` is firstly assigning the key to a key group based on the max parallelism, then distribute the assigned key group to a sub task.
If we wanna keep the splits distribution behavior of new source on par with the legacy source, we should using the same partitioning strategy. Otherwise, we can simply use the method you mentioned above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3856371387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b07ee0df16a4c56bab40bfcdb47c775c36d3758 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3859237038
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
}, {
"hash" : "840e7eb77889632c2a93274c73d00252b012c26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753",
"triggerID" : "840e7eb77889632c2a93274c73d00252b012c26f",
"triggerType" : "PUSH"
}, {
"hash" : "c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11755",
"triggerID" : "c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11755)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766996385
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
Since we don't set `maxParallelism` for the source operator, it's inferred here: https://github.com/apache/flink/blob/e8ac5b12b4f7188b86cb3ecd9fc9ee5d9ccc3410/flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyGroupRangeAssignment.java#L137
numberOfChannels is the source parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850973516
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 927e507f70906fb42135ca453c4a4414d28b8932 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677)
* c6306dd367d2d76656cd092910f737ce6bf66743 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766957224
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
How do we determine maxParallelism, numberOfChannels?
Assume maxParallelism == source maxParallelism and numberOfChannels == current parallelism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3859214925
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
}, {
"hash" : "840e7eb77889632c2a93274c73d00252b012c26f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753",
"triggerID" : "840e7eb77889632c2a93274c73d00252b012c26f",
"triggerType" : "PUSH"
}, {
"hash" : "c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11755",
"triggerID" : "c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 840e7eb77889632c2a93274c73d00252b012c26f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753)
* c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11755)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766842844
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
keyBy will add an internal stream partitioner `KeyGroupStreamPartitioner`, we can reuse its distribution logic directly: `KeyGroupRangeAssignment.assignKeyToParallelOperator(...)`.
https://github.com/apache/flink/blob/e8ac5b12b4f7188b86cb3ecd9fc9ee5d9ccc3410/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java#L63
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766918433
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/DefaultHoodieSplitProvider.java:
##########
@@ -37,29 +40,45 @@ public class DefaultHoodieSplitProvider implements HoodieSplitProvider {
public static final int DEFAULT_SPLIT_QUEUE_SIZE = 20;
private Queue<HoodieSourceSplit> pendingSplits;
+ private HoodieSplitAssigner splitAssigner;
- public DefaultHoodieSplitProvider() {
+ public DefaultHoodieSplitProvider(HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new ConcurrentLinkedDeque<>();
+ this.splitAssigner = splitAssigner;
}
/**
* Creates a DefaultHoodieSplitProvider with a custom comparator for ordering splits.
*
- * @param comparator the comparator to use for ordering splits
+ * @param comparator the comparator to use for ordering splits (must not be null)
+ * @param splitAssigner the assigner that assigns each split to a task ID (must not be null)
*/
- public DefaultHoodieSplitProvider(SerializableComparator<HoodieSourceSplit> comparator) {
- ValidationUtils.checkArgument(comparator != null,
- "The hoodie source split comparator can't be null");
+ public DefaultHoodieSplitProvider(
+ SerializableComparator<HoodieSourceSplit> comparator,
+ HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new PriorityBlockingQueue<>(DEFAULT_SPLIT_QUEUE_SIZE, comparator);
+ this.splitAssigner = splitAssigner;
}
@Override
- public Option<HoodieSourceSplit> getNext(@Nullable String hostname) {
- if (!pendingSplits.isEmpty()) {
- return Option.of(pendingSplits.poll());
+ public Option<HoodieSourceSplit> getNext(int subTaskId, @Nullable String hostname) {
+ List<HoodieSourceSplit> visited = new ArrayList<>();
+ HoodieSourceSplit targetSplit = null;
+ while (!pendingSplits.isEmpty()) {
+ HoodieSourceSplit split = pendingSplits.poll();
+ int target = splitAssigner.assign(split);
+ if (subTaskId != target) {
+ visited.add(split);
+ } else {
+ targetSplit = split;
+ break;
+ }
}
- return Option.empty();
+ if (!visited.isEmpty()) {
+ pendingSplits.addAll(visited);
Review Comment:
Yes, That is the easiest way if we don't want to limit the pending splits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766557001
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/DefaultHoodieSplitProvider.java:
##########
Review Comment:
this api should return the same future, not creating new future every time, like using
```java
private final AtomicReference<CompletableFuture<Void>> availableFuture =
new AtomicReference<>(new CompletableFuture<>());
```
then `onDiscoveredSplits()` and `onUnassignedSplits()` should complete it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766566297
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/HoodieSourceSplit.java:
##########
@@ -36,7 +36,7 @@
*/
@Getter
public class HoodieSourceSplit implements SourceSplit, Serializable {
- public static AtomicInteger SPLIT_COUNTER = new AtomicInteger(0);
+ public static AtomicInteger SPLIT_COUNTER = new AtomicInteger(-1);
Review Comment:
let's call it split_id_gen then, split "counter" is for counting, so the var name can be misleading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3851297579
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8f6a404ce39753f66b7c99798f9bcc8e4f47da41 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683)
* 14527710cce409336ad47a7c7c0cbc80d8e15289 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766988177
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/DefaultHoodieSplitProvider.java:
##########
Review Comment:
Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3855958619
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14527710cce409336ad47a7c7c0cbc80d8e15289 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686)
* 2b07ee0df16a4c56bab40bfcdb47c775c36d3758 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766459961
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/HoodieSourceSplit.java:
##########
@@ -36,7 +36,7 @@
*/
@Getter
public class HoodieSourceSplit implements SourceSplit, Serializable {
- public static AtomicInteger SPLIT_COUNTER = new AtomicInteger(0);
+ public static AtomicInteger SPLIT_COUNTER = new AtomicInteger(-1);
Review Comment:
can you add a doc to explain why this counter starts with -1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766711374
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/enumerator/AbstractHoodieSplitEnumerator.java:
##########
Review Comment:
It will not, handleSplitRequest will be triggered periodically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766807000
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
btw, 1 should be the default split assigner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3858166853
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11686",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
}, {
"hash" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710",
"triggerID" : "2b07ee0df16a4c56bab40bfcdb47c775c36d3758",
"triggerType" : "PUSH"
}, {
"hash" : "840e7eb77889632c2a93274c73d00252b012c26f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753",
"triggerID" : "840e7eb77889632c2a93274c73d00252b012c26f",
"triggerType" : "PUSH"
}, {
"hash" : "c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2b07ee0df16a4c56bab40bfcdb47c775c36d3758 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11710)
* 840e7eb77889632c2a93274c73d00252b012c26f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11753)
* c5df5706e8cc487ed29e1695ca7cf6d0c9ca7f97 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3849872370
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bbd60b5060f1b0b384402e87751567e37aaffb13 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661)
* 414cdb19c16c9b25308caf0908eea73f97780617 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766712826
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/HoodieSource.java:
##########
@@ -126,16 +130,26 @@ private SplitEnumerator<HoodieSourceSplit, HoodieSplitEnumeratorState> createEnu
SplitEnumeratorContext<HoodieSourceSplit> enumContext,
@Nullable HoodieSplitEnumeratorState enumeratorState) {
HoodieSplitProvider splitProvider;
+ HoodieSplitAssigner splitAssigner;
+
+ if (OptionsResolver.isMorWithBucketIndexUpsert(scanContext.getConf())) {
+ splitAssigner = new HoodieSplitBucketAssigner(enumContext.currentParallelism());
+ } else {
+ splitAssigner = new DefaultHoodieSplitAssigner(enumContext.currentParallelism());
+ }
+
if (enumeratorState == null) {
- splitProvider = new DefaultHoodieSplitProvider(new HoodieSourceSplitComparator());
+ splitProvider = new DefaultHoodieSplitProvider(
+ new HoodieSourceSplitComparator(),
+ splitAssigner);
} else {
LOG.info(
"Hoodie source restored {} splits from state for table {}",
enumeratorState.getPendingSplitStates().size(),
metaClient.getTableConfig().getTableName());
List<HoodieSourceSplit> pendingSplits =
enumeratorState.getPendingSplitStates().stream().map(HoodieSourceSplitState::split).collect(Collectors.toList());
- splitProvider = new DefaultHoodieSplitProvider();
+ splitProvider = new DefaultHoodieSplitProvider(splitAssigner);
Review Comment:
Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3851288691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
}, {
"hash" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14527710cce409336ad47a7c7c0cbc80d8e15289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8f6a404ce39753f66b7c99798f9bcc8e4f47da41 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683)
* 14527710cce409336ad47a7c7c0cbc80d8e15289 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766525710
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/enumerator/AbstractHoodieSplitEnumerator.java:
##########
Review Comment:
this break here will stop assigning to the remaining readers if 1 reader does not have split to assign to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766825505
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
Do we need to achieve any feature parity with Flink keyby?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3851081601
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6306dd367d2d76656cd092910f737ce6bf66743 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681)
* 8f6a404ce39753f66b7c99798f9bcc8e4f47da41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2767570178
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
+1 to the feature parity for the convenience of user migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #18082:
URL: https://github.com/apache/hudi/pull/18082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3849567918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bbd60b5060f1b0b384402e87751567e37aaffb13 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850997229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 927e507f70906fb42135ca453c4a4414d28b8932 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677)
* c6306dd367d2d76656cd092910f737ce6bf66743 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681)
* 8f6a404ce39753f66b7c99798f9bcc8e4f47da41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3851132420
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
}, {
"hash" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681",
"triggerID" : "c6306dd367d2d76656cd092910f737ce6bf66743",
"triggerType" : "PUSH"
}, {
"hash" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683",
"triggerID" : "8f6a404ce39753f66b7c99798f9bcc8e4f47da41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6306dd367d2d76656cd092910f737ce6bf66743 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11681)
* 8f6a404ce39753f66b7c99798f9bcc8e4f47da41 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11683)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766478121
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/DefaultHoodieSplitProvider.java:
##########
@@ -37,29 +40,45 @@ public class DefaultHoodieSplitProvider implements HoodieSplitProvider {
public static final int DEFAULT_SPLIT_QUEUE_SIZE = 20;
private Queue<HoodieSourceSplit> pendingSplits;
+ private HoodieSplitAssigner splitAssigner;
- public DefaultHoodieSplitProvider() {
+ public DefaultHoodieSplitProvider(HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new ConcurrentLinkedDeque<>();
+ this.splitAssigner = splitAssigner;
}
/**
* Creates a DefaultHoodieSplitProvider with a custom comparator for ordering splits.
*
- * @param comparator the comparator to use for ordering splits
+ * @param comparator the comparator to use for ordering splits (must not be null)
+ * @param splitAssigner the assigner that assigns each split to a task ID (must not be null)
*/
- public DefaultHoodieSplitProvider(SerializableComparator<HoodieSourceSplit> comparator) {
- ValidationUtils.checkArgument(comparator != null,
- "The hoodie source split comparator can't be null");
+ public DefaultHoodieSplitProvider(
+ SerializableComparator<HoodieSourceSplit> comparator,
+ HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new PriorityBlockingQueue<>(DEFAULT_SPLIT_QUEUE_SIZE, comparator);
+ this.splitAssigner = splitAssigner;
}
@Override
- public Option<HoodieSourceSplit> getNext(@Nullable String hostname) {
- if (!pendingSplits.isEmpty()) {
- return Option.of(pendingSplits.poll());
+ public Option<HoodieSourceSplit> getNext(int subTaskId, @Nullable String hostname) {
+ List<HoodieSourceSplit> visited = new ArrayList<>();
+ HoodieSourceSplit targetSplit = null;
+ while (!pendingSplits.isEmpty()) {
+ HoodieSourceSplit split = pendingSplits.poll();
+ int target = splitAssigner.assign(split);
+ if (subTaskId != target) {
+ visited.add(split);
+ } else {
+ targetSplit = split;
+ break;
+ }
}
- return Option.empty();
+ if (!visited.isEmpty()) {
+ pendingSplits.addAll(visited);
Review Comment:
this impl makes getNext() O(n) as it needs to scan the queue until a match. should use a different data structure to handle per-subtask assignment, like multimap for e.g.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766469490
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/HoodieSourceSplit.java:
##########
@@ -36,7 +36,7 @@
*/
@Getter
public class HoodieSourceSplit implements SourceSplit, Serializable {
- public static AtomicInteger SPLIT_COUNTER = new AtomicInteger(0);
+ public static AtomicInteger SPLIT_COUNTER = new AtomicInteger(-1);
Review Comment:
Just to make the first split id starts from 0 which matches the first task id 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766497739
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/split/DefaultHoodieSplitProvider.java:
##########
@@ -37,29 +40,45 @@ public class DefaultHoodieSplitProvider implements HoodieSplitProvider {
public static final int DEFAULT_SPLIT_QUEUE_SIZE = 20;
private Queue<HoodieSourceSplit> pendingSplits;
+ private HoodieSplitAssigner splitAssigner;
- public DefaultHoodieSplitProvider() {
+ public DefaultHoodieSplitProvider(HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new ConcurrentLinkedDeque<>();
+ this.splitAssigner = splitAssigner;
}
/**
* Creates a DefaultHoodieSplitProvider with a custom comparator for ordering splits.
*
- * @param comparator the comparator to use for ordering splits
+ * @param comparator the comparator to use for ordering splits (must not be null)
+ * @param splitAssigner the assigner that assigns each split to a task ID (must not be null)
*/
- public DefaultHoodieSplitProvider(SerializableComparator<HoodieSourceSplit> comparator) {
- ValidationUtils.checkArgument(comparator != null,
- "The hoodie source split comparator can't be null");
+ public DefaultHoodieSplitProvider(
+ SerializableComparator<HoodieSourceSplit> comparator,
+ HoodieSplitAssigner splitAssigner) {
this.pendingSplits = new PriorityBlockingQueue<>(DEFAULT_SPLIT_QUEUE_SIZE, comparator);
+ this.splitAssigner = splitAssigner;
}
@Override
- public Option<HoodieSourceSplit> getNext(@Nullable String hostname) {
- if (!pendingSplits.isEmpty()) {
- return Option.of(pendingSplits.poll());
+ public Option<HoodieSourceSplit> getNext(int subTaskId, @Nullable String hostname) {
+ List<HoodieSourceSplit> visited = new ArrayList<>();
+ HoodieSourceSplit targetSplit = null;
+ while (!pendingSplits.isEmpty()) {
+ HoodieSourceSplit split = pendingSplits.poll();
+ int target = splitAssigner.assign(split);
+ if (subTaskId != target) {
+ visited.add(split);
+ } else {
+ targetSplit = split;
+ break;
+ }
}
- return Option.empty();
+ if (!visited.isEmpty()) {
+ pendingSplits.addAll(visited);
Review Comment:
Considered about it. Just feel pending splits are bounded. I also consider adding a config for defining the max pending splits. Which one do you prefer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2766922918
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/DefaultHoodieSplitAssigner.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+import java.io.Serializable;
+
+/**
+ * Default implementation of {@link HoodieSplitAssigner} that assigns Hoodie
+ * source splits to task IDs using round-robin distribution based on split number.
+ */
+public class DefaultHoodieSplitAssigner implements HoodieSplitAssigner, Serializable {
Review Comment:
It is true. As it runs in JM, no Serlization needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18082:
URL: https://github.com/apache/hudi/pull/18082#discussion_r2767229072
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/assign/HoodieSplitAssigner.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.source.assign;
+
+import org.apache.hudi.source.split.HoodieSourceSplit;
+
+/**
+ * Interface for assigning {@link HoodieSourceSplit} instances to task IDs.
+ */
+public interface HoodieSplitAssigner {
Review Comment:
should it be enough to calculate the murmur hash code of the file group ids and shuffle by the code:
```java
MathUtils.murmurHash(keyHash) % parallelism;
```
@cshuo did you know why the max parallelism is used for Flink `datastream.keyBy`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850029631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 414cdb19c16c9b25308caf0908eea73f97780617 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support flink split distribution strategy [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18082:
URL: https://github.com/apache/hudi/pull/18082#issuecomment-3850436809
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11661",
"triggerID" : "bbd60b5060f1b0b384402e87751567e37aaffb13",
"triggerType" : "PUSH"
}, {
"hash" : "414cdb19c16c9b25308caf0908eea73f97780617",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663",
"triggerID" : "414cdb19c16c9b25308caf0908eea73f97780617",
"triggerType" : "PUSH"
}, {
"hash" : "6e07495142a88908c64304650574c739c144132f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673",
"triggerID" : "6e07495142a88908c64304650574c739c144132f",
"triggerType" : "PUSH"
}, {
"hash" : "927e507f70906fb42135ca453c4a4414d28b8932",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677",
"triggerID" : "927e507f70906fb42135ca453c4a4414d28b8932",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 414cdb19c16c9b25308caf0908eea73f97780617 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11663)
* 6e07495142a88908c64304650574c739c144132f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11673)
* 927e507f70906fb42135ca453c4a4414d28b8932 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org