You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2025/04/17 10:49:10 UTC
[PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
cshuo opened a new pull request, #13163:
URL: https://github.com/apache/hudi/pull/13163
… writers
### Change Logs
Fix inconsistency of ordering value type between reader & writers
### Impact
none
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13163:
URL: https://github.com/apache/hudi/pull/13163#discussion_r2049809598
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/ConsistentBucketStreamWriteFunctionWrapper.java:
##########
@@ -58,7 +58,9 @@ public void invoke(I record) throws Exception {
HoodieFlinkInternalRow hoodieRecord = toHoodieFunction.map((RowData) record);
ScalaCollector<HoodieFlinkInternalRow> collector = ScalaCollector.getInstance();
assignFunction.processElement(hoodieRecord, null, collector);
- writeFunction.processElement(collector.getVal(), null, null);
+ for (HoodieFlinkInternalRow row: collector.getVal()) {
Review Comment:
`collector` is used to collect emitted records from `asisgnFunction`, it's possible `asisgnFunction` emits more than one records for one time processElement, e.g., for case global inedx, it may emits 1 DELETE + 1 INSERT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13163:
URL: https://github.com/apache/hudi/pull/13163#issuecomment-2812493877
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924",
"triggerID" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63aed2dc9fed8ec830144a4e3fc32fc9d52b2543 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13163:
URL: https://github.com/apache/hudi/pull/13163#issuecomment-2812658238
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924",
"triggerID" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"triggerType" : "PUSH"
}, {
"hash" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925",
"triggerID" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63aed2dc9fed8ec830144a4e3fc32fc9d52b2543 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924)
* 4aaa9ccc9b29caa3148abdc1f3b84719a2b61942 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13163:
URL: https://github.com/apache/hudi/pull/13163#issuecomment-2812490698
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63aed2dc9fed8ec830144a4e3fc32fc9d52b2543 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13163:
URL: https://github.com/apache/hudi/pull/13163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13163:
URL: https://github.com/apache/hudi/pull/13163#discussion_r2049806028
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/ScalaCollector.java:
##########
@@ -20,27 +20,35 @@
import org.apache.flink.util.Collector;
+import java.util.ArrayList;
+import java.util.List;
+
/**
* A mock {@link Collector} that used in {@link TestFunctionWrapper}.
*/
class ScalaCollector<T> implements Collector<T> {
Review Comment:
The class name implies that the result should be singleton.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13163:
URL: https://github.com/apache/hudi/pull/13163#discussion_r2049805495
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/ConsistentBucketStreamWriteFunctionWrapper.java:
##########
@@ -58,7 +58,9 @@ public void invoke(I record) throws Exception {
HoodieFlinkInternalRow hoodieRecord = toHoodieFunction.map((RowData) record);
ScalaCollector<HoodieFlinkInternalRow> collector = ScalaCollector.getInstance();
assignFunction.processElement(hoodieRecord, null, collector);
- writeFunction.processElement(collector.getVal(), null, null);
+ for (HoodieFlinkInternalRow row: collector.getVal()) {
Review Comment:
why change this line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13163:
URL: https://github.com/apache/hudi/pull/13163#discussion_r2049810358
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/ScalaCollector.java:
##########
@@ -20,27 +20,35 @@
import org.apache.flink.util.Collector;
+import java.util.ArrayList;
+import java.util.List;
+
/**
* A mock {@link Collector} that used in {@link TestFunctionWrapper}.
*/
class ScalaCollector<T> implements Collector<T> {
Review Comment:
`Scalar`? ok, then we should rename it if multiple emitted records is expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13163:
URL: https://github.com/apache/hudi/pull/13163#issuecomment-2814311500
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924",
"triggerID" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"triggerType" : "PUSH"
}, {
"hash" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925",
"triggerID" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"triggerType" : "PUSH"
}, {
"hash" : "637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4940",
"triggerID" : "637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4aaa9ccc9b29caa3148abdc1f3b84719a2b61942 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925)
* 637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4940)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13163:
URL: https://github.com/apache/hudi/pull/13163#issuecomment-2814308353
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924",
"triggerID" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"triggerType" : "PUSH"
}, {
"hash" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925",
"triggerID" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"triggerType" : "PUSH"
}, {
"hash" : "637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4aaa9ccc9b29caa3148abdc1f3b84719a2b61942 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925)
* 637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13163:
URL: https://github.com/apache/hudi/pull/13163#issuecomment-2812654768
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924",
"triggerID" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"triggerType" : "PUSH"
}, {
"hash" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63aed2dc9fed8ec830144a4e3fc32fc9d52b2543 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924)
* 4aaa9ccc9b29caa3148abdc1f3b84719a2b61942 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13163:
URL: https://github.com/apache/hudi/pull/13163#issuecomment-2812838829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924",
"triggerID" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"triggerType" : "PUSH"
}, {
"hash" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925",
"triggerID" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4aaa9ccc9b29caa3148abdc1f3b84719a2b61942 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9319] Fix inconsistency of ordering value type between reader &… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13163:
URL: https://github.com/apache/hudi/pull/13163#issuecomment-2814408799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4924",
"triggerID" : "63aed2dc9fed8ec830144a4e3fc32fc9d52b2543",
"triggerType" : "PUSH"
}, {
"hash" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4925",
"triggerID" : "4aaa9ccc9b29caa3148abdc1f3b84719a2b61942",
"triggerType" : "PUSH"
}, {
"hash" : "637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4940",
"triggerID" : "637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 637dd78ba9b732e10ecec44fb172e3cfa8bc9f5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4940)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org