You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/01/20 01:06:22 UTC
[PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
voonhous opened a new pull request, #17953:
URL: https://github.com/apache/hudi/pull/17953
### Describe the issue this Pull Request addresses
Fixes: https://github.com/apache/hudi/issues/17938
For more details, please check the issue above.
The full E2E test should be covered in `org.apache.hudi.integ.ITTestHoodieDemo#testParquetDemo`, specifically: `org.apache.hudi.integ.ITTestHoodieDemo#ingestSecondBatchAndHiveSync`, which is currently disabled.
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
Ensure that `StringProcessor` in `MercifulJsonConverter` returns `org.apache.avro.util.Utf8` instead of `java.lang.String`.
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
Ordering value comparison will now pass without ClassCastExceptions.
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
Medium, this might affect other Dataflows. Will check if CI succeeds.
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
None
### Contributor's checklist
- [X] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [X] Enough context is provided in the sections above
- [X] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3770809355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3775921648
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
}, {
"hash" : "101565697c864068e65af6e3351c7f83680f9036",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248",
"triggerID" : "101565697c864068e65af6e3351c7f83680f9036",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* 6c68151439f2742ae7adf5a74283bcb2712bfbf4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239)
* 101565697c864068e65af6e3351c7f83680f9036 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3770602477
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d3ced28564cd27af238f7666e6e2dd0d3cf8368 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3773923576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* f8038c7037656cc40e9d2f50d45b3e46ad0211b5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234)
* f9d1932bae7cccd329d2d0b8810a2f0940156dc3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3774671137
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* 6c68151439f2742ae7adf5a74283bcb2712bfbf4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17953:
URL: https://github.com/apache/hudi/pull/17953#discussion_r2711072298
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -249,7 +249,7 @@ public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecor
Comparable oldOrderingValue = olderRecord.getOrderingValue();
HoodieSchema newSchema = recordContext.getSchemaFromBufferRecord(newerRecord);
if (!olderRecord.isCommitTimeOrderingDelete()
- && oldOrderingValue.compareTo(newOrderingValue) > 0) {
+ && OrderingValues.compare(oldOrderingValue, newOrderingValue) > 0) {
Review Comment:
I have updated the issue, it'll be good if you can read through my detailed analysis of:
1. **WHAT** is the ROOT CAUSE of the issue
2. **HOW** to trigger the issue and i have even included 2 test cases, 1 summarised, and another with an E2E flow
- I have also added within this test 16 test cases, all of which covers _tableType{COW,MOR}, transformer{true,false}, HoodieRecordType{SPARK,AVRO}_
- Using this, we can see how the types differs with each and every of the use cases/entrypoints
3. **WHERE** and **WHEN**, i.e. the code path that follows which will trigger the issue
While i do agree that what we have looks hacky, it'll be extremely helpful if you can at least take a look at my analysis and help to provide insights as to why:
1. HoodieRecordType.AVRO with Json sources are generating Strings for the problematic flow
2. If Strings are intended, then the older records should be read out in String type too, instead of `Avro.UTF8`
3. If Strings are not intended, then we should convert them to `Avro.UTF8`, but this will cause other separate issues when i tried doing that.
As can be seen, I have tried:
1. Fixing the issue at source, i.e. Since Side_A is not in the correct format, i tried changing that side to what is expected to make things work, but this did not work
2. Fixing the issue closer to the source, i.e. I Change the format of Side_B, and that did not work too as i realised that will cause too many cascading changes
3. When fixing the issue at source Side_A and Side_B did not work, i resorted to normalizing, which I felt is the correct choice at this point in time.
As mentioned, i do agree that this is hacky, but i feel what i am doing is justified FOR NOW. Until someone provides better information AFTER reading my analysis and point out what i have missed, my fix will stand for now until i have more time to revisit this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3773273782
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225)
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* f8038c7037656cc40e9d2f50d45b3e46ad0211b5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3773936934
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* f8038c7037656cc40e9d2f50d45b3e46ad0211b5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234)
* f9d1932bae7cccd329d2d0b8810a2f0940156dc3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3776082430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
}, {
"hash" : "101565697c864068e65af6e3351c7f83680f9036",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248",
"triggerID" : "101565697c864068e65af6e3351c7f83680f9036",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* 101565697c864068e65af6e3351c7f83680f9036 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3770606874
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d3ced28564cd27af238f7666e6e2dd0d3cf8368 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3773259535
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225)
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* f8038c7037656cc40e9d2f50d45b3e46ad0211b5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Allow String ordering fields can work with JSON src with COW [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17953:
URL: https://github.com/apache/hudi/pull/17953#discussion_r2711459373
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -249,7 +249,7 @@ public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecor
Comparable oldOrderingValue = olderRecord.getOrderingValue();
HoodieSchema newSchema = recordContext.getSchemaFromBufferRecord(newerRecord);
if (!olderRecord.isCommitTimeOrderingDelete()
- && oldOrderingValue.compareTo(newOrderingValue) > 0) {
+ && OrderingValues.compare(oldOrderingValue, newOrderingValue) > 0) {
Review Comment:
> TLDR of the root cause: MercifulJsonConverter.StringProcessor returns java.lang.String directly instead of wrapping it in Utf8. This is inconsistent with Avro's standard behavior where strings should be Utf8 by default.
Fixing the `MercifulJsonConverter.StringProcessor` to return avro `Utf8` looks like the right fix, for Flink and Spark engine row to avro generic record transformation, they all return `Uft8` string though.
Can we fix the test `TestJsonKafkaSource.testJsonKafkaSourceWithEncodedDecimals` for the row projection, I guess there is a gap between the avro record to InternalRow transformation. The other 2 UT failues looks like a easy fix.
In any case, let's not add the special handling(type conversion) in the comparison, it's costly also error prone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3770709130
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d3ced28564cd27af238f7666e6e2dd0d3cf8368 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223)
* 4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3770625829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d3ced28564cd27af238f7666e6e2dd0d3cf8368 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223)
* 4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17953:
URL: https://github.com/apache/hudi/pull/17953#discussion_r2706629513
##########
hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java:
##########
@@ -444,10 +445,10 @@ private static class StringProcessor extends JsonFieldProcessor {
@Override
public Pair<Boolean, Object> convert(Object value, String name, HoodieSchema schema) {
if (value instanceof String) {
- return Pair.of(true, value);
+ return Pair.of(true, new Utf8((String) value));
Review Comment:
is the converter only used for avro record type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Allow String ordering fields can work with JSON src with COW [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17953:
URL: https://github.com/apache/hudi/pull/17953#discussion_r2712515705
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -249,7 +249,7 @@ public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecor
Comparable oldOrderingValue = olderRecord.getOrderingValue();
HoodieSchema newSchema = recordContext.getSchemaFromBufferRecord(newerRecord);
if (!olderRecord.isCommitTimeOrderingDelete()
- && oldOrderingValue.compareTo(newOrderingValue) > 0) {
+ && OrderingValues.compare(oldOrderingValue, newOrderingValue) > 0) {
Review Comment:
Ignore the above exception in https://github.com/apache/hudi/issues/17938#issuecomment-3776309973.
The issue was because `MercifulJsonToRowConverter extends MercifulJsonConverter`. It inherited the `Utf8` conversion causing other paths to throw the error we see.
Overriding it again in `MercifulJsonToRowConverter` to produce the correct String type worked again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Allow String ordering fields can work with JSON src with COW [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3778487251
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
}, {
"hash" : "101565697c864068e65af6e3351c7f83680f9036",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248",
"triggerID" : "101565697c864068e65af6e3351c7f83680f9036",
"triggerType" : "PUSH"
}, {
"hash" : "41f4112b3a88e3bec16a1712d6366a7bb0a96004",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11258",
"triggerID" : "41f4112b3a88e3bec16a1712d6366a7bb0a96004",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* 41f4112b3a88e3bec16a1712d6366a7bb0a96004 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11258)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Allow String ordering fields can work with JSON src with COW [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #17953:
URL: https://github.com/apache/hudi/pull/17953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3774186430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* f9d1932bae7cccd329d2d0b8810a2f0940156dc3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3773245781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225)
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17953:
URL: https://github.com/apache/hudi/pull/17953#discussion_r2708677496
##########
hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java:
##########
@@ -444,10 +445,10 @@ private static class StringProcessor extends JsonFieldProcessor {
@Override
public Pair<Boolean, Object> convert(Object value, String name, HoodieSchema schema) {
if (value instanceof String) {
- return Pair.of(true, value);
+ return Pair.of(true, new Utf8((String) value));
Review Comment:
> is the converter only used for avro record type?
Looks to be the case now IIUC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3774370142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* f9d1932bae7cccd329d2d0b8810a2f0940156dc3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237)
* 6c68151439f2742ae7adf5a74283bcb2712bfbf4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3773521734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* f8038c7037656cc40e9d2f50d45b3e46ad0211b5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3774458808
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* f9d1932bae7cccd329d2d0b8810a2f0940156dc3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237)
* 6c68151439f2742ae7adf5a74283bcb2712bfbf4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3775885154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
}, {
"hash" : "101565697c864068e65af6e3351c7f83680f9036",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "101565697c864068e65af6e3351c7f83680f9036",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* 6c68151439f2742ae7adf5a74283bcb2712bfbf4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239)
* 101565697c864068e65af6e3351c7f83680f9036 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17953:
URL: https://github.com/apache/hudi/pull/17953#discussion_r2710950092
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -249,7 +249,7 @@ public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecor
Comparable oldOrderingValue = olderRecord.getOrderingValue();
HoodieSchema newSchema = recordContext.getSchemaFromBufferRecord(newerRecord);
if (!olderRecord.isCommitTimeOrderingDelete()
- && oldOrderingValue.compareTo(newOrderingValue) > 0) {
+ && OrderingValues.compare(oldOrderingValue, newOrderingValue) > 0) {
Review Comment:
we should ensure the ordring value is generated with engine type instead of hack the conversion in comparison, if avro record type is used here, let's figure out why a `String` ordering value is generated for spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Allow String ordering fields can work with JSON src with COW [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3778103670
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
}, {
"hash" : "101565697c864068e65af6e3351c7f83680f9036",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248",
"triggerID" : "101565697c864068e65af6e3351c7f83680f9036",
"triggerType" : "PUSH"
}, {
"hash" : "41f4112b3a88e3bec16a1712d6366a7bb0a96004",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11258",
"triggerID" : "41f4112b3a88e3bec16a1712d6366a7bb0a96004",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* 101565697c864068e65af6e3351c7f83680f9036 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248)
* 41f4112b3a88e3bec16a1712d6366a7bb0a96004 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11258)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Allow String ordering fields can work with JSON src with COW [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3778090701
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11225",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
}, {
"hash" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af1e3eb4c00e1fdcab71655b1605e4c516805910",
"triggerType" : "PUSH"
}, {
"hash" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11234",
"triggerID" : "f8038c7037656cc40e9d2f50d45b3e46ad0211b5",
"triggerType" : "PUSH"
}, {
"hash" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11237",
"triggerID" : "f9d1932bae7cccd329d2d0b8810a2f0940156dc3",
"triggerType" : "PUSH"
}, {
"hash" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11239",
"triggerID" : "6c68151439f2742ae7adf5a74283bcb2712bfbf4",
"triggerType" : "PUSH"
}, {
"hash" : "101565697c864068e65af6e3351c7f83680f9036",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248",
"triggerID" : "101565697c864068e65af6e3351c7f83680f9036",
"triggerType" : "PUSH"
}, {
"hash" : "41f4112b3a88e3bec16a1712d6366a7bb0a96004",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "41f4112b3a88e3bec16a1712d6366a7bb0a96004",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af1e3eb4c00e1fdcab71655b1605e4c516805910 UNKNOWN
* 101565697c864068e65af6e3351c7f83680f9036 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11248)
* 41f4112b3a88e3bec16a1712d6366a7bb0a96004 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17953:
URL: https://github.com/apache/hudi/pull/17953#discussion_r2706706618
##########
hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java:
##########
@@ -444,10 +445,10 @@ private static class StringProcessor extends JsonFieldProcessor {
@Override
public Pair<Boolean, Object> convert(Object value, String name, HoodieSchema schema) {
if (value instanceof String) {
- return Pair.of(true, value);
+ return Pair.of(true, new Utf8((String) value));
Review Comment:
Not sure, gonna trigger the CI to see where the tests are failing.
From what I can see via tracing, it's used in any JSON source that has `HoodieRecordType.AVRO`.
If it's SPARK, it will use RowConverter utils.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17953:
URL: https://github.com/apache/hudi/pull/17953#discussion_r2706707984
##########
hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java:
##########
@@ -444,10 +445,10 @@ private static class StringProcessor extends JsonFieldProcessor {
@Override
public Pair<Boolean, Object> convert(Object value, String name, HoodieSchema schema) {
if (value instanceof String) {
- return Pair.of(true, value);
+ return Pair.of(true, new Utf8((String) value));
Review Comment:
Okay, checking Azure, the tests below are failing:
```
TestHoodieDeltaStreamer.testLogicalTypesWithJsonSource:829->HoodieDeltaStreamerTestBase.syncOnce:846
```
Will investigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Ensure that MercifulJsonConverter returns Avro Utf8 string [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17953:
URL: https://github.com/apache/hudi/pull/17953#issuecomment-3770690253
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223",
"triggerID" : "8d3ced28564cd27af238f7666e6e2dd0d3cf8368",
"triggerType" : "PUSH"
}, {
"hash" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d3ced28564cd27af238f7666e6e2dd0d3cf8368 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11223)
* 4cd3bc395a37e8fb1de3cb0aed0f674ecfdd7f82 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org