You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2025/10/26 09:32:46 UTC
[PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
lokeshj1703 opened a new pull request, #14159:
URL: https://github.com/apache/hudi/pull/14159
### Describe the issue this Pull Request addresses
https://github.com/apache/hudi/issues/14158
### Summary and Changelog
Currently MySqlDebeziumAvroPayload is not handled with deltastreamer. We are overwriting cfg.payloadClassName based on inferred merge configs. The payloadClassName is required for upgrade handling with EightToNineUpgradeHandler to table version 9.
Also currently in EightToNineUpgradeHandler we only check for payload class in HoodieTableConfig. For some older tables with COW, the payload class is not available in the table config. In such cases we should also check for payload class provided in write config as well.
### Impact
Fixes upgrade with MySqlDebeziumAvroPayload to table version 9
### Risk Level
medium. Tested locally
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3450070084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5cec299dfab38cfdd83290146e7ed0647f74c059 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337)
* 0212e66c850bcce551fd24e904af558e427bc420 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3453640615
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
}, {
"hash" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361",
"triggerID" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"triggerType" : "PUSH"
}, {
"hash" : "2693ed18d86139f8694ebef9ff878ece2b6ec7e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2693ed18d86139f8694ebef9ff878ece2b6ec7e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22eeda6a2ebb10efac2b53022241efc4a9a28419 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361)
* 2693ed18d86139f8694ebef9ff878ece2b6ec7e0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3448266170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5cec299dfab38cfdd83290146e7ed0647f74c059 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3450307207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0212e66c850bcce551fd24e904af558e427bc420 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14159:
URL: https://github.com/apache/hudi/pull/14159#discussion_r2466187173
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -1145,6 +1149,16 @@ private Pair<HoodieWriteConfig, Schema> getHoodieClientConfigAndWriterSchema(Sch
if (!StringUtils.isNullOrEmpty(cfg.recordMergeStrategyId)) {
builder.withRecordMergeStrategyId(cfg.recordMergeStrategyId);
}
+
+ if (metaClient != null) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (tableConfig.getLegacyPayloadClass().equals(MySqlDebeziumAvroPayload.class.getCanonicalName())
+ && cfg.sourceOrderingFields.equals(DebeziumConstants.ADDED_SEQ_COL_NAME)) {
Review Comment:
lets add java docs as to why we are doing this
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -2261,6 +2262,33 @@ public void testDeltaStreamerWithMultipleOrderingFields(HoodieTableType tableTyp
UtilitiesTestBase.Helpers.deleteFileFromDfs(fs, tableBasePath);
}
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testDeltaStreamerFailureWithChangingOrderingFields(HoodieTableType tableType) throws Exception {
Review Comment:
is it possible to add tests for MysqlDebeziumPayload to test the happy path? i.e. when ordering field switches in table config during upgrade, but user is still using same old source ordering field?
##########
hudi-common/src/main/java/org/apache/hudi/common/model/debezium/MySqlDebeziumAvroPayload.java:
##########
@@ -108,4 +113,8 @@ static boolean isCurrentSeqLatest(String currentSeq, String newSeq) {
long newPos = Long.parseLong(newFilePos[1]);
return newPos < currentPos;
}
+
+ public static String getOrderingFields() {
Review Comment:
not sure why do we need this public method.
we can make : 54 public and let callers directly access public static variable.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -1145,6 +1149,16 @@ private Pair<HoodieWriteConfig, Schema> getHoodieClientConfigAndWriterSchema(Sch
if (!StringUtils.isNullOrEmpty(cfg.recordMergeStrategyId)) {
builder.withRecordMergeStrategyId(cfg.recordMergeStrategyId);
}
+
+ if (metaClient != null) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (tableConfig.getLegacyPayloadClass().equals(MySqlDebeziumAvroPayload.class.getCanonicalName())
Review Comment:
can we check table version as well here.
I assume `tableConfig.getLegacyPayloadClass()` will only be present in table v9, but having explicit checks would be better.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToNineUpgradeHandler.java:
##########
@@ -224,13 +224,13 @@ private void reconcileMergePropertiesConfig(Map<ConfigProperty, String> tablePro
}
}
- private void reconcileOrderingFieldsConfig(Map<ConfigProperty, String> tablePropsToAdd,
- Set<ConfigProperty> tablePropsToRemove,
- HoodieTableConfig tableConfig) {
- String payloadClass = tableConfig.getPayloadClass();
+ private void reconcileOrderingFieldsConfig(Map<ConfigProperty, String> tablePropsToAdd, Set<ConfigProperty> tablePropsToRemove,
+ HoodieTableConfig tableConfig, HoodieWriteConfig config) {
+ String payloadClass = tableConfig.getPayloadClassIfPresent()
Review Comment:
do we have tests for this case?
i.e. payload class missing from tableConfig, but we fetch it from incoming writer props in EightToNinceUpgradeHandler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3448267874
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5cec299dfab38cfdd83290146e7ed0647f74c059 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3448306505
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5cec299dfab38cfdd83290146e7ed0647f74c059 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3451205444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #14159:
URL: https://github.com/apache/hudi/pull/14159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3453810649
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
}, {
"hash" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361",
"triggerID" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"triggerType" : "PUSH"
}, {
"hash" : "2693ed18d86139f8694ebef9ff878ece2b6ec7e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9367",
"triggerID" : "2693ed18d86139f8694ebef9ff878ece2b6ec7e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2693ed18d86139f8694ebef9ff878ece2b6ec7e0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9367)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3453643932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
}, {
"hash" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361",
"triggerID" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"triggerType" : "PUSH"
}, {
"hash" : "2693ed18d86139f8694ebef9ff878ece2b6ec7e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9367",
"triggerID" : "2693ed18d86139f8694ebef9ff878ece2b6ec7e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22eeda6a2ebb10efac2b53022241efc4a9a28419 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361)
* 2693ed18d86139f8694ebef9ff878ece2b6ec7e0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9367)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3452740736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
}, {
"hash" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 10579a83aa781987c21a4f450085687fb0e0e2eb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355)
* 22eeda6a2ebb10efac2b53022241efc4a9a28419 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3453126556
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
}, {
"hash" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361",
"triggerID" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22eeda6a2ebb10efac2b53022241efc4a9a28419 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3450941822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0212e66c850bcce551fd24e904af558e427bc420 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352)
* 49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3451267731
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354)
* 10579a83aa781987c21a4f450085687fb0e0e2eb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #14159:
URL: https://github.com/apache/hudi/pull/14159#discussion_r2466658875
##########
hudi-common/src/main/java/org/apache/hudi/common/model/debezium/MySqlDebeziumAvroPayload.java:
##########
@@ -108,4 +113,8 @@ static boolean isCurrentSeqLatest(String currentSeq, String newSeq) {
long newPos = Long.parseLong(newFilePos[1]);
return newPos < currentPos;
}
+
+ public static String getOrderingFields() {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14159:
URL: https://github.com/apache/hudi/pull/14159#discussion_r2467232378
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -1145,6 +1149,19 @@ private Pair<HoodieWriteConfig, Schema> getHoodieClientConfigAndWriterSchema(Sch
if (!StringUtils.isNullOrEmpty(cfg.recordMergeStrategyId)) {
builder.withRecordMergeStrategyId(cfg.recordMergeStrategyId);
}
+
+ if (metaClient != null) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ // After upgrade to table version 9 with MySqlDebeziumAvroPayload, ordering fields are changed from
+ // `_event_seq` to `_event_bin_file,_event_pos`. The logic here ensures that deltastreamer config is updated
+ // if it points to older ordering field `_event_seq`.
+ if (tableConfig.getTableVersion().equals(HoodieTableVersion.NINE) && tableConfig.getLegacyPayloadClass().equals(MySqlDebeziumAvroPayload.class.getCanonicalName())
Review Comment:
greater than or equal to Nine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3454157192
<img width="1224" height="561" alt="image" src="https://github.com/user-attachments/assets/947862e9-cc89-401f-935c-9a2b6bf22e4c" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3451706647
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 10579a83aa781987c21a4f450085687fb0e0e2eb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3450074926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5cec299dfab38cfdd83290146e7ed0647f74c059 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337)
* 0212e66c850bcce551fd24e904af558e427bc420 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3450950463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0212e66c850bcce551fd24e904af558e427bc420 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352)
* 49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14159:
URL: https://github.com/apache/hudi/pull/14159#discussion_r2463855224
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -88,9 +89,10 @@ public class HoodieStreamerUtils {
*/
public static Option<JavaRDD<HoodieRecord>> createHoodieRecords(HoodieStreamer.Config cfg, TypedProperties props, Option<JavaRDD<GenericRecord>> avroRDDOptional,
SchemaProvider schemaProvider, HoodieRecord.HoodieRecordType recordType, boolean autoGenerateRecordKeys,
- String instantTime, Option<BaseErrorTableWriter> errorTableWriter) {
+ String instantTime, Option<BaseErrorTableWriter> errorTableWriter, HoodieTableConfig tableConfig) {
boolean shouldCombine = cfg.filterDupes || cfg.operation.equals(WriteOperationType.UPSERT);
- boolean shouldUseOrderingField = shouldCombine && !StringUtils.isNullOrEmpty(cfg.sourceOrderingFields);
+ String orderingFieldsStr = tableConfig.getOrderingFieldsStr().orElse(cfg.sourceOrderingFields);
Review Comment:
and we need tests for this.
for other payloads, if source ordering field is changed in subsequent commits, writes should fail.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToNineUpgradeHandler.java:
##########
@@ -127,22 +127,22 @@ && isComplexKeyGeneratorWithSingleRecordKeyField(tableConfig)) {
metaClient.getTableConfig().getTableVersion());
}
// Handle merge mode config.
- reconcileMergeModeConfig(tablePropsToAdd, tablePropsToRemove, tableConfig);
+ reconcileMergeModeConfig(tablePropsToAdd, tablePropsToRemove, tableConfig, config);
// Handle partial update mode config.
- reconcilePartialUpdateModeConfig(tablePropsToAdd, tableConfig);
+ reconcilePartialUpdateModeConfig(tablePropsToAdd, tableConfig, config);
// Handle merge properties config.
reconcileMergePropertiesConfig(tablePropsToAdd, tableConfig, config);
// Handle payload class configs.
- reconcilePayloadClassConfig(tablePropsToAdd, tablePropsToRemove, tableConfig);
+ reconcilePayloadClassConfig(tablePropsToAdd, tablePropsToRemove, tableConfig, config);
// Handle ordering fields config.
- reconcileOrderingFieldsConfig(tablePropsToAdd, tablePropsToRemove, tableConfig);
+ reconcileOrderingFieldsConfig(tablePropsToAdd, tablePropsToRemove, tableConfig, config);
return new UpgradeDowngrade.TableConfigChangeSet(tablePropsToAdd, tablePropsToRemove);
}
- private void reconcileMergeModeConfig(Map<ConfigProperty, String> tablePropsToAdd,
- Set<ConfigProperty> tablePropsToRemove,
- HoodieTableConfig tableConfig) {
- String payloadClass = tableConfig.getPayloadClass();
+ private void reconcileMergeModeConfig(Map<ConfigProperty, String> tablePropsToAdd, Set<ConfigProperty> tablePropsToRemove,
+ HoodieTableConfig tableConfig, HoodieWriteConfig config) {
Review Comment:
were you able to find the actual code block why we don't serialize payload class to hoodie.properties w/ HoodieStreamer for COW table ? @lokeshj1703
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -207,6 +195,13 @@ static Option<String> getPayloadClassNameIfPresent(Properties props) {
payloadClassName = ConfigUtils.getStringWithAltKeys(props, PAYLOAD_CLASS_NAME);
} else if (props.containsKey("hoodie.datasource.write.payload.class")) {
payloadClassName = props.getProperty("hoodie.datasource.write.payload.class");
+ } else if (ConfigUtils.containsConfigProperty(props, RECORD_MERGE_MODE)
+ && ConfigUtils.getStringWithAltKeys(props, RECORD_MERGE_MODE, StringUtils.EMPTY_STRING)
Review Comment:
instead, why not leverage `HoodieTableConig.propsMap()` within EightToNineUpgradeHandler to check if payload class is set or not within HoodieTableConfig and we don't need any of these changes.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -207,6 +195,13 @@ static Option<String> getPayloadClassNameIfPresent(Properties props) {
payloadClassName = ConfigUtils.getStringWithAltKeys(props, PAYLOAD_CLASS_NAME);
} else if (props.containsKey("hoodie.datasource.write.payload.class")) {
payloadClassName = props.getProperty("hoodie.datasource.write.payload.class");
+ } else if (ConfigUtils.containsConfigProperty(props, RECORD_MERGE_MODE)
+ && ConfigUtils.getStringWithAltKeys(props, RECORD_MERGE_MODE, StringUtils.EMPTY_STRING)
Review Comment:
can you help me understand, why do we need this refactoring ?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -88,9 +89,10 @@ public class HoodieStreamerUtils {
*/
public static Option<JavaRDD<HoodieRecord>> createHoodieRecords(HoodieStreamer.Config cfg, TypedProperties props, Option<JavaRDD<GenericRecord>> avroRDDOptional,
SchemaProvider schemaProvider, HoodieRecord.HoodieRecordType recordType, boolean autoGenerateRecordKeys,
- String instantTime, Option<BaseErrorTableWriter> errorTableWriter) {
+ String instantTime, Option<BaseErrorTableWriter> errorTableWriter, HoodieTableConfig tableConfig) {
boolean shouldCombine = cfg.filterDupes || cfg.operation.equals(WriteOperationType.UPSERT);
- boolean shouldUseOrderingField = shouldCombine && !StringUtils.isNullOrEmpty(cfg.sourceOrderingFields);
+ String orderingFieldsStr = tableConfig.getOrderingFieldsStr().orElse(cfg.sourceOrderingFields);
Review Comment:
we don't want to make this default and siltently move on.
lets re-use table config only for mysqldebezium payload and add a warn log if it differs from `sourceOrderingField` set in top level config.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -207,6 +195,13 @@ static Option<String> getPayloadClassNameIfPresent(Properties props) {
payloadClassName = ConfigUtils.getStringWithAltKeys(props, PAYLOAD_CLASS_NAME);
} else if (props.containsKey("hoodie.datasource.write.payload.class")) {
payloadClassName = props.getProperty("hoodie.datasource.write.payload.class");
+ } else if (ConfigUtils.containsConfigProperty(props, RECORD_MERGE_MODE)
+ && ConfigUtils.getStringWithAltKeys(props, RECORD_MERGE_MODE, StringUtils.EMPTY_STRING)
Review Comment:
we really need to know if payload class is set in TableConfig or not mainly for EightToNineUpgradeHandler right.
we may need it for other callers. can we ensure we try not to change other callers unintentionally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #14159:
URL: https://github.com/apache/hudi/pull/14159#discussion_r2464547996
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -207,6 +195,13 @@ static Option<String> getPayloadClassNameIfPresent(Properties props) {
payloadClassName = ConfigUtils.getStringWithAltKeys(props, PAYLOAD_CLASS_NAME);
} else if (props.containsKey("hoodie.datasource.write.payload.class")) {
payloadClassName = props.getProperty("hoodie.datasource.write.payload.class");
+ } else if (ConfigUtils.containsConfigProperty(props, RECORD_MERGE_MODE)
+ && ConfigUtils.getStringWithAltKeys(props, RECORD_MERGE_MODE, StringUtils.EMPTY_STRING)
Review Comment:
Its a simple refactoring. The changes are made to function `getPayloadClassNameIfPresent` which is only used by getPayloadClassName in HoodieRecordPayload and the functionality added in upgrade handler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3451260749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354)
* 10579a83aa781987c21a4f450085687fb0e0e2eb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #14159:
URL: https://github.com/apache/hudi/pull/14159#discussion_r2464722207
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -88,9 +89,10 @@ public class HoodieStreamerUtils {
*/
public static Option<JavaRDD<HoodieRecord>> createHoodieRecords(HoodieStreamer.Config cfg, TypedProperties props, Option<JavaRDD<GenericRecord>> avroRDDOptional,
SchemaProvider schemaProvider, HoodieRecord.HoodieRecordType recordType, boolean autoGenerateRecordKeys,
- String instantTime, Option<BaseErrorTableWriter> errorTableWriter) {
+ String instantTime, Option<BaseErrorTableWriter> errorTableWriter, HoodieTableConfig tableConfig) {
boolean shouldCombine = cfg.filterDupes || cfg.operation.equals(WriteOperationType.UPSERT);
- boolean shouldUseOrderingField = shouldCombine && !StringUtils.isNullOrEmpty(cfg.sourceOrderingFields);
+ String orderingFieldsStr = tableConfig.getOrderingFieldsStr().orElse(cfg.sourceOrderingFields);
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #14159:
URL: https://github.com/apache/hudi/pull/14159#discussion_r2466658342
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToNineUpgradeHandler.java:
##########
@@ -224,13 +224,13 @@ private void reconcileMergePropertiesConfig(Map<ConfigProperty, String> tablePro
}
}
- private void reconcileOrderingFieldsConfig(Map<ConfigProperty, String> tablePropsToAdd,
- Set<ConfigProperty> tablePropsToRemove,
- HoodieTableConfig tableConfig) {
- String payloadClass = tableConfig.getPayloadClass();
+ private void reconcileOrderingFieldsConfig(Map<ConfigProperty, String> tablePropsToAdd, Set<ConfigProperty> tablePropsToRemove,
+ HoodieTableConfig tableConfig, HoodieWriteConfig config) {
+ String payloadClass = tableConfig.getPayloadClassIfPresent()
Review Comment:
Addressed
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -1145,6 +1149,16 @@ private Pair<HoodieWriteConfig, Schema> getHoodieClientConfigAndWriterSchema(Sch
if (!StringUtils.isNullOrEmpty(cfg.recordMergeStrategyId)) {
builder.withRecordMergeStrategyId(cfg.recordMergeStrategyId);
}
+
+ if (metaClient != null) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (tableConfig.getLegacyPayloadClass().equals(MySqlDebeziumAvroPayload.class.getCanonicalName())
+ && cfg.sourceOrderingFields.equals(DebeziumConstants.ADDED_SEQ_COL_NAME)) {
Review Comment:
Addressed
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -1145,6 +1149,16 @@ private Pair<HoodieWriteConfig, Schema> getHoodieClientConfigAndWriterSchema(Sch
if (!StringUtils.isNullOrEmpty(cfg.recordMergeStrategyId)) {
builder.withRecordMergeStrategyId(cfg.recordMergeStrategyId);
}
+
+ if (metaClient != null) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (tableConfig.getLegacyPayloadClass().equals(MySqlDebeziumAvroPayload.class.getCanonicalName())
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bug: Fix upgrade handling for MySqlDebeziumAvroPayload with deltastreamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14159:
URL: https://github.com/apache/hudi/pull/14159#issuecomment-3452746125
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9337",
"triggerID" : "5cec299dfab38cfdd83290146e7ed0647f74c059",
"triggerType" : "PUSH"
}, {
"hash" : "0212e66c850bcce551fd24e904af558e427bc420",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9352",
"triggerID" : "0212e66c850bcce551fd24e904af558e427bc420",
"triggerType" : "PUSH"
}, {
"hash" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9354",
"triggerID" : "49c1ab81ac8b6b409e9ad56e96ed631a6a5b84fc",
"triggerType" : "PUSH"
}, {
"hash" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355",
"triggerID" : "10579a83aa781987c21a4f450085687fb0e0e2eb",
"triggerType" : "PUSH"
}, {
"hash" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361",
"triggerID" : "22eeda6a2ebb10efac2b53022241efc4a9a28419",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 10579a83aa781987c21a4f450085687fb0e0e2eb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9355)
* 22eeda6a2ebb10efac2b53022241efc4a9a28419 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9361)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org