You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2023/08/31 14:01:25 UTC
[GitHub] [hudi] codope opened a new pull request, #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
codope opened a new pull request, #9590:
URL: https://github.com/apache/hudi/pull/9590
### Change Logs
There are three configs in the `hoodie.properties` that hold classnames; `KEY_GENERATOR_CLASS_NAME`, `BOOTSTRAP_INDEX_CLASS_NAME`, `PAYLOAD_CLASS_NAME`. This PR adds enums for them and sets the enum in table config while inferring the classname from the enum in the getters.
### Impact
Table properties will not have any classname. The class name configs have only been depreacated but not removed for compatibility.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1716298552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 2cae9ba66bd537eb95161d789aad239e11a2e7ed Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1719377321
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
}, {
"hash" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887",
"triggerID" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"triggerType" : "PUSH"
}, {
"hash" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19906",
"triggerID" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"triggerType" : "PUSH"
}, {
"hash" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19909",
"triggerID" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 64e168c445a760c24d4287e6c5e63cc06628a105 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19909)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1711530562
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* c57e3b13a0f5eb6976b59a34e7b55625a9c74852 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737)
* f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1719109599
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
}, {
"hash" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887",
"triggerID" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"triggerType" : "PUSH"
}, {
"hash" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19906",
"triggerID" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"triggerType" : "PUSH"
}, {
"hash" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 4d355db5be1374e72f3a71bc56247888ef5cf4d0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887)
* 0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19906)
* 64e168c445a760c24d4287e6c5e63cc06628a105 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325526998
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ConfigUtils.java:
##########
@@ -78,7 +79,9 @@ public static String getOrderingField(Properties properties) {
*/
public static String getPayloadClass(Properties properties) {
String payloadClass = null;
- if (properties.containsKey(HoodieTableConfig.PAYLOAD_CLASS_NAME.key())) {
+ if (properties.containsKey(HoodieTableConfig.PAYLOAD_TYPE.key())) {
Review Comment:
Can `RecordPayloadType#getPayloadClassName` be used instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1711689319
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740)
* 388d94f01fc4dd96b56d796a7ff8edc3d1f393f8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1713715609
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 388d94f01fc4dd96b56d796a7ff8edc3d1f393f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743)
* 9525a30ed355aade0302b8b9673960d9d146fb5b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325734361
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -269,6 +270,8 @@ private Pair<String, String> genKeyGenClassAndPartitionColumns() {
keyGenClass = props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key());
} else if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), null))) {
keyGenClass = HoodieSparkKeyGeneratorFactory.getKeyGeneratorClassName(props);
+ } else if (StringUtils.nonEmpty(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key(), null))) {
+ keyGenClass = KeyGeneratorType.valueOf(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key())).getClassName();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1711519697
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e44c614583ebf6b398cf7a216141ae7870105262 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578)
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* c57e3b13a0f5eb6976b59a34e7b55625a9c74852 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737)
* f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1711449324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e44c614583ebf6b398cf7a216141ae7870105262 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578)
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1324238668
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -151,6 +152,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. "
+ "This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective");
+ public static final ConfigProperty<String> WRITE_PAYLOAD_TYPE = ConfigProperty
+ .key("hoodie.datasource.write.payload.type")
+ .defaultValue(RecordPayloadType.OVERWRITE_LATEST_AVRO.name())
Review Comment:
HUDI-6854 for changing default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1718930742
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
}, {
"hash" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887",
"triggerID" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"triggerType" : "PUSH"
}, {
"hash" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 4d355db5be1374e72f3a71bc56247888ef5cf4d0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887)
* 0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325523969
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -269,6 +270,8 @@ private Pair<String, String> genKeyGenClassAndPartitionColumns() {
keyGenClass = props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key());
} else if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), null))) {
keyGenClass = HoodieSparkKeyGeneratorFactory.getKeyGeneratorClassName(props);
+ } else if (StringUtils.nonEmpty(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key(), null))) {
+ keyGenClass = KeyGeneratorType.valueOf(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key())).getClassName();
Review Comment:
Can the `KeyGeneratorType#getKeyGeneratorClassName` be used instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1719132475
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
}, {
"hash" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887",
"triggerID" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"triggerType" : "PUSH"
}, {
"hash" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19906",
"triggerID" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"triggerType" : "PUSH"
}, {
"hash" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19909",
"triggerID" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19906)
* 64e168c445a760c24d4287e6c5e63cc06628a105 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19909)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1719582321
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64e168c445a760c24d4287e6c5e63cc06628a105 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1718943221
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
}, {
"hash" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887",
"triggerID" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"triggerType" : "PUSH"
}, {
"hash" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19906",
"triggerID" : "0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 4d355db5be1374e72f3a71bc56247888ef5cf4d0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887)
* 0ca69f5d02b4919b5f8d81f4f6c7ff65213bf0a5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19906)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6780] Introduce enums instead of classnames in table properties [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1771817705
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1031,6 +1050,9 @@ public PropertyBuilder fromProperties(Properties properties) {
setBootstrapIndexClass(
hoodieConfig.getString(HoodieTableConfig.BOOTSTRAP_INDEX_CLASS_NAME));
}
+ if (hoodieConfig.contains(HoodieTableConfig.BOOTSTRAP_INDEX_TYPE)) {
Review Comment:
good catch! This condition should call `setBootstrapIndexClass`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1701145615
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e44c614583ebf6b398cf7a216141ae7870105262 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1701220202
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e44c614583ebf6b398cf7a216141ae7870105262 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1711675475
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* c57e3b13a0f5eb6976b59a34e7b55625a9c74852 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737)
* f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740)
* 388d94f01fc4dd96b56d796a7ff8edc3d1f393f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1324228818
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -86,11 +86,11 @@ private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partit
if (!StringUtils.isNullOrEmpty(partitionFields)) {
int numPartFields = partitionFields.split(",").length;
if (numPartFields == 1) {
- return KeyGeneratorType.SIMPLE;
+ return KeyGeneratorType.SIMPLE_KEYGEN;
}
- return KeyGeneratorType.COMPLEX;
+ return KeyGeneratorType.COMPLEX_KEYGEN;
Review Comment:
Yeah earlier I had kept the same. But, while fixing some tests, I realized there is index type SIMPLE and keygen type SIMPLE and with static imports it might not be very readable. So, I added the suffix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325786435
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ConfigUtils.java:
##########
@@ -77,13 +79,7 @@ public static String getOrderingField(Properties properties) {
* Get payload class.
*/
public static String getPayloadClass(Properties properties) {
- String payloadClass = null;
- if (properties.containsKey(HoodieTableConfig.PAYLOAD_CLASS_NAME.key())) {
- payloadClass = properties.getProperty(HoodieTableConfig.PAYLOAD_CLASS_NAME.key());
- } else if (properties.containsKey("hoodie.datasource.write.payload.class")) {
- payloadClass = properties.getProperty("hoodie.datasource.write.payload.class");
- }
- return payloadClass;
+ return RecordPayloadType.getPayloadClassName(new HoodieConfig(properties));
}
Review Comment:
Great fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325834236
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -269,6 +270,8 @@ private Pair<String, String> genKeyGenClassAndPartitionColumns() {
keyGenClass = props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key());
} else if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), null))) {
keyGenClass = HoodieSparkKeyGeneratorFactory.getKeyGeneratorClassName(props);
+ } else if (StringUtils.nonEmpty(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key(), null))) {
+ keyGenClass = KeyGeneratorType.valueOf(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key())).getClassName();
Review Comment:
Reasonable ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1711506905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e44c614583ebf6b398cf7a216141ae7870105262 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578)
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* c57e3b13a0f5eb6976b59a34e7b55625a9c74852 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1713702085
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 388d94f01fc4dd96b56d796a7ff8edc3d1f393f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743)
* 9525a30ed355aade0302b8b9673960d9d146fb5b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1716146147
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 9525a30ed355aade0302b8b9673960d9d146fb5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793)
* 2cae9ba66bd537eb95161d789aad239e11a2e7ed Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325470130
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -86,11 +86,11 @@ private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partit
if (!StringUtils.isNullOrEmpty(partitionFields)) {
int numPartFields = partitionFields.split(",").length;
if (numPartFields == 1) {
- return KeyGeneratorType.SIMPLE;
+ return KeyGeneratorType.SIMPLE_KEYGEN;
}
- return KeyGeneratorType.COMPLEX;
+ return KeyGeneratorType.COMPLEX_KEYGEN;
Review Comment:
I've reverted the suffix in each enum. Didn't want to keep the suffix in some for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325788308
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -269,6 +270,8 @@ private Pair<String, String> genKeyGenClassAndPartitionColumns() {
keyGenClass = props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key());
} else if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), null))) {
keyGenClass = HoodieSparkKeyGeneratorFactory.getKeyGeneratorClassName(props);
+ } else if (StringUtils.nonEmpty(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key(), null))) {
+ keyGenClass = KeyGeneratorType.valueOf(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key())).getClassName();
Review Comment:
Is this resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1323833329
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -86,11 +86,11 @@ private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partit
if (!StringUtils.isNullOrEmpty(partitionFields)) {
int numPartFields = partitionFields.split(",").length;
if (numPartFields == 1) {
- return KeyGeneratorType.SIMPLE;
+ return KeyGeneratorType.SIMPLE_KEYGEN;
}
- return KeyGeneratorType.COMPLEX;
+ return KeyGeneratorType.COMPLEX_KEYGEN;
Review Comment:
Not sure we have all these `_KEYGEN` as suffix because it is implied implicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1719602626
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19909",
"triggerID" : "64e168c445a760c24d4287e6c5e63cc06628a105",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64e168c445a760c24d4287e6c5e63cc06628a105 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19909)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1717301579
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
}, {
"hash" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 2cae9ba66bd537eb95161d789aad239e11a2e7ed Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874)
* 4d355db5be1374e72f3a71bc56247888ef5cf4d0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1322357771
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -540,10 +566,17 @@ public void setTableVersion(HoodieTableVersion tableVersion) {
* Read the payload class for HoodieRecords from the table properties.
*/
public String getPayloadClass() {
- // There could be tables written with payload class from com.uber.hoodie. Need to transparently
- // change to org.apache.hudi
- return getStringOrDefault(PAYLOAD_CLASS_NAME).replace("com.uber.hoodie",
- "org.apache.hudi");
+ String payloadClassName;
+ if (contains(PAYLOAD_TYPE)) {
+ payloadClassName = RecordPayloadType.valueOf(getString(PAYLOAD_TYPE)).getClassName();
+ } else if (contains(PAYLOAD_CLASS_NAME)) {
+ payloadClassName = getString(PAYLOAD_CLASS_NAME);
+ } else {
+ payloadClassName = RecordPayloadType.valueOf(PAYLOAD_TYPE.defaultValue()).getClassName();
+ }
Review Comment:
I don't think so, it's very hard to keep the logic int consistency because they are scatered all around, such code are likely to incur new bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1318515074
##########
hudi-common/src/main/java/org/apache/hudi/common/model/RecordPayloadType.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.config.EnumDescription;
+import org.apache.hudi.common.config.EnumFieldDescription;
+import org.apache.hudi.common.model.debezium.MySqlDebeziumAvroPayload;
+import org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload;
+
+/**
+ * Payload to use for record.
+ */
+@EnumDescription("Payload to use for merging records")
+public enum RecordPayloadType {
+ @EnumFieldDescription("Provides support for seamlessly applying changes captured via Amazon Database Migration Service onto S3.")
+ AWS_DMS_AVRO(AWSDmsAvroPayload.class.getName()),
+
+ @EnumFieldDescription("Honors ordering field in both preCombine and combineAndGetUpdateValue.")
+ HOODIE_AVRO_DEFAULT(DefaultHoodieRecordPayload.class.getName()),
Review Comment:
For write config, users can set both classname or enum. For compatibility reasons, I introduced a new config e.g. `WRITE_PAYLOAD_TYPE`, so that it does not break for users who are setting existing config. This PR ensures that the classnames don't get written to `hoodie.properties`. #8566 handles for write configs. It does not fix table properties. After this PR, #8566 should be rebased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1321299763
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -540,10 +566,17 @@ public void setTableVersion(HoodieTableVersion tableVersion) {
* Read the payload class for HoodieRecords from the table properties.
*/
public String getPayloadClass() {
- // There could be tables written with payload class from com.uber.hoodie. Need to transparently
- // change to org.apache.hudi
- return getStringOrDefault(PAYLOAD_CLASS_NAME).replace("com.uber.hoodie",
- "org.apache.hudi");
+ String payloadClassName;
+ if (contains(PAYLOAD_TYPE)) {
+ payloadClassName = RecordPayloadType.valueOf(getString(PAYLOAD_TYPE)).getClassName();
+ } else if (contains(PAYLOAD_CLASS_NAME)) {
+ payloadClassName = getString(PAYLOAD_CLASS_NAME);
+ } else {
+ payloadClassName = RecordPayloadType.valueOf(PAYLOAD_TYPE.defaultValue()).getClassName();
+ }
Review Comment:
I understand your concern about code reusability but I think enums should be dumb. This logic depends on table config so it's better to keep here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6780] Introduce enums instead of classnames in table properties [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1771761379
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1031,6 +1050,9 @@ public PropertyBuilder fromProperties(Properties properties) {
setBootstrapIndexClass(
hoodieConfig.getString(HoodieTableConfig.BOOTSTRAP_INDEX_CLASS_NAME));
}
+ if (hoodieConfig.contains(HoodieTableConfig.BOOTSTRAP_INDEX_TYPE)) {
Review Comment:
I don't think this is correct? @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1320930702
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -151,6 +152,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. "
+ "This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective");
+ public static final ConfigProperty<String> WRITE_PAYLOAD_TYPE = ConfigProperty
+ .key("hoodie.datasource.write.payload.type")
+ .defaultValue(RecordPayloadType.OVERWRITE_LATEST_AVRO.name())
Review Comment:
Should use `HOODIE_AVRO_DEFAULT` as the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1712127002
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 388d94f01fc4dd96b56d796a7ff8edc3d1f393f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #9590:
URL: https://github.com/apache/hudi/pull/9590
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1323321914
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -540,10 +566,17 @@ public void setTableVersion(HoodieTableVersion tableVersion) {
* Read the payload class for HoodieRecords from the table properties.
*/
public String getPayloadClass() {
- // There could be tables written with payload class from com.uber.hoodie. Need to transparently
- // change to org.apache.hudi
- return getStringOrDefault(PAYLOAD_CLASS_NAME).replace("com.uber.hoodie",
- "org.apache.hudi");
+ String payloadClassName;
+ if (contains(PAYLOAD_TYPE)) {
+ payloadClassName = RecordPayloadType.valueOf(getString(PAYLOAD_TYPE)).getClassName();
+ } else if (contains(PAYLOAD_CLASS_NAME)) {
+ payloadClassName = getString(PAYLOAD_CLASS_NAME);
+ } else {
+ payloadClassName = RecordPayloadType.valueOf(PAYLOAD_TYPE.defaultValue()).getClassName();
+ }
Review Comment:
hmm.. i think you are right. this has a chance of introducing bugs.
i've refactored as suggested. please take a look at `KeyGeneratorType`, `RecordPayloadType`, `BootstrapIndexType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325788308
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -269,6 +270,8 @@ private Pair<String, String> genKeyGenClassAndPartitionColumns() {
keyGenClass = props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key());
} else if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), null))) {
keyGenClass = HoodieSparkKeyGeneratorFactory.getKeyGeneratorClassName(props);
+ } else if (StringUtils.nonEmpty(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key(), null))) {
+ keyGenClass = KeyGeneratorType.valueOf(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key())).getClassName();
Review Comment:
Is this resolved? I mean to replace all the if else related with keygen class name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1323834878
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -447,7 +450,14 @@ public void validateTableProperties(Properties properties) {
// Meta fields can be disabled only when either {@code SimpleKeyGenerator}, {@code ComplexKeyGenerator}, {@code NonpartitionedKeyGenerator} is used
if (!getTableConfig().populateMetaFields()) {
- String keyGenClass = properties.getProperty(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key(), "org.apache.hudi.keygen.SimpleKeyGenerator");
+ String keyGenClass;
+ if (properties.containsKey(HoodieTableConfig.KEY_GENERATOR_TYPE.key())) {
+ keyGenClass = KeyGeneratorType.valueOf(properties.getProperty(HoodieTableConfig.KEY_GENERATOR_TYPE.key())).getClassName();
+ } else if (properties.containsKey(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key())) {
+ keyGenClass = properties.getProperty(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key());
+ } else {
+ keyGenClass = "org.apache.hudi.keygen.SimpleKeyGenerator";
Review Comment:
Can we also move this constant class name "org.apache.hudi.keygen.SimpleKeyGenerator" into `KeyGeneratorType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325213089
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -86,11 +86,11 @@ private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partit
if (!StringUtils.isNullOrEmpty(partitionFields)) {
int numPartFields = partitionFields.split(",").length;
if (numPartFields == 1) {
- return KeyGeneratorType.SIMPLE;
+ return KeyGeneratorType.SIMPLE_KEYGEN;
}
- return KeyGeneratorType.COMPLEX;
+ return KeyGeneratorType.COMPLEX_KEYGEN;
Review Comment:
> with static imports it might not be very readable.
Not very persuasive from my comprehension, just fix these two variables should be okay if you think they are confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1312518092
##########
hudi-common/src/main/java/org/apache/hudi/common/model/RecordPayloadType.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.config.EnumDescription;
+import org.apache.hudi.common.config.EnumFieldDescription;
+import org.apache.hudi.common.model.debezium.MySqlDebeziumAvroPayload;
+import org.apache.hudi.common.model.debezium.PostgresDebeziumAvroPayload;
+
+/**
+ * Payload to use for record.
+ */
+@EnumDescription("Payload to use for merging records")
+public enum RecordPayloadType {
+ @EnumFieldDescription("Provides support for seamlessly applying changes captured via Amazon Database Migration Service onto S3.")
+ AWS_DMS_AVRO(AWSDmsAvroPayload.class.getName()),
+
+ @EnumFieldDescription("Honors ordering field in both preCombine and combineAndGetUpdateValue.")
+ HOODIE_AVRO_DEFAULT(DefaultHoodieRecordPayload.class.getName()),
Review Comment:
Are these options expected to be used by users? Then there might be in-consistency for table config and write config, for write config, we still prefer the class name ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1320932463
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -602,18 +635,26 @@ public String getPartitionFieldProp() {
* Read the payload class for HoodieRecords from the table properties.
*/
public String getBootstrapIndexClass() {
- // There could be tables written with payload class from com.uber.hoodie. Need to transparently
- // change to org.apache.hudi
- return getStringOrDefault(BOOTSTRAP_INDEX_CLASS_NAME, getDefaultBootstrapIndexClass(props));
+ if (!props.getBoolean(BOOTSTRAP_INDEX_ENABLE.key(), BOOTSTRAP_INDEX_ENABLE.defaultValue())) {
+ return BootstrapIndexType.NO_OP.getClassName();
+ }
+ String bootstrapIndexClassName;
+ if (contains(BOOTSTRAP_INDEX_TYPE)) {
+ bootstrapIndexClassName = BootstrapIndexType.valueOf(getString(BOOTSTRAP_INDEX_TYPE)).getClassName();
+ } else if (contains(BOOTSTRAP_INDEX_CLASS_NAME)) {
+ bootstrapIndexClassName = getString(BOOTSTRAP_INDEX_CLASS_NAME);
+ } else {
+ bootstrapIndexClassName = BootstrapIndexType.valueOf(BOOTSTRAP_INDEX_TYPE.defaultValue()).getClassName();
Review Comment:
It's better we move this method into the enum class itself, so that there is no similiar patched logic scattered everywhere.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -602,18 +635,26 @@ public String getPartitionFieldProp() {
* Read the payload class for HoodieRecords from the table properties.
*/
public String getBootstrapIndexClass() {
- // There could be tables written with payload class from com.uber.hoodie. Need to transparently
- // change to org.apache.hudi
- return getStringOrDefault(BOOTSTRAP_INDEX_CLASS_NAME, getDefaultBootstrapIndexClass(props));
+ if (!props.getBoolean(BOOTSTRAP_INDEX_ENABLE.key(), BOOTSTRAP_INDEX_ENABLE.defaultValue())) {
+ return BootstrapIndexType.NO_OP.getClassName();
+ }
+ String bootstrapIndexClassName;
+ if (contains(BOOTSTRAP_INDEX_TYPE)) {
+ bootstrapIndexClassName = BootstrapIndexType.valueOf(getString(BOOTSTRAP_INDEX_TYPE)).getClassName();
+ } else if (contains(BOOTSTRAP_INDEX_CLASS_NAME)) {
+ bootstrapIndexClassName = getString(BOOTSTRAP_INDEX_CLASS_NAME);
+ } else {
+ bootstrapIndexClassName = BootstrapIndexType.valueOf(BOOTSTRAP_INDEX_TYPE.defaultValue()).getClassName();
+ }
+ return bootstrapIndexClassName;
}
public static String getDefaultBootstrapIndexClass(Properties props) {
HoodieConfig hoodieConfig = new HoodieConfig(props);
- String defaultClass = BOOTSTRAP_INDEX_CLASS_NAME.defaultValue();
if (!hoodieConfig.getBooleanOrDefault(BOOTSTRAP_INDEX_ENABLE)) {
- defaultClass = NO_OP_BOOTSTRAP_INDEX_CLASS;
+ return BootstrapIndexType.NO_OP.getClassName();
}
- return defaultClass;
+ return BootstrapIndexType.valueOf(BOOTSTRAP_INDEX_TYPE.defaultValue()).getClassName();
Review Comment:
Move this into `BootstrapIndexType`
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ConfigUtils.java:
##########
@@ -78,7 +79,9 @@ public static String getOrderingField(Properties properties) {
*/
public static String getPayloadClass(Properties properties) {
String payloadClass = null;
- if (properties.containsKey(HoodieTableConfig.PAYLOAD_CLASS_NAME.key())) {
+ if (properties.containsKey(HoodieTableConfig.PAYLOAD_TYPE.key())) {
+ payloadClass = RecordPayloadType.valueOf(properties.getProperty(HoodieTableConfig.PAYLOAD_TYPE.key())).getClassName();
Review Comment:
Not sure why we need this method separately, can we move it altogether into `RecordPayloadType`.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -447,7 +450,14 @@ public void validateTableProperties(Properties properties) {
// Meta fields can be disabled only when either {@code SimpleKeyGenerator}, {@code ComplexKeyGenerator}, {@code NonpartitionedKeyGenerator} is used
if (!getTableConfig().populateMetaFields()) {
- String keyGenClass = properties.getProperty(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key(), "org.apache.hudi.keygen.SimpleKeyGenerator");
+ String keyGenClass;
+ if (properties.containsKey(HoodieTableConfig.KEY_GENERATOR_TYPE.key())) {
+ keyGenClass = KeyGeneratorType.valueOf(properties.getProperty(HoodieTableConfig.KEY_GENERATOR_TYPE.key())).getClassName();
+ } else if (properties.containsKey(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key())) {
+ keyGenClass = properties.getProperty(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME.key());
+ } else {
+ keyGenClass = "org.apache.hudi.keygen.SimpleKeyGenerator";
Review Comment:
Move this into `KeyGeneratorType`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -697,7 +738,13 @@ public HoodieCDCSupplementalLoggingMode cdcSupplementalLoggingMode() {
}
public String getKeyGeneratorClassName() {
- return getString(KEY_GENERATOR_CLASS_NAME);
+ String keyGeneratorClassName = null;
+ if (contains(KEY_GENERATOR_TYPE)) {
+ keyGeneratorClassName = KeyGeneratorType.valueOf(getString(KEY_GENERATOR_TYPE)).getClassName();
+ } else if (contains(KEY_GENERATOR_CLASS_NAME)) {
+ keyGeneratorClassName = getString(KEY_GENERATOR_CLASS_NAME);
+ }
Review Comment:
Move this into `KeyGeneratorType`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -540,10 +566,17 @@ public void setTableVersion(HoodieTableVersion tableVersion) {
* Read the payload class for HoodieRecords from the table properties.
*/
public String getPayloadClass() {
- // There could be tables written with payload class from com.uber.hoodie. Need to transparently
- // change to org.apache.hudi
- return getStringOrDefault(PAYLOAD_CLASS_NAME).replace("com.uber.hoodie",
- "org.apache.hudi");
+ String payloadClassName;
+ if (contains(PAYLOAD_TYPE)) {
+ payloadClassName = RecordPayloadType.valueOf(getString(PAYLOAD_TYPE)).getClassName();
+ } else if (contains(PAYLOAD_CLASS_NAME)) {
+ payloadClassName = getString(PAYLOAD_CLASS_NAME);
+ } else {
+ payloadClassName = RecordPayloadType.valueOf(PAYLOAD_TYPE.defaultValue()).getClassName();
+ }
Review Comment:
It's better we move this method into the enum class itself, so that there is no similiar patched logic scattered everywhere.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1243,7 +1250,11 @@ public String getPreCombineField() {
}
public String getWritePayloadClass() {
- return getString(WRITE_PAYLOAD_CLASS_NAME);
+ RecordPayloadType payloadType = RecordPayloadType.valueOf(getString(WRITE_PAYLOAD_TYPE));
+ if (payloadType.equals(RecordPayloadType.CUSTOM)) {
+ return getString(WRITE_PAYLOAD_CLASS_NAME);
Review Comment:
How about the user does not set up the payload type explicitly, is there any inference for that?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -230,7 +234,11 @@ object HoodieWriterUtils {
val diffConfigs = StringBuilder.newBuilder
if (null != tableConfig) {
- val tableConfigKeyGen = tableConfig.getString(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME)
+ val tableConfigKeyGen = if (tableConfig.getProps.containsKey(HoodieTableConfig.KEY_GENERATOR_TYPE)) {
+ KeyGeneratorType.valueOf(tableConfig.getString(HoodieTableConfig.KEY_GENERATOR_TYPE)).getClassName
+ } else {
+ tableConfig.getString(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME)
Review Comment:
refactor to reuse utilities from `KeyGeneratorType` instead.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -194,7 +194,11 @@ object HoodieWriterUtils {
}
val datasourceKeyGen = getOriginKeyGenerator(params)
- val tableConfigKeyGen = tableConfig.getString(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME)
+ val tableConfigKeyGen = if (tableConfig.getProps.containsKey(HoodieTableConfig.KEY_GENERATOR_TYPE)) {
+ KeyGeneratorType.valueOf(tableConfig.getString(HoodieTableConfig.KEY_GENERATOR_TYPE)).getClassName
+ } else {
+ tableConfig.getString(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME)
Review Comment:
refactor to reuse utilities from `KeyGeneratorType` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1321297397
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -151,6 +152,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. "
+ "This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective");
+ public static final ConfigProperty<String> WRITE_PAYLOAD_TYPE = ConfigProperty
+ .key("hoodie.datasource.write.payload.type")
+ .defaultValue(RecordPayloadType.OVERWRITE_LATEST_AVRO.name())
Review Comment:
We can take up changing the default in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1321298553
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1243,7 +1250,11 @@ public String getPreCombineField() {
}
public String getWritePayloadClass() {
- return getString(WRITE_PAYLOAD_CLASS_NAME);
+ RecordPayloadType payloadType = RecordPayloadType.valueOf(getString(WRITE_PAYLOAD_TYPE));
+ if (payloadType.equals(RecordPayloadType.CUSTOM)) {
+ return getString(WRITE_PAYLOAD_CLASS_NAME);
Review Comment:
Actually, this method is not used. I am going to remove it. However, to your point about inference, we do that in `HoodieTableConfig` or `HoodieWriterUtils`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1321301288
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -697,7 +738,13 @@ public HoodieCDCSupplementalLoggingMode cdcSupplementalLoggingMode() {
}
public String getKeyGeneratorClassName() {
- return getString(KEY_GENERATOR_CLASS_NAME);
+ String keyGeneratorClassName = null;
+ if (contains(KEY_GENERATOR_TYPE)) {
+ keyGeneratorClassName = KeyGeneratorType.valueOf(getString(KEY_GENERATOR_TYPE)).getClassName();
+ } else if (contains(KEY_GENERATOR_CLASS_NAME)) {
+ keyGeneratorClassName = getString(KEY_GENERATOR_CLASS_NAME);
+ }
Review Comment:
please check my comment about not keeping this logic in enum at https://github.com/apache/hudi/pull/9590/files#r1321299763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1714129172
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 9525a30ed355aade0302b8b9673960d9d146fb5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1717368808
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
}, {
"hash" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887",
"triggerID" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 2cae9ba66bd537eb95161d789aad239e11a2e7ed Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874)
* 4d355db5be1374e72f3a71bc56247888ef5cf4d0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1717546308
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19874",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
}, {
"hash" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887",
"triggerID" : "4d355db5be1374e72f3a71bc56247888ef5cf4d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 4d355db5be1374e72f3a71bc56247888ef5cf4d0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19887)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9590:
URL: https://github.com/apache/hudi/pull/9590#issuecomment-1716087065
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e44c614583ebf6b398cf7a216141ae7870105262",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19578",
"triggerID" : "e44c614583ebf6b398cf7a216141ae7870105262",
"triggerType" : "PUSH"
}, {
"hash" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57ba4408f8b2abf3af75e5eddb768200a0ea7654",
"triggerType" : "PUSH"
}, {
"hash" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19737",
"triggerID" : "c57e3b13a0f5eb6976b59a34e7b55625a9c74852",
"triggerType" : "PUSH"
}, {
"hash" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19740",
"triggerID" : "f9bf5ff45d3ff60132a39f84b4f7b8cbb80b1117",
"triggerType" : "PUSH"
}, {
"hash" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19743",
"triggerID" : "388d94f01fc4dd96b56d796a7ff8edc3d1f393f8",
"triggerType" : "PUSH"
}, {
"hash" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793",
"triggerID" : "9525a30ed355aade0302b8b9673960d9d146fb5b",
"triggerType" : "PUSH"
}, {
"hash" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2cae9ba66bd537eb95161d789aad239e11a2e7ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57ba4408f8b2abf3af75e5eddb768200a0ea7654 UNKNOWN
* 9525a30ed355aade0302b8b9673960d9d146fb5b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19793)
* 2cae9ba66bd537eb95161d789aad239e11a2e7ed UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1323835958
##########
hudi-common/src/main/java/org/apache/hudi/keygen/constant/KeyGeneratorType.java:
##########
@@ -33,31 +37,92 @@
public enum KeyGeneratorType {
@EnumFieldDescription("Simple key generator, which takes names of fields to be used for recordKey and partitionPath as configs.")
- SIMPLE,
+ SIMPLE_KEYGEN("org.apache.hudi.keygen.SimpleKeyGenerator"),
+ @EnumFieldDescription("Simple key generator, which takes names of fields to be used for recordKey and partitionPath as configs.")
+ SIMPLE_AVRO_KEYGEN("org.apache.hudi.keygen.SimpleAvroKeyGenerator"),
@EnumFieldDescription("Complex key generator, which takes names of fields to be used for recordKey and partitionPath as configs.")
- COMPLEX,
+ COMPLEX_KEYGEN("org.apache.hudi.keygen.ComplexKeyGenerator"),
+ @EnumFieldDescription("Complex key generator, which takes names of fields to be used for recordKey and partitionPath as configs.")
+ COMPLEX_AVRO_KEYGEN("org.apache.hudi.keygen.ComplexAvroKeyGenerator"),
@EnumFieldDescription("Timestamp-based key generator, that relies on timestamps for partitioning field. Still picks record key by name.")
- TIMESTAMP,
+ TIMESTAMP_KEYGEN("org.apache.hudi.keygen.TimestampBasedKeyGenerator"),
+ @EnumFieldDescription("Timestamp-based key generator, that relies on timestamps for partitioning field. Still picks record key by name.")
+ TIMESTAMP_AVRO_KEYGEN("org.apache.hudi.keygen.TimestampBasedAvroKeyGenerator"),
@EnumFieldDescription("This is a generic implementation type of KeyGenerator where users can configure record key as a single field or "
+ " a combination of fields. Similarly partition path can be configured to have multiple fields or only one field. "
+ " This KeyGenerator expects value for prop \"hoodie.datasource.write.partitionpath.field\" in a specific format. "
+ " For example: "
+ " properties.put(\"hoodie.datasource.write.partitionpath.field\", \"field1:PartitionKeyType1,field2:PartitionKeyType2\").")
- CUSTOM,
+ CUSTOM_KEYGEN("org.apache.hudi.keygen.CustomKeyGenerator"),
+ @EnumFieldDescription("This is a generic implementation type of KeyGenerator where users can configure record key as a single field or "
+ + " a combination of fields. Similarly partition path can be configured to have multiple fields or only one field. "
+ + " This KeyGenerator expects value for prop \"hoodie.datasource.write.partitionpath.field\" in a specific format. "
+ + " For example: "
+ + " properties.put(\"hoodie.datasource.write.partitionpath.field\", \"field1:PartitionKeyType1,field2:PartitionKeyType2\").")
+ CUSTOM_AVRO_KEYGEN("org.apache.hudi.keygen.CustomAvroKeyGenerator"),
@EnumFieldDescription("Simple Key generator for non-partitioned tables.")
- NON_PARTITION,
+ NON_PARTITION_KEYGEN("org.apache.hudi.keygen.NonpartitionedKeyGenerator"),
+ @EnumFieldDescription("Simple Key generator for non-partitioned tables.")
+ NON_PARTITION_AVRO_KEYGEN("org.apache.hudi.keygen.NonpartitionedAvroKeyGenerator"),
@EnumFieldDescription("Key generator for deletes using global indices.")
- GLOBAL_DELETE;
+ GLOBAL_DELETE_KEYGEN("org.apache.hudi.keygen.GlobalDeleteKeyGenerator"),
+ @EnumFieldDescription("Key generator for deletes using global indices.")
+ GLOBAL_DELETE_AVRO_KEYGEN("org.apache.hudi.keygen.GlobalAvroDeleteKeyGenerator"),
+
+ @EnumFieldDescription("Automatic record key generation.")
+ AUTO_RECORD_KEYGEN("org.apache.hudi.keygen.AutoRecordGenWrapperKeyGenerator"),
+ @EnumFieldDescription("Automatic record key generation.")
+ AUTO_RECORD_AVRO_KEYGEN("org.apache.hudi.keygen.AutoRecordGenWrapperAvroKeyGenerator"),
+
+ @EnumFieldDescription("A KeyGenerator which use the uuid as the record key.")
+ UUID_KEYGEN("org.apache.spark.sql.hudi.command.UuidKeyGenerator"),
+
+ @EnumFieldDescription("Custom key generator for the Hudi table metadata.")
+ HOODIE_METADATA_KEYGEN("org.apache.hudi.metadata.HoodieTableMetadataKeyGenerator"),
+
+ @EnumFieldDescription("Meant to be used internally for the spark sql MERGE INTO command.")
+ MERGE_INTO_KEYGEN("org.apache.spark.sql.hudi.command.MergeIntoKeyGenerator"),
+
+ @EnumFieldDescription("Custom spark-sql specific KeyGenerator overriding behavior handling TimestampType partition values.")
+ SQL_KEYGEN("org.apache.spark.sql.hudi.command.SqlKeyGenerator");
+
+ private final String className;
+
+ KeyGeneratorType(String className) {
+ this.className = className;
+ }
+
+ public String getClassName() {
+ return className;
+ }
+
+ public static KeyGeneratorType fromClassName(String className) {
+ for (KeyGeneratorType type : KeyGeneratorType.values()) {
+ if (type.getClassName().equals(className)) {
+ return type;
+ }
+ }
+ throw new IllegalArgumentException("No KeyGeneratorType found for class name: " + className);
+ }
public static List<String> getNames() {
List<String> names = new ArrayList<>(KeyGeneratorType.values().length);
Arrays.stream(KeyGeneratorType.values())
.forEach(x -> names.add(x.name()));
return names;
}
+
+ public static String getKeyGeneratorClassName(HoodieConfig config) {
Review Comment:
Add annotation `@Nullable` to the method if there is any chance it could return null value, or return `Option` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9590: [HUDI-6780] Introduce enums instead of classnames in table properties
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9590:
URL: https://github.com/apache/hudi/pull/9590#discussion_r1325808124
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -269,6 +270,8 @@ private Pair<String, String> genKeyGenClassAndPartitionColumns() {
keyGenClass = props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key());
} else if (StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_TYPE.key(), null))) {
keyGenClass = HoodieSparkKeyGeneratorFactory.getKeyGeneratorClassName(props);
+ } else if (StringUtils.nonEmpty(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key(), null))) {
+ keyGenClass = KeyGeneratorType.valueOf(props.getString(HoodieTableConfig.KEY_GENERATOR_TYPE.key())).getClassName();
Review Comment:
For this one I left the existing if-else blocks because one of them uses `HoodieSparkKeyGeneratorFactory` which used some infer keygen function. I did not want to change that order here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org