You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2024/08/22 16:17:13 UTC
[PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
lokeshj1703 opened a new pull request, #11816:
URL: https://github.com/apache/hudi/pull/11816
### Change Logs
Currently KeyGenUtils#inferKeyGeneratorTypeForAutoKeyGen does not support custom key generators but supports the other key generator types. This PR aims to add support for it.
Also HoodieCatalogTable should be modified to add support for storing partition fields with key generator partition type. This was left out in [HUDI-7996](https://issues.apache.org/jira/browse/HUDI-7996) because of no support for custom keygen with HoodieCatalogTable
### Impact
NA
### Risk level (write none, low medium or high below)
low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1754328160
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,92 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id,name',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
Review Comment:
Added tests and corresponding fix for this use case. I think there are other places where partition path field is derived but it is better to fix those in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2333740955
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22ea4eca27b62a60dab31df372dee047d28af449 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1749810088
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -67,28 +67,33 @@ public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
if (autoGenerateRecordKeys) {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2352080186
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
}, {
"hash" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488",
"triggerID" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c32178cb0da60a48c88f14071956bcc9760ac97a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457)
* 3a958f520217792f1e46d82c6ae5ba53783568b1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2352078219
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
}, {
"hash" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c32178cb0da60a48c88f14071956bcc9760ac97a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457)
* 3a958f520217792f1e46d82c6ae5ba53783568b1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2356109467
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
}, {
"hash" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488",
"triggerID" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"triggerType" : "PUSH"
}, {
"hash" : "7db628f48fede3c0b6e550d78c7be24d714b3bed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=514",
"triggerID" : "7db628f48fede3c0b6e550d78c7be24d714b3bed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7db628f48fede3c0b6e550d78c7be24d714b3bed Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=514)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1749809476
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -212,7 +211,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
} else {
val (recordName, namespace) = AvroConversionUtils.getAvroRecordNameAndNamespace(table.identifier.table)
val schema = SchemaConverters.toAvroType(dataSchema, nullable = false, recordName, namespace)
- val partitionColumns = if (table.partitionColumnNames.isEmpty) {
+ val partitionColumns = if (tableConfigs.contains(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())) {
Review Comment:
This is while creation of hudi table. So I have used the config used for setting partition fields.
I dont think we suggest using HoodieTableConfig.PARTITION_FIELDS while creation of new table?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/keygen/TestKeyGenUtils.java:
##########
@@ -45,12 +46,68 @@ public void testInferKeyGeneratorType() {
assertEquals(
KeyGeneratorType.COMPLEX,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1,partition2"));
+
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1"), "partition1:simple,partition2:timestamp"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1:simple"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1:simple,partition2:timestamp"));
+
assertEquals(
KeyGeneratorType.NON_PARTITION,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), ""));
assertEquals(
KeyGeneratorType.NON_PARTITION,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), null));
+
+ // Test key generator type with auto generation of record keys
+ assertEquals(
+ KeyGeneratorType.SIMPLE,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1"));
+ assertEquals(
+ KeyGeneratorType.COMPLEX,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1,partition2"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1:simple"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1:simple,partition2:timestamp"));
+ assertEquals(
+ KeyGeneratorType.NON_PARTITION,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), ""));
+ assertEquals(
+ KeyGeneratorType.NON_PARTITION,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), null));
+ }
+
+ @Test
+ public void testInferKeyGeneratorTypeFromPartitionFields() {
+ assertEquals(
+ KeyGeneratorType.SIMPLE,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1"));
+ assertEquals(
+ KeyGeneratorType.COMPLEX,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1,partition1"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1:simple"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1:timestamp"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1:simple,partition1:timestamp"));
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1751100197
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -66,29 +66,25 @@ public class KeyGenUtils {
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
- } else {
- if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
- }
- return KeyGeneratorType.COMPLEX;
- }
- return KeyGeneratorType.NON_PARTITION;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
Review Comment:
Why is `|| partitionKeyGeneratorType != KeyGeneratorType.SIMPLE` added?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2343122567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ad0c0707ea05e3214758188084148bd921a987c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348)
* 2c6812969a3334e7950f3b08ddbd22ac61c21c89 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1747504103
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -212,7 +211,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
} else {
val (recordName, namespace) = AvroConversionUtils.getAvroRecordNameAndNamespace(table.identifier.table)
val schema = SchemaConverters.toAvroType(dataSchema, nullable = false, recordName, namespace)
- val partitionColumns = if (table.partitionColumnNames.isEmpty) {
+ val partitionColumns = if (tableConfigs.contains(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())) {
Review Comment:
Why this instead of HoodieTableConfig.PARTITION_FIELDS?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,96 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id,name',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
+ | 'hoodie.insert.shuffle.parallelism' = '1',
+ | 'hoodie.upsert.shuffle.parallelism' = '1',
+ | 'hoodie.bulkinsert.shuffle.parallelism' = '1',
+ | 'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
+ | 'hoodie.keygen.timebased.output.dateformat' = '$dateFormat',
+ | 'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds'
+ |)
+ location '${tablePath}'
+ """.stripMargin)
+
+ testInserts(tableName, tsGenFunc, customPartitionFunc)
+ assertEquals(7, spark.sql(
Review Comment:
do we need this? Doesn't validate results ensure this already?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/keygen/TestKeyGenUtils.java:
##########
@@ -45,12 +46,68 @@ public void testInferKeyGeneratorType() {
assertEquals(
KeyGeneratorType.COMPLEX,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1,partition2"));
+
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1"), "partition1:simple,partition2:timestamp"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1:simple"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1:simple,partition2:timestamp"));
+
assertEquals(
KeyGeneratorType.NON_PARTITION,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), ""));
assertEquals(
KeyGeneratorType.NON_PARTITION,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), null));
+
+ // Test key generator type with auto generation of record keys
+ assertEquals(
+ KeyGeneratorType.SIMPLE,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1"));
+ assertEquals(
+ KeyGeneratorType.COMPLEX,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1,partition2"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1:simple"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1:simple,partition2:timestamp"));
+ assertEquals(
+ KeyGeneratorType.NON_PARTITION,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), ""));
+ assertEquals(
+ KeyGeneratorType.NON_PARTITION,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), null));
+ }
+
+ @Test
+ public void testInferKeyGeneratorTypeFromPartitionFields() {
+ assertEquals(
+ KeyGeneratorType.SIMPLE,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1"));
+ assertEquals(
+ KeyGeneratorType.COMPLEX,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1,partition1"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1:simple"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1:timestamp"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1:simple,partition1:timestamp"));
Review Comment:
partition1:simple,partition2:timestamp
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -67,28 +67,33 @@ public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
if (autoGenerateRecordKeys) {
Review Comment:
KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
return partitionKeyGeneratorType;
}
int numRecordKeyFields = recordsKeyFields.get().split(",").length;
return numRecordKeyFields == 1 ? KeyGeneratorType.SIMPLE : KeyGeneratorType.COMPLEX;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1750619654
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -212,7 +211,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
} else {
val (recordName, namespace) = AvroConversionUtils.getAvroRecordNameAndNamespace(table.identifier.table)
val schema = SchemaConverters.toAvroType(dataSchema, nullable = false, recordName, namespace)
- val partitionColumns = if (table.partitionColumnNames.isEmpty) {
+ val partitionColumns = if (tableConfigs.contains(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())) {
Review Comment:
thanks, that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2343363062
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c6812969a3334e7950f3b08ddbd22ac61c21c89 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385)
* 8b392cacf8db8b88fb9c78326d241e3cc44a64a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2348461238
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14f7227cf4fd12d6ebf47c9d1f2063526bd3e005 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453)
* c32178cb0da60a48c88f14071956bcc9760ac97a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2348322706
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7f310bb6edd6cfe77163fdea69434bf1599d8d6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388)
* 14f7227cf4fd12d6ebf47c9d1f2063526bd3e005 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2305161072
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 512cdd632574e18a05585ae61cfe8c9171aebaff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2337618237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ad0c0707ea05e3214758188084148bd921a987c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1751101788
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -66,29 +66,25 @@ public class KeyGenUtils {
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
- } else {
- if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
- }
- return KeyGeneratorType.COMPLEX;
- }
- return KeyGeneratorType.NON_PARTITION;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
+ return partitionKeyGeneratorType;
}
+ int numRecordKeyFields = recordsKeyFields.get().split(",").length;
+ return numRecordKeyFields == 1 ? KeyGeneratorType.SIMPLE : KeyGeneratorType.COMPLEX;
}
// When auto record key gen is enabled, our inference will be based on partition path only.
- private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partitionFields) {
+ static KeyGeneratorType inferKeyGeneratorTypeFromPartitionFields(String partitionFields) {
if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- if (numPartFields == 1) {
+ String[] partitonFields = partitionFields.split(",");
+ if (partitonFields[0].contains(BaseKeyGenerator.CUSTOM_KEY_GENERATOR_SPLIT_REGEX)) {
+ return KeyGeneratorType.CUSTOM;
Review Comment:
What about other engines? Do they have similar infer logic that needs to be fixed?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/keygen/TestKeyGenUtils.java:
##########
@@ -45,12 +46,68 @@ public void testInferKeyGeneratorType() {
assertEquals(
KeyGeneratorType.COMPLEX,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1,partition2"));
+
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1"), "partition1:simple,partition2:timestamp"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1:simple"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), "partition1:simple,partition2:timestamp"));
+
assertEquals(
KeyGeneratorType.NON_PARTITION,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), ""));
assertEquals(
KeyGeneratorType.NON_PARTITION,
KeyGenUtils.inferKeyGeneratorType(Option.of("col1,col2"), null));
+
+ // Test key generator type with auto generation of record keys
+ assertEquals(
+ KeyGeneratorType.SIMPLE,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1"));
+ assertEquals(
+ KeyGeneratorType.COMPLEX,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1,partition2"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1:simple"));
+ assertEquals(
+ KeyGeneratorType.CUSTOM,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), "partition1:simple,partition2:timestamp"));
+ assertEquals(
+ KeyGeneratorType.NON_PARTITION,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), ""));
+ assertEquals(
+ KeyGeneratorType.NON_PARTITION,
+ KeyGenUtils.inferKeyGeneratorType(Option.empty(), null));
+ }
+
+ @Test
+ public void testInferKeyGeneratorTypeFromPartitionFields() {
+ assertEquals(
+ KeyGeneratorType.SIMPLE,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1"));
+ assertEquals(
+ KeyGeneratorType.COMPLEX,
+ KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1,partition1"));
Review Comment:
```suggestion
KeyGenUtils.inferKeyGeneratorTypeFromPartitionFields("partition1,partition2"));
```
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -313,7 +315,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
KeyGeneratorType.valueOf(originTableConfig(HoodieTableConfig.KEY_GENERATOR_TYPE.key)).getClassName)
} else {
val primaryKeys = table.properties.getOrElse(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName, table.storage.properties.get(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName)).toString
- val partitions = table.partitionColumnNames.mkString(",")
+ val partitionFieldsOpt = originTableConfig.get(HoodieTableConfig.PARTITION_FIELDS.key)
+ .orElse(sqlOptions.get(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key()))
Review Comment:
Avoid using `.key` and use `ConfigUtils` instead so alternative keys are considered.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,92 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id,name',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
Review Comment:
Can we write a test that do not use `hoodie.datasource.write.partitionpath.field` in `tblproperties`, by creating a table using Spark datasource with custom keygen and timestamp-based partition field, and using Spark SQL without passing in any write configs to make sure the partition path value can be correctly generated?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -66,29 +66,25 @@ public class KeyGenUtils {
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
- } else {
- if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
- }
- return KeyGeneratorType.COMPLEX;
- }
- return KeyGeneratorType.NON_PARTITION;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
Review Comment:
Should `KeyGeneratorType.COMPLEX` be included too based on the logic below.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -212,7 +211,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
} else {
val (recordName, namespace) = AvroConversionUtils.getAvroRecordNameAndNamespace(table.identifier.table)
val schema = SchemaConverters.toAvroType(dataSchema, nullable = false, recordName, namespace)
- val partitionColumns = if (table.partitionColumnNames.isEmpty) {
+ val partitionColumns = if (tableConfigs.contains(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())) {
+ tableConfigs(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())
+ } else if (table.partitionColumnNames.isEmpty) {
Review Comment:
Same here on avoiding `.key` calls
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2333321733
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e15f2e5209875fcfb6796dc94da7a095ddb99e9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1746745451
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,93 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+ val keyGenConfigs = TS_KEY_GEN_CONFIGS + ("hoodie.keygen.timebased.output.dateformat" -> dateFormat)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
+ | 'hoodie.insert.shuffle.parallelism' = '1',
+ | 'hoodie.upsert.shuffle.parallelism' = '1',
+ | 'hoodie.bulkinsert.shuffle.parallelism' = '1',
+ | 'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
+ | 'hoodie.keygen.timebased.output.dateformat' = '$dateFormat',
+ | 'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds'
+ |)
+ location '${tablePath}'
+ """.stripMargin)
+
+ testInserts(tableName, tsGenFunc, customPartitionFunc)
+ assertEquals(7, spark.sql(
+ s"""
+ | SELECT * from $tableName
+ | """.stripMargin).count())
+
+ // Validate ts field is still of type int in the table
+ validateTsFieldSchema(tablePath)
+ }
+ }
+ }
+
private def validateTsFieldSchema(tablePath: String, fieldName: String, expectedType: Schema.Type): Unit = {
val metaClient = createMetaClient(spark, tablePath)
val schemaResolver = new TableSchemaResolver(metaClient)
val nullableSchema = Schema.createUnion(Schema.create(Schema.Type.NULL), Schema.create(expectedType))
assertEquals(nullableSchema, schemaResolver.getTableAvroSchema(true).getField(fieldName).schema())
}
+ private def testInserts(tableName: String,
+ tsGenFunc: Integer => String,
+ partitionGenFunc: (Integer, String) => String): Unit = {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1753774436
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -66,29 +66,25 @@ public class KeyGenUtils {
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
- } else {
- if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
- }
- return KeyGeneratorType.COMPLEX;
- }
- return KeyGeneratorType.NON_PARTITION;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
+ return partitionKeyGeneratorType;
}
+ int numRecordKeyFields = recordsKeyFields.get().split(",").length;
+ return numRecordKeyFields == 1 ? KeyGeneratorType.SIMPLE : KeyGeneratorType.COMPLEX;
}
// When auto record key gen is enabled, our inference will be based on partition path only.
- private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partitionFields) {
+ static KeyGeneratorType inferKeyGeneratorTypeFromPartitionFields(String partitionFields) {
if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- if (numPartFields == 1) {
+ String[] partitonFields = partitionFields.split(",");
+ if (partitonFields[0].contains(BaseKeyGenerator.CUSTOM_KEY_GENERATOR_SPLIT_REGEX)) {
+ return KeyGeneratorType.CUSTOM;
Review Comment:
I did not find similar logic for other engines. It seems like the same logic is reused based on `org.apache.hudi.DataSourceWriteOptions#KEYGENERATOR_CLASS_NAME`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1764083620
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -65,30 +65,35 @@ public class KeyGenUtils {
*/
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
- boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
+ int numRecordKeyFields = recordsKeyFields.map(fields -> fields.split(",").length).orElse(0);
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
Review Comment:
I think a better approach is to take both `recordsKeyFields` and `partitionFields` and infer the keygen type based on both `numRecordKeyFields` and `numPartitionFields` so it's more clear. @lokeshj1703 You can take this in a follow-up PR and fix all places where `autoGenerateRecordKeys` is checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1758405249
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -313,7 +315,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
KeyGeneratorType.valueOf(originTableConfig(HoodieTableConfig.KEY_GENERATOR_TYPE.key)).getClassName)
} else {
val primaryKeys = table.properties.getOrElse(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName, table.storage.properties.get(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName)).toString
- val partitions = table.partitionColumnNames.mkString(",")
+ val partitionFieldsOpt = Option.apply(ConfigUtils.getStringWithAltKeys(originTableConfig.asJava.asInstanceOf[java.util.Map[String, Object]], HoodieTableConfig.PARTITION_FIELDS))
Review Comment:
I would need to add a scala config utils class for this which would lead to code duplication. These seem to be wrappers when converted. They are only delegating the call to scala data structure on Java calls.
https://www.scala-lang.org/api/current/scala/collection/JavaConverters$.html
https://stackoverflow.com/questions/8301947/what-is-the-difference-between-javaconverters-and-javaconversions-in-scala
Please let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2333251871
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2bf496e1ece338256f0bc116dd9936af68ebc138 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156)
* 5e15f2e5209875fcfb6796dc94da7a095ddb99e9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1746745167
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,93 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+ val keyGenConfigs = TS_KEY_GEN_CONFIGS + ("hoodie.keygen.timebased.output.dateformat" -> dateFormat)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id',
Review Comment:
I have updated the test to use primary key with two columns. Also added the validation for table config.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -67,28 +67,33 @@ public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
+ return inferKeyGeneratorTypeFromPartitionFields(partitionFields);
} else {
if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
+ KeyGeneratorType recordKeyGeneratorType = numRecordKeyFields == 1 ? KeyGeneratorType.SIMPLE : KeyGeneratorType.COMPLEX;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (partitionKeyGeneratorType == KeyGeneratorType.SIMPLE && recordKeyGeneratorType == KeyGeneratorType.COMPLEX) {
+ return recordKeyGeneratorType;
+ } else {
+ return partitionKeyGeneratorType;
}
- return KeyGeneratorType.COMPLEX;
}
return KeyGeneratorType.NON_PARTITION;
}
}
// When auto record key gen is enabled, our inference will be based on partition path only.
- private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partitionFields) {
+ private static KeyGeneratorType inferKeyGeneratorTypeFromPartitionFields(String partitionFields) {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1774610413
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -65,30 +65,35 @@ public class KeyGenUtils {
*/
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
- boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
+ int numRecordKeyFields = recordsKeyFields.map(fields -> fields.split(",").length).orElse(0);
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
Review Comment:
@yihua I have added tests for SparkConfigUtils in https://github.com/apache/hudi/pull/12000. PTAL.
I find the existing logic ok for inferring the key generator type. For partition type it is not just numPartitionFields, it is also the custom keygen which is the case where fields have `:` separator. LMK if you think there is still some improvement possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1753520257
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -66,29 +66,25 @@ public class KeyGenUtils {
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
- } else {
- if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
- }
- return KeyGeneratorType.COMPLEX;
- }
- return KeyGeneratorType.NON_PARTITION;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
Review Comment:
KeyGeneratorType is SIMPLE only if both partition fields and record key fields have one field configured. For that reason we need additional check to know how many record key fields are present apart from partition fields.
For other partition types, we can infer based on partition type alone. This is aligned with older logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2343530821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b392cacf8db8b88fb9c78326d241e3cc44a64a0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387)
* d7f310bb6edd6cfe77163fdea69434bf1599d8d6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2306815922
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 512cdd632574e18a05585ae61cfe8c9171aebaff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142)
* 2bf496e1ece338256f0bc116dd9936af68ebc138 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2305296314
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 512cdd632574e18a05585ae61cfe8c9171aebaff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1754328160
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,92 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id,name',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
Review Comment:
Added tests and corresponding fix for this use case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2343683179
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7f310bb6edd6cfe77163fdea69434bf1599d8d6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2348325873
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7f310bb6edd6cfe77163fdea69434bf1599d8d6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388)
* 14f7227cf4fd12d6ebf47c9d1f2063526bd3e005 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1764078520
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -212,7 +211,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
} else {
val (recordName, namespace) = AvroConversionUtils.getAvroRecordNameAndNamespace(table.identifier.table)
val schema = SchemaConverters.toAvroType(dataSchema, nullable = false, recordName, namespace)
- val partitionColumns = if (table.partitionColumnNames.isEmpty) {
+ val partitionColumns = if (tableConfigs.contains(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())) {
+ ConfigUtils.getStringWithAltKeys(tableConfigs.asJava.asInstanceOf[java.util.Map[String, Object]], KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME)
Review Comment:
Synced offline. We can extract a common config util in Scala and reuse it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1774610413
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -65,30 +65,35 @@ public class KeyGenUtils {
*/
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
- boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
+ int numRecordKeyFields = recordsKeyFields.map(fields -> fields.split(",").length).orElse(0);
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
Review Comment:
@yihua I have added tests for SparkConfigUtils in https://github.com/apache/hudi/pull/12000. PTAL.
I find the existing logic ok for inferring the key generator type. For partition type it is not just numPartitionFields, it is also the custom keygen which is the case where fields have `:` separator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2333573699
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e15f2e5209875fcfb6796dc94da7a095ddb99e9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296)
* 22ea4eca27b62a60dab31df372dee047d28af449 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2343365416
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c6812969a3334e7950f3b08ddbd22ac61c21c89 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385)
* 8b392cacf8db8b88fb9c78326d241e3cc44a64a0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #11816:
URL: https://github.com/apache/hudi/pull/11816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2305158497
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 512cdd632574e18a05585ae61cfe8c9171aebaff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2306818331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 512cdd632574e18a05585ae61cfe8c9171aebaff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142)
* 2bf496e1ece338256f0bc116dd9936af68ebc138 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2306940635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2bf496e1ece338256f0bc116dd9936af68ebc138 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2343280829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c6812969a3334e7950f3b08ddbd22ac61c21c89 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2348598856
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c32178cb0da60a48c88f14071956bcc9760ac97a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2355828214
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
}, {
"hash" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488",
"triggerID" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"triggerType" : "PUSH"
}, {
"hash" : "7db628f48fede3c0b6e550d78c7be24d714b3bed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=514",
"triggerID" : "7db628f48fede3c0b6e550d78c7be24d714b3bed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a958f520217792f1e46d82c6ae5ba53783568b1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488)
* 7db628f48fede3c0b6e550d78c7be24d714b3bed Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=514)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2355823330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
}, {
"hash" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488",
"triggerID" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"triggerType" : "PUSH"
}, {
"hash" : "7db628f48fede3c0b6e550d78c7be24d714b3bed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7db628f48fede3c0b6e550d78c7be24d714b3bed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a958f520217792f1e46d82c6ae5ba53783568b1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488)
* 7db628f48fede3c0b6e550d78c7be24d714b3bed UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2352180102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=457",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
}, {
"hash" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488",
"triggerID" : "3a958f520217792f1e46d82c6ae5ba53783568b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a958f520217792f1e46d82c6ae5ba53783568b1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=488)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2343527996
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c6812969a3334e7950f3b08ddbd22ac61c21c89 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385)
* 8b392cacf8db8b88fb9c78326d241e3cc44a64a0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387)
* d7f310bb6edd6cfe77163fdea69434bf1599d8d6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1758304832
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,92 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id,name',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
Review Comment:
Those changes are addressed in https://github.com/apache/hudi/pull/11926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1758408388
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -66,29 +66,25 @@ public class KeyGenUtils {
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
- } else {
- if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
- }
- return KeyGeneratorType.COMPLEX;
- }
- return KeyGeneratorType.NON_PARTITION;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
Review Comment:
Addressed. I have tried to simplify the logic further. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1758450295
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -313,7 +315,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
KeyGeneratorType.valueOf(originTableConfig(HoodieTableConfig.KEY_GENERATOR_TYPE.key)).getClassName)
} else {
val primaryKeys = table.properties.getOrElse(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName, table.storage.properties.get(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName)).toString
- val partitions = table.partitionColumnNames.mkString(",")
+ val partitionFieldsOpt = Option.apply(ConfigUtils.getStringWithAltKeys(originTableConfig.asJava.asInstanceOf[java.util.Map[String, Object]], HoodieTableConfig.PARTITION_FIELDS))
Review Comment:
I made an attempt to avoid it. I created a new function in ConfigUtils which uses a key mapping function and used that instead in Scala.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -212,7 +211,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
} else {
val (recordName, namespace) = AvroConversionUtils.getAvroRecordNameAndNamespace(table.identifier.table)
val schema = SchemaConverters.toAvroType(dataSchema, nullable = false, recordName, namespace)
- val partitionColumns = if (table.partitionColumnNames.isEmpty) {
+ val partitionColumns = if (tableConfigs.contains(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())) {
+ ConfigUtils.getStringWithAltKeys(tableConfigs.asJava.asInstanceOf[java.util.Map[String, Object]], KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME)
Review Comment:
I made an attempt to avoid it. I created a new function in ConfigUtils which uses a key mapping function and used that instead in Scala.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2348387379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14f7227cf4fd12d6ebf47c9d1f2063526bd3e005 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453)
* c32178cb0da60a48c88f14071956bcc9760ac97a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1758539608
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -313,7 +315,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
KeyGeneratorType.valueOf(originTableConfig(HoodieTableConfig.KEY_GENERATOR_TYPE.key)).getClassName)
} else {
val primaryKeys = table.properties.getOrElse(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName, table.storage.properties.get(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName)).toString
- val partitions = table.partitionColumnNames.mkString(",")
+ val partitionFieldsOpt = Option.apply(ConfigUtils.getStringWithAltKeys(originTableConfig.asJava.asInstanceOf[java.util.Map[String, Object]], HoodieTableConfig.PARTITION_FIELDS))
Review Comment:
I prefer the older approach. LMK, I can revert the last commit if you feel the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2333252918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2bf496e1ece338256f0bc116dd9936af68ebc138 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156)
* 5e15f2e5209875fcfb6796dc94da7a095ddb99e9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1746539447
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,93 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+ val keyGenConfigs = TS_KEY_GEN_CONFIGS + ("hoodie.keygen.timebased.output.dateformat" -> dateFormat)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id',
Review Comment:
let's add a test to cover more than one primary keys. And also get the metaClient in the test to assert the table config correctly sets the keygen type.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -67,28 +67,33 @@ public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
+ return inferKeyGeneratorTypeFromPartitionFields(partitionFields);
} else {
if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
+ KeyGeneratorType recordKeyGeneratorType = numRecordKeyFields == 1 ? KeyGeneratorType.SIMPLE : KeyGeneratorType.COMPLEX;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (partitionKeyGeneratorType == KeyGeneratorType.SIMPLE && recordKeyGeneratorType == KeyGeneratorType.COMPLEX) {
+ return recordKeyGeneratorType;
+ } else {
+ return partitionKeyGeneratorType;
}
- return KeyGeneratorType.COMPLEX;
}
return KeyGeneratorType.NON_PARTITION;
}
}
// When auto record key gen is enabled, our inference will be based on partition path only.
- private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partitionFields) {
+ private static KeyGeneratorType inferKeyGeneratorTypeFromPartitionFields(String partitionFields) {
Review Comment:
Let's add unit tests for the static methods in this class. Tests should cover all different keygen types.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,93 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+ val keyGenConfigs = TS_KEY_GEN_CONFIGS + ("hoodie.keygen.timebased.output.dateformat" -> dateFormat)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
+ | 'hoodie.insert.shuffle.parallelism' = '1',
+ | 'hoodie.upsert.shuffle.parallelism' = '1',
+ | 'hoodie.bulkinsert.shuffle.parallelism' = '1',
+ | 'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
+ | 'hoodie.keygen.timebased.output.dateformat' = '$dateFormat',
+ | 'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds'
+ |)
+ location '${tablePath}'
+ """.stripMargin)
+
+ testInserts(tableName, tsGenFunc, customPartitionFunc)
+ assertEquals(7, spark.sql(
+ s"""
+ | SELECT * from $tableName
+ | """.stripMargin).count())
+
+ // Validate ts field is still of type int in the table
+ validateTsFieldSchema(tablePath)
+ }
+ }
+ }
+
private def validateTsFieldSchema(tablePath: String, fieldName: String, expectedType: Schema.Type): Unit = {
val metaClient = createMetaClient(spark, tablePath)
val schemaResolver = new TableSchemaResolver(metaClient)
val nullableSchema = Schema.createUnion(Schema.create(Schema.Type.NULL), Schema.create(expectedType))
assertEquals(nullableSchema, schemaResolver.getTableAvroSchema(true).getField(fieldName).schema())
}
+ private def testInserts(tableName: String,
+ tsGenFunc: Integer => String,
+ partitionGenFunc: (Integer, String) => String): Unit = {
Review Comment:
fix indentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2333576413
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e15f2e5209875fcfb6796dc94da7a095ddb99e9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296)
* 22ea4eca27b62a60dab31df372dee047d28af449 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1749808868
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,96 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id,name',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
+ | 'hoodie.insert.shuffle.parallelism' = '1',
+ | 'hoodie.upsert.shuffle.parallelism' = '1',
+ | 'hoodie.bulkinsert.shuffle.parallelism' = '1',
+ | 'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
+ | 'hoodie.keygen.timebased.output.dateformat' = '$dateFormat',
+ | 'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds'
+ |)
+ location '${tablePath}'
+ """.stripMargin)
+
+ testInserts(tableName, tsGenFunc, customPartitionFunc)
+ assertEquals(7, spark.sql(
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2337469003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22ea4eca27b62a60dab31df372dee047d28af449 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297)
* 7ad0c0707ea05e3214758188084148bd921a987c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2337466463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22ea4eca27b62a60dab31df372dee047d28af449 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297)
* 7ad0c0707ea05e3214758188084148bd921a987c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1757633288
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -313,7 +315,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
KeyGeneratorType.valueOf(originTableConfig(HoodieTableConfig.KEY_GENERATOR_TYPE.key)).getClassName)
} else {
val primaryKeys = table.properties.getOrElse(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName, table.storage.properties.get(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName)).toString
- val partitions = table.partitionColumnNames.mkString(",")
+ val partitionFieldsOpt = Option.apply(ConfigUtils.getStringWithAltKeys(originTableConfig.asJava.asInstanceOf[java.util.Map[String, Object]], HoodieTableConfig.PARTITION_FIELDS))
Review Comment:
`originTableConfig.asJava.asInstanceOf[java.util.Map[String, Object]` may incur non-trivial overhead. Could you add a new util method that can be used on Scala Map?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -212,7 +211,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
} else {
val (recordName, namespace) = AvroConversionUtils.getAvroRecordNameAndNamespace(table.identifier.table)
val schema = SchemaConverters.toAvroType(dataSchema, nullable = false, recordName, namespace)
- val partitionColumns = if (table.partitionColumnNames.isEmpty) {
+ val partitionColumns = if (tableConfigs.contains(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())) {
+ ConfigUtils.getStringWithAltKeys(tableConfigs.asJava.asInstanceOf[java.util.Map[String, Object]], KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME)
Review Comment:
Similar here on the conversion.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -66,29 +66,25 @@ public class KeyGenUtils {
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
- } else {
- if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
- }
- return KeyGeneratorType.COMPLEX;
- }
- return KeyGeneratorType.NON_PARTITION;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
Review Comment:
Make sense. In that case should all logic of checking record key and partition fields be put in the same method `inferKeyGeneratorTypeFromPartitionFields`? The code in the current shape is confusing. Still keep the structure like:
```
if (autoGenerateRecordKeys) {
return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
}
...
```
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -442,13 +442,92 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table with custom key generator") {
+ withTempDir { tmp => {
+ val tableName = generateTableName
+ val tablePath = tmp.getCanonicalPath + "/" + tableName
+ val writePartitionFields = "ts:timestamp"
+ val dateFormat = "yyyy/MM/dd"
+ val tsGenFunc = (ts: Integer) => TS_FORMATTER_FUNC_WITH_FORMAT.apply(ts, dateFormat)
+ val customPartitionFunc = (ts: Integer, _: String) => "ts=" + tsGenFunc.apply(ts)
+
+ spark.sql(
+ s"""
+ |create table ${tableName} (
+ | `id` INT,
+ | `name` STRING,
+ | `price` DECIMAL(5, 1),
+ | `ts` INT,
+ | `segment` STRING
+ |) using hudi
+ |tblproperties (
+ | 'primaryKey' = 'id,name',
+ | 'type' = 'mor',
+ | 'preCombineField'='name',
+ | 'hoodie.datasource.write.keygenerator.class' = '$CUSTOM_KEY_GEN_CLASS_NAME',
+ | 'hoodie.datasource.write.partitionpath.field' = '$writePartitionFields',
Review Comment:
Let's JIRA the follow up and mark it to fix 1.0.0 release.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -66,29 +66,25 @@ public class KeyGenUtils {
public static KeyGeneratorType inferKeyGeneratorType(
Option<String> recordsKeyFields, String partitionFields) {
boolean autoGenerateRecordKeys = !recordsKeyFields.isPresent();
- if (autoGenerateRecordKeys) {
- return inferKeyGeneratorTypeForAutoKeyGen(partitionFields);
- } else {
- if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- int numRecordKeyFields = recordsKeyFields.get().split(",").length;
- if (numPartFields == 1 && numRecordKeyFields == 1) {
- return KeyGeneratorType.SIMPLE;
- }
- return KeyGeneratorType.COMPLEX;
- }
- return KeyGeneratorType.NON_PARTITION;
+ KeyGeneratorType partitionKeyGeneratorType = inferKeyGeneratorTypeFromPartitionFields(partitionFields);
+ if (autoGenerateRecordKeys || partitionKeyGeneratorType != KeyGeneratorType.SIMPLE) {
+ return partitionKeyGeneratorType;
}
+ int numRecordKeyFields = recordsKeyFields.get().split(",").length;
+ return numRecordKeyFields == 1 ? KeyGeneratorType.SIMPLE : KeyGeneratorType.COMPLEX;
}
// When auto record key gen is enabled, our inference will be based on partition path only.
- private static KeyGeneratorType inferKeyGeneratorTypeForAutoKeyGen(String partitionFields) {
+ static KeyGeneratorType inferKeyGeneratorTypeFromPartitionFields(String partitionFields) {
if (!StringUtils.isNullOrEmpty(partitionFields)) {
- int numPartFields = partitionFields.split(",").length;
- if (numPartFields == 1) {
+ String[] partitonFields = partitionFields.split(",");
+ if (partitonFields[0].contains(BaseKeyGenerator.CUSTOM_KEY_GENERATOR_SPLIT_REGEX)) {
+ return KeyGeneratorType.CUSTOM;
Review Comment:
I created HUDI-7613 for you to validate write/query with CustomKeyGenerator on Flink and Hive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2348384345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
}, {
"hash" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=387",
"triggerID" : "8b392cacf8db8b88fb9c78326d241e3cc44a64a0",
"triggerType" : "PUSH"
}, {
"hash" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388",
"triggerID" : "d7f310bb6edd6cfe77163fdea69434bf1599d8d6",
"triggerType" : "PUSH"
}, {
"hash" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453",
"triggerID" : "14f7227cf4fd12d6ebf47c9d1f2063526bd3e005",
"triggerType" : "PUSH"
}, {
"hash" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c32178cb0da60a48c88f14071956bcc9760ac97a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7f310bb6edd6cfe77163fdea69434bf1599d8d6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=388)
* 14f7227cf4fd12d6ebf47c9d1f2063526bd3e005 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=453)
* c32178cb0da60a48c88f14071956bcc9760ac97a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1754117966
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -212,7 +211,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
} else {
val (recordName, namespace) = AvroConversionUtils.getAvroRecordNameAndNamespace(table.identifier.table)
val schema = SchemaConverters.toAvroType(dataSchema, nullable = false, recordName, namespace)
- val partitionColumns = if (table.partitionColumnNames.isEmpty) {
+ val partitionColumns = if (tableConfigs.contains(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())) {
+ tableConfigs(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key())
+ } else if (table.partitionColumnNames.isEmpty) {
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -313,7 +315,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
KeyGeneratorType.valueOf(originTableConfig(HoodieTableConfig.KEY_GENERATOR_TYPE.key)).getClassName)
} else {
val primaryKeys = table.properties.getOrElse(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName, table.storage.properties.get(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName)).toString
- val partitions = table.partitionColumnNames.mkString(",")
+ val partitionFieldsOpt = originTableConfig.get(HoodieTableConfig.PARTITION_FIELDS.key)
+ .orElse(sqlOptions.get(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key()))
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11816:
URL: https://github.com/apache/hudi/pull/11816#issuecomment-2343125313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=142",
"triggerID" : "512cdd632574e18a05585ae61cfe8c9171aebaff",
"triggerType" : "PUSH"
}, {
"hash" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=156",
"triggerID" : "2bf496e1ece338256f0bc116dd9936af68ebc138",
"triggerType" : "PUSH"
}, {
"hash" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=296",
"triggerID" : "5e15f2e5209875fcfb6796dc94da7a095ddb99e9",
"triggerType" : "PUSH"
}, {
"hash" : "22ea4eca27b62a60dab31df372dee047d28af449",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=297",
"triggerID" : "22ea4eca27b62a60dab31df372dee047d28af449",
"triggerType" : "PUSH"
}, {
"hash" : "7ad0c0707ea05e3214758188084148bd921a987c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348",
"triggerID" : "7ad0c0707ea05e3214758188084148bd921a987c",
"triggerType" : "PUSH"
}, {
"hash" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385",
"triggerID" : "2c6812969a3334e7950f3b08ddbd22ac61c21c89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ad0c0707ea05e3214758188084148bd921a987c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=348)
* 2c6812969a3334e7950f3b08ddbd22ac61c21c89 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=385)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8034] Support custom key generator with HoodieCatalogTable [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11816:
URL: https://github.com/apache/hudi/pull/11816#discussion_r1758539608
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala:
##########
@@ -313,7 +315,9 @@ class HoodieCatalogTable(val spark: SparkSession, var table: CatalogTable) exten
KeyGeneratorType.valueOf(originTableConfig(HoodieTableConfig.KEY_GENERATOR_TYPE.key)).getClassName)
} else {
val primaryKeys = table.properties.getOrElse(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName, table.storage.properties.get(SQL_KEY_TABLE_PRIMARY_KEY.sqlKeyName)).toString
- val partitions = table.partitionColumnNames.mkString(",")
+ val partitionFieldsOpt = Option.apply(ConfigUtils.getStringWithAltKeys(originTableConfig.asJava.asInstanceOf[java.util.Map[String, Object]], HoodieTableConfig.PARTITION_FIELDS))
Review Comment:
I think I prefer the older approach. LMK, I can revert the last commit if you feel the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org