You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2025/03/07 04:14:23 UTC
[PR] [HUDI-9132] Avoid empty string row key to avoid delete failure [hudi]
linliu-code opened a new pull request, #12929:
URL: https://github.com/apache/hudi/pull/12929
### Change Logs
Empty row key string could fail the initialization of SimpleKeyGenerator during its validation.
### Impact
Fixes a corner case.
### Risk level (write none, low medium or high below)
Low.
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2709149529
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4092",
"triggerID" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"triggerType" : "PUSH"
}, {
"hash" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109",
"triggerID" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"triggerType" : "PUSH"
}, {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109)
* c20f1782305eea9c97fda955e159bd564e3d75ff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12929:
URL: https://github.com/apache/hudi/pull/12929#discussion_r1988032109
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -80,18 +80,7 @@ trait ProvidesHoodieConfig extends Logging {
HiveSyncConfigHolder.HIVE_SUPPORT_TIMESTAMP_TYPE.key -> hiveSyncConfig.getBoolean(HiveSyncConfigHolder.HIVE_SUPPORT_TIMESTAMP_TYPE).toString
)
- val overridingOpts = Map[String, String](
- "path" -> hoodieCatalogTable.tableLocation,
- RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
- DATABASE_NAME.key -> hoodieCatalogTable.table.database,
Review Comment:
This is missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2711928859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
}, {
"hash" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"triggerType" : "PUSH"
}, {
"hash" : "e0db605d7821385d6517131fc27edc78924f121f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4138",
"triggerID" : "e0db605d7821385d6517131fc27edc78924f121f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20f1782305eea9c97fda955e159bd564e3d75ff Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119)
* 28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4 UNKNOWN
* e0db605d7821385d6517131fc27edc78924f121f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4138)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #12929:
URL: https://github.com/apache/hudi/pull/12929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2708464042
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4092",
"triggerID" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"triggerType" : "PUSH"
}, {
"hash" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109",
"triggerID" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54b876b645fcd5a0df6277cd3934f51dc3b32616 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4092)
* 2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2711908594
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
}, {
"hash" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"triggerType" : "PUSH"
}, {
"hash" : "e0db605d7821385d6517131fc27edc78924f121f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e0db605d7821385d6517131fc27edc78924f121f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20f1782305eea9c97fda955e159bd564e3d75ff Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119)
* 28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4 UNKNOWN
* e0db605d7821385d6517131fc27edc78924f121f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete operation on partitioned table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12929:
URL: https://github.com/apache/hudi/pull/12929#discussion_r1985807178
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -424,20 +424,28 @@ trait ProvidesHoodieConfig extends Logging {
HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS.key -> hiveSyncConfig.getStringOrDefault(HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS)
)
+ val overridingOpts = buildOverridingOpts(hoodieCatalogTable, tableConfig, path)
+ combineOptions(hoodieCatalogTable, tableConfig, sparkSession.sqlContext.conf,
+ defaultOpts = defaultOpts, overridingOpts = overridingOpts)
+ }
+
+ def buildOverridingOpts(hoodieCatalogTable: HoodieCatalogTable,
+ tableConfig: HoodieTableConfig,
+ path: String): Map[String, String] = {
val overridingOpts = Map(
"path" -> path,
- RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
TBL_NAME.key -> tableConfig.getTableName,
- DATABASE_NAME.key -> hoodieCatalogTable.table.database,
HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL,
PARTITIONPATH_FIELD.key -> getPartitionPathFieldWriteConfig(
tableConfig.getKeyGeneratorClassName, tableConfig.getPartitionFieldProp, hoodieCatalogTable)
)
-
- combineOptions(hoodieCatalogTable, tableConfig, sparkSession.sqlContext.conf,
- defaultOpts = defaultOpts, overridingOpts = overridingOpts)
+ if (hoodieCatalogTable.primaryKeys.isEmpty) {
Review Comment:
did we check if Update command and Insert command is good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2711843808
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20f1782305eea9c97fda955e159bd564e3d75ff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2708501815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4092",
"triggerID" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"triggerType" : "PUSH"
}, {
"hash" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109",
"triggerID" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2709211651
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4092",
"triggerID" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"triggerType" : "PUSH"
}, {
"hash" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109",
"triggerID" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"triggerType" : "PUSH"
}, {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20f1782305eea9c97fda955e159bd564e3d75ff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12929:
URL: https://github.com/apache/hudi/pull/12929#discussion_r1988056634
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -80,18 +80,7 @@ trait ProvidesHoodieConfig extends Logging {
HiveSyncConfigHolder.HIVE_SUPPORT_TIMESTAMP_TYPE.key -> hiveSyncConfig.getBoolean(HiveSyncConfigHolder.HIVE_SUPPORT_TIMESTAMP_TYPE).toString
)
- val overridingOpts = Map[String, String](
- "path" -> hoodieCatalogTable.tableLocation,
- RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
- DATABASE_NAME.key -> hoodieCatalogTable.table.database,
Review Comment:
fixed.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -556,4 +533,35 @@ object ProvidesHoodieConfig {
private def filterNullValues(opts: Map[String, String]): Map[String, String] =
opts.filter { case (_, v) => v != null }
+ private def buildOverridingOpts(hoodieCatalogTable: HoodieCatalogTable,
+ preCombineField: String): Map[String, String] = {
+ buildCommonOverridingOpts(hoodieCatalogTable) ++ Map(
+ PRECOMBINE_FIELD.key -> preCombineField
+ )
+ }
+
+ private def buildOverridingOptsForDelete(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ buildCommonOverridingOpts(hoodieCatalogTable) ++ Map(
+ OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL
+ )
+ }
+
+ private def buildCommonOverridingOpts(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val baseOpts = Map(
+ "path" -> hoodieCatalogTable.tableLocation,
+ TABLE_NAME.key -> tableConfig.getTableName,
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12929:
URL: https://github.com/apache/hudi/pull/12929#discussion_r1985914401
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -424,20 +424,28 @@ trait ProvidesHoodieConfig extends Logging {
HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS.key -> hiveSyncConfig.getStringOrDefault(HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS)
)
+ val overridingOpts = buildOverridingOpts(hoodieCatalogTable, tableConfig, path)
+ combineOptions(hoodieCatalogTable, tableConfig, sparkSession.sqlContext.conf,
+ defaultOpts = defaultOpts, overridingOpts = overridingOpts)
+ }
+
+ def buildOverridingOpts(hoodieCatalogTable: HoodieCatalogTable,
+ tableConfig: HoodieTableConfig,
+ path: String): Map[String, String] = {
val overridingOpts = Map(
"path" -> path,
- RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
TBL_NAME.key -> tableConfig.getTableName,
- DATABASE_NAME.key -> hoodieCatalogTable.table.database,
HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL,
PARTITIONPATH_FIELD.key -> getPartitionPathFieldWriteConfig(
tableConfig.getKeyGeneratorClassName, tableConfig.getPartitionFieldProp, hoodieCatalogTable)
)
-
- combineOptions(hoodieCatalogTable, tableConfig, sparkSession.sqlContext.conf,
- defaultOpts = defaultOpts, overridingOpts = overridingOpts)
+ if (hoodieCatalogTable.primaryKeys.isEmpty) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2708463555
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4092",
"triggerID" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"triggerType" : "PUSH"
}, {
"hash" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54b876b645fcd5a0df6277cd3934f51dc3b32616 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4092)
* 2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12929:
URL: https://github.com/apache/hudi/pull/12929#discussion_r1988053182
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -556,4 +533,35 @@ object ProvidesHoodieConfig {
private def filterNullValues(opts: Map[String, String]): Map[String, String] =
opts.filter { case (_, v) => v != null }
+ private def buildOverridingOpts(hoodieCatalogTable: HoodieCatalogTable,
+ preCombineField: String): Map[String, String] = {
+ buildCommonOverridingOpts(hoodieCatalogTable) ++ Map(
+ PRECOMBINE_FIELD.key -> preCombineField
+ )
+ }
+
+ private def buildOverridingOptsForDelete(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ buildCommonOverridingOpts(hoodieCatalogTable) ++ Map(
+ OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL
+ )
+ }
+
+ private def buildCommonOverridingOpts(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val baseOpts = Map(
+ "path" -> hoodieCatalogTable.tableLocation,
+ TABLE_NAME.key -> tableConfig.getTableName,
Review Comment:
```suggestion
TBL_NAME.key -> tableConfig.getTableName,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2711899187
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
}, {
"hash" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20f1782305eea9c97fda955e159bd564e3d75ff Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119)
* 28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2712048691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
}, {
"hash" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"triggerType" : "PUSH"
}, {
"hash" : "e0db605d7821385d6517131fc27edc78924f121f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4138",
"triggerID" : "e0db605d7821385d6517131fc27edc78924f121f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4 UNKNOWN
* e0db605d7821385d6517131fc27edc78924f121f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4138)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key to avoid delete failure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2705488722
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54b876b645fcd5a0df6277cd3934f51dc3b32616 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete operation on partitioned table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12929:
URL: https://github.com/apache/hudi/pull/12929#discussion_r1985807178
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -424,20 +424,28 @@ trait ProvidesHoodieConfig extends Logging {
HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS.key -> hiveSyncConfig.getStringOrDefault(HoodieSyncConfig.META_SYNC_PARTITION_EXTRACTOR_CLASS)
)
+ val overridingOpts = buildOverridingOpts(hoodieCatalogTable, tableConfig, path)
+ combineOptions(hoodieCatalogTable, tableConfig, sparkSession.sqlContext.conf,
+ defaultOpts = defaultOpts, overridingOpts = overridingOpts)
+ }
+
+ def buildOverridingOpts(hoodieCatalogTable: HoodieCatalogTable,
+ tableConfig: HoodieTableConfig,
+ path: String): Map[String, String] = {
val overridingOpts = Map(
"path" -> path,
- RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
TBL_NAME.key -> tableConfig.getTableName,
- DATABASE_NAME.key -> hoodieCatalogTable.table.database,
HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL,
PARTITIONPATH_FIELD.key -> getPartitionPathFieldWriteConfig(
tableConfig.getKeyGeneratorClassName, tableConfig.getPartitionFieldProp, hoodieCatalogTable)
)
-
- combineOptions(hoodieCatalogTable, tableConfig, sparkSession.sqlContext.conf,
- defaultOpts = defaultOpts, overridingOpts = overridingOpts)
+ if (hoodieCatalogTable.primaryKeys.isEmpty) {
Review Comment:
did we check if Update command is good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2711895521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
}, {
"hash" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20f1782305eea9c97fda955e159bd564e3d75ff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4119)
* 28a2a54d31bb9dcc82583eda18a0f3d9ea0c7bd4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2709148576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4092",
"triggerID" : "54b876b645fcd5a0df6277cd3934f51dc3b32616",
"triggerType" : "PUSH"
}, {
"hash" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109",
"triggerID" : "2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b",
"triggerType" : "PUSH"
}, {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2aef5f7e00b078cc9ea400e4e0ba862d817a1c5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4109)
* c20f1782305eea9c97fda955e159bd564e3d75ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9132] Avoid empty string row key for delete and update operations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12929:
URL: https://github.com/apache/hudi/pull/12929#issuecomment-2711840249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c20f1782305eea9c97fda955e159bd564e3d75ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20f1782305eea9c97fda955e159bd564e3d75ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org