You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2025/03/12 11:14:21 UTC
[PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
TheR1sing3un opened a new pull request, #12961:
URL: https://github.com/apache/hudi/pull/12961
For the record we need to delete, we only need to read the `hoodie_meta_fields`, `record_keys` and the columns involved in the delete condition from the table, which can greatly reduce the amount of read data when deleting.
> benchmark in our production
- 1000 columns
- execute `delete from table where col-a = 'xxx'`
- 50,000,000 records per partition
> before optimized
<img width="2508" alt="image" src="https://github.com/user-attachments/assets/79b1a53b-3ff4-4fdc-bc35-ee8989fb20b5" />
> after optimized
<img width="2524" alt="image" src="https://github.com/user-attachments/assets/350af7ee-38cf-477e-800b-7dfb6f16517c" />
In our wide table scenario, column pruning will greatly reduce the amount of data scanning and shuffle overhead in the entire process.
### Change Logs
1. introduce schema pruning for delete record
### Impact
improve delete performance especially tables with more columns
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1992803821
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -74,4 +88,13 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
Seq.empty[Row]
}
+
+ def tryPruningDeleteRecordSchema(query: LogicalPlan, requiredColNames: Seq[String]): LogicalPlan = {
+ val filteredOutput = query.output.filter(attr => isMetaField(attr.name) || requiredColNames.contains(attr.name))
+ if (filteredOutput == query.output) {
+ query
+ } else {
+ Project(filteredOutput, query)
+ }
Review Comment:
Can you elaborate what kind of `InternalRow` this project could produce?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1992882320
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -74,4 +88,13 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
Seq.empty[Row]
}
+
+ def tryPruningDeleteRecordSchema(query: LogicalPlan, requiredColNames: Seq[String]): LogicalPlan = {
+ val filteredOutput = query.output.filter(attr => isMetaField(attr.name) || requiredColNames.contains(attr.name))
+ if (filteredOutput == query.output) {
+ query
+ } else {
+ Project(filteredOutput, query)
+ }
Review Comment:
Does the schema change after the projection?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1996096804
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -41,9 +43,28 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
val condition = sparkAdapter.extractDeleteCondition(dft)
+ val config = if (sparkSession.sqlContext.conf.getConfString(SPARK_SQL_OPTIMIZED_WRITES.key()
+ , SPARK_SQL_OPTIMIZED_WRITES.defaultValue()) == "true") {
+ buildHoodieDeleteTableConfig(catalogTable, sparkSession) + (SPARK_SQL_WRITES_PREPPED_KEY -> "true")
+ } else {
+ buildHoodieDeleteTableConfig(catalogTable, sparkSession)
+ }
+
+ val recordKeysStr = config.getOrElse(HoodieTableConfig.RECORDKEY_FIELDS.key(), "")
+ val recordKeys = recordKeysStr.split(",").filter(_.nonEmpty)
+
+ // get all columns which are used in condition
+ val conditionColumns = if (condition == null) {
+ Seq.empty[String]
+ } else {
+ condition.references.map(_.name).toSeq
+ }
+
+ val requiredCols = recordKeys ++ conditionColumns
Review Comment:
Do we need the partition path data column(s) here too (or `_hoodie_partition_path` suffices)? Have you verified that deletes work properly on a partitioned table?
Another nuances is, Spark `DELETE` statement does not use precombine field so it is fine to ignore the field now. However, if we want to support streaming deletes, i.e., based on event time in precombine field, the precombine field needs to be added in this case.
Would be good to add javadocs/comments to clarify these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1996461612
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -41,9 +43,28 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
val condition = sparkAdapter.extractDeleteCondition(dft)
+ val config = if (sparkSession.sqlContext.conf.getConfString(SPARK_SQL_OPTIMIZED_WRITES.key()
+ , SPARK_SQL_OPTIMIZED_WRITES.defaultValue()) == "true") {
+ buildHoodieDeleteTableConfig(catalogTable, sparkSession) + (SPARK_SQL_WRITES_PREPPED_KEY -> "true")
+ } else {
+ buildHoodieDeleteTableConfig(catalogTable, sparkSession)
+ }
+
+ val recordKeysStr = config.getOrElse(HoodieTableConfig.RECORDKEY_FIELDS.key(), "")
+ val recordKeys = recordKeysStr.split(",").filter(_.nonEmpty)
+
+ // get all columns which are used in condition
+ val conditionColumns = if (condition == null) {
+ Seq.empty[String]
+ } else {
+ condition.references.map(_.name).toSeq
+ }
+
+ val requiredCols = recordKeys ++ conditionColumns
Review Comment:
in TestDeleteTable.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2717518297
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66b520af14d34fd5f292ce5f77d51ccbdabbc0d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2719683538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
}, {
"hash" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183",
"triggerID" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"triggerType" : "PUSH"
}, {
"hash" : "3cf98d7e7127adb78fbf4943b98df1a472c44c33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cf98d7e7127adb78fbf4943b98df1a472c44c33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66b520af14d34fd5f292ce5f77d51ccbdabbc0d2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170)
* 3bd707504ef02f98fc3b53e0a3867ac8b9542072 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183)
* 3cf98d7e7127adb78fbf4943b98df1a472c44c33 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2717811816
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66b520af14d34fd5f292ce5f77d51ccbdabbc0d2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2719641908
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
}, {
"hash" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66b520af14d34fd5f292ce5f77d51ccbdabbc0d2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170)
* 3bd707504ef02f98fc3b53e0a3867ac8b9542072 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1992831553
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -74,4 +88,13 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
Seq.empty[Row]
}
+
+ def tryPruningDeleteRecordSchema(query: LogicalPlan, requiredColNames: Seq[String]): LogicalPlan = {
+ val filteredOutput = query.output.filter(attr => isMetaField(attr.name) || requiredColNames.contains(attr.name))
+ if (filteredOutput == query.output) {
+ query
+ } else {
+ Project(filteredOutput, query)
+ }
Review Comment:
> Can you elaborate what kind of `InternalRow` this project could produce?
`GenericRowWithSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2717519082
@danny0405 @yihua Hi, Danny, Ethan, looking forward to your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1993052432
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -74,4 +88,13 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
Seq.empty[Row]
}
+
+ def tryPruningDeleteRecordSchema(query: LogicalPlan, requiredColNames: Seq[String]): LogicalPlan = {
+ val filteredOutput = query.output.filter(attr => isMetaField(attr.name) || requiredColNames.contains(attr.name))
+ if (filteredOutput == query.output) {
+ query
+ } else {
+ Project(filteredOutput, query)
+ }
Review Comment:
Does the delete records handler follow this contract then? Do they use the right schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1992904044
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -74,4 +88,13 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
Seq.empty[Row]
}
+
+ def tryPruningDeleteRecordSchema(query: LogicalPlan, requiredColNames: Seq[String]): LogicalPlan = {
+ val filteredOutput = query.output.filter(attr => isMetaField(attr.name) || requiredColNames.contains(attr.name))
+ if (filteredOutput == query.output) {
+ query
+ } else {
+ Project(filteredOutput, query)
+ }
Review Comment:
> Does the schema change after the projection?
This ends up generating only the record with schema in `project`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12961:
URL: https://github.com/apache/hudi/pull/12961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2719951482
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
}, {
"hash" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183",
"triggerID" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"triggerType" : "PUSH"
}, {
"hash" : "3cf98d7e7127adb78fbf4943b98df1a472c44c33",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4185",
"triggerID" : "3cf98d7e7127adb78fbf4943b98df1a472c44c33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3cf98d7e7127adb78fbf4943b98df1a472c44c33 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1993108496
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -74,4 +88,13 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
Seq.empty[Row]
}
+
+ def tryPruningDeleteRecordSchema(query: LogicalPlan, requiredColNames: Seq[String]): LogicalPlan = {
+ val filteredOutput = query.output.filter(attr => isMetaField(attr.name) || requiredColNames.contains(attr.name))
+ if (filteredOutput == query.output) {
+ query
+ } else {
+ Project(filteredOutput, query)
+ }
Review Comment:
> Does the delete records handler follow this contract then? Do they use the right schema?
In subsequent processing, there is no relationship with schema.
<img width="1464" alt="image" src="https://github.com/user-attachments/assets/ec35c03c-3a9e-44db-bc49-d1fca8fe1d15" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1992573658
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -735,6 +735,15 @@ object DataSourceWriteOptions {
.withDocumentation("Controls whether overwrite use dynamic or static mode, if not configured, " +
"respect spark.sql.sources.partitionOverwriteMode")
+ val DELETE_RECORD_SCHEMA_PRUNING: ConfigProperty[String] = ConfigProperty
+ .key("hoodie.datasource.delete.record.schema.pruning.enabled")
+ .defaultValue("true")
Review Comment:
> Can we eliminate this option because it looks like a pure optimization.
Of course we can. Done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2719692131
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
}, {
"hash" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183",
"triggerID" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"triggerType" : "PUSH"
}, {
"hash" : "3cf98d7e7127adb78fbf4943b98df1a472c44c33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cf98d7e7127adb78fbf4943b98df1a472c44c33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3bd707504ef02f98fc3b53e0a3867ac8b9542072 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183)
* 3cf98d7e7127adb78fbf4943b98df1a472c44c33 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1992508548
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -735,6 +735,15 @@ object DataSourceWriteOptions {
.withDocumentation("Controls whether overwrite use dynamic or static mode, if not configured, " +
"respect spark.sql.sources.partitionOverwriteMode")
+ val DELETE_RECORD_SCHEMA_PRUNING: ConfigProperty[String] = ConfigProperty
+ .key("hoodie.datasource.delete.record.schema.pruning.enabled")
+ .defaultValue("true")
Review Comment:
Can we eliminate this option because it looks like a pure optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2719643650
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
}, {
"hash" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183",
"triggerID" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66b520af14d34fd5f292ce5f77d51ccbdabbc0d2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170)
* 3bd707504ef02f98fc3b53e0a3867ac8b9542072 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2719749092
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
}, {
"hash" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183",
"triggerID" : "3bd707504ef02f98fc3b53e0a3867ac8b9542072",
"triggerType" : "PUSH"
}, {
"hash" : "3cf98d7e7127adb78fbf4943b98df1a472c44c33",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4185",
"triggerID" : "3cf98d7e7127adb78fbf4943b98df1a472c44c33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3bd707504ef02f98fc3b53e0a3867ac8b9542072 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4183)
* 3cf98d7e7127adb78fbf4943b98df1a472c44c33 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12961:
URL: https://github.com/apache/hudi/pull/12961#issuecomment-2717521756
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170",
"triggerID" : "66b520af14d34fd5f292ce5f77d51ccbdabbc0d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 66b520af14d34fd5f292ce5f77d51ccbdabbc0d2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9166] Introduce schema pruning for delete-record [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12961:
URL: https://github.com/apache/hudi/pull/12961#discussion_r1996097709
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala:
##########
@@ -41,9 +43,28 @@ case class DeleteHoodieTableCommand(dft: DeleteFromTable) extends HoodieLeafRunn
val condition = sparkAdapter.extractDeleteCondition(dft)
+ val config = if (sparkSession.sqlContext.conf.getConfString(SPARK_SQL_OPTIMIZED_WRITES.key()
+ , SPARK_SQL_OPTIMIZED_WRITES.defaultValue()) == "true") {
+ buildHoodieDeleteTableConfig(catalogTable, sparkSession) + (SPARK_SQL_WRITES_PREPPED_KEY -> "true")
+ } else {
+ buildHoodieDeleteTableConfig(catalogTable, sparkSession)
+ }
+
+ val recordKeysStr = config.getOrElse(HoodieTableConfig.RECORDKEY_FIELDS.key(), "")
+ val recordKeys = recordKeysStr.split(",").filter(_.nonEmpty)
+
+ // get all columns which are used in condition
+ val conditionColumns = if (condition == null) {
+ Seq.empty[String]
+ } else {
+ condition.references.map(_.name).toSeq
+ }
+
+ val requiredCols = recordKeys ++ conditionColumns
Review Comment:
Do existing tests cover these cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org