You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2024/12/04 17:01:50 UTC
[PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
jonvex opened a new pull request, #12422:
URL: https://github.com/apache/hudi/pull/12422
### Change Logs
bootstrap tables, virtual key tables, and tables with enums do not support partial updates currently
### Impact
prevent table from being in unusable state before the write occurs
### Risk level (write none, low medium or high below)
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870022992
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -432,6 +432,26 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
false
}
+ // validate that we can support partial updates
+ if (writePartialUpdates) {
+ if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
+ throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ }
+
+ if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
+ throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ }
+
+ if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
+ throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ }
+
+ if (parameters.getOrElse(DataSourceReadOptions.SCHEMA_EVOLUTION_ENABLED.key(),
Review Comment:
No, we need to check the value and that only has a method for string configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2518947467
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
}, {
"hash" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8a92daed26fd954797a197f7bdff0d86a2a174b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267)
* f6c2ef3e2119071cfabae7b5a20ed388d1db97ee UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2518949068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
}, {
"hash" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273",
"triggerID" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8a92daed26fd954797a197f7bdff0d86a2a174b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267)
* f6c2ef3e2119071cfabae7b5a20ed388d1db97ee Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870506887
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -435,20 +436,21 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
// validate that we can support partial updates
if (writePartialUpdates) {
if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
- throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for bootstrap tables.")
}
if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
- throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for virtual key tables.")
}
- if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
- throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ val tableSchemaResolver = new TableSchemaResolver(hoodieCatalogTable.metaClient)
Review Comment:
No. That is a spark schema, not an avro schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870126762
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestPartialUpdateForMergeInto.scala:
##########
@@ -66,6 +64,17 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
testPartialUpdateWithInserts("mor", "parquet")
}
+ test("Test partial update with schema on read enabled") {
Review Comment:
To test enum schema, I used TestHoodieFileGroupReaderOnHive to generate a table with an enum column. Then I copied the table to my tmp dir. Then I ran the spark commands:
```
sparkSession.sql("create table my_table using HUDI location '/tmp/myTable/'");
sparkSession.sql("select * from my_table").show(5,false);
sparkSession.sql("merge into my_table as target using (select '92b7d667-cc77-4462-9c05-fcd809ab3aec' as _row_key, 0.51 as begin_lat, 0 as timestamp) source on source._row_key = target._row_key when matched then update set begin_lat = source.begin_lat");
```
And this triggered the enum error:
<img width="1665" alt="image" src="https://github.com/user-attachments/assets/bec1a8de-75d7-426e-97db-f093bb28b1bc">
But I had to put in a fix to use table schema resolver to get the schema because fullSchema is just a spark schema converted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870138158
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestPartialUpdateForMergeInto.scala:
##########
@@ -66,6 +64,17 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
testPartialUpdateWithInserts("mor", "parquet")
}
+ test("Test partial update with schema on read enabled") {
Review Comment:
To test virtual keys I added
```
options.put(HoodieTableConfig.POPULATE_META_FIELDS.key(), "false");
```
to basicOptions() in TestBootstrapReadBase
Then I ran
```
sparkSession.sql("create table my_table using HUDI location '" + hudiBasePath + "'");
sparkSession.sql("select * from my_table").show(5,false);
sparkSession.sql("merge into my_table as target using (select '92b7d667-cc77-4462-9c05-fcd809ab3aec' as _row_key, 0.51 as begin_lat, 0 as timestamp) source on source._row_key = target._row_key when matched then update set begin_lat = source.begin_lat");
```
and this triggered the error:
<img width="1603" alt="image" src="https://github.com/user-attachments/assets/07fb92a6-6662-4c42-90a5-3a326ccc4abe">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870375965
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -435,20 +436,21 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
// validate that we can support partial updates
if (writePartialUpdates) {
if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
- throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for bootstrap tables.")
}
if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
- throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for virtual key tables.")
}
- if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
- throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ val tableSchemaResolver = new TableSchemaResolver(hoodieCatalogTable.metaClient)
Review Comment:
Is this because the `fullSchema` comes from the input data which may not contain all columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870026838
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestPartialUpdateForMergeInto.scala:
##########
@@ -66,6 +64,17 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
testPartialUpdateWithInserts("mor", "parquet")
}
+ test("Test partial update with schema on read enabled") {
Review Comment:
I didn't think it was worth it for CI time to test the other cases:
The first 2 just check a table config and we have a ton of confidence that table config works correctly.
The enum one should have the validation unit tested so I'll add that.
And the one I did test was spark config propagation which we are less confident on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870033806
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestPartialUpdateForMergeInto.scala:
##########
@@ -66,6 +64,17 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
testPartialUpdateWithInserts("mor", "parquet")
}
+ test("Test partial update with schema on read enabled") {
Review Comment:
OK, quickly test them locally to make sure there is no surprise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2519003437
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
}, {
"hash" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273",
"triggerID" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"triggerType" : "PUSH"
}, {
"hash" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279",
"triggerID" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f6c2ef3e2119071cfabae7b5a20ed388d1db97ee Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273)
* 8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2518929069
@yihua https://issues.apache.org/jira/browse/HUDI-8657. Previous linked ticket was the wrong one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2518668093
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8a92daed26fd954797a197f7bdff0d86a2a174b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870669576
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -432,6 +433,27 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
false
}
+ // validate that we can support partial updates
+ if (writePartialUpdates) {
+ if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
+ throw new HoodieNotSupportedException("Partial updates are not supported for bootstrap tables. " + partialUpdateFailureString)
+ }
+
+ if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
+ throw new HoodieNotSupportedException("Partial updates are not supported for virtual key tables. " + partialUpdateFailureString)
+ }
+
+ val tableSchemaResolver = new TableSchemaResolver(hoodieCatalogTable.metaClient)
+ if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(tableSchemaResolver.getTableAvroSchema)) {
+ throw new HoodieNotSupportedException("Partial updates are not supported for tables with enum columns. " + partialUpdateFailureString)
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870420590
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -435,20 +436,21 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
// validate that we can support partial updates
if (writePartialUpdates) {
if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
- throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for bootstrap tables.")
}
if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
- throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for virtual key tables.")
}
- if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
- throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ val tableSchemaResolver = new TableSchemaResolver(hoodieCatalogTable.metaClient)
Review Comment:
Let's see if we can avoid re-reading the table schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870009972
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1488,4 +1488,23 @@ public static Comparable<?> unwrapAvroValueWrapper(Object avroValueWrapper) {
}
}
+ public static boolean containsUnsupportedTypesForFileGroupReader(Schema schema) {
Review Comment:
Could you also copy the javadocs of the method from #12390?
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -432,6 +432,26 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
false
}
+ // validate that we can support partial updates
+ if (writePartialUpdates) {
+ if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
+ throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ }
+
+ if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
+ throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ }
+
+ if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
+ throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ }
+
+ if (parameters.getOrElse(DataSourceReadOptions.SCHEMA_EVOLUTION_ENABLED.key(),
Review Comment:
Could `SparkConfigUtils#containsConfigProperty` be used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870014011
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestPartialUpdateForMergeInto.scala:
##########
@@ -66,6 +64,17 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
testPartialUpdateWithInserts("mor", "parquet")
}
+ test("Test partial update with schema on read enabled") {
Review Comment:
Could you test other cases too? If it's not easy to add unit tests, test them locally through Spark Shell and report the results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2519224343
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
}, {
"hash" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273",
"triggerID" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"triggerType" : "PUSH"
}, {
"hash" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279",
"triggerID" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"triggerType" : "PUSH"
}, {
"hash" : "e4f46f6f867fbbad3f4d1b689f3237aa4dd55841",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2286",
"triggerID" : "e4f46f6f867fbbad3f4d1b689f3237aa4dd55841",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279)
* e4f46f6f867fbbad3f4d1b689f3237aa4dd55841 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2286)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2518927479
@yihua https://issues.apache.org/jira/browse/HUDI-8321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2519004611
The previous behavior before the partial updates feature was to require the user to use `update *` or the missing columns would be set to null. This was because PartialUpdateAvroPayload can be leveraged to simulate the desired behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2518670924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8a92daed26fd954797a197f7bdff0d86a2a174b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2519145826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
}, {
"hash" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273",
"triggerID" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"triggerType" : "PUSH"
}, {
"hash" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279",
"triggerID" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870608944
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -432,6 +433,27 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
false
}
+ // validate that we can support partial updates
+ if (writePartialUpdates) {
+ if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
+ throw new HoodieNotSupportedException("Partial updates are not supported for bootstrap tables. " + partialUpdateFailureString)
+ }
+
+ if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
+ throw new HoodieNotSupportedException("Partial updates are not supported for virtual key tables. " + partialUpdateFailureString)
+ }
+
+ val tableSchemaResolver = new TableSchemaResolver(hoodieCatalogTable.metaClient)
+ if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(tableSchemaResolver.getTableAvroSchema)) {
+ throw new HoodieNotSupportedException("Partial updates are not supported for tables with enum columns. " + partialUpdateFailureString)
Review Comment:
Let's remove this check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870083591
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestPartialUpdateForMergeInto.scala:
##########
@@ -66,6 +64,17 @@ class TestPartialUpdateForMergeInto extends HoodieSparkSqlTestBase {
testPartialUpdateWithInserts("mor", "parquet")
}
+ test("Test partial update with schema on read enabled") {
Review Comment:
In test bootstrap read I added on line 86
```
sparkSession.sql("create table my_table using HUDI location '" + bootstrapTargetPath + "'");
sparkSession.sql("select * from my_table").show(5,false);
sparkSession.sql("merge into my_table as target using (select '92b7d667-cc77-4462-9c05-fcd809ab3aec' as _row_key, 0.51 as begin_lat, 0 as timestamp) source on source._row_key = target._row_key when matched then update set begin_lat = source.begin_lat");
```
and this triggered the bootstrap error:
<img width="1625" alt="image" src="https://github.com/user-attachments/assets/cc74865c-bf9f-4a33-9680-b46fc70683d9">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870378956
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -435,20 +436,21 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
// validate that we can support partial updates
if (writePartialUpdates) {
if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
- throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for bootstrap tables.")
}
if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
- throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for virtual key tables.")
}
- if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
- throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ val tableSchemaResolver = new TableSchemaResolver(hoodieCatalogTable.metaClient)
Review Comment:
No, this is because full schema is from
```
private def getTableSchema: Schema = {
val (structName, nameSpace) = AvroConversionUtils
.getAvroRecordNameAndNamespace(hoodieCatalogTable.tableName)
AvroConversionUtils.convertStructTypeToAvroSchema(
new StructType(targetTableSchema), structName, nameSpace)
}
```
spark schema doesn't have enums, so when we convert to avro, the enum field a string field. So containsUnsupportedTypesForFileGroupReader will never return true. We need to get the table schema directly from avro
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2518818750
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8a92daed26fd954797a197f7bdff0d86a2a174b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in unsupported scenarios with partial updates in MIT [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #12422:
URL: https://github.com/apache/hudi/pull/12422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870003534
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -432,6 +432,26 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
false
}
+ // validate that we can support partial updates
+ if (writePartialUpdates) {
+ if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
+ throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
Review Comment:
Use `HoodieNotSupportedException` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2519222001
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
}, {
"hash" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273",
"triggerID" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"triggerType" : "PUSH"
}, {
"hash" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279",
"triggerID" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"triggerType" : "PUSH"
}, {
"hash" : "e4f46f6f867fbbad3f4d1b689f3237aa4dd55841",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e4f46f6f867fbbad3f4d1b689f3237aa4dd55841",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279)
* e4f46f6f867fbbad3f4d1b689f3237aa4dd55841 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870377249
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -432,6 +432,26 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
false
}
+ // validate that we can support partial updates
+ if (writePartialUpdates) {
+ if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
+ throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ }
+
+ if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
+ throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ }
+
+ if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
+ throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ }
+
+ if (parameters.getOrElse(DataSourceReadOptions.SCHEMA_EVOLUTION_ENABLED.key(),
Review Comment:
We can add a new util method. Not a blocking issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870380411
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -435,20 +436,21 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
// validate that we can support partial updates
if (writePartialUpdates) {
if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
- throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for bootstrap tables.")
}
if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
- throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for virtual key tables.")
}
- if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
- throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ val tableSchemaResolver = new TableSchemaResolver(hoodieCatalogTable.metaClient)
Review Comment:
Sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12422:
URL: https://github.com/apache/hudi/pull/12422#discussion_r1870415488
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -435,20 +436,21 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
// validate that we can support partial updates
if (writePartialUpdates) {
if (hoodieCatalogTable.tableConfig.getBootstrapBasePath.isPresent) {
- throw new UnsupportedOperationException("Partial updates are not supported for bootstrap tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for bootstrap tables.")
}
if (!hoodieCatalogTable.tableConfig.populateMetaFields()) {
- throw new UnsupportedOperationException("Partial updates are not supported for virtual key tables.")
+ throw new HoodieNotSupportedException("Partial updates are not supported for virtual key tables.")
}
- if (HoodieAvroUtils.containsUnsupportedTypesForFileGroupReader(fullSchema)) {
- throw new UnsupportedOperationException("Partial updates are not supported for tables with enum columns")
+ val tableSchemaResolver = new TableSchemaResolver(hoodieCatalogTable.metaClient)
Review Comment:
Could we use `hoodieCatalogTable.tableSchema`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2519002015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
}, {
"hash" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273",
"triggerID" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"triggerType" : "PUSH"
}, {
"hash" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8a92daed26fd954797a197f7bdff0d86a2a174b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267)
* f6c2ef3e2119071cfabae7b5a20ed388d1db97ee Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273)
* 8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8652] Throw exception in MIT if we try to write with partial updates but has unsupported features [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12422:
URL: https://github.com/apache/hudi/pull/12422#issuecomment-2519385503
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2267",
"triggerID" : "b8a92daed26fd954797a197f7bdff0d86a2a174b",
"triggerType" : "PUSH"
}, {
"hash" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2273",
"triggerID" : "f6c2ef3e2119071cfabae7b5a20ed388d1db97ee",
"triggerType" : "PUSH"
}, {
"hash" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2279",
"triggerID" : "8e6901a9acb47bbb48bbea5bd4d0a5647d829a2d",
"triggerType" : "PUSH"
}, {
"hash" : "e4f46f6f867fbbad3f4d1b689f3237aa4dd55841",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2286",
"triggerID" : "e4f46f6f867fbbad3f4d1b689f3237aa4dd55841",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4f46f6f867fbbad3f4d1b689f3237aa4dd55841 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2286)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org