You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "JoyJoyJo (via GitHub)" <gi...@apache.org> on 2025/02/01 08:21:40 UTC
[PR] [HUDI-8954]Reconstruct writer schema while clustering by spark row writer [hudi]
JoyJoyJo opened a new pull request, #12758:
URL: https://github.com/apache/hudi/pull/12758
### Change Logs
Reconstruct writer schema in HoodieDatasetBulkInsertHelper.bulkinsert while clustering as row.
### Impact
keep avro schema and parquet schema aligned
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [Y] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [Y] Change Logs and Impact were stated clearly
- [Y] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2628969622
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954]Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2628851351
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1499737ea7fb631fd0885419ed5ad612e8d9ee8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12758:
URL: https://github.com/apache/hudi/pull/12758#discussion_r1938714270
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java:
##########
@@ -80,7 +81,7 @@ public HoodieData<WriteStatus> performClusteringWithRecordsAsRow(Dataset<Row> in
Dataset<Row> repartitionedRecords = partitioner.repartitionRecords(inputRecords, numOutputGroups);
return HoodieDatasetBulkInsertHelper.bulkInsert(repartitionedRecords, instantTime, getHoodieTable(), newConfig,
- partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata);
+ partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata, WriteOperationType.CLUSTER);
Review Comment:
should we fix all the table services?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2628873307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
}, {
"hash" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336",
"triggerID" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1499737ea7fb631fd0885419ed5ad612e8d9ee8 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335)
* a9faa24733d1d54adbca6afd960a9f452d382f26 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12758:
URL: https://github.com/apache/hudi/pull/12758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2628905440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
}, {
"hash" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336",
"triggerID" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9faa24733d1d54adbca6afd960a9f452d382f26 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on code in PR #12758:
URL: https://github.com/apache/hudi/pull/12758#discussion_r1938909394
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java:
##########
@@ -80,7 +81,7 @@ public HoodieData<WriteStatus> performClusteringWithRecordsAsRow(Dataset<Row> in
Dataset<Row> repartitionedRecords = partitioner.repartitionRecords(inputRecords, numOutputGroups);
return HoodieDatasetBulkInsertHelper.bulkInsert(repartitionedRecords, instantTime, getHoodieTable(), newConfig,
- partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata);
+ partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata, WriteOperationType.CLUSTER);
Review Comment:
I think this just happens in clustering for cow table (enable row writer). Because spark uses a more permissive schema (all fields nullable) when reading records of older versions of parquet files.
The schemas used to read and write records in clustering (disable row writer or mor table) and other table service are aligned with avro schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954]Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2628852874
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
}, {
"hash" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1499737ea7fb631fd0885419ed5ad612e8d9ee8 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335)
* a9faa24733d1d54adbca6afd960a9f452d382f26 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954]Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2628852376
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
}, {
"hash" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1499737ea7fb631fd0885419ed5ad612e8d9ee8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335)
* a9faa24733d1d54adbca6afd960a9f452d382f26 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2630332179
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
}, {
"hash" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336",
"triggerID" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"triggerType" : "PUSH"
}, {
"hash" : "d5ce9b1de010ea03800b6999523ea7417adfa5f4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3345",
"triggerID" : "d5ce9b1de010ea03800b6999523ea7417adfa5f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5ce9b1de010ea03800b6999523ea7417adfa5f4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2630143492
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
}, {
"hash" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336",
"triggerID" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"triggerType" : "PUSH"
}, {
"hash" : "d5ce9b1de010ea03800b6999523ea7417adfa5f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5ce9b1de010ea03800b6999523ea7417adfa5f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9faa24733d1d54adbca6afd960a9f452d382f26 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336)
* d5ce9b1de010ea03800b6999523ea7417adfa5f4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2630145764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3335",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
}, {
"hash" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336",
"triggerID" : "a9faa24733d1d54adbca6afd960a9f452d382f26",
"triggerType" : "PUSH"
}, {
"hash" : "d5ce9b1de010ea03800b6999523ea7417adfa5f4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3345",
"triggerID" : "d5ce9b1de010ea03800b6999523ea7417adfa5f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9faa24733d1d54adbca6afd960a9f452d382f26 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3336)
* d5ce9b1de010ea03800b6999523ea7417adfa5f4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954] Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on code in PR #12758:
URL: https://github.com/apache/hudi/pull/12758#discussion_r1938909394
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java:
##########
@@ -80,7 +81,7 @@ public HoodieData<WriteStatus> performClusteringWithRecordsAsRow(Dataset<Row> in
Dataset<Row> repartitionedRecords = partitioner.repartitionRecords(inputRecords, numOutputGroups);
return HoodieDatasetBulkInsertHelper.bulkInsert(repartitionedRecords, instantTime, getHoodieTable(), newConfig,
- partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata);
+ partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata, WriteOperationType.CLUSTER);
Review Comment:
It does not happen in other table service except clustering as row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8954]Reconstruct writer schema while clustering by spark row writer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12758:
URL: https://github.com/apache/hudi/pull/12758#issuecomment-2628849849
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1499737ea7fb631fd0885419ed5ad612e8d9ee8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1499737ea7fb631fd0885419ed5ad612e8d9ee8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org