You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2025/01/08 02:06:19 UTC
[PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
cshuo opened a new pull request, #12598:
URL: https://github.com/apache/hudi/pull/12598
…ring
### Change Logs
In Flink SQL, primary key constraint can be defined, and the type of pk field become non-null，while spark has no primary key constraint, so they have discrepancies in the schema of the underlying files, e.g., parquet.
we can make a schema reconciliation during the async clustering reading.
### Impact
Consolidating flink clustering by reconcile schemas to tolerate different nullabilities of record key.
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1908102489
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java:
##########
@@ -591,4 +596,24 @@ private static void findAnyMissingFieldsRec(Schema incomingSchema,
public static String createSchemaErrorString(String errorMessage, Schema writerSchema, Schema tableSchema) {
return String.format("%s\nwriterSchema: %s\ntableSchema: %s", errorMessage, writerSchema, tableSchema);
}
+
+ /**
+ * Create a new schema by change the nullabilities of the given columns.
+ *
+ * @param schema original schema
+ * @param nullableUpdateCols columns that would be updated with given nullability
+ * @param nullable nullability of column type
+ * @return a new schema with the nullabilities of the given columns updated
+ */
+ public static Schema createSchemaWithNullabilityUpdate(
Review Comment:
Rename to `forceNullableColumns`, we should also avoid to recreate the schema if the field is already nullable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2589432788
If some fields have NOT NULL constraint added, should these fields also be converted to nullable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2579193540
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8886f4b1e6c514512f4a4b583a27b9d395feab9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722)
* d4aa17d628f132b9617a37dd6f1ecac68fd3af61 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2576563170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4ee02941422419b50107aea3a7e24b5c6740661 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2592660400
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
}, {
"hash" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759",
"triggerID" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"triggerType" : "PUSH"
}, {
"hash" : "aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2870",
"triggerID" : "aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a77e69e63f0770d7176b1da335ec6f07cfa814b1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759)
* aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2870)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2579274784
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4aa17d628f132b9617a37dd6f1ecac68fd3af61 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2579517665
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
}, {
"hash" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759",
"triggerID" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4aa17d628f132b9617a37dd6f1ecac68fd3af61 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751)
* a77e69e63f0770d7176b1da335ec6f07cfa814b1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1908311946
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java:
##########
@@ -605,14 +605,16 @@ public static String createSchemaErrorString(String errorMessage, Schema writerS
* @param nullable nullability of column type
* @return a new schema with the nullabilities of the given columns updated
*/
- public static Schema createSchemaWithNullabilityUpdate(
+ public static Schema forceNullableColumns(
Schema schema, List<String> nullableUpdateCols, boolean nullable) {
Review Comment:
nullableUpdateCols -> columns. We can eliminate the flag `nullalbe` because it is always true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2577313983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e588b2ba9a27751150d2b424531825f182d30ad Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714)
* f8886f4b1e6c514512f4a4b583a27b9d395feab9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2576568567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4ee02941422419b50107aea3a7e24b5c6740661 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713)
* 8e588b2ba9a27751150d2b424531825f182d30ad UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1906439350
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringOperator.java:
##########
@@ -344,6 +349,27 @@ private Iterator<RowData> readRecordsForGroupBaseFiles(List<ClusteringOperation>
return new ConcatenatingIterator<>(iteratorsForPartition);
}
+ /**
+ * Since there exists discrepancies between flink and spark dealing with nullability of primary key field,
+ * and there may be some files written by spark, we reconcile read schema against write schema to promote
+ * the nullability, so that schema validating would not fail.
+ *
+ * @param clusteringOperation the cluster operation
+ * @return schema that has nullability constraints reconciled
+ */
+ private Schema reconcileSchemaWithNullability(ClusteringOperation clusteringOperation) {
+ String instantTs = StringUtils.isNullOrEmpty(clusteringOperation.getDataFilePath())
+ ? FSUtils.getCommitTime(clusteringOperation.getDeltaFilePaths().get(0))
Review Comment:
No, the `TableSchemaResolver` would trigger file lising of the commit metadata which is a pressure to filesystem. Fetch the schema from file is straight-forward:
```
private Schema fetchSchemaFromFiles(Iterator<String> filePaths) throws IOException {
Schema schema = null;
while (filePaths.hasNext() && schema == null) {
StoragePath filePath = new StoragePath(filePaths.next());
if (FSUtils.isLogFile(filePath)) {
// this is a log file
schema = readSchemaFromLogFile(filePath);
} else {
schema = HoodieIOFactory.getIOFactory(metaClient.getStorage())
.getFileFormatUtils(filePath).readAvroSchema(metaClient.getStorage(), filePath);
}
}
return schema;
}
```
Another way is to force make the record key fields nullable if they are not if a nullable schema can be used to read non-nullable schema data file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2576714890
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e588b2ba9a27751150d2b424531825f182d30ad Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2576564719
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4ee02941422419b50107aea3a7e24b5c6740661 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1906336062
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringOperator.java:
##########
@@ -344,6 +349,27 @@ private Iterator<RowData> readRecordsForGroupBaseFiles(List<ClusteringOperation>
return new ConcatenatingIterator<>(iteratorsForPartition);
}
+ /**
+ * Since there exists discrepancies between flink and spark dealing with nullability of primary key field,
+ * and there may be some files written by spark, we reconcile read schema against write schema to promote
+ * the nullability, so that schema validating would not fail.
+ *
+ * @param clusteringOperation the cluster operation
+ * @return schema that has nullability constraints reconciled
+ */
+ private Schema reconcileSchemaWithNullability(ClusteringOperation clusteringOperation) {
+ String instantTs = StringUtils.isNullOrEmpty(clusteringOperation.getDataFilePath())
+ ? FSUtils.getCommitTime(clusteringOperation.getDeltaFilePaths().get(0))
Review Comment:
As for where to get the file schema, actually, files with same commit time share the same write schema, and getTableAvroSchema in `TableSchemaResolver` is more efficient since there is a cache, what do you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2591531840
In the issue #12561, all fields (include pk and not null fields) are nullable when parquet file was clustered by spark row writer. If we just focus on nullability of pk field, I am worried that flink will still not be able to perform clustering on those files that have been clustered by spark row writer in the presence of NOT NULL constraints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2592883927
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
}, {
"hash" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759",
"triggerID" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"triggerType" : "PUSH"
}, {
"hash" : "aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2870",
"triggerID" : "aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2870)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2591429593
> If some fields have NOT NULL constraint added, should these fields also be converted to nullable?
spark also supports NOT NULL constraint, not like primary key constraint, in which flink and spark have discrepancy. So I think we may first focus on nullability of pk field first in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2576621713
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4ee02941422419b50107aea3a7e24b5c6740661 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713)
* 8e588b2ba9a27751150d2b424531825f182d30ad Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2592655308
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
}, {
"hash" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759",
"triggerID" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"triggerType" : "PUSH"
}, {
"hash" : "aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a77e69e63f0770d7176b1da335ec6f07cfa814b1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759)
* aa8a6ad4bb3fa56d278b8d8c4600b2f2d963b553 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1906260009
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringOperator.java:
##########
@@ -344,6 +349,27 @@ private Iterator<RowData> readRecordsForGroupBaseFiles(List<ClusteringOperation>
return new ConcatenatingIterator<>(iteratorsForPartition);
}
+ /**
+ * Since there exists discrepancies between flink and spark dealing with nullability of primary key field,
+ * and there may be some files written by spark, we reconcile read schema against write schema to promote
+ * the nullability, so that schema validating would not fail.
+ *
+ * @param clusteringOperation the cluster operation
+ * @return schema that has nullability constraints reconciled
+ */
+ private Schema reconcileSchemaWithNullability(ClusteringOperation clusteringOperation) {
+ String instantTs = StringUtils.isNullOrEmpty(clusteringOperation.getDataFilePath())
+ ? FSUtils.getCommitTime(clusteringOperation.getDeltaFilePaths().get(0))
Review Comment:
Currently, the Flink clustering only work on append only table, all the data files should be in parquet format, we can fetch the record key fields, then the file schema from the parquet footer, and reconcile the record key fields if the nullability discrepency exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1908102789
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringOperator.java:
##########
@@ -344,6 +347,22 @@ private Iterator<RowData> readRecordsForGroupBaseFiles(List<ClusteringOperation>
return new ConcatenatingIterator<>(iteratorsForPartition);
}
+ /**
+ * Since there exists discrepancies between flink and spark dealing with nullability of primary key field,
+ * and there may be some files written by spark, we force update the nullability of record key to nullable,
+ * so that schema validating would not fail.
+ *
+ * @param schema the original schema to be updated
+ * @return schema that has nullability constraints reconciled
+ */
+ private Schema reconcileSchemaWithRecordKeyNullability(Schema schema) {
+ Option<String[]> recordKeyOp = table.getMetaClient().getTableConfig().getRecordKeyFields();
Review Comment:
The default record key field is `uuid`, we should check the validity first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2579195164
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8886f4b1e6c514512f4a4b583a27b9d395feab9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722)
* d4aa17d628f132b9617a37dd6f1ecac68fd3af61 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2576567218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4ee02941422419b50107aea3a7e24b5c6740661 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713)
* 8e588b2ba9a27751150d2b424531825f182d30ad UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1906496811
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringOperator.java:
##########
@@ -344,6 +349,27 @@ private Iterator<RowData> readRecordsForGroupBaseFiles(List<ClusteringOperation>
return new ConcatenatingIterator<>(iteratorsForPartition);
}
+ /**
+ * Since there exists discrepancies between flink and spark dealing with nullability of primary key field,
+ * and there may be some files written by spark, we reconcile read schema against write schema to promote
+ * the nullability, so that schema validating would not fail.
+ *
+ * @param clusteringOperation the cluster operation
+ * @return schema that has nullability constraints reconciled
+ */
+ private Schema reconcileSchemaWithNullability(ClusteringOperation clusteringOperation) {
+ String instantTs = StringUtils.isNullOrEmpty(clusteringOperation.getDataFilePath())
+ ? FSUtils.getCommitTime(clusteringOperation.getDeltaFilePaths().get(0))
Review Comment:
The second way sounds more efficient, I'll verify it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2579697225
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
}, {
"hash" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759",
"triggerID" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a77e69e63f0770d7176b1da335ec6f07cfa814b1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2577317308
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e588b2ba9a27751150d2b424531825f182d30ad Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714)
* f8886f4b1e6c514512f4a4b583a27b9d395feab9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2577451737
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8886f4b1e6c514512f4a4b583a27b9d395feab9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2579510282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2713",
"triggerID" : "e4ee02941422419b50107aea3a7e24b5c6740661",
"triggerType" : "PUSH"
}, {
"hash" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2714",
"triggerID" : "8e588b2ba9a27751150d2b424531825f182d30ad",
"triggerType" : "PUSH"
}, {
"hash" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2722",
"triggerID" : "f8886f4b1e6c514512f4a4b583a27b9d395feab9",
"triggerType" : "PUSH"
}, {
"hash" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751",
"triggerID" : "d4aa17d628f132b9617a37dd6f1ecac68fd3af61",
"triggerType" : "PUSH"
}, {
"hash" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a77e69e63f0770d7176b1da335ec6f07cfa814b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4aa17d628f132b9617a37dd6f1ecac68fd3af61 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2751)
* a77e69e63f0770d7176b1da335ec6f07cfa814b1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2592249379
> used to read non-nullable schema data file, I'm ok to force all non-null fields being nullable for cluster reading. cc
+1 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12598:
URL: https://github.com/apache/hudi/pull/12598#issuecomment-2591558781
> In the issue #12561, all fields (include pk and not null fields) are nullable when parquet file was clustered by spark row writer. If we just focus on nullability of pk field, I am worried that flink will still not be able to perform clustering on those files that have been clustered by spark row writer in the presence of NOT NULL constraints.
Checked the #12561 again, the scenario you mentioned is a problem. As we have already checked nullable schema can be used to read non-nullable schema data file, I'm ok to force all non-null fields being nullable for cluster reading. cc @danny0405 WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12598:
URL: https://github.com/apache/hudi/pull/12598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8841] Fix schema validating exception during flink async cluste… [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12598:
URL: https://github.com/apache/hudi/pull/12598#discussion_r1906961811
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringOperator.java:
##########
@@ -344,6 +349,27 @@ private Iterator<RowData> readRecordsForGroupBaseFiles(List<ClusteringOperation>
return new ConcatenatingIterator<>(iteratorsForPartition);
}
+ /**
+ * Since there exists discrepancies between flink and spark dealing with nullability of primary key field,
+ * and there may be some files written by spark, we reconcile read schema against write schema to promote
+ * the nullability, so that schema validating would not fail.
+ *
+ * @param clusteringOperation the cluster operation
+ * @return schema that has nullability constraints reconciled
+ */
+ private Schema reconcileSchemaWithNullability(ClusteringOperation clusteringOperation) {
+ String instantTs = StringUtils.isNullOrEmpty(clusteringOperation.getDataFilePath())
+ ? FSUtils.getCommitTime(clusteringOperation.getDeltaFilePaths().get(0))
Review Comment:
cc @danny0405 updated using the second way, and added a test to verify nullable schema can be used to read non-nullable schema data file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org