You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2025/12/10 05:20:48 UTC
[PR] feat(schema): Port findNestedFieldSchema util for HoodieSchema [hudi]
voonhous opened a new pull request, #17546:
URL: https://github.com/apache/hudi/pull/17546
### Describe the issue this Pull Request addresses
Add `findNestedFieldSchema` helper for working with and manipulating the HoodieSchema objects.
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
Add a helper to `HoodieSchemaUtils` that mirrors the functionality of `AvroSchemaUtils` for `HoodieSchema` types for the function `findNestedFieldSchema`. Test cases are also ported over to `HoodieSchemaUtils`
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
None
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
Low
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Port AvroSchemaUtils util for HoodieSchema [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous closed pull request #17546: feat(schema): Port AvroSchemaUtils util for HoodieSchema
URL: https://github.com/apache/hudi/pull/17546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Port AvroSchemaUtils util for HoodieSchema [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17546:
URL: https://github.com/apache/hudi/pull/17546#discussion_r2607090333
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -573,4 +574,89 @@ public static HoodieSchema getRecordKeySchema() {
return RECORD_KEY_SCHEMA;
}
+ /**
+ * Finds the schema of a nested field given a dot-separated field name.
+ * This is equivalent to {@link AvroSchemaUtils#findNestedFieldSchema(Schema, String, boolean)} but operates on HoodieSchema.
+ *
+ * <p>Unlike {@link #findNestedField(HoodieSchema, String)}, this method returns just the schema
+ * of the leaf field (unwrapped from union if nullable), not the field with its parent lineage.
+ *
+ * @param schema the schema to search in
+ * @param fieldName the dot-separated field name (e.g., "user.profile.name")
+ * @param allowsMissingField if true, returns Option.empty() when field is not found;
+ * if false, throws HoodieSchemaException
+ * @return Option containing the schema of the nested field, or Option.empty() if not found and allowsMissingField is true
+ * @throws HoodieSchemaException if field is not found and allowsMissingField is false
+ * @since 1.2.0
+ */
+ public static Option<HoodieSchema> findNestedFieldSchema(HoodieSchema schema, String fieldName, boolean allowsMissingField) {
+ if (fieldName == null || fieldName.isEmpty()) {
+ return Option.empty();
+ }
+ String[] parts = fieldName.split("\\.");
+
+ HoodieSchema currentSchema = schema;
+ for (String part : parts) {
+ HoodieSchema nonNullSchema = getNonNullTypeFromUnion(currentSchema);
+ if (nonNullSchema.getType() != HoodieSchemaType.RECORD) {
+ if (allowsMissingField) {
+ return Option.empty();
+ }
+ throw new HoodieSchemaException(fieldName + " not a field in " + schema);
+ }
+ Option<HoodieSchemaField> foundFieldOpt = nonNullSchema.getField(part);
+ if (foundFieldOpt.isEmpty()) {
+ if (allowsMissingField) {
+ return Option.empty();
+ }
+ throw new HoodieSchemaException(fieldName + " not a field in " + schema);
+ }
+ currentSchema = foundFieldOpt.get().schema();
+ }
+ return Option.of(getNonNullTypeFromUnion(currentSchema));
+ }
+
+ /**
+ * Finds the type of a nested field given a dot-separated field name.
+ * This is equivalent to {@link AvroSchemaUtils#findNestedFieldType(Schema, String)} but operates on HoodieSchema.
+ *
+ * @param schema the schema to search in
+ * @param fieldName the dot-separated field name (e.g., "user.profile.name")
+ * @return Option containing the HoodieSchemaType of the nested field, or Option.empty() if not found
+ * @since 1.2.0
+ */
+ public static Option<HoodieSchemaType> findNestedFieldType(HoodieSchema schema, String fieldName) {
+ return findNestedFieldSchema(schema, fieldName, false).map(HoodieSchema::getType);
+ }
+
+ /**
+ * Returns true if the schema contains a field with the provided name.
+ * This is equivalent to {@link AvroSchemaUtils#containsFieldInSchema(Schema, String)} but operates on HoodieSchema.
+ *
+ * @param schema the schema to search in
+ * @param fieldName the field name to check for
+ * @return true if the field exists in the schema, false otherwise
+ * @since 1.2.0
+ */
+ public static boolean containsFieldInSchema(HoodieSchema schema, String fieldName) {
Review Comment:
My preference here is to reduce the number of static methods and just use `schema.getField` directly. It will be easier for developers to understand. All of these static methods will lead to more confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Port AvroSchemaUtils util for HoodieSchema [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17546:
URL: https://github.com/apache/hudi/pull/17546#discussion_r2607354104
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -573,4 +574,89 @@ public static HoodieSchema getRecordKeySchema() {
return RECORD_KEY_SCHEMA;
}
+ /**
+ * Finds the schema of a nested field given a dot-separated field name.
+ * This is equivalent to {@link AvroSchemaUtils#findNestedFieldSchema(Schema, String, boolean)} but operates on HoodieSchema.
+ *
+ * <p>Unlike {@link #findNestedField(HoodieSchema, String)}, this method returns just the schema
+ * of the leaf field (unwrapped from union if nullable), not the field with its parent lineage.
+ *
+ * @param schema the schema to search in
+ * @param fieldName the dot-separated field name (e.g., "user.profile.name")
+ * @param allowsMissingField if true, returns Option.empty() when field is not found;
+ * if false, throws HoodieSchemaException
+ * @return Option containing the schema of the nested field, or Option.empty() if not found and allowsMissingField is true
+ * @throws HoodieSchemaException if field is not found and allowsMissingField is false
+ * @since 1.2.0
+ */
+ public static Option<HoodieSchema> findNestedFieldSchema(HoodieSchema schema, String fieldName, boolean allowsMissingField) {
+ if (fieldName == null || fieldName.isEmpty()) {
+ return Option.empty();
+ }
+ String[] parts = fieldName.split("\\.");
+
+ HoodieSchema currentSchema = schema;
+ for (String part : parts) {
+ HoodieSchema nonNullSchema = getNonNullTypeFromUnion(currentSchema);
+ if (nonNullSchema.getType() != HoodieSchemaType.RECORD) {
+ if (allowsMissingField) {
+ return Option.empty();
+ }
+ throw new HoodieSchemaException(fieldName + " not a field in " + schema);
+ }
+ Option<HoodieSchemaField> foundFieldOpt = nonNullSchema.getField(part);
+ if (foundFieldOpt.isEmpty()) {
+ if (allowsMissingField) {
+ return Option.empty();
+ }
+ throw new HoodieSchemaException(fieldName + " not a field in " + schema);
+ }
+ currentSchema = foundFieldOpt.get().schema();
+ }
+ return Option.of(getNonNullTypeFromUnion(currentSchema));
+ }
+
+ /**
+ * Finds the type of a nested field given a dot-separated field name.
+ * This is equivalent to {@link AvroSchemaUtils#findNestedFieldType(Schema, String)} but operates on HoodieSchema.
+ *
+ * @param schema the schema to search in
+ * @param fieldName the dot-separated field name (e.g., "user.profile.name")
+ * @return Option containing the HoodieSchemaType of the nested field, or Option.empty() if not found
+ * @since 1.2.0
+ */
+ public static Option<HoodieSchemaType> findNestedFieldType(HoodieSchema schema, String fieldName) {
+ return findNestedFieldSchema(schema, fieldName, false).map(HoodieSchema::getType);
+ }
+
+ /**
+ * Returns true if the schema contains a field with the provided name.
+ * This is equivalent to {@link AvroSchemaUtils#containsFieldInSchema(Schema, String)} but operates on HoodieSchema.
+ *
+ * @param schema the schema to search in
+ * @param fieldName the field name to check for
+ * @return true if the field exists in the schema, false otherwise
+ * @since 1.2.0
+ */
+ public static boolean containsFieldInSchema(HoodieSchema schema, String fieldName) {
Review Comment:
Sure, will close this PR then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Port findNestedFieldSchema util for HoodieSchema [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17546:
URL: https://github.com/apache/hudi/pull/17546#issuecomment-3635444733
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3782063a5935ef0cc79418f4124c1e755e843261",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3782063a5935ef0cc79418f4124c1e755e843261",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3782063a5935ef0cc79418f4124c1e755e843261 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Port AvroSchemaUtils util for HoodieSchema [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17546:
URL: https://github.com/apache/hudi/pull/17546#discussion_r2607069709
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -573,4 +574,89 @@ public static HoodieSchema getRecordKeySchema() {
return RECORD_KEY_SCHEMA;
}
+ /**
+ * Finds the schema of a nested field given a dot-separated field name.
+ * This is equivalent to {@link AvroSchemaUtils#findNestedFieldSchema(Schema, String, boolean)} but operates on HoodieSchema.
+ *
+ * <p>Unlike {@link #findNestedField(HoodieSchema, String)}, this method returns just the schema
+ * of the leaf field (unwrapped from union if nullable), not the field with its parent lineage.
+ *
+ * @param schema the schema to search in
+ * @param fieldName the dot-separated field name (e.g., "user.profile.name")
+ * @param allowsMissingField if true, returns Option.empty() when field is not found;
+ * if false, throws HoodieSchemaException
+ * @return Option containing the schema of the nested field, or Option.empty() if not found and allowsMissingField is true
+ * @throws HoodieSchemaException if field is not found and allowsMissingField is false
+ * @since 1.2.0
+ */
+ public static Option<HoodieSchema> findNestedFieldSchema(HoodieSchema schema, String fieldName, boolean allowsMissingField) {
Review Comment:
This can simply call `findNestedField` and then return the schema to reduce the code duplication
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org